Chapter 18: Hybrid Queries

Datalevin's indexes are most useful when they work together. Because Datalog, full-text search, vector similarity, text embedding search, document indexes, and the KV store can all live in the same storage system, you can combine them in a single query.

By default, full-text, vector, embedding, and idoc indexes are updated synchronously with source datoms. Full-text, vector, and embedding domains can opt into :indexing-mode :async; in that case source datoms commit with durable index jobs and index queries are eventually consistent until the worker catches up.

In Datalevin, indexes are programmable capabilities. This chapter demonstrates how to build hybrid queries that bridge structured logic and unstructured search.

Unless stated otherwise, snippets that call fulltext with an attribute argument, such as :doc/content, assume that attribute has :db/fulltext true and :db.fulltext/autoDomain true.

1. Unified Retrieval: The Single-Engine Advantage

In Datalevin, a hybrid query is a single Datalog expression. The engine manages joins between different index types automatically.

This single-engine model has consequences beyond convenience:

• Consistency. In the default synchronous mode, specialized indexes are updated in the same transaction as the source datoms, so a query over one database snapshot does not combine a full-text index from one committed state with a vector index from another. If a domain uses async indexing, source datoms may be visible before its secondary-index job has caught up.
• No dual writes. A Datalevin transaction records the source datoms and, in synchronous mode, the secondary-index updates. In async mode it records the source datoms and durable index jobs. You do not run change-data-capture pipelines or reconciliation jobs to keep separate search and vector stores in step with the system of record.
• No fan-out. Results are joined inside Datalevin over local indexes rather than serialized to several services and merged afterward. The query joins on entity ids instead of doing an application-level merge.
• One operational surface. One database environment to back up and restore, one access-control boundary, and one failure domain, instead of several systems that can fail or drift independently.

2. Index Return Formats

All specialized index functions return datoms for consistent destructuring. By default, they all return [e a v]. Different options return additional values.

The shared entity id (?e) enables seamless joins across all index types, as depicted in Figure 18.1.

The implication is that the entity id is a universal join key. In a multi-store architecture, each system returns identifiers in its own space, such as Lucene document ids, vector-store ids, or SQL primary keys, and the application maintains translation tables to line them up. In Datalevin, every index returns datoms over the same entity-id space, so joining a full-text hit to a vector hit to a graph fact is just unification on ?e. A practical consequence is that adding a new index type to an existing model, such as embeddings over text you already store, can compose with existing facts without building an id-mapping layer first.

Full-text scores and vector distances are different quantities. A full-text score is a relevance score where higher is better. A vector distance is a metric distance where lower is usually closer. They are not on the same scale, so production code should not combine the raw numbers directly.

3. Evidence Fusion

When a query combines full-text search and vector similarity, there are two different problems you might be solving.

The first problem is intersection: find entities that satisfy every retrieval condition. A single Datalog query with shared ?e variables does exactly that. If a full-text clause and a vector clause both bind ?e, the result contains entities that appear in both candidate sets.

The second problem is fusion: accept candidates from several ranked lists and combine their ranks into one final ranking. This is useful when either keyword evidence or semantic evidence can be enough to make an item relevant. Reciprocal rank fusion (RRF) is a simple, robust way to do this without pretending that full-text scores and vector distances share a numeric scale [1].

3.1 Reciprocal Rank Fusion

RRF fuses ranked lists by rank position. A common formula is 1.0 / (60 + rank), where rank is 1 for the first item, 2 for the second item, and so on. The constant 60 dampens the effect of any one list. The same entity can appear in the full-text list, vector list, and graph list; add its reciprocal-rank contributions from each list, then sort by the fused score.

The practical pattern is:

1. Run each retrieval lens with enough :top headroom: full-text, vector, embedding, idoc, or graph expansion.
2. Convert each result list to (entity-id, rank, source) records.
3. Sum 1 / (60 + rank) per entity id.
4. Sort by the fused score.
5. Hydrate or filter the fused entity ids with ordinary Datalog facts: permissions, recency, status, ownership, and business rules.

Use the same entity id as the fusion key. This is where Datalevin's return format matters: every index returns datoms in the same entity-id space, so there is no external id-mapping table to maintain.

(defn rrf-score [rank]
  (/ 1.0 (+ 60 rank)))

(defn ranked [source hits]
  (map-indexed
    (fn [idx hit]
      (assoc hit
             :source source
             :rank   (inc idx)
             :rrf    (rrf-score (inc idx))))
    hits))

(defn rrf-fuse [& source->hits]
  (->> source->hits
       (mapcat (fn [[source hits]] (ranked source hits)))
       (group-by :e)
       (map (fn [[e hits]]
              {:e         e
               :rrf-score (reduce + (map :rrf hits))
               :sources   (set (map :source hits))}))
       (sort-by :rrf-score >)))

(rrf-fuse
  [:text   [{:e 10} {:e 20} {:e 30}]]
  [:vector [{:e 20} {:e 40} {:e 10}]])

static double rrfScore(int rank) {
    return 1.0 / (60.0 + rank);
}

Map<Object, Double> scores = new HashMap<>();

BiConsumer<String, List<Map<String, Object>>> addList =
    (source, hits) -> {
        for (int i = 0; i < hits.size(); i++) {
            Object e = hits.get(i).get("e");
            scores.merge(e, rrfScore(i + 1), Double::sum);
        }
    };

addList.accept("text", textHits);
addList.accept("vector", vectorHits);

List<Object> fused =
    scores.entrySet().stream()
        .sorted(Map.Entry.<Object, Double>comparingByValue().reversed())
        .map(Map.Entry::getKey)
        .toList();

def rrf_score(rank):
    return 1.0 / (60 + rank)


def add_list(scores, hits):
    for index, hit in enumerate(hits, start=1):
        e = hit["e"]
        scores[e] = scores.get(e, 0.0) + rrf_score(index)


scores = {}
add_list(scores, text_hits)
add_list(scores, vector_hits)

fused = [
    e for e, score in sorted(
        scores.items(),
        key=lambda item: item[1],
        reverse=True,
    )
]

function rrfScore(rank) {
  return 1.0 / (60 + rank);
}

function addList(scores, hits) {
  hits.forEach((hit, index) => {
    const e = hit.e;
    scores.set(e, (scores.get(e) ?? 0) + rrfScore(index + 1));
  });
}

const scores = new Map();
addList(scores, textHits);
addList(scores, vectorHits);

const fused = [...scores.entries()]
  .sort((a, b) => b[1] - a[1])
  .map(([e]) => e);

3.2 Datalog-Native Evidence Fusion

The application-side RRF example above is useful when you want literal rank-position fusion over lists returned to application code. There is also a fully Datalog-native alternative: model each retrieval lens as evidence, assign each kind of evidence a weight, and aggregate those weights inside the query.

This is not exactly RRF unless rank positions are already represented as data. Datalog relations are unordered, so a query does not derive "the third result in the vector list" as a rank fact by itself. But for many retrieval pipelines, weighted evidence fusion is the right in-query solution: keyword evidence, semantic evidence, tag evidence, permissions, and business rules all remain in one Datalog query.

(def fusion-rules
  '[[(evidence ?e ?weight ?source ?query ?query-vec ?query-tags)
     [(fulltext $ :doc/content ?query {:top 50}) [[?e _ _]]]
     [(ground 1.0) ?weight]
     [(ground :text) ?source]]

    [(evidence ?e ?weight ?source ?query ?query-vec ?query-tags)
     [(vec-neighbors $ :doc/embedding ?query-vec {:top 50}) [[?e _ _]]]
     [(ground 1.0) ?weight]
     [(ground :vector) ?source]]

    [(evidence ?e ?weight ?source ?query ?query-vec ?query-tags)
     [?e :doc/tag ?tag]
     [(contains? ?query-tags ?tag)]
     [(ground 0.25) ?weight]
     [(ground :tag) ?source]]])

(d/q '[:find ?e ?title (sum ?weight)
       :with ?source
       :in $ % ?query ?query-vec ?query-tags ?allowed-owner-set
       :where
       (evidence ?e ?weight ?source ?query ?query-vec ?query-tags)
       [?e :doc/title ?title]
       [?e :doc/owner ?owner]
       [(contains? ?allowed-owner-set ?owner)]
       :order-by [2 :desc 1 :asc]]
     db
     fusion-rules
     user-query
     query-embedding
     query-tags
     allowed-owner-set)

Object fusionRules = Datalevin.edn("["
    + "[(evidence ?e ?weight ?source ?query ?query-vec ?query-tags) "
    + " [(fulltext $ :doc/content ?query {:top 50}) [[?e _ _]]] "
    + " [(ground 1.0) ?weight] "
    + " [(ground :text) ?source]] "
    + "[(evidence ?e ?weight ?source ?query ?query-vec ?query-tags) "
    + " [(vec-neighbors $ :doc/embedding ?query-vec {:top 50}) [[?e _ _]]] "
    + " [(ground 1.0) ?weight] "
    + " [(ground :vector) ?source]] "
    + "[(evidence ?e ?weight ?source ?query ?query-vec ?query-tags) "
    + " [?e :doc/tag ?tag] "
    + " [(contains? ?query-tags ?tag)] "
    + " [(ground 0.25) ?weight] "
    + " [(ground :tag) ?source]]]");

conn.query("[:find ?e ?title (sum ?weight) " +
           " :with ?source " +
           " :in $ % ?query ?query-vec ?query-tags ?allowed-owner-set " +
           " :where " +
           " (evidence ?e ?weight ?source ?query ?query-vec ?query-tags) " +
           " [?e :doc/title ?title] " +
           " [?e :doc/owner ?owner] " +
           " [(contains? ?allowed-owner-set ?owner)] " +
           " :order-by [2 :desc 1 :asc]]",
           fusionRules,
           userQuery,
           queryEmbedding,
           queryTags,
           allowedOwnerSet);

fusion_rules = interop().read_edn(
    '[[ (evidence ?e ?weight ?source ?query ?query-vec ?query-tags) '
    '   [(fulltext $ :doc/content ?query {:top 50}) [[?e _ _]]] '
    '   [(ground 1.0) ?weight] '
    '   [(ground :text) ?source]] '
    ' [ (evidence ?e ?weight ?source ?query ?query-vec ?query-tags) '
    '   [(vec-neighbors $ :doc/embedding ?query-vec {:top 50}) [[?e _ _]]] '
    '   [(ground 1.0) ?weight] '
    '   [(ground :vector) ?source]] '
    ' [ (evidence ?e ?weight ?source ?query ?query-vec ?query-tags) '
    '   [?e :doc/tag ?tag] '
    '   [(contains? ?query-tags ?tag)] '
    '   [(ground 0.25) ?weight] '
    '   [(ground :tag) ?source]]]'
)

conn.query(
    '[:find ?e ?title (sum ?weight) '
    ' :with ?source '
    ' :in $ % ?query ?query-vec ?query-tags ?allowed-owner-set '
    ' :where '
    ' (evidence ?e ?weight ?source ?query ?query-vec ?query-tags) '
    ' [?e :doc/title ?title] '
    ' [?e :doc/owner ?owner] '
    ' [(contains? ?allowed-owner-set ?owner)] '
    ' :order-by [2 :desc 1 :asc]]',
    fusion_rules,
    user_query,
    query_embedding,
    query_tags,
    allowed_owner_set,
)

const fusionRules = await interop().readEdn(
  '[[ (evidence ?e ?weight ?source ?query ?query-vec ?query-tags) ' +
  '   [(fulltext $ :doc/content ?query {:top 50}) [[?e _ _]]] ' +
  '   [(ground 1.0) ?weight] ' +
  '   [(ground :text) ?source]] ' +
  ' [ (evidence ?e ?weight ?source ?query ?query-vec ?query-tags) ' +
  '   [(vec-neighbors $ :doc/embedding ?query-vec {:top 50}) [[?e _ _]]] ' +
  '   [(ground 1.0) ?weight] ' +
  '   [(ground :vector) ?source]] ' +
  ' [ (evidence ?e ?weight ?source ?query ?query-vec ?query-tags) ' +
  '   [?e :doc/tag ?tag] ' +
  '   [(contains? ?query-tags ?tag)] ' +
  '   [(ground 0.25) ?weight] ' +
  '   [(ground :tag) ?source]]]'
);

await conn.query(
  '[:find ?e ?title (sum ?weight) ' +
  ' :with ?source ' +
  ' :in $ % ?query ?query-vec ?query-tags ?allowed-owner-set ' +
  ' :where ' +
  ' (evidence ?e ?weight ?source ?query ?query-vec ?query-tags) ' +
  ' [?e :doc/title ?title] ' +
  ' [?e :doc/owner ?owner] ' +
  ' [(contains? ?allowed-owner-set ?owner)] ' +
  ' :order-by [2 :desc 1 :asc]]',
  fusionRules,
  userQuery,
  queryEmbedding,
  queryTags,
  allowedOwnerSet
);

The :with ?source clause is important. It keeps one row per evidence source before aggregation, so text evidence and vector evidence for the same entity are both counted even when they have the same numeric weight. The :order-by clause uses result column 2, the aggregate score, to rank entities by total evidence.

The ground clauses in the rules introduce literal values such as the evidence weight and source label. Each branch mentions only the inputs it actually uses. The outer query binds ?query, ?query-vec, and ?query-tags before calling evidence, so those bound rule-call arguments are carried through even when a branch does not inspect them. The rule head itself gives all evidence definitions the same arity.

Literal rank-position RRF is usually an application-side fusion step unless ranks have already been represented as data. Datalog-native weighted fusion is the better fit when the fusion policy itself should be queryable data: rules produce evidence, aggregation sums it, and strict filters remain ordinary Datalog clauses. If a condition must be mandatory, keep it as a Datalog clause rather than treating it as just another evidence source.

4. Order of Execution: How the Optimizer Thinks

When you mix different index types, the query optimizer uses cost-based analysis for ordinary Datalog clauses and joins the result tuples produced by specialized index functions.

A candidate set is the intermediate set of entities produced by one clause or index function. Specialized functions such as fulltext, vec-neighbors, embedding-neighbors, and idoc-match create candidate sets from their own indexes. Ordinary Datalog clauses then join, filter, or extend those candidates.

• Selective Retrieval First: Keep :top, domains, and index-specific filters tight so specialized functions produce useful candidate sets.
• Join Order: All index results are sets of entity ids. The join is an intersection of these ids, an extremely fast operation in a memory-mapped architecture.

You usually do not need to reorder clauses manually. State constraints clearly, then tune the candidate-producing options when a search/vector function returns too much or too little.

The deeper point is to think in terms of selectivity. Let the specialized function that produces the smallest, most relevant candidate set drive the query, and let cheap Datalog clauses prune what remains. In retrieval-first queries, cost is usually dominated by the size of the candidate sets that search functions return, not just by the number of clauses. That also makes :top a recall-versus-cost knob: set it too low and a genuine match can be dropped before the structured filters ever see it; set it too high and downstream joins do needless work. When in doubt, widen :top enough that the real answer is reliably in the candidate set, then rely on the structured clauses to enforce correctness.

5. Mixing KV and Datalog

KV and Datalog storage do not have to live in separate files or separate services. A Datalevin Datalog store is built on the same DLMDB substrate as the key-value APIs, so application-owned KV data can live beside Datalog data in the same database environment. Use separate DBIs for those KV datasets when they need their own key/value encoding, range access pattern, or lifecycle, while Datalog keeps managing its own internal DBIs.

For performance-sensitive lookups, combine raw key-value layer access with Datalog using function bindings or descriptor-backed runtime UDFs. The example below registers :kv/check-blacklist, a small deterministic function that reads the relevant same-store KV DBI.

This pattern is useful when a check is naturally key-addressed and operational: deny lists, rollout flags, short-lived suppression tables, or availability markers that are maintained outside the main entity model. Keep durable domain facts in Datalog. Use the KV DBI for the compact lookup table, then let the query apply that table as one more filter.

The underlying KV layer is a first-class capability, allowing custom logic and specialized indexes. At application startup, register the UDF against the same registry used to open the connection. The registered function can close over the backing KV handle:

(require '[datalevin.core :as d]
         '[datalevin.udf :as udf])

(def blacklist-fn
  {:udf/lang :app
   :udf/kind :query-fn
   :udf/id   :kv/check-blacklist})

(def registry (udf/create-registry))

(def conn
  (d/get-conn "/tmp/app" schema
    {:runtime-opts {:udf-registry registry}}))

(let [kv (d/datalog-kv conn)]
  (d/open-dbi kv "blacklisted-users")
  (udf/register! registry blacklist-fn
    (fn [user-id]
      (boolean
        (d/get-value kv "blacklisted-users" user-id :string :data)))))

import datalevin.Connection;
import datalevin.Datalevin;
import datalevin.KV;
import datalevin.KVType;
import datalevin.UdfRegistry;

import java.util.Map;

UdfRegistry registry = Datalevin.udfRegistry();
var blacklistFn = Datalevin.queryUdf(":kv/check-blacklist")
    .lang("app")
    .build();

Connection conn = Datalevin.createConn(
    "/tmp/app",
    schema,
    Map.of(":runtime-opts", Map.of(":udf-registry", registry)));

KV kv = conn.datalogKV();
kv.openDbi("blacklisted-users");

registry.register(blacklistFn, args -> {
    String userId = (String) args.get(0);
    return kv.getValue("blacklisted-users",
                       userId,
                       KVType.STRING,
                       KVType.DATA,
                       true) != null;
});

from datalevin import connect, create_udf_registry, udf_descriptor

registry = create_udf_registry()
blacklist_fn = udf_descriptor(":kv/check-blacklist", lang=":app")

conn = connect(
    "/tmp/app",
    schema=schema,
    opts={":runtime-opts": {":udf-registry": registry}},
)

kv = conn.datalog_kv()
kv.open_dbi("blacklisted-users")

registry.register(
    blacklist_fn,
    lambda user_id: kv.get_value(
        "blacklisted-users", user_id, ":string", ":data", True
    ) is not None,
)

import {
  connect,
  createUdfRegistry,
  udfDescriptor
} from "datalevin-node";

const registry = await createUdfRegistry();
const blacklistFn = udfDescriptor(":kv/check-blacklist", { lang: ":app" });

const conn = await connect("/tmp/app", {
  schema,
  opts: { ":runtime-opts": { ":udf-registry": registry } }
});

const kv = await conn.datalogKv();
await kv.openDbi("blacklisted-users");

await registry.register(blacklistFn, async (userId) =>
  (await kv.getValue("blacklisted-users", userId, {
    kType: ":string",
    vType: ":data",
    ignoreKey: true
  })) !== null
);

Once registered, the descriptor can be passed to queries and called through the portable udf form:

(d/q '[:find ?e
       :in $ ?blacklist-fn
       :where [?e :user/id ?id]
              [(udf ?blacklist-fn ?id) ?blocked?]
              [(false? ?blocked?)]]
     db blacklist-fn)

conn.query("[:find ?e " +
           " :in $ ?blacklist-fn " +
           " :where [?e :user/id ?id] " +
           "        [(udf ?blacklist-fn ?id) ?blocked?] " +
           "        [(false? ?blocked?)]]",
           blacklistFn);

conn.query(
    '[:find ?e '
    ' :in $ ?blacklist-fn '
    ' :where [?e :user/id ?id] '
    '        [(udf ?blacklist-fn ?id) ?blocked?] '
    '        [(false? ?blocked?)]]',
    blacklist_fn,
)

await conn.query(
  '[:find ?e ' +
  ' :in $ ?blacklist-fn ' +
  ' :where [?e :user/id ?id] ' +
  '        [(udf ?blacklist-fn ?id) ?blocked?] ' +
  '        [(false? ?blocked?)]]',
  blacklistFn
);

The Datalog query uses a function binding through the portable udf form: the function receives a value from the query and returns another value that the query can bind. The callable receives only the arguments that appear after the descriptor in the udf clause. In the query above, it receives ?id; it does not receive db as an implicit first argument. Descriptor-backed UDFs are the portable shape for server deployments and code that should be configured through an application registry.

Keep query UDFs deterministic and cheap. If a function performs network calls, writes state, or scans a large KV range, it can make query behavior surprising and hard to optimize.

When a write must update Datalog datoms and application-owned KV DBIs atomically, use the with-transaction pattern from Chapter 6.

6. Practical Pattern: Hybrid Retrieval with Structured Filters

The database pattern behind Retrieval-Augmented Generation (RAG) is the same pattern this chapter has been building toward: use full-text and vector indexes to produce candidates, then join those candidates with ordinary Datalog facts for permission, recency, ownership, status, and business rules.

The important database point is that the filter should run in the same query as retrieval. That avoids a separate step where the system fetches candidates and then has to remember to filter them before handing context to another component. In a fan-out architecture, filters often run after results are merged from several services, which is where access-control leaks and stale data can slip in. In a single Datalevin query, an entity that fails a permission or policy clause is not part of the result, no matter which index surfaced it.

Chapter 25 returns to this shape in the agent-memory setting. There, the problem is no longer just candidate retrieval; it is recall and context assembly: choosing the right lenses, ranking evidence, preserving provenance, and formatting a small packet for an LLM. This chapter stays at the database level: how to make heterogeneous indexes participate in one planned query.

7. End-to-End Example: The "Smart" Product Search

After the retrieval pieces are clear, a larger query can combine all of them. This example finds products that:

1. Match the term "clojure" in the description (full-text)
2. Are semantically similar to a user's interest vector or query text
3. Are in stock and in a specific category (Datalog)
4. Have specific nested metadata (idoc)
5. Pass a fast operational policy check stored in a KV DBI (KV)

This is the intersection pattern from Section 3: every retrieval and structured condition must bind the same ?e. Use fusion when either textual or semantic evidence should be enough to admit a candidate.

This example assumes :product/desc has :db/fulltext true and :db.fulltext/autoDomain true, :product/embedding is a vector attribute, :product/metadata is an idoc attribute, and the KV-backed policy check is registered as the descriptor-backed query UDF :kv/sellable-sku?, using the pattern from Section 5.

Chapter 14 introduced idoc-match: it searches inside a path-indexed document value. In this example it filters products whose nested metadata says {:discounted true} without requiring that nested field to become a separate Datalog attribute.

(def sellable-fn
  {:udf/lang :app
   :udf/kind :query-fn
   :udf/id   :kv/sellable-sku?})

(d/q '[:find ?title ?dist
       :in $ ?search-term ?target-vec ?category ?sellable-fn
       :where
       ;; full-text search
       [(fulltext $ :product/desc ?search-term) [[?e _ _]]]
       ;; vector search with distance
       [(vec-neighbors $ :product/embedding ?target-vec
                       {:top 20 :display :refs+dists}) [[?e _ _ ?dist]]]
       ;; structured filters
       [?e :product/status :product.status/in-stock]
       [?e :product/category ?category]
       [?e :product/sku ?sku]
       [?e :product/title ?title]
       ;; KV-backed policy check
       [(udf ?sellable-fn ?sku) ?sellable?]
       [(true? ?sellable?)]
       ;; idoc metadata filter
       [(idoc-match $ :product/metadata {:discounted true}) [[?e _ _]]]]
     db "clojure" user-embedding :books sellable-fn)

var sellableFn = Datalevin.queryUdf(":kv/sellable-sku?")
    .lang("app")
    .build();

conn.query("[:find ?title ?dist " +
           " :in $ ?search-term ?target-vec ?category ?sellable-fn " +
           " :where " +
           " [(fulltext $ :product/desc ?search-term) [[?e _ _]]] " +
           " [(vec-neighbors $ :product/embedding ?target-vec " +
           "                 {:top 20 :display :refs+dists}) [[?e _ _ ?dist]]] " +
           " [?e :product/status :product.status/in-stock] " +
           " [?e :product/category ?category] " +
           " [?e :product/sku ?sku] " +
           " [?e :product/title ?title] " +
           " [(udf ?sellable-fn ?sku) ?sellable?] " +
           " [(true? ?sellable?)] " +
           " [(idoc-match $ :product/metadata {:discounted true}) [[?e _ _]]]]",
           "clojure", userEmbedding, Datalevin.kw(":books"), sellableFn);

from datalevin import udf_descriptor

sellable_fn = udf_descriptor(":kv/sellable-sku?", lang=":app")

conn.query(
    '[:find ?title ?dist '
    ' :in $ ?search-term ?target-vec ?category ?sellable-fn '
    ' :where '
    ' [(fulltext $ :product/desc ?search-term) [[?e _ _]]] '
    ' [(vec-neighbors $ :product/embedding ?target-vec '
    '                 {:top 20 :display :refs+dists}) [[?e _ _ ?dist]]] '
    ' [?e :product/status :product.status/in-stock] '
    ' [?e :product/category ?category] '
    ' [?e :product/sku ?sku] '
    ' [?e :product/title ?title] '
    ' [(udf ?sellable-fn ?sku) ?sellable?] '
    ' [(true? ?sellable?)] '
    ' [(idoc-match $ :product/metadata {:discounted true}) [[?e _ _]]]]',
    "clojure",
    user_embedding,
    ":books",
    sellable_fn,
)

import { udfDescriptor } from "datalevin-node";

const sellableFn = udfDescriptor(":kv/sellable-sku?", { lang: ":app" });

await conn.query(
  '[:find ?title ?dist ' +
  ' :in $ ?search-term ?target-vec ?category ?sellable-fn ' +
  ' :where ' +
  ' [(fulltext $ :product/desc ?search-term) [[?e _ _]]] ' +
  ' [(vec-neighbors $ :product/embedding ?target-vec ' +
  '                 {:top 20 :display :refs+dists}) [[?e _ _ ?dist]]] ' +
  ' [?e :product/status :product.status/in-stock] ' +
  ' [?e :product/category ?category] ' +
  ' [?e :product/sku ?sku] ' +
  ' [?e :product/title ?title] ' +
  ' [(udf ?sellable-fn ?sku) ?sellable?] ' +
  ' [(true? ?sellable?)] ' +
  ' [(idoc-match $ :product/metadata {:discounted true}) [[?e _ _]]]]',
  'clojure',
  userEmbedding,
  ':books',
  sellableFn
);

The shared ?e variable joins all results together. The _ placeholders discard returned attribute and value positions that this query does not need. The ?sellable-fn input names ordinary application code that can read a small KV DBI, for example a same-file operational table of suppressed SKUs, real-time availability flags, or rollout policy. Register it with the UDF registry used to open the connection, as in Section 5.

Summary

Datalevin's index model is about composition. Full-text, vector, embedding, idoc, graph, relational, and KV-backed checks can participate in one Datalog query because they all return or constrain the same entity-id space.

• Universal join key: Specialized indexes return datoms keyed by entity id, so a full-text hit, vector neighbor, idoc match, and ordinary fact can meet on ?e without an application-side id mapping table.
• Intersection vs. fusion: Use a single query with shared variables when all conditions must hold. Use reciprocal-rank fusion or weighted Datalog evidence when several retrieval lenses should contribute evidence and any one lens may be enough to surface a candidate.
• Planner-friendly composition: Specialized indexes produce bounded candidate sets; ordinary Datalog clauses then join, filter, and hydrate those candidates in the same engine.
• Operational composition: Descriptor-backed UDFs let query-time logic read small same-store KV DBIs without giving up the portable server-safe UDF model.
• Consistency: Indexes are synchronous by default, with explicit async modes for expensive secondary indexing when ingestion throughput matters more than immediate index visibility.

Treating every index as a programmable capability lets Datalevin answer queries that would otherwise require several services and an application-level merge step.

References

[1] Gordon V. Cormack, Charles L. A. Clarke, and Stefan Buettcher, "Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods," Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '09), 2009. DOI: https://doi.org/10.1145/1571941.1572114.