DatalevinDatalevin
Log in
Part III - Modeling Across Paradigms

Chapter 14: Document Modeling

Datalevin gives you two ways to model document-shaped data: logical documents, built from normal datoms and component references, and indexed documents (idoc), stored as single values with path-level indexes. By default an idoc indexes every scalar leaf path, and schema or domain options can restrict indexing to selected path prefixes.

Figure 14.1 contrasts the two representations.

Logical documents vs. idoc, modeling the same nested user data: the logical document (:db/isComponent) decomposes it into a user entity that owns separate profile and contact component entities plus tag datoms - many facts that join, pull, and update independently; the native idoc (:db.type/idoc) stores the whole nested map as one datom value on :user/metadata with a path index over configured leaf paths, all leaf paths by default, such as profile.name, profile.age, contact.email, and tags. The two models can coexist on the same entity

Use logical documents when the nested structure is part of your core domain model. Use idoc when the nested structure is flexible metadata, imported JSON, Markdown-derived structure, or application-defined data that should be searched by path without forcing every field into the schema. Section 6 shows a boundary technique for querying nested data without storing it at all.

1. Logical Documents with :db/isComponent

The most common way to model durable domain documents in Datalevin is to use component attributes. A component reference stores a nested entity as ordinary datoms, but treats that entity as owned by its parent.

Components require single ownership

Use component attributes for single-owner logical documents. If a nested record can be shared by more than one parent, model it with ordinary references or as an idoc value instead.

1.1 Example: A Nested Blog Post

;; Schema: :post/comments owns comment entities.
{:post/slug     {:db/valueType :db.type/string
                 :db/unique    :db.unique/identity}
 :post/comments {:db/valueType   :db.type/ref
                 :db/cardinality :db.cardinality/many
                 :db/isComponent true}
 :user/email    {:db/valueType :db.type/string
                 :db/unique    :db.unique/identity}
 :comment/author {:db/valueType :db.type/ref}}

Schema schema = Datalevin.schema()
    .attr(":post/slug",
          Schema.attribute()
              .valueType(Schema.ValueType.STRING)
              .unique(Schema.Unique.IDENTITY))
    .attr(":post/comments",
          Schema.attribute()
              .valueType(Schema.ValueType.REF)
              .cardinality(Schema.Cardinality.MANY)
              .isComponent(true))
    .attr(":user/email",
          Schema.attribute()
              .valueType(Schema.ValueType.STRING)
              .unique(Schema.Unique.IDENTITY))
    .attr(":comment/author",
          Schema.attribute()
              .valueType(Schema.ValueType.REF));

schema = {
    ":post/slug": {
        ":db/valueType": ":db.type/string",
        ":db/unique": ":db.unique/identity"
    },
    ":post/comments": {
        ":db/valueType": ":db.type/ref",
        ":db/cardinality": ":db.cardinality/many",
        ":db/isComponent": True
    },
    ":user/email": {
        ":db/valueType": ":db.type/string",
        ":db/unique": ":db.unique/identity"
    },
    ":comment/author": {
        ":db/valueType": ":db.type/ref"
    }
}

const schema = {
  ":post/slug": {
    ":db/valueType": ":db.type/string",
    ":db/unique": ":db.unique/identity"
  },
  ":post/comments": {
    ":db/valueType": ":db.type/ref",
    ":db/cardinality": ":db.cardinality/many",
    ":db/isComponent": true
  },
  ":user/email": {
    ":db/valueType": ":db.type/string",
    ":db/unique": ":db.unique/identity"
  },
  ":comment/author": {
    ":db/valueType": ":db.type/ref"
  }
};

Logical documents are still normal facts. Every nested field can be typed, indexed, joined, pulled, and updated independently. This is useful when nested data has identity, relationships, constraints, or a lifecycle that should remain visible to Datalog.

For example, comments can be modeled as components of a post while their authors remain separate top-level entities, addressed through a unique email lookup ref:

(d/transact! conn
  [{:user/name "Ada"
    :user/email "ada@example.com"}
   {:post/slug "indexes-as-capabilities"
    :post/title "Indexes as Capabilities"
    :post/comments [{:comment/body "This clarifies the mental model."
                     :comment/author [:user/email "ada@example.com"]}]}])

Object ada = List.of(":user/email", "ada@example.com");

conn.transact(Datalevin.tx()
    .entity(Tx.entity()
        .put(":user/name", "Ada")
        .put(":user/email", "ada@example.com"))
    .entity(Tx.entity()
        .put(":post/slug", "indexes-as-capabilities")
        .put(":post/title", "Indexes as Capabilities")
        .put(":post/comments", List.of(
            Tx.entity()
                .put(":comment/body", "This clarifies the mental model.")
                .put(":comment/author", ada)
                .build()))));

conn.transact([
    {":user/name": "Ada",
     ":user/email": "ada@example.com"},
    {":post/slug": "indexes-as-capabilities",
     ":post/title": "Indexes as Capabilities",
     ":post/comments": [
         {":comment/body": "This clarifies the mental model.",
          ":comment/author": [":user/email", "ada@example.com"]}]}
])

await conn.transact([
  {
    ":user/name": "Ada",
    ":user/email": "ada@example.com"
  },
  {
    ":post/slug": "indexes-as-capabilities",
    ":post/title": "Indexes as Capabilities",
    ":post/comments": [
      {
        ":comment/body": "This clarifies the mental model.",
        ":comment/author": [":user/email", "ada@example.com"]
      }
    ]
  }
]);

For logical documents, use pull patterns to navigate nested paths:

(d/pull db
        [:post/title {:post/comments [:comment/body {:comment/author [:user/name]}]}]
        [:post/slug "indexes-as-capabilities"])

conn.pull(
    "[:post/title {:post/comments [:comment/body {:comment/author [:user/name]}]}]",
    List.of(":post/slug", "indexes-as-capabilities"));

conn.pull(
    [":post/title", {":post/comments": [":comment/body",
                                        {":comment/author": [":user/name"]}]}],
    [":post/slug", "indexes-as-capabilities"])

await conn.pull(
  [':post/title', { ':post/comments': [':comment/body',
                                       { ':comment/author': [':user/name'] }] }],
  [':post/slug', 'indexes-as-capabilities']
);

2. Native Indexed Documents with idoc

An idoc stores a nested document as one datom value, while Datalevin maintains an index for paths inside the document, as shown in Figure 14.2. This gives you document-database-style path queries while still using Datalog joins and transactions.

Indexed documents: a nested indexed-document value is flattened to a path-to-value index whose indexed leaf paths are queried with idoc-match and idoc-get

idoc indexes are orthogonal to full-text and vector indexes. The same entity can have structured facts, full-text fields, embeddings, and idoc metadata, and a single Datalog query can combine all of those constraints.

2.1 Defining an idoc Attribute

Declare an idoc attribute with :db/valueType :db.type/idoc. The optional :db/idocFormat controls how payloads are interpreted, the optional :db/domain gives the backing index a stable domain name, and the optional :db.idoc/indexedPaths / :db.idoc/excludedPaths keys control which path prefixes are indexed. Section 2.3 explains path selectors.

(def schema
  {:user/metadata {:db/valueType :db.type/idoc
                   :db/domain    "user_metadata"
                   :db.idoc/indexedPaths [:theme :profile :tags :contact]
                   :db.idoc/excludedPaths [[:profile :secret]]}
   :user/raw-json {:db/valueType  :db.type/idoc
                   :db/idocFormat :json}
   :doc/title     {:db/valueType :db.type/string}
   :doc/markdown  {:db/valueType  :db.type/idoc
                   :db/idocFormat :markdown}})

Schema schema = Datalevin.schema()
    .attr(":user/metadata",
          Schema.attribute()
              .valueType(Schema.ValueType.IDOC)
              .prop(":db/domain", "user_metadata")
              .idocIndexedPaths("theme", "profile", "tags", "contact")
              .idocExcludedPaths(
                  List.of("profile", "secret")))
    .attr(":user/raw-json",
          Schema.attribute()
              .valueType(Schema.ValueType.IDOC)
              .prop(":db/idocFormat", Datalevin.kw(":json")))
    .attr(":doc/title",
          Schema.attribute()
              .valueType(Schema.ValueType.STRING))
    .attr(":doc/markdown",
          Schema.attribute()
              .valueType(Schema.ValueType.IDOC)
              .prop(":db/idocFormat", Datalevin.kw(":markdown")));

from datalevin import idoc_attr

schema = {
    ":user/metadata": idoc_attr(
        domain="user_metadata",
        indexed_paths=["theme", "profile", "tags", "contact"],
        excluded_paths=[["profile", "secret"]]),
    ":user/raw-json": idoc_attr(format=":json"),
    ":doc/title": {
        ":db/valueType": ":db.type/string"
    },
    ":doc/markdown": idoc_attr(format=":markdown")
}

import { idocAttr } from "datalevin-node";

const schema = {
  ":user/metadata": idocAttr({
    domain: "user_metadata",
    indexedPaths: ["theme", "profile", "tags", "contact"],
    excludedPaths: [["profile", "secret"]]
  }),
  ":user/raw-json": idocAttr({ format: ":json" }),
  ":doc/title": {
    ":db/valueType": ":db.type/string"
  },
  ":doc/markdown": idocAttr({ format: ":markdown" })
};

Supported :db/idocFormat values are :edn (the default), :json, and :markdown. If :db/domain is omitted, Datalevin derives a domain from the attribute name without the leading colon; for example, :user/metadata becomes "user/metadata". If no path controls are supplied, Datalevin indexes all scalar leaf paths in the document.

2.2 idoc Domains

An idoc domain names the backing path index for one idoc attribute. If :db/domain is omitted, Datalevin derives the domain from the attribute name. Most schemas can treat this as an internal index name and query documents by attribute.

This is related to, but not the same as, full-text, vector, or embedding domains. Those indexes use domain lists or store-level domain maps to configure retrieval and indexing behavior, and some support default or automatic domains (Chapters 16 and 17).

idoc has no :db.idoc/domains list or autoDomain flag. An idoc attribute has one idoc domain, selected by :db/domain or derived from the attribute name. Several idoc attributes may intentionally share the same domain, but that is an index design choice, not a ranking or retrieval policy.

Store-level :idoc-opts and :idoc-domains can configure which paths a domain indexes (Section 2.3), but they do not make one attribute participate in multiple domains. Markdown idoc values normalize heading text into path segments, so they should usually live in their own domains instead of sharing a domain with EDN or JSON idoc values. Section 3 focuses on attribute-scoped idoc-match queries; Chapter 31 lists the domain-scoped and full-DB forms.

2.3 Selective Path Indexing

By default, an idoc domain indexes every scalar leaf path in each document. For large or user-defined documents, this can index more paths than the application needs. Selective path indexing lets you keep only the path prefixes your queries use.

Path selectors may be a top-level keyword, a top-level string, or a vector of keyword/string path segments. They are prefix selectors: :profile includes leaf paths such as [:profile :age] and [:profile :name]. :db.idoc/indexedPaths is the include list for an attribute's domain. When it is absent, all paths are included. :db.idoc/excludedPaths is the omit list and wins over included paths.

The original document is still stored as the datom value. Selective indexing only controls which leaf paths enter the path dictionary and inverted index. A query cannot use idoc-match to find documents by an omitted path, but once a document has been found by another indexed path, idoc-get can still extract any value from the stored document.

Use schema-level path controls when the indexing policy belongs to one attribute. The same controls can also be supplied as store options: :idoc-opts applies defaults to every idoc domain, schema keys configure the domain used by an idoc attribute, and :idoc-domains configures a named domain directly.

For a single attribute and domain, later layers override earlier ones: :idoc-opts, then schema keys, then the matching :idoc-domains entry. If several idoc attributes share the same domain, Datalevin combines their include and exclude selectors. Excluded paths are still applied after included paths. The example below expresses the same policy as the schema keys in Section 2.1 at the store level; use one mechanism or the other for ordinary schemas.

(d/create-conn
  "/tmp/idoc-db"
  schema
  {:idoc-domains
   {"user_metadata" {:indexed-paths [:theme :profile :tags :contact]
                     :excluded-paths [[:profile :secret]]}}})

Connection conn = Datalevin.createConn(
    "/tmp/idoc-db",
    schema,
    Map.of(":idoc-domains",
           Map.of("user_metadata",
                  Map.of(":indexed-paths",
                         List.of("theme", "profile", "tags", "contact"),
                         ":excluded-paths",
                         List.of(List.of("profile", "secret"))))));

from datalevin import connect, idoc_domain

conn = connect(
    "/tmp/idoc-db",
    schema=schema,
    opts={":idoc-domains": {
        "user_metadata": idoc_domain(
            indexed_paths=["theme", "profile", "tags", "contact"],
            excluded_paths=[["profile", "secret"]])
    }})

import { connect, idocDomain } from "datalevin-node";

const conn = await connect("/tmp/idoc-db", {
  schema,
  opts: {
    ":idoc-domains": {
      user_metadata: idocDomain({
        indexedPaths: ["theme", "profile", "tags", "contact"],
        excludedPaths: [["profile", "secret"]]
      })
    }
  }
});

Path indexing choices are part of index configuration. Changing :db.idoc/indexedPaths, :db.idoc/excludedPaths, :idoc-opts, or :idoc-domains after data has been indexed requires an explicit rebuild, because the existing path dictionary and inverted index were built under the old selection policy.

2.4 Transacting Documents

idoc values must be document-like maps. Nested vectors are treated as arrays.

(d/transact! conn
  [{:db/id 101
    :user/metadata {:theme "dark"
                    :profile {:age 30 :name "Alice" :secret "internal"}
                    :tags ["beta" "admin"]
                    :contact {:email "alice@example.com"}}
    :user/raw-json "{\"middle\":null,\"theme\":\"dark\"}"}])

conn.transact(Datalevin.tx()
    .entity(Tx.entity(101)
        .put(":user/metadata", Map.of(
            "theme", "dark",
            "profile", Map.of("age", 30, "name", "Alice",
                              "secret", "internal"),
            "tags", List.of("beta", "admin"),
            "contact", Map.of("email", "alice@example.com")))
        .put(":user/raw-json", "{\"middle\":null,\"theme\":\"dark\"}")));

conn.transact([
    {":db/id": 101,
     ":user/metadata": {
         "theme": "dark",
         "profile": {"age": 30, "name": "Alice", "secret": "internal"},
         "tags": ["beta", "admin"],
         "contact": {"email": "alice@example.com"}},
     ":user/raw-json": '{"middle":null,"theme":"dark"}'}
])

await conn.transact([
  { ":db/id": 101,
    ":user/metadata": {
      theme: "dark",
      profile: { age: 30, name: "Alice", secret: "internal" },
      tags: ["beta", "admin"],
      contact: { email: "alice@example.com" }
    },
    ":user/raw-json": "{\"middle\":null,\"theme\":\"dark\"}" }
]);

Because [:profile :secret] is excluded, (idoc-match $ :user/metadata {:profile {:secret "internal"}}) does not find this document. A query that finds the document by another indexed path, such as {:theme "dark"}, can still bind ?doc and read (idoc-get ?doc :profile :secret).

idoc input rules

  • The top-level value must be a map.
  • Keys may be keywords or strings, depending on the format and client language.
  • Lists are not allowed; use vectors for arrays.
  • nil values are normalized to :json/null.
  • :json/null is reserved and rejected on input.
  • For :edn format, string payloads are read as EDN and must yield a map.
  • JSON and Markdown attributes accept string payloads in those formats.

2.5 Markdown Documents

Markdown idoc values are useful when the source data is authored as prose but you still want to query by section. Datalevin parses Markdown headings into nested map paths and stores the section text as values.

(d/transact! conn
  [{:db/id 201
    :doc/title "Search Guide"
    :doc/markdown
    "# Guide

## Install

Run `clj -M:dev`.

## Configure Search

Set `:index-position? true` for phrase search."}])

String markdown = "# Guide\n\n" +
    "## Install\n\n" +
    "Run `clj -M:dev`.\n\n" +
    "## Configure Search\n\n" +
    "Set `:index-position? true` for phrase search.";

conn.transact(Datalevin.tx()
    .entity(Tx.entity(201)
        .put(":doc/title", "Search Guide")
        .put(":doc/markdown", markdown)));

markdown = """# Guide

## Install

Run `clj -M:dev`.

## Configure Search

Set `:index-position? true` for phrase search."""

conn.transact([
    {":db/id": 201,
     ":doc/title": "Search Guide",
     ":doc/markdown": markdown}
])

const markdown = `# Guide

## Install

Run \`clj -M:dev\`.

## Configure Search

Set \`:index-position? true\` for phrase search.`;

await conn.transact([
  {":db/id": 201,
   ":doc/title": "Search Guide",
   ":doc/markdown": markdown}
]);

The parsed idoc value is equivalent to the following nested structure:

{:guide {:install "Run clj -M:dev."
         :configure-search "Set :index-position? true for phrase search."}}

Map<Object, Object> parsed = Map.of(
    Datalevin.kw(":guide"), Map.of(
        Datalevin.kw(":install"), "Run clj -M:dev.",
        Datalevin.kw(":configure-search"),
        "Set :index-position? true for phrase search."));

parsed = {
    ":guide": {
        ":install": "Run clj -M:dev.",
        ":configure-search": "Set :index-position? true for phrase search."}}

const parsed = {
  ":guide": {
    ":install": "Run clj -M:dev.",
    ":configure-search": "Set :index-position? true for phrase search."
  }
};

Heading text is normalized to keyword-like paths: case is folded, leading numbering is removed, punctuation is dropped, and spaces become hyphens. Inline Markdown markup is stripped from the stored text. Content before the first heading is rejected, because Datalevin needs a heading path for the text.

You can match a section and extract its text in the same query:

(d/q '[:find ?title ?install
       :where
       [?e :doc/title ?title]
       [(idoc-match $ :doc/markdown
                    {:guide {:install "Run clj -M:dev."}})
        [[?e _ ?doc]]]
       [(idoc-get ?doc :guide :install) ?install]]
     db)

Object sections = conn.query(
    "[:find ?title ?install " +
    " :where [?e :doc/title ?title] " +
    "        [(idoc-match $ :doc/markdown " +
    "                     {:guide {:install \"Run clj -M:dev.\"}}) " +
    "         [[?e _ ?doc]]] " +
    "        [(idoc-get ?doc :guide :install) ?install]]");

sections = conn.query(
    '[:find ?title ?install '
    ' :where [?e :doc/title ?title] '
    '        [(idoc-match $ :doc/markdown '
    '                     {:guide {:install "Run clj -M:dev."}}) '
    '         [[?e _ ?doc]]] '
    '        [(idoc-get ?doc :guide :install) ?install]]')

const sections = await conn.query(
  "[:find ?title ?install " +
  " :where [?e :doc/title ?title] " +
  "        [(idoc-match $ :doc/markdown " +
  "                     {:guide {:install \"Run clj -M:dev.\"}}) " +
  "         [[?e _ ?doc]]] " +
  "        [(idoc-get ?doc :guide :install) ?install]]");

For Markdown idoc values, query paths are normalized too. This means string paths that look like headings work:

(d/q '[:find ?e
       :where
       [(idoc-match $ :doc/markdown
                    {"Guide" {"Configure Search"
                              "Set :index-position? true for phrase search."}})
        [[?e _ _]]]]
     db)

Object matches = conn.query(
    "[:find ?e " +
    " :where [(idoc-match $ :doc/markdown " +
    "          {\"Guide\" {\"Configure Search\" " +
    "                    \"Set :index-position? true for phrase search.\"}}) " +
    "         [[?e _ _]]]]");

matches = conn.query(
    '[:find ?e '
    ' :where '
    ' [(idoc-match $ :doc/markdown '
    '              {"Guide" {"Configure Search" '
    '                        "Set :index-position? true for phrase search."}}) '
    '  [[?e _ _]]]]')

const matches = await conn.query(
  `[:find ?e
    :where
    [(idoc-match $ :doc/markdown
                 {"Guide" {"Configure Search"
                           "Set :index-position? true for phrase search."}})
     [[?e _ _]]]]`);

Use Markdown idoc values for section-level lookup over authored documents. If a section needs independent identity, permissions, relationships, or versioning, model it as normal component entities instead.

2.6 Patching Documents

Use :db.fn/patchIdoc for partial updates when you do not want to resend the whole document.

(d/transact! conn
  [[:db.fn/patchIdoc 101 :user/metadata
    [[:set    [:profile :display-name] "Alice A."]
     [:unset  [:profile :middle]]
     [:update [:tags] :conj "active"]]]])

conn.transact(Datalevin.tx()
    .raw(Datalevin.edn(
        "[:db.fn/patchIdoc 101 :user/metadata " +
        " [[:set [:profile :display-name] \"Alice A.\"] " +
        "  [:unset [:profile :middle]] " +
        "  [:update [:tags] :conj \"active\"]]]")));

conn.transact([
    [":db.fn/patchIdoc", 101, ":user/metadata",
     [[":set", [":profile", ":display-name"], "Alice A."],
      [":unset", [":profile", ":middle"]],
      [":update", [":tags"], ":conj", "active"]]]
])

await conn.transact([
  [":db.fn/patchIdoc", 101, ":user/metadata",
    [[":set", [":profile", ":display-name"], "Alice A."],
     [":unset", [":profile", ":middle"]],
     [":update", [":tags"], ":conj", "active"]]]
]);

Patch paths are vectors of map keys and vector indices. Supported operations are:

Patching cardinality-many documents needs the old value

For cardinality-many idoc attributes, provide the old document value so Datalevin can identify which value should be patched.

3. Querying Path-Indexed Documents

Use idoc-match inside Datalog to find entities with matching nested document content. It returns matching datoms as [e a v] triples, so the result can be joined with ordinary facts.

;; Find users with a dark theme in their metadata document.
(d/q '[:find ?e
       :where [(idoc-match $ :user/metadata {:theme "dark"}) [[?e _ _]]]]
     db)

conn.query("[:find ?e " +
           " :where [(idoc-match $ :user/metadata {:theme \"dark\"}) [[?e _ _]]]]");

conn.query('[:find ?e '
           ' :where [(idoc-match $ :user/metadata {:theme "dark"}) [[?e _ _]]]]')

await conn.query('[:find ?e ' +
                 ' :where [(idoc-match $ :user/metadata {:theme "dark"}) [[?e _ _]]]]');

Nested maps express nested path matches:

(d/q '[:find ?e
       :where [(idoc-match $ :user/metadata
                            {:profile {:age 30 :name "Alice"}})
               [[?e _ _]]]]
     db)

conn.query("[:find ?e " +
           " :where [(idoc-match $ :user/metadata " +
           "          {:profile {:age 30 :name \"Alice\"}}) [[?e _ _]]]]");

conn.query('[:find ?e '
           ' :where [(idoc-match $ :user/metadata '
           '          {:profile {:age 30 :name "Alice"}}) [[?e _ _]]]]')

await conn.query('[:find ?e ' +
                 ' :where [(idoc-match $ :user/metadata ' +
                 '          {:profile {:age 30 :name "Alice"}}) [[?e _ _]]]]');

Vectors are treated as arrays. A match succeeds if any element matches:

(d/q '[:find ?e
       :where [(idoc-match $ :user/metadata {:tags "admin"}) [[?e _ _]]]]
     db)

conn.query("[:find ?e " +
           " :where [(idoc-match $ :user/metadata {:tags \"admin\"}) [[?e _ _]]]]");

conn.query('[:find ?e '
           ' :where [(idoc-match $ :user/metadata {:tags "admin"}) [[?e _ _]]]]')

await conn.query('[:find ?e ' +
                 ' :where [(idoc-match $ :user/metadata {:tags "admin"}) [[?e _ _]]]]');

3.1 Logical Combinators

Use [:and ...], [:or ...], and [:not ...] inside query maps:

(d/q '[:find ?e
       :where [(idoc-match $ :user/metadata
                            {:profile [:or {:age 30} {:age 40}]})
               [[?e _ _]]]]
     db)

conn.query("[:find ?e " +
           " :where [(idoc-match $ :user/metadata " +
           "          {:profile [:or {:age 30} {:age 40}]}) " +
           "         [[?e _ _]]]]");

conn.query('[:find ?e '
           ' :where [(idoc-match $ :user/metadata '
           '          {:profile [:or {:age 30} {:age 40}]}) '
           '         [[?e _ _]]]]')

await conn.query('[:find ?e ' +
                 ' :where [(idoc-match $ :user/metadata ' +
                 '          {:profile [:or {:age 30} {:age 40}]}) ' +
                 '         [[?e _ _]]]]');

3.2 Predicates and Ranges

Predicates can appear inline in query maps, or as path predicates supplied as query inputs:

;; Inline predicate.
(d/q '[:find ?e
       :where [(idoc-match $ :user/metadata {:profile {:age (> 21)}})
               [[?e _ _]]]]
     db)

;; Path predicate supplied as data.
(d/q '[:find ?e
       :in $ ?q
       :where [(idoc-match $ :user/metadata ?q) [[?e _ _]]]]
     db
     '(>= [:profile :age] 30))

;; Range via multi-arity comparison.
(d/q '[:find ?e
       :in $ ?q
       :where [(idoc-match $ :user/metadata ?q) [[?e _ _]]]]
     db
     '(< 20 [:profile :age] 40))

// Inline predicate.
conn.query("[:find ?e " +
           " :where [(idoc-match $ :user/metadata " +
           "          {:profile {:age (> 21)}}) [[?e _ _]]]]");

// Path predicate supplied as data.
conn.query("[:find ?e " +
           " :in $ ?q " +
           " :where [(idoc-match $ :user/metadata ?q) [[?e _ _]]]]",
           Datalevin.edn("(>= [:profile :age] 30)"));

// Range via multi-arity comparison.
conn.query("[:find ?e " +
           " :in $ ?q " +
           " :where [(idoc-match $ :user/metadata ?q) [[?e _ _]]]]",
           Datalevin.edn("(< 20 [:profile :age] 40)"));

# Inline predicate.
conn.query('[:find ?e '
           ' :where [(idoc-match $ :user/metadata '
           '          {:profile {:age (> 21)}}) [[?e _ _]]]]')

# Path predicate supplied as data.
conn.query('[:find ?e '
           ' :in $ ?q '
           ' :where [(idoc-match $ :user/metadata ?q) [[?e _ _]]]]',
           interop().read_edn("(>= [:profile :age] 30)"))

# Range via multi-arity comparison.
conn.query('[:find ?e '
           ' :in $ ?q '
           ' :where [(idoc-match $ :user/metadata ?q) [[?e _ _]]]]',
           interop().read_edn("(< 20 [:profile :age] 40)"))

// Inline predicate.
await conn.query('[:find ?e ' +
                 ' :where [(idoc-match $ :user/metadata ' +
                 '          {:profile {:age (> 21)}}) [[?e _ _]]]]');

// Path predicate supplied as data.
await conn.query('[:find ?e ' +
                 ' :in $ ?q ' +
                 ' :where [(idoc-match $ :user/metadata ?q) [[?e _ _]]]]',
                 await interop().readEdn("(>= [:profile :age] 30)"));

// Range via multi-arity comparison.
await conn.query('[:find ?e ' +
                 ' :in $ ?q ' +
                 ' :where [(idoc-match $ :user/metadata ?q) [[?e _ _]]]]',
                 await interop().readEdn("(< 20 [:profile :age] 40)"));

Supported predicates are nil?, >, >=, <, and <=.

3.3 Wildcard Paths

Use wildcard path segments when the exact nested key is not known:

;; Match any single key under :profile with value >= 30.
(d/q '[:find ?e
       :in $ ?q
       :where [(idoc-match $ :user/metadata ?q) [[?e _ _]]]]
     db
     '(>= [:profile :?] 30))

;; Match a name field at any depth.
(d/q '[:find ?e
       :where [(idoc-match $ :user/metadata {:* {:name "Alice"}})
               [[?e _ _]]]]
     db)

// Match any single key under profile with value >= 30.
conn.query("[:find ?e " +
           " :in $ ?q " +
           " :where [(idoc-match $ :user/metadata ?q) [[?e _ _]]]]",
           Datalevin.edn("(>= [:profile :?] 30)"));

// Match a name field at any depth.
conn.query("[:find ?e " +
           " :where [(idoc-match $ :user/metadata {:* {:name \"Alice\"}}) " +
           "         [[?e _ _]]]]");

# Match any single key under profile with value >= 30.
conn.query('[:find ?e '
           ' :in $ ?q '
           ' :where [(idoc-match $ :user/metadata ?q) [[?e _ _]]]]',
           interop().read_edn("(>= [:profile :?] 30)"))

# Match a name field at any depth.
conn.query('[:find ?e '
           ' :where [(idoc-match $ :user/metadata {:* {:name "Alice"}}) '
           '         [[?e _ _]]]]')

// Match any single key under profile with value >= 30.
await conn.query('[:find ?e ' +
                 ' :in $ ?q ' +
                 ' :where [(idoc-match $ :user/metadata ?q) [[?e _ _]]]]',
                 await interop().readEdn("(>= [:profile :?] 30)"));

// Match a name field at any depth.
await conn.query('[:find ?e ' +
                 ' :where [(idoc-match $ :user/metadata {:* {:name "Alice"}}) ' +
                 '         [[?e _ _]]]]');

:? matches exactly one path segment. :* matches any depth, including zero segments.

3.4 Null Matching

To match null values, use (nil?):

(d/q '[:find ?e
       :where [(idoc-match $ :user/raw-json {"middle" (nil?)}) [[?e _ _]]]]
     db)

conn.query("[:find ?e " +
           " :where [(idoc-match $ :user/raw-json {\"middle\" (nil?)}) " +
           "         [[?e _ _]]]]");

conn.query('[:find ?e '
           ' :where [(idoc-match $ :user/raw-json {"middle" (nil?)}) '
           '         [[?e _ _]]]]')

await conn.query('[:find ?e ' +
                 ' :where [(idoc-match $ :user/raw-json {"middle" (nil?)}) ' +
                 '         [[?e _ _]]]]');

Match JSON keys as strings

JSON keys are strings, so JSON documents should be matched with "middle", not :middle.

3.5 Extracting Values with idoc-get

idoc-get extracts a nested value from an idoc document inside a Datalog query:

(d/q '[:find ?e ?email
       :where [(idoc-match $ :user/metadata {:theme "dark"}) [[?e _ ?doc]]]
              [(idoc-get ?doc :contact :email) ?email]]
     db)

conn.query("[:find ?e ?email " +
           " :where [(idoc-match $ :user/metadata {:theme \"dark\"}) [[?e _ ?doc]]] " +
           "        [(idoc-get ?doc :contact :email) ?email]]");

conn.query('[:find ?e ?email '
           ' :where [(idoc-match $ :user/metadata {:theme "dark"}) [[?e _ ?doc]]] '
           '        [(idoc-get ?doc :contact :email) ?email]]')

await conn.query('[:find ?e ?email ' +
                 ' :where [(idoc-match $ :user/metadata {:theme "dark"}) [[?e _ ?doc]]] ' +
                 '        [(idoc-get ?doc :contact :email) ?email]]');

When a path traverses arrays, idoc-get returns a vector of matching values.

4. How idoc Indexes Work

Datalevin stores the idoc value itself as the datom value. The path index records enough path and value information to narrow candidates without duplicating full documents. Each domain has a separate path index, and indexing is synchronous during transactions. Unlike full-text and vector indexes, idoc has no async indexing mode. After a transaction commits, idoc-match sees the updated document index immediately.

idoc path indexes do not split the fact

idoc gives you indexed path filters, but the document remains one value from the perspective of ordinary datoms. If a nested field deserves identity, constraints, independent history, or joins as a first-class fact, model that part with normal attributes instead.

5. Choosing Between Logical Documents and idoc

Feature Logical document (:db/isComponent) Native idoc
Storage Decomposed into many datoms Stored as one datom value
Schema Attributes are declared explicitly Nested fields can evolve freely
Indexing Standard EAV/AVE and attribute indexes Path-level indexing, all leaf paths by default or selected prefixes
Querying Datalog joins, pull, entity API idoc-match and idoc-get
Updates Fine-grained datom updates patchIdoc or full replacement
Best for Core domain state and relationships Flexible metadata and imported documents

Use logical documents when nested data needs type checking, referential integrity, joins to other entities, or independent updates as facts. Use idoc when nested data is sparse, user-defined, imported from another document store, or expected to change shape frequently.

These models can coexist on the same entity. A product can have structured facts for price and inventory, component entities for variants, full-text fields for descriptions, and an idoc attribute for merchant-specific metadata.

6. Ad Hoc Queries over Nested Data

Sometimes you do not want to persist a nested value at all. You have a map from an API response, a fixture, or a one-off data export, and you want Datalog's joins and predicates for exploration. Chapter 8 showed that Datalevin can query an in-memory sequence of tuples. A small boundary helper can turn a nested value into such a relation.

The following helpers walk maps/objects and vectors/arrays and emit one tuple for each leaf value. By default, the tuple is the path segments followed by the leaf value.

(defn leaf-paths
  "Returns paths to scalar leaves in a nested map/vector."
  ([root]
   {:pre [(or (map? root) (vector? root))]}
   (leaf-paths [] root))
  ([parent x]
   (cond
     (map? x)
     (mapcat (fn [[k v]]
               (leaf-paths (conj parent k) v))
             x)

     (vector? x)
     (mapcat (fn [[i v]]
               (leaf-paths (conj parent i) v))
             (map-indexed vector x))

     :else
     [parent])))

(defn nested->tuples
  "Converts nested maps/vectors into tuples queryable by Datalog."
  ([x]
   (nested->tuples x nil))
  ([x {:keys [paths?] :or {paths? false}}]
   (with-meta
     (mapv (fn [path]
             (let [tuple (conj path (get-in x path))]
               (if paths?
                 (into [path] tuple)
                 tuple)))
           (leaf-paths x))
     {:datalevin.docs/original x})))

static List<List<Object>> leafPaths(Object root) {
    return leafPaths(List.of(), root);
}

static List<List<Object>> leafPaths(List<Object> parent, Object x) {
    if (x instanceof Map<?, ?> m) {
        List<List<Object>> paths = new ArrayList<>();
        for (Map.Entry<?, ?> entry : m.entrySet()) {
            ArrayList<Object> next = new ArrayList<>(parent);
            next.add(entry.getKey());
            paths.addAll(leafPaths(next, entry.getValue()));
        }
        return paths;
    }
    if (x instanceof List<?> xs) {
        List<List<Object>> paths = new ArrayList<>();
        for (int i = 0; i < xs.size(); i++) {
            ArrayList<Object> next = new ArrayList<>(parent);
            next.add(i);
            paths.addAll(leafPaths(next, xs.get(i)));
        }
        return paths;
    }
    return List.of(new ArrayList<>(parent));
}

static Object getIn(Object root, List<Object> path) {
    Object cur = root;
    for (Object step : path) {
        if (cur instanceof Map<?, ?> m) {
            cur = m.get(step);
        } else if (cur instanceof List<?> xs) {
            cur = xs.get(((Number) step).intValue());
        } else {
            throw new IllegalArgumentException("Cannot traverse " + step);
        }
    }
    return cur;
}

static List<List<Object>> nestedToTuples(Object x, boolean paths) {
    List<List<Object>> out = new ArrayList<>();
    for (List<Object> path : leafPaths(x)) {
        ArrayList<Object> tuple = new ArrayList<>(path);
        tuple.add(getIn(x, path));
        if (paths) {
            ArrayList<Object> withPath = new ArrayList<>();
            withPath.add(path);
            withPath.addAll(tuple);
            out.add(withPath);
        } else {
            out.add(tuple);
        }
    }
    return out;
}

def leaf_paths(root, path=()):
    if isinstance(root, dict):
        paths = []
        for key, value in root.items():
            paths.extend(leaf_paths(value, (*path, key)))
        return paths
    if isinstance(root, list):
        paths = []
        for i, value in enumerate(root):
            paths.extend(leaf_paths(value, (*path, i)))
        return paths
    return [list(path)]


def get_in(root, path):
    cur = root
    for step in path:
        cur = cur[step]
    return cur


def nested_to_tuples(x, paths=False):
    tuples = []
    for path in leaf_paths(x):
        tuple_ = [*path, get_in(x, path)]
        tuples.append([path, *tuple_] if paths else tuple_)
    return tuples

function leafPaths(root, path = []) {
  if (Array.isArray(root)) {
    return root.flatMap((value, i) => leafPaths(value, [...path, i]));
  }
  if (root !== null && typeof root === "object") {
    return Object.entries(root).flatMap(([key, value]) =>
      leafPaths(value, [...path, key]));
  }
  return [path];
}

function getIn(root, path) {
  return path.reduce((cur, step) => cur[step], root);
}

function nestedToTuples(x, { paths = false } = {}) {
  return leafPaths(x).map((path) => {
    const tuple = [...path, getIn(x, path)];
    return paths ? [path, ...tuple] : tuple;
  });
}

For example:

(def order-doc
  {:order/id "o-1001"
   :customer {:email "ada@example.com"
              :name  "Ada"}
   :lines    [{:sku "book" :qty 2}
              {:sku "pen"  :qty 5}]})

(def tuples (nested->tuples order-doc))
;; => [[:order/id "o-1001"]
;;     [:customer :email "ada@example.com"]
;;     [:customer :name "Ada"]
;;     [:lines 0 :sku "book"]
;;     [:lines 0 :qty 2]
;;     [:lines 1 :sku "pen"]
;;     [:lines 1 :qty 5]]

Map<Object, Object> orderDoc = Map.of(
    Datalevin.kw(":order/id"), "o-1001",
    Datalevin.kw(":customer"), Map.of(
        Datalevin.kw(":email"), "ada@example.com",
        Datalevin.kw(":name"), "Ada"),
    Datalevin.kw(":lines"), List.of(
        Map.of(Datalevin.kw(":sku"), "book",
               Datalevin.kw(":qty"), 2),
        Map.of(Datalevin.kw(":sku"), "pen",
               Datalevin.kw(":qty"), 5)));

List<List<Object>> tuples = nestedToTuples(orderDoc, false);
// => [[:order/id "o-1001"]
//     [:customer :email "ada@example.com"]
//     [:customer :name "Ada"]
//     [:lines 0 :sku "book"]
//     [:lines 0 :qty 2]
//     [:lines 1 :sku "pen"]
//     [:lines 1 :qty 5]]

order_doc = {
    ":order/id": "o-1001",
    ":customer": {":email": "ada@example.com",
                  ":name": "Ada"},
    ":lines": [{":sku": "book", ":qty": 2},
               {":sku": "pen", ":qty": 5}]}

tuples = nested_to_tuples(order_doc)
# => [[":order/id", "o-1001"],
#     [":customer", ":email", "ada@example.com"],
#     [":customer", ":name", "Ada"],
#     [":lines", 0, ":sku", "book"],
#     [":lines", 0, ":qty", 2],
#     [":lines", 1, ":sku", "pen"],
#     [":lines", 1, ":qty", 5]]

const orderDoc = {
  ":order/id": "o-1001",
  ":customer": {":email": "ada@example.com",
                ":name": "Ada"},
  ":lines": [{":sku": "book", ":qty": 2},
             {":sku": "pen", ":qty": 5}]
};

const tuples = nestedToTuples(orderDoc);
// => [[":order/id", "o-1001"],
//     [":customer", ":email", "ada@example.com"],
//     [":customer", ":name", "Ada"],
//     [":lines", 0, ":sku", "book"],
//     [":lines", 0, ":qty", 2],
//     [":lines", 1, ":sku", "pen"],
//     [":lines", 1, ":qty", 5]]

Now the vector index in the path can be used as a join key:

(d/q '[:find ?sku ?qty
       :where
       [:lines ?i :sku ?sku]
       [:lines ?i :qty ?qty]]
     tuples)
;; => #{["book" 2] ["pen" 5]}

Object lines = conn.query(
    "[:find ?sku ?qty " +
    " :in $ $data " +
    " :where [$data :lines ?i :sku ?sku] " +
    "        [$data :lines ?i :qty ?qty]]",
    tuples);
// => [["book", 2], ["pen", 5]]

lines = conn.query(
    '[:find ?sku ?qty '
    ' :in $ $data '
    ' :where [$data :lines ?i :sku ?sku] '
    '        [$data :lines ?i :qty ?qty]]',
    tuples)
# => [["book", 2], ["pen", 5]]

const lines = await conn.query(
  "[:find ?sku ?qty " +
  " :in $ $data " +
  " :where [$data :lines ?i :sku ?sku] " +
  "        [$data :lines ?i :qty ?qty]]",
  tuples);
// => [["book", 2], ["pen", 5]]

When the paths option is true, the full path is also carried as the first element of each tuple. This is useful when a query result should drive a later path lookup or update:

(def tuples-with-paths (nested->tuples order-doc {:paths? true}))

(d/q '[:find ?path ?sku
       :where
       [?path :lines ?i :sku ?sku]]
     tuples-with-paths)
;; => #{[[:lines 0 :sku] "book"]
;;      [[:lines 1 :sku] "pen"]}

List<List<Object>> tuplesWithPaths = nestedToTuples(orderDoc, true);

Object paths = conn.query(
    "[:find ?path ?sku " +
    " :in $ $data " +
    " :where [$data ?path :lines ?i :sku ?sku]]",
    tuplesWithPaths);
// => [[[":lines", 0, ":sku"], "book"],
//     [[":lines", 1, ":sku"], "pen"]]

tuples_with_paths = nested_to_tuples(order_doc, paths=True)

paths = conn.query(
    '[:find ?path ?sku '
    ' :in $ $data '
    ' :where [$data ?path :lines ?i :sku ?sku]]',
    tuples_with_paths)
# => [[[":lines", 0, ":sku"], "book"],
#     [[":lines", 1, ":sku"], "pen"]]

const tuplesWithPaths = nestedToTuples(orderDoc, {paths: true});

const paths = await conn.query(
  "[:find ?path ?sku " +
  " :in $ $data " +
  " :where [$data ?path :lines ?i :sku ?sku]]",
  tuplesWithPaths);
// => [[[":lines", 0, ":sku"], "book"],
//     [[":lines", 1, ":sku"], "pen"]]

Ad hoc tuple views are not persistence

This technique is for ad hoc analysis and local transformation. It does not create attributes, schema, indexes, lookup refs, or durable entities. If the nested data is part of the application's long-lived domain model, turn it into ordinary Datalevin facts or component entities. If it is flexible document data that should be stored and searched by path, use :db.type/idoc.

Summary

Datalevin lets you choose a representation for each part of your data. Use logical documents for structured domain state where facts, relationships, and constraints matter. Use idoc for flexible document-shaped data that should be stored as one value but queried by path. By default, idoc indexes every scalar leaf path; use selective path indexing when only a few prefixes should be searchable or when large user-defined documents would otherwise create more index surface area than the application needs. For one-off nested values, convert them to tuples and query them in memory instead of creating durable schema or idoc indexes. Both stored models participate in Datalog, so you can join document filters with graph, relational, full-text, and vector queries in one transactionally consistent database.

User Examples

Log in to create examples

No examples for this chapter yet.