Chapter 14: Document Modeling
Datalevin gives you two ways to model document-shaped data: logical documents, built from normal datoms and component references, and indexed documents (idoc), stored as single values with path-level indexes. By default an idoc indexes every scalar leaf path, and schema or domain options can restrict indexing to selected path prefixes.
Figure 14.1 contrasts the two representations.
Use logical documents when the nested structure is part of your core domain model. Use idoc when the nested structure is flexible metadata, imported JSON, Markdown-derived structure, or application-defined data that should be searched by path without forcing every field into the schema. Section 6 shows a boundary technique for querying nested data without storing it at all.
1. Logical Documents with :db/isComponent
The most common way to model durable domain documents in Datalevin is to use component attributes. A component reference stores a nested entity as ordinary datoms, but treats that entity as owned by its parent.
Components require single ownership
Use component attributes for single-owner logical documents. If a nested record can be shared by more than one parent, model it with ordinary references or as an idoc value instead.
1.1 Example: A Nested Blog Post
;; Schema: :post/comments owns comment entities.
{:post/slug {:db/valueType :db.type/string
:db/unique :db.unique/identity}
:post/comments {:db/valueType :db.type/ref
:db/cardinality :db.cardinality/many
:db/isComponent true}
:user/email {:db/valueType :db.type/string
:db/unique :db.unique/identity}
:comment/author {:db/valueType :db.type/ref}}
Schema schema = Datalevin.schema()
.attr(":post/slug",
Schema.attribute()
.valueType(Schema.ValueType.STRING)
.unique(Schema.Unique.IDENTITY))
.attr(":post/comments",
Schema.attribute()
.valueType(Schema.ValueType.REF)
.cardinality(Schema.Cardinality.MANY)
.isComponent(true))
.attr(":user/email",
Schema.attribute()
.valueType(Schema.ValueType.STRING)
.unique(Schema.Unique.IDENTITY))
.attr(":comment/author",
Schema.attribute()
.valueType(Schema.ValueType.REF));
schema = {
":post/slug": {
":db/valueType": ":db.type/string",
":db/unique": ":db.unique/identity"
},
":post/comments": {
":db/valueType": ":db.type/ref",
":db/cardinality": ":db.cardinality/many",
":db/isComponent": True
},
":user/email": {
":db/valueType": ":db.type/string",
":db/unique": ":db.unique/identity"
},
":comment/author": {
":db/valueType": ":db.type/ref"
}
}
const schema = {
":post/slug": {
":db/valueType": ":db.type/string",
":db/unique": ":db.unique/identity"
},
":post/comments": {
":db/valueType": ":db.type/ref",
":db/cardinality": ":db.cardinality/many",
":db/isComponent": true
},
":user/email": {
":db/valueType": ":db.type/string",
":db/unique": ":db.unique/identity"
},
":comment/author": {
":db/valueType": ":db.type/ref"
}
};
Logical documents are still normal facts. Every nested field can be typed, indexed, joined, pulled, and updated independently. This is useful when nested data has identity, relationships, constraints, or a lifecycle that should remain visible to Datalog.
For example, comments can be modeled as components of a post while their authors remain separate top-level entities, addressed through a unique email lookup ref:
(d/transact! conn
[{:user/name "Ada"
:user/email "ada@example.com"}
{:post/slug "indexes-as-capabilities"
:post/title "Indexes as Capabilities"
:post/comments [{:comment/body "This clarifies the mental model."
:comment/author [:user/email "ada@example.com"]}]}])
Object ada = List.of(":user/email", "ada@example.com");
conn.transact(Datalevin.tx()
.entity(Tx.entity()
.put(":user/name", "Ada")
.put(":user/email", "ada@example.com"))
.entity(Tx.entity()
.put(":post/slug", "indexes-as-capabilities")
.put(":post/title", "Indexes as Capabilities")
.put(":post/comments", List.of(
Tx.entity()
.put(":comment/body", "This clarifies the mental model.")
.put(":comment/author", ada)
.build()))));
from datalevin import tx
conn.transact(tx.data(
tx.entity(attrs={
"user/name": "Ada",
"user/email": "ada@example.com",
}),
tx.entity(attrs={
"post/slug": "indexes-as-capabilities",
"post/title": "Indexes as Capabilities",
"post/comments": [tx.entity(attrs={
"comment/body": "This clarifies the mental model.",
"comment/author": tx.lookup_ref(
"user/email", "ada@example.com"
),
})],
}),
))
import { tx } from "datalevin-node";
await conn.transact(tx.data(
tx.entity({
"user/name": "Ada",
"user/email": "ada@example.com"
}),
tx.entity({
"post/slug": "indexes-as-capabilities",
"post/title": "Indexes as Capabilities",
"post/comments": [tx.entity({
"comment/body": "This clarifies the mental model.",
"comment/author": tx.lookupRef("user/email", "ada@example.com")
})]
})
));
For logical documents, use pull patterns to navigate nested paths:
(d/pull db
[:post/title {:post/comments [:comment/body {:comment/author [:user/name]}]}]
[:post/slug "indexes-as-capabilities"])
conn.pull(
"[:post/title {:post/comments [:comment/body {:comment/author [:user/name]}]}]",
List.of(":post/slug", "indexes-as-capabilities"));
from datalevin import q, tx
author = q.pull_nested("comment/author", q.selector("user/name"))
comments = q.pull_nested(
"post/comments", q.selector("comment/body", author))
conn.pull(
q.selector("post/title", comments),
tx.lookup_ref("post/slug", "indexes-as-capabilities"),
)
import { q, tx } from "datalevin-node";
const author = q.pullNested("comment/author", q.selector("user/name"));
const comments = q.pullNested(
"post/comments", q.selector("comment/body", author)
);
await conn.pull(
q.selector("post/title", comments),
tx.lookupRef("post/slug", "indexes-as-capabilities")
);
2. Native Indexed Documents with idoc
An idoc stores a nested document as one datom value, while Datalevin maintains an index for paths inside the document, as shown in Figure 14.2. This gives you document-database-style path queries while still using Datalog joins and transactions.
idoc indexes are orthogonal to full-text and vector indexes. The same entity can have structured facts, full-text fields, embeddings, and idoc metadata, and a single Datalog query can combine all of those constraints.
2.1 Defining an idoc Attribute
Declare an idoc attribute with :db/valueType :db.type/idoc. The optional :db/idocFormat controls how payloads are interpreted, the optional :db/domain gives the backing index a stable domain name, and the optional :db.idoc/indexedPaths / :db.idoc/excludedPaths keys control which path prefixes are indexed. Section 2.3 explains path selectors.
(def schema
{:user/metadata {:db/valueType :db.type/idoc
:db/domain "user_metadata"
:db.idoc/indexedPaths [:theme :profile :tags :contact]
:db.idoc/excludedPaths [[:profile :secret]]}
:user/raw-json {:db/valueType :db.type/idoc
:db/idocFormat :json}
:doc/title {:db/valueType :db.type/string}
:doc/markdown {:db/valueType :db.type/idoc
:db/idocFormat :markdown}})
Schema schema = Datalevin.schema()
.attr(":user/metadata",
Schema.attribute()
.valueType(Schema.ValueType.IDOC)
.prop(":db/domain", "user_metadata")
.idocIndexedPaths("theme", "profile", "tags", "contact")
.idocExcludedPaths(
List.of("profile", "secret")))
.attr(":user/raw-json",
Schema.attribute()
.valueType(Schema.ValueType.IDOC)
.prop(":db/idocFormat", Datalevin.kw(":json")))
.attr(":doc/title",
Schema.attribute()
.valueType(Schema.ValueType.STRING))
.attr(":doc/markdown",
Schema.attribute()
.valueType(Schema.ValueType.IDOC)
.prop(":db/idocFormat", Datalevin.kw(":markdown")));
from datalevin import idoc_attr
schema = {
":user/metadata": idoc_attr(
domain="user_metadata",
indexed_paths=["theme", "profile", "tags", "contact"],
excluded_paths=[["profile", "secret"]]),
":user/raw-json": idoc_attr(format=":json"),
":doc/title": {
":db/valueType": ":db.type/string"
},
":doc/markdown": idoc_attr(format=":markdown")
}
import { idocAttr } from "datalevin-node";
const schema = {
":user/metadata": idocAttr({
domain: "user_metadata",
indexedPaths: ["theme", "profile", "tags", "contact"],
excludedPaths: [["profile", "secret"]]
}),
":user/raw-json": idocAttr({ format: ":json" }),
":doc/title": {
":db/valueType": ":db.type/string"
},
":doc/markdown": idocAttr({ format: ":markdown" })
};
Supported :db/idocFormat values are :edn (the default), :json, and :markdown. If :db/domain is omitted, Datalevin derives a domain from the attribute name without the leading colon; for example, :user/metadata becomes "user/metadata". If no path controls are supplied, Datalevin indexes all scalar leaf paths in the document.
2.2 idoc Domains
An idoc domain names the backing path index for one idoc attribute. If :db/domain is omitted, Datalevin derives the domain from the attribute name. Most schemas can treat this as an internal index name and query documents by attribute.
This is related to, but not the same as, full-text, vector, or embedding domains. Those indexes use domain lists or store-level domain maps to configure retrieval and indexing behavior, and some support default or automatic domains (Chapters 16 and 17).
idoc has no :db.idoc/domains list or autoDomain flag. An idoc attribute has one idoc domain, selected by :db/domain or derived from the attribute name. Several idoc attributes may intentionally share the same domain, but that is an index design choice, not a ranking or retrieval policy.
Store-level :idoc-opts and :idoc-domains can configure which paths a domain indexes (Section 2.3), but they do not make one attribute participate in multiple domains. Markdown idoc values normalize heading text into path segments, so they should usually live in their own domains instead of sharing a domain with EDN or JSON idoc values. Section 3 focuses on attribute-scoped idoc-match queries; Chapter 31 lists the domain-scoped and full-DB forms.
2.3 Selective Path Indexing
By default, an idoc domain indexes every scalar leaf path in each document. For large or user-defined documents, this can index more paths than the application needs. Selective path indexing lets you keep only the path prefixes your queries use.
Path selectors may be a top-level keyword, a top-level string, or a vector of keyword/string path segments. They are prefix selectors: :profile includes leaf paths such as [:profile :age] and [:profile :name]. :db.idoc/indexedPaths is the include list for an attribute's domain. When it is absent, all paths are included. :db.idoc/excludedPaths is the omit list and wins over included paths.
The original document is still stored as the datom value. Selective indexing only controls which leaf paths enter the path dictionary and inverted index. A query cannot use idoc-match to find documents by an omitted path, but once a document has been found by another indexed path, idoc-get can still extract any value from the stored document.
Use schema-level path controls when the indexing policy belongs to one attribute. The same controls can also be supplied as store options: :idoc-opts applies defaults to every idoc domain, schema keys configure the domain used by an idoc attribute, and :idoc-domains configures a named domain directly.
For a single attribute and domain, later layers override earlier ones: :idoc-opts, then schema keys, then the matching :idoc-domains entry. If several idoc attributes share the same domain, Datalevin combines their include and exclude selectors. Excluded paths are still applied after included paths. The example below expresses the same policy as the schema keys in Section 2.1 at the store level; use one mechanism or the other for ordinary schemas.
(d/create-conn
"/tmp/idoc-db"
schema
{:idoc-domains
{"user_metadata" {:indexed-paths [:theme :profile :tags :contact]
:excluded-paths [[:profile :secret]]}}})
Connection conn = Datalevin.createConn(
"/tmp/idoc-db",
schema,
Map.of(":idoc-domains",
Map.of("user_metadata",
Map.of(":indexed-paths",
List.of("theme", "profile", "tags", "contact"),
":excluded-paths",
List.of(List.of("profile", "secret"))))));
from datalevin import connect, idoc_domain
conn = connect(
"/tmp/idoc-db",
schema=schema,
opts={":idoc-domains": {
"user_metadata": idoc_domain(
indexed_paths=["theme", "profile", "tags", "contact"],
excluded_paths=[["profile", "secret"]])
}})
import { connect, idocDomain } from "datalevin-node";
const conn = await connect("/tmp/idoc-db", {
schema,
opts: {
":idoc-domains": {
user_metadata: idocDomain({
indexedPaths: ["theme", "profile", "tags", "contact"],
excludedPaths: [["profile", "secret"]]
})
}
}
});
Path indexing choices are part of index configuration. Changing :db.idoc/indexedPaths, :db.idoc/excludedPaths, :idoc-opts, or :idoc-domains after data has been indexed requires an explicit rebuild, because the existing path dictionary and inverted index were built under the old selection policy.
2.4 Transacting Documents
idoc values must be document-like maps. Nested vectors are treated as arrays.
(d/transact! conn
[{:db/id 101
:user/metadata {:theme "dark"
:profile {:age 30 :name "Alice" :secret "internal"}
:tags ["beta" "admin"]
:contact {:email "alice@example.com"}}
:user/raw-json "{\"middle\":null,\"theme\":\"dark\"}"}])
conn.transact(Datalevin.tx()
.entity(Tx.entity(101)
.put(":user/metadata", Map.of(
"theme", "dark",
"profile", Map.of("age", 30, "name", "Alice",
"secret", "internal"),
"tags", List.of("beta", "admin"),
"contact", Map.of("email", "alice@example.com")))
.put(":user/raw-json", "{\"middle\":null,\"theme\":\"dark\"}")));
conn.transact(tx.data(tx.entity(101, {
"user/metadata": {
"theme": "dark",
"profile": {"age": 30, "name": "Alice", "secret": "internal"},
"tags": ["beta", "admin"],
"contact": {"email": "alice@example.com"},
},
"user/raw-json": '{"middle":null,"theme":"dark"}',
})))
await conn.transact(tx.data(tx.entity(101, {
"user/metadata": {
theme: "dark",
profile: { age: 30, name: "Alice", secret: "internal" },
tags: ["beta", "admin"],
contact: { email: "alice@example.com" }
},
"user/raw-json": "{\"middle\":null,\"theme\":\"dark\"}"
})));
Because [:profile :secret] is excluded, (idoc-match $ :user/metadata {:profile {:secret "internal"}}) does not find this document. A query that finds the document by another indexed path, such as {:theme "dark"}, can still bind ?doc and read (idoc-get ?doc :profile :secret).
idoc input rules
- The top-level value must be a map.
- Keys may be keywords or strings, depending on the format and client language.
- Lists are not allowed; use vectors for arrays.
nilvalues are normalized to:json/null.:json/nullis reserved and rejected on input.- For
:ednformat, string payloads are read as EDN and must yield a map. - JSON and Markdown attributes accept string payloads in those formats.
The Python and JavaScript transaction above stores native string keys, so the matching query examples below also use ordinary strings for paths such as "profile" and "age". Use q.kw instead when the stored document actually contains EDN keyword keys; query operators such as :or, :?, and :* remain keywords regardless of the document's key type.
2.5 Markdown Documents
Markdown idoc values are useful when the source data is authored as prose but you still want to query by section. Datalevin parses Markdown headings into nested map paths and stores the section text as values.
(d/transact! conn
[{:db/id 201
:doc/title "Search Guide"
:doc/markdown
"# Guide
## Install
Run `clj -M:dev`.
## Configure Search
Set `:index-position? true` for phrase search."}])
String markdown = "# Guide\n\n" +
"## Install\n\n" +
"Run `clj -M:dev`.\n\n" +
"## Configure Search\n\n" +
"Set `:index-position? true` for phrase search.";
conn.transact(Datalevin.tx()
.entity(Tx.entity(201)
.put(":doc/title", "Search Guide")
.put(":doc/markdown", markdown)));
markdown = """# Guide
## Install
Run `clj -M:dev`.
## Configure Search
Set `:index-position? true` for phrase search."""
conn.transact(tx.data(tx.entity(201, {
"doc/title": "Search Guide",
"doc/markdown": markdown,
})))
const markdown = `# Guide
## Install
Run \`clj -M:dev\`.
## Configure Search
Set \`:index-position? true\` for phrase search.`;
await conn.transact(tx.data(tx.entity(201, {
"doc/title": "Search Guide",
"doc/markdown": markdown
})));
The parsed idoc value is equivalent to the following nested structure:
{:guide {:install "Run clj -M:dev."
:configure-search "Set :index-position? true for phrase search."}}
Map<Object, Object> parsed = Map.of(
Datalevin.kw(":guide"), Map.of(
Datalevin.kw(":install"), "Run clj -M:dev.",
Datalevin.kw(":configure-search"),
"Set :index-position? true for phrase search."));
parsed = {
":guide": {
":install": "Run clj -M:dev.",
":configure-search": "Set :index-position? true for phrase search."}}
const parsed = {
":guide": {
":install": "Run clj -M:dev.",
":configure-search": "Set :index-position? true for phrase search."
}
};
Heading text is normalized to keyword-like paths: case is folded, leading numbering is removed, punctuation is dropped, and spaces become hyphens. Inline Markdown markup is stripped from the stored text. Content before the first heading is rejected, because Datalevin needs a heading path for the text.
You can match a section and extract its text in the same query:
(d/q '[:find ?title ?install
:where
[?e :doc/title ?title]
[(idoc-match $ :doc/markdown
{:guide {:install "Run clj -M:dev."}})
[[?e _ ?doc]]]
[(idoc-get ?doc :guide :install) ?install]]
db)
Object sections = conn.query(
"[:find ?title ?install " +
" :where [?e :doc/title ?title] " +
" [(idoc-match $ :doc/markdown " +
" {:guide {:install \"Run clj -M:dev.\"}}) " +
" [[?e _ ?doc]]] " +
" [(idoc-get ?doc :guide :install) ?install]]");
from datalevin import q
e, title, document, install = map(
q.var, ("e", "title", "doc", "install"))
markdown_match = {
q.kw("guide"): {q.kw("install"): "Run clj -M:dev."},
}
sections = conn.query(q.query(
find=q.relation(title, install),
where=[
q.datom(e, "doc/title", title),
q.idoc_match(
markdown_match,
[e, q.IGNORE, document],
attribute="doc/markdown",
),
q.bind(
"idoc-get", install, document,
q.kw("guide"), q.kw("install"),
),
],
))
import { q } from "datalevin-node";
const e = q.var("e");
const title = q.var("title");
const document = q.var("doc");
const install = q.var("install");
const markdownMatch = new Map([
[q.kw("guide"), new Map([
[q.kw("install"), "Run clj -M:dev."]
])]
]);
const sections = await conn.query(q.query({
find: q.relation(title, install),
where: [
q.datom(e, "doc/title", title),
q.idocMatch(
markdownMatch,
[e, q.IGNORE, document],
{ attribute: "doc/markdown" }
),
q.bind(
"idoc-get", install, document,
q.kw("guide"), q.kw("install")
)
]
}));
For Markdown idoc values, query paths are normalized too. This means string paths that look like headings work:
(d/q '[:find ?e
:where
[(idoc-match $ :doc/markdown
{"Guide" {"Configure Search"
"Set :index-position? true for phrase search."}})
[[?e _ _]]]]
db)
Object matches = conn.query(
"[:find ?e " +
" :where [(idoc-match $ :doc/markdown " +
" {\"Guide\" {\"Configure Search\" " +
" \"Set :index-position? true for phrase search.\"}}) " +
" [[?e _ _]]]]");
from datalevin import q
e = q.var("e")
heading_match = {
"Guide": {
"Configure Search":
"Set :index-position? true for phrase search.",
},
}
matches = conn.query(q.query(
find=q.relation(e),
where=[q.idoc_match(
heading_match,
[e, q.IGNORE, q.IGNORE],
attribute="doc/markdown",
)],
))
import { q } from "datalevin-node";
const e = q.var("e");
const headingMatch = {
Guide: {
"Configure Search":
"Set :index-position? true for phrase search."
}
};
const matches = await conn.query(q.query({
find: q.relation(e),
where: [q.idocMatch(
headingMatch,
[e, q.IGNORE, q.IGNORE],
{ attribute: "doc/markdown" }
)]
}));
Use Markdown idoc values for section-level lookup over authored documents. If a section needs independent identity, permissions, relationships, or versioning, model it as normal component entities instead.
2.6 Patching Documents
Use :db.fn/patchIdoc for partial updates when you do not want to resend the whole document.
(d/transact! conn
[[:db.fn/patchIdoc 101 :user/metadata
[[:set [:profile :display-name] "Alice A."]
[:unset [:profile :middle]]
[:update [:tags] :conj "active"]]]])
conn.transact(Datalevin.tx()
.raw(Datalevin.edn(
"[:db.fn/patchIdoc 101 :user/metadata " +
" [[:set [:profile :display-name] \"Alice A.\"] " +
" [:unset [:profile :middle]] " +
" [:update [:tags] :conj \"active\"]]]")));
conn.transact(tx.data(tx.patch_idoc(
101,
"user/metadata",
[
tx.patch_set(["profile", "display-name"], "Alice A."),
tx.patch_unset(["profile", "middle"]),
tx.patch_update(["tags"], "conj", "active"),
],
)))
await conn.transact(tx.data(tx.patchIdoc(
101,
"user/metadata",
[
tx.patchSet(["profile", "display-name"], "Alice A."),
tx.patchUnset(["profile", "middle"]),
tx.patchUpdate(["tags"], "conj", "active")
]
)));
Patch paths are vectors of map keys and vector indices. Supported operations are:
:setsets the value at a path.:unsetremoves a map key or vector element.:updateapplies these operations::conjappends one or more values to a vector. If the path is absent, it starts from an empty vector.:mergemerges one or more maps into the map at the path. If the path is absent, it starts from an empty map.:assocsets one or more key/value pairs in the map at the path. If the path is absent, it starts from an empty map.:dissocremoves one or more keys from the map at the path. If the path is absent, it starts from an empty map.:incincrements an integer counter at the path. If the path is absent, it starts from0.:decdecrements an integer counter at the path. If the path is absent, it starts from0.
Patching cardinality-many documents needs the old value
For cardinality-many idoc attributes, provide the old document value so Datalevin can identify which value should be patched.
3. Querying Path-Indexed Documents
Use idoc-match inside Datalog to find entities with matching nested document content. It returns matching datoms as [e a v] triples, so the result can be joined with ordinary facts.
;; Find users with a dark theme in their metadata document.
(d/q '[:find ?e
:where [(idoc-match $ :user/metadata {:theme "dark"}) [[?e _ _]]]
:limit 20]
db)
conn.query("[:find ?e " +
" :where [(idoc-match $ :user/metadata {:theme \"dark\"}) [[?e _ _]]]" +
" :limit 20]");
from datalevin import q
e = q.var("e")
conn.query(q.query(
find=q.relation(e),
where=[q.idoc_match(
{"theme": "dark"},
[e, q.IGNORE, q.IGNORE],
attribute="user/metadata",
)],
limit=20,
))
import { q } from "datalevin-node";
const e = q.var("e");
await conn.query(q.query({
find: q.relation(e),
where: [q.idocMatch(
{ theme: "dark" },
[e, q.IGNORE, q.IGNORE],
{ attribute: "user/metadata" }
)],
limit: 20
}));
idoc-match provides no ranked order, so a limited result without :order-by has no ordering guarantee and an offset does not define stable pagination. Query-level :offset and :limit apply after surrounding joins and filters, not to raw document matches. Chapter 21, Section 1.4 explains when the optimizer can avoid reading the entire matching relation.
Nested maps express nested path matches:
(d/q '[:find ?e
:where [(idoc-match $ :user/metadata
{:profile {:age 30 :name "Alice"}})
[[?e _ _]]]]
db)
conn.query("[:find ?e " +
" :where [(idoc-match $ :user/metadata " +
" {:profile {:age 30 :name \"Alice\"}}) [[?e _ _]]]]");
from datalevin import q
e = q.var("e")
profile_match = {
"profile": {
"age": 30,
"name": "Alice",
},
}
conn.query(q.query(
find=q.relation(e),
where=[q.idoc_match(
profile_match,
[e, q.IGNORE, q.IGNORE],
attribute="user/metadata",
)],
))
import { q } from "datalevin-node";
const e = q.var("e");
const profileMatch = {
profile: { age: 30, name: "Alice" }
};
await conn.query(q.query({
find: q.relation(e),
where: [q.idocMatch(
profileMatch,
[e, q.IGNORE, q.IGNORE],
{ attribute: "user/metadata" }
)]
}));
Vectors are treated as arrays. A match succeeds if any element matches:
(d/q '[:find ?e
:where [(idoc-match $ :user/metadata {:tags "admin"}) [[?e _ _]]]]
db)
conn.query("[:find ?e " +
" :where [(idoc-match $ :user/metadata {:tags \"admin\"}) [[?e _ _]]]]");
from datalevin import q
e = q.var("e")
conn.query(q.query(
find=q.relation(e),
where=[q.idoc_match(
{"tags": "admin"},
[e, q.IGNORE, q.IGNORE],
attribute="user/metadata",
)],
))
import { q } from "datalevin-node";
const e = q.var("e");
await conn.query(q.query({
find: q.relation(e),
where: [q.idocMatch(
{ tags: "admin" },
[e, q.IGNORE, q.IGNORE],
{ attribute: "user/metadata" }
)]
}));
3.1 Logical Combinators
Use [:and ...], [:or ...], and [:not ...] inside query maps:
(d/q '[:find ?e
:where [(idoc-match $ :user/metadata
{:profile [:or {:age 30} {:age 40}]})
[[?e _ _]]]]
db)
conn.query("[:find ?e " +
" :where [(idoc-match $ :user/metadata " +
" {:profile [:or {:age 30} {:age 40}]}) " +
" [[?e _ _]]]]");
from datalevin import q
e = q.var("e")
age_30_or_40 = {
"profile": [
q.kw("or"),
{"age": 30},
{"age": 40},
],
}
conn.query(q.query(
find=q.relation(e),
where=[q.idoc_match(
age_30_or_40,
[e, q.IGNORE, q.IGNORE],
attribute="user/metadata",
)],
))
import { q } from "datalevin-node";
const e = q.var("e");
const age30Or40 = {
profile: [
q.kw("or"),
{ age: 30 },
{ age: 40 }
]
};
await conn.query(q.query({
find: q.relation(e),
where: [q.idocMatch(
age30Or40,
[e, q.IGNORE, q.IGNORE],
{ attribute: "user/metadata" }
)]
}));
3.2 Predicates and Ranges
Predicates can appear inline in query maps, or as path predicates supplied as query inputs:
;; Inline predicate.
(d/q '[:find ?e
:where [(idoc-match $ :user/metadata {:profile {:age (> 21)}})
[[?e _ _]]]]
db)
;; Path predicate supplied as data.
(d/q '[:find ?e
:in $ ?q
:where [(idoc-match $ :user/metadata ?q) [[?e _ _]]]]
db
'(>= [:profile :age] 30))
;; Range via multi-arity comparison.
(d/q '[:find ?e
:in $ ?q
:where [(idoc-match $ :user/metadata ?q) [[?e _ _]]]]
db
'(< 20 [:profile :age] 40))
// Inline predicate.
conn.query("[:find ?e " +
" :where [(idoc-match $ :user/metadata " +
" {:profile {:age (> 21)}}) [[?e _ _]]]]");
// Path predicate supplied as data.
conn.query("[:find ?e " +
" :in $ ?q " +
" :where [(idoc-match $ :user/metadata ?q) [[?e _ _]]]]",
Datalevin.edn("(>= [:profile :age] 30)"));
// Range via multi-arity comparison.
conn.query("[:find ?e " +
" :in $ ?q " +
" :where [(idoc-match $ :user/metadata ?q) [[?e _ _]]]]",
Datalevin.edn("(< 20 [:profile :age] 40)"));
from datalevin import q
e = q.var("e")
match_query = q.var("q")
hits = q.relation_binding(e, q.IGNORE, q.IGNORE)
greater_than_21 = q.edn_list(q.sym(">"), 21)
# Inline predicate.
conn.query(q.query(
find=q.relation(e),
where=[q.idoc_match(
{"profile": {"age": greater_than_21}},
hits,
attribute="user/metadata",
)],
))
idoc_query = q.query(
find=q.relation(e),
inputs=[q.DB, match_query],
where=[q.idoc_match(
match_query, hits, attribute="user/metadata",
)],
)
# Path predicate supplied as data.
conn.query(
idoc_query,
q.edn_list(q.sym(">="), ["profile", "age"], 30),
)
# Range via multi-arity comparison.
conn.query(
idoc_query,
q.edn_list(q.sym("<"), 20, ["profile", "age"], 40),
)
import { q } from "datalevin-node";
const e = q.var("e");
const matchQuery = q.var("q");
const hits = q.relationBinding(e, q.IGNORE, q.IGNORE);
const greaterThan21 = q.ednList(q.sym(">"), 21);
// Inline predicate.
await conn.query(q.query({
find: q.relation(e),
where: [q.idocMatch(
{ profile: { age: greaterThan21 } },
hits,
{ attribute: "user/metadata" }
)]
}));
const idocQuery = q.query({
find: q.relation(e),
inputs: [q.DB, matchQuery],
where: [q.idocMatch(
matchQuery, hits, { attribute: "user/metadata" }
)]
});
// Path predicate supplied as data.
await conn.query(
idocQuery,
q.ednList(q.sym(">="), ["profile", "age"], 30)
);
// Range via multi-arity comparison.
await conn.query(
idocQuery,
q.ednList(q.sym("<"), 20, ["profile", "age"], 40)
);
Supported predicates are nil?, >, >=, <, and <=. These predicate forms are EDN lists, not vectors. In Python and JavaScript, q.edn_list/q.ednList preserves that distinction without parsing an EDN string; use q.sym for the predicate operator. q.expression is for expressions in Datalog query grammar.
3.3 Wildcard Paths
Use wildcard path segments when the exact nested key is not known:
;; Match any single key under :profile with value >= 30.
(d/q '[:find ?e
:in $ ?q
:where [(idoc-match $ :user/metadata ?q) [[?e _ _]]]]
db
'(>= [:profile :?] 30))
;; Match a name field at any depth.
(d/q '[:find ?e
:where [(idoc-match $ :user/metadata {:* {:name "Alice"}})
[[?e _ _]]]]
db)
// Match any single key under profile with value >= 30.
conn.query("[:find ?e " +
" :in $ ?q " +
" :where [(idoc-match $ :user/metadata ?q) [[?e _ _]]]]",
Datalevin.edn("(>= [:profile :?] 30)"));
// Match a name field at any depth.
conn.query("[:find ?e " +
" :where [(idoc-match $ :user/metadata {:* {:name \"Alice\"}}) " +
" [[?e _ _]]]]");
from datalevin import q
e = q.var("e")
match_query = q.var("q")
hits = q.relation_binding(e, q.IGNORE, q.IGNORE)
idoc_query = q.query(
find=q.relation(e),
inputs=[q.DB, match_query],
where=[q.idoc_match(
match_query, hits, attribute="user/metadata",
)],
)
# Match any single key under profile with value >= 30.
conn.query(
idoc_query,
q.edn_list(q.sym(">="), ["profile", q.kw("?")], 30),
)
# Match a name field at any depth.
conn.query(q.query(
find=q.relation(e),
where=[q.idoc_match(
{q.kw("*"): {"name": "Alice"}},
hits,
attribute="user/metadata",
)],
))
import { q } from "datalevin-node";
const e = q.var("e");
const matchQuery = q.var("q");
const hits = q.relationBinding(e, q.IGNORE, q.IGNORE);
const idocQuery = q.query({
find: q.relation(e),
inputs: [q.DB, matchQuery],
where: [q.idocMatch(
matchQuery, hits, { attribute: "user/metadata" }
)]
});
// Match any single key under profile with value >= 30.
await conn.query(
idocQuery,
q.ednList(q.sym(">="), ["profile", q.kw("?")], 30)
);
// Match a name field at any depth.
await conn.query(q.query({
find: q.relation(e),
where: [q.idocMatch(
new Map([[q.kw("*"), { name: "Alice" }]]),
hits,
{ attribute: "user/metadata" }
)]
}));
:? matches exactly one path segment. :* matches any depth, including zero segments.
3.4 Null Matching
To match null values, use (nil?):
(d/q '[:find ?e
:where [(idoc-match $ :user/raw-json {"middle" (nil?)}) [[?e _ _]]]]
db)
conn.query("[:find ?e " +
" :where [(idoc-match $ :user/raw-json {\"middle\" (nil?)}) " +
" [[?e _ _]]]]");
from datalevin import q
e = q.var("e")
nil_predicate = q.edn_list(q.sym("nil?"))
conn.query(q.query(
find=q.relation(e),
where=[q.idoc_match(
{"middle": nil_predicate},
q.relation_binding(e, q.IGNORE, q.IGNORE),
attribute="user/raw-json",
)],
))
import { q } from "datalevin-node";
const e = q.var("e");
const nilPredicate = q.ednList(q.sym("nil?"));
await conn.query(q.query({
find: q.relation(e),
where: [q.idocMatch(
{ middle: nilPredicate },
q.relationBinding(e, q.IGNORE, q.IGNORE),
{ attribute: "user/raw-json" }
)]
}));
Match JSON keys as strings
JSON keys are strings, so JSON documents should be matched with "middle", not :middle.
3.5 Extracting Values with idoc-get
idoc-get extracts a nested value from an idoc document inside a Datalog query:
(d/q '[:find ?e ?email
:where [(idoc-match $ :user/metadata {:theme "dark"}) [[?e _ ?doc]]]
[(idoc-get ?doc :contact :email) ?email]]
db)
conn.query("[:find ?e ?email " +
" :where [(idoc-match $ :user/metadata {:theme \"dark\"}) [[?e _ ?doc]]] " +
" [(idoc-get ?doc :contact :email) ?email]]");
from datalevin import q
e, document, email = map(q.var, ("e", "doc", "email"))
conn.query(q.query(
find=q.relation(e, email),
where=[
q.idoc_match(
{"theme": "dark"},
[e, q.IGNORE, document],
attribute="user/metadata",
),
q.bind(
"idoc-get", email, document,
"contact", "email",
),
],
))
import { q } from "datalevin-node";
const e = q.var("e");
const document = q.var("doc");
const email = q.var("email");
await conn.query(q.query({
find: q.relation(e, email),
where: [
q.idocMatch(
{ theme: "dark" },
[e, q.IGNORE, document],
{ attribute: "user/metadata" }
),
q.bind(
"idoc-get", email, document,
"contact", "email"
)
]
}));
When a path traverses arrays, idoc-get returns a vector of matching values.
4. How idoc Indexes Work
Datalevin stores the idoc value itself as the datom value. The path index records enough path and value information to narrow candidates without duplicating full documents. Each domain has a separate path index, and indexing is synchronous during transactions. Unlike full-text and vector indexes, idoc has no async indexing mode. After a transaction commits, idoc-match sees the updated document index immediately.
idoc path indexes do not split the fact
idoc gives you indexed path filters, but the document remains one value from the perspective of ordinary datoms. If a nested field deserves identity, constraints, independent history, or joins as a first-class fact, model that part with normal attributes instead.
4.1 Benchmark Snapshot
The September 2026 indexed-document benchmark compares Datalevin idoc with PostgreSQL JSONB, SQLite JSON1, and MongoDB. It uses 10,000 synthetic documents, 10,000 measured operations, a fixed mix of nested equality, range, wildcard, and array queries, and aligned strict durability settings. Correctness is checked against an independent oracle, and the reported pass runs in a separate JVM from the full warmup pass. Measurements were collected on a 12-core Apple Silicon Mac with Datalevin 1.1.0, PostgreSQL 18.4, SQLite 3.51.1, and MongoDB 7.0.35.
The single-thread results below are completed operations per second. Workload A mixes reads and updates, C is read-only, and F uses read-modify-write.
| Workload | Datalevin | PostgreSQL | SQLite | MongoDB |
|---|---|---|---|---|
| A | 3,358/s | 2,270/s | 438/s | 309/s |
| C | 11,454/s | 2,039/s | 437/s | 2,868/s |
| F | 2,680/s | 1,789/s | 416/s | 192/s |
Datalevin led all three single-thread workloads. On the C run, Datalevin's median latency across the five document-query shapes ranged from 0.051 to 0.328 ms.
These results show the benefit of path indexes for this document and query mix; they are not a comparison between idoc and normalized Datalevin datoms, and they do not replace application-specific measurement. Choose between the two Datalevin models by the semantic trade-offs in the next section.
5. Choosing Between Logical Documents and idoc
| Feature | Logical document (:db/isComponent) |
Native idoc |
|---|---|---|
| Storage | Decomposed into many datoms | Stored as one datom value |
| Schema | Attributes are declared explicitly | Nested fields can evolve freely |
| Indexing | Standard EAV/AVE and attribute indexes | Path-level indexing, all leaf paths by default or selected prefixes |
| Querying | Datalog joins, pull, entity API | idoc-match and idoc-get |
| Updates | Fine-grained datom updates | patchIdoc or full replacement |
| Best for | Core domain state and relationships | Flexible metadata and imported documents |
Use logical documents when nested data needs type checking, referential integrity, joins to other entities, or independent updates as facts. Use idoc when nested data is sparse, user-defined, imported from another document store, or expected to change shape frequently.
These models can coexist on the same entity. A product can have structured facts for price and inventory, component entities for variants, full-text fields for descriptions, and an idoc attribute for merchant-specific metadata.
6. Ad Hoc Queries over Nested Data
Sometimes you do not want to persist a nested value at all. You have a map from an API response, a fixture, or a one-off data export, and you want Datalog's joins and predicates for exploration. Chapter 8 showed that Datalevin can query an in-memory sequence of tuples. A small boundary helper can turn a nested value into such a relation.
The following helpers walk maps/objects and vectors/arrays and emit one tuple for each leaf value. By default, the tuple is the path segments followed by the leaf value.
(defn leaf-paths
"Returns paths to scalar leaves in a nested map/vector."
([root]
{:pre [(or (map? root) (vector? root))]}
(leaf-paths [] root))
([parent x]
(cond
(map? x)
(mapcat (fn [[k v]]
(leaf-paths (conj parent k) v))
x)
(vector? x)
(mapcat (fn [[i v]]
(leaf-paths (conj parent i) v))
(map-indexed vector x))
:else
[parent])))
(defn nested->tuples
"Converts nested maps/vectors into tuples queryable by Datalog."
([x]
(nested->tuples x nil))
([x {:keys [paths?] :or {paths? false}}]
(with-meta
(mapv (fn [path]
(let [tuple (conj path (get-in x path))]
(if paths?
(into [path] tuple)
tuple)))
(leaf-paths x))
{:datalevin.docs/original x})))
static List<List<Object>> leafPaths(Object root) {
return leafPaths(List.of(), root);
}
static List<List<Object>> leafPaths(List<Object> parent, Object x) {
if (x instanceof Map<?, ?> m) {
List<List<Object>> paths = new ArrayList<>();
for (Map.Entry<?, ?> entry : m.entrySet()) {
ArrayList<Object> next = new ArrayList<>(parent);
next.add(entry.getKey());
paths.addAll(leafPaths(next, entry.getValue()));
}
return paths;
}
if (x instanceof List<?> xs) {
List<List<Object>> paths = new ArrayList<>();
for (int i = 0; i < xs.size(); i++) {
ArrayList<Object> next = new ArrayList<>(parent);
next.add(i);
paths.addAll(leafPaths(next, xs.get(i)));
}
return paths;
}
return List.of(new ArrayList<>(parent));
}
static Object getIn(Object root, List<Object> path) {
Object cur = root;
for (Object step : path) {
if (cur instanceof Map<?, ?> m) {
cur = m.get(step);
} else if (cur instanceof List<?> xs) {
cur = xs.get(((Number) step).intValue());
} else {
throw new IllegalArgumentException("Cannot traverse " + step);
}
}
return cur;
}
static List<List<Object>> nestedToTuples(Object x, boolean paths) {
List<List<Object>> out = new ArrayList<>();
for (List<Object> path : leafPaths(x)) {
ArrayList<Object> tuple = new ArrayList<>(path);
tuple.add(getIn(x, path));
if (paths) {
ArrayList<Object> withPath = new ArrayList<>();
withPath.add(path);
withPath.addAll(tuple);
out.add(withPath);
} else {
out.add(tuple);
}
}
return out;
}
def leaf_paths(root, path=()):
if isinstance(root, dict):
paths = []
for key, value in root.items():
paths.extend(leaf_paths(value, (*path, key)))
return paths
if isinstance(root, list):
paths = []
for i, value in enumerate(root):
paths.extend(leaf_paths(value, (*path, i)))
return paths
return [list(path)]
def get_in(root, path):
cur = root
for step in path:
cur = cur[step]
return cur
def nested_to_tuples(x, paths=False):
tuples = []
for path in leaf_paths(x):
tuple_ = [*path, get_in(x, path)]
tuples.append([path, *tuple_] if paths else tuple_)
return tuples
function leafPaths(root, path = []) {
if (Array.isArray(root)) {
return root.flatMap((value, i) => leafPaths(value, [...path, i]));
}
if (root !== null && typeof root === "object") {
return Object.entries(root).flatMap(([key, value]) =>
leafPaths(value, [...path, key]));
}
return [path];
}
function getIn(root, path) {
return path.reduce((cur, step) => cur[step], root);
}
function nestedToTuples(x, { paths = false } = {}) {
return leafPaths(x).map((path) => {
const tuple = [...path, getIn(x, path)];
return paths ? [path, ...tuple] : tuple;
});
}
For example:
(def order-doc
{:order/id "o-1001"
:customer {:email "ada@example.com"
:name "Ada"}
:lines [{:sku "book" :qty 2}
{:sku "pen" :qty 5}]})
(def tuples (nested->tuples order-doc))
;; => [[:order/id "o-1001"]
;; [:customer :email "ada@example.com"]
;; [:customer :name "Ada"]
;; [:lines 0 :sku "book"]
;; [:lines 0 :qty 2]
;; [:lines 1 :sku "pen"]
;; [:lines 1 :qty 5]]
Map<Object, Object> orderDoc = Map.of(
Datalevin.kw(":order/id"), "o-1001",
Datalevin.kw(":customer"), Map.of(
Datalevin.kw(":email"), "ada@example.com",
Datalevin.kw(":name"), "Ada"),
Datalevin.kw(":lines"), List.of(
Map.of(Datalevin.kw(":sku"), "book",
Datalevin.kw(":qty"), 2),
Map.of(Datalevin.kw(":sku"), "pen",
Datalevin.kw(":qty"), 5)));
List<List<Object>> tuples = nestedToTuples(orderDoc, false);
// => [[:order/id "o-1001"]
// [:customer :email "ada@example.com"]
// [:customer :name "Ada"]
// [:lines 0 :sku "book"]
// [:lines 0 :qty 2]
// [:lines 1 :sku "pen"]
// [:lines 1 :qty 5]]
order_doc = {
":order/id": "o-1001",
":customer": {":email": "ada@example.com",
":name": "Ada"},
":lines": [{":sku": "book", ":qty": 2},
{":sku": "pen", ":qty": 5}]}
tuples = nested_to_tuples(order_doc)
# => [[":order/id", "o-1001"],
# [":customer", ":email", "ada@example.com"],
# [":customer", ":name", "Ada"],
# [":lines", 0, ":sku", "book"],
# [":lines", 0, ":qty", 2],
# [":lines", 1, ":sku", "pen"],
# [":lines", 1, ":qty", 5]]
const orderDoc = {
":order/id": "o-1001",
":customer": {":email": "ada@example.com",
":name": "Ada"},
":lines": [{":sku": "book", ":qty": 2},
{":sku": "pen", ":qty": 5}]
};
const tuples = nestedToTuples(orderDoc);
// => [[":order/id", "o-1001"],
// [":customer", ":email", "ada@example.com"],
// [":customer", ":name", "Ada"],
// [":lines", 0, ":sku", "book"],
// [":lines", 0, ":qty", 2],
// [":lines", 1, ":sku", "pen"],
// [":lines", 1, ":qty", 5]]
Now the vector index in the path can be used as a join key:
(d/q '[:find ?sku ?qty
:where
[:lines ?i :sku ?sku]
[:lines ?i :qty ?qty]]
tuples)
;; => #{["book" 2] ["pen" 5]}
Object lines = conn.query(
"[:find ?sku ?qty " +
" :in $ $data " +
" :where [$data :lines ?i :sku ?sku] " +
" [$data :lines ?i :qty ?qty]]",
tuples);
// => [["book", 2], ["pen", 5]]
from datalevin import q
data = q.source("data")
index, sku, quantity = map(q.var, ("i", "sku", "qty"))
lines = conn.query(q.query(
find=q.relation(sku, quantity),
inputs=[q.DB, data],
where=[
q.pattern(q.kw("lines"), index, q.kw("sku"), sku,
from_source=data),
q.pattern(q.kw("lines"), index, q.kw("qty"), quantity,
from_source=data),
],
), tuples)
# => [["book", 2], ["pen", 5]]
import { q } from "datalevin-node";
const data = q.source("data");
const index = q.var("i");
const sku = q.var("sku");
const quantity = q.var("qty");
const lines = await conn.query(q.query({
find: q.relation(sku, quantity),
inputs: [q.DB, data],
where: [
q.patternFrom(data, q.kw("lines"), index, q.kw("sku"), sku),
q.patternFrom(data, q.kw("lines"), index, q.kw("qty"), quantity)
]
}), tuples);
// => [["book", 2], ["pen", 5]]
When the paths option is true, the full path is also carried as the first element of each tuple. This is useful when a query result should drive a later path lookup or update:
(def tuples-with-paths (nested->tuples order-doc {:paths? true}))
(d/q '[:find ?path ?sku
:where
[?path :lines ?i :sku ?sku]]
tuples-with-paths)
;; => #{[[:lines 0 :sku] "book"]
;; [[:lines 1 :sku] "pen"]}
List<List<Object>> tuplesWithPaths = nestedToTuples(orderDoc, true);
Object paths = conn.query(
"[:find ?path ?sku " +
" :in $ $data " +
" :where [$data ?path :lines ?i :sku ?sku]]",
tuplesWithPaths);
// => [[[":lines", 0, ":sku"], "book"],
// [[":lines", 1, ":sku"], "pen"]]
tuples_with_paths = nested_to_tuples(order_doc, paths=True)
from datalevin import q
data = q.source("data")
path, index, sku = map(q.var, ("path", "i", "sku"))
paths = conn.query(q.query(
find=q.relation(path, sku),
inputs=[q.DB, data],
where=[q.pattern(
path, q.kw("lines"), index, q.kw("sku"), sku,
from_source=data,
)],
), tuples_with_paths)
# => [[[":lines", 0, ":sku"], "book"],
# [[":lines", 1, ":sku"], "pen"]]
const tuplesWithPaths = nestedToTuples(orderDoc, {paths: true});
import { q } from "datalevin-node";
const data = q.source("data");
const path = q.var("path");
const index = q.var("i");
const sku = q.var("sku");
const paths = await conn.query(q.query({
find: q.relation(path, sku),
inputs: [q.DB, data],
where: [q.patternFrom(
data, path, q.kw("lines"), index, q.kw("sku"), sku
)]
}), tuplesWithPaths);
// => [[[":lines", 0, ":sku"], "book"],
// [[":lines", 1, ":sku"], "pen"]]
Ad hoc tuple views are not persistence
This technique is for ad hoc analysis and local transformation. It does not create attributes, schema, indexes, lookup refs, or durable entities. If the nested data is part of the application's long-lived domain model, turn it into ordinary Datalevin facts or component entities. If it is flexible document data that should be stored and searched by path, use :db.type/idoc.
Summary
Datalevin lets you choose a representation for each part of your data. Use logical documents for structured domain state where facts, relationships, and constraints matter. Use idoc for flexible document-shaped data that should be stored as one value but queried by path. By default, idoc indexes every scalar leaf path; use selective path indexing when only a few prefixes should be searchable or when large user-defined documents would otherwise create more index surface area than the application needs. For one-off nested values, convert them to tuples and query them in memory instead of creating durable schema or idoc indexes. Both stored models participate in Datalog, so you can join document filters with graph, relational, full-text, and vector queries in one transactionally consistent database. For eligible finite relation queries, the optimizer can page idoc-match incrementally and stop after enough final rows survive the surrounding logic.
User Examples
Log in to create examplesNo examples for this chapter yet.
