DatalevinDatalevin
Part I - Foundations: A Fact-First Database

Chapter 3: Datalevin Mental Model

Transitioning to Datalevin often requires a straightforward but fundamental shift in how you think about data. If you are coming from a SQL or document database background, your mental model likely revolves around "containers": table rows with fixed columns or documents with nested structures. In these systems, the container is the primary unit of truth.

Datalevin asks you to look deeper, past the containers, to the atomic facts themselves. Moving from SQL to Datalevin is less like moving between different spreadsheet programs and more like moving from a spreadsheet to a knowledge graph: a network of things and relationships represented as facts. This chapter provides the narrative framework to help you navigate this shift.

The chapter moves through four stages:

1. Think in Datoms, Not Containers

In traditional databases, if you want to record that Alice's email address is alice@example.com, you must first decide which "box" that fact belongs in. You might create a Users table and add a row for Alice. The fact of her email address is then trapped within the context of that row and those other columns.

At the conceptual level, Datalevin discards these rigid boxes. Instead, it stores data as a collection of atomic facts called datoms. Each datom is a simple statement of truth represented as a triple: [entity attribute value].

From a traditional perspective, e and a are the two coordinates in a two-dimensional table view of values, and the value v is contained in the cell identified by the coordinates.

However, that view is limited to a single table. A better view is to imagine a giant ledger where every single cell of every spreadsheet in your entire company is cut out and pasted as an individual line. Each line tells you exactly which "row" it came from (Entity), what "column" it represents (Attribute), and what the "value" is.

This fact-first approach is powerful because it is additive. To add a new piece of information, like a user's preferred language, you do not need to "alter a table." You just add another datom: [101 :user/preferred-language "en"].

2. Attributes Over Tables: The Fluid Entity

Because facts are stored individually, the concept of a "table" disappears. In its place is attribute-centric modeling: you describe the behavior of each attribute instead of defining a fixed row shape for each entity type.

In SQL, the schema belongs to the table: a users table might declare columns such as id, email, and display_name, while an orders table declares columns such as id, customer_id, status, and total. In Datalevin, the schema belongs to the attribute. A schema is a map from attribute names to behaviors: value type, cardinality, uniqueness, full-text indexing for searching text, references to other entities, and other properties. You define what a :user/email is, for example, a string and perhaps a value that must be unique, but you do not define what a "User" is in a rigid sense.

Three schema terms appear often:

To update a value, for example, Alice's email address, you do not modify a row; you transact a new fact about the same attribute of the same entity. For a cardinality-one attribute, the new value replaces the old value for that entity and attribute. For a cardinality-many attribute, the new value is added to the entity's set of values. Uniqueness is a separate concern: it prevents two entities from claiming the same unique value and lets you look up an entity by a domain value such as an email address.

As illustrated in Figure 3.1, an entity in Datalevin is just a collection of datoms that happen to share the same entity id. A single id could have attributes associated with a :user, a :customer, and an :employee simultaneously. You do not "join" separate tables; you simply query for an id that possesses all those attributes.

One entity as many datoms: a row or document is really the set of datoms that share an entity id

This open-ended shape lets your data model evolve alongside your application. You never have to worry about "breaking the schema" by adding a new attribute to an existing entity, because there is no rigid "row" to break. Entity types are conventions you create by naming your attributes, such as the person/ namespace in :person/name; they are not constraints enforced by the storage engine.

3. Data-Oriented Application Design

The previous two sections describe Datalevin's database model, but they also point to a broader application design principle: data-oriented programming. In this style, the essential state of the application is represented as plain, inspectable data, and behavior is expressed as functions and queries over that data rather than as hidden mutation inside object graphs [1].

Datalevin is a natural fit for this style. Schemas are maps. Transactions are data. Queries are data. Rules are data. Entity maps are data. Even a relationship is not an object pointer with private behavior; it is an explicit fact whose value is another entity id. That means the important parts of the application can be printed, logged, diffed, tested, stored, transmitted, and reasoned about with ordinary tools.

This does not mean applications have no behavior. It means behavior stays at the edges as named operations over data:

This is why later chapters repeatedly prefer data shapes over framework machinery. A transaction map is easier to inspect than an object with many mutating methods. A Datalog query is easier to test than a traversal hidden in a service class. A fact with provenance is easier to audit than a cached field whose origin is unclear. Part VI applies the same principle to agent memory: episodes, facts, tasks, evidence, and working-memory slots are ordinary entities and attributes, not a special memory subsystem.

The practical payoff is not just aesthetic. Data-oriented programs are easier to debug, migrate, replay, and explain because the application state has a durable, queryable shape. Datalevin gives that style a database substrate: facts remain small, attributes remain additive, and queries compose across domains.

4. Operational Basics: Transactions and Tempids

This section switches from motivation to mechanics. It is not the full transaction API; Chapter 6 covers that. The goal here is to give you enough vocabulary to understand how facts enter the database and how new entities can refer to each other in one atomic change.

A database is not only a place to read facts. It is also a system for changing facts without leaving the application in a half-updated state. A transaction is a single atomic unit of change: all changes in the transaction succeed together, or none are applied. Transaction processing is the database discipline of making those changes atomic, isolated, and durable while many users or programs read and write concurrently [2].

A deeper motivation for transactions is that a database acts as a surrogate for the part of the external world your application cares about. Real-world events do not happen halfway. A customer either placed an order or did not. A payment either settled or did not. A job either moved from pending to running or it did not. The database may need several internal writes to record one such event, but the application wants the event itself to appear as one coherent change.

This is the core workload of Online Transaction Processing (OLTP) systems: applications record orders, messages, jobs, profile edits, tool results, and other current operational state. OLTP databases are usually optimized for many small concurrent reads and writes over current state, not for large historical scans or immutable log analysis. Datalevin is an OLTP database.

In Datalevin, a transaction is the boundary around assertions and retractions. An assertion adds a fact; a retraction removes a fact. One transaction can create entities, update cardinality-one attributes, add cardinality-many values, retract facts, and update custom key-value data in the same Datalevin store. After the transaction commits, readers see a consistent database state.

A small example makes this concrete. In transaction data, :db/id identifies the entity being written. If its value is a negative integer or string, Datalevin treats it as a temporary id, or tempid, and replaces it with a permanent entity id when the transaction commits. In the example below, MIT uses tempid -10 because Alice's :person/school value points to it. Alice does not need :db/id because no other new entity refers to Alice.

;; The schema defines the behavior of attributes, not the shape of rows.
(def schema
  {:person/name    {:db/valueType :db.type/string}
   :person/email   {:db/valueType :db.type/string}
   :person/school  {:db/valueType :db.type/ref} ; A reference to the id of another entity
   :school/name    {:db/valueType :db.type/string}
   :school/country {:db/valueType :db.type/string}})

;; Data is written as a collection of entities here.
;; Notice how 'Alice' and 'MIT' are just sets of attributes.
(d/transact! conn
  [{:person/name "Alice" :person/email "alice@example.com" :person/school -10}
   {:db/id -10 :school/name "MIT" :school/country "USA"}])
// The schema defines the behavior of attributes, not the shape of rows.
Schema schema = Datalevin.schema()
    .attr(":person/name",
          Schema.attribute().valueType(Schema.ValueType.STRING))
    .attr(":person/email",
          Schema.attribute().valueType(Schema.ValueType.STRING))
    .attr(":person/school",
          Schema.attribute().valueType(Schema.ValueType.REF)) // A reference to the id of another entity
    .attr(":school/name",
          Schema.attribute().valueType(Schema.ValueType.STRING))
    .attr(":school/country",
          Schema.attribute().valueType(Schema.ValueType.STRING));

// Data is written as a collection of entities.
// Notice how 'Alice' and 'MIT' are just sets of attributes.
conn.transact(Datalevin.tx()
    .entity(Tx.entity()
        .put(":person/name", "Alice")
        .put(":person/email", "alice@example.com")
        .put(":person/school", -10))
    .entity(Tx.entity(-10)
        .put(":school/name", "MIT")
        .put(":school/country", "USA")));
# The schema defines the behavior of attributes, not the shape of rows.
schema = {
    ":person/name":    {":db/valueType": ":db.type/string"},
    ":person/email":   {":db/valueType": ":db.type/string"},
    ":person/school":  {":db/valueType": ":db.type/ref"},  # A reference to the id of another entity
    ":school/name":    {":db/valueType": ":db.type/string"},
    ":school/country": {":db/valueType": ":db.type/string"}}

# Data is written as a collection of entities.
# Notice how 'Alice' and 'MIT' are just sets of attributes.
conn.transact([
    {":person/name": "Alice",
     ":person/email": "alice@example.com",
     ":person/school": -10},
    {":db/id": -10,
     ":school/name": "MIT",
     ":school/country": "USA"}])
// The schema defines the behavior of attributes, not the shape of rows.
const schema = {
  ":person/name":    { ":db/valueType": ":db.type/string" },
  ":person/email":   { ":db/valueType": ":db.type/string" },
  ":person/school":  { ":db/valueType": ":db.type/ref" },  // A reference to the id of another entity
  ":school/name":    { ":db/valueType": ":db.type/string" },
  ":school/country": { ":db/valueType": ":db.type/string" }
};

// Data is written as a collection of entities.
// Notice how 'Alice' and 'MIT' are just sets of attributes.
await conn.transact([
  { ":person/name": "Alice",
    ":person/email": "alice@example.com",
    ":person/school": -10 },
  { ":db/id": -10,
    ":school/name": "MIT",
    ":school/country": "USA" }
]);

Tempids are local to one transaction. In map-style transaction data, if the transaction does not need to connect new entities to one another, :db/id can be omitted and Datalevin will assign permanent entity ids automatically.

Warning

Do not put application identifiers in :db/id. A permanent Datalevin entity id is a system-managed long integer, not a UUID, email, slug, or external primary key.

The value is local to one database: two databases with the same application data may assign different eids. Use a separate unique identity attribute, such as :user/id or :user/email, for application identity. Application code can then address the entity with a lookup ref instead of depending on the internal id.

Chapter 6 covers transaction data shapes, transaction reports, listeners, transaction functions, and durability settings in detail. For now, keep one mental model in view: Datalevin changes the database in coherent committed steps, not by mutating one fact at a time in isolation.

With the basic write vocabulary in place, the modeling habit worth practicing is to think in datoms: store each durable fact in one place, then connect entities with references. Suppose a customer places many orders. You could copy the customer's email address into every order, but then one email change would require many writes. In Datalevin, you usually store the customer's email once on the customer entity, and store a reference from each order to that customer.

The connection is itself a fact. If entity 1001 is a customer and entity 2001 is an order, the relationship can be represented as another datom:

The third datom says that order 2001 points to customer 1001. Nothing is embedded inside the order, and the customer's email is not copied into the order. The facts stay separate, but the shared entity id connects the order facts to the customer facts.

Database terminology calls this normalization [3]: representing each durable fact once and connecting related entities by references. The opposite is denormalization: copying or nesting the same information into several larger records so each read can fetch a preassembled shape. Both styles have uses. If an order should always show the customer's current email address, store the email on the customer entity and link the order to that customer. If the order must preserve the email address used at purchase time, store that value on the order as its own historical fact. Datalevin's fact model makes the normalized style natural, while still letting you store intentional snapshots when the domain needs them.

The database term for following this kind of connection is a join. A join combines facts that share a value. In this case, a query can read the :order/customer value from the order, treat that value as the customer entity id, and then read the customer's email from that entity.

In Datalevin, all facts are stored in a unified set of indexes. An index is a sorted data access path optimized for a particular lookup order. Joining an Order to a Customer is a matter of looking up datoms that share a value.

This is why storing facts once works well in Datalevin. Updates stay granular: you can update a customer's email without touching thousands of order records. Indexes stay useful because every datom participates in the core indexes, and AVE indexing is automatic for every attribute. When each fact is stored once, the engine can use its EAV (Entity-Attribute-Value) and AVE (Attribute-Value-Entity) indexes to jump directly to the relevant facts.

Query planning also gets clearer. Datalevin's cost-based optimizer estimates how many datoms each query clause might match, then chooses an execution order. Storing each fact once gives the optimizer a cleaner picture of how to order joins for speed. This estimate is also called cardinality in query planning; do not confuse it with the schema property :db/cardinality, covered in Chapter 5's attribute properties section. For an example of how this affects many-join query performance, see Chapter 1's "Table Storage Makes Query Planning Harder."

6. Datalog: Querying as Logic Programming

The next three sections are query intuition, not a complete language reference (see Chapter 8). They show how the fact model affects ordinary reads, search, and reusable query logic.

If SQL is like giving the database a set of instructions on how to assemble a result table, Datalog is like giving the database a description of the answer you want. That declarative style comes from logic programming [4]: programs are expressed as logical clauses, and evaluation searches for values that make those clauses true.

You are not telling the engine to "JOIN table A with table B." You are stating: "I am looking for an entity ?user-eid that has attribute :user/name with value 'Alice', and that same entity ?user-eid must also have an attribute :user/email whose value is ?email."

(d/q '[:find ?email
       :where
       [?user-eid :user/name "Alice"]
       [?user-eid :user/email ?email]]
     db)
Object results = conn.query(
    "[:find ?email " +
    " :where " +
    " [?user-eid :user/name \"Alice\"] " +
    " [?user-eid :user/email ?email]]");
results = conn.query("""
    [:find ?email
     :where
     [?user-eid :user/name "Alice"]
     [?user-eid :user/email ?email]]""")
const results = await conn.query(
  `[:find ?email
    :where
    [?user-eid :user/name "Alice"]
    [?user-eid :user/email ?email]]`);

The syntax has a few pieces:

The engine takes these statements and finds the set of values that makes all of them true simultaneously. Because joins are implicit (represented by the shared variable ?user-eid), the query stays declarative as joins are added: you state the facts that must agree, and the engine chooses the index walks and join order.

Now that you have seen db used as a query input, it is worth being precise about what it is. Datalevin's DB object is a read handle to storage plus cached metadata about the storage state, not a durable immutable snapshot. In application code, keep and share the connection. When a Clojure read API needs a DB object, call (d/db conn) at the read site and use that object for the operation. Connection-level APIs in Java, Python, and JavaScript do this for ordinary reads.

The useful mental model is perception, not possession. The database is the surrogate of an external world that changes. Calling d/db is how Clojure code takes a read handle for the operation it is about to perform. After writes, read again from the connection instead of treating a previous read object as live application state.

Attributes are queryable too. In a datom pattern, the entity, attribute, and value positions can all be variables. For example, this query asks which attributes are present on entities that have a :person/name:

(d/q '[:find ?attr
       :where
       [?person-eid :person/name]
       [?person-eid ?attr]]
     db)

The first pattern finds entities with a :person/name; the second pattern binds ?attr to each attribute asserted for those same entities.

7. Unified Retrieval: Plugging into the Datom Flow

One of the most powerful aspects of the Datalevin mental model is that "special" features like full-text search, vector similarity, and document indexing are not separate services running beside the database. Full-text search finds text by terms (Chapter 16), vector similarity finds nearby numeric vectors (Chapter 17), and document indexing (chapter 14) makes paths inside nested document values queryable. They are integrated directly into the Datalog flow.

When you perform a full-text search, the search function behaves just like any other Datalog clause. It returns a set of datoms ([e a v]) that can be immediately joined with other relational facts. The examples below destructure each returned datom as [[?doc-eid _ _]]: keep the entity id, and ignore the attribute and value. _ is a placeholder symbol for any value, and Datalevin always ignores it instead of binding it to a variable.

The query assumes :doc/body is declared with :db/fulltext true and :db.fulltext/autoDomain true. In the fulltext call, $ means "the database input to this query," the same database handle supplied to the query call.

;; A hybrid query combines text search with exact metadata filters.
(d/q '[:find ?title
       :where
       [(fulltext $ :doc/body "clojure") [[?doc-eid _ _]]] ; Keep eid from returned datoms
       [?doc-eid :doc/status "published"]                  ; Standard datom join
       [?doc-eid :doc/title ?title]]                       ; Retrieve the title
     db)
// A hybrid query combines text search with exact metadata filters.
Object results = conn.query(
    "[:find ?title " +
    " :where " +
    " [(fulltext $ :doc/body \"clojure\") [[?doc-eid _ _]]] " +
    " [?doc-eid :doc/status \"published\"] " +
    " [?doc-eid :doc/title ?title]]");
# A hybrid query combines text search with exact metadata filters.
results = conn.query("""
    [:find ?title
     :where
     [(fulltext $ :doc/body "clojure") [[?doc-eid _ _]]]
     [?doc-eid :doc/status "published"]
     [?doc-eid :doc/title ?title]]""")
// A hybrid query combines text search with exact metadata filters.
const results = await conn.query(
  `[:find ?title
    :where
    [(fulltext $ :doc/body "clojure") [[?doc-eid _ _]]]
    [?doc-eid :doc/status "published"]
    [?doc-eid :doc/title ?title]]`);

This unified retrieval model means you do not have to sync your database with an external search engine. With the default synchronous indexing mode, search results are transactionally consistent with your data: they reflect the same committed database state as the ordinary datoms in the query.

8. Rules: Reusable Query Logic

Rules are how you encapsulate and reuse logic in Datalevin. A rule is a named Datalog pattern. Query clauses can call the rule by name, and Datalevin expands that call into the rule body during query evaluation. If you find yourself frequently querying for "Active Users who have Premium subscriptions," you can define a rule called premium-user.

Rules are supplied to a query as an input. In :in $ %, $ is the database input and % is Datalevin's rule-set input. The clause (premium-user ?user-eid) calls the rule and binds ?user-eid to each entity that satisfies the rule body.

(def user-rules
  '[[(premium-user ?user-eid)
     [?user-eid :user/status :user.status/active]
     [?sub-eid :subscription/user ?user-eid]
     [?sub-eid :subscription/tier :subscription.tier/premium]
     [?sub-eid :subscription/status :subscription.status/active]]])

(d/q '[:find ?email
       :in $ %
       :where
       (premium-user ?user-eid)
       [?user-eid :user/email ?email]]
     db user-rules)
Object userRules = Datalevin.edn(
    "[[(premium-user ?user-eid) " +
    "  [?user-eid :user/status :user.status/active] " +
    "  [?sub-eid :subscription/user ?user-eid] " +
    "  [?sub-eid :subscription/tier :subscription.tier/premium] " +
    "  [?sub-eid :subscription/status :subscription.status/active]]]");

Object emails = conn.query(
    "[:find ?email " +
    " :in $ % " +
    " :where " +
    " (premium-user ?user-eid) " +
    " [?user-eid :user/email ?email]]",
    userRules);
from datalevin import interop

raw = interop()
user_rules = raw.read_edn("""
    [[(premium-user ?user-eid)
      [?user-eid :user/status :user.status/active]
      [?sub-eid :subscription/user ?user-eid]
      [?sub-eid :subscription/tier :subscription.tier/premium]
      [?sub-eid :subscription/status :subscription.status/active]]]""")

emails = conn.query("""
    [:find ?email
     :in $ %
     :where
     (premium-user ?user-eid)
     [?user-eid :user/email ?email]]""",
    user_rules)
import { interop } from "datalevin-node";

const raw = interop();
const userRules = await raw.readEdn(`
    [[(premium-user ?user-eid)
      [?user-eid :user/status :user.status/active]
      [?sub-eid :subscription/user ?user-eid]
      [?sub-eid :subscription/tier :subscription.tier/premium]
      [?sub-eid :subscription/status :subscription.status/active]]]`);

const emails = await conn.query(
  `[:find ?email
    :in $ %
    :where
    (premium-user ?user-eid)
    [?user-eid :user/email ?email]]`,
  userRules);

The rule body records what the policy means: the user is active, has a subscription entity, and that subscription is both premium and active. If the policy changes, the calling queries can stay focused on the business concept rather than repeating all of those clauses.

Rules can also be recursive, meaning a rule can call itself. Recursion is essential for graph traversal because it lets a query follow an unknown number of edges. A classic example is finding ancestors in a family tree:

(def ancestry-rules
  '[[(ancestor ?x ?y)      ; Rule name and arguments
     [?x :person/parent ?y]] ; Case 1: y is the direct parent of x

    [(ancestor ?x ?y)      ; Case 2: Recursion
     [?x :person/parent ?z] ; z is the parent of x
     (ancestor ?z ?y)]])   ; y is an ancestor of z
// Rules are read as EDN data.
Object ancestryRules = Datalevin.edn(
    "[[(ancestor ?x ?y) " +
    "  [?x :person/parent ?y]] " +
    " [(ancestor ?x ?y) " +
    "  [?x :person/parent ?z] " +
    "  (ancestor ?z ?y)]]");
from datalevin import interop

# Rules are read as EDN data.
raw = interop()
ancestry_rules = raw.read_edn("""
    [[(ancestor ?x ?y)
      [?x :person/parent ?y]]
     [(ancestor ?x ?y)
      [?x :person/parent ?z]
      (ancestor ?z ?y)]]""")
import { interop } from "datalevin-node";

// Rules are read as EDN data.
const raw = interop();
const ancestryRules = await raw.readEdn(`
    [[(ancestor ?x ?y)
      [?x :person/parent ?y]]
     [(ancestor ?x ?y)
      [?x :person/parent ?z]
      (ancestor ?z ?y)]]`);

By defining this rule, the database has a named ancestor relation. The query below calls that rule as (ancestor 42 ?ancestor):

(d/q '[:find ?ancestor
       :in $ %
       :where
       (ancestor 42 ?ancestor)]
     db ancestry-rules)
Object ancestors = conn.query(
    "[:find ?ancestor " +
    " :in $ % " +
    " :where " +
    " (ancestor 42 ?ancestor)]",
    ancestryRules);
ancestors = conn.query("""
    [:find ?ancestor
     :in $ %
     :where
     (ancestor 42 ?ancestor)]""",
    ancestry_rules)
const ancestors = await conn.query(
  `[:find ?ancestor
    :in $ %
    :where
    (ancestor 42 ?ancestor)]`,
  ancestryRules);

You can now ask "Who are all the ancestors of entity 42?" and the engine will navigate the graph for you. Chapter 9 discusses rules and their use in detail.

9. Three Views of the Same Database

After seeing transaction data, queries, and rules, it helps to name the layers Datalevin keeps separate. Most application code lives in the logical view: datoms, schema, transactions, and queries. Beneath that, Datalevin stores those facts in sorted indexes. At runtime, the Datalog engine connects the two. These are not three different databases. They are three ways of looking at the same set of facts, as shown in Figure 3.2.

The three views of a Datalevin database: a query resolves from the logical view (datoms and schema), through the query view (the Datalog planner and evaluator), down to the storage view (DLMDB indexes of sorted keys read with range scans)

Logical View: Datoms and Schema

This is the view you use when designing your application. You think in entities, attributes, values, references, and schema. At this level, a fact such as "Alice's email is alice@example.com" is simply a datom: [101 :user/email "alice@example.com"].

The logical view is intentionally small. It gives you a uniform way to describe people, documents, orders, permissions, embeddings, and relationships without inventing a new container shape for every case.

Storage View: Indexes and Sorted Keys

The storage view is where logical facts become sorted indexes. Datalevin persists them in DLMDB, its extension of LMDB, so common access patterns can be served by ordered key lookups and range scans.

You do not normally model at this layer. It matters when you want to understand performance or use Datalevin's direct key-value API for simple, low-level state.

Query View: Datalog, Planning, and Evaluation

The query view is the Datalog engine. It evaluates Datalog queries by translating logical clauses into index lookups and joins over the storage layer. A cost-based optimizer decides the order for retrieving and combining facts.

This view is what lets you stay focused on the shape of the answer instead of manual traversal. You describe the relationships that must hold, and the engine handles joins, filtering, recursion, and lookup order.

Summary

The Datalevin mental model is about moving from "data-in-boxes" to "data-as-facts."

Using this model gives you flexibility that is harder to get from fixed table-shaped or document-shaped records.

References

[1] Yehonathan Sharvit, Data-Oriented Programming, Manning, 2022.

[2] Jim Gray and Andreas Reuter, Transaction Processing: Concepts and Techniques, Morgan Kaufmann, 1993.

[3] E. F. Codd, "Further Normalization of the Data Base Relational Model," IBM Research Report RJ909, August 31, 1971. Republished in Randall J. Rustin, ed., Data Base Systems: Courant Computer Science Symposia Series 6, Prentice-Hall, 1972.

[4] Robert A. Kowalski, "Predicate Logic as Programming Language," IFIP Congress, 1974, pp. 569-574. URL: https://www.doc.ic.ac.uk/~rak/papers/IFIP74.pdf.

No examples for this chapter yet.