DatalevinDatalevin
Log in

Preface

SQL has been the default language of application databases for half a century. That default is now holding application state back. Datalevin1 is a database built to replace SQL databases at the center of application systems: it stores data as small facts and queries those facts with Datalog [1].

Modern applications rarely fit neatly into SQL's table-first model. A product may need transactions, graph relationships, documents, full-text search, and vector similarity in the same workflow. The usual response is to split these capabilities across services or stretch a SQL database with extensions. Both responses create costs. Services copy data and push consistency into application code. SQL-centered designs have their own tax: queries live as strings inside application code, composition pushes teams toward ORMs or query builders, dialect differences leak through, and non-relational workloads retreat into extension-specific syntax.

Complex queries expose a deeper structural problem. In conventional row-shaped SQL storage, the values a query cares about are packed inside rows, where cross-condition statistics are hard to maintain, and missing values are often represented with NULL. The planner must estimate which conditions are selective and which joins will stay small; bad estimates can turn a reasonable query into a brittle performance problem.

Datalevin replaces the table as the center of gravity with the fact. A fact is a statement such as "Alice follows Bob" or "document 42 is in English": a logical assertion with one entity, one attribute, and one value. Together, those three elements form a triple. Datalevin combines the basic model with LMDB [2], a fast key-value store, and a careful physical layout. The result can represent relational records, graph relationships, documents, search hits, and vector neighbors as different views over the same durable facts.

To query these facts, Datalevin uses Datalog. SQL joined relational algebra to a structured-English surface, part of an era that tried to make computing approachable by making programs read like business prose. That choice was useful for interactive database work, but it is awkward inside application programs: a query becomes a string in one language describing work for another.

Datalog takes a different path. It comes from logic programming and asks which facts must be true together. Datalevin uses a user-friendly form of Datalog and makes it the main application query language: one query model for relational joins, graph traversal, rules, document predicates, full-text search, and vector or embedding similarity.

A query is just data. Even before the syntax is explained, the query reads like a small description of the facts to match:

[:find ?person-eid
 :where [?person-eid :person/name "Alice"]]

"[:find ?person-eid :where [?person-eid :person/name \"Alice\"]]"

"""[:find ?person-eid
    :where [?person-eid :person/name "Alice"]]"""

`[:find ?person-eid
  :where [?person-eid :person/name "Alice"]]`

Application code can parse, generate, and test such data directly.

Why Datalevin Can Replace SQL Database

Datalevin is not meant to be one more database beside SQL databases. Its case as the center of application state rests on two practical advantages: better developer ergonomics and better performance on complex application queries.

Developer ergonomics. In my experience, many developers do not miss SQL once they use Datalog in a real application. Here, "developers" includes both human programmers and LLMs that generate or revise application queries. This matters because a language that replaces SQL must support composition, inspection, testing, and generation in normal application code. A fuller example appears in the next section. There are two concrete reasons: Datalog is more composable in programs, and its declarative model stays close to application facts.

Datalevin queries are data structures, not strings assembled inside another language. A program can build a query by adding or removing pieces, combining common parts, or supplying parameters without concatenating SQL text or depending on a SQL query builder. The query remains ordinary data that can be inspected, transformed, tested, logged, and generated by tools.

Datalog is truly declarative and it maps directly to the fact model. A query describes the facts that must be true, and the engine finds the matching values. The programmer states what relationships and constraints should hold in the result, not how the engine should implement them. The system takes responsibility for planning the query, choosing an efficient execution order, and handling the database-specific details needed to produce the answer.

Together, composable query data and declarative constraints let a query express the logic of the application domain directly. The programmer can write in terms of people, documents, relationships, events, and constraints, instead of first translating that logic into database-specific jargon. This helps human developers, new team members, and LLMs for the same reason: there is less distance between the question being asked and the query that answers it.

Performance for complex application queries. The performance advantage follows from the same design. Datalevin's fact indexes keep data close to the shapes that Datalog queries ask for. This addresses the planner problem described earlier: the engine can count and sample candidate facts under query constraints, giving the query planner better information and reducing unnecessary intermediate work. Integrated full-text, vector, embedding, and document indexes also avoid shuttling candidate ids between separate systems before applying logical constraints.

The published Datalevin benchmarks make the challenge concrete. They should be read as reproducible workload observations, not universal guarantees. Datalevin's Join Order Benchmark (JOB) run pits IMDb-derived complex join queries against PostgreSQL and SQLite [3], where Datalevin is more than 2x faster than PostgreSQL; against SQLite, the reported speedup is already more than 4x when counting only the queries SQLite completed; including its timed-out queries would make the advantage larger. In the LDBC Social Network Benchmark run comparing graph query performance with Neo4j [4], Datalevin is much faster than Neo4j on the Interactive Short queries, while the complex-query results are closer and mixed. The claim is not that Datalevin wins every workload. The point is sharper: for the complex queries in real application workloads, Datalevin's fact indexes and Datalog optimizer attack structural weaknesses in row-shaped SQL storage and systems split across separate databases.

Together, these advantages define Datalevin's purpose: retire SQL as the default database model for day-to-day application state. Application data belongs in a model that can be queried, searched, composed, and evolved without SQL strings, ORM ceremony, extension mini-languages, or cross-system glue code.

The same Datalevin engine can run embedded, as a server, or from the command line. For AI-oriented applications, embedding and text-generation providers are built in, and a bundled MCP server exposes Datalevin to AI agents without a separate adapter. Datalevin is the database for applications that have outgrown SQL as their center of gravity.

A Look in Code

The example below has four parts:

  1. Declare two attributes: :person/name is a unique identity, and :person/follows is a reference to another person.
  2. Open a connection.
  3. Write three people: Alice follows Bob, and Bob follows Carol.
  4. Query for Alice's friend and friend-of-a-friend.
(require '[datalevin.core :as d])

(def schema
  {:person/name    {:db/unique :db.unique/identity}
   :person/follows {:db/valueType :db.type/ref}})

(def conn (d/get-conn "/tmp/preface-demo" schema))

(d/transact! conn
  [{:db/id -1
    :person/name "Alice"
    :person/follows -2}
   {:db/id -2
    :person/name "Bob"
    :person/follows -3}
   {:db/id -3
    :person/name "Carol"}])

(d/q '[:find ?friend ?friend-of-friend
       :where
       ;; Start from Alice.
       [?alice-eid :person/name "Alice"]

       ;; Follow Alice -> Bob.
       [?alice-eid :person/follows ?friend-eid]
       [?friend-eid :person/name ?friend]

       ;; Follow Bob -> Carol.
       [?friend-eid :person/follows ?fof-eid]
       [?fof-eid :person/name ?friend-of-friend]]
     (d/db conn))
;; => #{["Bob" "Carol"]}

(d/close conn)

import datalevin.*;

Schema schema = Datalevin.schema()
    .attr(":person/name",
          Schema.attribute().unique(Schema.Unique.IDENTITY))
    .attr(":person/follows",
          Schema.attribute().valueType(Schema.ValueType.REF));

Connection conn = Datalevin.getConn("/tmp/preface-demo", schema);

conn.transact(Datalevin.tx()
    .entity(Tx.entity(-1)
        .put(":person/name", "Alice")
        .put(":person/follows", -2))
    .entity(Tx.entity(-2)
        .put(":person/name", "Bob")
        .put(":person/follows", -3))
    .entity(Tx.entity(-3)
        .put(":person/name", "Carol")));

Object results = conn.query("""
    [:find ?friend ?friend-of-friend
     :where
            ;; Start from Alice.
            [?alice-eid :person/name "Alice"]

            ;; Follow Alice -> Bob.
            [?alice-eid :person/follows ?friend-eid]
            [?friend-eid :person/name ?friend]

            ;; Follow Bob -> Carol.
            [?friend-eid :person/follows ?fof-eid]
            [?fof-eid :person/name ?friend-of-friend]]""");
// => [["Bob", "Carol"]]

conn.close();

from datalevin import connect

schema = {
    ":person/name": {":db/unique": ":db.unique/identity"},
    ":person/follows": {":db/valueType": ":db.type/ref"}}

with connect("/tmp/preface-demo", schema=schema) as conn:
    conn.transact([
        {":db/id": -1,
         ":person/name": "Alice",
         ":person/follows": -2},
        {":db/id": -2,
         ":person/name": "Bob",
         ":person/follows": -3},
        {":db/id": -3,
         ":person/name": "Carol"}])

    results = conn.query("""
        [:find ?friend ?friend-of-friend
         :where
                ;; Start from Alice.
                [?alice-eid :person/name "Alice"]

                ;; Follow Alice -> Bob.
                [?alice-eid :person/follows ?friend-eid]
                [?friend-eid :person/name ?friend]

                ;; Follow Bob -> Carol.
                [?friend-eid :person/follows ?fof-eid]
                [?fof-eid :person/name ?friend-of-friend]]""")
    # => [["Bob", "Carol"]]

import { connect } from "datalevin-node";

const schema = {
  ":person/name": { ":db/unique": ":db.unique/identity" },
  ":person/follows": { ":db/valueType": ":db.type/ref" }
};

const conn = await connect("/tmp/preface-demo", { schema });

try {
  await conn.transact([
    { ":db/id": -1, ":person/name": "Alice", ":person/follows": -2 },
    { ":db/id": -2, ":person/name": "Bob", ":person/follows": -3 },
    { ":db/id": -3, ":person/name": "Carol" }
  ]);

  const results = await conn.query(
    `[:find ?friend ?friend-of-friend
      :where
             ;; Start from Alice.
             [?alice-eid :person/name "Alice"]

             ;; Follow Alice -> Bob.
             [?alice-eid :person/follows ?friend-eid]
             [?friend-eid :person/name ?friend]

             ;; Follow Bob -> Carol.
             [?friend-eid :person/follows ?fof-eid]
             [?fof-eid :person/name ?friend-of-friend]]`);
  // => [["Bob", "Carol"]]
} finally {
  await conn.close();
}

Read the query from top to bottom. It first finds Alice's entity id, follows :person/follows to Bob's entity id, reads Bob's name, follows one more :person/follows reference to Carol's entity id, and reads Carol's name. The result is the pair ["Bob" "Carol"].

The point here is Datalevin's basic shape: data is represented as facts, relationships are ordinary values, and queries are data. The database answers questions by matching facts, following references, and applying logic. Later chapters explain the syntax and APIs in detail.

How to Read the Examples

The print version of this guide shows Clojure example code for its conciseness. The web version at https://datalevin.org also shows Java, Python, and JavaScript examples. You do not need to know Clojure to follow the code listings; read them as EDN data plus a small number of Datalevin function calls. EDN [9] is a data notation similar to JSON. In EDN data, text after a semicolon, such as ; or ;;, is considered comment, not part of the data. Appendix B covers EDN. EDN has readers and printers in many languages, and Datalevin exposes EDN helpers or typed builders where useful.

The code examples follow one set of conventions:

Teaching examples use this variable convention: an internal Datalevin entity id ends in -eid, such as ?person-eid or ?order-eid. An application or domain identifier ends in -id, such as ?person-id or ?order-id. Variables such as ?name or ?total hold ordinary values.

Built for Performance and Flexibility

Datalevin's implementation is built for both performance and flexibility. Storage is performance-critical, so Datalevin uses a close-to-metal native code implementation for the transactional storage layer. The layers above storage need flexibility to support tight integration between the data model and a variety of paradigms and capabilities.

From those needs, a deliberately layered implementation follows. At the bottom, Datalevin uses LMDB as the native storage foundation for its robust transaction mechanism and raw read performance. Above that foundation, Datalevin uses the JVM for its portability, maturity, tooling, and extensive integration ecosystem.

In particular, Datalevin's higher layers use a data-oriented implementation style [5]: schemas, transactions, queries, and plans are represented as plain immutable data structures that can be inspected, transformed, tested, and passed around. This keeps the implementation close to the database's data model and makes the code easier to adapt to evolving requirements. It also reduces complexity by separating data from behavior: facts and queries remain explicit data inside the engine, not state hidden behind a large object hierarchy. Immutable data also helps with concurrency, because it can be shared, snapshotted, compared, and passed between threads with less hidden mutable state.

This layered approach leads to a mixed language implementation. LMDB is written in C. A Java layer packages and exposes the native LMDB operations to managed code. On top, a Clojure layer implements schema, queries, planning, and high-level APIs. The same implementation is exposed through Clojure, Java, Python, and JavaScript/Node.js APIs. Appendix A gives installation details for these language environments and links to the current compatibility matrix.

The result is a pragmatic stack: native code where the storage operations must be fast and obey strict transaction boundaries, and data-oriented managed code where database semantics must evolve. The remaining chapters cover this design in practice.

Why This Guide Exists

SQL is not a law of nature. As a successful compromise from an earlier era of application architecture, it has table-shaped records, string-shaped queries, and a sharp divide between storage, search, logic, and application code. Datalevin is a compact database system designed to replace that compromise for modern applications. It can be embedded in applications written in multiple languages, run as a server, and used from scripts.

This guide exists to make the replacement practical. A serious SQL replacement cannot stop at a nicer query language. It must explain transactions, schema, identity, indexes, performance, deployment, failure modes, and the discipline needed to model application state as facts rather than tables plus glue code.

The first goal is to teach Datalevin as a complete database, not just a query language: how to open it, transact data, read data, design schemas, model relationships, use the key-value layer, query with Datalog, tune storage, and operate the system.

The second goal is to change the modeling reflex. Datalevin works best when you stop treating rows, documents, graphs, search results, and similarity results as separate conceptual universes. They are different projections over durable facts. Once that shift lands, much of the machinery around SQL applications starts to look accidental.

The third goal is to show why this matters now. Large language models and agentic applications are pushing databases into a new role. A useful AI system needs persistent memory, scoped task state, retrieval with permissions and provenance, and a way to integrate new observations into a coherent long-term model. Part VI shows how Datalevin can serve as that memory substrate.

How This Guide Is Organized

Part I builds the foundation. It explains why Datalevin exists, how to get started, how to think in facts rather than containers, how LMDB shapes the storage model, and how attributes, entities, and namespaces work.

Part II covers the core APIs. You will transact data atomically, read with lookup, pull, and entity APIs, learn Datalog fundamentals, rules, recursion, and derived knowledge, and then drop down to the direct key-value API when sorted keys or custom storage structures are the right tool.

Part III is about modeling. It shows how relational, graph, and document patterns map onto Datalevin's fact-first model, and how to make schema choices that can evolve without losing clarity.

Part IV treats indexes as capabilities. You will learn the core EAV and AVE indexes, full-text search, vector and embedding search, and hybrid queries that combine those capabilities.

Part V is about performance and operations. It covers durability, storage tuning, batching, ingestion, query planning and diagnostics, recursive-rule execution, deployment, security, replication, monitoring, and production checklists.

Part VI applies the database to intelligent systems. It follows a path from persistent agent memory to episodic, semantic, and working memory; then to recall and context assembly; finally to apperception, truth maintenance, and stateful AI application patterns.

Part VII is reference material. It covers installation, runtimes, and deployment modes; EDN, the data notation used throughout this guide; schema keys; Datalog built-ins; and the core, key-value, and client APIs.

Who This Guide Is For

This guide is for engineers who build data-intensive applications and are ready to stop treating SQL as inevitable. If tables, SQL strings, ORMs, side indexes, and auxiliary services have made ordinary application data harder to model and change, this guide is written for you. You may be building an embedded application, a backend service, a knowledge graph, a search-heavy product, a durable workflow system, or an AI application that needs persistent memory.

That does not mean every byte in a system belongs in Datalevin. Warehouses, analytics stores, logging stores, object stores, and specialized infrastructure still have their jobs. The target here is SQL's default role as the place where application state, relationships, constraints, search, and derived knowledge are forced to meet.

This guide assumes basic database literacy: transactions, indexes, schemas, and querying. You do not need prior Datalog experience. This guide introduces Datalog from first principles, then returns to it throughout the modeling, search, performance, and intelligent-systems chapters.

How to Read This Guide

If you are new to Datalevin, read Parts I and II in order. They establish the mental model and the APIs that later chapters assume.

If you are designing an application schema, continue into Part III before optimizing anything. Good Datalevin design starts with clear facts, identities, and relationships.

If you are building search or retrieval features, read Parts III and IV together. The most useful retrieval systems combine modeling choices with the right indexes.

If you are operating Datalevin in production, read Part V after the foundations. The operational chapters make more sense once you know how data, indexes, and queries are represented.

If you are building AI systems, resist the urge to jump straight to Part VI. Persistent memory and agent state are only reliable when the underlying data model, transactions, indexes, and query semantics are clear. You should not consider Part VI to be a separate AI appendix; it is the natural consequence of the earlier parts.

For source code, issues, release history, and project documentation, use the Datalevin GitHub repository.

What You Should Take Away

By the end of this guide, you should be able to:

The larger argument of this guide is this: the application database should be a logical substrate, a place where facts are stored, relationships are traversed, knowledge is derived, evidence is preserved, and intelligent systems can maintain state over time. Datalevin is built around that idea.

Acknowledgments

Datalevin stands on a line of ideas and software that made this project possible. I am grateful to Rich Hickey, whose EDN-based Datalog flavor in Datomic [6] showed how logic queries could become a practical application programming interface; to Nikita Prokopov, whose DataScript [7] code was the starting point for Datalevin; and to Howard Chu, whose LMDB code became the storage foundation on which Datalevin is built [2].

I also want to thank the Datalevin users in the Clojure community. Their questions, bug reports, feature requests, examples, and production experience have shaped the project in ways that are hard to separate from the code itself. Thanks also to everyone who contributed code, wrote about Datalevin, or supported the work through GitHub Sponsors: Dennis Heihoff, Anders Murphy, Aleksandr Bogdanov, Nils Grünwald, Christophe Grand, Garrett Hopper, Daniel Vingo, Samuel Ludwig, itonomi, Amar Mehta, Ryan Domigan, Lars Rune Nøstdal, Clay Hopperdietzel, Isaac Ballone, and many others I have missed.

Datalevin was originally motivated by needs at Juji [8], where intelligent applications required persistent memory, fast retrieval, and a more flexible way to represent user and agent state. I am grateful to Michelle Zhou, Amar Mehta, Wenxi Chen, Ransom Williams, Taejun Song, and other colleagues at Juji, whose work made those needs concrete and whose applications kept the project grounded in real systems.

This guide has already benefited from volunteer reviewers in the Datalevin community who read draft chapters, ran examples, questioned unclear explanations, improved phrasing, and made suggestions: Estevo U. C. Castro, Max Rothman, Norbert Wójtowicz, Jeroen van Dijk, Siavash Mohammady, Weidong Cai, Thomas Moerman, Amar Mehta, and Vlad Poh. I am grateful for their contributions. Any remaining mistakes are my own responsibility.

Huahai Yang
July 2026

References

[1] Stefano Ceri, Georg Gottlob, and Letizia Tanca, "What You Always Wanted to Know About Datalog (And Never Dared to Ask)," IEEE Transactions on Knowledge and Data Engineering 1(1):146-166, 1989. URL: https://hdl.handle.net/11311/665510.

[2] Symas, "Lightning Memory-Mapped Database (LMDB)." URL: https://www.symas.com/symas-embedded-database-lmdb.

[3] Datalevin project, "Join Order Benchmark," benchmark writeup and implementation. URL: https://github.com/datalevin/datalevin/tree/master/benchmarks/JOB-bench.

[4] Datalevin project, "LDBC SNB Benchmark," benchmark writeup and implementation. URL: https://github.com/datalevin/datalevin/tree/master/benchmarks/LDBC-SNB-bench.

[5] Yehonathan Sharvit, Data-Oriented Programming, Manning, 2022.

[6] Rich Hickey, "Datomic." URL: https://www.datomic.com/.

[7] Nikita Prokopov, "DataScript." URL: https://github.com/tonsky/datascript.

[8] Juji Inc. URL: https://juji.io/.

[9] EDN format project, "Extensible Data Notation." URL: https://github.com/edn-format/edn.

  1. The name joins LMDB's "Lightning" with "levin", a Middle English word for lightning. ↩

User Examples

Log in to create examples

No examples for this chapter yet.