DatalevinDatalevin
Log in
Part I - Foundations: A Fact-First Database

Chapter 1: Why Datalevin

This chapter makes the preface's claim concrete: Datalevin is built to replace SQL's default role at the center of application state, not to sit beside SQL as one more specialized database.

The problem is larger than SQL syntax. Modern applications need transactional updates, relational joins, graph traversal, document-style flexibility, full-text search, and vector or embedding search in the same workflows. A SQL-centered design usually responds in one of two ways: stretch tables with extensions, or split capabilities across specialized stores. Both paths push incidental knowledge into application code: which rows own which relationships, which data was copied into a search index, which JSON fields duplicate relational columns, and which vectors must be refreshed after an update. This knowledge is not essential to the business domain. As it accumulates, the application becomes harder to change and more likely to sink into the software engineering "tar pit" [1].

Datalevin's answer is to replace the table as the center of gravity with the fact. A fact can participate in a row-like record, a graph edge, document metadata, a search result, or a vector-neighbor workflow without leaving the same durable database model. Datalog then queries those facts as ordinary data rather than as strings assembled inside application code.

Datalevin combines this fact-first model with a high-performance key-value storage layer, integrated indexes, and search capabilities.

The central idea is locality. Facts, indexes, documents, search entries, and vector data stay close enough that one ACID (atomicity, consistency, isolation, and durability) transaction can keep them consistent and one query model can combine them.

This chapter explains how that design replaces SQL's table-first compromise for day-to-day application state, how Datalevin unifies multiple access patterns, and where it fits in a modern stack.

1. A Unifying Data Model

At the Datalog layer, Datalevin represents facts as EAV triples: Entity, Attribute, and Value. Each assertion is one small, independently queryable fact, called a datom (data atom). Modeling data in terms of datoms produces a model that is compact, sparse, and composable.

Semantically, the four terms mean:

The database meaning is more specific:

Because the relationship between entity and attribute is open, the data model can be sparse: only asserted datoms are present in the database. Missing facts are represented by the absence of a datom rather than by placeholder values. This does not prevent storing a boolean false as an actual value; in that case, the datom is present and its value is false.

When surfaced in query functions, datoms are commonly represented as [e a v]. For example, the fact that entity 101 has Alice's email address is the datom [101 :user/email "alice@example.com"].

This tiny shape is the first new concept to internalize. A datom is smaller than a row and flatter than a document, but it can express both. A group of datoms that share the same entity id can look like a row. A graph edge can be a datom whose value is another entity id. A document-like aggregate can be modeled as ordinary facts when its fields should join, validate, reference other entities, or change independently. Chapter 14 introduces the complementary choice: storing a nested map or JSON-like payload as one document value while Datalevin indexes its internal paths. The uniform shape makes these choices composable instead of isolated. Figure 1.1 shows those three readings of the same [e a v] shape.

One datom shape can look row-like when facts share an entity id, graph-like when a ref datom's value is another entity id, and document-like when component facts are connected by refs

What Is an Entity?

Entities also deserve a precise definition. An entity is not a table, class, or JSON object. It is the bundle of datoms anchored by one eid: the currently asserted properties and relationships that share the same database-local handle. If entity 101 has :user/email, :customer/tier, and :employee/manager, Datalevin does not need three separate containers to hold those facts. They are simply assertions about the same entity. This gives the model room to evolve as the application learns more about the thing being represented.

This makes Datalevin's notion of entity close to the metaphysical idea known as bundle theory: an object can be understood as the bundle of properties that belong together, rather than as a hidden substance underneath those properties [2]. The analogy should not be pushed too far: a Datalevin eid is not the object itself.

Because eids are database-local handles, they should not be confused with stable application identifiers; two databases containing the same application data may assign different sets of eids. Application domain identifiers such as user IDs, order IDs, document IDs, UUIDs, and slugs should be stored as ordinary attribute values, often with the :db.unique/identity property (see Chapter 3 for details). Internal relationships can still use entity references, so the database does not have to make every relationship depend on those application-facing identifiers.

Aggregate Models vs Fact-First Model

SQL tables and document databases are typically aggregate-first: a row or document is the main unit of mutation and retrieval, and columns or nested fields are scoped under that container.

Fact-first systems start one level lower: each EAV assertion is atomic and independently queryable. Cross-aggregate joins, inference, and graph traversal can then operate over the same facts rather than over application-owned aggregates.

Instead of asking "which table owns this field?" you ask "what fact am I asserting?" Instead of asking "which document should duplicate this data?" you ask "which entity does this fact describe, and which other entities does it refer to?" You can still return an entity as a map, pull a nested shape for an API response, or index a JSON-like document for path queries. The difference is that these aggregate shapes are views over durable facts, not the only way to see the data. The same facts can also be joined relationally, traversed as a graph, searched by text, or combined with search constraints.

Compared with rigid table layouts, EAV handles sparse and evolving data naturally. The important shift is that there is no hidden wide table where every entity has a slot for every possible attribute. Datalevin stores the facts that exist. If no one has asserted :user/timezone for an entity, there is no :user/timezone datom for that entity. Queries that need the value match the fact; queries that care about absence can ask for missing facts explicitly. When the domain needs a meaningful value such as "unknown", "not applicable", or "withheld", model that value as data. Otherwise absence is simply absence, not a placeholder null.

Example

If only some users have a :user/timezone, only those users need that fact. If a later feature adds :user/preferred-model, existing entities do not need to be rewritten. The model grows by adding attributes and facts.

2. One Engine, Multiple Access Patterns

Datalevin exposes several complementary capabilities in one runtime. Each capability can be useful by itself, but the more important strength is that they compose over the same database state. As illustrated in Figure 1.2, Datalevin uses a single unified data model to support multiple database capabilities. This section gives an overview, while the remaining chapters fill in the details.

Datalevin unified data model

The same model also gives application code a clean place to observe change. When a write commits, Datalevin can report what changed and notify application listeners, so an embedded application does not have to poll or scatter change detection across search, graph, document, and cache subsystems. Chapter 6 covers that transaction-observation pattern after introducing transaction reports.

Relational and Graph Queries with Datalog

Datalog provides declarative joins, recursive rules, and reusable query composition. In practice, this lets Datalevin support both relational workloads and graph traversal workloads through one query model.

The same query language can ask for "all orders whose customer is in France" and "all dependencies reachable from this package." The first feels relational: facts are joined by shared variables. The second feels graph-oriented: entity references are followed as edges. In Datalevin both are expressed as logical patterns over datoms.

The underlying idea is classic Datalog: describe the facts that must be true, and let the engine find bindings for the variables [3]. Classic Datalog literature writes rules in Prolog syntax:

friend_of_friend(X, Z) :- follows(X, Y), follows(Y, Z).

This is a classic friend-of-friend query. The notation is abstract. The precise meanings of :-, , and . are not obvious from the notation alone, and the symbol followed by parentheses can look like a function call.

Datalevin keeps the logic but writes queries as EDN data structures. The same query looks like this:

[:find ?z-eid
 :in $ ?x-eid
 :where
 [?x-eid :person/follows ?y-eid]
 [?y-eid :person/follows ?z-eid]]

Read :find as the requested output, :in as the inputs supplied by the caller, and :where as the facts that must hold. $ is the database input. Names that start with ? are variables, and attributes such as :person/follows come from the schema. Combined, the example asks: find ?z-eid, such that ?x-eid follows ?y-eid, and ?y-eid follows ?z-eid.

Each vector under :where is one fact pattern. [?doc-eid :doc/lang "en"] reads: bind an entity whose :doc/lang value is "en". [?order-eid :order/customer ?customer-eid] reads: bind an order entity and the customer entity it points to. When the same variable appears in more than one pattern, Datalevin keeps only bindings that make all of those patterns true. In the friend-of-friend query above, ?y-eid is the join: it is the person followed by ?x-eid and also the person who follows ?z-eid.

Direct Key-Value Access

The same key-value substrate that persists triples and indexes is also available directly as a durable key-value API. That substrate is DLMDB, Datalevin's extension of LMDB, the memory-mapped key-value store covered in Chapter 4. This low-level API is useful for low-latency state access, caches, and simple lookup-oriented components.

Use the key-value API when the access pattern really is just "given this key, read or update this value" or "read this key range." Examples include local application state, checkpoints, registries, counters, queues, or cached results. In those cases, forcing everything through a logical query layer would add complexity without adding much value.

Path-Indexed Documents with idoc

Datalevin supports path-indexed access to documents in EDN, JSON, and Markdown formats using the :db.type/idoc data type (Chapter 14). You can query nested structures with idoc-match, including logical combinators and path predicates, while keeping the documents in the same transactional store.

This is useful because real applications rarely receive perfectly normalized facts at the boundary. Here, normalized means that each durable fact is stored once and related facts are connected by references, rather than copied or nested into every larger record that might need them. Applications receive JSON from APIs, EDN configuration, parsed Markdown metadata, tool outputs, and model-produced structures whose shape may change over time. An idoc lets you keep such nested data as a document while still indexing paths inside it.

For example, a documentation page might have structured metadata such as {:module {:name "search" :status "stable"}}. You may not want to promote every nested field to its own top-level schema attribute on day one. With idoc, Datalevin can still answer questions about paths inside the document and join those answers with ordinary facts about the same entity.

Idoc does not replace normal datoms. It complements them. Use datoms for core domain facts with stable meaning and relationships. Use idoc for nested, semi-structured, imported, or metadata-heavy values where path search is more important than turning every leaf into a named attribute.

Integrated Full-Text, Vector, and Embedding Search

Datalevin includes built-in full-text search, similarity search over vectors supplied by the application, and text embedding search. Because these indexes live in the same database system, you can combine search results and exact logical conditions in one Datalog query instead of synchronizing external search services.

These search modes answer different questions:

In many systems, these capabilities live outside the primary database. The application first asks a search engine for candidate ids, then asks the database for structured facts, then performs more filtering in application code. Datalevin's integrated approach lets word search, meaning-based search, document-path search, and exact logical constraints meet inside one query.

The practical result is that an application can choose the access pattern that fits the problem without leaving the same data model. A query can ask for "documents like this question, containing this phrase, visible to this user, and marked stable" as one database question rather than as a sequence of cross-system calls.

3. Why Not Just SQL?

A possible objection to a SQL replacement built on Datalog is that modern SQL databases keep adding capabilities: JSON columns, full-text search, graph extensions, vector indexes, procedural languages, recursive CTEs, and extension ecosystems. For example, if PostgreSQL can do more and more of these jobs, why not stay with familiar SQL?

Familiarity is a real argument for SQL, but it is not the same as fit. Many teams stay with SQL because it is transactional, durable, mature, and already operationally understood. Many developers are not enthusiastic SQL users so much as resigned SQL users.

They want something more composable, more directly usable from programs, and less tied to table containers. But the available alternatives often give up too much: transactions, durability, joins, ad hoc queries, deployment simplicity, or operational maturity.

Datalevin is aimed at SQL's default application-state role. It is a database for developers who want to replace SQL in application systems, not merely wrap it, hide it behind an ORM, or add another specialized service beside it.

There are three reasons that go deeper than syntactic cosmetics, and they build on one another: SQL is awkward as an application interface, table-shaped storage makes query planning harder, and stacking SQL extensions turns every cross-cutting question into an integration problem.

SQL Is Awkward as an Application Interface

The first reason is about programming ergonomics. SQL succeeded as a standard, but standardization should not be confused with a good programming interface. SQL descends from SEQUEL, introduced as "a Structured English Query Language" [4]. That origin still shows in SQL's large English-like surface. The original ambition was to make database work feel closer to ordinary business language. That was useful for interactive querying, but it is awkward inside programs: SQL queries are usually strings assembled by a host language. Complex SQL often forces programmers to think in terms of table aliases, join syntax, and container-specific mechanics before they can express the relationship they actually mean.

The ORM and query-builder ecosystem is evidence that raw SQL is awkward at application boundaries. These tools soften many rough edges: SQL string construction, weak host-language composition, dialect differences, type mismatches, row/object mismatch, and boilerplate around joins, identity, transactions, lazy loading, and migrations.

Even tools that reject classic ORM behavior, such as query builders, LINQ-style systems, jOOQ, Ecto, and Prisma, are still trying to make SQL less like raw SQL and more like host-language data or typed expressions.

Datalog gives the programmer a smaller and more uniform surface. A query is data, not a string. It says which facts must be true, not which joins to perform step by step.

Reusing the same variable name connects facts. Rules name reusable logic. Recursive queries use the same logical form as non-recursive queries. The result is not less relational; it is a higher-level way to write relational logic. Relational algebra remains the substrate, but the user does not have to speak in join operators to ask a relational question [3].

The ergonomic difference is not limited to database specialists. In practice, experienced developers often prefer Datalog once they have used it for real application queries, because the clauses stay close to the facts they are asking for.

The contrast is even sharper for people without years of SQL habits: interns, domain experts learning to inspect data, and application developers who do not want to become query-planner whisperers before they can ask a basic question. A small set of data patterns, shared variables, and named rules is easier to teach than a large string language with dialects, aliases, join forms, grouping edge cases, and host-language impedance mismatch. This developer ergonomics goal was one of the original motivations for Datalevin.

The same simplicity matters for LLM-generated queries. Text-to-SQL is a hard problem not only because natural language is ambiguous, but also because SQL is a large, dialect-sensitive target language. A model must choose table aliases, join syntax, grouping rules, subquery shapes, vendor functions, and sometimes planner-sensitive rewrites.

Recent text-to-SQL results show the gap clearly. Spider 2.0 was designed around realistic enterprise workflows, and its authors reported much lower success rates than on older benchmarks such as Spider 1.0 and BIRD [5]. A later PExA result reported 70.2% execution accuracy on Spider 2.0 [6], which is real progress but still too much uncertainty for an automatic application interface.

Datalevin's Datalog is a more regular generation target because its surface is smaller and query structure is data. Most of the work is choosing facts, attributes, variables, predicates, and rules. The syntax is often compact compared with SQL's structured-English surface, and there are fewer dialect decisions for the model to hallucinate. Even though public Datalog training data is much smaller than SQL training data, the language itself gives both humans and LLMs a simpler target.

Datalevin's attribute-centered schema also gives an LLM better semantic handles. An attribute such as :order/total can carry its own type, cardinality, uniqueness, index behavior, and :db/doc documentation wherever that fact appears. SQL systems often support table and column comments, but they are vendor-specific metadata on table containers. In Datalevin, attribute documentation is ordinary schema metadata. It can be read, exported, and fed to a query-writing assistant along with the same attribute names used in Datalog clauses. That makes the schema more useful as prompt context and reduces the guesswork involved in choosing the right facts.

Table Storage Makes Query Planning Harder

The second reason is query planning. SQL databases are table-first systems. Data is bundled into row or column containers, and the optimizer must infer how many rows will survive predicates and joins over those containers.

Real application data is usually neither evenly distributed nor independent. A few customers may account for most orders; a few tags may appear on most documents; some vendors, regions, products, and refund patterns may strongly move together. Data distributions are called skewed when some values are much more common than others. They are correlated when one predicate being true changes the odds of another predicate being true. Skewed and correlated data make a query optimizer's job difficult because formulas based on uniformity or independence can produce wildly inaccurate estimates for them. Traditional estimators therefore depend on approximations such as histograms and, in some systems, learned models.

This difficult problem has a name: cardinality estimation, or estimating result sizes. It has been repeatedly identified as one of the central problems in query optimization in the database research literature [7,8]. For container-based storage, there is no cheap general answer. The counts the planner needs are not directly present in the storage layout, so the optimizer has to approximate them.

Consider a production-style question: find refunded late orders since a given time, for products from one vendor, shipped to customers in one region. In Datalevin, the logical form of that question is a set of fact patterns:

[:find ?order-id ?customer-email ?product-name
 :in $ ?vendor-id ?region ?since
 :where
 ;; Find the vendor and its products.
 [?vendor-eid :vendor/id ?vendor-id]
 [?product-eid :product/vendor ?vendor-eid]
 [?product-eid :product/name ?product-name]

 ;; Follow line items from products to orders.
 [?line-eid :line-item/product ?product-eid]
 [?line-eid :line-item/order ?order-eid]
 [?order-eid :order/id ?order-id]

 ;; Find the customer and restrict the shipping region.
 [?order-eid :order/customer ?customer-eid]
 [?customer-eid :customer/email ?customer-email]
 [?customer-eid :customer/address ?address-eid]
 [?address-eid :address/region ?region]

 ;; Require a late shipment for the same order.
 [?shipment-eid :shipment/order ?order-eid]
 [?shipment-eid :shipment/status :shipment.status/late]

 ;; Require a recent refund for the same order.
 [?refund-eid :refund/order ?order-eid]
 [?refund-eid :refund/created-at ?refund-time]
 [(>= ?refund-time ?since)]]

The actual execution plan in a database engine is not straightforward. The engine must decide whether to start from the vendor's products, the recent refunds, the late shipments, the region, or some other selective fact, and then join outward.

This kind of query is not exotic. It appears whenever an application keeps data normalized and asks compositional questions: which documents can this user see, which orders match a constraint spanning products, vendors, shipments, refunds, and customers, which clinical cases satisfy rules spanning diagnoses, providers, prescriptions, and coverage, or which papers connect authors, affiliations, topics, institutions, and collaborators. If a system never seems to have such queries, that is often because the complexity has been moved into denormalized records, application-side filtering, or cache materialization rather than disappearing.

In a SQL engine, this kind of question is usually a many-join query over table-shaped containers. The optimizer must estimate selectivity: how many rows survive each predicate and each join. The fact counts it needs are bundled inside rows, indexes, histograms, and assumptions about value independence.

When the estimates are wrong, the optimizer can start from a broad table, build huge intermediate results, and only later discover that a different predicate would have narrowed the search immediately. This makes complex SQL performance hard to predict. Large join queries become optimizer-sensitive.

For example, PostgreSQL's own documentation [9] notes that possible join orders grow exponentially, that exhaustive planning becomes impractical beyond roughly ten input tables, and that the planner switches to heuristic search for many-relation queries. It also documents techniques for constraining join order to reduce planning time. That is the practical pressure behind advice to denormalize, split a query, materialize intermediate results, or hand-guide the planner when SQL joins become too large or too sensitive to cardinality estimation [7].

A triplestore changes the shape of the problem. In Datalevin, a datom is an explicit cell: entity, attribute, value. Missing facts are absent rather than stored as nullable positions. Attributes are globally scoped. EAV (Entity-Attribute-Value) and AVE (Attribute-Value-Entity) indexes make many counts direct, and larger estimates can be sampled under the same query conditions that execution will use.

Datalevin still has to optimize queries, but it starts from a storage model that exposes the units the optimizer needs to count or sample [8]. That is why an optimizing Datalog engine on a triplestore has advantages: it can plan the complex question as one query without making the application hard-code a join order, denormalize the model, or split the question into many smaller queries.

Join Order Benchmark (JOB) is the benchmark version of this problem [10]. It is not a benchmark of simple key lookups or single-table scans, nor an academic exercise with synthetic data. JOB is based on the real IMDb dataset, where value frequencies and relationships have the same nonuniform shape seen in many production datasets. It requires joins across many related entities: 113 analytical queries, with query shapes that range from a few joins to more than a dozen. Its queries have enough joins that a poor query plan can produce orders of magnitude more intermediate rows than a good one. In Datalevin's published JOB run [11,12], the same workload was run against both PostgreSQL and SQLite. Treat these numbers as a self-published workload observation, not a general performance guarantee. In that run, Datalevin finished the workload in 71 seconds, compared with 171 seconds for PostgreSQL. SQLite completed the non-timeout portion in 295 seconds and hit timeout on 9 queries, so the reported "more than 4x faster than SQLite" summary counts only the queries SQLite completed.

Stacking Extensions Becomes an Integration Tax

The third reason is integration. SQL extensions are valuable; they are one reason systems such as PostgreSQL remain useful for so many applications. The problem is not that extensions exist. The issue is composition.

Each extension often brings its own syntax, operators, indexes, cost model, and limitations. JSON paths, text search, vector similarity, recursive queries, and procedural code may all be available in one SQL database. But the application still has to compose several feature-specific surfaces and understand how the planner handles their interaction.

That flexibility is helpful when a workload needs one extension in isolation. It becomes a tax when a single application question needs search, document structure, relationships, similarity, and exact constraints to work together.

For example, suppose a documentation assistant needs to find pages that mention "vector", belong to a stable module in nested metadata, are visible to the current user, and are close to the user's question in embedding space. In a PostgreSQL-style design, this usually means combining a text-search expression, a JSON path or containment expression, a vector-similarity expression, and joins through permission tables.

Each part can be indexed, but each part also has its own operators, index choices, selectivity estimates, and tuning rules. The engineering cost is not only syntax. The application may need to maintain generated search columns or embedding columns, create several different index types, tune a query plan whose estimates cross extension boundaries, or fetch candidate ids from one subsystem and filter them in another. The runtime cost is extra memory traffic, intermediate result sets, and CPU spent reconciling candidates that were found through different access paths.

Datalevin's approach is to make these capabilities participate in one logical query model over the same facts. The same documentation question can be written as one Datalog query whose parts say, in one place: this entity matches the text, this entity has the required nested metadata, this entity is visible to the user, this entity is close in meaning to the question, and this entity has the exact attributes required by the application.

The cost shifts from application-level stitching to database-level planning over one fact-oriented model. Section 4 shows a smaller concrete version of this pattern.

Datalevin's argument is that a fact-first model, paired with Datalog, is a better foundation for applications whose data is relational, graph-shaped, sparse, evolving, searchable by words and meaning, and increasingly used by AI systems.

4. A Minimal Unified Query Example

Suppose you are building a documentation assistant. You want to find English documents that mention a term in free text, while also requiring a structured module status inside nested document metadata.

The examples below all perform the same four steps:

  1. Declare a small schema. :doc/body is a full-text attribute, :doc/idoc is a nested-document attribute, and :doc/lang is an ordinary string fact.
  2. Open a throwaway in-memory database.
  3. Transact one document entity with a body, a language, and nested metadata.
  4. Run one query that asks for document entity ids satisfying three constraints.

Read the query around ?doc-eid. The full-text clause finds documents whose :doc/body mentions the search term, the idoc clause keeps documents whose nested metadata has {:module {:status "stable"}}, and the Datalog clause requires the exact fact [?doc-eid :doc/lang "en"]. The repeated ?doc-eid variable is the join: all three clauses must describe the same document entity.

(require '[datalevin.core :as d])

;; Schema is optional, but this example declares it for type checking and indexes.
(def schema
  {:doc/id   {:db/valueType :db.type/string
              :db/unique    :db.unique/identity}
   :doc/body {:db/valueType :db.type/string
              :db/fulltext  true
              :db.fulltext/autoDomain true}
   :doc/lang {:db/valueType :db.type/string}
   :doc/idoc {:db/valueType :db.type/idoc}})

;; Create a throwaway in-memory DB connection.
(def conn (d/create-conn nil schema {:kv-opts {:inmemory? true}}))

;; Transact one entity.
(d/transact! conn
  [{:db/id    -1
    :doc/id   "guide/search-overview"
    :doc/lang "en"
    :doc/body "Datalevin adds idoc indexing and vector search."
    :doc/idoc {:module {:name "search" :status "stable"}}}])

;; Run the query.
(d/q '[:find ?doc-eid
       :in $ ?term
       :where
       ;; fulltext/idoc-match return many document refs, each shaped as [e a v].
       ;; The outer vector binds each row; the inner vector destructures one row.
       [(fulltext $ :doc/body ?term) [[?doc-eid _ _]]]
       [(idoc-match $ :doc/idoc {:module {:status "stable"}}) [[?doc-eid _ _]]]
       [?doc-eid :doc/lang "en"]]
     (d/db conn) "vector")

(d/close conn)

import datalevin.*;

import java.util.Map;

// Schema is optional, but this example declares it for type checking and indexes.
Schema schema = Datalevin.schema()
    .attr(":doc/id",
          Schema.attribute()
              .valueType(Schema.ValueType.STRING)
              .unique(Schema.Unique.IDENTITY))
    .attr(":doc/body",
          Schema.attribute()
              .valueType(Schema.ValueType.STRING)
              .fulltext(true)
              .prop(":db.fulltext/autoDomain", true))
    .attr(":doc/lang",
          Schema.attribute().valueType(Schema.ValueType.STRING))
    .attr(":doc/idoc",
          Schema.attribute().valueType(Schema.ValueType.IDOC));

// Create a throwaway in-memory DB connection.
Connection conn = Datalevin.createConn(
    null,
    schema,
    Map.of(":kv-opts", Map.of(":inmemory?", true)));

try {
    // Transact one entity.
    conn.transact(Datalevin.tx()
        .entity(Tx.entity(-1)
            .put(":doc/id", "guide/search-overview")
            .put(":doc/lang", "en")
            .put(":doc/body", "Datalevin adds idoc indexing and vector search.")
            .put(":doc/idoc",
                 Datalevin.mapOf("module",
                     Datalevin.mapOf("name", "search", "status", "stable")))));

    // Run the query.
    Object results = conn.query(
        "[:find ?doc-eid " +
        " :in $ ?term " +
        " :where " +
        " [(fulltext $ :doc/body ?term) [[?doc-eid _ _]]] " +
        " [(idoc-match $ :doc/idoc {:module {:status \"stable\"}}) [[?doc-eid _ _]]] " +
        " [?doc-eid :doc/lang \"en\"]]",
        "vector");
} finally {
    conn.close();
}

from datalevin import connect

# Schema is optional, but this example declares it for type checking and indexes.
schema = {
    ":doc/id": {":db/valueType": ":db.type/string",
                ":db/unique": ":db.unique/identity"},
    ":doc/body": {":db/valueType": ":db.type/string",
                  ":db/fulltext": True,
                  ":db.fulltext/autoDomain": True},
    ":doc/lang": {":db/valueType": ":db.type/string"},
    ":doc/idoc": {":db/valueType": ":db.type/idoc"}}

# Create a throwaway in-memory DB connection.
with connect(None,
             schema=schema,
             opts={":kv-opts": {":inmemory?": True}}) as conn:
    # Transact an entity.
    conn.transact([
        {":db/id": -1,
         ":doc/id": "guide/search-overview",
         ":doc/lang": "en",
         ":doc/body": "Datalevin adds idoc indexing and vector search.",
         ":doc/idoc": {"module": {"name": "search", "status": "stable"}}}])

    # Run the query.
    results = conn.query("""
        [:find ?doc-eid
         :in $ ?term
         :where
         [(fulltext $ :doc/body ?term) [[?doc-eid _ _]]]
         [(idoc-match $ :doc/idoc {:module {:status "stable"}}) [[?doc-eid _ _]]]
         [?doc-eid :doc/lang "en"]]""",
        "vector")

import { connect } from "datalevin-node";

// Schema is optional, but this example declares it for type checking and indexes.
const schema = {
  ":doc/id": {
    ":db/valueType": ":db.type/string",
    ":db/unique": ":db.unique/identity"
  },
  ":doc/body": { ":db/valueType": ":db.type/string",
                 ":db/fulltext": true,
                 ":db.fulltext/autoDomain": true },
  ":doc/lang": { ":db/valueType": ":db.type/string" },
  ":doc/idoc": { ":db/valueType": ":db.type/idoc" }
};

// Create a throwaway in-memory DB connection.
const conn = await connect(null, {
  schema,
  opts: { ":kv-opts": { ":inmemory?": true } }
});

try {
  // Transact an entity.
  await conn.transact([
    { ":db/id": -1,
      ":doc/id": "guide/search-overview",
      ":doc/lang": "en",
      ":doc/body": "Datalevin adds idoc indexing and vector search.",
      ":doc/idoc": { "module": { "name": "search", "status": "stable" } } }
  ]);

  // Run the query.
  const results = await conn.query(
    `[:find ?doc-eid
      :in $ ?term
      :where
      [(fulltext $ :doc/body ?term) [[?doc-eid _ _]]]
      [(idoc-match $ :doc/idoc {:module {:status "stable"}}) [[?doc-eid _ _]]]
      [?doc-eid :doc/lang "en"]]`,
    "vector");
} finally {
  await conn.close();
}

In the query, $ represents the database. The double brackets in [[?doc-eid _ _]] are a binding pattern for a function that returns multiple rows. The outer vector means "for each returned row"; the inner vector destructures one [e a v] row.

The key point here is composition: word search, structure-aware filtering, and exact logical predicates are executed in one query context over one database state. In a split architecture, this often requires multiple systems and intermediate joins. Here it is a single declarative query over shared facts.

5. One Model Across Deployment Modes

The design choices described above are meant to survive contact with real deployment. A query should still be data when it moves from a local prototype to a production service. Attributes should carry the same meaning in application code, shell scripts, and AI tools. The application should not have to redesign its data model merely because the operational topology changes.

Datalevin therefore treats deployment flexibility as part of the design criteria. Deployment mode should not remake the model.

You can begin with an embedded database in the same process as the application, move to client/server when sharing or centralized operations are needed, use replicas or HA1 when availability requirements grow, and still script the same database from the command line. The facts, attributes, transactions, queries, and indexes remain the same across those modes.

Datalevin is designed to run where your application runs:

This flexibility lets teams start small and evolve deployment architecture without changing core data and queries. The same schema, datoms, Datalog queries, idoc values, and search indexes remain identical as you move from a local prototype to a server deployment or an AI tool integration. Operational shape can change later; the conceptual model does not have to be redesigned each time. That continuity is the practical point: Datalevin is meant to feel like one database across development, production, automation, and AI integration, rather than a collection of special-case systems connected by glue code.

6. Where Datalevin Fits

Given these design choices, Datalevin fits where SQL has become the default home for application state mainly because it is familiar, durable, and operationally serious, not because SQL is the right language.

Datalevin targets that default role: the operational application database that must transact, query, search, evolve, and feed application or AI logic without scattering state across extensions, ORMs, and side services.

Datalevin is especially strong when one operational database system should support:

This combination appears in more places than it may first seem. A SaaS product may need accounts, permissions, audit events, search, and recommendation features. A developer tool may need package dependency graphs, documentation search, local state, and structured metadata. An AI application may need episodic memory, semantic facts, vector recall, source documents, and strict filters that prevent irrelevant context from entering a prompt.

Datalevin is especially attractive when the boundaries between these needs are porous. If a search result must be filtered by permissions, joined to ownership metadata, checked against a lifecycle state, and then ranked with meaning-based signals, keeping those facts in one engine simplifies the application. In SQL, that kind of query often crosses ordinary joins, JSON expressions, search operators, vector extensions, permission tables, and application-side filtering. In Datalevin, the same application question can stay inside one fact-oriented query.

Another advantage is the ability to absorb change. Datalevin lets you start with a small schema, add declared structure where the application has learned enough to justify it, and keep optional facts absent rather than storing placeholder nulls. Declared schema is enforced when transactions write data. Because facts are isolated datoms and attributes are additive, new features can add new attributes without rewriting old entities or forcing every record through a table migration. Stable domain identifiers, lookup refs, and unique attributes let the application keep its own identities while Datalevin manages database-local entity ids. This makes the model well suited to systems whose shape changes as the product, data, or AI workflow matures.

Together, these choices make Datalevin a good default candidate for new applications whose data is already relational, graph-shaped, searchable, document-shaped, or AI-facing. It is also a modernization path for mature systems where SQL has become a friction point: too many hand-written joins, too much ORM ceremony, too many side indexes and auxiliary stores, or too much application code compensating for queries the database should be able to express directly.

That does not mean Datalevin should be the only database for all cases. If your workload is narrowly specialized, a single-purpose engine may still be the better choice. For example, a pure analytical warehouse, a massive append-only log pipeline, or a dedicated search cluster may be better served by systems built only for those jobs.

The target is broad but concrete: SQL's default role as the application database whose state must be modeled, queried, searched, evolved, and reasoned over in one coherent system. For developers who have been waiting for a database that keeps SQL's practical seriousness while offering a simpler, more composable query model, Datalevin is designed to be that replacement.

Summary

Datalevin is built to replace SQL database as the default center for application state when workloads cross boundaries between key-value storage, logical querying, graph relationships, document modeling, full-text search, and vector or embedding search. Its core move is to treat facts as the shared substrate and indexes as composable capabilities over that substrate. The remaining chapters develop these concepts in depth.

References

[1] Frederick P. Brooks, Jr., The Mythical Man-Month: Essays on Software Engineering, Anniversary Edition, Addison-Wesley, 1995.

[2] Howard Robinson and Ralph Weir, "Substance," The Stanford Encyclopedia of Philosophy, substantive revision May 6, 2024, especially Section 3.2, "Bundle theories versus substrata and thin particulars." URL: https://plato.stanford.edu/entries/substance/.

[3] Stefano Ceri, Georg Gottlob, and Letizia Tanca, "What You Always Wanted to Know About Datalog (And Never Dared to Ask)," IEEE Transactions on Knowledge and Data Engineering 1(1):146-166, 1989. URL: https://hdl.handle.net/11311/665510.

[4] Donald D. Chamberlin and Raymond F. Boyce, "SEQUEL: A Structured English Query Language," in Proceedings of the 1974 ACM SIGFIDET Workshop on Data Description, Access and Control, 1974, pp. 249-264. DOI: https://doi.org/10.1145/800296.811515.

[5] Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu, "Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows," arXiv:2411.07763, 2024. URL: https://arxiv.org/abs/2411.07763.

[6] Tanmay Parekh, Ella Hofmann-Coyle, Shuyi Wang, Sachith Sri Ram Kothur, Srivas Prasad, and Yunmo Chen, "PExA: Parallel Exploration Agent for Complex Text-to-SQL," arXiv:2604.22934, 2026. URL: https://arxiv.org/abs/2604.22934.

[7] Yuxing Han, Ziniu Wu, Peizhi Wu, Rong Zhu, Jingyi Yang, Liang Wei Tan, Kai Zeng, Gao Cong, Yanzhao Qin, Andreas Pfadler, Zhengping Qian, Jingren Zhou, Jiangneng Li, and Bin Cui, "Cardinality Estimation in DBMS: A Comprehensive Benchmark Evaluation," Proceedings of the VLDB Endowment 15(4), 2022. URL: https://arxiv.org/abs/2109.05877.

[8] Viktor Leis, Bernhard Radke, Andrey Gubichev, Alfons Kemper, and Thomas Neumann, "Cardinality Estimation Done Right: Index-Based Join Sampling," CIDR 2017. URL: https://www.cidrdb.org/cidr2017/papers/p9-leis-cidr17.pdf.

[9] PostgreSQL Global Development Group, "Controlling the Planner with Explicit JOIN Clauses," PostgreSQL Documentation. URL: https://www.postgresql.org/docs/current/explicit-joins.html.

[10] Viktor Leis, Andrey Gubichev, Atanas Mirchev, Peter Boncz, Alfons Kemper, and Thomas Neumann, "How Good Are Query Optimizers, Really?" Proceedings of the VLDB Endowment 9(3):204-215, 2015. URL: https://www.vldb.org/pvldb/vol9/p204-leis.pdf.

[11] Huahai Yang, "Competing for the JOB with a Triplestore," yyhh.org, 2024. URL: https://yyhh.org/blog/2024/09/competing-for-the-job-with-a-triplestore/.

[12] Datalevin project, "Join Order Benchmark," benchmark writeup and implementation. URL: https://github.com/datalevin/datalevin/tree/master/benchmarks/JOB-bench.

[13] Babashka project, "Babashka." URL: https://babashka.org/.

[14] Model Context Protocol project, "What is the Model Context Protocol (MCP)?" URL: https://modelcontextprotocol.io/.

  1. HA means high availability: a deployment designed to keep the service available when a server fails. Non-HA replication means read-only replicas can serve reads, but they do not automatically become the writer if the primary fails. Consensus-lease HA is Datalevin's HA server mode: servers use a consensus protocol to agree which server holds the time-limited write lease, so another server can be promoted when the leader is no longer safe to use. Chapter 22 covers the operational details. ↩

User Examples

Log in to create examples

No examples for this chapter yet.