Appendix F: Key-Value API

This appendix is a compact reference for public Datalevin KV functions exposed from datalevin.core. Chapter 10 is the tutorial introduction. Chapters 19 and 20 cover durability, snapshots, WAL, re-indexing, and ingestion strategy. Appendix A covers host-language binding conventions.

1. Store Lifecycle

This table lists common option groups, not every accepted key.

Chapter 19 covers WAL, snapshot, checkpoint, and retention tuning in detail. WAL is disabled by default; for a new store, when it is enabled and no durability profile is specified, the profile defaults to :strict. Existing stores retain their persisted profile as Chapter 19 describes. Without WAL, the default LMDB environment flags are #{:nordahead :notls}. Persistent WAL environments add :writemap when no explicit :flags option is supplied; WAL also enables LMDB :nosync, with durability supplied by the WAL profile and checkpoints explicitly syncing LMDB. Chapter 19 explains these defaults, and Chapter 20, Section 1.4 covers embedded write collection options.

d/datalog-kv returns a borrowed handle that is safe to retain and reuse for as long as its owning Datalog connection or raw DB remains open. Unlike the Datalog DB object returned by d/db, it does not need to be obtained again after writes. Close the owner, not the borrowed KV handle. re-index is an offline maintenance operation; use it only when other threads or programs are not accessing the same store.

For a Datalog database that also stores application KV DBIs through datalog-kv, the dtlv dump/dtlv load auto mode can round-trip the mixed store: Datalog schema and datoms plus user-created KV DBIs in the same LMDB environment. Appendix E lists the datalevin.main functions and shows the command-line form. Use copy when you want a physical database copy; use dump/load when you want a portable text export/import workflow.

2. DBI Management

Use regular DBIs for one value per key and list DBIs for sorted multi-values per key. :validate-data? checks keys and values against KV type descriptors; it is independent of Datalog schema validation. :key-type and :value-type name registered custom types for ordered keys and list items; Section 12 covers them.

3. Writes and Transactions

Transaction functions:

Transaction data forms:

key-type, value-type, and flags are optional. Common write flags include :nooverwrite, :nodupdata, :current, :reserve, :append, :appenddup, and :multiple. Use :append only with input already sorted in the target DBI's key order.

KV keys and put values cannot be nil. For list DBI writes, prefer put-list-items and del-list-items; they wrap the lower-level :put-list and :del-list transaction operators.

update-kv supports single-value DBIs only. Missing keys pass nil to f; exceptions or invalid/nil results abort the update. Use an inter-fn for remote stores; local calls also accept ordinary Clojure functions. The function may run more than once, so use it only to compute the new value. Passing an explicit transaction's KV handle makes the update part of that transaction. Chapter 10, Section 5.1 shows counter updates in all four languages.

prepare-update-kv fixes the function, DBI, types, and extra arguments. Each invocation atomically updates the supplied key and returns :transacted. For remote stores, use inter-fn; the preparation reuses the serialized and validated function per connection and registers again after reconnect or handle eviction. Keep the owning store open while reusing the preparation.

4. Encoding Helpers

For raw key and value encode and decode:

Data types:

KV type descriptors use unqualified keywords, not Datalog value-type keywords such as :db.type/string. Prefer one consistent key type, or one tuple key type, per application DBI.

For DBI operations, a registered custom type name can select a declared custom key or list item type (Section 12). Custom values need the store's type registration and payload storage; raw put-buffer and read-buffer calls do not encode or decode complete custom values on their own.

:ignore is not a put-buffer or read-buffer descriptor. It is a range-scan sentinel accepted by scan APIs when the caller wants to skip value decoding.

5. Point Reads and Order Statistics

Rank and sampling functions use DLMDB order-statistics support.

5.1 Prepared Point Reads

Prepared KV reads are available in Datalevin 1.2.0. Open the DBI before preparing the read. The Clojure forms are:

Execute with (d/execute-prepared prepared key) or (prepared key). Missing keys return nil. Defaults and custom-type resolution match get-value: :data selects a declared custom key or list-item type when applicable.

Types may be omitted to use :data. Returning only the value is the default; set ignoreKey / ignore_key to false to return the key and value together. Missing keys return null in Java and JavaScript, or None in Python.

Prepare once and reuse the read while the store is open; no separate close is needed. Each call reads the current value. To read uncommitted changes inside with-transaction-kv, prepare using its transaction handle. Chapter 7, Section 5 includes a complete example and remote behavior.

6. Key Ranges

Range specs use the keywords in Section 11. Use get-range for bounded results and range-seq for large scans.

7. Range Predicates and Visitors

raw-pred? controls whether callbacks receive raw storage objects or decoded values, and the raw shape depends on the helper. visit-key-range receives a raw key ByteBuffer; visit and range/list-range predicates over key/value pairs receive an IKV object; visit-list for one key receives a raw value ByteBuffer. Raw callbacks expose short-lived buffers; decode or copy data inside the callback if it must outlive that call.

8. List DBI Operations

Write functions:

Read functions:

List DBIs are opened with open-list-dbi. Each key maps to a sorted set of values, and list items must fit within the LMDB key-size limit.

9. WAL and Snapshots

10. Environment Flags

Functions:

Flags:

For a Datalog connection, call datalog-kv first and apply set-env-flags, get-env-flags, or sync to the backing KV handle.

11. Range Keywords

The same range shapes are used for key ranges and list value ranges.

12. Custom Data Types

register-type defines a custom type for ordered KV keys, ordered list items, and Datalog attribute values. All DBIs and Datalog attributes in the same database share the registration, which persists when the database is closed and reopened. Registration through a remote handle requires database alter permission. Chapter 10, Section 3.4 introduces custom types with examples.

12.1 Type Definitions

handle is a local or remote KV handle or Datalog connection. type-name is a namespaced keyword distinct from the built-in type names. A definition has these fields:

Supported scalar types for order keys are :keyword, :symbol, :string, :boolean, :long, :double, :float, :instant, :uuid, :bytes, :bigint, :bigdec, and :id. Use KV type names such as :long. For a tuple, declare a nonempty vector of these scalar types, excluding :bytes and :id. The order function must return a value of the declared type; for a tuple, it must return a vector with exactly the declared number of components, even for a one-component tuple.

Supply functions as datalevin.interpret/inter-fn values or UDF descriptors. An ordinary Clojure fn cannot be stored in a type definition. A UDF descriptor identifies a runtime implementation by language, role, id, and optional version:

{:udf/lang :python :udf/kind :order-fn
 :udf/id :task/order :udf/version 1}

Use UDF kind :order-fn for the order function, :serializer for the payload encoder, and :deserializer for the payload decoder. The serializer accepts a complete value and returns bytes; the deserializer reconstructs the value from those bytes.

inter-fn source and captured values persist with the registration. UDF implementations must be supplied at runtime: pass {:runtime-opts {:udf-registry registry}} when opening a local KV store or Datalog connection. Recreate the registry before reopening the database in a new process. Appendix E, Section 11.2 lists the registry helpers.

Registering the same definition again is safe; a conflicting definition is rejected. Neither increasing :version nor replacing a UDF implementation rebuilds indexes or converts stored payloads. For an incompatible change, register a new type name and migrate values into a new DBI or attribute while the old type's functions remain available.

12.2 DBI Declarations

DBI declarations select a registered type for a field:

A custom :value-type requires a list DBI. Declarations persist and cannot be changed by reopening the DBI with different options. Omitting an operation's type argument, or passing the default :data, selects the declared custom type. Other explicit types must match the declaration. A custom type cannot be used for an undeclared key or list item.

12.3 Language Bindings

Host-language registration and descriptor factories:

Definition maps use the field names above. String keys may include a leading colon. Java also accepts KVType scalar and tuple declarations. Python and JavaScript descriptor factories default to their respective host languages.

Python and JavaScript can also store and return instances of application classes. Define the order function and payload encoder and decoder, then bind the class to the type definition in the runtime registry before opening the database:

from dataclasses import dataclass
import json
from datalevin import UdfDescriptor, create_udf_registry, open_kv

@dataclass
class Task:
    priority: int
    name: str

registry = create_udf_registry()
registry.order_udf("task/order")(lambda task: task.priority)
registry.serializer_udf("task/encode")(
    lambda task: json.dumps([task.priority, task.name]).encode("utf-8"))
registry.deserializer_udf("task/decode")(
    lambda payload: Task(*json.loads(payload)))

definition = {
    "index": {"type": ":long",
              "order-fn": UdfDescriptor.order_fn("task/order")},
    "payload": {"serialize": UdfDescriptor.serializer("task/encode"),
                "deserialize": UdfDescriptor.deserializer("task/decode")},
}
registry.bind_native_type("app/task", Task, definition)

with open_kv("/tmp/datalevin-custom-python",
             opts={":runtime-opts": {":udf-registry": registry}}) as kv:
    kv.register_type("app/task", definition)
    kv.open_dbi("tasks", {":key-type": ":app/task"})
    kv.transact([(":put", Task(3, "Compile"), "queued")], dbi_name="tasks")
    assert kv.get_value("tasks", Task(3, "Compile")) == "queued"

import { UdfDescriptor, createUdfRegistry, openKv } from "datalevin-node";

class Task {
  constructor(priority, name) {
    this.priority = priority;
    this.name = name;
  }
}

const registry = await createUdfRegistry();
await registry.orderUdf("task/order", task => task.priority);
await registry.serializerUdf("task/encode", task =>
  Buffer.from(JSON.stringify([task.priority.toString(), task.name])));
await registry.deserializerUdf("task/decode", payload => {
  const [priority, name] = JSON.parse(payload.toString("utf8"));
  return new Task(BigInt(priority), name);
});
const definition = {
  index: { type: ":long", "order-fn": UdfDescriptor.orderFn("task/order") },
  payload: {
    serialize: UdfDescriptor.serializer("task/encode"),
    deserialize: UdfDescriptor.deserializer("task/decode")
  }
};
await registry.bindNativeType("app/task", Task, definition);

const kv = await openKv("/tmp/datalevin-custom-javascript", {
  ":runtime-opts": { ":udf-registry": registry }
});
try {
  await kv.registerType("app/task", definition);
  await kv.openDbi("tasks", { ":key-type": ":app/task" });
  await kv.transact([[":put", new Task(3n, "Compile"), "queued"]],
                    { dbiName: "tasks" });
  console.log(await kv.getValue("tasks", new Task(3n, "Compile"))); // queued
} finally {
  await kv.close();
}

Exact matching compares decoded values using Python == or JavaScript's node:util.isDeepStrictEqual. JavaScript also accepts an equals argument to bindNativeType for custom equality. Native class bindings exist only at runtime and must be supplied again when reopening the database in a new process.

Native instances can be custom keys, list items, Datalog attribute values, range endpoints, lookup references, and query inputs. They also work with remote operations and queries that spill to disk. An entity returns a custom attribute with cardinality many as a Python list or a JavaScript Set.

Native instances require a custom type declaration. They cannot be stored in untyped :data payloads or passed to helpers that convert values before a database handle is available. Within the JVM, native values currently share a single hash code, which can make large hash joins, maps, sets, and deduplication expensive.

12.4 Remote Databases

For remote databases, order and payload functions execute on the server. Install UDF bindings there through datalevin.server/*server-runtime-opts-fn*; client registries are not sent to the server. To reconstruct native class instances on the client, also supply the caller's registry through :runtime-opts when opening the remote handle.