Appendix F: Key-Value API

This appendix is a compact reference for public Datalevin KV functions exposed from datalevin.core. Chapter 10 is the tutorial introduction. Chapters 19 and 20 cover durability, snapshots, WAL, re-indexing, and ingestion strategy. Appendix A covers host-language binding conventions.

1. Store Lifecycle

Function	Common forms	Purpose
`open-kv`	`(d/open-kv dir)`, `(d/open-kv dir opts)`	Open a local KV store or a remote `dtlv://` KV database.
`close-kv`	`(d/close-kv db)`	Close a KV store.
`closed-kv?`	`(d/closed-kv? db)`	Return true if the KV store is closed.
`dir`	`(d/dir db)`	Return the path or URI string for the store.
`datalog-kv`	`(d/datalog-kv conn)`, `(d/datalog-kv db)`	Return the KV handle backing a Datalog connection or DB.
`copy`	`(d/copy db dest)`, `(d/copy db dest compact?)`	Copy a Datalog or KV database to another directory, optionally compacting.
`sync`	`(d/sync db)`	Force a synchronous flush to disk.
`re-index`	`(d/re-index db opts)`	Dump, clear, reload, and rebuild a KV store with new options.

This table lists common option groups, not every accepted key.

Common option group	Keys
Capacity and environment	`:mapsize`, `:max-dbs`, `:max-readers`, `:flags`, `:temp?`, `:inmemory?`
Store identity	`:db-name`, `:db-identity`
WAL commit path	`:wal?`, `:wal-durability-profile`, `:wal-group-commit`, `:wal-group-commit-ms`
LMDB checkpointing	`:lmdb-sync-interval-ms`, `:lmdb-sync-interval`, `:wal-checkpoint-stale-threshold-ms`
WAL segments and retention	`:wal-segment-max-bytes`, `:wal-segment-max-ms`, segment preallocation options, `:wal-retention-bytes`, `:wal-retention-ms`, retention backpressure options
Snapshots	`:snapshot-scheduler?`, `:snapshot-dir`, `:snapshot-interval-ms`, `:snapshot-max-lsn-delta`, `:snapshot-max-log-bytes-delta`, `:snapshot-max-age-ms`, snapshot contention and defer options
Client	`:client-opts`
Spill to disk	`:spill-opts`
HA	`:ha-mode`, `:ha-control-plane`, lease timing, fencing, promotion, and clock-skew options

Chapter 19 covers WAL, snapshot, checkpoint, and retention tuning in detail. The default LMDB environment flags are #{:nordahead :notls}. datalog-kv returns a borrowed handle owned by the Datalog connection. re-index is an offline maintenance operation; use it only when other threads or programs are not accessing the same store.

For a Datalog database that also stores application KV DBIs through datalog-kv, the dtlv dump/dtlv load auto mode can round-trip the mixed store: Datalog schema and datoms plus user-created KV DBIs in the same LMDB environment. Appendix E lists the datalevin.main functions and shows the command-line form. Use copy when you want a physical database copy; use dump/load when you want a portable text export/import workflow.

2. DBI Management

Function	Common forms	Purpose
`open-dbi`	`(d/open-dbi db name)`, `(d/open-dbi db name opts)`	Open a regular named DBI.
`open-list-dbi`	`(d/open-list-dbi db name)`, `(d/open-list-dbi db name opts)`	Open a DUPSORT/list DBI.
`clear-dbi`	`(d/clear-dbi db name)`	Remove data from a DBI but leave the DBI open.
`drop-dbi`	`(d/drop-dbi db name)`	Clear and delete a DBI.
`list-dbis`	`(d/list-dbis db)`	Return names of sub-databases in the store.
`stat`	`(d/stat db)`, `(d/stat db name)`	Return LMDB statistics for the environment or a DBI.
`entries`	`(d/entries db name)`	Return the number of entries in a DBI.

Use regular DBIs for one value per key and list DBIs for sorted multi-values per key. :validate-data? checks keys and values against KV type descriptors; it is independent of Datalog schema validation.

Option or flag group	Values
DBI options	`:key-size`, `:val-size`, `:validate-data?`, `:closed-schema?`, `:flags`
Default DBI flags	`#{:create :counted :prefix-compression}`
Common LMDB flags	`:create`, `:reversekey`, `:dupsort`, `:integerkey`, `:dupfixed`, `:integerdup`, `:reversedup`, `:counted`, `:prefix-compression`

3. Writes and Transactions

Transaction functions:

Function	Common forms	Purpose
`transact-kv`	`(d/transact-kv db txs)`, `(d/transact-kv db name txs k-type v-type)`	Synchronously apply KV write operations.
`transact-kv-async`	`(d/transact-kv-async db txs)`, `(d/transact-kv-async db name txs k-type v-type callback)`	Batch asynchronous KV writes; returns a future.
`with-transaction-kv`	`(d/with-transaction-kv [tx db] body...)`, `(d/with-transaction-kv [tx db {:timeout-ms n}] body...)`	Run reads and writes in one explicit KV transaction, optionally with a timeout.
`abort-transact-kv`	`(d/abort-transact-kv tx)`	Roll back writes from inside `with-transaction-kv`.

Transaction data forms:

Transaction item	Shape
Self-contained put	`[:put dbi-name key value]`, `[:put dbi-name key value key-type value-type]`, or `[:put dbi-name key value key-type value-type flags]`
Self-contained delete	`[:del dbi-name key]` or `[:del dbi-name key key-type]`
Single-DBI put	`[:put key value]` or `[:put key value flags]`
Single-DBI delete	`[:del key]`
List operators	`:put-list` and `:del-list` use the same transaction shapes as puts, with a sequential value collection.

key-type, value-type, and flags are optional. Common write flags include :nooverwrite, :nodupdata, :current, :reserve, :append, :appenddup, and :multiple. Use :append only with input already sorted in the target DBI's key order.

KV keys and put values cannot be nil. For list DBI writes, prefer put-list-items and del-list-items; they wrap the lower-level :put-list and :del-list transaction operators.

4. Encoding Helpers

For raw key and value encode and decode:

Function	Common forms	Purpose
`put-buffer`	`(d/put-buffer bf x)`, `(d/put-buffer bf x x-type)`	Encode a value into a `ByteBuffer`. Mostly useful in raw visitors.
`read-buffer`	`(d/read-buffer bf)`, `(d/read-buffer bf v-type)`	Decode a value from a `ByteBuffer`. Mostly useful in raw visitors.
`k`	`(d/k kv)`	Return the raw key buffer from an `IKV` visitor object.
`v`	`(d/v kv)`	Return the raw value buffer from an `IKV` visitor object.

Data types:

Descriptor form	Meaning
`:data`	General serialized data.
`:string`, `:long`, `:float`, `:double`, `:bigint`, `:bigdec`	Scalar numeric or string encodings.
`:bytes`, `:keyword`, `:symbol`, `:boolean`, `:instant`, `:uuid`	Other scalar encodings.
`[:string :instant]`, `[:long]`	Tuple key or value encodings.

KV type descriptors use unqualified keywords, not Datalog value-type keywords such as :db.type/string. Prefer one consistent key type, or one tuple key type, per application DBI.

:ignore is not a put-buffer or read-buffer descriptor. It is a range-scan sentinel accepted by scan APIs when the caller wants to skip value decoding.

5. Point Reads and Order Statistics

Function	Common forms	Purpose
`get-value`	`(d/get-value db name k)`, `(d/get-value db name k k-type v-type ignore-key?)`	Get one key's value, or `[k v]` when `ignore-key?` is false.
`get-rank`	`(d/get-rank db name k)`, `(d/get-rank db name k k-type)`	Return a key's zero-based rank in sorted key order.
`get-by-rank`	`(d/get-by-rank db name rank)`, `(d/get-by-rank db name rank k-type v-type ignore-key?)`	Return the value or pair at a zero-based key rank.
`sample-kv`	`(d/sample-kv db name n)`, `(d/sample-kv db name n k-type v-type ignore-key?)`	Return `n` random samples from a DBI, or `nil` if `n` exceeds the entry count.

Rank and sampling functions use DLMDB order-statistics support.

6. Key Ranges

Function	Common forms	Purpose
`get-first`	`(d/get-first db name k-range)`, `(d/get-first db name k-range k-type v-type ignore-key?)`	Return the first value or pair in a key range.
`get-first-n`	`(d/get-first-n db name n k-range)`, `(d/get-first-n db name n k-range k-type v-type ignore-key?)`	Return the first `n` values or pairs in a key range.
`get-range`	`(d/get-range db name k-range)`, `(d/get-range db name k-range k-type v-type ignore-key?)`	Eagerly return values or pairs in a key range.
`range-seq`	`(d/range-seq db name k-range)`, `(d/range-seq db name k-range k-type v-type ignore-key? opts)`	Lazily stream a key range in batches; close it with `with-open`.
`key-range`	`(d/key-range db name k-range)`, `(d/key-range db name k-range k-type)`	Return only keys in a range.
`range-count`	`(d/range-count db name k-range)`, `(d/range-count db name k-range k-type)`	Count KV pairs in a key range without materializing them.
`key-range-count`	`(d/key-range-count db name k-range k-type)`, `(d/key-range-count db name k-range k-type cap)`	Count keys in a range, optionally stopping at `cap`.
`key-range-list-count`	`(d/key-range-list-count db name k-range k-type)`, `(d/key-range-list-count db name k-range k-type cap)`	Count list items across keys in a list DBI key range.

Range specs use the keywords in Section 11. Use get-range for bounded results and range-seq for large scans.

7. Range Predicates and Visitors

Function	Common forms	Purpose
`get-some`	`(d/get-some db name pred k-range ...)`	Return the first KV pair whose predicate is true.
`range-filter`	`(d/range-filter db name pred k-range ...)`	Eagerly return KV pairs whose predicate is true.
`range-keep`	`(d/range-keep db name pred k-range ...)`	Return non-nil predicate results over a key range.
`range-some`	`(d/range-some db name pred k-range ...)`	Return the first logical true predicate result.
`range-filter-count`	`(d/range-filter-count db name pred k-range ...)`	Count KV pairs whose predicate is true.
`visit-key-range`	`(d/visit-key-range db name visitor k-range ...)`	Visit only keys in a key range for side effects.
`visit`	`(d/visit db name visitor k-range ...)`	Visit KV pairs in a key range for side effects.

raw-pred? controls whether callbacks receive raw storage objects or decoded values, and the raw shape depends on the helper. visit-key-range receives a raw key ByteBuffer; visit and range/list-range predicates over key/value pairs receive an IKV object; visit-list for one key receives a raw value ByteBuffer. Raw callbacks expose short-lived buffers; decode or copy data inside the callback if it must outlive that call.

8. List DBI Operations

Write functions:

Function	Common forms	Purpose
`put-list-items`	`(d/put-list-items db name k vs k-type v-type)`	Add multiple values under one key.
`del-list-items`	`(d/del-list-items db name k k-type)`, `(d/del-list-items db name k vs k-type v-type)`	Delete a whole list or selected list items.

Read functions:

Function	Common forms	Purpose
`get-list`	`(d/get-list db name k k-type v-type)`	Return all values for one key.
`visit-list`	`(d/visit-list db name visitor k k-type v-type raw-pred?)`	Visit values for one key.
`list-count`	`(d/list-count db name k k-type)`	Count values associated with one key.
`in-list?`	`(d/in-list? db name k v k-type v-type)`	Test whether one value is present under a key.
`list-range`	`(d/list-range db name k-range k-type v-range v-type)`	Return key/value pairs matching both ranges.
`list-range-count`	`(d/list-range-count db name k-range k-type)`	Approximate count across a key range; ignores value range boundaries.
`list-range-first`	`(d/list-range-first db name k-range k-type v-range v-type)`	Return the first matching list item.
`list-range-first-n`	`(d/list-range-first-n db name n k-range k-type v-range v-type)`	Return the first `n` matching list items.
`list-range-filter`	`(d/list-range-filter db name pred k-range k-type v-range v-type raw-pred?)`	Filter list range results with a predicate.
`list-range-keep`	`(d/list-range-keep db name pred k-range k-type v-range v-type raw-pred?)`	Return non-nil predicate results over list range matches.
`list-range-filter-count`	`(d/list-range-filter-count db name pred k-range k-type v-range v-type raw-pred?)`	Count list range matches whose predicate is true.
`list-range-some`	`(d/list-range-some db name pred k-range k-type v-range v-type raw-pred?)`	Return the first logical true predicate result.
`visit-list-range`	`(d/visit-list-range db name visitor k-range k-type v-range v-type raw-pred?)`	Visit list range matches for side effects.

List DBIs are opened with open-list-dbi. Each key maps to a sorted set of values, and list items must fit within the LMDB key-size limit.

9. WAL and Snapshots

Function	Common forms	Purpose
`txlog-watermarks`	`(d/txlog-watermarks db)`	Return WAL watermarks and runtime sync state, or `{:wal? false}` when WAL is disabled.
`open-tx-log`	`(d/open-tx-log db from-lsn)`, `(d/open-tx-log db from-lsn upto-lsn)`	Read committed WAL records from an LSN range.
`force-lmdb-sync!`	`(d/force-lmdb-sync! db)`	Force an LMDB environment sync for the store.
`create-snapshot!`	`(d/create-snapshot! db)`	Create or rotate LMDB snapshots and update WAL snapshot bookkeeping.
`list-snapshots`	`(d/list-snapshots db)`	Return available snapshot metadata.
`gc-txlog-segments!`	`(d/gc-txlog-segments! db)`, `(d/gc-txlog-segments! db retain-floor-lsn)`	Garbage-collect WAL segments, optionally retaining records from a floor LSN.

10. Environment Flags

Functions:

Function	Common forms	Purpose
`set-env-flags`	`(d/set-env-flags db flags on?)`	Set or clear LMDB environment flags.
`get-env-flags`	`(d/get-env-flags db)`	Return currently active LMDB environment flags.

Flags:

Common flag group	Keywords
Read and locking	`:rdonly-env`, `:notls`, `:nolock`
Durability and write behavior	`:nosync`, `:nometasync`, `:writemap`, `:mapasync`
Filesystem and memory	`:nordahead`, `:nomeminit`, `:nosubdir`, `:fixedmap`

For a Datalog connection, call datalog-kv first and apply set-env-flags, get-env-flags, or sync to the backing KV handle.

11. Range Keywords

Range type	Meaning
`:all` / `:all-back`	Whole keyspace, forward or backward.
`:at-least` / `:at-most-back`	Inclusive lower bound, forward or backward.
`:at-most` / `:at-least-back`	Inclusive upper bound, forward or backward.
`:closed` / `:closed-back`	Inclusive lower and upper bounds.
`:closed-open` / `:closed-open-back`	Inclusive lower bound, exclusive upper bound.
`:open-closed` / `:open-closed-back`	Exclusive lower bound, inclusive upper bound.
`:open` / `:open-back`	Exclusive lower and upper bounds.
`:greater-than` / `:less-than-back`	Exclusive lower bound.
`:less-than` / `:greater-than-back`	Exclusive upper bound.

The same range shapes are used for key ranges and list value ranges.

User Examples

No examples for this chapter yet.