Chapter 20: High-Throughput Ingestion

High write throughput is a different problem from low-latency reads or interactive transactions. This chapter collects the tools Datalevin offers for loading large volumes of data quickly: WAL mode and asynchronous transactions for durable online ingestion, non-durable LMDB flags for workloads that accept weaker crash behavior, pre-sorting to keep B+Tree inserts cheap, and the init-db/fill-db bulk-load path that bypasses normal transaction processing entirely. It closes with a decision flow for choosing among them. For the durability trade-offs behind these choices, see Chapter 19.

The short version is:

If this is an offline load and you can prepare final datoms, use init-db or fill-db.
If weaker crash behavior is acceptable, use non-durable LMDB flags.
If the workload is interactive or modest, use ordinary transact! batches.
If durable online ingestion needs more throughput, use manual batches with transact-async, often with WAL mode.

1. Online Ingestion: WAL and Async Transactions

For ingestion while the database remains queryable, the usual recipe is manual batches submitted through transact-async, often with WAL mode enabled. Chapter 19 explains WAL durability profiles and LSN maintenance; here the point is how to submit work without making every caller wait for its own commit. The crash-durability boundary still comes from the selected LMDB or WAL durability settings.

The async transaction API submits ordinary transaction data and returns a future or promise. Datalevin can then batch concurrent submissions internally. In the example below, each batch-N is a manually formed transaction chunk; sizing is covered next.

;; Fire multiple async transactions
(def f1 (d/transact-async conn batch-1))
(def f2 (d/transact-async conn batch-2))
(def f3 (d/transact-async conn batch-3))

;; Block on the last one as a queue barrier.
@f3

import java.util.Map;
import java.util.concurrent.CompletableFuture;

CompletableFuture<Map<?, ?>> f1 = conn.transactAsync(batch1);
CompletableFuture<Map<?, ?>> f2 = conn.transactAsync(batch2);
CompletableFuture<Map<?, ?>> f3 = conn.transactAsync(batch3);

// Block on the last one as a queue barrier.
f3.join();

future1 = conn.transact_async(batch_1)
future2 = conn.transact_async(batch_2)
future3 = conn.transact_async(batch_3)

# Block on the last one as a queue barrier.
future3.result(timeout=30)

const p1 = conn.transactAsync(batch1);
const p2 = conn.transactAsync(batch2);
const p3 = conn.transactAsync(batch3);

// Block on the last one as a queue barrier.
await p3;

Block on the last submitted future when later work depends on the whole burst. Since async transactions for the same database are queued in order, the last realized future is a barrier for prior submissions in that burst. Keep the individual futures or promises when you need to observe each transaction's success or failure separately.

1.1 Manual Batching

Manual batching means grouping many transaction values into each transact! or transact-async call instead of writing one row or entity at a time. The goal is to amortize transaction setup and commit overhead while keeping memory use, validation cost, retry cost, and transaction-report size bounded.

There is no universal batch size. Start with batches large enough to amortize per-transaction overhead, such as thousands of simple entity maps, then measure with representative data. Each batch is still one transaction: if it fails, the whole batch fails, and the retry unit is that batch. transact-async can batch concurrent submissions internally, and that adaptive batching compounds with manual batching; it does not replace shaping the input into sensible chunks. The sorting and cache-tuning sections below use this same batch shape after the input has been ordered for cheaper B+Tree writes.

1.2 Commit Barriers After Async Work

For embedded applications that need both good throughput and deterministic handoff points, use a sequence of async transactions followed by an empty blocking transaction:

;; Async writes for throughput
(d/transact-async conn batch-1)
(d/transact-async conn batch-2)
(d/transact-async conn batch-3)

;; Empty d/transact queues behind the prior async work and blocks
(d/transact conn [])

import java.util.List;

// Async writes for throughput.
conn.transactAsync(batch1);
conn.transactAsync(batch2);
conn.transactAsync(batch3);

// Empty transact queues behind the prior async work and blocks.
conn.transact(List.of());

# Async writes for throughput.
conn.transact_async(batch_1)
conn.transact_async(batch_2)
conn.transact_async(batch_3)

# Empty transact queues behind the prior async work and blocks.
conn.transact([])

// Async writes for throughput.
conn.transactAsync(batch1);
conn.transactAsync(batch2);
conn.transactAsync(batch3);

// Empty transact queues behind the prior async work and blocks.
await conn.transact([]);

The empty transaction is a barrier because transact is the blocking form of transact-async: it is submitted through the same async transaction queue after the earlier async transactions and blocks until its own future is realized. A successful return means the barrier itself has committed and all earlier submissions have finished; if those earlier transactions succeeded, they have committed too. Keep the futures or promises when you need to observe each result separately; waiting on the last submitted async result is also a queue barrier. Crash durability still follows the current durability mode. Under WAL :relaxed, committed LSNs can temporarily be ahead of durable LSNs; use the watermarks from Chapter 19 when an operator needs to confirm the durable boundary.

Async barriers do not change durability

Use the empty blocking transaction as a handoff point for commit ordering, not as a stronger crash-durability guarantee. Under relaxed WAL, use watermarks when the operator needs to confirm what is durable on storage.

In the default direct LMDB path, writes still coordinate through the single-writer B+Tree transaction model. WAL mode can improve online ingestion through sequential append and group commit, but most jobs should combine manual batches with transact-async and measure before adding writer threads.

2. Non-Durable LMDB Flags

WAL mode and async transactions are the first tools to try when data must remain durable. This section is for workloads where weaker crash behavior is acceptable: one-time imports, cache rebuilds, temporary KV stores, derived indexes, test fixtures, disposable data, jobs with an external source of truth, or cases where losing recent writes after a crash simply does not matter. For those cases, LMDB exposes environment flags that reduce disk-sync work at commit time. These flags can apply to both Datalog connections and raw KV stores because both ultimately write through LMDB. When you supply a full :flags value, include Datalevin's default flags unless you intentionally want to replace them.

Flags	What changes	Use when	Risk
`:nometasync`	Sync data pages but not meta pages on commit	You want a modest speedup while preserving database integrity after a crash	The last transaction may be lost
`:nosync`	Skip the normal synchronous flush on commit and leave flushing to the OS	Lost or corrupted recent work is acceptable	A system crash may corrupt the database
`:writemap` + `:mapasync`	Use a writable memory map with asynchronous flushing	High raw write speed, when the weaker crash boundary is acceptable	A crash may corrupt the database; buggy native code can overwrite mapped DB memory; some OSes preallocate the full map size

The Datalevin write benchmark [3] tests these flags with both transact! and transact-async, and with both transact-kv and transact-kv-async. In the KV pure-write benchmark, :nosync and :writemap/:mapasync move throughput from the durable baseline into the hundreds of thousands to over a million writes per second depending on batch size. The same benchmark also shows that KV batching has a stronger effect than Datalog batching, so do not treat environment flags as a substitute for large batches.

(require '[datalevin.core :as d])
(require '[datalevin.constants :as c])

;; Raw KV store using non-durable flags.
(def kv
  (d/open-kv "/tmp/import-kv"
             {:flags (-> c/default-env-flags
                         (conj :writemap)
                         (conj :mapasync))}))

;; Datalog connection using the same LMDB flags.
(def conn
  (d/get-conn "/tmp/import-dl" schema
              {:kv-opts {:flags (conj c/default-env-flags :nometasync)}}))

import datalevin.*;
import java.util.List;
import java.util.Map;

// Raw KV store using non-durable flags.
KV kv = Datalevin.openKV("/tmp/import-kv",
    Map.of(":flags", List.of(":nordahead", ":notls", ":writemap", ":mapasync")));

// Datalog connection using the same LMDB flags.
Connection conn = Datalevin.getConn("/tmp/import-dl", schema,
    Map.of(":kv-opts", Map.of(":flags", List.of(":nordahead", ":notls",
                                               ":nometasync"))));

from datalevin import connect, open_kv

# Raw KV store using non-durable flags.
kv = open_kv("/tmp/import-kv",
    opts={":flags": [":nordahead", ":notls", ":writemap", ":mapasync"]})

# Datalog connection using the same LMDB flags.
conn = connect("/tmp/import-dl", schema=schema,
    opts={":kv-opts": {":flags": [":nordahead", ":notls", ":nometasync"]}})

import { connect, openKv } from "datalevin-node";

// Raw KV store using non-durable flags.
const kv = await openKv("/tmp/import-kv",
  { ":flags": [":nordahead", ":notls", ":writemap", ":mapasync"] });

// Datalog connection using the same LMDB flags.
const conn = await connect("/tmp/import-dl", {
  schema,
  opts: { ":kv-opts": { ":flags": [":nordahead", ":notls", ":nometasync"] } }
});

If you use :nosync or :writemap/:mapasync, make the crash boundary an explicit application choice. The data might be disposable, reproducible from a durable source, protected by a higher-level log, or simply not worth preserving after a system crash. For raw KV jobs that need a handoff point, call the KV sync operation at explicit checkpoints or before handing the database to normal durable workloads. For online systems with irreplaceable writes, prefer WAL mode with async transactions and tune durability there instead.

Use weak sync only for rebuildable data

Non-durable LMDB flags can make imports much faster, but they move failure handling out of the database and into your recovery plan. Use them for repeatable loads, caches, and derived data, not for primary OLTP writes that must survive a crash.

3. Preparing Transaction Batches

After you decide to use transaction-based ingestion, prepare each batch so it is cheap to apply. Two practical levers are ordering the input before transaction submission and disabling cache work that does not help a one-pass bulk job.

3.1 Pre-Sorting Your Data

A B+Tree performs best when keys are inserted in sorted order. If you insert random entity ids and random values, the engine must "jump around" the B+Tree, leading to frequent cache misses and page splits. A page split happens when a B+Tree page becomes full and must be divided into two pages, with parent pages updated to point at the new key ranges. Random insert order causes more of this rebalancing work; sorted input lets the tree append and fill pages more predictably.

If you are performing a bulk import (e.g., from a CSV or a SQL dump), sort the transaction data before it reaches the database.

Sort before the write path

Pre-sorting is an import-shaping step, not something the database should discover after work arrives. Group entity-local facts for EAV locality, and sort prepared datoms by attribute and value when AVE locality matters.

Group by entity id: This ensures attributes for a single entity are written to the EAV index contiguously.
For prepared datoms, also sort by attribute and value where practical: this makes updates to the AVE index more localized. For entity maps, entity grouping is usually the easiest useful first step.

;; Sort data by entity id
(let [sorted-datoms (sort-by :db/id raw-data)]
  (doseq [batch (partition-all 5000 sorted-datoms)]
    (d/transact! conn batch)))

// Sort data by entity id
rawData.sort(Comparator.comparing(m -> (Long) m.get(":db/id")));
List<List<Map>> batches = partition(rawData, 5000);
for (List<Map> batch : batches) {
    conn.transact(batch);
}

# Sort data by entity id
sorted_datoms = sorted(raw_data, key=lambda x: x[":db/id"])
for batch in partition_all(sorted_datoms, 5000):
    conn.transact(batch)

// Sort data by entity id
const sortedDatoms = rawData.sort((a, b) => a[":db/id"] - b[":db/id"]);
for (const batch of partitionAll(sortedDatoms, 5000)) {
  await conn.transact(batch);
}

3.2 Disable the Datalog Index Cache During Bulk Transactions

Datalevin keeps a small Datalog index cache for normal interactive workloads. During a large bulk transaction job, that cache can add memory pressure and cache maintenance work without helping much, especially when the import touches a wide range of entities and attributes only once. In every binding, set the cache limit to 0 for the import phase, then restore the previous limit:

(let [db             (d/db conn)
      previous-limit (d/datalog-index-cache-limit db)]
  (try
    (d/datalog-index-cache-limit db 0)
    (doseq [batch (partition-all 5000 sorted-datoms)]
      (d/transact! conn batch))
    (finally
      (d/datalog-index-cache-limit (d/db conn) previous-limit))))

long previousLimit = conn.datalogIndexCacheLimit();

try {
    conn.datalogIndexCacheLimit(0);
    for (List<Map<String, Object>> batch : partition(sortedDatoms, 5000)) {
        conn.transact(batch);
    }
} finally {
    conn.datalogIndexCacheLimit(previousLimit);
}

previous_limit = conn.datalog_index_cache_limit()

try:
    conn.datalog_index_cache_limit(0)
    for batch in partition_all(sorted_datoms, 5000):
        conn.transact(batch)
finally:
    conn.datalog_index_cache_limit(previous_limit)

const previousLimit = await conn.datalogIndexCacheLimit();

try {
  await conn.datalogIndexCacheLimit(0);
  for (const batch of partitionAll(sortedDatoms, 5000)) {
    await conn.transact(batch);
  }
} finally {
  await conn.datalogIndexCacheLimit(previousLimit);
}

This is a bulk-ingestion tuning knob, not a general query-performance setting. Leave the cache enabled for mixed read/write application traffic unless a measured workload shows that disabling it helps.

Restore cache settings after imports

Disabling the Datalog index cache is useful for one-pass bulk transaction jobs. Save the previous limit and restore it afterward so normal interactive traffic keeps the read-side cache behavior it expects.

4. Bulk Load: `d/init-db` and `d/fill-db`

For the highest-throughput bulk load into a database, bypass normal transaction processing using d/init-db and d/fill-db. Note that these functions bypass the WAL and will not appear in the transaction log.

Bulk load bypasses transaction semantics

init-db and fill-db are for prepared datoms with final numeric ids. Use normal transactions when you need tempids, lookup refs, upserts, transaction functions, transaction-log append, or transaction-level integrity checks.

d/init-db: Load a collection of prepared datoms into a fresh, empty database
d/fill-db: Bulk load additional datoms into an existing database

4.1 Assigning Entity Ids for Prepared Datoms

init-db and fill-db do not accept transaction maps, tempids, lookup refs, or upserts. They accept prepared Datom values. That means the subject entity id and every :db.type/ref value must already be the final numeric entity id.

A simple technique works well for relational imports whose source tables already have integer primary keys: reserve a non-overlapping entity-id range for each source table, then convert every primary key and foreign key with the same function [1,2]. For example:

(def schema
  {:user/id        {:db/valueType :db.type/long
                    :db/unique    :db.unique/identity}
   :user/name      {:db/valueType :db.type/string}
   :order/id       {:db/valueType :db.type/long
                    :db/unique    :db.unique/identity}
   :order/customer {:db/valueType :db.type/ref}
   :order/total    {:db/valueType :db.type/double}})

;; Reserve disjoint numeric ranges. This is the same idea used by the
;; JOB benchmark, where each imported table has its own eid base.
(def user-base  1000000)
(def order-base 2000000)

(defn user-eid [source-id]
  (+ user-base (parse-long source-id)))

(defn order-eid [source-id]
  (+ order-base (parse-long source-id)))

(def users
  [{:id "1" :name "Alice"}
   {:id "2" :name "Bob"}])

(def orders
  [{:id "10" :customer-id "1" :total 42.5}
   {:id "11" :customer-id "2" :total 19.0}])

(defn user-datoms [{:keys [id name]}]
  (let [eid (user-eid id)]
    [(d/datom eid :user/id (parse-long id))
     (d/datom eid :user/name name)]))

(defn order-datoms [{:keys [id customer-id total]}]
  (let [eid (order-eid id)]
    [(d/datom eid :order/id (parse-long id))
     (d/datom eid :order/customer (user-eid customer-id))
     (d/datom eid :order/total total)]))

(def prepared-datoms
  (vec (concat (mapcat user-datoms users)
               (mapcat order-datoms orders))))

;; Load into empty database
(def db (d/init-db prepared-datoms "/tmp/import-db" schema))

import datalevin.*;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;

static long userEid(String sourceId) {
    return 1_000_000L + Long.parseLong(sourceId);
}

static long orderEid(String sourceId) {
    return 2_000_000L + Long.parseLong(sourceId);
}

static List<Object> userDatoms(Map<String, ?> user) {
    String id = (String) user.get("id");
    long eid = userEid(id);

    return Datalevin.listOf(
        Datalevin.datom(eid, ":user/id", Long.parseLong(id)),
        Datalevin.datom(eid, ":user/name", user.get("name")));
}

static List<Object> orderDatoms(Map<String, ?> order) {
    String id = (String) order.get("id");
    String customerId = (String) order.get("customerId");
    long eid = orderEid(id);

    return Datalevin.listOf(
        Datalevin.datom(eid, ":order/id", Long.parseLong(id)),
        Datalevin.datom(eid, ":order/customer", userEid(customerId)),
        Datalevin.datom(eid, ":order/total", order.get("total")));
}

Map<?, ?> schema = Map.of(
    ":user/id", Map.of(
        ":db/valueType", ":db.type/long",
        ":db/unique", ":db.unique/identity"),
    ":user/name", Map.of(":db/valueType", ":db.type/string"),
    ":order/id", Map.of(
        ":db/valueType", ":db.type/long",
        ":db/unique", ":db.unique/identity"),
    ":order/customer", Map.of(":db/valueType", ":db.type/ref"),
    ":order/total", Map.of(":db/valueType", ":db.type/double"));

List<Map<String, ?>> users = List.of(
    Map.of("id", "1", "name", "Alice"),
    Map.of("id", "2", "name", "Bob"));

List<Map<String, ?>> orders = List.of(
    Map.of("id", "10", "customerId", "1", "total", 42.5),
    Map.of("id", "11", "customerId", "2", "total", 19.0));

List<Object> preparedDatoms = new ArrayList<>();
for (Map<String, ?> user : users) {
    preparedDatoms.addAll(userDatoms(user));
}
for (Map<String, ?> order : orders) {
    preparedDatoms.addAll(orderDatoms(order));
}

try (Connection conn =
         Datalevin.initDb(preparedDatoms, "/tmp/import-db", schema)) {
    // Bulk-loaded database is ready to query.
}

from datalevin import datom, init_db

schema = {
    ":user/id": {
        ":db/valueType": ":db.type/long",
        ":db/unique": ":db.unique/identity",
    },
    ":user/name": {":db/valueType": ":db.type/string"},
    ":order/id": {
        ":db/valueType": ":db.type/long",
        ":db/unique": ":db.unique/identity",
    },
    ":order/customer": {":db/valueType": ":db.type/ref"},
    ":order/total": {":db/valueType": ":db.type/double"},
}

USER_BASE = 1_000_000
ORDER_BASE = 2_000_000


def user_eid(source_id):
    return USER_BASE + int(source_id)


def order_eid(source_id):
    return ORDER_BASE + int(source_id)


users = [
    {"id": "1", "name": "Alice"},
    {"id": "2", "name": "Bob"},
]

orders = [
    {"id": "10", "customer_id": "1", "total": 42.5},
    {"id": "11", "customer_id": "2", "total": 19.0},
]


def user_datoms(user):
    eid = user_eid(user["id"])
    return [
        datom(eid, ":user/id", int(user["id"])),
        datom(eid, ":user/name", user["name"]),
    ]


def order_datoms(order):
    eid = order_eid(order["id"])
    return [
        datom(eid, ":order/id", int(order["id"])),
        datom(eid, ":order/customer", user_eid(order["customer_id"])),
        datom(eid, ":order/total", order["total"]),
    ]

prepared_datoms = [
    item
    for row in users
    for item in user_datoms(row)
] + [
    item
    for row in orders
    for item in order_datoms(row)
]

with init_db(prepared_datoms, dir="/tmp/import-db", schema=schema) as conn:
    # Bulk-loaded database is ready to query.
    pass

import { datom, initDb } from "datalevin-node";

const schema = {
  ":user/id": {
    ":db/valueType": ":db.type/long",
    ":db/unique": ":db.unique/identity"
  },
  ":user/name": { ":db/valueType": ":db.type/string" },
  ":order/id": {
    ":db/valueType": ":db.type/long",
    ":db/unique": ":db.unique/identity"
  },
  ":order/customer": { ":db/valueType": ":db.type/ref" },
  ":order/total": { ":db/valueType": ":db.type/double" }
};

const userBase = 1_000_000;
const orderBase = 2_000_000;

const userEid = sourceId => userBase + Number.parseInt(sourceId, 10);
const orderEid = sourceId => orderBase + Number.parseInt(sourceId, 10);

const users = [
  { id: "1", name: "Alice" },
  { id: "2", name: "Bob" }
];

const orders = [
  { id: "10", customerId: "1", total: 42.5 },
  { id: "11", customerId: "2", total: 19.0 }
];

function userDatoms(user) {
  const eid = userEid(user.id);
  return [
    datom(eid, ":user/id", Number.parseInt(user.id, 10)),
    datom(eid, ":user/name", user.name)
  ];
}

function orderDatoms(order) {
  const eid = orderEid(order.id);
  return [
    datom(eid, ":order/id", Number.parseInt(order.id, 10)),
    datom(eid, ":order/customer", userEid(order.customerId)),
    datom(eid, ":order/total", order.total)
  ];
}

const preparedDatoms = [
  ...users.flatMap(userDatoms),
  ...orders.flatMap(orderDatoms)
];

const conn = await initDb(preparedDatoms, {
  dir: "/tmp/import-db",
  schema
});

try {
  // Bulk-loaded database is ready to query.
} finally {
  await conn.close();
}

The same id functions are used on both sides of the relationship: :order/customer stores (user-eid customer-id), not the external customer id and not a lookup ref. The source ids are also stored as unique identity attributes (:user/id, :order/id) so later normal transactions can use lookup refs such as [:user/id 1]. Treat Datalevin's numeric entity ids as internal implementation ids even when you assign them during a bulk load. They are not a durable interchange format: another database loaded from the same source data may use different eids, while the unique source-id attributes stay stable.

If the source does not have dense integer ids, build an eid-by-key map first: collect stable keys such as [:user external-id] and [:order external-id], sort or otherwise deterministically order them, assign fresh positive integers, and use that map when emitting subject ids and reference values.

4.2 Streaming with `fill-db`

For a large import, it is common to stream one table at a time with fill-db. Use the same id functions for every chunk so references remain consistent:

(def db
  (-> (d/init-db (mapcat user-datoms users)
                 "/tmp/import-db"
                 schema
                 {:closed-schema? true})
      (d/fill-db (mapcat order-datoms orders))))

List<Object> userChunk = new ArrayList<>();
for (Map<String, ?> user : users) {
    userChunk.addAll(userDatoms(user));
}

List<Object> orderChunk = new ArrayList<>();
for (Map<String, ?> order : orders) {
    orderChunk.addAll(orderDatoms(order));
}

try (Connection conn = Datalevin.initDb(
         userChunk,
         "/tmp/import-db",
         schema,
         Map.of(":closed-schema?", true))) {
    conn.fillDb(orderChunk);
}

user_chunk = [
    item
    for row in users
    for item in user_datoms(row)
]

order_chunk = [
    item
    for row in orders
    for item in order_datoms(row)
]

with init_db(user_chunk,
             dir="/tmp/import-db",
             schema=schema,
             opts={":closed-schema?": True}) as conn:
    conn.fill_db(order_chunk)

const userChunk = users.flatMap(userDatoms);
const orderChunk = orders.flatMap(orderDatoms);

const conn = await initDb(userChunk, {
  dir: "/tmp/import-db",
  schema,
  opts: { ":closed-schema?": true }
});

try {
  await conn.fillDb(orderChunk);
} finally {
  await conn.close();
}

4.3 Bulk-Load Safety Checks

With fill-db, new subject ids must not collide with existing entities. The simplest plan is to use ids greater than the database's current maximum eid, or to reserve non-overlapping ranges up front. In Clojure, inspect this with (d/max-eid db); in Java, use conn.maxEid(). In Python and JavaScript, prefer deterministic reserved ranges in the import plan.

Prepared datoms must already be valid

Bulk-load helpers still expect values that match the declared schema. Ensure there are no duplicate entity-id ranges, no unresolved references, and every item uses the prepared datom shape accepted by the binding, such as d/datom, Datalevin.datom, datom(...), or compact [eid attr value] tuples/arrays.

5. Choosing an Ingestion Strategy

After the individual tools are clear, Figure 20.1 expands the decision preview from the chapter introduction into a complete flow.

Summary: The Ingestion Checklist

When you need to ingest data at scale, follow this checklist:

Use init-db/fill-db when prepared datoms are available: This is the highest-throughput path for static data sets, but it bypasses normal transaction processing and WAL.
Use non-durable LMDB flags only when weaker crash behavior is acceptable: :nometasync, :nosync, and :writemap/:mapasync can improve write speed, but they change durability (i.e., crash behavior).
Use ordinary transaction batches for interactive or modest write rates: Keep tempids, lookup refs, upserts, transaction functions, and integrity checks in the normal transaction path.
Use WAL mode when OLTP data must stay queryable: Choose the durability profile using Chapter 19's operational guidance.
Use async transactions for higher durable online throughput: d/transact-async provides adaptive batching. Use an empty blocking transaction as an explicit queue barrier, or wait on returned futures or promises before dependent work.
Combine async submission with manual batching: The effects of auto-batching and manual batching compound.
Prepare transaction batches before submitting them: Sort by entity id before transacting, and disable the Datalog index cache during large bulk transaction jobs when measured workloads justify it.

For the full durability discussion and maintenance implications, see Chapter 19. These patterns let you choose the right ingestion path without hiding the trade-offs around durability, transaction semantics, and operational recovery.

References

[1] Huahai Yang, "Competing for the JOB with a Triplestore," yyhh.org, 2024. URL: https://yyhh.org/blog/2024/09/competing-for-the-job-with-a-triplestore/.

[2] Viktor Leis, Andrey Gubichev, Atanas Mirchev, Peter Boncz, Alfons Kemper, and Thomas Neumann, "How Good Are Query Optimizers, Really?" Proceedings of the VLDB Endowment 9(3):204-215, 2015. URL: https://www.vldb.org/pvldb/vol9/p204-leis.pdf.

[3] Datalevin project, "Write Benchmark," benchmark implementation and results. URL: https://github.com/datalevin/datalevin/tree/master/benchmarks/write-bench.

User Examples

No examples for this chapter yet.