● v2.0.27 — The DAG Engine

NEDB

Content-addressed Merkle DAG for embedded data. Every document version is an immutable, BLAKE2b-verified object — with bi-temporal queries, causal write provenance, and a Rust v2 engine. Instant warm start via MANIFEST. No corruption possible.

View on GitHub NEDB Studio →

$ pip install nedb-engine $ npm install nedb-engine $ NEDBD_DAG=1 NEDB_TMK=<hex> nedbd --data ./data

dag_engine.py

from nedb import NEDB

# v2 DAG engine: content-addressed objects
db = NEDB("./data", dag=True)   # or NEDBD_DAG=1

# Every put becomes an immutable, BLAKE2b-verified
# object in the Merkle DAG. No AOF, no replay.
db.put("claims", "c1", {
    "fact": "Earth orbits Sun",
    "valid_from": "1543-01-01",
    "caused_by": "obs:heliocentric-model",
    "confidence": 0.9999,
})

# The Merkle head is a real BLAKE2b root over the
# full DAG of all versions ever written.
db.head()                  # "b2:7af3..." (DAG root)

# Bi-temporal queries unchanged
db.query(
    'FROM claims AS OF 200'
    ' VALID AS OF "2024-06-15"'
    ' WHERE confidence > 0.9'
)

# Cold start is instant — no log replay
assert db.verify()     # DAG integrity check

v2.0.27 · The DAG Engine

Content-addressed Merkle DAG — instant warm start, deferred cold scan, zero corruption

NEDB v2.0.27 ships the production DAG substrate: every document version is hashed with BLAKE2b and stored as an immutable, addressable object. On restart, MANIFEST restores seq and the Merkle head in O(1). On the first open of an existing dataset, the daemon accepts connections immediately and runs a deferred background cold scan with a live progress bar; reads work instantly while writes return 503 startup in progress until the scan completes. The Merkle head is updated incrementally on every write — never recomputed.

🌐

Content-addressed objects

Every put writes an immutable object keyed by its BLAKE2b digest. Identical values dedup automatically. Document history is a real DAG of versions, not a sequential log.

⚡

MANIFEST warm start

nedbd loads seq and the current Merkle head in O(1) from a small MANIFEST file. Subsequent restarts after the first open are instant — no scan, no replay, independent of dataset size.

⏳

Deferred cold scan

First open of an existing dataset accepts connections immediately. A background thread runs the integrity scan with a live progress bar (rate + ETA); reads serve instantly, writes return 503 startup in progress until the startup_ready gate flips.

🛡

No corruption possible

Objects are immutable and hash-addressed; a corrupted byte changes its hash and the DAG simply ignores it. There is no partially-written AOF tail to recover.

🧮

Rust v2 engine

Run with nedbd --dag or NEDBD_DAG=1 to launch the nedbd-v2 binary — tokio + axum with TCP_NODELAY, IdIndex sharded across 256 subdirectories for million-doc scale, and O(1) incremental Merkle head updates.

🔗

O(1) Merkle head

The BLAKE2b Merkle head is updated incrementally on every write — never recomputed. Every response carries the current root over the full DAG of every version ever written, anchorable on-chain, comparable across replicas.

🗑

Tombstone deletes

Deletes are tombstone objects in the DAG. History never disappears; AS OF still sees the pre-delete value, and the head hash still commits to every version ever written.

$ nedbd --dag --data ./data # launch v2 engine $ NEDBD_DAG=1 NEDB_TMK=<hex> nedbd --data ./data # env-driven, encrypted $ curl http://127.0.0.1:7070/events # SSE — real-time log stream

Production

In production at vision.interchained.org

NEDB v2.0.27 is serving the Vision DAG engine at vision.interchained.org: 1,310,703 sequences indexed, AES-256-GCM encrypted at rest, live at block height 620,989. Warm starts return in O(1); cold scans run deferred behind the live socket.

Operation (10k writes / 100k reads / 30k objects, Intel iMac)	Throughput	p50 / p99
Sequential writes	418/s	2.3 ms / 3.3 ms
Point-lookup reads	478/s	2.0 ms / 3.0 ms
ORDER BY queries	489/s	1.8 ms / 4.3 ms
Batch writes (500 ops/req)	1,104 ops/s	0.9 ms / 1.2 ms
Tamper-verify (30k objects)	~21k BLAKE2b/sec	1.38 s total

Why NEDB

The first embedded database that answers “why does the agent believe X?”

NEDB seals caused_by, evidence, and confidence directly into its Merkle DAG — so every fact carries a cryptographically-linked audit trail. Combine that with bi-temporal queries and a concurrent Rust core and you get something no other embedded database offers: four-dimensional auditability at embedded-database speed.

🕘

Bi-temporal Queries

VALID AS OF "date" for business time + AS OF seq for transaction time. Ask the four-dimensional question: “what did the system know at seq 200 about what was true on 2024-06-15?”

🔗

Causal Write Provenance

caused_by, evidence, confidence sealed in the DAG. TRACE caused_by walks backward (why?) and TRACE caused_by REVERSE walks forward (what did this cause?).

⚡

Concurrent Group-Commit

nedbd handles parallel reads and batched writes safely via a lock-free sequencer. ~15 K writes/s under load — without sacrificing durability or chain integrity.

🔄

SQL + Redis + MongoDB Adapters

All three wire protocols speak the same Rust engine. Migrate from any stack without changing your queries — and add RESP2 for redis-cli compatibility via NEDBD_RESP2_PORT=6380.

🔒

AES-256-GCM Encryption

Set NEDB_TMK=<32-byte-hex> and every object is encrypted at rest. The DAG remains verifiable; keys never touch storage.

🔧

Self-Healing DAG

Structural gaps are detected and auto-repaired on open. Combined with the Rust native core (napi-rs for Node, PyO3 for Python), the same compiled engine powers both runtimes.

NQL Grammar

One query language for all four dimensions

NQL composes cleanly — any clause is optional, order is consistent, and the same syntax works across the Python, Node, SQL, Redis, and MongoDB adapters. Identical grammar in the v1 AOF engine and the v2 DAG engine.

Clause	Description
FROM <coll>	Target collection
AS OF <seq>	Transaction-time travel — read DB state as it existed at sequence number
VALID AS OF "<date>"	Valid-time travel — filter records whose validity window includes the given date
WHERE <expr>	Filter with equality, comparison, boolean logic, and nested field access
SEARCH "text"	Full-text search via incremental inverted index
ORDER BY <field>	Ordered result set (ascending/descending)
TRAVERSE <rel>	Graph traversal over adjacency-list edges (time-travels with the rest)
TRACE caused_by	Backward causal trace — follows the provenance chain: why does the agent believe X?
TRACE caused_by REVERSE	Forward causal trace — what downstream facts did this event produce?
LIMIT n	Cap result count
GROUP BY <field> COUNT\|SUM\|AVG\|MIN\|MAX	Aggregations, evaluated after all filters

Release highlights

What's new

v2.0.27

Warm start + deferred cold scan + SSE log stream

MANIFEST loads seq and the Merkle head in O(1) on every restart after the first open. Deferred background cold scan accepts connections immediately while reads serve instantly; writes return 503 startup in progress until the startup_ready gate flips. New GET /events SSE endpoint for real-time log streaming. NEDBD_HOST now defaults to 127.0.0.1 (security fix). TCP_NODELAY on the axum listener eliminates the macOS loopback Nagle delay. IdIndex sharded into 256 subdirectories for 1M+ doc scale. Production: 1,310,703 sequences indexed in vision.interchained.org.

v2.0.4

The DAG Engine

Content-addressed Merkle DAG storage. Every document version is an immutable, BLAKE2b-verified object. nedbd --dag (or NEDBD_DAG=1) runs the Rust v2 engine. Instant cold start, no corruption possible, real Merkle head on every response.

v1.0.5

Bi-temporal stability & Studio GA

Production-hardened VALID AS OF indexing, improved self-healing on corrupt segments, and NEDB Studio goes generally available at studio.interchained.org — prompt-to-database GUI with live chain inspection.

v1.0.x

Rust native core (napi-rs + PyO3)

Same compiled Rust core surfaces to Node via napi-rs and to Python via maturin/PyO3. Both npm install nedb-engine and pip install nedb-engine ship native binaries with a pure fallback.

v0.9.x

RESP2 wire protocol + adapters

nedbd now speaks RESP2 — set NEDBD_RESP2_PORT=6380 and connect any redis-cli client. SQL and MongoDB adapters landed alongside, all sharing the same engine.

v0.8.x

Causal Write Provenance + bi-temporal

caused_by, evidence, and confidence are now first-class fields sealed in the hash chain. TRACE caused_by [REVERSE] introduced. Bi-temporal VALID AS OF clause added alongside existing AS OF seq.

v0.7.x

Concurrent group-commit sequencer + AES-256-GCM

nedbd gained a lock-free sequencer for parallel reads and batched writes (~15 K writes/s). At-rest encryption via NEDB_TMK and self-healing chain repair on open also shipped in this cycle.

Full history on GitHub Releases.

Run it as a server

nedbd

pip install nedb-engine or npm install nedb-engine ships the daemon. Pass --dag (or set NEDBD_DAG=1) to launch the v2 Rust engine binary nedbd-v2 with the content-addressed Merkle DAG. Connect clients over HTTP, RESP2, SQL, or MongoDB wire — whichever fits your stack. Watch writes happen live via the /events SSE stream. MANIFEST gives O(1) warm starts; the deferred cold scan keeps the socket open immediately.

shell

$ nedbd                              # v1 AOF engine
$ nedbd --dag                        # v2 DAG engine (nedbd-v2)
$ NEDBD_DAG=1 nedbd                  # env-driven equivalent
$ NEDBD_DAG=1 NEDB_TMK=<hex> \
    nedbd --data ./data               # DAG + AES-256-GCM
$ NEDBD_RESP2_PORT=6380 nedbd       # also speaks redis-cli

# Create a database and query — same API on v1 and v2
$ curl -X POST localhost:7070/v1/databases \
    -d '{"name":"agents"}'

$ curl -X POST localhost:7070/v1/databases/agents/query \
    -d '{"nql":"FROM beliefs AS OF 200 VALID AS OF \"2024-06-15\" WHERE confidence > 0.8 TRACE caused_by"}'

# Real-time log stream (SSE) — new in v2.0.27
$ curl http://127.0.0.1:7070/events
# event: write
# data: {"seq":42,"coll":"beliefs","head":"b2:7af3…"}

# env: NEDBD_DAG · NEDBD_HOST (127.0.0.1) · NEDBD_PORT
#      NEDBD_DATA · NEDBD_TOKEN · NEDB_TMK (AES-256-GCM key)

Install

Two lines to a database

$ pip install nedb-engine $ npm install nedb-engine $ NEDBD_DAG=1 NEDB_TMK=<hex> nedbd --data ./data

NEDB() is in-memory; NEDB("./path") is durable. nedbd runs it as a server; nedbd --dag launches the v2 DAG engine. The Rust core ships as a native binary with a pure fallback. Set NEDB_TMK for AES-256-GCM at-rest encryption. Read the format spec or explore your data visually in NEDB Studio.