One principle, three generations: state is a pure function of a sealed log. Everything else — time travel, provenance, proofs — falls out of that sentence.
The pipeline
┌──────────────────────────────────────────────────────────┐
put/del → │ OpLog (BLAKE2b hash chain · per-client nonce · │ ← single source of truth
link │ idempotency keys · causal provenance fields) │
└───────────────┬──────────────────────────────────────────┘
deterministic fold │ (state = pure function of the log)
┌──────────────┬──────────┴──────┬───────────────┬────────────────┐
▼ ▼ ▼ ▼ ▼
MVCC store Relations Indexes CauseMap BlobStore
(time-travel) (graph+AS OF) eq/ord/search (reverse index) (Cascade CDC)
┌─────────────────────────────────┐
Thread-safe → │ Sequencer (group-commit) │ ← single writer, parallel readers
│ — one committer thread/db │
│ — batch fsync │
└─────────────────────────────────┘The provenance machinery is load-bearing, not metadata stapled on: CauseMap is a
reverse index over causal edges (that is why TRACE … REVERSE is as fast as forward),
and the Relations layer backs both TRAVERSE and the AS OF joins. The
daemon is a single-writer group-commit sequencer — parallel reads, batched durable writes, one
hash chain per database, zero write-write races.
Three storage generations
| Gen | Layout | Trade |
|---|---|---|
| v1 · AOF | append-only log, replay on open | simple, universal fallback; O(n) restart |
| v2 · DAG | content-addressed loose objects, one file per version, MANIFEST for O(1) warm start | trivially atomic writes; file-metadata churn at scale (~185 writes/s ceiling on busy disks) |
| v3 · segments | append-only packs, one fsync per batch, .idx sidecars, compaction | cost tracks bytes not object count; opt-in, dual-read compatible with v2 |
Same API across all three — a v1 store auto-migrates to the DAG on first --dag
startup, and v2 stores open in v3 mode without a rewrite. Cold starts accept connections
immediately (reads serve from the DAG; writes return 503 until the scan gate flips, progress
streams over /events); every restart after the first is O(1) from the MANIFEST.
Encryption
AES-256-GCM at rest with a TMK/DEK double-envelope: a 32-byte hex master key
(NEDB_TMK) wraps per-database data keys. Key handling fails closed — a
malformed provided key is an error, never a silent fall-through to plaintext. Because a node's
hash is a function of its ciphertext, state-root leaves commit to logical content rather than
object hashes (see Provenance).
Repo layout
python/nedb/ reference engine (pure Python — always-works baseline)
rust/
nedb-v2/ v2/v3 DAG engine (tokio + axum + BLAKE2b) — the core everything binds
nesql-cli/ the nesql CLI
nedb-core/ v1 Rust engine
nedb-py/ maturin PyO3 binding → PyPI native wheels
nedb-node/ napi-rs binding → npm native addons
nedb-wrap/ the wrap_* surface at native speed
vendor/postgresql/ PostgreSQL 17.4 grammar, vendored (gram.y, kwlist.h, …)
distributions/ crypto-database + aof-db submodules (tri-distribution)
bench/ benchmarks.py + RESULTS.md — the dated numbers
vectors/ state_root_v1.json — cross-engine test vectors
tests/ engine + concurrent + causal + bitemporal + deploy + perf suitestools/build_docs_site.py.