NEDB Docs / Running it / Durability

NEDB's durability model was not designed on a whiteboard — it was found by killing a real engine at every persistence boundary and by filling a real filesystem to zero free blocks. Three defect classes came out of that campaign, all fixed and regression-tested.

Flush contracts

db.try_flush_all()?;      // Result<()> — use this when the outcome matters
db.flush_all();           // still logs; for ticker / Drop, nowhere to propagate

A failed flush used to silently discard acknowledged writes: the id-index buffer cleared entries regardless of whether the disk write landed, so a flush that hit ENOSPC lost rows while verify() reported every object healthy — the objects were durable; the index entries that made them findable were gone. An entry now leaves the WAL only when its write actually landed. Flush errors are observable (try_flush_all, try_flush_manifest, IdIndex::try_flush_write_buf).

Embedded bindings flush on a cadence: NedbCore.open() (Node + Python) runs the 1-second manifest ticker exactly as nedbd does, so a SIGKILL / OOM / power cut loses at most one tick of acknowledged writes. NEDB_FLUSH_MS tunes or disables it. Durable stores also flush on Ctrl+C/SIGTERM — automatic in the bindings, Db::install_exit_flush(Arc<Db>) for standalone Rust binaries. nedb-inspector warns when a durable open lacks flush-on-exit wiring.

The v3 segment store

v2 stores every version as its own content-addressed file: trivially atomic, corruption-proof, but each write costs file-create + fsync + rename plus a directory B-tree update — on a busy disk that caps sustained writes around ~185/s, and the bottleneck is the number of files touched, not bytes.

v3 batches objects into append-only segment packs; a batch commits with a single fsync. Flush cost scales with bytes, not object count. Each segment carries a checksummed .idx sidecar so reopen rebuilds the in-memory index from the sidecar (missing or corrupt sidecar falls back to a full scan-and-heal — slower, never fatal). Enabling is non-destructive: old loose objects stay fully readable, only new writes go to segments.

nedbd-v2 --dag-v3 --data /var/lib/nedb     # or NEDB_DAG_V3=1
Measured on a real blockchain node (itcd, FlushStateToDisk on live chainstate): 2,549 coins / 366 kB went from minutes on v2 to 1.71 s on v3 — and the larger batch finishes faster than the smaller one, because cost is the per-batch fsync, not the coin count.

Repair

Every object carries its own coll, id and seq — the id index is fully derivable, nothing is invented:

nedb-cli repair ./data
# repaired: 203 id-index entr(ies) rebuilt, 203 node(s) verified, flushed

The self-healing pass on open repairs structural gaps only; content tampering is never masked — a tampered log verifies false, on purpose.

Known sharp edges (documented, not hidden)

  • since()'s cursor is exclusive and seqs start at 0, so the very first write (seq 0) is unreachable through any cursor value — ten writes drain as nine records. Changing the convention would break existing consumers; gate on ScanStatus.seq_index_ready, not scan_complete.
  • compact() prunes superseded versions and tombstones — AS OF can no longer reach them. Nothing invokes it automatically: not on the HTTP surface, not in the CLI, not on a timer. A compacted store answers "not available at that sequence", never a stale value.
  • NEDB_FAST_FSYNC trades F_FULLFSYNC for plain fsync(2) on macOS (faster, weaker durability guarantee; no-op elsewhere).