Architecture
One Go process, ndb. It listens on a unix socket (and optional TCP), answers HTTP/JSON, and owns one data directory.
- Direct mode: your app appends lines to its own
.ndjsonfile. The tailer picks them up (fsnotify plus a 5 s poll). ndb never writes these files. - Engine mode: an SDK sends
insert,updateordeleteover the unix socket or TCP. The writer group-commits them to the WAL withfsyncbefore it answers. - Both feed the in-memory table and its indexes. The sealer moves old rows into Parquet segments by size or age.
find,aggregateandwatchread memory and segments over HTTP. The newest version of each id wins.
Guarantees
- One daemon owns a data directory. A second process gets a lock error, so two daemons never corrupt the same files.
- Your NDJSON is the truth (the WAL in engine mode, your own files in direct mode). Segments, indexes and checkpoints are derived: delete them and ndb builds them again.
- A query sees the data as of its start. Writes that happen while it runs do not change its answer.
- A write in engine mode is answered only after it is on disk (
durability: strict), and a crash never leaves half a write. - The SDKs and
curlgive the same answer, because the daemon owns the meaning of every operator. /healthanswers503 recoveringuntil the data directory is open, so a load balancer waits for you.- Version 1 is one node: no replication, no clustering, no joins, no SQL.