Show HN: SNKV – KV store on SQLite's B-tree with 11x less memory than RocksDB

3 points

4 months ago

While working with RocksDB, I noticed it consumed significantly more memory than expected for small workloads. On a 1M record benchmark, it peaked at ~121 MB RSS. That got me thinking: what if I skip the query layer entirely and use SQLite purely as a storage engine? So I started digging into SQLite internals — specifically the B-tree, pager, and OS layers — to understand how data is actually stored and managed under the hood. I documented what I learned along the way:

B-tree operations: https://github.com/hash-anu/snkv/blob/master/internal/BTREE_... Pager operations: https://github.com/hash-anu/snkv/blob/master/internal/PAGER_... OS layer operations: https://github.com/hash-anu/snkv/blob/master/internal/OS_LAY...

Once I understood the core pieces, I built a simple key-value store directly on top of SQLite's B-tree layer — reusing its storage management, paging, and file I/O without touching the SQL engine at all.

I then added column family support, WAL and journal modes, and transactions. For easier integration, I generated a single amalgamated header (similar to SQLite's approach): https://github.com/hash-anu/snkv

Benchmarks (1M records, sync-on-commit enabled for both):

                    RocksDB     SNKV

Sequential writes: 237K/s 181K/s Random reads: 95K/s 154K/s Sequential scan: 1.78M/s 5.95M/s Mixed workload: 51K/s 97K/s Peak memory (RSS): 121 MB 10.8 MB (~11x less)

RocksDB's LSM-tree design gives it a clear write advantage, and with tuning (bloom filters, larger caches) its read numbers would improve too. The memory gap is structural though — RocksDB carries significantly more overhead even when constrained to a small footprint configuration.

The full KV layer documentation is here: https://github.com/hash-anu/snkv/blob/master/internal/KVSTOR...

Curious if others have hit the same memory wall with RocksDB on small workloads, or taken a different approach to this tradeoff.