Evolutionary, JSON data store (keeping the full revision history)

2 points

3 years ago

I'm working on an evolutionary data store in my spare time[1]. It is based on the idea to get rid of the need for a second trx log (the WAL) by using a persistent tree of tries (preserving the previous revision through copy on write and path copying to the root) indexes as the log itself with only a single permitted read/write txn concurrently and in parallel to N read-only txns, which are bound to specific revisions during the start. The idea is, that the system atomically swaps the tree root to the new version (replicated). If something fails the log can simply be truncated to the former tree root.

A speciality is, that the secondary indexes are also naturally versioned and stored in the leaf nodes/pages of keyed tries.

Thus, the system has many similarities to Git (structural sharing of unchanged nodes) and ZFS snapshots and checksums stored in parent pages[2]. You can of course simply execute time travel queries on the whole revision history, add commit comments and the author...

The system not only copies full data pages, but it applies a sliding snapshot versioning algorithm to keep storage space to a minimum.

It's best suited for fast flash drives with fast random reads and sequential writes. Data is never overwritten, thus audit trails are given for free.

The system stores find granular JSON nodes, thus the structure and size of an object has almost no limits. A path summary is built, which is an unordered set of all paths to leaf nodes in the tree and enables various optimizations. Furthermore a rolling hash is optionally built, whereas during inserts all ancestor node hashes are adapted.

Furthermore it optionally keeps track of update operations and the ctx nodes involved during txn commits. Thus, you can easily get the changes between revisions, you can check the full history of nodes, as well as navigate in time to the first revision, the last revision, the next and previous revision of a node...

You can also open a revision at a specific system time revert to a revision and commit a new version while preserving all revisions in-between.

As said one feature is, that the objects can be arbitrarily nested, thus almost no limits in the number and updates are cheap.

A dated Jupyter notebook with some examples can be found in [3] and overall documentation in [4].

The query engine[5] Brackit is retargetable (a couple of interfaces and rewrite rules have to be implemented for DB systems) and especially finds implicit joins and applies known algorithms from the relational DB systems world to optimize joins and aggregate functions due to set-oriented processing of the operators.[6]

I've given an interview in [7], but I'm usually very nervous, so don't judge too harshly.

Give it a try and happy coding!

Kind regards

Johannes

[1] https://sirix.io | https://github.com/sirixdb/sirix

[2] https://sirix.io/docs/concepts.html

[3] https://colab.research.google.com/drive/1NNn1nwSbK6hAekzo1YbED52RI3NMqqbG#scrollTo=CBWQIvc0Ov3P

[4] https://sirix.io/docs/

[5] http://brackit.io

[6] https://colab.research.google.com/drive/19eC-UfJVm_gCjY--koOWN50sgiFa5hSC

[7] https://youtu.be/Ee-5ruydgqo?si=Ift73d49w84RJWb2