Show HN: Shared – Triptych for data exchange and persistence

14 points

4 years ago

Hi HN! I'm Alex, a tech enthusiast who likes to build stuff from scratch. Often my programs have data to store or a configuration file to read and/or edit. Therefore, I design relational databases [1] and write SQL queries to manage complex data with relationships, and for other needs, I used to do unmaintainable hacks (custom file formats) with plain old files. But that was before I discovered Pickle [2], the Python data serialization module. A simple way to store a representation of memory-living data collections on local disk.

Therefore, Pickle became my favorite way to store data, until one day I needed to exchange data with another program written in another language. Pickle is Python-specific by the way. Due to this embarrassing situation, I needed to find a way to "replace" Pickle. The new method should be language-agnostic.

CSV [3], INI file [4] and JSON [5] respectively entered the room.

As an open standard file format with a formal specification, JSON sounds very serious. The Internet speaks in JSON [6]. Therefore, I adopted JSON. So at this point, for my personal projects, I had SQLite for interaction with relational databases, files to store binary data, and JSON for everything else. But one thing is missing: a single tool to unify these APIs under the same umbrella, and so with a single installation I could do data persistence and exchange wherever I want.

Instead of one, for aesthetic reasons, I built two libraries: Litedao for SQL related stuff and Shared for data collections (dict [7], set, list) and binary data. Litedao was designed to use SQLite and Shared to use JSON and files. I had from my point of view a smooth programmer experience when it came to the data of my personal programs. But since I need SQL less than JSON, the Litedao library caught my attention less. Therefore, more bugs have been fixed in Shared and more functionalities have been added.

Due to this situation, I decided to integrate Litedao into Shared so that it could get more attention from me. It worked ! Therefore, Shared becomes a lightweight library for data exchange and persistence that smoothly handles collections (dict, list, set), binary data, and SQL queries. For aesthetic and functional reasons, Shared is a triptych whose classes share similar interfaces: Document, Dossier, and Database. The Document class allows individual access to files that are likely to be edited manually by a human, example: a configuration file. The Dossier class stores collections and binary data in a dossier without bothering the programmer about how the data is actually saved. The Database class allows interaction with SQLite databases.

Among other functionalities, there is a command-line interface for exchanging data between programs, Autosave mode, Readonly mode, and Temporary Data mode.

My project, still in Beta, is written in Python and I'm still working on it while using it intensively. Thanks for reading, let me know if you have any feedback and feel free to ask questions.

[1] https://www3.ntu.edu.sg/home/ehchua/programming/sql/Relation...

[2] https://docs.python.org/3/library/pickle.html

[3] https://en.wikipedia.org/wiki/Comma-separated_values

[4] https://en.wikipedia.org/wiki/INI_file

[5] https://en.wikipedia.org/wiki/JSON

[6] https://stackoverflow.blog/2022/06/02/a-beginners-guide-to-j...

[7] https://docs.python.org/3/tutorial/datastructures.html#dicti...