Show HN: New AI Dataset Based on LibGen and Sci-Hub

4 points

3 years ago

We recently began extracting the text layers of scholarly publications and books to include in our database. This encompasses sources such as scimag, libgen, and the latest zlib leaks. Our project, named the Standard Template Construct, also features a distributed search engine and incorporates various AI routines to handle the text corpus.

Today we have releases our first dataset, STC230908. This dataset contains approximately 75,000 book texts, 1.3 million scholarly paper texts, and 24 million abstracts, including the years from 2021 to 2023.

We're currently in the process of preparing the next version of the dataset, which will include an additional 300,000 books.

How to Access

Short Instructions:

Install IPFS and launch it. pip3 install stc-geck && geck - documents More details: the dataset is released in IPFS and replicated to multiple nodes. It is in format of database for the search engine that we use in STC. GECK is the library that embeds this search engine and allows to stream all contained data in easy way.

Even more detailed Instructions: https://github.com/nexus-stc/stc/tree/master/geck