Show HN: RustyRAG lowest-latency open-source RAG on GitHub

1 point

3 months ago

I built an open-source RAG API in Rust to see how low I could push latency without a GPU.

RustyRAG v0.2 hits sub-200ms on localhost and sub-600ms from Azure North Central US to a browser in Brazil. 977 PDFs, 56K chunks in Milvus, 3 sources per response.

Key changes in v0.2: switched to Cerebras/Groq for LLM inference, replaced Cohere with Jina AI local embeddings (v5-text-nano-retrieval), and added optional contextual retrieval via LLM-generated chunk prefixes.