We have been building SecureAI Tools -- an open-source application layer for ChatGPT and ChatPDF-like AI tools.

It works with locally running LLMs as well as with OpenAI-compatible APIs. For local LLMs, it supports Ollama which supports all the gguf/ggml models.

Currently, it has two features: Chat-with-LLM, and Chat-with-PDFs. It is optimized for self-hosting use cases and comes with basic user management features.

Here are some quick demos:

  * Chat with documents using OpenAI's GPT3.5 model: https://www.youtube.com/watch?v=Br2D3G9O47s
  * Chat with documents using a locally running Mistral model (M2 MacBook): https://www.youtube.com/watch?v=UvRHL6f_w74