Hugh and Vince here.
LLMonitor helps you record, trace & search your LLM queries and chatbot conversations.
You can also capture user feedback on your frontend and correlate it with backend LLM queries then use that to fine-tune your own models.
The project started has an internal tool in our previous (failed) AI startup.
We’re aware the LLM observability space is very crowded. Apart from being open-source, we differentiate with:
- Model-agnostic and minimal lock-in (no MITM of requests).
- High focus on DX and dashboard clarity.
- Support for complex scenarios: e.g. a chatbot that invokes multiple agents & tools before giving an answer.
- Targeting early-stage startups + indie developers.
Two of our big focuses in the near future are:
- Tests (evaluations) written in either code or English (with GPT-4) to know when queries go wrong.
- OpenTelemetry support to expand traces to the rest of the stack.
(Both of those have turned out to be much more complicated than anticipated.)
One decision we made is to stick with Postgres for everything instead of going OLAP with something like Clickhouse. As we are writing & updating many rows concurrently for the traces, Postgres seems to be the best tool for that. It has been sufficient for our analytics and search needs at the moment. We hope that also makes it easier for people to self-host and contribute.
The entire codebase is licensed under Apache 2.0 and we’re fully bootstrapped. Being open-source has already begun to foster a community, e.g. with people building integrations for tooling we wouldn't have been able to support otherwise.
To be honest we're still quite embarrassed with the state of the product, but I guess we’d be doing something wrong if that wasn’t the case.
We’d love to hear your feedback - thank you!
Landing page: https://llmonitor.com