Most people building with LLMs today don’t have quantified evaluation and usage metrics on the interactions between users and their product. The only solution is to read every message (or a sample) to get a sense of what is going on. You can't improve your product without understanding who your users are and how they are using it. Nobody would launch a website without standard analytics today; the same principle should apply to LLM products.

We made phospho to analyze the large amounts of text from user inputs and LLM app outputs, and give you quantified and actionable insights.

You first log messages and set up semantic events. Eg: “user is talking about sports”, “assistant didn’t quote the source”. We then run asynchronous jobs to detect if events are present in the text or not. To do so, we use GPT-4 for the first few events, and then downsize to smaller fine-tuned models (cheaper & faster). It works with any LLM provider (OpenAI, Mistral, Ollama…). No proxy, no monkey patch, and no OpenAI key needed.

You can link phospho to your users’ feedback, and even use the platform to annotate some messages yourself. This helps you design step by step a custom evaluation pipeline that runs automatically, fits your needs, and enables you to iterate.

Results are available in dashboards, as dataframes, or via API. You can also directly leverage the events in your app to trigger actions in real-time with the API or via webhooks.

Deploy everything with Docker, or use the hosted cloud version. We have Python/Javascript SDK and an API. License is Apache 2.0.

Give it a spin and see where we’re at: https://github.com/phospho-app/phospho We’re interested in both feature requests and roasts. Let us know what you think!