One of the biggest challenges I see clients face is understanding how their assistants are performing in production. Evals are great for catching regressions, but they can’t surface the blind spots in your AI’s behavior.

This gets even more challenging for conversational AI products that don’t have a single “correct” answer. Different users cohorts want different experiences. That makes measurement tricky.

Coming from a product analytics background, my default instinct is always: “instrument the product!” However, tracking generic events like user_sent_message doesn’t tell you much.

What you really want are insights like:

  - How frequently do users request to speak with a human when interacting with a customer support agent? 

  - Which user journeys trigger self-reflection during a session with an AI therapist?

  - What percentage of the time does an AI tutor's explanation leave the student confused?

  - Analyzes your conversation transcripts

  - Auto-generates a rich event schema

  - Tags each message with relevant events and event properties

  - Sends the events to your analytics tool (currently supports Amplitude and PostHog)