"Continuous-eval" came from our efforts to measure, validate and improve the reliability of a finance AI copilot we were developing for banks.

End-to-end evaluation was not enough for us. We wanted to have granular evaluations that help pinpoint the bottlenecks and identify what / how to improve.

We’ve since developed more metrics and made the framework more flexible so it can evaluate components like agent tool use, code change, retrieval steps, etc.

Let us know what you think of our approach to GenAI App evaluation.