for the past weeks, we’ve been developing an open-source evaluation and testing framework for computer vision models. Today we’ve released the first alpha version and would love to get your feedback and support.

Github: https://github.com/moonwatcher-ai/moonwatcher

*What problems are we solving?*

- *Manual, error-prone evaluation:* Assessing model quality is still a manual and error-prone process. Of course, aggregation metrics exist, but they usually overlook that the model works differently on some parts of the data. - *Lack of a single source of truth:* Teams struggle to align on AI quality. There are multiple metrics and not all stakeholders understand their meaning and implication. Moreover, manual evaluations from different model versions are stored in Notion, Jira, Google Docs et al which makes it difficult to find reliable data about model quality. - *Testing for compliance:* The AI Act is coming into force in the next months. Becoming compliant requires teams to fully understand the capacities and limitations of their models and document them. One way of doing that is through testing. BY THE WAY: Some companies out there charge an between 100k-300k for a certification. We believe that there needs to be an open-source alternative that ensures a vibrant ecosystem which can release innovative products without paying a fortune.

*Features*

*Open-Source Package*

- *Automated Checks:* There is a set of automated checks that you can run. For now, we’ve started with image features such as brightness and saturation. In the future, we plan to develop more complex automation checks based on image content, bounding box size etc. - *Customizable Checks:* Of course you can write your own custom checks. - *Quick Demos:* We’ve set up demos that help you understand how it all works.

*Web App*

- *Visualize Results:* You can visualize the test results and browse relevant images to debug failure cases. In the future, we want to allow non-technical team members to use the app to create tests and align on model results. - *Share Insights:* Non-devs are used to using non-dev tools. We believe that it’s important to establish a common ground where engineers and non-technical stakeholders can communicate to foster a common understanding of model quality. - *Try the Demo:* Log in at [app.moonwatcher.ai/sign-in](http://app.moonwatcher.ai/sign-in) with: - Email: [[email protected]](mailto:[email protected]) - Password: Demo123!

Check out the repo for more details, and feel free to contribute or leave feedback: https://github.com/moonwatcher-ai/moonwatcher

Reach out at [[email protected]](mailto:[email protected]) for questions, support, or collaboration. Looking forward to your feedback and suggestions!