I lead product for Vectara (https://vectara.com) and we recently worked with OpenSource connections to both evaluate our new home-grown embedding model (Boomerang) as well as to help users start more quantitatively evaluating these systems on their own data/with their own queries.

OSC maintains a fantastic open source tool, Quepid, and we worked with them to integrate Vectara (and to use it to quantitatively evaluate Boomerang). We're hoping this allows more vector/hybrid players to be more transparent about the quality of their systems and any models they use instead of everyone relying on and gaming a benchmark like BIER.

More details on OSC's eval can be found at https://opensourceconnections.com/blog/2023/10/11/learning-t...