This leads me to believe there is growing yet largely unnoticed issue is taking root in recent AI research. I've termed this the "AI Reproducibility Crisis".

The principle is simple, if accessible private models are silently changing in time, then previous results cannot be replicated.

*Key Issues*:

  1. Users have reported significant performance shifts post the May release.

  2. Beyond community discussions, academic studies are showing differences in performance across time http://arxiv.org/abs/2307.09009.

  3. It appears difficult to replicate previous benchmark evals, see our effort here - https://github.com/emrgnt-cmplxty/zero-shot-replication.

  4. The centralized approach of major providers amplifies these concerns, underscoring the essential need for research autonomy.

  - Lean towards open-source foundational models for transparency.

  - Clearly annotate the date of model access when relying on private providers. Push these providers to provide model / inference specifiers to delineate any changes on their end.

  - Advocate for continuous third-party benchmarking of LLM providers to monitor model changes. This is an initiative we're currently starting w/ some help from a great academic group.