Publishing LLM benchmarks on the Internet can cause contamination, since the benchmark can leak into training data or be used to tune models. A common mitigation is to keep the benchmark private and host a leaderboard, but that does not fully solve the problem. Participants can still overfit to a hidden test set through repeated queries. We propose CapBencher, a simple protocol that intentionally caps the best possible accuracy on a public or private benchmark. If a model scores above the cap, it is a strong black-box warning signal of leakage or gaming.
Show HN: Give your LLM benchmark a built-in alarm for leakage and gaming | Heykuki News