Benchmarks Are Broken. Here's Who's Fixing Them.

AI benchmarks are marketing. The UK government just started treating them like science.

The UK AI Security Institute has partnered with the EvalEval Coalition to publish evaluation results using a shared infrastructure — a reporting schema called Every Eval Ever and an open platform called Evaluation Cards. The goal is blunt: make AI benchmark results reproducible and verifiable, rather than the current system where labs self-report scores in incompatible formats with enough methodological ambiguity to drive a truck through.

This matters more than it sounds. Right now, when Anthropic says Claude Opus 5.5 scores X on some benchmark, you’re trusting Anthropic’s implementation of that benchmark, Anthropic’s sampling strategy, Anthropic’s prompt format, and Anthropic’s decision about which results to publish. OpenAI does the same. So does Google. The benchmarks aren’t standardized. The evaluation environments aren’t standardized. Running the evals again may be prohibitively expensive. What gets released to the public is a number stripped of every piece of context required to interpret it. That’s not science — it’s a press release formatted to look like data.

The EvalEval-AISI collaboration is trying to fix this by creating a common structure for reporting evaluation results alongside the information needed to reproduce them. AISI has previously developed OptStop, a tool for more efficient evaluations, and HiBayES, a framework for more statistically rigorous analysis. This partnership puts that infrastructure into public practice rather than leaving it as internal methodology. The EvalEval Coalition built the Every Eval Ever schema specifically to address the reproducibility gap — what was the prompt format, what was the sampling temperature, what was the pass/fail criterion, what was the model’s system prompt? The Evaluation Cards platform makes that context public and searchable.

Everyone says the problem with AI safety benchmarks is that models are being trained on the test data — that labs optimize for the evals rather than the underlying capability. That’s real. But the deeper problem is that we can’t even verify the claim, because we don’t have reproducible results. You can’t catch goodharting if you can’t run the eval yourself and get the same number. Right now, most people can’t.

The power structure here is worth naming. Labs control their own benchmark results. They choose what to submit to leaderboards, how to format the prompts, and whether to include error bars. An external evaluator can run the same benchmark and get a meaningfully different score — and there’s no shared standard for which result is authoritative. This isn’t hypothetical: the EvalEval post explicitly notes that “results are reported across many formats, platforms, and outlets, often without enough information to reproduce them.” That’s a diplomatic way of saying the field is epistemically broken.

The mutual insurer analogy from a recent Lawfare piece on AI governance is useful here. Markets figured out long ago that self-certification is insufficient for high-stakes systems. You don’t let banks audit their own capital ratios. You don’t let pharmaceutical companies run their own approval trials without regulatory oversight of the methodology. AI is being evaluated almost entirely by the entities with the strongest financial interest in the outcome. AISI stepping in to standardize the methodology — and EvalEval providing the public infrastructure — is the first real crack in that arrangement.

The AISI collaboration started at a joint workshop alongside NeurIPS 2025. The fact that it’s taken until September 2026 to produce a shared reporting standard tells you something about the pace of governance in this space. The models are moving faster than the epistemics. That gap is where the risk lives.

The labs will resist this, quietly. Not with press releases opposing reproducibility — no one opposes reproducibility out loud — but with the standard friction of institutions that benefit from opacity: incompatible data formats, proprietary model interfaces that prevent external replication, and the argument that their internal methodology is rigorous enough to not require external verification. Watch for those. They’re the tells.

Independent evaluation infrastructure is the unsexy intervention that makes every other AI safety conversation possible. You cannot govern what you cannot measure. And right now, the measurement is owned by the thing being governed.