Every lab quotes the same benchmarks. How many can you train on?

A model release is a wall of percentages. What the table never tells you is how each number was scored — and a fully open test set is, by definition, a trainable one.

So we took the benchmarks the frontier labs actually cite and tagged each by access model and contamination risk. 47 benchmarks; 33 are fully open, only 13 are held-out, private, or live.

Five ways a benchmark guards its answers.

Openness is a spectrum. At one end the whole test set sits on Hugging Face; at the other the questions never leave the maintainer’s server. Each step up makes the score harder to game and harder to fake.

Copper marks the contamination-resistant tiers — the ones whose numbers age well.

Every benchmark, by category and access.

Grouped by what they measure, sorted within each group from most open to most guarded. The contamination column is our read on how likely the test set has already been seen.

Red = high contamination risk · copper = low. “Sat.” flags a near-solved benchmark.

An open test set stops measuring the moment it’s famous.

Once a benchmark is public and widely cited, its questions end up in the next pretraining crawl. Scores climb whether or not the underlying capability moved — the number measures memorization as much as skill. That’s why 26% of the canonical suite carries a high contamination risk, and why the held-out and live tiers are the ones worth betting a decision on.

The fix isn’t a new leaderboard — it’s a hold-out set the model has never seen, with a reward you can verify in code. That is exactly what we build.

Need a number you can actually trust?

If the public benchmarks for your capability are saturated or contaminated, the leaderboard can’t tell your models apart. We build private, contamination-resistant, verifiable-reward evals on a hold-out set — designed to discriminate where the open ones no longer do.