Benchmark saturation: when every model scores 100%

The signs of a saturated set

• Perfect cells. All 11 configurations in our effort ladder passed 16/16 each (95% interval 81% to 100%). In our consistency study, 7 of 9 model-and-prompt cells passed 10/10 each (72% to 100%). Haiku 4.5's other 2 cells scored 0/10 (0% to 28%) and 1/10 (2% to 40%). Their intervals do not overlap the perfect cells on the same prompts.
• Overlapping intervals. Even perfect cells have wide 95% Wilson intervals. They do not establish equal ability.
• Only format misses. Our five-task study had 127/130 passes (98%; 95% interval 93.4% to 99.2%). All 3 non-passes came from one configuration on one task. Right answers had extra working lines, which the exact-text validator rejects.

Our CLI latency study also hit a ceiling: 194/194 evaluated runs passed (95% interval 98.1% to 100%). It excludes 18 runs and 18 diagnostic receipts. Pass rate cannot separate these runs.

Our own sets hit the ceiling

On five short tasks, 8 of 9 configurations passed every call: seven scored 15/15 (95% interval 80% to 100%); one scored 10/10 (72% to 100%).

The hard head-to-head followed with eight harder tasks. Still, 6 of 7 configurations passed every call: four scored 24/24 (95% interval 86% to 100%); two scored 16/16 (81% to 100%).

Haiku 4.5 scored 11/24 strictly (46%; 28% to 65%). Its interval overlaps none of the six. The set separates Haiku from them, but cannot separate those six.

The effort ladder reused those 8 tasks across 11 configurations: Sonnet 5.5, Opus 5.5 and GPT-6.1 Sol at different efforts. Six were new; five reused hard-set cells. All scored 16/16 strictly (81% to 100% each). Pooled: 176/176 (95% interval 97.9% to 100%), with 0 format misses and 0 wrong answers. This combines configurations; it is not any one model's interval.

What a ceiling hides

• Uncertainty. A 16/16 interval reaches down to 81%. Calculation: about 19 percentage points below the observed perfect score. This describes one rate, not the difference between models. “No difference found” does not mean “no difference”. More calls help slowly: 24/24 reaches down to 86%.
• Weak spots. Haiku's easy-set 15/15 (95% interval 80% to 100%) overlapped every configuration. The hard set exposed a gap: 11/24 strictly (46%; 28% to 65%). Lenient grading accepts its 5 format misses: 16/24 (47% to 82%).
• Hidden splits. In our memory study, Sonnet 5.5 scored 57/60 or 60/60 on rules the code shows, across conditions. Their 95% intervals are 86% to 98% and 94% to 100%. Team-only checks differed: 6/15 without memory (40%; 20% to 64%), versus 15/15 with an 11-line curated file (80% to 100%). Those intervals do not overlap. Pooling hides the split.
• Selection. In SWE-bench Verified, 14 instances fall in “every panel model solved it”. Each of the 11 panel models scores 14/14 by definition (descriptive 95% Wilson interval 78% to 100%). Bands come from the panel's results. Selection guarantees perfection here; it does not estimate performance on new tasks.

What still differed between configurations

Time, tokens and calculated list-price cost differed at the ceiling:

• Time. Effort-ladder Sonnet 5.5 medians were 5.8 s at low effort and 8.8 s at high (n = 16 each). Single-call ranges: 2.8 to 20 s and 2.9 to 35.8 s. These overlap; medians describe this run, not a ranking. Ranges are not confidence intervals.
• Tokens. The same cells' output medians were 667 and 1,192 tokens (n = 16 each). Single-call ranges overlapped: 176 to 2,263 and 220 to 4,187 tokens.
• Cost. Calculation: all 16 calls' reported tokens × list price, divided by 16 strict passes. Cache reads and writes use separate rates. Low effort cost $0.0122 per pass; high cost $0.0167. Calculated per-call ranges: $0.0051 to $0.0261 and $0.0056 to $0.0453. Subscription calls make this a calculation, not a bill.
• Hard set. The six perfect configurations had different latency and token medians, and calculated costs. Their per-call time ranges overlap. Sonnet's median was 7.7 s (range 2.3 to 34.8 s; n = 24). GPT-6.1 Sol at high effort: 18.1 s (range 11.7 to 92.2 s; n = 16). These are call ranges, not confidence intervals.

Tools were off, on one host. Each CLI adds its prompt and start-up time. Hard-set routes ran on different days; reused ladder cells ran in a different hour. These compare model-and-CLI configurations.

Repeated calls on fixed tasks, and checks within one memory session, are not independent samples of new tasks. Wilson intervals describe these fixed sets.

What to do when a set saturates

1. Pilot harder tasks; keep ones models pass only sometimes. This is design advice; our dataset does not test it.
2. Show per-task results beside pooled rates.
3. Separate format misses from wrong answers. Haiku's 13 hard-set non-passes: 5 format misses, 8 wrong answers.
4. Report time, tokens and cost beside pass rate. Give time medians and ranges.
5. Show n and the interval beside every 100%. Say “ceiling” when a set hits one.

Frequently asked questions

What does it mean when a benchmark is saturated?

Most models score at or near the maximum, so scores stop separating them. The effort ladder above shows this across 11 configurations. Its pooled interval describes all configurations together.

Does 100% mean two models are equal?

No. Perfect observed scores retain uncertainty. The calculation of about 19 points above concerns one rate, not the difference between cells. Equal scores show both clear this bar.

How do you make a benchmark harder?

Pilot tasks that models sometimes fail. Our eight-task follow-up separated Haiku from six perfect configurations, but could not separate those six. We did not test the pilot step.

What still separates models at the ceiling?

Time, tokens and calculated list-price cost can differ. The Sonnet and GPT-6.1 Sol time ranges above overlap. They are not confidence intervals; the comparison remains unclear, not a ranking.