Most AI model comparisons are ties: 44 of 1,196 rows show a clear gap

TL;DR

• Our comparison pages hold 187 pairs and 1,196 rows (dataset of 2026-10-06). Only 44 rows show a gap: the intervals or run ranges do not overlap. In the title, "ties" means "no side is ahead". By strict label, 471 rows are ties and 681 are unclear.
• 621 rows are provider list prices. A list price has no interval, so no price row can name a side. Only 269 rows meet the evidence rules for naming a side. 44 of those 269 show a gap (16%, a calculation).
• Speed and route decide most gaps. 29 of the 44 are times. The other 15 are outcome rates, and each involves Claude Haiku 4.5.
• No row separates Sonnet 5.5, Opus 5.5, Fable 5.1 and GPT-6.1 Sol on quality. All 28 outcome-rate rows among them are ties. All 130 effort rows are ties or unclear.
• Are LLM benchmark differences significant? These counts use overlap rules, not significance tests. Check n (the number of runs), intervals and any paired test before you believe "X beats Y".

The count: 44 of 1,196

We read the winner field of every comparison row. The compiler sets it with fixed rules. We judged no row again.

The 1,152 rows without a gap break down like this:

• 621 price rows. 316 are unclear. 305 are ties, because both sides list the same price.
• 153 ties, where the 95% intervals overlap.
• 76 unclear, where run ranges overlap.
• 161 unclear, with no interval or range (143) or too few runs (18).
• 141 rows with no better direction, such as a token count: 128 unclear and 13 ties.

Across all rows, 269 rows meet the evidence rules for naming a side. Each has a better direction and a 95% interval (168), a run range (91) or a p50 to p95 band (10). The 44 gaps sit in 15 of the 187 pairs.

What "a gap" means here

The compiler names a side only when a metric has a better direction (a higher pass rate, a lower time) and:

1. Outcome rates: the 95% Wilson intervals do not overlap. If they overlap, the row is a tie. Higher pass rates are better; lower failure rates are better.
2. Times: the fastest-to-slowest ranges do not overlap, with 5 or more runs per side.
3. Routing times: the p50 to p95 bands do not overlap, with 5 or more runs per side.

The run-count rule stops 18 rows. A scheduler repair took a median 15.0 s (13.9 to 15.9 s) with Claude Code. It took 61.2 s (59.9 to 69.5 s) with the Codex CLI. The ranges do not overlap, but each side had only 3 runs. These use Sonnet 5.5 and GPT-6.1 Sol at medium effort, so both model and route differ.

A range is not a confidence interval. A tie means "our overlap rule does not separate them", not "equal". Overlapping individual intervals can still occur with a significant paired test.

What decides: speed and route

Time rows meet our gap rule more often than rate rows. Of 90 eligible time rows with a range or band, 29 show a gap (32%, a calculation). Of 168 rate rows with an interval, 15 do (9%, a calculation). None of the 11 cost rows with eligible ranges shows a gap; these costs are calculations. The 44 gaps are 10 CLI times, 10 routing times, 7 repeated-prompt times, 2 coding-session times and 15 Haiku outcome rates.

CLI start-up. On a one-word answer, Claude Code with Haiku 4.5 reached first model output in a median 1,461 ms (1,206 to 2,308 ms). The Codex CLI, on its default model, took 5,059 ms (4,391 to 5,478 ms). Each side had 5 runs. The models differ, so this row does not separate the CLI from the model.

CLI vs API. GPT-6.1 Sol at high effort answered one line in a median 4.19 s (3.81 to 4.69 s) through the Codex CLI. Through the OpenAI API it took 1.52 s (1.35 to 2.23 s). Each side had 5 runs. All 8 CLI-vs-API gap rows name the API side as faster.

Routing decisions. A rule-based policy decided in a median 1.42 µs (p95 2.33 µs, 20,000 decisions). Claude Sonnet 5.5 at low effort took 2,597 ms (p95 4,298 ms). Haiku 4.5 with thinking on took 12,543 ms (p95 34,481 ms). Each Claude router made 82 calls through Claude Code.

These are wall times, not model compute times. The policy is an in-process microbenchmark. Jev’s rows use a direct HTTPS route, so routes differ. In all 10 routing rows, the slower median sits above the faster side's p95.

Coding sessions. On 6 hidden-test tasks, Sonnet in Claude Code took a median 23.1 s (18.7 to 44.5 s). Sol in Codex took 113.4 s (78.5 to 221.9 s); each side had 12 sessions. Codex followed the tester’s standing instructions and did extra work, so this compares configured systems.

Repeated prompts. On a code-fix prompt run 10 times, Claude Code with Sonnet 5.5 took a median 2.67 s (2.32 to 4.34 s). The Codex CLI with GPT-6.1 Sol at medium effort took 11.3 s (9.08 to 14.85 s).

What decides on quality: Haiku on these tasks

Each of the 15 outcome-rate rows with a gap involves Claude Haiku 4.5. This does not rank Haiku on all work:

• Hard set (7 rows). Haiku passed 11/24 strictly (95% interval 28% to 65%). Sonnet 5.5, Opus 5.5 and Fable 5.1 each passed 24/24 (86% to 100%). GPT-6.1 Sol passed 16/16 (81% to 100%). Haiku passed 16/24 under lenient grading (47% to 82%); three rows count this too. There were 30 blocked Codex launch receipts; the pass-rate cells exclude them.
• Repeated prompts (4 rows). Haiku passed 0/10 on an exact-number prompt (0% to 28%) and 1/10 on a JSON prompt (2% to 40%). Sonnet 5.5 and GPT-6.1 Sol each passed 10/10 on both (72% to 100%). Haiku’s JSON answers were correct in 10/10 calls (95% interval 72% to 100%) but had 9 format misses.
• Agent memory (4 rows). With a 211-line handbook, Haiku passed 3/10 sessions in full (11% to 60%) and Sonnet 15/15 (80% to 100%). All four rows name Sonnet. The fourth is below.

One row counts a bad outcome. "A stale README command: who still ran it?" counts sessions that ran a test command that fails on Node 25. With 60-line raw notes as memory, Haiku ran it in 10/10 sessions (72% to 100%) and Sonnet in 0/15 (0% to 20%). This chart sets lower as better. The compiler correctly names Sonnet for avoiding the stale command.

What never decides

• Quality at the top. All 28 outcome-rate rows among Sonnet 5.5, Opus 5.5, Fable 5.1 and GPT-6.1 Sol (Codex CLI) are ties. Their only gaps are 3 time rows. This group includes the interim SWE-bench probe, where model and platform build differ together.
• Effort. All 11 effort-ladder configurations passed 16/16 (81% to 100% each). The 16 effort comparisons hold 130 rows: 31 ties and 99 unclear. Five configurations reuse hard-set receipts; they were not rerun.
• SWE-bench Verified. Agent resolved 25/33 (59% to 87%). All 66 resolved-rate rows in the study are ties.

Three limits help explain these ties and unclear rows:

1. The ceiling. 16/16 gives a 95% Wilson interval of 81% to 100%, so two perfect cells overlap. These task sets hit a ceiling. The interval extends to about 81%; it does not measure quality on harder work.
2. Small n. On five short tasks, Sonnet 5.5 passed 12/15 (55% to 93%) and Opus 5.5 passed 15/15 (80% to 100%). The rates are 20 percentage points apart (a calculation), and still a tie under our overlap rule. The three Sonnet misses were format misses, not wrong final answers.
3. Overlapping ranges. On the same tasks, Sonnet took a median 2.31 s (2.17 to 7.73 s) and Opus 2.75 s (2.47 to 8.91 s). Each side had 15 calls; these are run ranges.

How to read any "X beats Y" claim

1. Find n, the interval and any paired test. Without a span, our comparison rule leaves the gap unclear.
2. Check the overlap and the ceiling. Overlap blocks a winner under our rule; it does not prove equality or replace a paired test. Near 100% on both sides, the task set may have hit a ceiling.
3. Rank inside one metric. Our leaderboard has no composite score.
4. Check the route and the date. At high effort, Sol took a median 1.52 s through the API (1.35 to 2.23 s) and 4.19 s through Codex (3.81 to 4.69 s). Each side had 5 runs. Counts change with new studies.

More: how to read AI benchmarks honestly and Wilson confidence intervals.

How we measured

A Python check read the winner field of every row in the public dataset of 2026-10-06. It grouped the 44 gaps by chart and row type. The counts and shares are calculations from the dataset. They describe comparison rows, not independent tests or a share of all AI benchmarks.

Sources: public dataset, short-task receipts, hard-task receipts, CLI/API runs, repeated-prompt receipts, routing summary, coding sessions and memory study.

Caveats

• The counts are a snapshot. New studies change them.
• Rows are not independent. The 7 hard-set rows reuse one Haiku result. The 8 CLI-vs-API rows reuse 4 selected sets of 5 runs. Repeated calls on the same tasks do not provide independent task samples.
• The rule is cautious on purpose. It is not a significance test. A paired test can detect a difference despite overlapping individual intervals.
• A third party reports the provider prices. We did not measure them.
• Our task sets are small: 8 hard tasks, 5 short tasks, 6 coding-agent tasks, 5 memory tasks and 33 SWE-bench Verified instances.
• Disclosure: I build Agent. These comparisons combine our own runs, public SWE-bench runs and reported provider prices.

What to read next

• Does reasoning effort buy quality?
• When tasks get hard
• Pairs: Haiku vs Sonnet, Sonnet vs Opus

Run the comparison on your own work

Agent keeps receipts for your own tasks: model, route, tokens, time, cost and validation result. Try Agent and see your numbers.