Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head
On short tasks with strict validators, how do the Claude Code models and efforts compare with Codex on pass rate, speed and tokens?
Published · 9 charts · Download the data or a carousel
98%
The answer
127 of 130 calls passed (98%, 95% interval 93% to 99%), so on these short tasks pass rate barely separates the configurations (non-passes: Claude Sonnet 5.5 · Claude Code on Multi-step shift arithmetic ×3 ("Output does not exactly match the expected text")). Every non-pass ended on the expected answer (2292) but added working lines, which the exact-text validator rejects by design: a format miss, not a wrong answer. Speed separates them more: the fastest configuration was Claude Fable 5.1 · Claude Code at a median 1.9 s per call, the slowest GPT-6.1 Sol (low) · Codex CLI at 6.3 s. Claude Code medians ran from 1.9 s to 4.4 s. Codex CLI medians ran from 5.6 s to 6.3 s. Single calls vary a lot (see the ranges), so neighbouring configurations are not separated. Codex CLI sends a median 12,124 input tokens per call against 2,130 for Claude Code, mostly the CLI’s own context. At list price (a calculation; the calls ran on subscriptions), the cheapest passing answer came from Claude Sonnet 5.5 · Claude Code at $0.0062. On speed, cost per pass and pass rate together, the frontier is Claude Fable 5.1 · Claude Code, Claude Sonnet 5.5 · Claude Code, Claude Opus 5.5 (high) · Claude Code, Claude Opus 5.5 · Claude Code, Claude Opus 5.5 (low) · Claude Code; with 10 to 15 calls per configuration, small gaps inside it are within the spread of single calls. A harder follow-up with eight tasks and strict validators: /benchmarks/hard-model-head-to-head.
Where each call’s input tokens come from. Pick up to three configurations to compare.
Claude Opus 5.5 (low) · Claude Code2,081 tokens per call
1,463 + 618 = 2,081
GPT-6.1 Sol (medium) · Codex CLI12,123 tokens per call
5,180 + 6,943 = 12,123
Ribbon thickness is tokens per call, on one scale for every configuration. Only parts that sum to the total are drawn. First shown: the largest and the smallest total.
| Item | Cache read | Other input | n |
|---|---|---|---|
| Claude Fable 5.1 · Claude Code | 2,760 | 473 | 15 |
| Claude Sonnet 5.5 · Claude Code | 1,401 | 685 | 15 |
| Claude Opus 5.5 (high) · Claude Code | 1,463 | 619 | 15 |
| Claude Opus 5.5 · Claude Code | 1,401 | 680 | 15 |
| Claude Opus 5.5 (low) · Claude Code | 1,463 | 618 | 15 |
| Claude Haiku 4.5 · Claude Code | 0 | 3,790 | 15 |
| GPT-6.1 Sol (high) · Codex CLI | 6,716 | 5,406 | 15 |
| GPT-6.1 Sol (medium) · Codex CLI | 5,180 | 6,943 | 15 |
| GPT-6.1 Sol (low) · Codex CLI | 8,064 | 4,059 | 10 |
9 rows, 2 series: Cache read, Other input. Cache read: highest GPT-6.1 Sol (low) · Codex CLI 8,064 (n 10). Lowest Claude Haiku 4.5 · Claude Code 0 (n 15). Other input: highest GPT-6.1 Sol (medium) · Codex CLI 6,943 (n 15). Lowest Claude Fable 5.1 · Claude Code 473 (n 15).
Notesn 10–15 per row
Mean per call, split into prompt-cache reads and other input
The prompts are a few hundred tokens; most input is the CLI’s own system prompt and tool context. Input counts include cache reads, as the vendors report them.
Source: Provider head-to-head: Claude Code models vs Codex efforts
Live story
Drawn live in the page from the same data as the charts below. Play it, or download it as a video from the player.
Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head
127 of 130 calls passed, so speed and tokens separate the models: Fable 5.1 was fastest at 1.9 s median.
Transcript
- Head-to-head · 130 timed calls · 9 configurations. Haiku vs Sonnet vs Opus vs Fable vs Codex. Five short tasks with strict validators. Every call kept, nothing retried.
- 127 of 130 calls passed. Pass rate barely separates them; speed and tokens do. Calls that passed their validator: 98% (127/130) (n = 130, 95% CI 93–99%). Median input tokens per call: Codex CLI vs Claude Code: 12,124 vs 2,130 (n = 130). Cheapest passing answer (list-price calculation): Sonnet 5.5: $0.0062 (n = 15). Caveat: The tasks are short and easy; pass rate saturates. Latency and tokens carry the signal. A harder follow-up with eight tasks and strict validators: /benchmarks/hard-model-head-to-head.
- Fable 5.1 finishes first at 1.9 s. The Codex CLI needs 5.6–6.3 s. Chart: Median total time per call · real time (n = 10–15 each). Caveat: CLI timings include CLI start-up and the CLI’s own system prompt.
- What the CLI sends: a median 12,124 input tokens per call on the Codex CLI, 2,130 on Claude Code. Chart: Input tokens per call: what the CLI sends (n = 10–15 each). Caveat: The prompt cache stayed at the provider default, so cache counters differ by route and by call order.
- Speed vs cost per passing answer. Ringed: no other setup is faster, cheaper per pass and as accurate. Chart: Speed, cost and quality frontier (n = 10–15 each). Calculation, not a run. Caveat: Few repetitions per cell (2 or 3 per task). Medians with ranges, not intervals.
- Open benchmarks: intervals, sources and every failure kept.
Key numbers
9
Configurations compared
1.9s
Claude Fable 5.1 · Claude Code
Fastest configuration (median total time)
n = 15
6.3s
GPT-6.1 Sol (low) · Codex CLI
Slowest configuration (median total time)
n = 10
$0.0062
Claude Sonnet 5.5 · Claude Code
Lowest list-price cost per passing answer (calculation)
n = 15
12,124
Median input tokens per call, Codex CLI vs Claude Code
vs 2,130 · n = 130
The charts
Hover or focus a row for its exact value, interval and sample. Each chart has a Table view.
Every interval overlaps every other: this chart does not order these rows.
| Item | Pass rate | 95% interval | n |
|---|---|---|---|
| Claude Fable 5.1 · Claude Code | 100% | 80%–100% | 15 |
| Claude Sonnet 5.5 · Claude Code | 80% | 55%–93% | 15 |
| Claude Opus 5.5 (high) · Claude Code | 100% | 80%–100% | 15 |
| Claude Opus 5.5 · Claude Code | 100% | 80%–100% | 15 |
| Claude Opus 5.5 (low) · Claude Code | 100% | 80%–100% | 15 |
| Claude Haiku 4.5 · Claude Code | 100% | 80%–100% | 15 |
| GPT-6.1 Sol (high) · Codex CLI | 100% | 80%–100% | 15 |
| GPT-6.1 Sol (medium) · Codex CLI | 100% | 80%–100% | 15 |
| GPT-6.1 Sol (low) · Codex CLI | 100% | 72%–100% | 10 |
9 rows. Highest Claude Fable 5.1 · Claude Code 100% (95% interval 80%–100%, n 15). Lowest Claude Sonnet 5.5 · Claude Code 80% (95% interval 55%–93%, n 15). All intervals overlap.
NotesWhiskers: 95% Wilson intervaln 10–15 per row8 of 9 at 100%: this task set cannot separate them.
Every call counts; failures and timeouts are non-passes
Whiskers are 95% Wilson intervals. The tasks are short and most configurations pass nearly all of them, so pass rate does not separate the models here.
Source: Provider head-to-head: Claude Code models vs Codex efforts
Every lane’s run range overlaps another lane’s, so the chart shows no finish order.
| Item | Total time per call | Range (lowest–highest run) | n |
|---|---|---|---|
| Claude Fable 5.1 · Claude Code | 1.9 s | 1.4 s–9.8 s | 15 |
| Claude Sonnet 5.5 · Claude Code | 2.3 s | 2.2 s–7.7 s | 15 |
| Claude Opus 5.5 (high) · Claude Code | 2.7 s | 2.5 s–11.8 s | 15 |
| Claude Opus 5.5 · Claude Code | 2.8 s | 2.5 s–8.9 s | 15 |
| Claude Opus 5.5 (low) · Claude Code | 2.8 s | 2.4 s–6.6 s | 15 |
| Claude Haiku 4.5 · Claude Code | 4.4 s | 3.2 s–23.6 s | 15 |
| GPT-6.1 Sol (high) · Codex CLI | 5.6 s | 4.1 s–19.5 s | 15 |
| GPT-6.1 Sol (medium) · Codex CLI | 5.7 s | 4.1 s–25.5 s | 15 |
| GPT-6.1 Sol (low) · Codex CLI | 6.3 s | 4.7 s–10.5 s | 10 |
9 rows. Slowest GPT-6.1 Sol (low) · Codex CLI 6.3 s (range 4.7 s–10.5 s, n 10). Fastest Claude Fable 5.1 · Claude Code 1.9 s (range 1.4 s–9.8 s, n 15). All run ranges overlap.
NotesLines: fastest–slowest run (not an interval)n 10–15 per row
Median per configuration; whiskers = fastest and slowest call
One host, one network, one day. Whiskers are a range, not a confidence interval.
Source: Provider head-to-head: Claude Code models vs Codex efforts
Every lane’s run range overlaps another lane’s, so the chart shows no finish order.
| Item | Time to first useful output | Range (lowest–highest run) | n |
|---|---|---|---|
| Claude Fable 5.1 · Claude Code | 1.2 s | 1 s–7.9 s | 15 |
| Claude Sonnet 5.5 · Claude Code | 1.6 s | 1 s–6.4 s | 15 |
| Claude Opus 5.5 (high) · Claude Code | 2 s | 1.4 s–9.9 s | 15 |
| Claude Opus 5.5 · Claude Code | 1.9 s | 1.6 s–7.2 s | 15 |
| Claude Opus 5.5 (low) · Claude Code | 2.4 s | 1.5 s–4.9 s | 15 |
| Claude Haiku 4.5 · Claude Code | 3.6 s | 2.8 s–22.3 s | 15 |
| GPT-6.1 Sol (high) · Codex CLI | 5.3 s | 3.6 s–16.4 s | 15 |
| GPT-6.1 Sol (medium) · Codex CLI | 5.1 s | 3.4 s–17.8 s | 15 |
| GPT-6.1 Sol (low) · Codex CLI | 5.1 s | 4 s–8.5 s | 10 |
9 rows. Slowest GPT-6.1 Sol (high) · Codex CLI 5.3 s (range 3.6 s–16.4 s, n 15). Fastest Claude Fable 5.1 · Claude Code 1.2 s (range 1 s–7.9 s, n 15). All run ranges overlap.
NotesLines: fastest–slowest run (not an interval)n 10–15 per row
Median per configuration; whiskers = fastest and slowest call
One host, one network, one day. Whiskers are a range, not a confidence interval.
Source: Provider head-to-head: Claude Code models vs Codex efforts
- Output tokens
- of which reasoning tokens (inner bar)
| Item | Output tokens | Reasoning tokens | n |
|---|---|---|---|
| Claude Fable 5.1 · Claude Code | 64 | 0 | 15 |
| Claude Sonnet 5.5 · Claude Code | 107 | 0 | 15 |
| Claude Opus 5.5 (high) · Claude Code | 78 | 34 | 15 |
| Claude Opus 5.5 · Claude Code | 64 | 0 | 15 |
| Claude Opus 5.5 (low) · Claude Code | 64 | 0 | 15 |
| Claude Haiku 4.5 · Claude Code | 367 | 297 | 15 |
| GPT-6.1 Sol (high) · Codex CLI | 42 | 21 | 15 |
| GPT-6.1 Sol (medium) · Codex CLI | 42 | 20 | 15 |
| GPT-6.1 Sol (low) · Codex CLI | 42 | 20 | 10 |
9 rows, 2 series: Output tokens, Reasoning tokens. Output tokens: highest Claude Haiku 4.5 · Claude Code 367 (n 15). Lowest GPT-6.1 Sol (low) · Codex CLI 42 (n 10). Reasoning tokens: highest Claude Haiku 4.5 · Claude Code 297 (n 15). Lowest Claude Opus 5.5 (low) · Claude Code 0 (n 15).
Notesn 10–15 per row
Median per configuration; reasoning tokens where the CLI reports them
Vendors count reasoning differently; compare within a vendor first. A median of 0 can mean the CLI did not report reasoning for most calls.
Source: Provider head-to-head: Claude Code models vs Codex efforts
Every interval overlaps every other: this chart does not order these rows.
| Item | Cost per call | Range (lowest–highest run) | n |
|---|---|---|---|
| Claude Fable 5.1 · Claude Code | $0.0099 | $0.0049–$0.058 | 15 |
| Claude Sonnet 5.5 · Claude Code | $0.0036 | $0.0034–$0.01 | 15 |
| Claude Opus 5.5 (high) · Claude Code | $0.0069 | $0.0059–$0.027 | 15 |
| Claude Opus 5.5 · Claude Code | $0.0069 | $0.0059–$0.022 | 15 |
| Claude Opus 5.5 (low) · Claude Code | $0.0069 | $0.0058–$0.018 | 15 |
| Claude Haiku 4.5 · Claude Code | $0.0057 | $0.0051–$0.018 | 15 |
| GPT-6.1 Sol (high) · Codex CLI | $0.01 | $0.0066–$0.028 | 15 |
| GPT-6.1 Sol (medium) · Codex CLI | $0.01 | $0.0054–$0.027 | 15 |
| GPT-6.1 Sol (low) · Codex CLI | $0.0077 | $0.0074–$0.026 | 10 |
List-price calculation, not a run. 9 rows. Highest GPT-6.1 Sol (high) · Codex CLI $0.01 (range $0.0066–$0.028, n 15). Lowest Claude Sonnet 5.5 · Claude Code $0.0036 (range $0.0034–$0.01, n 15). All run ranges overlap.
NotesLines: lowest–highest run (not an interval)n 10–15 per row
Reported tokens × list price; the calls ran on subscriptions
Calculation, not a bill: the calls ran on flat subscriptions. Whiskers = cheapest and most expensive call.
Sources: Provider head-to-head: Claude Code models vs Codex efforts, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices
| Point | USD per call (list-price calculation) | Median total seconds | n |
|---|---|---|---|
| Claude Fable 5.1 · Claude Code | $0.0099 | 1.9 s | 15 |
| Claude Sonnet 5.5 · Claude Code | $0.0036 | 2.3 s | 15 |
| Claude Opus 5.5 (high) · Claude Code | $0.0069 | 2.7 s | 15 |
| Claude Opus 5.5 · Claude Code | $0.0069 | 2.8 s | 15 |
| Claude Opus 5.5 (low) · Claude Code | $0.0069 | 2.8 s | 15 |
| Claude Haiku 4.5 · Claude Code | $0.0057 | 4.4 s | 15 |
| GPT-6.1 Sol (high) · Codex CLI | $0.01 | 5.6 s | 15 |
| GPT-6.1 Sol (medium) · Codex CLI | $0.01 | 5.7 s | 15 |
| GPT-6.1 Sol (low) · Codex CLI | $0.0077 | 6.3 s | 10 |
List-price calculation, not a run. 9 points: Median total seconds against USD per call (list-price calculation). USD per call (list-price calculation) runs from $0.0036 to $0.01; Median total seconds from 1.9 s to 6.3 s. Highlighted: Claude Fable 5.1 · Claude Code, Claude Sonnet 5.5 · Claude Code, Claude Opus 5.5 (high) · Claude Code, Claude Opus 5.5 · Claude Code, Claude Opus 5.5 (low) · Claude Code, Claude Haiku 4.5 · Claude Code.
Notesn 10–15 per point
Median total time and median list-price cost per call
Lower-left is faster and cheaper. Cost is a calculation from tokens.
Sources: Provider head-to-head: Claude Code models vs Codex efforts, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices
Hover or focus a bar for its ratio to Claude Sonnet 5.5 (the lowest value): a ratio of list-price calculations, not a measurement.
| Item | Cost per pass | n |
|---|---|---|
| Claude Sonnet 5.5 · Claude Code | $0.0062 | 15 |
| Claude Opus 5.5 (low) · Claude Code | $0.0083 | 15 |
| Claude Haiku 4.5 · Claude Code | $0.0084 | 15 |
| GPT-6.1 Sol (low) · Codex CLI | $0.01 | 10 |
| Claude Opus 5.5 · Claude Code | $0.01 | 15 |
| Claude Opus 5.5 (high) · Claude Code | $0.01 | 15 |
| GPT-6.1 Sol (high) · Codex CLI | $0.013 | 15 |
| GPT-6.1 Sol (medium) · Codex CLI | $0.016 | 15 |
| Claude Fable 5.1 · Claude Code | $0.021 | 15 |
List-price calculation, not a run. 9 rows. Highest Claude Fable 5.1 · Claude Code $0.021 (n 15). Lowest Claude Sonnet 5.5 · Claude Code $0.0062 (n 15).
Notesn 10–15 per row
All calls in a configuration, failures included, divided by its passes
Calculation, not a bill: reported tokens × list price; the calls ran on flat subscriptions. A failed call still costs, so a lower pass rate raises the cost per pass. Highlighted bars are on the speed/cost/quality frontier.
Sources: Provider head-to-head: Claude Code models vs Codex efforts, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices
- Claude Code
- Codex CLI
Haloed: on the frontier (5 of 9). The frontier also uses a measure this plot does not show, so no line is drawn.
| Point | Series | USD per passing answer (list-price calculation) | Median total seconds | n |
|---|---|---|---|---|
| Claude Fable 5.1 · Claude Code | Claude Code | $0.021 | 1.9 s | 15 |
| Claude Sonnet 5.5 · Claude Code | Claude Code | $0.0062 | 2.3 s | 15 |
| Claude Opus 5.5 (high) · Claude Code | Claude Code | $0.01 | 2.7 s | 15 |
| Claude Opus 5.5 · Claude Code | Claude Code | $0.01 | 2.8 s | 15 |
| Claude Opus 5.5 (low) · Claude Code | Claude Code | $0.0083 | 2.8 s | 15 |
| Claude Haiku 4.5 · Claude Code | Claude Code | $0.0084 | 4.4 s | 15 |
| GPT-6.1 Sol (high) · Codex CLI | Codex CLI | $0.013 | 5.6 s | 15 |
| GPT-6.1 Sol (medium) · Codex CLI | Codex CLI | $0.016 | 5.7 s | 15 |
| GPT-6.1 Sol (low) · Codex CLI | Codex CLI | $0.01 | 6.3 s | 10 |
List-price calculation, not a run. 9 points: Median total seconds against USD per passing answer (list-price calculation). USD per passing answer (list-price calculation) runs from $0.0062 to $0.021; Median total seconds from 1.9 s to 6.3 s. Highlighted: Claude Fable 5.1 · Claude Code, Claude Sonnet 5.5 · Claude Code, Claude Opus 5.5 (high) · Claude Code, Claude Opus 5.5 · Claude Code, Claude Opus 5.5 (low) · Claude Code.
Notesn 10–15 per point
Median seconds against list-price cost per passing answer; pass rate in the note
Lower-left is better. Highlighted points are on the frontier: no other configuration is at least as fast, as cheap per pass and as accurate. Frontier: Claude Fable 5.1 · Claude Code; Claude Sonnet 5.5 · Claude Code; Claude Opus 5.5 (high) · Claude Code; Claude Opus 5.5 · Claude Code; Claude Opus 5.5 (low) · Claude Code. Pass rates: Claude Fable 5.1 · Claude Code 15/15; Claude Sonnet 5.5 · Claude Code 12/15; Claude Opus 5.5 (high) · Claude Code 15/15; Claude Opus 5.5 · Claude Code 15/15; Claude Opus 5.5 (low) · Claude Code 15/15; Claude Haiku 4.5 · Claude Code 15/15; GPT-6.1 Sol (high) · Codex CLI 15/15; GPT-6.1 Sol (medium) · Codex CLI 15/15; GPT-6.1 Sol (low) · Codex CLI 10/10. Costs are calculations from tokens; medians come from small samples.
Sources: Provider head-to-head: Claude Code models vs Codex efforts, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices
Tables
Pass matrix: configuration × task
Configuration by column: each cell is k of n from the table; a cell opens its model
- Fix a buggy median function
- Extract invoice fields to JSON
- Multi-step shift arithmetic
- Refactor recursion to iteration
- Classify six support tickets
Shade is the share k of n in each cell. Totals and other columns are in the Table view.
| Configuration | Fix a buggy median function | Extract invoice fields to JSON | Multi-step shift arithmetic | Refactor recursion to iteration | Classify six support tickets | Median total (s) |
|---|---|---|---|---|---|---|
| Claude Fable 5.1 · Claude Code | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 1.9 s |
| Claude Sonnet 5.5 · Claude Code | 3/3 | 3/3 | 0/3 | 3/3 | 3/3 | 2.3 s |
| Claude Opus 5.5 (high) · Claude Code | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 2.7 s |
| Claude Opus 5.5 · Claude Code | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 2.8 s |
| Claude Opus 5.5 (low) · Claude Code | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 2.8 s |
| Claude Haiku 4.5 · Claude Code | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 4.4 s |
| GPT-6.1 Sol (high) · Codex CLI | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 5.6 s |
| GPT-6.1 Sol (medium) · Codex CLI | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 5.7 s |
| GPT-6.1 Sol (low) · Codex CLI | 2/2 | 2/2 | 2/2 | 2/2 | 2/2 | 6.3 s |
Counts from the table, not an intervaln = 2–3 per cell
9 rows by 5 columns. 44 of 45 k/n cells are full (n of n) and 1 are zero.
Method
- Protocol declared before the first call: 5 cases with deterministic validators (behavioral checks in a sandbox, canonical JSON or exact text).
- Claude Code: Haiku, Sonnet, Opus and Fable at default effort, plus Opus at low and high, 3 repetitions. Codex: GPT-6.1 Sol at low, medium and high, 2 repetitions.
- Isolation: fresh empty working folder, tools off, no MCP servers, no session persistence, one turn. One call at a time per account.
- Every attempt is kept. Nothing is retried. A cell that did not run is shown as trimmed.
- All 130 planned calls ran. Nothing was trimmed or retried, and no run hit a usage or rate limit.
- Cost per passing answer: list price × reported tokens for every call in the configuration (cache reads and writes priced separately), divided by its passes.
Caveats
- The tasks are short and easy; pass rate saturates. Latency and tokens carry the signal. A harder follow-up with eight tasks and strict validators: /benchmarks/hard-model-head-to-head.
- Few repetitions per cell (2 or 3 per task). Medians with ranges, not intervals.
- CLI timings include CLI start-up and the CLI’s own system prompt.
- One host, one network, one day.
- List-price costs are calculations; the calls used flat subscriptions.
- The prompt cache stayed at the provider default, so cache counters differ by route and by call order.
- Haiku 4.5 reported reasoning tokens on most calls under the CLI default, which explains much of its extra time and output.
- Haiku and Fable are not in the platform runner catalog, so these calls used the runner’s CLI functions directly; each receipt records the model the CLI reported.
Sources
Provider head-to-head: Claude Code models vs Codex efforts
Five short tasks with deterministic validators, declared protocol, every attempt kept.
Repricing calculation
Recorded token counts multiplied by the list prices in the price-list sources above. A calculation, not a run: a different model would have used a different number of tokens and reached different outcomes.
Anthropic list prices (Claude models)
Prices as listed by the vendor on 2026-09-21 and recorded in the product price table. Cache reads at the listed rate, one-hour cache writes at twice the input price.
Token prices as listed by the vendor on 2026-10-03.
Download the data
The study as JSON (with its sources), every chart point as CSV, or the whole public dataset. Free to reuse under CC BY 4.0: credit Agent and link the study.
Share it as a carousel
Square slides made in your browser from the charts on this page, with the same numbers, intervals and notes, and a captions file for alt text.
Cite as: Agent public benchmarks, “Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head”, updated October 5, 2026, https://agent.sasid.ai/benchmarks/model-head-to-head.
Explainers that cite this study
Read the methods and terms in the context of these recorded results.
- Benchmark saturation: when every model scores 100%
- Cost per correct answer: the LLM price that counts failures
- How to read AI benchmarks honestly
- The Pareto frontier: reading LLM cost against quality
- pass@k explained: pass@1, pass@k and pass^k with real runs
- Strict grading and format misses: when a right answer fails
- Time to first token (TTFT), explained with CLI and API timings
- Tokens per call and the CLI context tax
More comparisons based on this study (9)
These pages reuse this study’s recorded rows. Read each page’s original sample, comparability and ceiling limits.
- Claude Fable 5.1 vs GPT-6.1 Sol (Codex CLI)
- Claude Haiku 4.5 vs Claude Fable 5.1
- Claude Haiku 4.5 vs GPT-6.1 Sol (Codex CLI)
- Claude Opus 5.5: high vs default effort
- Claude Opus 5.5: low vs default effort
- Claude Opus 5.5: low vs high effort
- GPT-6.1 Sol (Codex CLI): low vs high effort
- GPT-6.1 Sol (Codex CLI): low vs medium effort
- GPT-6.1 Sol (Codex CLI): medium vs high effort
Models and comparisons in this study
- Claude Sonnet 5.5
- Claude Opus 5.5
- Claude Haiku 4.5
- Claude Fable 5.1
- GPT-6.1 Sol (Codex CLI)
- Claude Code
- Codex CLI
- Claude Sonnet 5.5 vs Claude Opus 5.5
- Claude Haiku 4.5 vs Claude Sonnet 5.5
- Claude Opus 5.5 vs Claude Fable 5.1
- Claude Sonnet 5.5 vs Claude Fable 5.1
- Claude Haiku 4.5 vs Claude Opus 5.5
- Claude Code vs Codex CLI
- Claude Sonnet 5.5 vs GPT-6.1 Sol (Codex CLI)
- Claude Opus 5.5 vs GPT-6.1 Sol (Codex CLI)
- All comparisons
Write-ups on this study
A latency budget for voice agents: which LLM steps fit in one turn?
136.5 ms for Jev, 0.82 s for a small-model API, 2.79 s to 3.79 s for Codex CLI: which steps fit a voice agent latency budget? A thought experiment.
Best LLM for JSON output? Haiku, Sonnet and GPT-6.1 Sol, 10 runs each
Best LLM for JSON output? On one prompt, Sonnet and GPT-6.1 Sol passed 10/10 (95%: 72–100%); Haiku passed 1/10 (2–40%). CLI results, not JSON mode.
Claude Code cost per task: a price ladder from one decision to one agent run
$0.087 per Claude Code coding session at API list prices, $0.0036 per short call, $2.81 per full agent attempt. Calculations on recorded tokens, not bills.
Claude Fable 5.1 vs Opus 5.5 vs Sonnet 5.5: speed, tokens and price tested
24 of 24: Claude Fable 5.1, Opus 5.5 and Sonnet 5.5 each passed every hard task. Fable cost 3.3x Opus and 6.5x Sonnet per pass (list-price calculation).
Claude Haiku 4.5 vs Sonnet 5.5: all 80 comparison rows, and where the small model loses
Haiku 4.5 vs Sonnet 5.5 on 80 rows: Sonnet ahead on 14, Haiku on none, 31 ties. Hard tasks 11/24 vs 24/24, plus speed, memory and price.
Format misses vs wrong answers: your LLM eval may be failing right answers
5 of 13 failed calls on our hard set were right answers in the wrong format. How to grade LLM output strictly, test validators first and report both numbers.
GPT-6.1 Sol vs Claude Sonnet 5.5 vs Opus 5.5: every row we measured
Of 70 comparison rows for GPT-6.1 Sol, Claude Sonnet 5.5 and Opus 5.5, only 2 have a winner (speed). All 15 pass-rate rows tie. Tokens, price and route differ.
How many runs do you need to compare two AI models? A sample-size table
Calculation: 100 runs can separate an observed 8-point gap at a perfect score. See Wilson intervals, paired tests and limits for LLM eval sample sizes.
Is Claude Haiku cheaper than Sonnet? Cost per correct answer, with retries
Haiku 4.5 lists at half the price of Sonnet 5.5, yet calculated cost per correct hard answer was 4.7x higher. Retry maths, two exceptions and every source.
LLM API pricing comparison, October 2026: Claude vs GPT vs Gemini per million tokens
21 LLM API prices per million tokens, October 2026: Claude, GPT, Gemini and more. Blended prices span 100x. Price per token is not price per task.
Most AI model comparisons are ties: 44 of 1,196 rows show a clear gap
44 of 1,196 AI model comparison rows show a gap under our overlap rules. Most gaps are timing rows. No effort row separates quality.
The cheapest way to run an AI coding agent: 7 levers from measured runs
7 levers that may cut an AI coding agent's bill, sized from our data: prompt cache 3.9x, Fable/Sonnet cost per pass 6.5x, and 5 more. List-price calculations.
What is the best AI model for coding? Our data says four models tie
4 models tied at the top of our hard coding set: Sonnet 5.5, Opus 5.5, Fable 5.1 and GPT-6.1 Sol. Only Haiku 4.5 separated. A tier list built from intervals.
What thinking costs: reasoning tokens per call for Claude and GPT-6.1 Sol
Haiku 4.5 spent a median 4,556 reasoning tokens per hard call, Sonnet 5.5 585, GPT-6.1 Sol 150. List price per 1,000 calls: $22.78, $5.85, $1.50 (calculation).
Which Claude model is fastest? It depends on the task, and on thinking
1.94 s was the lowest median on short calls (Fable 5.1). 7.75 s on hard calls (Sonnet 5.5). Haiku 4.5 took 4.43 s and 39.01 s with default thinking.
Which Claude model should you use? A task-by-task guide from our measurements
Which Claude model fits your job? Sonnet passed 24/24 hard calls (95% interval 86%–100%) at $0.01435 per pass (calculation). The top models hit a ceiling.
Why is Claude Code slow? Where the seconds go in a coding CLI call, and what to change
A one-word Claude Code call took a median 2.5 s, with 1.7 s outside the model (n = 5). Where the rest goes: thinking, effort, route. Measured splits and fixes.
AI coding benchmarks roundup, October 2026: sixteen studies, every number in one place
Sixteen AI benchmark studies on one page: SWE-bench, Claude Code vs Codex CLI, agent memory, effort, caching, routing, decision models and provider prices.
Claude Code vs Codex CLI on hidden tests: when every agent passes, what differs?
36 hidden-test sessions: Sonnet 5.5 and Opus 5.5 in Claude Code, GPT-6.1 Sol in Codex CLI. All passed. Time, tool calls, diffs and one confound differ.
Claude Code vs Codex CLI: the hidden context tax and what it does to speed
Codex CLI sent a median 12,124 input tokens per call; Claude Code sent 2,130. What a coding CLI's hidden context costs, and how much time the route adds.
Claude Sonnet 5.5 vs Opus 5.5: when is Opus worth the price?
Sonnet 5.5 and Opus 5.5 tied on every quality test we ran, easy, hard and agentic. Opus cost 1.6x to 2.6x per unit of work. Where the gap comes from.
Claude vs Codex on hard tasks: GPT-6.1 Sol joins the hard set
GPT-6.1 Sol in the Codex CLI passed 16/16 hard tasks at medium and high effort. Sonnet, Opus and Fable passed 24/24. What separates them: time and cost.
How to estimate your AI coding bill from real token mixes
Estimate AI coding costs from recorded token mixes: an agent task at $2.64, a hard call at $0.014, a routing decision at $0.005. Calculations, limits stated.
When tasks get hard: Claude Haiku vs Sonnet vs Opus vs Fable (and where Codex is)
120 calls on 8 hard tasks with strict validators. Sonnet, Opus and Fable passed 24/24; Haiku 11/24. Speed, tokens and cost per pass. Codex results: see update.
Claude Code vs Codex CLI vs the API: latency, time to first token and hidden prompts
194 timed runs. Codex CLI took 3.5x as long as the OpenAI API for a one-line answer and sent 19,551 input tokens instead of 17. What a coding CLI adds.
Claude Haiku vs Sonnet vs Opus vs Fable vs Codex: 130 timed calls, head to head
130 timed calls on five validated tasks. Pass rate, latency, hidden input tokens and cost per passing answer for Claude Code models and Codex CLI.
More studies
All benchmarksHaiku vs Sonnet vs Opus vs Fable vs GPT-6.1 Sol on 8 hard tasks
152 calls on 8 hard tasks with strict validators. Pass rate with 95% intervals, format misses, speed, tokens and cost per pass.
GPT-6.1 Sol vs Claude Opus 5.5, Sonnet 5.5 and Haiku 4.5 on 4 harder tasks
56 counted calls on 4 harder tasks with strict validators: GPT-6.1 Sol, Claude Opus 5.5, Sonnet 5.5, Haiku 4.5. Pass rate, intervals, speed, cost.
Where the seconds go: first text, output speed and prompt size for 6 LLMs
60 calls on 6 models: time to first text, tokens per second, and what a 1k, 16k or 64k prompt adds. Claude Code and Codex CLI.