9 measured metrics · 5 calculated · 3 studies
Claude Opus 5.5lowvshigheffort
“default”: the effort flag was not passed and the CLI chose.
No row separates them: 3 ties, 11 unclear.
The verdict
Claude Opus 5.5 at low effort and Claude Opus 5.5 at high effort share 9 measured metrics and 5 list-price calculations from 3 studies. Only the effort setting differs between the two sides of a row; the route and the task set are the same. No row separates them: every interval or run range overlaps, too few runs were recorded, no interval was recorded, or more is not better for that metric. The rows are 3 ties and 11 unclear; each row says why. Calculation rows are derived from list prices and recorded counts; they are not bills or runs. Some rows rest on small samples (n = 15 at the smallest).
Headline metrics
How far apart the two sides are on the headline metrics. Length is the ratio of the two values; it is not a winner.
Bar length is the ratio of the two values on a log scale, pointing to the larger one. Larger is not better for time, tokens or cost. A bar has a side’s color only when that side is ahead in the data; gray means the data does not separate them.
| Metric | Claude Opus 5.5 at low effort | Claude Opus 5.5 at high effort | n | Interval or range | Outcome | Basis | Study |
|---|---|---|---|---|---|---|---|
| Pass rate on five validated tasks | 100% (15/15)Claude Code · effort low · five short validated tasks | 100% (15/15)Claude Code · effort high · five short validated tasks | 15 | 95% CI: 80%–100% vs 80%–100% | Tie | The 95% intervals overlap (Claude Opus 5.5 at low effort 80% to 100%; Claude Opus 5.5 at high effort 80% to 100%), so this sample cannot separate them. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Total time per call | 2.83 sClaude Code · effort low · five short validated tasks | 2.71 sClaude Code · effort high · five short validated tasks | 15 | range: 2.4 s–6.6 s vs 2.5 s–11.8 s | Unclear | The run ranges (fastest to slowest) overlap (Claude Opus 5.5 at low effort 2.35 s to 6.62 s; Claude Opus 5.5 at high effort 2.45 s to 11.8 s); the medians alone do not show a reliable difference. A range is not a confidence interval. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Time to first useful output | 2.39 sClaude Code · effort low · five short validated tasks | 2.04 sClaude Code · effort high · five short validated tasks | 15 | range: 1.5 s–4.9 s vs 1.4 s–9.9 s | Unclear | The run ranges (fastest to slowest) overlap (Claude Opus 5.5 at low effort 1.45 s to 4.90 s; Claude Opus 5.5 at high effort 1.40 s to 9.94 s); the medians alone do not show a reliable difference. A range is not a confidence interval. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Input tokens per call: what the CLI sends (Cache read) | 1,463Claude Code · effort low · five short validated tasks | 1,463Claude Code · effort high · five short validated tasks | 15 | none recorded | Tie | Same value. More or fewer is not better by itself for this metric. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Output tokens per call (Output tokens) | 64Claude Code · effort low · five short validated tasks | 78Claude Code · effort high · five short validated tasks | 15 | none recorded | Unclear | More or fewer tokens is not better or worse by itself; this row describes behaviour, not a winner. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| List-price cost per passing answer (calculation)Calculation | $0.0083Claude Code · effort low · five short validated tasks | $0.010Claude Code · effort high · five short validated tasks | 15 | none recorded | Unclear | No interval or range was recorded for either side, so the gap ($0.0083 vs $0.010) is not tested against run-to-run variation. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
Marks: 95% intervals (Wilson for rates); fastest–slowest run ranges (not intervals)n is shown per side on every rowHollow marks: list-price calculations, not runs
6 headline metrics as the ratio of the two values. None of them separates the sides in the data. Widest ratio: List-price cost per passing answer (calculation), 1.3x (High effort larger).
Watch it build
A live story drawn from the same dataset in your browser. It shows the same numbers, notes and interval labels as this page.
Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on eight hard tasks
All 11 configurations passed 16/16 strictly (95% interval 81%–100%): more effort bought no extra passes on this set (a ceiling). Median output tokens rose with effort; time ranges overlap. Costs are list-price calculations.
Transcript
- Effort ladder · 176 calls · 11 configurations. Does more effort buy quality? Low, medium and high effort on eight hard tasks. Sonnet 5.5 and Opus 5.5 via Claude Code, GPT-6.1 Sol via Codex CLI.
- 176 of 176 calls passed strictly, at every effort level. No format misses and no wrong answers. Calls that passed strictly, every effort: 100% (176/176) (n = 176, 95% CI 98–100%). Model and effort configurations: 11 (6 new, 5 reference) (n = 176). Format misses and wrong answers: 0 · 0 (n = 176, format · wrong). Caveat: Reference cells ran in a different batch and hour than the new cells (provider load can differ), on the same host, CLI versions, cases and flags.
- All 11 configurations passed 16/16. A 16/16 has a 95% interval of 81%–100%: the set has a ceiling, so no effort level wins. Chart: Strict pass rate by effort · 95% Wilson intervals (n = 16 each). Caveat: Every cell passed every call, so the set has a ceiling: a perfect 16/16 has a 95% interval of 81% to 100%. This study cannot show that effort does not matter on harder work; it shows that these 8 tasks do not need more than low effort.
- Median time: Sonnet 5.5 5.8 s at low, 8.8 s at high; Opus 5.5 7.5 s at low, 10.1 s at high; GPT-6.1 Sol 13.6 s at low, 18.1 s at high. Within each model the ranges overlap: no effort is ranked faster. Chart: Median total time per call · whiskers: fastest to slowest call (n = 16 each). Caveat: Each cell has only 2 calls per task. Medians of 16 calls move with a few slow calls; the ranges are wide and overlap.
- More effort writes more: Sonnet 5.5 used a median 667 / 770 / 1,192 output tokens at low / medium / high. Chart: Median output and reasoning tokens per call (n = 16 each). Caveat: Claude Code and Codex CLI rows pair a CLI with a model; each CLI adds its own system prompt and start-up time. Effort levels are not the same scale across vendors.
- List-price cost per strict pass, low → high effort, a calculation: Sonnet $0.0122 → $0.0167, Opus $0.0212 → $0.0337, GPT-6.1 Sol $0.0128 → $0.0151. Chart: List-price cost per strict pass · calculation (n = 16 each). Calculation, not a run. Caveat: List-price costs are calculations; the calls used flat subscriptions.
- 16 effort pairs, 172 comparison rows: no effort level wins any row. The rest are ties or unclear. Effort pairs compared (same model, same route): 16. Comparison rows across those pairs: 172. Rows where one effort level wins: 0 (of 172). Caveat: Every cell passed every call, so the set has a ceiling: a perfect 16/16 has a 95% interval of 81% to 100%. This study cannot show that effort does not matter on harder work; it shows that these 8 tasks do not need more than low effort.
- Start at low effort and measure. Every call, interval and cost online.
Metric by metric
Both values of a row come from the same chart in the same study. Each row has its own axis. The shaded band is where the two intervals or ranges overlap: a side is ahead only when they do not.
- Low effort
- High effort
- 95% interval
- fastest–slowest run (not an interval)
- where the two overlap
- hollow: list-price calculation
Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head
- Pass rate on five validated tasks100% (15/15)n 15100% (15/15)n 15TiePass rate on five validated tasks: Low effort 100% (15/15) (n 15, 95% interval 80%–100%); High effort 100% (15/15) (n 15, 95% interval 80%–100%). Tie.
- Total time per call2.83 sn 152.71 sn 15UnclearTotal time per call: Low effort 2.83 s (n 15, run range 2.4 s–6.6 s); High effort 2.71 s (n 15, run range 2.5 s–11.8 s). Unclear.
- Time to first useful output2.39 sn 152.04 sn 15UnclearTime to first useful output: Low effort 2.39 s (n 15, run range 1.5 s–4.9 s); High effort 2.04 s (n 15, run range 1.4 s–9.9 s). Unclear.
- Input tokens per call: what the CLI sends (Cache read)1,463n 151,463n 15TieInput tokens per call: what the CLI sends (Cache read): Low effort 1,463 (n 15); High effort 1,463 (n 15). Tie.
- Input tokens per call: what the CLI sends (Other input)618n 15619n 15UnclearInput tokens per call: what the CLI sends (Other input): Low effort 618 (n 15); High effort 619 (n 15). Unclear.
- Output tokens per call (Output tokens)64n 1578n 15UnclearOutput tokens per call (Output tokens): Low effort 64 (n 15); High effort 78 (n 15). Unclear.
- List-price cost per call (calculation)Calculation$0.0069n 15$0.0069n 15UnclearList-price cost per call (calculation), calculation: Low effort $0.0069 (n 15, run range $0.0058–$0.018); High effort $0.0069 (n 15, run range $0.0059–$0.027). Unclear.
- List-price cost per passing answer (calculation)Calculation$0.0083n 15$0.010n 15UnclearList-price cost per passing answer (calculation), calculation: Low effort $0.0083 (n 15); High effort $0.010 (n 15). Unclear.
Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks
- Strict pass rate by effort on eight hard tasks100% (16/16)n 16100% (16/16)n 16TieStrict pass rate by effort on eight hard tasks: Low effort 100% (16/16) (n 16, 95% interval 81%–100%); High effort 100% (16/16) (n 16, 95% interval 81%–100%). Tie.
- Total time per call by effort on hard tasks7.50 sn 1610.1 sn 16UnclearTotal time per call by effort on hard tasks: Low effort 7.50 s (n 16, run range 3.3 s–15.8 s); High effort 10.1 s (n 16, run range 3.6 s–63 s). Unclear.
- Output tokens per call by effort on hard tasks (Output tokens)594n 161,052n 16UnclearOutput tokens per call by effort on hard tasks (Output tokens): Low effort 594 (n 16); High effort 1,052 (n 16). Unclear.
- List-price cost per strict pass by effort (calculation)Calculation$0.021n 16$0.034n 16UnclearList-price cost per strict pass by effort (calculation), calculation: Low effort $0.021 (n 16); High effort $0.034 (n 16). Unclear.
How much of an AI bill is thinking? Reasoning tokens by model and effort
- Reasoning cost per strict pass by effort, with the total (calculation) (Reasoning cost per strict pass)Calculation$0.0050n 16$0.018n 16UnclearReasoning cost per strict pass by effort, with the total (calculation) (Reasoning cost per strict pass), calculation: Low effort $0.0050 (n 16); High effort $0.018 (n 16). Unclear.
- Reasoning cost per strict pass by effort, with the total (calculation) (Total cost per strict pass)Calculation$0.021n 16$0.034n 16UnclearReasoning cost per strict pass by effort, with the total (calculation) (Total cost per strict pass), calculation: Low effort $0.021 (n 16); High effort $0.034 (n 16). Unclear.
| Metric | Claude Opus 5.5 at low effort | Claude Opus 5.5 at high effort | n | Interval or range | Outcome | Basis | Study |
|---|---|---|---|---|---|---|---|
| Pass rate on five validated tasks | 100% (15/15)Claude Code · effort low · five short validated tasks | 100% (15/15)Claude Code · effort high · five short validated tasks | 15 | 95% CI: 80%–100% vs 80%–100% | Tie | The 95% intervals overlap (Claude Opus 5.5 at low effort 80% to 100%; Claude Opus 5.5 at high effort 80% to 100%), so this sample cannot separate them. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Total time per call | 2.83 sClaude Code · effort low · five short validated tasks | 2.71 sClaude Code · effort high · five short validated tasks | 15 | range: 2.4 s–6.6 s vs 2.5 s–11.8 s | Unclear | The run ranges (fastest to slowest) overlap (Claude Opus 5.5 at low effort 2.35 s to 6.62 s; Claude Opus 5.5 at high effort 2.45 s to 11.8 s); the medians alone do not show a reliable difference. A range is not a confidence interval. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Time to first useful output | 2.39 sClaude Code · effort low · five short validated tasks | 2.04 sClaude Code · effort high · five short validated tasks | 15 | range: 1.5 s–4.9 s vs 1.4 s–9.9 s | Unclear | The run ranges (fastest to slowest) overlap (Claude Opus 5.5 at low effort 1.45 s to 4.90 s; Claude Opus 5.5 at high effort 1.40 s to 9.94 s); the medians alone do not show a reliable difference. A range is not a confidence interval. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Input tokens per call: what the CLI sends (Cache read) | 1,463Claude Code · effort low · five short validated tasks | 1,463Claude Code · effort high · five short validated tasks | 15 | none recorded | Tie | Same value. More or fewer is not better by itself for this metric. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Input tokens per call: what the CLI sends (Other input) | 618Claude Code · effort low · five short validated tasks | 619Claude Code · effort high · five short validated tasks | 15 | none recorded | Unclear | More or fewer tokens is not better or worse by itself; this row describes behaviour, not a winner. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Output tokens per call (Output tokens) | 64Claude Code · effort low · five short validated tasks | 78Claude Code · effort high · five short validated tasks | 15 | none recorded | Unclear | More or fewer tokens is not better or worse by itself; this row describes behaviour, not a winner. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| List-price cost per call (calculation)Calculation | $0.0069Claude Code · effort low · five short validated tasks | $0.0069Claude Code · effort high · five short validated tasks | 15 | range: $0.0058–$0.018 vs $0.0059–$0.027 | Unclear | The run ranges (fastest to slowest) overlap (Claude Opus 5.5 at low effort $0.0058 to $0.018; Claude Opus 5.5 at high effort $0.0059 to $0.027); the medians alone do not show a reliable difference. A range is not a confidence interval. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| List-price cost per passing answer (calculation)Calculation | $0.0083Claude Code · effort low · five short validated tasks | $0.010Claude Code · effort high · five short validated tasks | 15 | none recorded | Unclear | No interval or range was recorded for either side, so the gap ($0.0083 vs $0.010) is not tested against run-to-run variation. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Strict pass rate by effort on eight hard tasks | 100% (16/16)Claude Code · effort low · eight hard validated tasks, effort ladder | 100% (16/16)Claude Code · effort high · eight hard validated tasks, effort ladder | 16 | 95% CI: 81%–100% vs 81%–100% | Tie | The 95% intervals overlap (Claude Opus 5.5 at low effort 81% to 100%; Claude Opus 5.5 at high effort 81% to 100%), so this sample cannot separate them. | Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks |
| Total time per call by effort on hard tasks | 7.50 sClaude Code · effort low · eight hard validated tasks, effort ladder | 10.1 sClaude Code · effort high · eight hard validated tasks, effort ladder | 16 | range: 3.3 s–15.8 s vs 3.6 s–63 s | Unclear | The run ranges (fastest to slowest) overlap (Claude Opus 5.5 at low effort 3.34 s to 15.8 s; Claude Opus 5.5 at high effort 3.63 s to 63.0 s); the medians alone do not show a reliable difference. A range is not a confidence interval. | Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks |
| Output tokens per call by effort on hard tasks (Output tokens) | 594Claude Code · effort low · eight hard validated tasks, effort ladder | 1,052Claude Code · effort high · eight hard validated tasks, effort ladder | 16 | none recorded | Unclear | More or fewer tokens is not better or worse by itself; this row describes behaviour, not a winner. | Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks |
| List-price cost per strict pass by effort (calculation)Calculation | $0.021Claude Code · effort low · eight hard validated tasks, effort ladder | $0.034Claude Code · effort high · eight hard validated tasks, effort ladder | 16 | none recorded | Unclear | No interval or range was recorded for either side, so the gap ($0.021 vs $0.034) is not tested against run-to-run variation. | Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks |
| Reasoning cost per strict pass by effort, with the total (calculation) (Reasoning cost per strict pass)Calculation | $0.0050Claude Code · effort low | $0.018Claude Code · effort high | 16 | none recorded | Unclear | More or fewer usd is not better or worse by itself; this row describes behaviour, not a winner. | How much of an AI bill is thinking? Reasoning tokens by model and effort |
| Reasoning cost per strict pass by effort, with the total (calculation) (Total cost per strict pass)Calculation | $0.021Claude Code · effort low | $0.034Claude Code · effort high | 16 | none recorded | Unclear | More or fewer usd is not better or worse by itself; this row describes behaviour, not a winner. | How much of an AI bill is thinking? Reasoning tokens by model and effort |
Marks: 95% intervals (Wilson for rates); fastest–slowest run ranges (not intervals)n is shown per side on every rowHollow marks: list-price calculations, not runs
14 rows from 3 studies. No row separates them: 3 ties, 11 unclear.
When to pick which
Only from the rows above. A tie is not a reason to pick either side.
When to pick Claude Opus 5.5 at low effort
- List-price cost per strict pass by effort (calculation): $0.021 vs $0.034. A list-price calculation, not a measured difference. Calculation
- Reasoning cost per strict pass by effort, with the total (calculation) (Reasoning cost per strict pass): $0.0050 vs $0.018. A list-price calculation, not a measured difference. Calculation
- Reasoning cost per strict pass by effort, with the total (calculation) (Total cost per strict pass): $0.021 vs $0.034. A list-price calculation, not a measured difference. Calculation
When to pick Claude Opus 5.5 at high effort
No row in this data puts Claude Opus 5.5 at high effort ahead of Claude Opus 5.5 at low effort. Pick on other grounds (price, access, the tasks you run), or measure your own workload.
Side by side
The study charts, showing only Claude Opus 5.5 at each effort it ran; the two compared settings are highlighted. Open a study for every configuration.
Every interval overlaps every other: this chart does not order these rows.
| Item | Pass rate | 95% interval | n |
|---|---|---|---|
| Claude Opus 5.5 (high) · Claude Code | 100% | 80%–100% | 15 |
| Claude Opus 5.5 · Claude Code | 100% | 80%–100% | 15 |
| Claude Opus 5.5 (low) · Claude Code | 100% | 80%–100% | 15 |
3 rows. All at 100%.
NotesWhiskers: 95% Wilson intervaln = 15 per row3 of 3 at 100%: this task set cannot separate them.
Every call counts; failures and timeouts are non-passes
Whiskers are 95% Wilson intervals. The tasks are short and most configurations pass nearly all of them, so pass rate does not separate the models here.
Source: Provider head-to-head: Claude Code models vs Codex efforts
Every lane’s run range overlaps another lane’s, so the chart shows no finish order.
| Item | Total time per call | Range (lowest–highest run) | n |
|---|---|---|---|
| Claude Opus 5.5 (high) · Claude Code | 2.7 s | 2.5 s–11.8 s | 15 |
| Claude Opus 5.5 · Claude Code | 2.8 s | 2.5 s–8.9 s | 15 |
| Claude Opus 5.5 (low) · Claude Code | 2.8 s | 2.4 s–6.6 s | 15 |
3 rows. Slowest Claude Opus 5.5 (low) · Claude Code 2.8 s (range 2.4 s–6.6 s, n 15). Fastest Claude Opus 5.5 (high) · Claude Code 2.7 s (range 2.5 s–11.8 s, n 15). All run ranges overlap.
NotesLines: fastest–slowest run (not an interval)n = 15 per row
Median per configuration; whiskers = fastest and slowest call
One host, one network, one day. Whiskers are a range, not a confidence interval.
Source: Provider head-to-head: Claude Code models vs Codex efforts
Every lane’s run range overlaps another lane’s, so the chart shows no finish order.
| Item | Time to first useful output | Range (lowest–highest run) | n |
|---|---|---|---|
| Claude Opus 5.5 (high) · Claude Code | 2 s | 1.4 s–9.9 s | 15 |
| Claude Opus 5.5 · Claude Code | 1.9 s | 1.6 s–7.2 s | 15 |
| Claude Opus 5.5 (low) · Claude Code | 2.4 s | 1.5 s–4.9 s | 15 |
3 rows. Slowest Claude Opus 5.5 (low) · Claude Code 2.4 s (range 1.5 s–4.9 s, n 15). Fastest Claude Opus 5.5 · Claude Code 1.9 s (range 1.6 s–7.2 s, n 15). All run ranges overlap.
NotesLines: fastest–slowest run (not an interval)n = 15 per row
Median per configuration; whiskers = fastest and slowest call
One host, one network, one day. Whiskers are a range, not a confidence interval.
Source: Provider head-to-head: Claude Code models vs Codex efforts
- Cache read
- Other input
Totals are the sum of the parts shown. Shares are calculated from the same values.
| Item | Cache read | Other input | n |
|---|---|---|---|
| Claude Opus 5.5 (high) · Claude Code | 1,463 | 619 | 15 |
| Claude Opus 5.5 · Claude Code | 1,401 | 680 | 15 |
| Claude Opus 5.5 (low) · Claude Code | 1,463 | 618 | 15 |
3 rows, 2 series: Cache read, Other input. Cache read: highest Claude Opus 5.5 (high) · Claude Code 1,463 (n 15). Lowest Claude Opus 5.5 · Claude Code 1,401 (n 15). Other input: highest Claude Opus 5.5 · Claude Code 680 (n 15). Lowest Claude Opus 5.5 (low) · Claude Code 618 (n 15).
Notesn = 15 per row
Mean per call, split into prompt-cache reads and other input
The prompts are a few hundred tokens; most input is the CLI’s own system prompt and tool context. Input counts include cache reads, as the vendors report them.
Source: Provider head-to-head: Claude Code models vs Codex efforts
| Item | Output tokens | n |
|---|---|---|
| Claude Opus 5.5 (high) · Claude Code | 78 | 15 |
| Claude Opus 5.5 · Claude Code | 64 | 15 |
| Claude Opus 5.5 (low) · Claude Code | 64 | 15 |
3 rows. Highest Claude Opus 5.5 (high) · Claude Code 78 (n 15). Lowest Claude Opus 5.5 (low) · Claude Code 64 (n 15).
Notesn = 15 per row
Median per configuration; reasoning tokens where the CLI reports them
Vendors count reasoning differently; compare within a vendor first. A median of 0 can mean the CLI did not report reasoning for most calls.
Source: Provider head-to-head: Claude Code models vs Codex efforts
Every interval overlaps every other: this chart does not order these rows.
| Item | Cost per call | Range (lowest–highest run) | n |
|---|---|---|---|
| Claude Opus 5.5 (high) · Claude Code | $0.0069 | $0.0059–$0.027 | 15 |
| Claude Opus 5.5 · Claude Code | $0.0069 | $0.0059–$0.022 | 15 |
| Claude Opus 5.5 (low) · Claude Code | $0.0069 | $0.0058–$0.018 | 15 |
List-price calculation, not a run. 3 rows. Highest Claude Opus 5.5 (high) · Claude Code $0.0069 (range $0.0059–$0.027, n 15). Lowest Claude Opus 5.5 (low) · Claude Code $0.0069 (range $0.0058–$0.018, n 15). All run ranges overlap.
NotesLines: lowest–highest run (not an interval)n = 15 per row
Reported tokens × list price; the calls ran on subscriptions
Calculation, not a bill: the calls ran on flat subscriptions. Whiskers = cheapest and most expensive call.
Sources: Provider head-to-head: Claude Code models vs Codex efforts, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices
Hover or focus a bar for its ratio to Claude Opus 5.5 (low) (the lowest value): a ratio of list-price calculations, not a measurement.
| Item | Cost per pass | n |
|---|---|---|
| Claude Opus 5.5 (low) · Claude Code | $0.0083 | 15 |
| Claude Opus 5.5 · Claude Code | $0.01 | 15 |
| Claude Opus 5.5 (high) · Claude Code | $0.01 | 15 |
List-price calculation, not a run. 3 rows. Highest Claude Opus 5.5 (high) · Claude Code $0.01 (n 15). Lowest Claude Opus 5.5 (low) · Claude Code $0.0083 (n 15).
Notesn = 15 per row
All calls in a configuration, failures included, divided by its passes
Calculation, not a bill: reported tokens × list price; the calls ran on flat subscriptions. A failed call still costs, so a lower pass rate raises the cost per pass. Highlighted bars are on the speed/cost/quality frontier.
Sources: Provider head-to-head: Claude Code models vs Codex efforts, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices
Every interval overlaps every other: this chart does not order these rows.
| Item | Strict pass | 95% interval | n |
|---|---|---|---|
| Claude Opus 5.5 (low) · Claude Code | 100% | 81%–100% | 16 |
| Claude Opus 5.5 (medium) · Claude Code | 100% | 81%–100% | 16 |
| Claude Opus 5.5 (high) · Claude Code | 100% | 81%–100% | 16 |
| Claude Opus 5.5 · Claude Code | 100% | 81%–100% | 16 |
4 rows. All at 100%.
NotesWhiskers: 95% Wilson intervaln = 16 per row4 of 4 at 100%: this task set cannot separate them.
Every configuration: 8 tasks × 2 repetitions. Whiskers are 95% Wilson intervals
Whiskers are 95% Wilson intervals. A format miss never counts as a pass. Reference cells (efforts the hard head-to-head already ran) are reused, not rerun; Claude reference cells keep repetitions 1-2, so every cell is 8 tasks × 2 repetitions.
Source: Effort ladder: the hard task set at each effort level
Every lane’s run range overlaps another lane’s, so the chart shows no finish order.
| Item | Total time per call by effort on hard tasks | Range (lowest–highest run) | n |
|---|---|---|---|
| Claude Opus 5.5 (low) · Claude Code | 7.5 s | 3.3 s–15.8 s | 16 |
| Claude Opus 5.5 (medium) · Claude Code | 9.7 s | 4.8 s–31.4 s | 16 |
| Claude Opus 5.5 (high) · Claude Code | 10.1 s | 3.6 s–63 s | 16 |
| Claude Opus 5.5 · Claude Code | 9.2 s | 4.2 s–27.2 s | 16 |
4 rows. Slowest Claude Opus 5.5 (high) · Claude Code 10.1 s (range 3.6 s–63 s, n 16). Fastest Claude Opus 5.5 (low) · Claude Code 7.5 s (range 3.3 s–15.8 s, n 16). All run ranges overlap.
NotesLines: fastest–slowest run (not an interval)n = 16 per row
Median per configuration; whiskers = fastest and slowest call
Whiskers are a range (fastest and slowest call), not a confidence interval. One host, one network. Reference cells (efforts the hard head-to-head already ran) are reused, not rerun; Claude reference cells keep repetitions 1-2, so every cell is 8 tasks × 2 repetitions. The reference cells ran in a different hour.
Source: Effort ladder: the hard task set at each effort level
| Item | Output tokens | n |
|---|---|---|
| Claude Opus 5.5 (low) · Claude Code | 594 | 16 |
| Claude Opus 5.5 (medium) · Claude Code | 853 | 16 |
| Claude Opus 5.5 (high) · Claude Code | 1,052 | 16 |
| Claude Opus 5.5 · Claude Code | 945 | 16 |
4 rows. Highest Claude Opus 5.5 (high) · Claude Code 1,052 (n 16). Lowest Claude Opus 5.5 (low) · Claude Code 594 (n 16).
Notesn = 16 per row
Median per configuration; reasoning tokens as the CLI reports them
Reasoning tokens are part of the output tokens where the CLI reports them; 0 can mean "not reported". Their content is never captured. More tokens is not better or worse by itself.
Source: Effort ladder: the hard task set at each effort level
Hover or focus a bar for its ratio to Claude Opus 5.5 (low) (the lowest value): a ratio of list-price calculations, not a measurement.
| Item | Cost per strict pass | n |
|---|---|---|
| Claude Opus 5.5 (low) · Claude Code | $0.021 | 16 |
| Claude Opus 5.5 (medium) · Claude Code | $0.029 | 16 |
| Claude Opus 5.5 (high) · Claude Code | $0.034 | 16 |
| Claude Opus 5.5 · Claude Code | $0.029 | 16 |
List-price calculation, not a run. 4 rows. Highest Claude Opus 5.5 (high) · Claude Code $0.034 (n 16). Lowest Claude Opus 5.5 (low) · Claude Code $0.021 (n 16).
Notesn = 16 per row
All calls in a configuration divided by its strict passes
Calculation, not a bill: reported tokens × list price, cache reads and writes priced as in the hard head-to-head; the calls ran on flat subscriptions. Codex input includes its own system prompt and tool schemas, so a Claude-vs-Codex gap is partly the CLI.
Sources: Effort ladder: the hard task set at each effort level, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices
- Reasoning cost per strict pass
- Total cost per strict pass (square)
Gap labels, Total cost per strict pass vs Reasoning cost per strict pass: Total cost per strict pass is x% higher (+) or lower (−) than Reasoning cost per strict pass, calculated from the two values shown (the change counted from Reasoning cost per strict pass’s value).
| Item | Reasoning cost per strict pass | Total cost per strict pass | n |
|---|---|---|---|
| Claude Opus 5.5 (low) · Claude Code | $0.005 | $0.021 | 16 |
| Claude Opus 5.5 (medium) · Claude Code | $0.013 | $0.029 | 16 |
| Claude Opus 5.5 (high) · Claude Code | $0.018 | $0.034 | 16 |
| Claude Opus 5.5 · Claude Code | $0.013 | $0.029 | 16 |
List-price calculation, not a run. 4 rows, 2 series: Reasoning cost per strict pass, Total cost per strict pass. Reasoning cost per strict pass: highest Claude Opus 5.5 (high) · Claude Code $0.018 (n 16). Lowest Claude Opus 5.5 (low) · Claude Code $0.005 (n 16). Total cost per strict pass: highest Claude Opus 5.5 (high) · Claude Code $0.034 (n 16). Lowest Claude Opus 5.5 (low) · Claude Code $0.021 (n 16).
Notesn = 16 per row
List price ÷ strict passes; every cell is 8 tasks × 2 repetitions
Calculation, not a bill: reported tokens × list price, divided by the cell's strict passes; the calls ran on flat subscriptions. The effort-ladder cells: new calls plus reference cells reused from the hard head-to-head (Claude repetitions 1-2 only). "Default" means the effort flag was not passed. The total is the same value as the effort-ladder cost-per-pass chart. Effort levels are not the same scale across vendors, and the reference cells ran in a different batch and hour.
Sources: Reasoning token bill (calculation), Provider head-to-head, hard set: eight hard tasks with strict validators, Effort ladder: the hard task set at each effort level, Provider head-to-head: Claude Code models vs Codex efforts, Anthropic list prices (Claude models), OpenAI list prices
How a row is called
- AheadThe 95% intervals do not overlap, or the run ranges or p50–p95 bands do not overlap with at least 5 runs per side.
- TieThe values match, or both sit at the same ceiling.
- UnclearThe intervals or ranges overlap, too few runs were recorded, no interval was recorded, or more is not better (token counts are never a win).
- CalculationDerived from list prices and recorded counts. Not a bill and not a run.
The dataset compiler makes every call; this page only draws it. No composite score, no rank.
Questions
- Is Claude Opus 5.5 better at low effort or high effort?
- Claude Opus 5.5 at low effort and Claude Opus 5.5 at high effort share 9 measured metrics and 5 list-price calculations from 3 studies. Only the effort setting differs between the two sides of a row; the route and the task set are the same. No row separates them: every interval or run range overlaps, too few runs were recorded, no interval was recorded, or more is not better for that metric. The rows are 3 ties and 11 unclear; each row says why. Calculation rows are derived from list prices and recorded counts; they are not bills or runs. Some rows rest on small samples (n = 15 at the smallest).
- How were the two effort settings measured?
- They share 9 measured metrics and 5 list-price calculations from 3 public studies: Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head; Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks; How much of an AI bill is thinking? Reasoning tokens by model and effort. Only the effort setting differs between the two sides of a row.
- How does pass rate on five validated tasks change from low effort to high effort?
- Claude Opus 5.5 at low effort: 100% (15/15) (n = 15, 95% interval 80% to 100%). Claude Opus 5.5 at high effort: 100% (15/15) (n = 15, 95% interval 80% to 100%). The 95% intervals overlap (Claude Opus 5.5 at low effort 80% to 100%; Claude Opus 5.5 at high effort 80% to 100%), so this sample cannot separate them.
- How does total time per call change from low effort to high effort?
- Claude Opus 5.5 at low effort: 2.83 s (n = 15, run range 2.4 s to 6.6 s). Claude Opus 5.5 at high effort: 2.71 s (n = 15, run range 2.5 s to 11.8 s). The run ranges (fastest to slowest) overlap (Claude Opus 5.5 at low effort 2.35 s to 6.62 s; Claude Opus 5.5 at high effort 2.45 s to 11.8 s); the medians alone do not show a reliable difference. A range is not a confidence interval.
- How does time to first useful output change from low effort to high effort?
- Claude Opus 5.5 at low effort: 2.39 s (n = 15, run range 1.5 s to 4.9 s). Claude Opus 5.5 at high effort: 2.04 s (n = 15, run range 1.4 s to 9.9 s). The run ranges (fastest to slowest) overlap (Claude Opus 5.5 at low effort 1.45 s to 4.90 s; Claude Opus 5.5 at high effort 1.40 s to 9.94 s); the medians alone do not show a reliable difference. A range is not a confidence interval.
- How does strict pass rate by effort on eight hard tasks change from low effort to high effort?
- Claude Opus 5.5 at low effort: 100% (16/16) (n = 16, 95% interval 81% to 100%). Claude Opus 5.5 at high effort: 100% (16/16) (n = 16, 95% interval 81% to 100%). The 95% intervals overlap (Claude Opus 5.5 at low effort 81% to 100%; Claude Opus 5.5 at high effort 81% to 100%), so this sample cannot separate them.
The studies behind this page
Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head
130 timed calls on five validated tasks: Claude Haiku, Sonnet, Opus and Fable against Codex GPT-6.1 Sol. Pass rate, speed, tokens, cost per pass.
Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks
176 calls on 8 hard tasks at low, medium, high and default effort: Sonnet, Opus and GPT-6.1 Sol. Pass rate with 95% intervals, time, tokens, cost per pass.
How much of an AI bill is thinking? Reasoning tokens by model and effort
Reasoning tokens in 378 recorded calls by model and effort: share of output, list-price cost per call and per pass, and time. Calculations.