14 measured metrics · 12 calculated · 4 studies
Claude Opus 5.5highvsdefaulteffort
“default”: the effort flag was not passed and the CLI chose.
No row separates them: 4 ties, 22 unclear.
The verdict
Claude Opus 5.5 at high effort and Claude Opus 5.5 at default effort share 14 measured metrics and 12 list-price calculations from 4 studies. Only the effort setting differs between the two sides of a row; the route and the task set are the same. "default" means the effort flag was not passed, so its level is the CLI’s choice. No row separates them: every interval or run range overlaps, too few runs were recorded, no interval was recorded, or more is not better for that metric. The rows are 4 ties and 22 unclear; each row says why. Calculation rows are derived from list prices and recorded counts; they are not bills or runs. Some rows rest on small samples (n = 15 at the smallest).
Headline metrics
How far apart the two sides are on the headline metrics. Length is the ratio of the two values; it is not a winner.
Bar length is the ratio of the two values on a log scale, pointing to the larger one. Larger is not better for time, tokens or cost. A bar has a side’s color only when that side is ahead in the data; gray means the data does not separate them.
| Metric | Claude Opus 5.5 at high effort | Claude Opus 5.5 at default effort | n | Interval or range | Outcome | Basis | Study |
|---|---|---|---|---|---|---|---|
| Pass rate on five validated tasks | 100% (15/15)Claude Code · effort high · five short validated tasks | 100% (15/15)Claude Code · five short validated tasks | 15 | 95% CI: 80%–100% vs 80%–100% | Tie | The 95% intervals overlap (Claude Opus 5.5 at high effort 80% to 100%; Claude Opus 5.5 at default effort 80% to 100%), so this sample cannot separate them. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Total time per call | 2.71 sClaude Code · effort high · five short validated tasks | 2.75 sClaude Code · five short validated tasks | 15 | range: 2.5 s–11.8 s vs 2.5 s–8.9 s | Unclear | The run ranges (fastest to slowest) overlap (Claude Opus 5.5 at high effort 2.45 s to 11.8 s; Claude Opus 5.5 at default effort 2.47 s to 8.91 s); the medians alone do not show a reliable difference. A range is not a confidence interval. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Time to first useful output | 2.04 sClaude Code · effort high · five short validated tasks | 1.92 sClaude Code · five short validated tasks | 15 | range: 1.4 s–9.9 s vs 1.6 s–7.2 s | Unclear | The run ranges (fastest to slowest) overlap (Claude Opus 5.5 at high effort 1.40 s to 9.94 s; Claude Opus 5.5 at default effort 1.56 s to 7.23 s); the medians alone do not show a reliable difference. A range is not a confidence interval. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Input tokens per call: what the CLI sends (Cache read) | 1,463Claude Code · effort high · five short validated tasks | 1,401Claude Code · five short validated tasks | 15 | none recorded | Unclear | More or fewer tokens is not better or worse by itself; this row describes behaviour, not a winner. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Output tokens per call (Output tokens) | 78Claude Code · effort high · five short validated tasks | 64Claude Code · five short validated tasks | 15 | none recorded | Unclear | More or fewer tokens is not better or worse by itself; this row describes behaviour, not a winner. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| List-price cost per passing answer (calculation)Calculation | $0.010Claude Code · effort high · five short validated tasks | $0.010Claude Code · five short validated tasks | 15 | none recorded | Unclear | No interval or range was recorded for either side, so the gap ($0.010 vs $0.010) is not tested against run-to-run variation. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
Marks: 95% intervals (Wilson for rates); fastest–slowest run ranges (not intervals)n is shown per side on every rowHollow marks: list-price calculations, not runs
6 headline metrics as the ratio of the two values. None of them separates the sides in the data. Widest ratio: Output tokens per call (Output tokens), 1.2x (High effort larger).
Watch it build
A live story drawn from the same dataset in your browser. It shows the same numbers, notes and interval labels as this page.
Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on eight hard tasks
All 11 configurations passed 16/16 strictly (95% interval 81%–100%): more effort bought no extra passes on this set (a ceiling). Median output tokens rose with effort; time ranges overlap. Costs are list-price calculations.
Transcript
- Effort ladder · 176 calls · 11 configurations. Does more effort buy quality? Low, medium and high effort on eight hard tasks. Sonnet 5.5 and Opus 5.5 via Claude Code, GPT-6.1 Sol via Codex CLI.
- 176 of 176 calls passed strictly, at every effort level. No format misses and no wrong answers. Calls that passed strictly, every effort: 100% (176/176) (n = 176, 95% CI 98–100%). Model and effort configurations: 11 (6 new, 5 reference) (n = 176). Format misses and wrong answers: 0 · 0 (n = 176, format · wrong). Caveat: Reference cells ran in a different batch and hour than the new cells (provider load can differ), on the same host, CLI versions, cases and flags.
- All 11 configurations passed 16/16. A 16/16 has a 95% interval of 81%–100%: the set has a ceiling, so no effort level wins. Chart: Strict pass rate by effort · 95% Wilson intervals (n = 16 each). Caveat: Every cell passed every call, so the set has a ceiling: a perfect 16/16 has a 95% interval of 81% to 100%. This study cannot show that effort does not matter on harder work; it shows that these 8 tasks do not need more than low effort.
- Median time: Sonnet 5.5 5.8 s at low, 8.8 s at high; Opus 5.5 7.5 s at low, 10.1 s at high; GPT-6.1 Sol 13.6 s at low, 18.1 s at high. Within each model the ranges overlap: no effort is ranked faster. Chart: Median total time per call · whiskers: fastest to slowest call (n = 16 each). Caveat: Each cell has only 2 calls per task. Medians of 16 calls move with a few slow calls; the ranges are wide and overlap.
- More effort writes more: Sonnet 5.5 used a median 667 / 770 / 1,192 output tokens at low / medium / high. Chart: Median output and reasoning tokens per call (n = 16 each). Caveat: Claude Code and Codex CLI rows pair a CLI with a model; each CLI adds its own system prompt and start-up time. Effort levels are not the same scale across vendors.
- List-price cost per strict pass, low → high effort, a calculation: Sonnet $0.0122 → $0.0167, Opus $0.0212 → $0.0337, GPT-6.1 Sol $0.0128 → $0.0151. Chart: List-price cost per strict pass · calculation (n = 16 each). Calculation, not a run. Caveat: List-price costs are calculations; the calls used flat subscriptions.
- 16 effort pairs, 172 comparison rows: no effort level wins any row. The rest are ties or unclear. Effort pairs compared (same model, same route): 16. Comparison rows across those pairs: 172. Rows where one effort level wins: 0 (of 172). Caveat: Every cell passed every call, so the set has a ceiling: a perfect 16/16 has a 95% interval of 81% to 100%. This study cannot show that effort does not matter on harder work; it shows that these 8 tasks do not need more than low effort.
- Start at low effort and measure. Every call, interval and cost online.
Metric by metric
Both values of a row come from the same chart in the same study. Each row has its own axis. The shaded band is where the two intervals or ranges overlap: a side is ahead only when they do not.
- High effort
- Default effort
- 95% interval
- fastest–slowest run (not an interval)
- where the two overlap
- hollow: list-price calculation
Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head
- Pass rate on five validated tasks100% (15/15)n 15100% (15/15)n 15TiePass rate on five validated tasks: High effort 100% (15/15) (n 15, 95% interval 80%–100%); Default effort 100% (15/15) (n 15, 95% interval 80%–100%). Tie.
- Total time per call2.71 sn 152.75 sn 15UnclearTotal time per call: High effort 2.71 s (n 15, run range 2.5 s–11.8 s); Default effort 2.75 s (n 15, run range 2.5 s–8.9 s). Unclear.
- Time to first useful output2.04 sn 151.92 sn 15UnclearTime to first useful output: High effort 2.04 s (n 15, run range 1.4 s–9.9 s); Default effort 1.92 s (n 15, run range 1.6 s–7.2 s). Unclear.
- Input tokens per call: what the CLI sends (Cache read)1,463n 151,401n 15UnclearInput tokens per call: what the CLI sends (Cache read): High effort 1,463 (n 15); Default effort 1,401 (n 15). Unclear.
- Input tokens per call: what the CLI sends (Other input)619n 15680n 15UnclearInput tokens per call: what the CLI sends (Other input): High effort 619 (n 15); Default effort 680 (n 15). Unclear.
- Output tokens per call (Output tokens)78n 1564n 15UnclearOutput tokens per call (Output tokens): High effort 78 (n 15); Default effort 64 (n 15). Unclear.
- List-price cost per call (calculation)Calculation$0.0069n 15$0.0069n 15UnclearList-price cost per call (calculation), calculation: High effort $0.0069 (n 15, run range $0.0059–$0.027); Default effort $0.0069 (n 15, run range $0.0059–$0.022). Unclear.
- List-price cost per passing answer (calculation)Calculation$0.010n 15$0.010n 15UnclearList-price cost per passing answer (calculation), calculation: High effort $0.010 (n 15); Default effort $0.010 (n 15). Unclear.
Haiku vs Sonnet vs Opus vs Fable vs GPT-6.1 Sol on 8 hard tasks
- Pass rate on eight hard tasks (Strict pass)100% (24/24)n 24100% (24/24)n 24TiePass rate on eight hard tasks (Strict pass): High effort 100% (24/24) (n 24, 95% interval 86%–100%); Default effort 100% (24/24) (n 24, 95% interval 86%–100%). Tie.
- Pass rate on eight hard tasks (Lenient (format misses counted))100% (24/24)n 24100% (24/24)n 24TiePass rate on eight hard tasks (Lenient (format misses counted)): High effort 100% (24/24) (n 24, 95% interval 86%–100%); Default effort 100% (24/24) (n 24, 95% interval 86%–100%). Tie.
- Total time per call on hard tasks (separate batches)11.0 sn 249.18 sn 24UnclearTotal time per call on hard tasks (separate batches): High effort 11.0 s (n 24, run range 3.6 s–63 s); Default effort 9.18 s (n 24, run range 4.2 s–27.2 s). Unclear.
- Time to first useful output on hard tasks7.13 sn 246.78 sn 24UnclearTime to first useful output on hard tasks: High effort 7.13 s (n 24, run range 2.2 s–56.2 s); Default effort 6.78 s (n 24, run range 2.4 s–21.8 s). Unclear.
- Output tokens per call on hard tasks (Output tokens)1,052n 24945n 24UnclearOutput tokens per call on hard tasks (Output tokens): High effort 1,052 (n 24); Default effort 945 (n 24). Unclear.
- List-price cost per strict pass on hard tasks (calculation)Calculation$0.033n 24$0.028n 24UnclearList-price cost per strict pass on hard tasks (calculation), calculation: High effort $0.033 (n 24); Default effort $0.028 (n 24). Unclear.
Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks
- Strict pass rate by effort on eight hard tasks100% (16/16)n 16100% (16/16)n 16TieStrict pass rate by effort on eight hard tasks: High effort 100% (16/16) (n 16, 95% interval 81%–100%); Default effort 100% (16/16) (n 16, 95% interval 81%–100%). Tie.
- Total time per call by effort on hard tasks10.1 sn 169.18 sn 16UnclearTotal time per call by effort on hard tasks: High effort 10.1 s (n 16, run range 3.6 s–63 s); Default effort 9.18 s (n 16, run range 4.2 s–27.2 s). Unclear.
- Output tokens per call by effort on hard tasks (Output tokens)1,052n 16945n 16UnclearOutput tokens per call by effort on hard tasks (Output tokens): High effort 1,052 (n 16); Default effort 945 (n 16). Unclear.
- List-price cost per strict pass by effort (calculation)Calculation$0.034n 16$0.029n 16UnclearList-price cost per strict pass by effort (calculation), calculation: High effort $0.034 (n 16); Default effort $0.029 (n 16). Unclear.
How much of an AI bill is thinking? Reasoning tokens by model and effort
- Reasoning share of output tokens per call on hard tasks (calculation)Calculation54.4%n 2454.8%n 24UnclearReasoning share of output tokens per call on hard tasks (calculation), calculation: High effort 54.4% (n 24, run range 36%–96%); Default effort 54.8% (n 24, run range 30%–96%). Unclear.
- List-price cost per call: reasoning, remaining output and input (calculation) (Reasoning (output tokens))Calculation$0.018n 24$0.013n 24UnclearList-price cost per call: reasoning, remaining output and input (calculation) (Reasoning (output tokens)), calculation: High effort $0.018 (n 24); Default effort $0.013 (n 24). Unclear.
- List-price cost per call: reasoning, remaining output and input (calculation) (Remaining output (visible-answer estimate))Calculation$0.0080n 24$0.0080n 24UnclearList-price cost per call: reasoning, remaining output and input (calculation) (Remaining output (visible-answer estimate)), calculation: High effort $0.0080 (n 24); Default effort $0.0080 (n 24). Unclear.
- List-price cost per call: reasoning, remaining output and input (calculation) (Input (prompt, cache priced))Calculation$0.0074n 24$0.0077n 24UnclearList-price cost per call: reasoning, remaining output and input (calculation) (Input (prompt, cache priced)), calculation: High effort $0.0074 (n 24); Default effort $0.0077 (n 24). Unclear.
- Reasoning cost per strict pass by effort, with the total (calculation) (Reasoning cost per strict pass)Calculation$0.018n 16$0.013n 16UnclearReasoning cost per strict pass by effort, with the total (calculation) (Reasoning cost per strict pass), calculation: High effort $0.018 (n 16); Default effort $0.013 (n 16). Unclear.
- Reasoning cost per strict pass by effort, with the total (calculation) (Total cost per strict pass)Calculation$0.034n 16$0.029n 16UnclearReasoning cost per strict pass by effort, with the total (calculation) (Total cost per strict pass), calculation: High effort $0.034 (n 16); Default effort $0.029 (n 16). Unclear.
- Reasoning share on short tasks vs hard tasks (calculation) (Eight hard tasks)Calculation54.4%n 2454.8%n 24UnclearReasoning share on short tasks vs hard tasks (calculation) (Eight hard tasks), calculation: High effort 54.4% (n 24, run range 36%–96%); Default effort 54.8% (n 24, run range 30%–96%). Unclear.
- Reasoning share on short tasks vs hard tasks (calculation) (Five short tasks)Calculation43.6%n 150%n 15UnclearReasoning share on short tasks vs hard tasks (calculation) (Five short tasks), calculation: High effort 43.6% (n 15, run range 0%–93%); Default effort 0% (n 15, run range 0%–93%). Unclear.
| Metric | Claude Opus 5.5 at high effort | Claude Opus 5.5 at default effort | n | Interval or range | Outcome | Basis | Study |
|---|---|---|---|---|---|---|---|
| Pass rate on five validated tasks | 100% (15/15)Claude Code · effort high · five short validated tasks | 100% (15/15)Claude Code · five short validated tasks | 15 | 95% CI: 80%–100% vs 80%–100% | Tie | The 95% intervals overlap (Claude Opus 5.5 at high effort 80% to 100%; Claude Opus 5.5 at default effort 80% to 100%), so this sample cannot separate them. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Total time per call | 2.71 sClaude Code · effort high · five short validated tasks | 2.75 sClaude Code · five short validated tasks | 15 | range: 2.5 s–11.8 s vs 2.5 s–8.9 s | Unclear | The run ranges (fastest to slowest) overlap (Claude Opus 5.5 at high effort 2.45 s to 11.8 s; Claude Opus 5.5 at default effort 2.47 s to 8.91 s); the medians alone do not show a reliable difference. A range is not a confidence interval. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Time to first useful output | 2.04 sClaude Code · effort high · five short validated tasks | 1.92 sClaude Code · five short validated tasks | 15 | range: 1.4 s–9.9 s vs 1.6 s–7.2 s | Unclear | The run ranges (fastest to slowest) overlap (Claude Opus 5.5 at high effort 1.40 s to 9.94 s; Claude Opus 5.5 at default effort 1.56 s to 7.23 s); the medians alone do not show a reliable difference. A range is not a confidence interval. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Input tokens per call: what the CLI sends (Cache read) | 1,463Claude Code · effort high · five short validated tasks | 1,401Claude Code · five short validated tasks | 15 | none recorded | Unclear | More or fewer tokens is not better or worse by itself; this row describes behaviour, not a winner. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Input tokens per call: what the CLI sends (Other input) | 619Claude Code · effort high · five short validated tasks | 680Claude Code · five short validated tasks | 15 | none recorded | Unclear | More or fewer tokens is not better or worse by itself; this row describes behaviour, not a winner. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Output tokens per call (Output tokens) | 78Claude Code · effort high · five short validated tasks | 64Claude Code · five short validated tasks | 15 | none recorded | Unclear | More or fewer tokens is not better or worse by itself; this row describes behaviour, not a winner. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| List-price cost per call (calculation)Calculation | $0.0069Claude Code · effort high · five short validated tasks | $0.0069Claude Code · five short validated tasks | 15 | range: $0.0059–$0.027 vs $0.0059–$0.022 | Unclear | The run ranges (fastest to slowest) overlap (Claude Opus 5.5 at high effort $0.0059 to $0.027; Claude Opus 5.5 at default effort $0.0059 to $0.022); the medians alone do not show a reliable difference. A range is not a confidence interval. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| List-price cost per passing answer (calculation)Calculation | $0.010Claude Code · effort high · five short validated tasks | $0.010Claude Code · five short validated tasks | 15 | none recorded | Unclear | No interval or range was recorded for either side, so the gap ($0.010 vs $0.010) is not tested against run-to-run variation. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Pass rate on eight hard tasks (Strict pass) | 100% (24/24)Claude Code · effort high · eight hard validated tasks | 100% (24/24)Claude Code · eight hard validated tasks | 24 | 95% CI: 86%–100% vs 86%–100% | Tie | The 95% intervals overlap (Claude Opus 5.5 at high effort 86% to 100%; Claude Opus 5.5 at default effort 86% to 100%), so this sample cannot separate them. | Haiku vs Sonnet vs Opus vs Fable vs GPT-6.1 Sol on 8 hard tasks |
| Pass rate on eight hard tasks (Lenient (format misses counted)) | 100% (24/24)Claude Code · effort high · eight hard validated tasks | 100% (24/24)Claude Code · eight hard validated tasks | 24 | 95% CI: 86%–100% vs 86%–100% | Tie | The 95% intervals overlap (Claude Opus 5.5 at high effort 86% to 100%; Claude Opus 5.5 at default effort 86% to 100%), so this sample cannot separate them. | Haiku vs Sonnet vs Opus vs Fable vs GPT-6.1 Sol on 8 hard tasks |
| Total time per call on hard tasks (separate batches) | 11.0 sClaude Code · effort high · eight hard validated tasks | 9.18 sClaude Code · eight hard validated tasks | 24 | range: 3.6 s–63 s vs 4.2 s–27.2 s | Unclear | The run ranges (fastest to slowest) overlap (Claude Opus 5.5 at high effort 3.63 s to 63.0 s; Claude Opus 5.5 at default effort 4.24 s to 27.2 s); the medians alone do not show a reliable difference. A range is not a confidence interval. | Haiku vs Sonnet vs Opus vs Fable vs GPT-6.1 Sol on 8 hard tasks |
| Time to first useful output on hard tasks | 7.13 sClaude Code · effort high · eight hard validated tasks | 6.78 sClaude Code · eight hard validated tasks | 24 | range: 2.2 s–56.2 s vs 2.4 s–21.8 s | Unclear | The run ranges (fastest to slowest) overlap (Claude Opus 5.5 at high effort 2.15 s to 56.2 s; Claude Opus 5.5 at default effort 2.39 s to 21.8 s); the medians alone do not show a reliable difference. A range is not a confidence interval. | Haiku vs Sonnet vs Opus vs Fable vs GPT-6.1 Sol on 8 hard tasks |
| Output tokens per call on hard tasks (Output tokens) | 1,052Claude Code · effort high · eight hard validated tasks | 945Claude Code · eight hard validated tasks | 24 | none recorded | Unclear | More or fewer tokens is not better or worse by itself; this row describes behaviour, not a winner. | Haiku vs Sonnet vs Opus vs Fable vs GPT-6.1 Sol on 8 hard tasks |
| List-price cost per strict pass on hard tasks (calculation)Calculation | $0.033Claude Code · effort high · eight hard validated tasks | $0.028Claude Code · eight hard validated tasks | 24 | none recorded | Unclear | No interval or range was recorded for either side, so the gap ($0.033 vs $0.028) is not tested against run-to-run variation. | Haiku vs Sonnet vs Opus vs Fable vs GPT-6.1 Sol on 8 hard tasks |
| Strict pass rate by effort on eight hard tasks | 100% (16/16)Claude Code · effort high · eight hard validated tasks, effort ladder | 100% (16/16)Claude Code · eight hard validated tasks, effort ladder | 16 | 95% CI: 81%–100% vs 81%–100% | Tie | The 95% intervals overlap (Claude Opus 5.5 at high effort 81% to 100%; Claude Opus 5.5 at default effort 81% to 100%), so this sample cannot separate them. | Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks |
| Total time per call by effort on hard tasks | 10.1 sClaude Code · effort high · eight hard validated tasks, effort ladder | 9.18 sClaude Code · eight hard validated tasks, effort ladder | 16 | range: 3.6 s–63 s vs 4.2 s–27.2 s | Unclear | The run ranges (fastest to slowest) overlap (Claude Opus 5.5 at high effort 3.63 s to 63.0 s; Claude Opus 5.5 at default effort 4.24 s to 27.2 s); the medians alone do not show a reliable difference. A range is not a confidence interval. | Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks |
| Output tokens per call by effort on hard tasks (Output tokens) | 1,052Claude Code · effort high · eight hard validated tasks, effort ladder | 945Claude Code · eight hard validated tasks, effort ladder | 16 | none recorded | Unclear | More or fewer tokens is not better or worse by itself; this row describes behaviour, not a winner. | Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks |
| List-price cost per strict pass by effort (calculation)Calculation | $0.034Claude Code · effort high · eight hard validated tasks, effort ladder | $0.029Claude Code · eight hard validated tasks, effort ladder | 16 | none recorded | Unclear | No interval or range was recorded for either side, so the gap ($0.034 vs $0.029) is not tested against run-to-run variation. | Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks |
| Reasoning share of output tokens per call on hard tasks (calculation)Calculation | 54.4%Claude Code · effort high | 54.8%Claude Code | 24 | range: 36%–96% vs 30%–96% | Unclear | More or fewer percent is not better or worse by itself; this row describes behaviour, not a winner. | How much of an AI bill is thinking? Reasoning tokens by model and effort |
| List-price cost per call: reasoning, remaining output and input (calculation) (Reasoning (output tokens))Calculation | $0.018Claude Code · effort high | $0.013Claude Code | 24 | none recorded | Unclear | More or fewer usd is not better or worse by itself; this row describes behaviour, not a winner. | How much of an AI bill is thinking? Reasoning tokens by model and effort |
| List-price cost per call: reasoning, remaining output and input (calculation) (Remaining output (visible-answer estimate))Calculation | $0.0080Claude Code · effort high | $0.0080Claude Code | 24 | none recorded | Unclear | More or fewer usd is not better or worse by itself; this row describes behaviour, not a winner. | How much of an AI bill is thinking? Reasoning tokens by model and effort |
| List-price cost per call: reasoning, remaining output and input (calculation) (Input (prompt, cache priced))Calculation | $0.0074Claude Code · effort high | $0.0077Claude Code | 24 | none recorded | Unclear | More or fewer usd is not better or worse by itself; this row describes behaviour, not a winner. | How much of an AI bill is thinking? Reasoning tokens by model and effort |
| Reasoning cost per strict pass by effort, with the total (calculation) (Reasoning cost per strict pass)Calculation | $0.018Claude Code · effort high | $0.013Claude Code | 16 | none recorded | Unclear | More or fewer usd is not better or worse by itself; this row describes behaviour, not a winner. | How much of an AI bill is thinking? Reasoning tokens by model and effort |
| Reasoning cost per strict pass by effort, with the total (calculation) (Total cost per strict pass)Calculation | $0.034Claude Code · effort high | $0.029Claude Code | 16 | none recorded | Unclear | More or fewer usd is not better or worse by itself; this row describes behaviour, not a winner. | How much of an AI bill is thinking? Reasoning tokens by model and effort |
| Reasoning share on short tasks vs hard tasks (calculation) (Eight hard tasks)Calculation | 54.4%Claude Code · effort high | 54.8%Claude Code | 24 | range: 36%–96% vs 30%–96% | Unclear | More or fewer percent is not better or worse by itself; this row describes behaviour, not a winner. | How much of an AI bill is thinking? Reasoning tokens by model and effort |
| Reasoning share on short tasks vs hard tasks (calculation) (Five short tasks)Calculation | 43.6%Claude Code · effort high | 0%Claude Code | 15 | range: 0%–93% vs 0%–93% | Unclear | More or fewer percent is not better or worse by itself; this row describes behaviour, not a winner. | How much of an AI bill is thinking? Reasoning tokens by model and effort |
Marks: 95% intervals (Wilson for rates); fastest–slowest run ranges (not intervals)n is shown per side on every rowHollow marks: list-price calculations, not runs
26 rows from 4 studies. No row separates them: 4 ties, 22 unclear.
When to pick which
Only from the rows above. A tie is not a reason to pick either side.
When to pick Claude Opus 5.5 at high effort
No row in this data puts Claude Opus 5.5 at high effort ahead of Claude Opus 5.5 at default effort. Pick on other grounds (price, access, the tasks you run), or measure your own workload.
When to pick Claude Opus 5.5 at default effort
No row in this data puts Claude Opus 5.5 at default effort ahead of Claude Opus 5.5 at high effort. Pick on other grounds (price, access, the tasks you run), or measure your own workload.
Side by side
The study charts, showing only Claude Opus 5.5 at each effort it ran; the two compared settings are highlighted. Open a study for every configuration.
Every interval overlaps every other: this chart does not order these rows.
| Item | Pass rate | 95% interval | n |
|---|---|---|---|
| Claude Opus 5.5 (high) · Claude Code | 100% | 80%–100% | 15 |
| Claude Opus 5.5 · Claude Code | 100% | 80%–100% | 15 |
| Claude Opus 5.5 (low) · Claude Code | 100% | 80%–100% | 15 |
3 rows. All at 100%.
NotesWhiskers: 95% Wilson intervaln = 15 per row3 of 3 at 100%: this task set cannot separate them.
Every call counts; failures and timeouts are non-passes
Whiskers are 95% Wilson intervals. The tasks are short and most configurations pass nearly all of them, so pass rate does not separate the models here.
Source: Provider head-to-head: Claude Code models vs Codex efforts
Every lane’s run range overlaps another lane’s, so the chart shows no finish order.
| Item | Total time per call | Range (lowest–highest run) | n |
|---|---|---|---|
| Claude Opus 5.5 (high) · Claude Code | 2.7 s | 2.5 s–11.8 s | 15 |
| Claude Opus 5.5 · Claude Code | 2.8 s | 2.5 s–8.9 s | 15 |
| Claude Opus 5.5 (low) · Claude Code | 2.8 s | 2.4 s–6.6 s | 15 |
3 rows. Slowest Claude Opus 5.5 (low) · Claude Code 2.8 s (range 2.4 s–6.6 s, n 15). Fastest Claude Opus 5.5 (high) · Claude Code 2.7 s (range 2.5 s–11.8 s, n 15). All run ranges overlap.
NotesLines: fastest–slowest run (not an interval)n = 15 per row
Median per configuration; whiskers = fastest and slowest call
One host, one network, one day. Whiskers are a range, not a confidence interval.
Source: Provider head-to-head: Claude Code models vs Codex efforts
Every lane’s run range overlaps another lane’s, so the chart shows no finish order.
| Item | Time to first useful output | Range (lowest–highest run) | n |
|---|---|---|---|
| Claude Opus 5.5 (high) · Claude Code | 2 s | 1.4 s–9.9 s | 15 |
| Claude Opus 5.5 · Claude Code | 1.9 s | 1.6 s–7.2 s | 15 |
| Claude Opus 5.5 (low) · Claude Code | 2.4 s | 1.5 s–4.9 s | 15 |
3 rows. Slowest Claude Opus 5.5 (low) · Claude Code 2.4 s (range 1.5 s–4.9 s, n 15). Fastest Claude Opus 5.5 · Claude Code 1.9 s (range 1.6 s–7.2 s, n 15). All run ranges overlap.
NotesLines: fastest–slowest run (not an interval)n = 15 per row
Median per configuration; whiskers = fastest and slowest call
One host, one network, one day. Whiskers are a range, not a confidence interval.
Source: Provider head-to-head: Claude Code models vs Codex efforts
- Cache read
- Other input
Totals are the sum of the parts shown. Shares are calculated from the same values.
| Item | Cache read | Other input | n |
|---|---|---|---|
| Claude Opus 5.5 (high) · Claude Code | 1,463 | 619 | 15 |
| Claude Opus 5.5 · Claude Code | 1,401 | 680 | 15 |
| Claude Opus 5.5 (low) · Claude Code | 1,463 | 618 | 15 |
3 rows, 2 series: Cache read, Other input. Cache read: highest Claude Opus 5.5 (high) · Claude Code 1,463 (n 15). Lowest Claude Opus 5.5 · Claude Code 1,401 (n 15). Other input: highest Claude Opus 5.5 · Claude Code 680 (n 15). Lowest Claude Opus 5.5 (low) · Claude Code 618 (n 15).
Notesn = 15 per row
Mean per call, split into prompt-cache reads and other input
The prompts are a few hundred tokens; most input is the CLI’s own system prompt and tool context. Input counts include cache reads, as the vendors report them.
Source: Provider head-to-head: Claude Code models vs Codex efforts
| Item | Output tokens | n |
|---|---|---|
| Claude Opus 5.5 (high) · Claude Code | 78 | 15 |
| Claude Opus 5.5 · Claude Code | 64 | 15 |
| Claude Opus 5.5 (low) · Claude Code | 64 | 15 |
3 rows. Highest Claude Opus 5.5 (high) · Claude Code 78 (n 15). Lowest Claude Opus 5.5 (low) · Claude Code 64 (n 15).
Notesn = 15 per row
Median per configuration; reasoning tokens where the CLI reports them
Vendors count reasoning differently; compare within a vendor first. A median of 0 can mean the CLI did not report reasoning for most calls.
Source: Provider head-to-head: Claude Code models vs Codex efforts
Every interval overlaps every other: this chart does not order these rows.
| Item | Cost per call | Range (lowest–highest run) | n |
|---|---|---|---|
| Claude Opus 5.5 (high) · Claude Code | $0.0069 | $0.0059–$0.027 | 15 |
| Claude Opus 5.5 · Claude Code | $0.0069 | $0.0059–$0.022 | 15 |
| Claude Opus 5.5 (low) · Claude Code | $0.0069 | $0.0058–$0.018 | 15 |
List-price calculation, not a run. 3 rows. Highest Claude Opus 5.5 (high) · Claude Code $0.0069 (range $0.0059–$0.027, n 15). Lowest Claude Opus 5.5 (low) · Claude Code $0.0069 (range $0.0058–$0.018, n 15). All run ranges overlap.
NotesLines: lowest–highest run (not an interval)n = 15 per row
Reported tokens × list price; the calls ran on subscriptions
Calculation, not a bill: the calls ran on flat subscriptions. Whiskers = cheapest and most expensive call.
Sources: Provider head-to-head: Claude Code models vs Codex efforts, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices
Hover or focus a bar for its ratio to Claude Opus 5.5 (low) (the lowest value): a ratio of list-price calculations, not a measurement.
| Item | Cost per pass | n |
|---|---|---|
| Claude Opus 5.5 (low) · Claude Code | $0.0083 | 15 |
| Claude Opus 5.5 · Claude Code | $0.01 | 15 |
| Claude Opus 5.5 (high) · Claude Code | $0.01 | 15 |
List-price calculation, not a run. 3 rows. Highest Claude Opus 5.5 (high) · Claude Code $0.01 (n 15). Lowest Claude Opus 5.5 (low) · Claude Code $0.0083 (n 15).
Notesn = 15 per row
All calls in a configuration, failures included, divided by its passes
Calculation, not a bill: reported tokens × list price; the calls ran on flat subscriptions. A failed call still costs, so a lower pass rate raises the cost per pass. Highlighted bars are on the speed/cost/quality frontier.
Sources: Provider head-to-head: Claude Code models vs Codex efforts, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices
- Strict pass
- Lenient (format misses counted)
| Item | Strict pass | Lenient (format misses counted) | 95% interval | n |
|---|---|---|---|---|
| Claude Opus 5.5 · Claude Code | 100% | 100% | Strict pass: 86%–100%; Lenient (format misses counted): 86%–100% | 24 |
| Claude Opus 5.5 (high) · Claude Code | 100% | 100% | Strict pass: 86%–100%; Lenient (format misses counted): 86%–100% | 24 |
2 rows, 2 series: Strict pass, Lenient (format misses counted). Strict pass: all at 100%. Lenient (format misses counted): all at 100%.
NotesWhiskers: 95% Wilson intervaln = 24 per row
Strict: the reply passes as given. Lenient: a correct answer in the wrong format also counts
Whiskers are 95% Wilson intervals. The strict pass is the result; the lenient reading is shown so a reader can see how much is format and how much is a wrong answer. A format miss never counts as a pass.
Source: Provider head-to-head, hard set: eight hard tasks with strict validators
Every lane’s run range overlaps another lane’s, so the chart shows no finish order.
| Item | Total time per call on hard tasks (separate batches) | Range (lowest–highest run) | n |
|---|---|---|---|
| Claude Opus 5.5 · Claude Code | 9.2 s | 4.2 s–27.2 s | 24 |
| Claude Opus 5.5 (high) · Claude Code | 11 s | 3.6 s–63 s | 24 |
2 rows. Slowest Claude Opus 5.5 (high) · Claude Code 11 s (range 3.6 s–63 s, n 24). Fastest Claude Opus 5.5 · Claude Code 9.2 s (range 4.2 s–27.2 s, n 24). All run ranges overlap.
NotesLines: fastest–slowest run (not an interval)n = 24 per row
Median per configuration; whiskers = fastest and slowest call
One host and network; the counted Claude and Codex batches ran hours apart. Host load was not controlled. Whiskers are a range, not a confidence interval. Highlighted: configurations that passed every call.
Source: Provider head-to-head, hard set: eight hard tasks with strict validators
Every lane’s run range overlaps another lane’s, so the chart shows no finish order.
| Item | Time to first useful output on hard tasks | Range (lowest–highest run) | n |
|---|---|---|---|
| Claude Opus 5.5 · Claude Code | 6.8 s | 2.4 s–21.8 s | 24 |
| Claude Opus 5.5 (high) · Claude Code | 7.1 s | 2.2 s–56.2 s | 24 |
2 rows. Slowest Claude Opus 5.5 (high) · Claude Code 7.1 s (range 2.2 s–56.2 s, n 24). Fastest Claude Opus 5.5 · Claude Code 6.8 s (range 2.4 s–21.8 s, n 24). All run ranges overlap.
NotesLines: fastest–slowest run (not an interval)n = 24 per row
Median per configuration; whiskers = fastest and slowest call
One host and network; the counted Claude and Codex batches ran hours apart. Host load was not controlled. Whiskers are a range, not a confidence interval. Highlighted: configurations that passed every call.
Source: Provider head-to-head, hard set: eight hard tasks with strict validators
| Item | Output tokens | n |
|---|---|---|
| Claude Opus 5.5 · Claude Code | 945 | 24 |
| Claude Opus 5.5 (high) · Claude Code | 1,052 | 24 |
2 rows. Highest Claude Opus 5.5 (high) · Claude Code 1,052 (n 24). Lowest Claude Opus 5.5 · Claude Code 945 (n 24).
Notesn = 24 per row
Median per configuration; reasoning tokens as the CLI reports them
Reasoning tokens are part of the output tokens where the CLI reports them. Their content is never captured. More tokens is not better or worse by itself.
Source: Provider head-to-head, hard set: eight hard tasks with strict validators
| Item | Cost per strict pass | n |
|---|---|---|
| Claude Opus 5.5 · Claude Code | $0.028 | 24 |
| Claude Opus 5.5 (high) · Claude Code | $0.033 | 24 |
List-price calculation, not a run. 2 rows. Highest Claude Opus 5.5 (high) · Claude Code $0.033 (n 24). Lowest Claude Opus 5.5 · Claude Code $0.028 (n 24).
Notesn = 24 per row
All calls in a configuration, failures and format misses included, divided by its strict passes
Calculation, not a bill: reported tokens × list price; the calls ran on a flat subscription. A failed call still costs, so a lower pass rate raises the cost per pass. Highlighted bars are on the quality-vs-cost frontier.
Sources: Provider head-to-head, hard set: eight hard tasks with strict validators, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices
Every interval overlaps every other: this chart does not order these rows.
| Item | Strict pass | 95% interval | n |
|---|---|---|---|
| Claude Opus 5.5 (low) · Claude Code | 100% | 81%–100% | 16 |
| Claude Opus 5.5 (medium) · Claude Code | 100% | 81%–100% | 16 |
| Claude Opus 5.5 (high) · Claude Code | 100% | 81%–100% | 16 |
| Claude Opus 5.5 · Claude Code | 100% | 81%–100% | 16 |
4 rows. All at 100%.
NotesWhiskers: 95% Wilson intervaln = 16 per row4 of 4 at 100%: this task set cannot separate them.
Every configuration: 8 tasks × 2 repetitions. Whiskers are 95% Wilson intervals
Whiskers are 95% Wilson intervals. A format miss never counts as a pass. Reference cells (efforts the hard head-to-head already ran) are reused, not rerun; Claude reference cells keep repetitions 1-2, so every cell is 8 tasks × 2 repetitions.
Source: Effort ladder: the hard task set at each effort level
Every lane’s run range overlaps another lane’s, so the chart shows no finish order.
| Item | Total time per call by effort on hard tasks | Range (lowest–highest run) | n |
|---|---|---|---|
| Claude Opus 5.5 (low) · Claude Code | 7.5 s | 3.3 s–15.8 s | 16 |
| Claude Opus 5.5 (medium) · Claude Code | 9.7 s | 4.8 s–31.4 s | 16 |
| Claude Opus 5.5 (high) · Claude Code | 10.1 s | 3.6 s–63 s | 16 |
| Claude Opus 5.5 · Claude Code | 9.2 s | 4.2 s–27.2 s | 16 |
4 rows. Slowest Claude Opus 5.5 (high) · Claude Code 10.1 s (range 3.6 s–63 s, n 16). Fastest Claude Opus 5.5 (low) · Claude Code 7.5 s (range 3.3 s–15.8 s, n 16). All run ranges overlap.
NotesLines: fastest–slowest run (not an interval)n = 16 per row
Median per configuration; whiskers = fastest and slowest call
Whiskers are a range (fastest and slowest call), not a confidence interval. One host, one network. Reference cells (efforts the hard head-to-head already ran) are reused, not rerun; Claude reference cells keep repetitions 1-2, so every cell is 8 tasks × 2 repetitions. The reference cells ran in a different hour.
Source: Effort ladder: the hard task set at each effort level
| Item | Output tokens | n |
|---|---|---|
| Claude Opus 5.5 (low) · Claude Code | 594 | 16 |
| Claude Opus 5.5 (medium) · Claude Code | 853 | 16 |
| Claude Opus 5.5 (high) · Claude Code | 1,052 | 16 |
| Claude Opus 5.5 · Claude Code | 945 | 16 |
4 rows. Highest Claude Opus 5.5 (high) · Claude Code 1,052 (n 16). Lowest Claude Opus 5.5 (low) · Claude Code 594 (n 16).
Notesn = 16 per row
Median per configuration; reasoning tokens as the CLI reports them
Reasoning tokens are part of the output tokens where the CLI reports them; 0 can mean "not reported". Their content is never captured. More tokens is not better or worse by itself.
Source: Effort ladder: the hard task set at each effort level
Hover or focus a bar for its ratio to Claude Opus 5.5 (low) (the lowest value): a ratio of list-price calculations, not a measurement.
| Item | Cost per strict pass | n |
|---|---|---|
| Claude Opus 5.5 (low) · Claude Code | $0.021 | 16 |
| Claude Opus 5.5 (medium) · Claude Code | $0.029 | 16 |
| Claude Opus 5.5 (high) · Claude Code | $0.034 | 16 |
| Claude Opus 5.5 · Claude Code | $0.029 | 16 |
List-price calculation, not a run. 4 rows. Highest Claude Opus 5.5 (high) · Claude Code $0.034 (n 16). Lowest Claude Opus 5.5 (low) · Claude Code $0.021 (n 16).
Notesn = 16 per row
All calls in a configuration divided by its strict passes
Calculation, not a bill: reported tokens × list price, cache reads and writes priced as in the hard head-to-head; the calls ran on flat subscriptions. Codex input includes its own system prompt and tool schemas, so a Claude-vs-Codex gap is partly the CLI.
Sources: Effort ladder: the hard task set at each effort level, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices
- Reasoning (output tokens)
- Remaining output (visible-answer estimate)
- Input (prompt, cache priced)
Totals are the sum of the parts shown. Shares are calculated from the same values.
| Item | Reasoning (output tokens) | Remaining output (visible-answer estimate) | Input (prompt, cache priced) | n |
|---|---|---|---|---|
| Claude Opus 5.5 (high) · Claude Code | $0.018 | $0.008 | $0.0074 | 24 |
| Claude Opus 5.5 · Claude Code | $0.013 | $0.008 | $0.0077 | 24 |
List-price calculation, not a run. 2 rows, 3 series: Reasoning (output tokens), Remaining output (visible-answer estimate), Input (prompt, cache priced). Reasoning (output tokens): highest Claude Opus 5.5 (high) · Claude Code $0.018 (n 24). Lowest Claude Opus 5.5 · Claude Code $0.013 (n 24). Remaining output (visible-answer estimate): highest Claude Opus 5.5 · Claude Code $0.008 (n 24). Lowest Claude Opus 5.5 (high) · Claude Code $0.008 (n 24).
Notesn = 24 per row
Mean per call on the hard tasks; the three parts add up to the call
Calculation, not a bill: reported tokens × list price (per million output tokens: Haiku 4.5 $5, Sonnet 5.5 $10, Opus 5.5 $20, Fable 5.1 $50 and GPT-6.1 Sol $10); the calls ran on flat subscriptions. Reasoning and remaining output both use the output price. The remainder estimates visible-answer tokens; its exact meaning depends on the CLI counters. Input is the whole prompt, with cache reads and writes priced as in the hard head-to-head. It includes CLI context; these receipts do not separate task tokens from CLI context. It changes with cache counters. Batch timing and cache behavior were not controlled. Means, not medians, so the parts add up.
Sources: Reasoning token bill (calculation), Provider head-to-head, hard set: eight hard tasks with strict validators, Effort ladder: the hard task set at each effort level, Provider head-to-head: Claude Code models vs Codex efforts, Anthropic list prices (Claude models), OpenAI list prices
- Reasoning cost per strict pass
- Total cost per strict pass (square)
Gap labels, Total cost per strict pass vs Reasoning cost per strict pass: Total cost per strict pass is x% higher (+) or lower (−) than Reasoning cost per strict pass, calculated from the two values shown (the change counted from Reasoning cost per strict pass’s value).
| Item | Reasoning cost per strict pass | Total cost per strict pass | n |
|---|---|---|---|
| Claude Opus 5.5 (low) · Claude Code | $0.005 | $0.021 | 16 |
| Claude Opus 5.5 (medium) · Claude Code | $0.013 | $0.029 | 16 |
| Claude Opus 5.5 (high) · Claude Code | $0.018 | $0.034 | 16 |
| Claude Opus 5.5 · Claude Code | $0.013 | $0.029 | 16 |
List-price calculation, not a run. 4 rows, 2 series: Reasoning cost per strict pass, Total cost per strict pass. Reasoning cost per strict pass: highest Claude Opus 5.5 (high) · Claude Code $0.018 (n 16). Lowest Claude Opus 5.5 (low) · Claude Code $0.005 (n 16). Total cost per strict pass: highest Claude Opus 5.5 (high) · Claude Code $0.034 (n 16). Lowest Claude Opus 5.5 (low) · Claude Code $0.021 (n 16).
Notesn = 16 per row
List price ÷ strict passes; every cell is 8 tasks × 2 repetitions
Calculation, not a bill: reported tokens × list price, divided by the cell's strict passes; the calls ran on flat subscriptions. The effort-ladder cells: new calls plus reference cells reused from the hard head-to-head (Claude repetitions 1-2 only). "Default" means the effort flag was not passed. The total is the same value as the effort-ladder cost-per-pass chart. Effort levels are not the same scale across vendors, and the reference cells ran in a different batch and hour.
Sources: Reasoning token bill (calculation), Provider head-to-head, hard set: eight hard tasks with strict validators, Effort ladder: the hard task set at each effort level, Provider head-to-head: Claude Code models vs Codex efforts, Anthropic list prices (Claude models), OpenAI list prices
- Eight hard tasks
- Five short tasks (square)
Gap labels, Five short tasks vs Eight hard tasks: Five short tasks is x percentage points higher (+) or lower (−) than Eight hard tasks, calculated from the two values shown; lines are the lowest–highest run (not an interval).
| Item | Eight hard tasks | Five short tasks | Range (lowest–highest run) | n |
|---|---|---|---|---|
| Claude Opus 5.5 · Claude Code | 55% | 0% | Eight hard tasks: 30%–96%; Five short tasks: 0%–93% | 24 |
| Claude Opus 5.5 (high) · Claude Code | 54% | 44% | Eight hard tasks: 36%–96%; Five short tasks: 0%–93% | 24 |
List-price calculation, not a run. 2 rows, 2 series: Eight hard tasks, Five short tasks. Eight hard tasks: highest Claude Opus 5.5 · Claude Code 55% (range 30%–96%, n 24). Lowest Claude Opus 5.5 (high) · Claude Code 54% (range 36%–96%, n 24). All run ranges overlap. Five short tasks: highest Claude Opus 5.5 (high) · Claude Code 44% (range 0%–93%, n 15). Lowest Claude Opus 5.5 · Claude Code 0% (range 0%–93%, n 15). All run ranges overlap.
NotesLines: lowest–highest run (not an interval)n 15–24 per row
Median call per configuration; five short tasks and eight hard tasks
Calculation from reported tokens, not a run: the median of per-call reasoning ÷ output. A median of 0% means at least half of the calls reported 0 reasoning tokens in the receipt. On a route that reports reasoning, we retain a numeric 0 as the recorded counter. It does not prove the model did no internal reasoning. The table says how many calls reported 0. The short tasks are five small validated tasks (a bug fix, a JSON extraction, an arithmetic problem, a refactor and a ticket classification). Per-call ranges are in the tables.
Sources: Reasoning token bill (calculation), Provider head-to-head, hard set: eight hard tasks with strict validators, Effort ladder: the hard task set at each effort level, Provider head-to-head: Claude Code models vs Codex efforts, Anthropic list prices (Claude models), OpenAI list prices
How a row is called
- AheadThe 95% intervals do not overlap, or the run ranges or p50–p95 bands do not overlap with at least 5 runs per side.
- TieThe values match, or both sit at the same ceiling.
- UnclearThe intervals or ranges overlap, too few runs were recorded, no interval was recorded, or more is not better (token counts are never a win).
- CalculationDerived from list prices and recorded counts. Not a bill and not a run.
The dataset compiler makes every call; this page only draws it. No composite score, no rank.
Questions
- Is Claude Opus 5.5 better at high effort or default effort (CLI chose)?
- Claude Opus 5.5 at high effort and Claude Opus 5.5 at default effort share 14 measured metrics and 12 list-price calculations from 4 studies. Only the effort setting differs between the two sides of a row; the route and the task set are the same. "default" means the effort flag was not passed, so its level is the CLI’s choice. No row separates them: every interval or run range overlaps, too few runs were recorded, no interval was recorded, or more is not better for that metric. The rows are 4 ties and 22 unclear; each row says why. Calculation rows are derived from list prices and recorded counts; they are not bills or runs. Some rows rest on small samples (n = 15 at the smallest).
- How were the two effort settings measured?
- They share 14 measured metrics and 12 list-price calculations from 4 public studies: Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head; Haiku vs Sonnet vs Opus vs Fable vs GPT-6.1 Sol on 8 hard tasks; Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks; How much of an AI bill is thinking? Reasoning tokens by model and effort. Only the effort setting differs between the two sides of a row.
- How does pass rate on five validated tasks change from high effort to default effort (CLI chose)?
- Claude Opus 5.5 at high effort: 100% (15/15) (n = 15, 95% interval 80% to 100%). Claude Opus 5.5 at default effort: 100% (15/15) (n = 15, 95% interval 80% to 100%). The 95% intervals overlap (Claude Opus 5.5 at high effort 80% to 100%; Claude Opus 5.5 at default effort 80% to 100%), so this sample cannot separate them.
- How does total time per call change from high effort to default effort (CLI chose)?
- Claude Opus 5.5 at high effort: 2.71 s (n = 15, run range 2.5 s to 11.8 s). Claude Opus 5.5 at default effort: 2.75 s (n = 15, run range 2.5 s to 8.9 s). The run ranges (fastest to slowest) overlap (Claude Opus 5.5 at high effort 2.45 s to 11.8 s; Claude Opus 5.5 at default effort 2.47 s to 8.91 s); the medians alone do not show a reliable difference. A range is not a confidence interval.
- How does time to first useful output change from high effort to default effort (CLI chose)?
- Claude Opus 5.5 at high effort: 2.04 s (n = 15, run range 1.4 s to 9.9 s). Claude Opus 5.5 at default effort: 1.92 s (n = 15, run range 1.6 s to 7.2 s). The run ranges (fastest to slowest) overlap (Claude Opus 5.5 at high effort 1.40 s to 9.94 s; Claude Opus 5.5 at default effort 1.56 s to 7.23 s); the medians alone do not show a reliable difference. A range is not a confidence interval.
- How does pass rate on eight hard tasks (Strict pass) change from high effort to default effort (CLI chose)?
- Claude Opus 5.5 at high effort: 100% (24/24) (n = 24, 95% interval 86% to 100%). Claude Opus 5.5 at default effort: 100% (24/24) (n = 24, 95% interval 86% to 100%). The 95% intervals overlap (Claude Opus 5.5 at high effort 86% to 100%; Claude Opus 5.5 at default effort 86% to 100%), so this sample cannot separate them.
The studies behind this page
Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head
130 timed calls on five validated tasks: Claude Haiku, Sonnet, Opus and Fable against Codex GPT-6.1 Sol. Pass rate, speed, tokens, cost per pass.
Haiku vs Sonnet vs Opus vs Fable vs GPT-6.1 Sol on 8 hard tasks
152 calls on 8 hard tasks with strict validators. Pass rate with 95% intervals, format misses, speed, tokens and cost per pass.
Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks
176 calls on 8 hard tasks at low, medium, high and default effort: Sonnet, Opus and GPT-6.1 Sol. Pass rate with 95% intervals, time, tokens, cost per pass.
How much of an AI bill is thinking? Reasoning tokens by model and effort
Reasoning tokens in 378 recorded calls by model and effort: share of output, list-price cost per call and per pass, and time. Calculations.