Agent
Sign in

Model · Anthropic

Claude Opus 5.5

Anthropic’s large Claude model, measured through Claude Code at low, medium, high and default effort.

136 values from 11 studies (75 are list-price calculations) · Updated

At a glance

The best-supported value per category: a 95% interval first, then a run range, then the larger n. Three separate values from separate studies, never one score.

Qualitypass rates, accuracy and scores

100% (24/24)

95% CI 86%–100% · n = 24

Pass rate on eight hard tasks (Strict pass)

Claude Code · eight hard validated tasks · Haiku vs Sonnet vs Opus vs Fable vs GPT-6.1 Sol on 8 hard tasks

Speedtime per call or decision

9.18s

range 4.2 s–27.2 s · n = 24

Total time per call on hard tasks (separate batches)

Claude Code · eight hard validated tasks · Haiku vs Sonnet vs Opus vs Fable vs GPT-6.1 Sol on 8 hard tasks

CostUS dollars per call, pass or decision

$0.0069

range $0.0059–$0.022 · n = 15 · list-price calculation

List-price cost per call (calculation)

Claude Code · five short validated tasks · Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head

Claude Opus 5.5 price per 1M tokens

Claude Opus 5.5 costs $4.00 per 1M input tokens and $20.00 per 1M output tokens at standard Claude API list prices (observed 2026-10-08). Cache reads cost $0.20 per 1M tokens.

Vendor list price · observed 2026-10-08Third-party-reported · snapshot 2026-10-06

Vendor list price

$4.00

Input, per 1M tokens

Vendor list price

$20.00

Output, per 1M tokens

Vendor list price

$0.20

Cache read, per 1M tokens

Vendor specification · 2026-10-08

1M

Context window, tokens

Cache writes are separate: $5.00 per 1M tokens for five minutes; $8.00 per 1M tokens for one hour. Standard global Claude API prices; other tiers, tools and taxes can add charges.

Source: Official Claude API prices · observed .

Context: Official model specification · observed . Route and provider limits can be lower than the model specification.

Recorded study price source: Anthropic list prices (Claude models) (). Prices as listed by the vendor on 2026-09-21 and recorded in the product price table. Cache reads at the listed rate, one-hour cache writes at twice the input price.

Providers in the dated snapshot

5 standard-tier providers · snapshot 2026-10-06 · USD per 1M tokens

Same snapshot price at all 5 providers
  • Anthropicfirst-party
  • Amazon Bedrock
  • Azure
  • Claude Platform on AWS
  • Google Vertex

Third-party-reported, not measured by Agent · Standard tier only: regional, flex, fast and priority tiers are left out · One row per provider: its cheapest standard endpoint

Claude Opus 5.5: 5 standard-tier providers, all at the same price ($4.00 input, $20.00 output per 1M tokens). Third-party-reported, snapshot 2026-10-06.

Source: OpenRouter public API: models and provider endpoints (snapshot) (). The diamond marks the first-party provider in this snapshot. Current availability is not checked.

Estimate your monthly costPrices of every model at every provider

Where it sits

Every measured value, grouped by study. Each row puts the value on its own track, with the other configurations of the same chart as muted dots. A range is the fastest to slowest recorded run and p50–p95 is the median to the 95th percentile; neither is a confidence interval. Use Table for the plain values.

Opus 5.5 vs Sonnet 5.5 as the Agent brain on SWE-bench Verified (interim)

3 values · open the study

Resolved on the same 3 SWE-bench Verified instances (interim)n = 3 · 95% CI 21%–94% · Agent · new build · SWE-bench Verified, interim paired probe
67% (2/3)
List-price cost per attempt (calculation)n = 3 · Agent · new build · SWE-bench Verified, interim paired probe
$7.59Calculation
Worker time per attemptn = 3 · range 9.8 min–25.3 min · Agent · new build · SWE-bench Verified, interim paired probe
20.3 min

Whiskers: 95% Wilson intervalLines: fastest–slowest run (not an interval)n beside each valueMuted dots: the other configurations on the same chartHollow: list-price calculation

Claude Opus 5.5 in Opus 5.5 vs Sonnet 5.5 as the Agent brain on SWE-bench Verified (interim): 3 values, first Resolved on the same 3 SWE-bench Verified instances (interim) 67% (2/3).

Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head

24 values · open the study

Pass rate on five validated tasksn = 15 · 95% CI 80%–100% · Claude Code · effort high · five short validated tasks
100% (15/15)
Pass rate on five validated tasksn = 15 · 95% CI 80%–100% · Claude Code · five short validated tasks
100% (15/15)
Pass rate on five validated tasksn = 15 · 95% CI 80%–100% · Claude Code · effort low · five short validated tasks
100% (15/15)
Total time per calln = 15 · range 2.5 s–11.8 s · Claude Code · effort high · five short validated tasks
2.71 s
Total time per calln = 15 · range 2.5 s–8.9 s · Claude Code · five short validated tasks
2.75 s
Total time per calln = 15 · range 2.4 s–6.6 s · Claude Code · effort low · five short validated tasks
2.83 s

Whiskers: 95% Wilson intervalLines: fastest–slowest run (not an interval)n beside each valueMuted dots: the other configurations on the same chartHollow: list-price calculation

Claude Opus 5.5 in Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head: 24 values, first Pass rate on five validated tasks 100% (15/15).

Haiku vs Sonnet vs Opus vs Fable vs GPT-6.1 Sol on 8 hard tasks

12 values · open the study

Pass rate on eight hard tasks (Strict pass)n = 24 · 95% CI 86%–100% · Claude Code · eight hard validated tasks
100% (24/24)
Pass rate on eight hard tasks (Strict pass)n = 24 · 95% CI 86%–100% · Claude Code · effort high · eight hard validated tasks
100% (24/24)
Pass rate on eight hard tasks (Lenient (format misses counted))n = 24 · 95% CI 86%–100% · Claude Code · eight hard validated tasks
100% (24/24)
Pass rate on eight hard tasks (Lenient (format misses counted))n = 24 · 95% CI 86%–100% · Claude Code · effort high · eight hard validated tasks
100% (24/24)
Total time per call on hard tasks (separate batches)n = 24 · range 4.2 s–27.2 s · Claude Code · eight hard validated tasks
9.18 s
Total time per call on hard tasks (separate batches)n = 24 · range 3.6 s–63 s · Claude Code · effort high · eight hard validated tasks
11.0 s

Whiskers: 95% Wilson intervalLines: fastest–slowest run (not an interval)n beside each valueMuted dots: the other configurations on the same chartHollow: list-price calculation

Claude Opus 5.5 in Haiku vs Sonnet vs Opus vs Fable vs GPT-6.1 Sol on 8 hard tasks: 12 values, first Pass rate on eight hard tasks (Strict pass) 100% (24/24).

Claude Code (Sonnet 5.5, Opus 5.5) vs Codex CLI on 6 hidden-test coding tasks

4 values · open the study

Coding sessions that passed every hidden checkn = 12 · 95% CI 76%–100% · Claude Code · six small repository tasks with hidden tests
100% (12/12)
Time per coding sessionn = 12 · range 29.8 s–186 s · Claude Code · six small repository tasks with hidden tests
56.9 s
Tool calls per coding sessionn = 12 · range 5–14 · Claude Code · six small repository tasks with hidden tests
7.5
List-price cost per passing coding session (calculation)n = 12 · Claude Code · six small repository tasks with hidden tests
$0.22Calculation

Whiskers: 95% Wilson intervalLines: fastest–slowest run (not an interval)n beside each valueMuted dots: the other configurations on the same chartHollow: list-price calculation

Claude Opus 5.5 in Claude Code (Sonnet 5.5, Opus 5.5) vs Codex CLI on 6 hidden-test coding tasks: 4 values, first Coding sessions that passed every hidden check 100% (12/12).

Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks

16 values · open the study

Strict pass rate by effort on eight hard tasksn = 16 · 95% CI 81%–100% · Claude Code · effort low · eight hard validated tasks, effort ladder
100% (16/16)
Strict pass rate by effort on eight hard tasksn = 16 · 95% CI 81%–100% · Claude Code · effort medium · eight hard validated tasks, effort ladder
100% (16/16)
Strict pass rate by effort on eight hard tasksn = 16 · 95% CI 81%–100% · Claude Code · effort high · eight hard validated tasks, effort ladder
100% (16/16)
Strict pass rate by effort on eight hard tasksn = 16 · 95% CI 81%–100% · Claude Code · eight hard validated tasks, effort ladder
100% (16/16)
Total time per call by effort on hard tasksn = 16 · range 3.3 s–15.8 s · Claude Code · effort low · eight hard validated tasks, effort ladder
7.50 s
Total time per call by effort on hard tasksn = 16 · range 4.8 s–31.4 s · Claude Code · effort medium · eight hard validated tasks, effort ladder
9.72 s

Whiskers: 95% Wilson intervalLines: fastest–slowest run (not an interval)n beside each valueMuted dots: the other configurations on the same chartHollow: list-price calculation

Claude Opus 5.5 in Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks: 16 values, first Strict pass rate by effort on eight hard tasks 100% (16/16).

Prompt caching and run-to-run consistency in Claude Code and Codex CLI

4 values · open the study

List-price cost of 5-question sessions with and without the cache (calculation) (With the cache, as recorded)n = 15 · Claude Code · calculation: 5-turn cached sessions over a fixed ledger
$0.26Calculation
List-price cost of 5-question sessions with and without the cache (calculation) (Without a cache: every input token at the input price)n = 15 · Claude Code · calculation: 5-turn cached sessions over a fixed ledger
$0.54Calculation
Time per turn: first turn vs later turns in a cached session (Turn 1 (writes the ledger to the cache))n = 3 · range 1.8 s–4.4 s · Claude Code · 5-turn cached sessions over a fixed ledger
1.90 s
Time per turn: first turn vs later turns in a cached session (Turns 2-5 (read the ledger from the cache))n = 12 · range 1.6 s–12.7 s · Claude Code · 5-turn cached sessions over a fixed ledger
2.40 s

Lines: fastest–slowest run (not an interval)n beside each valueMuted dots: the other configurations on the same chartHollow: list-price calculation

Claude Opus 5.5 in Prompt caching and run-to-run consistency in Claude Code and Codex CLI: 4 values, first List-price cost of 5-question sessions with and without the cache (calculation) (With the cache, as recorded) $0.26.

What if every call ran on Opus? Repricing real agent tokens

3 values · open the study

Thought experiment: the same tokens at other list pricescalculation: Agent’s recorded tokens at this model’s list price
$5.75Calculation
Thought experiment: what prompt caching saved (With caching (as recorded))calculation: Agent’s recorded tokens at this model’s list price
$143.83Calculation
Thought experiment: what prompt caching saved (Without caching)calculation: Agent’s recorded tokens at this model’s list price
$686.66Calculation

n beside each valueMuted dots: the other configurations on the same chartHollow: list-price calculation

Claude Opus 5.5 in What if every call ran on Opus? Repricing real agent tokens: 3 values, first Thought experiment: the same tokens at other list prices $5.75.

Prompt cache break-even: after how many reuses does a cached prefix cost less?

30 values · open the study

Reuses before a cached prefix costs less, by model and write type (calculation) (1-hour write (2× input), whole prefix new)cache read $0.2 per M · calculation: Agent’s recorded tokens at this model’s list price
1.05Calculation
Reuses before a cached prefix costs less, by model and write type (calculation) (1-hour write (2× input), whole prefix new)cache read $0.4 per M · calculation: Agent’s recorded tokens at this model’s list price
1.11Calculation
Reuses before a cached prefix costs less, by model and write type (calculation) (1-hour write, pooled n = 6 session share, 19% already cached (as recorded))cache read $0.2 per M · calculation: Agent’s recorded tokens at this model’s list price
0.67Calculation
Reuses before a cached prefix costs less, by model and write type (calculation) (1-hour write, pooled n = 6 session share, 19% already cached (as recorded))cache read $0.4 per M · calculation: Agent’s recorded tokens at this model’s list price
0.72Calculation
Reuses before a cached prefix costs less, by model and write type (calculation) (5-minute write (1.25× input, an assumption))cache read $0.2 per M · calculation: Agent’s recorded tokens at this model’s list price
0.26Calculation
Reuses before a cached prefix costs less, by model and write type (calculation) (5-minute write (1.25× input, an assumption))cache read $0.4 per M · calculation: Agent’s recorded tokens at this model’s list price
0.28Calculation

n beside each valueMuted dots: the other configurations on the same chartHollow: list-price calculation

Claude Opus 5.5 in Prompt cache break-even: after how many reuses does a cached prefix cost less?: 30 values, first Reuses before a cached prefix costs less, by model and write type (calculation) (1-hour write (2× input), whole prefix new) 1.05.

How much of an AI bill is thinking? Reasoning tokens by model and effort

20 values · open the study

Reasoning share of output tokens per call on hard tasks (calculation)n = 24 · range 30%–96% · Claude Code
54.8%Calculation
Reasoning share of output tokens per call on hard tasks (calculation)n = 24 · range 36%–96% · Claude Code · effort high
54.4%Calculation
List-price cost per call: reasoning, remaining output and input (calculation) (Reasoning (output tokens))n = 24 · Claude Code · effort high
$0.018Calculation
List-price cost per call: reasoning, remaining output and input (calculation) (Reasoning (output tokens))n = 24 · Claude Code
$0.013Calculation
List-price cost per call: reasoning, remaining output and input (calculation) (Remaining output (visible-answer estimate))n = 24 · Claude Code · effort high
$0.0080Calculation
List-price cost per call: reasoning, remaining output and input (calculation) (Remaining output (visible-answer estimate))n = 24 · Claude Code
$0.0080Calculation

Lines: lowest–highest run (not an interval)n beside each valueMuted dots: the other configurations on the same chartHollow: list-price calculation

Claude Opus 5.5 in How much of an AI bill is thinking? Reasoning tokens by model and effort: 20 values, first Reasoning share of output tokens per call on hard tasks (calculation) 54.8%.

Where the seconds go: first text, output speed and prompt size for 6 LLMs

10 values · open the study

Time to first text: a 250-line answer, six modelsn = 4 · range 1.7 s–2.4 s · Claude Code
1.97 s
Output speed after the first text: visible tokens per second (calculation)n = 4 · range 155–156 · Claude Code
156Calculation
Output speed in characters per second after the first text (calculation)n = 4 · range 345–349 · Claude Code
347Calculation
Time to first text as the prompt grows: 1kn = 3 · range 1.5 s–2 s · Claude Code
1.51 sCalculation
Time to first text as the prompt grows: 16kn = 3 · range 1.7 s–3 s · Claude Code
1.74 sCalculation
Time to first text as the prompt grows: 64kn = 3 · range 1.7 s–3.7 s · Claude Code
1.79 sCalculation

Lines: fastest–slowest run (not an interval)Whiskers: 95% Wilson intervaln beside each valueMuted dots: the other configurations on the same chartHollow: list-price calculation

Claude Opus 5.5 in Where the seconds go: first text, output speed and prompt size for 6 LLMs: 10 values, first Time to first text: a 250-line answer, six models 1.97 s.

GPT-6.1 Sol vs Claude Opus 5.5, Sonnet 5.5 and Haiku 4.5 on 4 harder tasks

10 values · open the study

Pass rate on 4 harder tasks (Strict pass)n = 12 · 95% CI 19%–68% · Claude Code
42% (5/12)
Pass rate on 4 harder tasks (Lenient (format misses counted))n = 12 · 95% CI 25%–75% · Claude Code
50% (6/12)
Calls that tried a tool although tools were offn = 12 · 95% CI 19%–68% · Claude Code
42% (5/12)
Strict pass rate by task: 10x10 nonogramn = 3 · 95% CI 44%–100% · Claude Code
100% (3/3)
Strict pass rate by task: Sudoku, 22 givensn = 3 · 95% CI 0%–56% · Claude Code
0% (0/3)
Strict pass rate by task: 6x6 Skyscrapersn = 3 · 95% CI 6.2%–79% · Claude Code
33% (1/3)

Whiskers: 95% Wilson intervalLines: fastest–slowest run (not an interval)n beside each valueMuted dots: the other configurations on the same chartHollow: list-price calculation

Claude Opus 5.5 in GPT-6.1 Sol vs Claude Opus 5.5, Sonnet 5.5 and Haiku 4.5 on 4 harder tasks: 10 values, first Pass rate on 4 harder tasks (Strict pass) 42% (5/12).

Effort profile

The effort ladder study ran the same eight hard tasks at each effort setting. Claude Opus 5.5 is emphasized; the other models stay on the chart for scale.

Median total time per call, by effort

  • Claude Sonnet 5.5 · Claude Code
  • Claude Opus 5.5 · Claude Code
  • GPT-6.1 Sol · Codex CLI

Seconds (median)

* default: the effort flag was not passed; its level is not known, so no line joins it.

4 efforts, 3 series: Claude Sonnet 5.5 · Claude Code, Claude Opus 5.5 · Claude Code, GPT-6.1 Sol · Codex CLI. Claude Sonnet 5.5 · Claude Code: slowest high 8.8 s (n 16). Fastest low 5.8 s (n 16). Claude Opus 5.5 · Claude Code: slowest high 10.1 s (n 16). Fastest low 7.5 s (n 16).

Notesn = 16 per row

One line per model and route; default = the effort flag was not passed

Medians only; the per-call ranges are in the total-time chart and they overlap. "default" is placed last because its level is not known: the CLI chose it.

Source: Effort ladder: the hard task set at each effort level

Compare Claude Opus 5.5

Each bar counts the rows of one comparison: a side ahead only where its interval or range is apart, otherwise a tie or unclear.

Watch

Live story · 211 sAgent Benchmarks, October 2026: 28 studies in one film

Agent Benchmarks, October 2026: 28 studies in one film

The headline of each of our 28 open benchmark studies, with sample sizes and intervals. Calculations labelled; failures counted.

Live story · 211 s · download it as video (16:9, 9:16 or 1:1) from the player · See the data

Transcript
  1. October 2026 roundup · 28 studies. Agent Benchmarks: every headline. Recorded runs, 95% intervals where they exist, every failure counted. Calculations labelled.
  2. SWE-bench Verified: Agent resolved 25 of 33. The public panel averaged 74.1%; the intervals overlap, so no rank. Agent resolved (one attempt each): 76% (25/33) (n = 33, 95% CI 59–87%). Public panel mean, same instances: 74.1% (n = 33). Model cost per resolved instance (calculation, notional): $3.71 (n = 25). Caveat: n = 33: intervals are wide. This is a defect-finding run, not a ranking.
  3. SWE-bench, interim (3 of 8 pairs graded): Opus 5.5 as Agent’s brain resolved 2 of 3, Sonnet 5.5 1 of 3. Exact McNemar p = 1.0: no difference yet. Opus cost 2.6× as much, a calculation. Opus 5.5 in Agent: resolved (interim): 67% (2/3) (n = 3, 95% CI 21–94%). Sonnet 5.5 in Agent: same instances: 33% (1/3) (n = 3, 95% CI 6–79%). List-price cost, Opus vs Sonnet (calculation): 2.6× (n = 3). Caveat: Different platform builds: Opus ran on 4f6f4027; the Sonnet attempts ran on f0ac3a8a and 236c0d3f. Platform changes can move results by themselves, so a gap compares Sonnet on an older platform with Opus on the current one, not the models alone.
  4. Blind critics preferred the AI pull request to the merged human one on 9 of 12 tasks at the latest attempt, 6 of 12 at the first. AI preferred, latest attempt: 75% (9/12) (n = 12, 95% CI 47–91%). AI preferred, first scored attempt: 50% (6/12) (n = 12, 95% CI 25–75%). Single critic verdicts for the AI change: 69% (91/132) (n = 132, 95% CI 61–76%). Caveat: Most critics are Anthropic models, and the worker runs on an Anthropic model. Same-family preference is possible; the per-critic chart shows the OpenAI critics’ share.
  5. Five short tasks, nine setups: 127 of 130 calls passed, so speed and tokens separate them. Fastest median: Fable 5.1 at 1.9 s. Calls that passed their validator: 98% (127/130) (n = 130, 95% CI 93–99%). Fastest median total time (Fable 5.1): 1.9 s (n = 15). Median input tokens per call: Codex CLI vs Claude Code: 12,124 vs 2,130 (n = 130). Caveat: CLI timings include CLI start-up and the CLI’s own system prompt.
  6. Eight hard tasks: Sonnet 5.5, Opus 5.5, Opus 5.5 high, GPT-6.1 Sol medium, Fable 5.1 and GPT-6.1 Sol high passed every call (24/24 or 16/16). Haiku 4.5 passed 11/24. Chart: Eight hard tasks · strict pass rate · 95% intervals (n = 16–24 each). Caveat: Claude Code and Codex CLI rows pair a CLI with a model, and each CLI adds its own system prompt and start-up time. A Claude-vs-GPT row compares the route + model pairs, not the models alone.
  7. Coding agents on hidden-test tasks: 36 of 36 sessions passed, so time separates them. Claude Code with Sonnet 5.5: median 23.1 s; Codex CLI took 4.9× as long, with extra standing instructions. Sessions that passed every hidden check: 100% (36/36) (n = 36, 95% CI 90–100%). Median time, Sonnet 5.5 in Claude Code: 23.1 s (n = 12). Median time, Codex CLI vs Claude Code with Sonnet: 4.9× (n = 12). Caveat: Codex CLI ran with the tester’s global AGENTS.md: --ignore-user-config does not switch that file off. 12 of 12 Codex sessions read the tester’s notes, 10 wrote a WORKLOG.md and 9 reported a commit attempt. Its time, tool calls and lines changed include that work. Claude Code ran with setting sources off, and no Claude session did any of it.
  8. Effort ladder: 11 model and effort settings passed 176 of 176 calls strictly (16/16 each, 81%–100%). No effort level wins any of 172 rows. Calls that passed strictly, low to high effort: 100% (176/176) (n = 176, 95% CI 98–100%). Model and effort configurations: 11 (6 new, 5 reference). Effort comparison rows where one level wins: 0 (of 172 rows · 16 pairs). Caveat: Every cell passed every call, so the set has a ceiling: a perfect 16/16 has a 95% interval of 81% to 100%. This study cannot show that effort does not matter on harder work; it shows that these 8 tasks do not need more than low effort.
  9. Caching, a list-price calculation: 50% less on Sonnet 5.5, 53% on Opus 5.5. Same prompt 10 times: 7 of 9 cells passed every time. Sonnet 5.5 session cost saved by the cache (calculation): 50% ($0.1350 vs $0.2698). Later sessions that reused an earlier session’s cache: 0 of 4. Model-and-prompt cells that passed 10 of 10: 7 of 9. Caveat: Cross-session reuse did not happen here. The cause was not tested: the CLI may add per-process context before the user message. Do not read it as a provider property.
  10. Agent memory: Sonnet 5.5 followed team-only rules 40% (6/15) of the time without memory and 100% (15/15) with an 11-line file. Without the late-fee rate it asked 7/9 times. Team knowledge, no memory: 40% (6/15) (n = 15, 95% CI 20–64%). Team knowledge, 11-line file: 100% (15/15) (n = 15, 95% CI 80–100%). Asked for the missing rate: 7/9. Caveat: One small synthetic repository and one author of the facts: the curated file is an upper bound written with knowledge of the tasks.
  11. System One arena: on 1,085 checkable decisions, Jev 1.13 answered 76.8% right; the best open model, Clef 27B, 69.9%. Jev 1.13: 76.8% (805/1048) (n = 1048, 95% CI 74–79%). Clef 27B: 69.9% (733/1048) (n = 1048, 95% CI 67–73%). Checkable decisions: 1,085. Caveat: Local models ran quantized on one Mac; the vendors measured full-precision weights on data-centre GPUs. Latency on other hardware will differ, and Jev latency includes the network.
  12. Routing: Jev 74/82, Haiku 4.5 73/82, Sonnet 5.5 77/82 exact, and the intervals overlap. List-price cost is where they differ. Chart: Cost per 1,000 routing decisions (n = 82–246 each). Calculation, not a run. Caveat: The case sets and question wording were revised in fix waves against Jev answers on 2026-10-04 and 2026-10-05, so Jev has a home advantage.
  13. Routing overhead: an in-process policy decides in 1.42 µs; Sonnet 5.5 as a router takes 2.60 s through the CLI, about 1.8 million times longer. Chart: Time per routing decision · log scale · median to p95 (n = 82–20000 each). Caveat: The policy is timed in process and the LLM routers through a CLI: this compares the two ways of routing as deployed, not two models on equal footing. A direct API call would skip the CLI time (shown separately).
  14. Provider prices, reported by OpenRouter: open-weight models vary up to 12.6x across providers. Closed models: one price, plus a 5.5% credit fee. Largest price spread (DeepSeek V4 Flash 0423): 12.6x (15 providers · blended price). Gateway price equals the first-party price: 10 of 10. Endpoints in the snapshot: 265 endpoints (52 providers, 27 models). Caveat: Every price is third-party-reported by OpenRouter’s API at 2026-10-06. Prices change often; refetch before relying on them.
  15. Recorded SWE-bench tokens, repriced: $87.23 on Sonnet 5.5, $143.83 on Opus 5.5, $343.33 with no prompt cache. At Sonnet 5.5 prices (the model that ran): $87.23 (n = 33). At Opus 5.5 prices: $143.83 (n = 33). Sonnet 5.5 without the prompt cache: $343.33 (n = 33). Calculation, not a run. Caveat: Every repriced figure is a calculation, not a run. Only the Sonnet 5.5 row matches the model that produced the tokens.
  16. What a CLI adds: for a one-line answer the Codex CLI was 3.5x slower than the API and sent 19,551 input tokens. Codex CLI vs OpenAI API, one-line answer: 3.5x slower (n = 30). Input tokens the Codex CLI sends for one line: 19,551 (n = 15). Scheduler repair, median: Claude Code vs Codex CLI: 15.0 s vs 61.2 s (n = 3, models differ too). Caveat: Small samples: 3 to 5 runs per configuration. Medians with ranges, not intervals.
  17. Three real pull requests: the latest build verified 1 of 3, at $11.06 notional. Guardrail refusals went 26 → 19. Verified deliveries, latest build: 1 of 3 (n = 3). Notional cost, latest build, 3 tasks: $11.06 (n = 3). Guardrail refusals, first vs latest slice: 26 → 19 (n = 3). Caveat: One attempt per cell: these are defect-finding runs, not rates.
  18. Agent-loop strict passes: Haiku 54% (13/24); Sonnet 100% (16/16). Attempts excluded for outside reads: 2 of 56. Claude Haiku 4.5 strict pass rate, agent loop: 54% (13/24) (n = 24, 95% CI 35–72%). Claude Sonnet 5.5 strict pass rate, agent loop: 100% (16/16) (n = 16, 95% CI 81–100%). Agent-loop attempts left out for reading outside the work folder: 2 of 56 (n = 56). Caveat: The Claude single-call cells ran in another batch on 2026-10-06 (03:23 to 04:02 UTC), with the same CLI version, tasks and validators; provider load can differ by hour.
  19. Haiku exact routing: thinking off 87% (71/82), on 89% (73/82). Paired McNemar p = 0.754; day and account differ. Claude Haiku 4.5 (thinking off): exact routing decisions: 87% (71/82) (n = 82, 95% CI 78–92%). Claude Haiku 4.5 (thinking on): exact routing decisions: 89% (73/82) (n = 82, 95% CI 80–94%). Paired exact test, thinking off vs on (exact McNemar p): 0.754 (n = 82). Caveat: The thinking-on and Sonnet arms are the recorded routing run of 2026-10-05, reused. The thinking-off arm ran on a different day and on a different Claude subscription account, so this is a confounded comparison. Day, account, CLI version and input-token differences can affect the results. The data cannot isolate the effect of thinking.
  20. Format misses: instructions 35% (17/48), JSON schema 0% (0/48). The schema still gave wrong values: 13% (6/48). Format-miss rate with instructions only, all models: 35% (17/48) (n = 48, 95% CI 23–50%). Format-miss rate with a JSON schema, all models: 0% (0/48) (n = 48, 95% CI 0–7%). Wrong-values rate with a JSON schema, all models: 13% (6/48) (n = 48, 95% CI 6–25%). Caveat: Small samples: Haiku 24 calls per mode, Sonnet 12 calls per mode and GPT-6.1 Sol 12 calls per mode. A 12/12 result has a 95% interval of 76% to 100%. This is not a minimum detectable difference.
  21. Later Claude sessions with substantial cache reuse: new folders 0 of 2 (95% interval 0% to 66%); a fixed folder 2 of 2 (95% interval 34% to 100%). This analysis is exploratory. Later sessions with at least 50% of turn-1 input cached, A: new folder each time: 0 of 2 (95% interval 0% to 66%) (n = 2, 95% CI 0–66%). Later sessions with at least 50% of turn-1 input cached, B: fixed folder: 2 of 2 (95% interval 34% to 100%) (n = 2, 95% CI 34–100%). Caveat: The surviving protocol file was created after all counted calls. Its claimed 00:32 UTC declaration is not supported by its file birth time. Amendment 1 and 2 state 00:36 and 00:37 UTC, but separate pre-edit copies do not verify those times. The current summary was regenerated at 07:41 UTC. Treat the analysis as exploratory.
  22. Cache break-even, a calculation: a new prefix with a one-hour write needs 2 reuses (the 3rd request). Recorded session payback: 2 turns (6 of 6 sessions). Reuses before a 1-hour cached prefix costs less, whole prefix new (calculation): 2 reuses (the 3rd request). Turn at which a recorded Claude Code session’s total input cost with the cache first fell below its cost with no cache (calculation on recorded tokens): 2 turns (6 of 6 sessions) (n = 6). Calculation, not a run. Caveat: The retained Claude protocol file was created at 14:43:55 UTC, after the first counted session at 14:35:39 UTC on 2026-10-06. Its declaration says 14:25 UTC, but file times do not verify that claim. Treat this as a retrospective protocol record.
  23. Exact unseen routing decisions: Jev 82% (46/56), Haiku 79% (44/56), Sonnet 88% (49/56). The intervals overlap; no rank. Jev 1.13 (TypeSafe): exact on unseen decisions: 82% (46/56) (n = 56, 95% CI 70–90%). Claude Haiku 4.5 · Claude Code: exact on unseen decisions: 79% (44/56) (n = 56, 95% CI 66–87%). Claude Sonnet 5.5 (low) · Claude Code: exact on unseen decisions: 88% (49/56) (n = 56, 95% CI 76–94%). Caveat: The case author and two of the three routers are Claude models, so a same-family label bias is possible. A second labeller (GPT-6.1 Sol) labelled every case blind; the secondary scoring keeps only the keys where its label is inside our acceptable set.
  24. Reasoning cost at high vs low effort, a calculation from recorded calls: Sonnet 2.2x ($0.0043 at low, $0.0094 at high); Opus 3.6x ($0.0050 at low, $0.0180 at high). Reasoning cost per call, high ÷ low effort, Claude Sonnet 5.5 · Claude Code (calculation, means): 2.2x ($0.0043 at low, $0.0094 at high) (n = 16). Reasoning cost per call, high ÷ low effort, Claude Opus 5.5 · Claude Code (calculation, means): 3.6x ($0.0050 at low, $0.0180 at high) (n = 16). Calculation, not a run. Caveat: The hard-set protocol file was created after its first counted call. Both effort-ladder protocol files were created after their batches ended. The short-set file predates its first call, but its top-up amendment timing is unverified. Batch receipts preserve protocol text, but we cannot verify all rules were written before inference. Treat these as exploratory calculations, not preregistered tests.
  25. Speed anatomy, a calculation: same-text token count ratio 1.8x. Extra time to first text with the larger prompt: Haiku +0.9 s; Sonnet +1.6 s. The same 4,327-character reply: most tokens ÷ fewest tokens across models (calculation): 1.8x (n = 23). Haiku: extra time to first text at 64k vs 1k (calculation): +0.9 s (n = 6). Sonnet: extra time to first text at 64k vs 1k (calculation): +1.6 s (n = 6). Calculation, not a run. Caveat: 4 calls per model in part A and 3 per size in part B. Medians of so few calls move with one slow call, and the ranges are not confidence intervals. The 95% Wilson intervals on the lookup rates are wide.
  26. Cost per correct answer, a calculation: Sonnet every time $0.0132; Haiku with one retry then Sonnet $0.0566. No policy ran. Cost per correct answer, Sonnet 5.5 every time (calculation): $0.0132 (n = 24). Cost per correct answer, Haiku with one retry then Sonnet (calculation): $0.0566 (n = 48). Calculation, not a run. Caveat: Advance registration is not verified. The hard-set protocol file birth time is 2026-10-06 04:03:37 UTC; its first counted call started at 03:23:59 UTC. The ladder protocol file birth time is 14:35:11 UTC; its first Claude call started at 14:21:50 UTC. Both files claim advance declaration, but the available file times do not support that claim. Copying could explain the times; we cannot establish it.
  27. Strict passes on four selected harder tasks: Sol 69% (11/16); Opus 42% (5/12); Sonnet 38% (6/16). Tasks were selected with a Sonnet pilot. GPT-6.1 Sol (medium) (Codex CLI): strict pass rate on the harder tasks: 69% (11/16) (n = 16, 95% CI 44–86%). Opus 5.5 (Claude Code): strict pass rate on the harder tasks: 42% (5/12) (n = 12, 95% CI 19–68%). Sonnet 5.5 (Claude Code): strict pass rate on the harder tasks: 38% (6/16) (n = 16, 95% CI 18–61%). Caveat: Selection effect: the study picked tasks that Sonnet did not pass twice in the pilot. Sonnet passed 2 of 8 pilot calls on the kept tasks and 6 of 16 counted calls (calculation: 25% and 38%; the intervals overlap). Selection can produce this pattern, but the run does not establish its cause.
  28. Latency budget, a calculation: steps within an assumed 300 ms budget 3 of 14 (median: 3 of 14); within 1,500 ms 4 of 14 (median: 8 of 14), at p95 or the observed maximum. No voice turn ran. Steps that fit a 300 ms budget at the slow end (calculation): 3 of 14 (median: 3 of 14) (n = 14). Steps that fit a 1,500 ms budget at the slow end (calculation): 4 of 14 (median: 8 of 14) (n = 14). Calculation, not a run. Caveat: Timing samples omit 0 failed or untimed matched explorer calls and 0 failed or untimed Claude head-to-head calls. A failure is not a fast successful step. No pass rate is estimated here.
  29. Routing cost at an assumed million decisions a day, a calculation: Jev $33.70; Sonnet $7,324. No load test ran. Daily cost at 1 million decisions, Jev (calculation): $33.70 (n = 82). Daily cost at 1 million decisions, Sonnet 5.5 router (calculation): $7,324 (n = 82). Calculation, not a run. Caveat: The routing and live Jev protocols predate the first counted calls by file birth time. The overhead protocol predates its microbenchmark output. Claude ran 82 calls per router, below its 120-call cap. Jev ran 246 counted calls, below its 300-call cap. One Haiku pilot and one Jev probe are excluded. All counted calls completed without call errors. The policy microbenchmark used 5,000 warm-up iterations and 64 synthetic contexts; the minimum and individual timing samples are unavailable, so we cannot rebuild its quantiles or full range. No separate validator-control receipt or Sonnet pilot is retained.
  30. 28 studies, one place. Intervals, sources and every failure kept.
Live story · 48 sDoes more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on eight hard tasks

Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on eight hard tasks

All 11 configurations passed 16/16 strictly (95% interval 81%–100%): more effort bought no extra passes on this set (a ceiling). Median output tokens rose with effort; time ranges overlap. Costs are list-price calculations.

Live story · 48 s · download it as video (16:9, 9:16 or 1:1) from the player · See the data

Transcript
  1. Effort ladder · 176 calls · 11 configurations. Does more effort buy quality? Low, medium and high effort on eight hard tasks. Sonnet 5.5 and Opus 5.5 via Claude Code, GPT-6.1 Sol via Codex CLI.
  2. 176 of 176 calls passed strictly, at every effort level. No format misses and no wrong answers. Calls that passed strictly, every effort: 100% (176/176) (n = 176, 95% CI 98–100%). Model and effort configurations: 11 (6 new, 5 reference) (n = 176). Format misses and wrong answers: 0 · 0 (n = 176, format · wrong). Caveat: Reference cells ran in a different batch and hour than the new cells (provider load can differ), on the same host, CLI versions, cases and flags.
  3. All 11 configurations passed 16/16. A 16/16 has a 95% interval of 81%–100%: the set has a ceiling, so no effort level wins. Chart: Strict pass rate by effort · 95% Wilson intervals (n = 16 each). Caveat: Every cell passed every call, so the set has a ceiling: a perfect 16/16 has a 95% interval of 81% to 100%. This study cannot show that effort does not matter on harder work; it shows that these 8 tasks do not need more than low effort.
  4. Median time: Sonnet 5.5 5.8 s at low, 8.8 s at high; Opus 5.5 7.5 s at low, 10.1 s at high; GPT-6.1 Sol 13.6 s at low, 18.1 s at high. Within each model the ranges overlap: no effort is ranked faster. Chart: Median total time per call · whiskers: fastest to slowest call (n = 16 each). Caveat: Each cell has only 2 calls per task. Medians of 16 calls move with a few slow calls; the ranges are wide and overlap.
  5. More effort writes more: Sonnet 5.5 used a median 667 / 770 / 1,192 output tokens at low / medium / high. Chart: Median output and reasoning tokens per call (n = 16 each). Caveat: Claude Code and Codex CLI rows pair a CLI with a model; each CLI adds its own system prompt and start-up time. Effort levels are not the same scale across vendors.
  6. List-price cost per strict pass, low → high effort, a calculation: Sonnet $0.0122 → $0.0167, Opus $0.0212 → $0.0337, GPT-6.1 Sol $0.0128 → $0.0151. Chart: List-price cost per strict pass · calculation (n = 16 each). Calculation, not a run. Caveat: List-price costs are calculations; the calls used flat subscriptions.
  7. 16 effort pairs, 172 comparison rows: no effort level wins any row. The rest are ties or unclear. Effort pairs compared (same model, same route): 16. Comparison rows across those pairs: 172. Rows where one effort level wins: 0 (of 172). Caveat: Every cell passed every call, so the set has a ceiling: a perfect 16/16 has a 95% interval of 81% to 100%. This study cannot show that effort does not matter on harder work; it shows that these 8 tasks do not need more than low effort.
  8. Start at low effort and measure. Every call, interval and cost online.

Write-ups that use this data

All posts

Turn the numbers into shipped work.

Agent runs these choices for you: a persistent AI worker with memory and rules, on your Claude and Codex subscriptions.

Try it on a past pull requestSee how Agent works