AClaude Sonnet 5.5Model · AnthropicvsBClaude Opus 5.5Model · Anthropic

One attempt per arm per instance, official grader · 95% Wilson intervals

2 rows. Highest Claude Opus 5.5 (Agent, new build) 67% (95% interval 21%–94%, n 3). Lowest Claude Sonnet 5.5 (Agent, older builds) 33% (95% interval 6.2%–79%, n 3). All intervals overlap.

Interim: 3 of 8 declared pairs are graded; 5 were never started; no resumed attempts are included here. With n = 3 the intervals span most of the axis, so this chart supports no ranking. The instances were chosen to hold Sonnet misses and resolves in equal numbers, so neither rate estimates SWE-bench Verified as a whole.

Sources: Agent on SWE-bench Verified with Opus 5.5 as the brain, paired with Sonnet 5.5 (interim), Agent on SWE-bench Verified, campaign 1 (25 instances), Agent on SWE-bench Verified, campaign 2 (8 compiled-extension instances)

Mean over the same 3 instances; notional cost from the platform price table

List-price calculation, not a run. 2 rows. Highest Claude Opus 5.5 (Agent, new build) $7.59 (n 3). Lowest Claude Sonnet 5.5 (Agent, older builds) $2.88 (n 3).

Calculation, not an invoice: the platform price table at each run commit applied to the recorded tokens of subscription runs (Opus 5.5: $4 input, $20 output, $0.20 cache read per million tokens). Totals $22.76 vs $8.64: 2.6×, a ratio of two calculations.

Sources: Agent on SWE-bench Verified with Opus 5.5 as the brain, paired with Sonnet 5.5 (interim), Agent on SWE-bench Verified, campaign 1 (25 instances), Agent on SWE-bench Verified, campaign 2 (8 compiled-extension instances), Repricing calculation, Anthropic list prices (Claude models)

Median minutes; whiskers = fastest and slowest of 3 attempts (not an interval)

2 rows. Slowest Claude Opus 5.5 (Agent, new build) 20.3 min (range 9.8 min–25.3 min, n 3). Fastest Claude Sonnet 5.5 (Agent, older builds) 9.4 min (range 4.7 min–15 min, n 3). All run ranges overlap.

Worker minutes from the run reports. Opus attempts ran one at a time on the newer build; the Sonnet attempts ran in earlier campaigns. 3 attempts per arm is too few to call a difference; a range is not a confidence interval.

Sources: Agent on SWE-bench Verified with Opus 5.5 as the brain, paired with Sonnet 5.5 (interim), Agent on SWE-bench Verified, campaign 1 (25 instances), Agent on SWE-bench Verified, campaign 2 (8 compiled-extension instances)

Every call counts; failures and timeouts are non-passes

2 rows. Highest Claude Opus 5.5 (high) · Claude Code 100% (95% interval 80%–100%, n 15). Lowest Claude Sonnet 5.5 · Claude Code 80% (95% interval 55%–93%, n 15). All intervals overlap.

Whiskers are 95% Wilson intervals. The tasks are short and most configurations pass nearly all of them, so pass rate does not separate the models here.

Source: Provider head-to-head: Claude Code models vs Codex efforts

Median per configuration; whiskers = fastest and slowest call

2 rows. Slowest Claude Opus 5.5 · Claude Code 2.8 s (range 2.5 s–8.9 s, n 15). Fastest Claude Sonnet 5.5 · Claude Code 2.3 s (range 2.2 s–7.7 s, n 15). All run ranges overlap.

One host, one network, one day. Whiskers are a range, not a confidence interval.

Source: Provider head-to-head: Claude Code models vs Codex efforts

Median per configuration; whiskers = fastest and slowest call

2 rows. Slowest Claude Opus 5.5 · Claude Code 1.9 s (range 1.6 s–7.2 s, n 15). Fastest Claude Sonnet 5.5 · Claude Code 1.6 s (range 1 s–6.4 s, n 15). All run ranges overlap.

One host, one network, one day. Whiskers are a range, not a confidence interval.

Source: Provider head-to-head: Claude Code models vs Codex efforts

Mean per call, split into prompt-cache reads and other input

2 rows, 2 series: Cache read, Other input. Cache read: all at 1,401. Other input: highest Claude Sonnet 5.5 · Claude Code 685 (n 15). Lowest Claude Opus 5.5 · Claude Code 680 (n 15).

The prompts are a few hundred tokens; most input is the CLI’s own system prompt and tool context. Input counts include cache reads, as the vendors report them.

Source: Provider head-to-head: Claude Code models vs Codex efforts

Median per configuration; reasoning tokens where the CLI reports them

2 rows, 2 series: Output tokens, Reasoning tokens. Output tokens: highest Claude Sonnet 5.5 · Claude Code 107 (n 15). Lowest Claude Opus 5.5 · Claude Code 64 (n 15). Reasoning tokens: all at 0.

Vendors count reasoning differently; compare within a vendor first. A median of 0 can mean the CLI did not report reasoning for most calls.

Source: Provider head-to-head: Claude Code models vs Codex efforts

Reported tokens × list price; the calls ran on subscriptions

List-price calculation, not a run. 2 rows. Highest Claude Opus 5.5 · Claude Code $0.0069 (range $0.0059–$0.022, n 15). Lowest Claude Sonnet 5.5 · Claude Code $0.0036 (range $0.0034–$0.01, n 15). All run ranges overlap.

Calculation, not a bill: the calls ran on flat subscriptions. Whiskers = cheapest and most expensive call.

Sources: Provider head-to-head: Claude Code models vs Codex efforts, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices

All calls in a configuration, failures included, divided by its passes

List-price calculation, not a run. 2 rows. Highest Claude Opus 5.5 · Claude Code $0.01 (n 15). Lowest Claude Sonnet 5.5 · Claude Code $0.0062 (n 15).

Calculation, not a bill: reported tokens × list price; the calls ran on flat subscriptions. A failed call still costs, so a lower pass rate raises the cost per pass. Highlighted bars are on the speed/cost/quality frontier.

Sources: Provider head-to-head: Claude Code models vs Codex efforts, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices

Strict: the reply passes as given. Lenient: a correct answer in the wrong format also counts

2 rows, 2 series: Strict pass, Lenient (format misses counted). Strict pass: all at 100%. Lenient (format misses counted): all at 100%.

Whiskers are 95% Wilson intervals. The strict pass is the result; the lenient reading is shown so a reader can see how much is format and how much is a wrong answer. A format miss never counts as a pass.

Source: Provider head-to-head, hard set: eight hard tasks with strict validators

Median per configuration; whiskers = fastest and slowest call

2 rows. Slowest Claude Opus 5.5 · Claude Code 9.2 s (range 4.2 s–27.2 s, n 24). Fastest Claude Sonnet 5.5 · Claude Code 7.8 s (range 2.3 s–34.8 s, n 24). All run ranges overlap.

One host and network; the counted Claude and Codex batches ran hours apart. Host load was not controlled. Whiskers are a range, not a confidence interval. Highlighted: configurations that passed every call.

Source: Provider head-to-head, hard set: eight hard tasks with strict validators

Median per configuration; whiskers = fastest and slowest call

2 rows. Slowest Claude Opus 5.5 · Claude Code 6.8 s (range 2.4 s–21.8 s, n 24). Fastest Claude Sonnet 5.5 · Claude Code 6 s (range 0.9 s–30.6 s, n 24). All run ranges overlap.

One host and network; the counted Claude and Codex batches ran hours apart. Host load was not controlled. Whiskers are a range, not a confidence interval. Highlighted: configurations that passed every call.

Source: Provider head-to-head, hard set: eight hard tasks with strict validators

Median per configuration; reasoning tokens as the CLI reports them

2 rows, 2 series: Output tokens, Reasoning tokens. Output tokens: highest Claude Sonnet 5.5 · Claude Code 1,050 (n 24). Lowest Claude Opus 5.5 · Claude Code 945 (n 24). Reasoning tokens: highest Claude Sonnet 5.5 · Claude Code 585 (n 24). Lowest Claude Opus 5.5 · Claude Code 529 (n 24).

Reasoning tokens are part of the output tokens where the CLI reports them. Their content is never captured. More tokens is not better or worse by itself.

Source: Provider head-to-head, hard set: eight hard tasks with strict validators

All calls in a configuration, failures and format misses included, divided by its strict passes

List-price calculation, not a run. 2 rows. Highest Claude Opus 5.5 · Claude Code $0.028 (n 24). Lowest Claude Sonnet 5.5 · Claude Code $0.014 (n 24).

Calculation, not a bill: reported tokens × list price; the calls ran on a flat subscription. A failed call still costs, so a lower pass rate raises the cost per pass. Highlighted bars are on the quality-vs-cost frontier.

Sources: Provider head-to-head, hard set: eight hard tasks with strict validators, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices

A pass needs every hidden check · 12 sessions per agent · 95% Wilson intervals

2 rows. All at 100%.

6 tasks × 2 repetitions per agent. All 36 sessions passed, so the chart sits at its ceiling: this task set cannot rank the agents on quality. Before the first session, every base repository failed its hidden checks and every reference patch passed them.

Source: Coding agents head-to-head: Claude Code and Codex CLI on 6 hidden-test tasks

Median wall time; whiskers = fastest and slowest of 12 sessions (not an interval)

2 rows. Slowest Claude Opus 5.5 · Claude Code 56.9 s (range 29.8 s–186 s, n 12). Fastest Claude Sonnet 5.5 · Claude Code 23.1 s (range 18.7 s–44.5 s, n 12). All run ranges overlap.

CLI process start to exit. One session at a time per lane; the Claude Code and Codex CLI lanes ran at the same time on one machine with different accounts. Codex time includes reading the tester’s notes, writing a work log and trying to commit. A range is not a confidence interval.

Source: Coding agents head-to-head: Claude Code and Codex CLI on 6 hidden-test tasks

Median; whiskers = fewest and most of 12 sessions (not an interval)

2 rows. All at 7.5.

Claude Code counts its tool calls (Bash, Read, Edit, Write, Glob, Grep). Codex CLI counts shell commands and file changes; it has no separate read tool, so it reads files with shell commands. Turns are not compared: Codex reports one turn per run.

Source: Coding agents head-to-head: Claude Code and Codex CLI on 6 hidden-test tasks

Reported tokens of all 12 sessions × list price, divided by the passes

List-price calculation, not a run. 2 rows. Highest Claude Opus 5.5 · Claude Code $0.22 (n 12). Lowest Claude Sonnet 5.5 · Claude Code $0.085 (n 12).

Calculation, not a bill: both CLIs ran on flat subscriptions. Claude cache writes are priced at 2× input, as in the other studies (Claude Code’s own estimate gives the same totals); Codex cached input at its cache-read price. Codex input includes its own system prompt and, here, the tester’s AGENTS.md.

Sources: Coding agents head-to-head: Claude Code and Codex CLI on 6 hidden-test tasks, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices

Every configuration: 8 tasks × 2 repetitions. Whiskers are 95% Wilson intervals

2 rows. All at 100%.

Whiskers are 95% Wilson intervals. A format miss never counts as a pass. Reference cells (efforts the hard head-to-head already ran) are reused, not rerun; Claude reference cells keep repetitions 1-2, so every cell is 8 tasks × 2 repetitions.

Source: Effort ladder: the hard task set at each effort level

Median per configuration; whiskers = fastest and slowest call

2 rows. Slowest Claude Opus 5.5 · Claude Code 9.2 s (range 4.2 s–27.2 s, n 16). Fastest Claude Sonnet 5.5 · Claude Code 8 s (range 2.3 s–21.6 s, n 16). All run ranges overlap.

Whiskers are a range (fastest and slowest call), not a confidence interval. One host, one network. Reference cells (efforts the hard head-to-head already ran) are reused, not rerun; Claude reference cells keep repetitions 1-2, so every cell is 8 tasks × 2 repetitions. The reference cells ran in a different hour.

Source: Effort ladder: the hard task set at each effort level

Median per configuration; reasoning tokens as the CLI reports them

2 rows, 2 series: Output tokens, Reasoning tokens. Output tokens: highest Claude Sonnet 5.5 · Claude Code 1,054 (n 16). Lowest Claude Opus 5.5 · Claude Code 945 (n 16). Reasoning tokens: highest Claude Sonnet 5.5 · Claude Code 668 (n 16). Lowest Claude Opus 5.5 · Claude Code 538 (n 16).

Reasoning tokens are part of the output tokens where the CLI reports them; 0 can mean "not reported". Their content is never captured. More tokens is not better or worse by itself.

Source: Effort ladder: the hard task set at each effort level

All calls in a configuration divided by its strict passes

List-price calculation, not a run. 2 rows. Highest Claude Opus 5.5 · Claude Code $0.029 (n 16). Lowest Claude Sonnet 5.5 · Claude Code $0.014 (n 16).

Calculation, not a bill: reported tokens × list price, cache reads and writes priced as in the hard head-to-head; the calls ran on flat subscriptions. Codex input includes its own system prompt and tool schemas, so a Claude-vs-Codex gap is partly the CLI.

Sources: Effort ladder: the hard task set at each effort level, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices

All recorded turns per model; the same reported tokens priced two ways

List-price calculation, not a run. 2 rows, 2 series: With the cache, as recorded, Without a cache: every input token at the input price. With the cache, as recorded: highest Claude Opus 5.5 · Claude Code $0.26 (n 15). Lowest Claude Sonnet 5.5 · Claude Code $0.14 (n 15). Without a cache: every input token at the input price: highest Claude Opus 5.5 · Claude Code $0.54 (n 15). Lowest Claude Sonnet 5.5 · Claude Code $0.27 (n 15).

Calculation, not a bill: the calls ran on a subscription. Cache reads at the cache-read price, 1-hour cache writes at 2× the input price (every write in this run was a 1-hour write). Codex CLI is not priced here: it reports no cache-write count.

Sources: Caching sessions and repeated prompts (Claude Code and Codex CLI), Cost with and without the prompt cache (calculation), Anthropic list prices (Claude models)

Median; whiskers = fastest and slowest turn

2 rows, 2 series: Turn 1 (writes the ledger to the cache), Turns 2-5 (read the ledger from the cache). Turn 1 (writes the ledger to the cache): slowest Claude Opus 5.5 · Claude Code 1.9 s (range 1.8 s–4.4 s, n 3). Fastest Claude Sonnet 5.5 · Claude Code 1.6 s (range 1.6 s–1.8 s, n 3). All run ranges overlap. Turns 2-5 (read the ledger from the cache): slowest Claude Opus 5.5 · Claude Code 2.4 s (range 1.6 s–12.7 s, n 12). Fastest Claude Sonnet 5.5 · Claude Code 1.6 s (range 1.4 s–5.6 s, n 12). All run ranges overlap.

Whiskers are a range (fastest and slowest turn), not a confidence interval. Turns ask different questions: the slow later turns are the counting question, which produced the most output.

Source: Caching sessions and repeated prompts (Claude Code and Codex CLI)

Prefix of 7,831 tokens (Sonnet 5.5) and 7,828 tokens (Opus 5.5), the rounded mean turn-1 input; n = 3 Sonnet sessions (range 7,831 to 7,832) and 3 Opus sessions (the same in every session); USD per 1,000 sessions

List-price calculation, not a run. 6 rows, 5 series: Claude Sonnet 5.5 (no cache), Claude Sonnet 5.5 (1-hour cache write), Claude Opus 5.5 (no cache), Claude Opus 5.5 (1-hour cache write, read $0.2 per M), Claude Opus 5.5 (1-hour cache write, read $0.4 per M). Claude Sonnet 5.5 (no cache): highest 20 turns $313. Lowest 1 turn $15.66. Claude Sonnet 5.5 (1-hour cache write): highest 20 turns $61.08. Lowest 1 turn $31.32.

Calculation, not a bill. The chart multiplies list prices by a prefix of 7,831 tokens (Sonnet) or 7,828 tokens (Opus). It uses a 1-hour write at 2× input and treats the whole prefix as new. It leaves out output and the tokens that each turn adds. The cache costs more at 1 to 2 turns and less from 3. The second Opus line uses a $0.4 cache read, which another table of the product lists. We did not check the vendor price. The table adds the 5-minute write.

Sources: Cost with and without the prompt cache (calculation), Caching sessions and repeated prompts (Claude Code and Codex CLI), Anthropic list prices (Claude models), OpenAI list prices

USD per 1,000 workloads of 10 requests that send the same prefix; each recorded session started in a new temporary folder, n = 4 later sessions; 0 met the read-more-than-write proxy, 95% Wilson 0% to 49%

List-price calculation, not a run. 2 rows, 3 series: No cache (the same either way), One 10-turn session, 1-hour cache, Ten 1-turn sessions, 1-hour cache (assumed no reuse of the new prefix). No cache (the same either way): highest Claude Opus 5.5 (cache read $0.2 per M) $313. Lowest Claude Sonnet 5.5 $157. One 10-turn session, 1-hour cache: highest Claude Opus 5.5 (cache read $0.2 per M) $76.71. Lowest Claude Sonnet 5.5 $45.42.

Calculation, not a bill: the same prefix, prices and 1-hour write as the cost-curve chart. Each recorded session ran in a new temporary working folder. On turn 1, 0 of 4 later sessions read more tokens than they wrote. This proxy cannot identify which earlier call supplied a cache entry. The split calculation assumes no shared reuse and a wholly new prefix, so it charges ten full writes. With reuse across sessions they would cost the same as one 10-turn session. The cause of the missing reuse was not tested here. The caching study protocol names the random temporary folder as one untested hypothesis. The table adds the 5-minute variants.

Sources: Cost with and without the prompt cache (calculation), Caching sessions and repeated prompts (Claude Code and Codex CLI), Anthropic list prices (Claude models), OpenAI list prices

Median call: reasoning tokens ÷ output tokens. Whiskers: lowest and highest call (16 to 24 calls per configuration)

List-price calculation, not a run. 2 rows. Highest Claude Opus 5.5 · Claude Code 55% (range 30%–96%, n 24). Lowest Claude Sonnet 5.5 · Claude Code 55% (range 0%–96%, n 24). All run ranges overlap.

Calculation from reported tokens, not a run. Each call gives reasoning ÷ output; the bar is the median of those shares. Whiskers are the lowest and highest call. They are a range, not a confidence interval. They are wide, so the medians describe this run and rank nothing. The pooled share (all reasoning tokens ÷ all output tokens) is in the table. We treat reasoning tokens as part of output tokens; the consistency check supports this accounting assumption. Each CLI reports its own count.

Sources: Reasoning token bill (calculation), Provider head-to-head, hard set: eight hard tasks with strict validators, Effort ladder: the hard task set at each effort level, Provider head-to-head: Claude Code models vs Codex efforts, Anthropic list prices (Claude models), OpenAI list prices

Mean per call on the hard tasks; the three parts add up to the call

List-price calculation, not a run. 2 rows, 3 series: Reasoning (output tokens), Remaining output (visible-answer estimate), Input (prompt, cache priced). Reasoning (output tokens): highest Claude Opus 5.5 · Claude Code $0.013 (n 24). Lowest Claude Sonnet 5.5 · Claude Code $0.0067 (n 24). Remaining output (visible-answer estimate): highest Claude Opus 5.5 · Claude Code $0.008 (n 24). Lowest Claude Sonnet 5.5 · Claude Code $0.0037 (n 24).

Calculation, not a bill: reported tokens × list price (per million output tokens: Haiku 4.5 $5, Sonnet 5.5 $10, Opus 5.5 $20, Fable 5.1 $50 and GPT-6.1 Sol $10); the calls ran on flat subscriptions. Reasoning and remaining output both use the output price. The remainder estimates visible-answer tokens; its exact meaning depends on the CLI counters. Input is the whole prompt, with cache reads and writes priced as in the hard head-to-head. It includes CLI context; these receipts do not separate task tokens from CLI context. It changes with cache counters. Batch timing and cache behavior were not controlled. Means, not medians, so the parts add up.

Sources: Reasoning token bill (calculation), Provider head-to-head, hard set: eight hard tasks with strict validators, Effort ladder: the hard task set at each effort level, Provider head-to-head: Claude Code models vs Codex efforts, Anthropic list prices (Claude models), OpenAI list prices

List price ÷ strict passes; every cell is 8 tasks × 2 repetitions

List-price calculation, not a run. 2 rows, 2 series: Reasoning cost per strict pass, Total cost per strict pass. Reasoning cost per strict pass: highest Claude Opus 5.5 · Claude Code $0.013 (n 16). Lowest Claude Sonnet 5.5 · Claude Code $0.0063 (n 16). Total cost per strict pass: highest Claude Opus 5.5 · Claude Code $0.029 (n 16). Lowest Claude Sonnet 5.5 · Claude Code $0.014 (n 16).

Calculation, not a bill: reported tokens × list price, divided by the cell's strict passes; the calls ran on flat subscriptions. The effort-ladder cells: new calls plus reference cells reused from the hard head-to-head (Claude repetitions 1-2 only). "Default" means the effort flag was not passed. The total is the same value as the effort-ladder cost-per-pass chart. Effort levels are not the same scale across vendors, and the reference cells ran in a different batch and hour.

Sources: Reasoning token bill (calculation), Provider head-to-head, hard set: eight hard tasks with strict validators, Effort ladder: the hard task set at each effort level, Provider head-to-head: Claude Code models vs Codex efforts, Anthropic list prices (Claude models), OpenAI list prices

Median call per configuration; five short tasks and eight hard tasks

List-price calculation, not a run. 2 rows, 2 series: Eight hard tasks, Five short tasks. Eight hard tasks: highest Claude Opus 5.5 · Claude Code 55% (range 30%–96%, n 24). Lowest Claude Sonnet 5.5 · Claude Code 55% (range 0%–96%, n 24). All run ranges overlap. Five short tasks: all at 0%.

Calculation from reported tokens, not a run: the median of per-call reasoning ÷ output. A median of 0% means at least half of the calls reported 0 reasoning tokens in the receipt. On a route that reports reasoning, we retain a numeric 0 as the recorded counter. It does not prove the model did no internal reasoning. The table says how many calls reported 0. The short tasks are five small validated tasks (a bug fix, a JSON extraction, an arithmetic problem, a refactor and a ticket classification). Per-call ranges are in the tables.

Sources: Reasoning token bill (calculation), Provider head-to-head, hard set: eight hard tasks with strict validators, Effort ladder: the hard task set at each effort level, Provider head-to-head: Claude Code models vs Codex efforts, Anthropic list prices (Claude models), OpenAI list prices

Median of 4 calls per model; whiskers = fastest and slowest call

2 rows. Slowest Claude Opus 5.5 · Claude Code 2 s (range 1.7 s–2.4 s, n 4). Fastest Claude Sonnet 5.5 · Claude Code 2 s (range 0.9 s–4.1 s, n 4). All run ranges overlap.

Dot = median; whiskers = fastest and slowest call (a range, not a confidence interval). The clock starts when the CLI starts, so first text includes CLI start-up (see chart cli-startup-tax) and any reasoning before the first word. One Mac, one network, two sessions on one night.

Source: LLM speed anatomy

Median of 4 calls per model; whiskers = slowest and fastest call

List-price calculation, not a run. 2 rows. Highest Claude Sonnet 5.5 · Claude Code 232 (range 230–233, n 4). Lowest Claude Opus 5.5 · Claude Code 156 (range 155–156, n 4). Not all run ranges overlap.

Calculation, not a measurement: visible output tokens (the CLI’s output token count minus its reported reasoning tokens) divided by the time from the first text to the end of the call. The denominator includes the CLI’s exit overhead; its size is not measured here. Whiskers are a range of calls, not a confidence interval. The calculation removes reported reasoning tokens. These receipts do not locate all reasoning in time.

Source: LLM speed anatomy

Only replies that matched all 250 lines; median per model, whiskers = slowest and fastest call

List-price calculation, not a run. 2 rows. Highest Claude Sonnet 5.5 · Claude Code 517 (range 513–519, n 4). Lowest Claude Opus 5.5 · Claude Code 347 (range 345–349, n 4). Not all run ranges overlap.

Calculation, not a measurement: the characters of a correct reply (the same text for every model) divided by the time from the first text to the end of the call. Each vendor counts the same text as a different number of tokens, so characters compare across models where tokens do not. The CLI’s exit time is inside that time. Whiskers are a range of calls, not a confidence interval.

Source: LLM speed anatomy

Median of 3 calls per size; whiskers = fastest and slowest call

List-price calculation, not a run. 3 rows, 4 series: Claude Haiku 4.5 · Claude Code, Claude Sonnet 5.5 · Claude Code, Claude Opus 5.5 · Claude Code, GPT-6.1 Sol (low) · Codex CLI. Claude Haiku 4.5 · Claude Code: slowest 64k 2.8 s (range 2.5 s–2.9 s, n 3). Fastest 1k 1.9 s (range 1.9 s–2 s, n 3). Not all run ranges overlap. Claude Sonnet 5.5 · Claude Code: slowest 64k 3.1 s (range 1.4 s–3.6 s, n 3). Fastest 1k 1.5 s (range 1.2 s–1.7 s, n 3). All run ranges overlap.

Each call used a new ledger seed. Cache-read counts stayed within the short-prompt baseline (see the cache table). This does not identify which tokens were cached. Sizes name the text we send; each model’s reported input tokens are in the table and include the CLI’s own prefix. The size calibration subtracts estimated prefixes from probe input counts; these are calculations, not measured prefix counts for each call. Whiskers are a range of calls, not a confidence interval.

Source: LLM speed anatomy

Median of 3 calls per bar; whiskers = fastest and slowest call

2 rows, 3 series: 1k prompt, 16k prompt, 64k prompt. 1k prompt: slowest Claude Opus 5.5 · Claude Code 1.8 s (range 1.8 s–2.4 s, n 3). Fastest Claude Sonnet 5.5 · Claude Code 1.8 s (range 1.6 s–2.1 s, n 3). All run ranges overlap. 16k prompt: slowest Claude Opus 5.5 · Claude Code 2.4 s (range 2.1 s–3.4 s, n 3). Fastest Claude Sonnet 5.5 · Claude Code 2.1 s (range 2 s–2.5 s, n 3). All run ranges overlap.

Whole call: CLI start-up, first text and the one-line answer. Whiskers are a range of calls, not a confidence interval. Each call used a new ledger.

Source: LLM speed anatomy

All sizes together per model; whiskers = 95% Wilson intervals

2 rows. Highest Claude Sonnet 5.5 · Claude Code 100% (95% interval 70%–100%, n 9). Lowest Claude Opus 5.5 · Claude Code 56% (95% interval 27%–81%, n 9). All intervals overlap.

Whiskers are 95% Wilson intervals. One lookup question per call; a reply with extra words is a format miss, not a pass. With 9 calls per model, a perfect score still has a wide interval.

Source: LLM speed anatomy