AClaude Opus 5.5Model · AnthropicvsBGPT-6.1 Sol (Codex CLI)Model · OpenAI

Every call counts; failures and timeouts are non-passes

2 rows. All at 100%.

Whiskers are 95% Wilson intervals. The tasks are short and most configurations pass nearly all of them, so pass rate does not separate the models here.

Source: Provider head-to-head: Claude Code models vs Codex efforts

Median per configuration; whiskers = fastest and slowest call

2 rows. Slowest GPT-6.1 Sol (high) · Codex CLI 5.6 s (range 4.1 s–19.5 s, n 15). Fastest Claude Opus 5.5 (high) · Claude Code 2.7 s (range 2.5 s–11.8 s, n 15). All run ranges overlap.

One host, one network, one day. Whiskers are a range, not a confidence interval.

Source: Provider head-to-head: Claude Code models vs Codex efforts

Median per configuration; whiskers = fastest and slowest call

2 rows. Slowest GPT-6.1 Sol (high) · Codex CLI 5.3 s (range 3.6 s–16.4 s, n 15). Fastest Claude Opus 5.5 (high) · Claude Code 2 s (range 1.4 s–9.9 s, n 15). All run ranges overlap.

One host, one network, one day. Whiskers are a range, not a confidence interval.

Source: Provider head-to-head: Claude Code models vs Codex efforts

Mean per call, split into prompt-cache reads and other input

2 rows, 2 series: Cache read, Other input. Cache read: highest GPT-6.1 Sol (high) · Codex CLI 6,716 (n 15). Lowest Claude Opus 5.5 (high) · Claude Code 1,463 (n 15). Other input: highest GPT-6.1 Sol (high) · Codex CLI 5,406 (n 15). Lowest Claude Opus 5.5 (high) · Claude Code 619 (n 15).

The prompts are a few hundred tokens; most input is the CLI’s own system prompt and tool context. Input counts include cache reads, as the vendors report them.

Source: Provider head-to-head: Claude Code models vs Codex efforts

Median per configuration; reasoning tokens where the CLI reports them

2 rows, 2 series: Output tokens, Reasoning tokens. Output tokens: highest Claude Opus 5.5 (high) · Claude Code 78 (n 15). Lowest GPT-6.1 Sol (high) · Codex CLI 42 (n 15). Reasoning tokens: highest Claude Opus 5.5 (high) · Claude Code 34 (n 15). Lowest GPT-6.1 Sol (high) · Codex CLI 21 (n 15).

Vendors count reasoning differently; compare within a vendor first. A median of 0 can mean the CLI did not report reasoning for most calls.

Source: Provider head-to-head: Claude Code models vs Codex efforts

Reported tokens × list price; the calls ran on subscriptions

List-price calculation, not a run. 2 rows. Highest GPT-6.1 Sol (high) · Codex CLI $0.01 (range $0.0066–$0.028, n 15). Lowest Claude Opus 5.5 (high) · Claude Code $0.0069 (range $0.0059–$0.027, n 15). All run ranges overlap.

Calculation, not a bill: the calls ran on flat subscriptions. Whiskers = cheapest and most expensive call.

Sources: Provider head-to-head: Claude Code models vs Codex efforts, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices

All calls in a configuration, failures included, divided by its passes

List-price calculation, not a run. 2 rows. Highest GPT-6.1 Sol (high) · Codex CLI $0.013 (n 15). Lowest Claude Opus 5.5 (high) · Claude Code $0.01 (n 15).

Calculation, not a bill: reported tokens × list price; the calls ran on flat subscriptions. A failed call still costs, so a lower pass rate raises the cost per pass. Highlighted bars are on the speed/cost/quality frontier.

Sources: Provider head-to-head: Claude Code models vs Codex efforts, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices

Strict: the reply passes as given. Lenient: a correct answer in the wrong format also counts

2 rows, 2 series: Strict pass, Lenient (format misses counted). Strict pass: all at 100%. Lenient (format misses counted): all at 100%.

Whiskers are 95% Wilson intervals. The strict pass is the result; the lenient reading is shown so a reader can see how much is format and how much is a wrong answer. A format miss never counts as a pass.

Source: Provider head-to-head, hard set: eight hard tasks with strict validators

Median per configuration; whiskers = fastest and slowest call

2 rows. Slowest GPT-6.1 Sol (high) · Codex CLI 18.1 s (range 11.7 s–92.2 s, n 16). Fastest Claude Opus 5.5 (high) · Claude Code 11 s (range 3.6 s–63 s, n 24). All run ranges overlap.

One host and network; the counted Claude and Codex batches ran hours apart. Host load was not controlled. Whiskers are a range, not a confidence interval. Highlighted: configurations that passed every call.

Source: Provider head-to-head, hard set: eight hard tasks with strict validators

Median per configuration; whiskers = fastest and slowest call

2 rows. Slowest GPT-6.1 Sol (high) · Codex CLI 12.7 s (range 8.9 s–75.9 s, n 16). Fastest Claude Opus 5.5 (high) · Claude Code 7.1 s (range 2.2 s–56.2 s, n 24). All run ranges overlap.

One host and network; the counted Claude and Codex batches ran hours apart. Host load was not controlled. Whiskers are a range, not a confidence interval. Highlighted: configurations that passed every call.

Source: Provider head-to-head, hard set: eight hard tasks with strict validators

Median per configuration; reasoning tokens as the CLI reports them

2 rows, 2 series: Output tokens, Reasoning tokens. Output tokens: highest Claude Opus 5.5 (high) · Claude Code 1,052 (n 24). Lowest GPT-6.1 Sol (high) · Codex CLI 436 (n 16). Reasoning tokens: highest Claude Opus 5.5 (high) · Claude Code 614 (n 24). Lowest GPT-6.1 Sol (high) · Codex CLI 225 (n 16).

Reasoning tokens are part of the output tokens where the CLI reports them. Their content is never captured. More tokens is not better or worse by itself.

Source: Provider head-to-head, hard set: eight hard tasks with strict validators

All calls in a configuration, failures and format misses included, divided by its strict passes

List-price calculation, not a run. 2 rows. Highest Claude Opus 5.5 (high) · Claude Code $0.033 (n 24). Lowest GPT-6.1 Sol (high) · Codex CLI $0.015 (n 16).

Calculation, not a bill: reported tokens × list price; the calls ran on a flat subscription. A failed call still costs, so a lower pass rate raises the cost per pass. Highlighted bars are on the quality-vs-cost frontier.

Sources: Provider head-to-head, hard set: eight hard tasks with strict validators, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices

A pass needs every hidden check · 12 sessions per agent · 95% Wilson intervals

2 rows. All at 100%.

6 tasks × 2 repetitions per agent. All 36 sessions passed, so the chart sits at its ceiling: this task set cannot rank the agents on quality. Before the first session, every base repository failed its hidden checks and every reference patch passed them.

Source: Coding agents head-to-head: Claude Code and Codex CLI on 6 hidden-test tasks

Median wall time; whiskers = fastest and slowest of 12 sessions (not an interval)

2 rows. Slowest GPT-6.1 Sol (medium, tester’s AGENTS.md) · Codex CLI 113 s (range 78.5 s–222 s, n 12). Fastest Claude Opus 5.5 · Claude Code 56.9 s (range 29.8 s–186 s, n 12). All run ranges overlap.

CLI process start to exit. One session at a time per lane; the Claude Code and Codex CLI lanes ran at the same time on one machine with different accounts. Codex time includes reading the tester’s notes, writing a work log and trying to commit. A range is not a confidence interval.

Source: Coding agents head-to-head: Claude Code and Codex CLI on 6 hidden-test tasks

Median; whiskers = fewest and most of 12 sessions (not an interval)

2 rows. Highest GPT-6.1 Sol (medium, tester’s AGENTS.md) · Codex CLI 12.5 (range 8–18, n 12). Lowest Claude Opus 5.5 · Claude Code 7.5 (range 5–14, n 12). All run ranges overlap.

Claude Code counts its tool calls (Bash, Read, Edit, Write, Glob, Grep). Codex CLI counts shell commands and file changes; it has no separate read tool, so it reads files with shell commands. Turns are not compared: Codex reports one turn per run.

Source: Coding agents head-to-head: Claude Code and Codex CLI on 6 hidden-test tasks

Reported tokens of all 12 sessions × list price, divided by the passes

List-price calculation, not a run. 2 rows. Highest Claude Opus 5.5 · Claude Code $0.22 (n 12). Lowest GPT-6.1 Sol (medium, tester’s AGENTS.md) · Codex CLI $0.098 (n 12).

Calculation, not a bill: both CLIs ran on flat subscriptions. Claude cache writes are priced at 2× input, as in the other studies (Claude Code’s own estimate gives the same totals); Codex cached input at its cache-read price. Codex input includes its own system prompt and, here, the tester’s AGENTS.md.

Sources: Coding agents head-to-head: Claude Code and Codex CLI on 6 hidden-test tasks, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices

Every configuration: 8 tasks × 2 repetitions. Whiskers are 95% Wilson intervals

2 rows. All at 100%.

Whiskers are 95% Wilson intervals. A format miss never counts as a pass. Reference cells (efforts the hard head-to-head already ran) are reused, not rerun; Claude reference cells keep repetitions 1-2, so every cell is 8 tasks × 2 repetitions.

Source: Effort ladder: the hard task set at each effort level

Median per configuration; whiskers = fastest and slowest call

2 rows. Slowest GPT-6.1 Sol (medium) · Codex CLI 13.1 s (range 8.5 s–61.6 s, n 16). Fastest Claude Opus 5.5 (medium) · Claude Code 9.7 s (range 4.8 s–31.4 s, n 16). All run ranges overlap.

Whiskers are a range (fastest and slowest call), not a confidence interval. One host, one network. Reference cells (efforts the hard head-to-head already ran) are reused, not rerun; Claude reference cells keep repetitions 1-2, so every cell is 8 tasks × 2 repetitions. The reference cells ran in a different hour.

Source: Effort ladder: the hard task set at each effort level

Median per configuration; reasoning tokens as the CLI reports them

2 rows, 2 series: Output tokens, Reasoning tokens. Output tokens: highest Claude Opus 5.5 (medium) · Claude Code 853 (n 16). Lowest GPT-6.1 Sol (medium) · Codex CLI 335 (n 16). Reasoning tokens: highest Claude Opus 5.5 (medium) · Claude Code 518 (n 16). Lowest GPT-6.1 Sol (medium) · Codex CLI 150 (n 16).

Reasoning tokens are part of the output tokens where the CLI reports them; 0 can mean "not reported". Their content is never captured. More tokens is not better or worse by itself.

Source: Effort ladder: the hard task set at each effort level

All calls in a configuration divided by its strict passes

List-price calculation, not a run. 2 rows. Highest Claude Opus 5.5 (medium) · Claude Code $0.029 (n 16). Lowest GPT-6.1 Sol (medium) · Codex CLI $0.026 (n 16).

Calculation, not a bill: reported tokens × list price, cache reads and writes priced as in the hard head-to-head; the calls ran on flat subscriptions. Codex input includes its own system prompt and tool schemas, so a Claude-vs-Codex gap is partly the CLI.

Sources: Effort ladder: the hard task set at each effort level, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices

Median call: reasoning tokens ÷ output tokens. Whiskers: lowest and highest call (16 to 24 calls per configuration)

List-price calculation, not a run. 2 rows. Highest GPT-6.1 Sol (high) · Codex CLI 57% (range 29%–91%, n 16). Lowest Claude Opus 5.5 (high) · Claude Code 54% (range 36%–96%, n 24). All run ranges overlap.

Calculation from reported tokens, not a run. Each call gives reasoning ÷ output; the bar is the median of those shares. Whiskers are the lowest and highest call. They are a range, not a confidence interval. They are wide, so the medians describe this run and rank nothing. The pooled share (all reasoning tokens ÷ all output tokens) is in the table. We treat reasoning tokens as part of output tokens; the consistency check supports this accounting assumption. Each CLI reports its own count.

Sources: Reasoning token bill (calculation), Provider head-to-head, hard set: eight hard tasks with strict validators, Effort ladder: the hard task set at each effort level, Provider head-to-head: Claude Code models vs Codex efforts, Anthropic list prices (Claude models), OpenAI list prices

Mean per call on the hard tasks; the three parts add up to the call

List-price calculation, not a run. 2 rows, 3 series: Reasoning (output tokens), Remaining output (visible-answer estimate), Input (prompt, cache priced). Reasoning (output tokens): highest Claude Opus 5.5 (high) · Claude Code $0.018 (n 24). Lowest GPT-6.1 Sol (high) · Codex CLI $0.0041 (n 16). Remaining output (visible-answer estimate): highest Claude Opus 5.5 (high) · Claude Code $0.008 (n 24). Lowest GPT-6.1 Sol (high) · Codex CLI $0.0029 (n 16).

Calculation, not a bill: reported tokens × list price (per million output tokens: Haiku 4.5 $5, Sonnet 5.5 $10, Opus 5.5 $20, Fable 5.1 $50 and GPT-6.1 Sol $10); the calls ran on flat subscriptions. Reasoning and remaining output both use the output price. The remainder estimates visible-answer tokens; its exact meaning depends on the CLI counters. Input is the whole prompt, with cache reads and writes priced as in the hard head-to-head. It includes CLI context; these receipts do not separate task tokens from CLI context. It changes with cache counters. Batch timing and cache behavior were not controlled. Means, not medians, so the parts add up.

Sources: Reasoning token bill (calculation), Provider head-to-head, hard set: eight hard tasks with strict validators, Effort ladder: the hard task set at each effort level, Provider head-to-head: Claude Code models vs Codex efforts, Anthropic list prices (Claude models), OpenAI list prices

List price ÷ strict passes; every cell is 8 tasks × 2 repetitions

List-price calculation, not a run. 2 rows, 2 series: Reasoning cost per strict pass, Total cost per strict pass. Reasoning cost per strict pass: highest Claude Opus 5.5 (medium) · Claude Code $0.013 (n 16). Lowest GPT-6.1 Sol (medium) · Codex CLI $0.0023 (n 16). Total cost per strict pass: highest Claude Opus 5.5 (medium) · Claude Code $0.029 (n 16). Lowest GPT-6.1 Sol (medium) · Codex CLI $0.026 (n 16).

Calculation, not a bill: reported tokens × list price, divided by the cell's strict passes; the calls ran on flat subscriptions. The effort-ladder cells: new calls plus reference cells reused from the hard head-to-head (Claude repetitions 1-2 only). "Default" means the effort flag was not passed. The total is the same value as the effort-ladder cost-per-pass chart. Effort levels are not the same scale across vendors, and the reference cells ran in a different batch and hour.

Sources: Reasoning token bill (calculation), Provider head-to-head, hard set: eight hard tasks with strict validators, Effort ladder: the hard task set at each effort level, Provider head-to-head: Claude Code models vs Codex efforts, Anthropic list prices (Claude models), OpenAI list prices

Median call per configuration; five short tasks and eight hard tasks

List-price calculation, not a run. 2 rows, 2 series: Eight hard tasks, Five short tasks. Eight hard tasks: highest GPT-6.1 Sol (high) · Codex CLI 57% (range 29%–91%, n 16). Lowest Claude Opus 5.5 (high) · Claude Code 54% (range 36%–96%, n 24). All run ranges overlap. Five short tasks: highest GPT-6.1 Sol (high) · Codex CLI 58% (range 0%–76%, n 15). Lowest Claude Opus 5.5 (high) · Claude Code 44% (range 0%–93%, n 15). All run ranges overlap.

Calculation from reported tokens, not a run: the median of per-call reasoning ÷ output. A median of 0% means at least half of the calls reported 0 reasoning tokens in the receipt. On a route that reports reasoning, we retain a numeric 0 as the recorded counter. It does not prove the model did no internal reasoning. The table says how many calls reported 0. The short tasks are five small validated tasks (a bug fix, a JSON extraction, an arithmetic problem, a refactor and a ticket classification). Per-call ranges are in the tables.

Sources: Reasoning token bill (calculation), Provider head-to-head, hard set: eight hard tasks with strict validators, Effort ladder: the hard task set at each effort level, Provider head-to-head: Claude Code models vs Codex efforts, Anthropic list prices (Claude models), OpenAI list prices

Median of 4 calls per model; whiskers = fastest and slowest call

2 rows. Slowest GPT-6.1 Sol (low) · Codex CLI 3.5 s (range 2.8 s–4.4 s, n 4). Fastest Claude Opus 5.5 · Claude Code 2 s (range 1.7 s–2.4 s, n 4). Not all run ranges overlap.

Dot = median; whiskers = fastest and slowest call (a range, not a confidence interval). The clock starts when the CLI starts, so first text includes CLI start-up (see chart cli-startup-tax) and any reasoning before the first word. One Mac, one network, two sessions on one night.

Source: LLM speed anatomy

Median of 4 calls per model; whiskers = slowest and fastest call

List-price calculation, not a run. 2 rows. Highest Claude Opus 5.5 · Claude Code 156 (range 155–156, n 4). Lowest GPT-6.1 Sol (low) · Codex CLI 80 (range 72–81, n 4). Not all run ranges overlap.

Calculation, not a measurement: visible output tokens (the CLI’s output token count minus its reported reasoning tokens) divided by the time from the first text to the end of the call. The denominator includes the CLI’s exit overhead; its size is not measured here. Whiskers are a range of calls, not a confidence interval. The calculation removes reported reasoning tokens. These receipts do not locate all reasoning in time.

Source: LLM speed anatomy

Only replies that matched all 250 lines; median per model, whiskers = slowest and fastest call

List-price calculation, not a run. 2 rows. Highest Claude Opus 5.5 · Claude Code 347 (range 345–349, n 4). Lowest GPT-6.1 Sol (low) · Codex CLI 323 (range 291–327, n 4). Not all run ranges overlap.

Calculation, not a measurement: the characters of a correct reply (the same text for every model) divided by the time from the first text to the end of the call. Each vendor counts the same text as a different number of tokens, so characters compare across models where tokens do not. The CLI’s exit time is inside that time. Whiskers are a range of calls, not a confidence interval.

Source: LLM speed anatomy

Median of 3 calls per size; whiskers = fastest and slowest call

List-price calculation, not a run. 3 rows, 4 series: Claude Haiku 4.5 · Claude Code, Claude Sonnet 5.5 · Claude Code, Claude Opus 5.5 · Claude Code, GPT-6.1 Sol (low) · Codex CLI. Claude Haiku 4.5 · Claude Code: slowest 64k 2.8 s (range 2.5 s–2.9 s, n 3). Fastest 1k 1.9 s (range 1.9 s–2 s, n 3). Not all run ranges overlap. Claude Sonnet 5.5 · Claude Code: slowest 64k 3.1 s (range 1.4 s–3.6 s, n 3). Fastest 1k 1.5 s (range 1.2 s–1.7 s, n 3). All run ranges overlap.

Each call used a new ledger seed. Cache-read counts stayed within the short-prompt baseline (see the cache table). This does not identify which tokens were cached. Sizes name the text we send; each model’s reported input tokens are in the table and include the CLI’s own prefix. The size calibration subtracts estimated prefixes from probe input counts; these are calculations, not measured prefix counts for each call. Whiskers are a range of calls, not a confidence interval.

Source: LLM speed anatomy

Median of 3 calls per bar; whiskers = fastest and slowest call

2 rows, 3 series: 1k prompt, 16k prompt, 64k prompt. 1k prompt: slowest GPT-6.1 Sol (low) · Codex CLI 3.4 s (range 3.4 s–4.9 s, n 3). Fastest Claude Opus 5.5 · Claude Code 1.8 s (range 1.8 s–2.4 s, n 3). Not all run ranges overlap. 16k prompt: slowest GPT-6.1 Sol (low) · Codex CLI 4.1 s (range 4 s–4.7 s, n 3). Fastest Claude Opus 5.5 · Claude Code 2.4 s (range 2.1 s–3.4 s, n 3). Not all run ranges overlap.

Whole call: CLI start-up, first text and the one-line answer. Whiskers are a range of calls, not a confidence interval. Each call used a new ledger.

Source: LLM speed anatomy

All sizes together per model; whiskers = 95% Wilson intervals

2 rows. Highest GPT-6.1 Sol (low) · Codex CLI 100% (95% interval 70%–100%, n 9). Lowest Claude Opus 5.5 · Claude Code 56% (95% interval 27%–81%, n 9). All intervals overlap.

Whiskers are 95% Wilson intervals. One lookup question per call; a reply with extra words is a format miss, not a pass. With 9 calls per model, a perfect score still has a wide interval.

Source: LLM speed anatomy

Strict: the reply passes as given. Lenient: a correct answer in the wrong format also counts

2 rows, 2 series: Strict pass, Lenient (format misses counted). Strict pass: highest GPT-6.1 Sol (medium) · Codex CLI 69% (95% interval 44%–86%, n 16). Lowest Claude Opus 5.5 · Claude Code 42% (95% interval 19%–68%, n 12). All intervals overlap. Lenient (format misses counted): highest GPT-6.1 Sol (medium) · Codex CLI 69% (95% interval 44%–86%, n 16). Lowest Claude Opus 5.5 · Claude Code 50% (95% interval 25%–75%, n 12). All intervals overlap.

Whiskers are 95% Wilson intervals. Strict passes decide the result. The lenient reading shows which failures contain a correct answer in the wrong format. A format miss never counts as a pass. The study picked tasks that Sonnet did not pass twice in a pilot. This selection can lower a Sonnet row. Counted calls are new calls.

Source: Harder tasks head-to-head

Share of calls whose reply held tool-call markup or whose tool call the CLI could not parse

2 rows. Highest Claude Opus 5.5 · Claude Code 42% (95% interval 19%–68%, n 12). Lowest GPT-6.1 Sol (medium) · Codex CLI 0% (95% interval 0%–19%, n 16). All intervals overlap.

Every prompt asked for the answer only. Both CLIs disabled tools. The Codex runner adds a developer instruction not to call tools. The Claude Code runner adds no such instruction. The validator decides the score. Tool-call markup is a separate flag; a flagged call can still pass if its whole reply meets the validator. It is a behaviour, not a quality score: more is not better. No call with a tool attempt passed. Whiskers are 95% Wilson intervals. The flag comes from the reply text, which is not published.

Source: Harder tasks head-to-head

One bar per configuration and task; each bar rests on only a few calls, so the 95% intervals are wide

3 rows, 4 series: GPT-6.1 Sol (medium) · Codex CLI, Claude Opus 5.5 · Claude Code, Claude Sonnet 5.5 · Claude Code, Claude Haiku 4.5 · Claude Code. GPT-6.1 Sol (medium) · Codex CLI: highest 6x6 Skyscrapers 100% (95% interval 51%–100%, n 4). Lowest Sudoku, 22 givens 25% (95% interval 4.6%–70%, n 4). All intervals overlap. Claude Opus 5.5 · Claude Code: highest 10x10 nonogram 100% (95% interval 44%–100%, n 3). Lowest Sudoku, 22 givens 0% (95% interval 0%–56%, n 3). All intervals overlap.

Each bar is 3 to 4 calls, so one call moves a bar by a quarter or a third. Whiskers are 95% Wilson intervals; with this few calls they overlap almost everywhere.

Source: Harder tasks head-to-head

Median per configuration; whiskers = fastest and slowest call

2 rows. Slowest GPT-6.1 Sol (medium) · Codex CLI 120 s (range 46.2 s–273 s, n 13). Fastest Claude Opus 5.5 · Claude Code 80.3 s (range 3.8 s–280 s, n 9). All run ranges overlap.

Median and range over the calls that completed. Completed calls include wrong answers and format misses. Only timeouts and tool-call parse errors are excluded from this run’s timings. Both count as non-passes in the outcomes chart. One Mac, one network, one session. Arena servers shared the Mac during part of the run. Host load was not controlled, so these times cannot isolate model speed. Whiskers are a range, not a confidence interval. Times include the CLI start-up and the CLI’s own system prompt. Highlighted: configurations that passed every call.

Source: Harder tasks head-to-head

Median per configuration; reasoning tokens as the CLI reports them

2 rows, 2 series: Output tokens, Reasoning tokens. Output tokens: highest Claude Opus 5.5 · Claude Code 8,420 (range 279–40,044, n 9). Lowest GPT-6.1 Sol (medium) · Codex CLI 4,994 (range 2,099–13,413, n 13). All run ranges overlap. Reasoning tokens: highest Claude Opus 5.5 · Claude Code 8,352 (range 21–9,897, n 9). Lowest GPT-6.1 Sol (medium) · Codex CLI 4,971 (range 2,070–13,372, n 13). All run ranges overlap.

Medians and minimum-to-maximum token ranges cover completed calls only. Ranges are not confidence intervals. The chart omits unknown reasoning counts. Reasoning tokens are part of the output tokens where the CLI reports them. Their content is never captured. Claude Code used an output-token cap setting of 16,000. Some reported totals exceeded it. Codex CLI had no cap. More tokens is not better or worse by itself.

Source: Harder tasks head-to-head

All calls in a configuration, failures and format misses included, divided by its strict passes

List-price calculation, not a run. 2 rows. Highest Claude Opus 5.5 · Claude Code $0.59 (n 12). Lowest GPT-6.1 Sol (medium) · Codex CLI $0.083 (n 16).

Calculation, not a bill: reported tokens × list price; the calls ran on flat subscriptions. A failed call still costs, so a lower pass rate raises the cost per pass. Timeout calls report no tokens and are not priced. Unpriced calls: GPT-6.1 Sol (medium) 3 of 16, Opus 5.5 1 of 12 and Sonnet 5.5 4 of 16. These cells show a lower bound. Assume each unpriced call cost its cell’s median priced call. This sensitivity calculation gives GPT-6.1 Sol (medium) $0.100, Opus 5.5 $0.633 and Sonnet 5.5 $0.303. Opus 5.5 figures are provisional: its cache-read price is under re-check. Highlights mark the observed frontier of these lower-bound costs. Unknown timeout costs can change it; this is not a cost ranking. Claude Haiku 4.5 · Claude Code had no strict pass, so it has no cost per pass.

Sources: Harder tasks head-to-head, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices