AClaude Haiku 4.5Model · AnthropicvsBClaude Fable 5.1Model · Anthropic

Every call counts; failures and timeouts are non-passes

2 rows. All at 100%.

Whiskers are 95% Wilson intervals. The tasks are short and most configurations pass nearly all of them, so pass rate does not separate the models here.

Source: Provider head-to-head: Claude Code models vs Codex efforts

Median per configuration; whiskers = fastest and slowest call

2 rows. Slowest Claude Haiku 4.5 · Claude Code 4.4 s (range 3.2 s–23.6 s, n 15). Fastest Claude Fable 5.1 · Claude Code 1.9 s (range 1.4 s–9.8 s, n 15). All run ranges overlap.

One host, one network, one day. Whiskers are a range, not a confidence interval.

Source: Provider head-to-head: Claude Code models vs Codex efforts

Median per configuration; whiskers = fastest and slowest call

2 rows. Slowest Claude Haiku 4.5 · Claude Code 3.6 s (range 2.8 s–22.3 s, n 15). Fastest Claude Fable 5.1 · Claude Code 1.2 s (range 1 s–7.9 s, n 15). All run ranges overlap.

One host, one network, one day. Whiskers are a range, not a confidence interval.

Source: Provider head-to-head: Claude Code models vs Codex efforts

Mean per call, split into prompt-cache reads and other input

2 rows, 2 series: Cache read, Other input. Cache read: highest Claude Fable 5.1 · Claude Code 2,760 (n 15). Lowest Claude Haiku 4.5 · Claude Code 0 (n 15). Other input: highest Claude Haiku 4.5 · Claude Code 3,790 (n 15). Lowest Claude Fable 5.1 · Claude Code 473 (n 15).

The prompts are a few hundred tokens; most input is the CLI’s own system prompt and tool context. Input counts include cache reads, as the vendors report them.

Source: Provider head-to-head: Claude Code models vs Codex efforts

Median per configuration; reasoning tokens where the CLI reports them

2 rows, 2 series: Output tokens, Reasoning tokens. Output tokens: highest Claude Haiku 4.5 · Claude Code 367 (n 15). Lowest Claude Fable 5.1 · Claude Code 64 (n 15). Reasoning tokens: highest Claude Haiku 4.5 · Claude Code 297 (n 15). Lowest Claude Fable 5.1 · Claude Code 0 (n 15).

Vendors count reasoning differently; compare within a vendor first. A median of 0 can mean the CLI did not report reasoning for most calls.

Source: Provider head-to-head: Claude Code models vs Codex efforts

Reported tokens × list price; the calls ran on subscriptions

List-price calculation, not a run. 2 rows. Highest Claude Fable 5.1 · Claude Code $0.0099 (range $0.0049–$0.058, n 15). Lowest Claude Haiku 4.5 · Claude Code $0.0057 (range $0.0051–$0.018, n 15). All run ranges overlap.

Calculation, not a bill: the calls ran on flat subscriptions. Whiskers = cheapest and most expensive call.

Sources: Provider head-to-head: Claude Code models vs Codex efforts, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices

All calls in a configuration, failures included, divided by its passes

List-price calculation, not a run. 2 rows. Highest Claude Fable 5.1 · Claude Code $0.021 (n 15). Lowest Claude Haiku 4.5 · Claude Code $0.0084 (n 15).

Calculation, not a bill: reported tokens × list price; the calls ran on flat subscriptions. A failed call still costs, so a lower pass rate raises the cost per pass. Highlighted bars are on the speed/cost/quality frontier.

Sources: Provider head-to-head: Claude Code models vs Codex efforts, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices

Strict: the reply passes as given. Lenient: a correct answer in the wrong format also counts

2 rows, 2 series: Strict pass, Lenient (format misses counted). Strict pass: highest Claude Fable 5.1 · Claude Code 100% (95% interval 86%–100%, n 24). Lowest Claude Haiku 4.5 · Claude Code 46% (95% interval 28%–65%, n 24). Not all intervals overlap. Lenient (format misses counted): highest Claude Fable 5.1 · Claude Code 100% (95% interval 86%–100%, n 24). Lowest Claude Haiku 4.5 · Claude Code 67% (95% interval 47%–82%, n 24). Not all intervals overlap.

Whiskers are 95% Wilson intervals. The strict pass is the result; the lenient reading is shown so a reader can see how much is format and how much is a wrong answer. A format miss never counts as a pass.

Source: Provider head-to-head, hard set: eight hard tasks with strict validators

Median per configuration; whiskers = fastest and slowest call

2 rows. Slowest Claude Haiku 4.5 · Claude Code 39 s (range 15.3 s–75.1 s, n 24). Fastest Claude Fable 5.1 · Claude Code 16.1 s (range 4.5 s–90 s, n 24). All run ranges overlap.

One host and network; the counted Claude and Codex batches ran hours apart. Host load was not controlled. Whiskers are a range, not a confidence interval. Highlighted: configurations that passed every call.

Source: Provider head-to-head, hard set: eight hard tasks with strict validators

Median per configuration; whiskers = fastest and slowest call

2 rows. Slowest Claude Haiku 4.5 · Claude Code 35.5 s (range 12.9 s–70.3 s, n 24). Fastest Claude Fable 5.1 · Claude Code 11.6 s (range 2 s–85.3 s, n 24). All run ranges overlap.

One host and network; the counted Claude and Codex batches ran hours apart. Host load was not controlled. Whiskers are a range, not a confidence interval. Highlighted: configurations that passed every call.

Source: Provider head-to-head, hard set: eight hard tasks with strict validators

Median per configuration; reasoning tokens as the CLI reports them

2 rows, 2 series: Output tokens, Reasoning tokens. Output tokens: highest Claude Haiku 4.5 · Claude Code 5,064 (n 24). Lowest Claude Fable 5.1 · Claude Code 1,366 (n 24). Reasoning tokens: highest Claude Haiku 4.5 · Claude Code 4,556 (n 24). Lowest Claude Fable 5.1 · Claude Code 889 (n 24).

Reasoning tokens are part of the output tokens where the CLI reports them. Their content is never captured. More tokens is not better or worse by itself.

Source: Provider head-to-head, hard set: eight hard tasks with strict validators

All calls in a configuration, failures and format misses included, divided by its strict passes

List-price calculation, not a run. 2 rows. Highest Claude Fable 5.1 · Claude Code $0.093 (n 24). Lowest Claude Haiku 4.5 · Claude Code $0.067 (n 24).

Calculation, not a bill: reported tokens × list price; the calls ran on a flat subscription. A failed call still costs, so a lower pass rate raises the cost per pass. Highlighted bars are on the quality-vs-cost frontier.

Sources: Provider head-to-head, hard set: eight hard tasks with strict validators, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices

Median call: reasoning tokens ÷ output tokens. Whiskers: lowest and highest call (16 to 24 calls per configuration)

List-price calculation, not a run. 2 rows. Highest Claude Haiku 4.5 · Claude Code 92% (range 76%–99%, n 24). Lowest Claude Fable 5.1 · Claude Code 64% (range 23%–97%, n 24). All run ranges overlap.

Calculation from reported tokens, not a run. Each call gives reasoning ÷ output; the bar is the median of those shares. Whiskers are the lowest and highest call. They are a range, not a confidence interval. They are wide, so the medians describe this run and rank nothing. The pooled share (all reasoning tokens ÷ all output tokens) is in the table. We treat reasoning tokens as part of output tokens; the consistency check supports this accounting assumption. Each CLI reports its own count.

Sources: Reasoning token bill (calculation), Provider head-to-head, hard set: eight hard tasks with strict validators, Effort ladder: the hard task set at each effort level, Provider head-to-head: Claude Code models vs Codex efforts, Anthropic list prices (Claude models), OpenAI list prices

Mean per call on the hard tasks; the three parts add up to the call

List-price calculation, not a run. 2 rows, 3 series: Reasoning (output tokens), Remaining output (visible-answer estimate), Input (prompt, cache priced). Reasoning (output tokens): highest Claude Fable 5.1 · Claude Code $0.054 (n 24). Lowest Claude Haiku 4.5 · Claude Code $0.024 (n 24). Remaining output (visible-answer estimate): highest Claude Fable 5.1 · Claude Code $0.019 (n 24). Lowest Claude Haiku 4.5 · Claude Code $0.0018 (n 24).

Calculation, not a bill: reported tokens × list price (per million output tokens: Haiku 4.5 $5, Sonnet 5.5 $10, Opus 5.5 $20, Fable 5.1 $50 and GPT-6.1 Sol $10); the calls ran on flat subscriptions. Reasoning and remaining output both use the output price. The remainder estimates visible-answer tokens; its exact meaning depends on the CLI counters. Input is the whole prompt, with cache reads and writes priced as in the hard head-to-head. It includes CLI context; these receipts do not separate task tokens from CLI context. It changes with cache counters. Batch timing and cache behavior were not controlled. Means, not medians, so the parts add up.

Sources: Reasoning token bill (calculation), Provider head-to-head, hard set: eight hard tasks with strict validators, Effort ladder: the hard task set at each effort level, Provider head-to-head: Claude Code models vs Codex efforts, Anthropic list prices (Claude models), OpenAI list prices

Median call per configuration; five short tasks and eight hard tasks

List-price calculation, not a run. 2 rows, 2 series: Eight hard tasks, Five short tasks. Eight hard tasks: highest Claude Haiku 4.5 · Claude Code 92% (range 76%–99%, n 24). Lowest Claude Fable 5.1 · Claude Code 64% (range 23%–97%, n 24). All run ranges overlap. Five short tasks: highest Claude Haiku 4.5 · Claude Code 90% (range 73%–98%, n 15). Lowest Claude Fable 5.1 · Claude Code 0% (range 0%–74%, n 15). All run ranges overlap.

Calculation from reported tokens, not a run: the median of per-call reasoning ÷ output. A median of 0% means at least half of the calls reported 0 reasoning tokens in the receipt. On a route that reports reasoning, we retain a numeric 0 as the recorded counter. It does not prove the model did no internal reasoning. The table says how many calls reported 0. The short tasks are five small validated tasks (a bug fix, a JSON extraction, an arithmetic problem, a refactor and a ticket classification). Per-call ranges are in the tables.

Sources: Reasoning token bill (calculation), Provider head-to-head, hard set: eight hard tasks with strict validators, Effort ladder: the hard task set at each effort level, Provider head-to-head: Claude Code models vs Codex efforts, Anthropic list prices (Claude models), OpenAI list prices

Median of 4 calls per model; whiskers = fastest and slowest call

2 rows. Slowest Claude Fable 5.1 · Claude Code 4.4 s (range 2.3 s–4.6 s, n 4). Fastest Claude Haiku 4.5 · Claude Code 4 s (range 2.8 s–6.4 s, n 4). All run ranges overlap.

Dot = median; whiskers = fastest and slowest call (a range, not a confidence interval). The clock starts when the CLI starts, so first text includes CLI start-up (see chart cli-startup-tax) and any reasoning before the first word. One Mac, one network, two sessions on one night.

Source: LLM speed anatomy

Median of 4 calls per model; whiskers = slowest and fastest call

List-price calculation, not a run. 2 rows. Highest Claude Haiku 4.5 · Claude Code 153 (range 153–216, n 4). Lowest Claude Fable 5.1 · Claude Code 123 (range 121–131, n 4). Not all run ranges overlap.

Calculation, not a measurement: visible output tokens (the CLI’s output token count minus its reported reasoning tokens) divided by the time from the first text to the end of the call. The denominator includes the CLI’s exit overhead; its size is not measured here. Whiskers are a range of calls, not a confidence interval. The calculation removes reported reasoning tokens. These receipts do not locate all reasoning in time.

Source: LLM speed anatomy

Only replies that matched all 250 lines; median per model, whiskers = slowest and fastest call

List-price calculation, not a run. 2 rows. Highest Claude Haiku 4.5 · Claude Code 547 (range 546–548, n 3). Lowest Claude Fable 5.1 · Claude Code 273 (range 270–293, n 4). Not all run ranges overlap.

Calculation, not a measurement: the characters of a correct reply (the same text for every model) divided by the time from the first text to the end of the call. Each vendor counts the same text as a different number of tokens, so characters compare across models where tokens do not. The CLI’s exit time is inside that time. Whiskers are a range of calls, not a confidence interval.

Source: LLM speed anatomy