AClaude Haiku 4.5Model · AnthropicvsBGPT-6 Luna (Codex CLI)Model · OpenAI

Same tasks and validators. Whiskers are 95% Wilson intervals

2 rows. Highest GPT-6 Luna (single call) · Codex CLI 63% (95% interval 39%–82%, n 16). Lowest Claude Haiku 4.5 (single call) · Claude Code 46% (95% interval 28%–65%, n 24). All intervals overlap.

Whiskers are 95% Wilson intervals. A format miss never counts as a pass; an attempt with no final answer (error, time-out, turn limit) counts as a fail. Contaminated attempts (a read outside the work folder) are left out. The Claude single-call cells are reference cells from the hard head-to-head (8 tasks × 3 repetitions), reused, not rerun.

Sources: Single call vs agent loop, Provider head-to-head, hard set: eight hard tasks with strict validators

Share of attempts per task that passed strictly; 2 or 3 attempts per task and configuration

5 rows, 6 series: Claude Haiku 4.5 (single call) · Claude Code, Claude Haiku 4.5 (agent loop) · Claude Code, Claude Sonnet 5.5 (single call) · Claude Code, Claude Sonnet 5.5 (agent loop) · Claude Code, GPT-6 Luna (single call) · Codex CLI, GPT-6 Luna (agent loop) · Codex CLI. Claude Haiku 4.5 (single call) · Claude Code: highest Interval merge fix 100% (95% interval 44%–100%, n 3). Lowest Room schedule 0% (95% interval 0%–56%, n 3). All intervals overlap. Claude Haiku 4.5 (agent loop) · Claude Code: highest Event-loop order 100% (95% interval 44%–100%, n 3). Lowest Room schedule 67% (95% interval 21%–94%, n 3). All intervals overlap.

Whiskers are 95% Wilson intervals on 2 or 3 attempts, so they are very wide: read this chart for where a configuration failed, not for a ranking. Contaminated attempts (a read outside the work folder) are left out.

Sources: Single call vs agent loop, Provider head-to-head, hard set: eight hard tasks with strict validators

Median per configuration; whiskers = fastest and slowest attempt

2 rows. Slowest Claude Haiku 4.5 (single call) · Claude Code 39 s (range 15.3 s–75.1 s, n 24). Fastest GPT-6 Luna (single call) · Codex CLI 5.2 s (range 3.6 s–11.3 s, n 16). Not all run ranges overlap.

Whiskers are a range (fastest and slowest attempt), not a confidence interval. Agent loop: wall time of the whole session, failures and time-outs included. One host, one network. The Claude single-call cells are reference cells from the hard head-to-head (8 tasks × 3 repetitions), reused, not rerun. The reference cells ran in a different hour.

Sources: Single call vs agent loop, Provider head-to-head, hard set: eight hard tasks with strict validators

Median per configuration; whiskers = fewest and most

2 rows, 2 series: Input tokens (cache reads included), Output tokens. Input tokens (cache reads included): highest GPT-6 Luna (single call) · Codex CLI 11,582 (range 11,526–11,818, n 16). Lowest Claude Haiku 4.5 (single call) · Claude Code 3,941 (range 3,879–4,221, n 24). Not all run ranges overlap. Output tokens: highest Claude Haiku 4.5 (single call) · Claude Code 5,064 (range 1,899–9,321, n 24). Lowest GPT-6 Luna (single call) · Codex CLI 345 (range 36–634, n 16). Not all run ranges overlap.

Whiskers are a range (fewest and most), not a confidence interval. Input counts the whole prompt of every model request in the attempt, cache reads and writes included; an agent loop re-sends its growing context each turn. Codex input includes its own system prompt and tool schemas. More tokens is not better or worse by itself.

Sources: Single call vs agent loop, Provider head-to-head, hard set: eight hard tasks with strict validators

Median per configuration; whiskers = fewest and most. A single call makes none

2 rows. Highest Claude Haiku 4.5 (agent loop) · Claude Code 3 (range 2–18, n 24). Lowest GPT-6 Luna (agent loop) · Codex CLI 0 (range 0–1, n 14). Not all run ranges overlap.

Whiskers are a range (fewest and most), not a confidence interval. Claude Code tools: shell, read, edit, write, glob, grep. Codex CLI: shell commands and file changes. The model chose whether to test its answer; the prompt allowed it but did not require it.

Source: Single call vs agent loop

All attempts in a configuration divided by its strict passes

List-price calculation, not a run. 2 rows. Highest Claude Haiku 4.5 (single call) · Claude Code $0.067 (n 24). Lowest GPT-6 Luna (single call) · Codex CLI $0.0012 (n 16).

Calculation, not a bill: reported tokens × list price for every attempt (per model when a session used several), cache reads at the cache-read price and cache writes at the one-hour write price, divided by the strict passes. The calls ran on flat subscriptions.

Sources: Single call vs agent loop, Provider head-to-head, hard set: eight hard tasks with strict validators, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices

Median of 4 calls per model; whiskers = fastest and slowest call

2 rows. Slowest Claude Haiku 4.5 · Claude Code 4 s (range 2.8 s–6.4 s, n 4). Fastest GPT-6 Luna (low) · Codex CLI 3.3 s (range 3.2 s–3.5 s, n 4). All run ranges overlap.

Dot = median; whiskers = fastest and slowest call (a range, not a confidence interval). The clock starts when the CLI starts, so first text includes CLI start-up (see chart cli-startup-tax) and any reasoning before the first word. One Mac, one network, two sessions on one night.

Source: LLM speed anatomy

Median of 4 calls per model; whiskers = slowest and fastest call

List-price calculation, not a run. 2 rows. Highest Claude Haiku 4.5 · Claude Code 153 (range 153–216, n 4). Lowest GPT-6 Luna (low) · Codex CLI 129 (range 56–259, n 4). All run ranges overlap.

Calculation, not a measurement: visible output tokens (the CLI’s output token count minus its reported reasoning tokens) divided by the time from the first text to the end of the call. The denominator includes the CLI’s exit overhead; its size is not measured here. Whiskers are a range of calls, not a confidence interval. The calculation removes reported reasoning tokens. These receipts do not locate all reasoning in time.

Source: LLM speed anatomy

Only replies that matched all 250 lines; median per model, whiskers = slowest and fastest call

List-price calculation, not a run. 2 rows. Highest Claude Haiku 4.5 · Claude Code 547 (range 546–548, n 3). Lowest GPT-6 Luna (low) · Codex CLI 524 (range 225–1,052, n 4). All run ranges overlap.

Calculation, not a measurement: the characters of a correct reply (the same text for every model) divided by the time from the first text to the end of the call. Each vendor counts the same text as a different number of tokens, so characters compare across models where tokens do not. The CLI’s exit time is inside that time. Whiskers are a range of calls, not a confidence interval.

Source: LLM speed anatomy