Coding CLI · OpenAI
Codex CLI
OpenAI’s coding CLI. Each measurement pairs it with one GPT model and effort; the context names them.
167 values from 12 studies (44 are list-price calculations) · Updated
At a glance
The best-supported value per category: a 95% interval first, then a run range, then the larger n. Three separate values from separate studies, never one score.
Qualitypass rates, accuracy and scores
100% (16/16)
95% CI 81%–100% · n = 16
Pass rate on eight hard tasks (Strict pass)
GPT-6.1 Sol · effort medium · eight hard validated tasks · Haiku vs Sonnet vs Opus vs Fable vs GPT-6.1 Sol on 8 hard tasks
Speedtime per call or decision
13.1s
range 8.5 s–61.6 s · n = 16
Total time per call on hard tasks (separate batches)
GPT-6.1 Sol · effort medium · eight hard validated tasks · Haiku vs Sonnet vs Opus vs Fable vs GPT-6.1 Sol on 8 hard tasks
CostUS dollars per call, pass or decision
$0.010
range $0.0054–$0.027 · n = 15 · list-price calculation
List-price cost per call (calculation)
GPT-6.1 Sol · effort medium · five short validated tasks · Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head
Where it sits
Every measured value, grouped by study. Each row puts the value on its own track, with the other configurations of the same chart as muted dots. A range is the fastest to slowest recorded run and p50–p95 is the median to the 95th percentile; neither is a confidence interval. Use Table for the plain values.
| Metric | Value | n | Interval or range | Configuration |
|---|---|---|---|---|
| Pass rate on five validated tasks | 100% (15/15) | 15 | 80%–100% (95% CI) | GPT-6.1 Sol · effort high · five short validated tasks |
| Pass rate on five validated tasks | 100% (15/15) | 15 | 80%–100% (95% CI) | GPT-6.1 Sol · effort medium · five short validated tasks |
| Pass rate on five validated tasks | 100% (10/10) | 10 | 72%–100% (95% CI) | GPT-6.1 Sol · effort low · five short validated tasks |
| Total time per call | 5.60 s | 15 | 4.1 s–19.5 s (range) | GPT-6.1 Sol · effort high · five short validated tasks |
| Total time per call | 5.65 s | 15 | 4.1 s–25.5 s (range) | GPT-6.1 Sol · effort medium · five short validated tasks |
| Total time per call | 6.26 s | 10 | 4.7 s–10.5 s (range) | GPT-6.1 Sol · effort low · five short validated tasks |
| Time to first useful output | 5.32 s | 15 | 3.6 s–16.4 s (range) | GPT-6.1 Sol · effort high · five short validated tasks |
| Time to first useful output | 5.05 s | 15 | 3.4 s–17.8 s (range) | GPT-6.1 Sol · effort medium · five short validated tasks |
| Time to first useful output | 5.14 s | 10 | 4 s–8.5 s (range) | GPT-6.1 Sol · effort low · five short validated tasks |
| Input tokens per call: what the CLI sends (Cache read) | 6,716 | 15 | — | GPT-6.1 Sol · effort high · five short validated tasks |
| Input tokens per call: what the CLI sends (Cache read) | 5,180 | 15 | — | GPT-6.1 Sol · effort medium · five short validated tasks |
| Input tokens per call: what the CLI sends (Cache read) | 8,064 | 10 | — | GPT-6.1 Sol · effort low · five short validated tasks |
| Input tokens per call: what the CLI sends (Other input) | 5,406 | 15 | — | GPT-6.1 Sol · effort high · five short validated tasks |
| Input tokens per call: what the CLI sends (Other input) | 6,943 | 15 | — | GPT-6.1 Sol · effort medium · five short validated tasks |
| Input tokens per call: what the CLI sends (Other input) | 4,059 | 10 | — | GPT-6.1 Sol · effort low · five short validated tasks |
| Output tokens per call (Output tokens) | 42 | 15 | — | GPT-6.1 Sol · effort high · five short validated tasks |
| Output tokens per call (Output tokens) | 42 | 15 | — | GPT-6.1 Sol · effort medium · five short validated tasks |
| Output tokens per call (Output tokens) | 42 | 10 | — | GPT-6.1 Sol · effort low · five short validated tasks |
| List-price cost per call (calculation) Calculation | $0.010 | 15 | $0.0066–$0.028 (range) | GPT-6.1 Sol · effort high · five short validated tasks |
| List-price cost per call (calculation) Calculation | $0.010 | 15 | $0.0054–$0.027 (range) | GPT-6.1 Sol · effort medium · five short validated tasks |
| List-price cost per call (calculation) Calculation | $0.0077 | 10 | $0.0074–$0.026 (range) | GPT-6.1 Sol · effort low · five short validated tasks |
| List-price cost per passing answer (calculation) Calculation | $0.010 | 10 | — | GPT-6.1 Sol · effort low · five short validated tasks |
| List-price cost per passing answer (calculation) Calculation | $0.013 | 15 | — | GPT-6.1 Sol · effort high · five short validated tasks |
| List-price cost per passing answer (calculation) Calculation | $0.016 | 15 | — | GPT-6.1 Sol · effort medium · five short validated tasks |
Whiskers: 95% Wilson intervalLines: fastest–slowest run (not an interval)n beside each valueMuted dots: the other configurations on the same chartHollow: list-price calculation
Codex CLI in Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head: 24 values, first Pass rate on five validated tasks 100% (15/15).
| Metric | Value | n | Interval or range | Configuration |
|---|---|---|---|---|
| Pass rate on eight hard tasks (Strict pass) | 100% (16/16) | 16 | 81%–100% (95% CI) | GPT-6.1 Sol · effort medium · eight hard validated tasks |
| Pass rate on eight hard tasks (Strict pass) | 100% (16/16) | 16 | 81%–100% (95% CI) | GPT-6.1 Sol · effort high · eight hard validated tasks |
| Pass rate on eight hard tasks (Lenient (format misses counted)) | 100% (16/16) | 16 | 81%–100% (95% CI) | GPT-6.1 Sol · effort medium · eight hard validated tasks |
| Pass rate on eight hard tasks (Lenient (format misses counted)) | 100% (16/16) | 16 | 81%–100% (95% CI) | GPT-6.1 Sol · effort high · eight hard validated tasks |
| Total time per call on hard tasks (separate batches) | 13.1 s | 16 | 8.5 s–61.6 s (range) | GPT-6.1 Sol · effort medium · eight hard validated tasks |
| Total time per call on hard tasks (separate batches) | 18.1 s | 16 | 11.7 s–92.2 s (range) | GPT-6.1 Sol · effort high · eight hard validated tasks |
| Time to first useful output on hard tasks | 10.2 s | 16 | 6.1 s–40.4 s (range) | GPT-6.1 Sol · effort medium · eight hard validated tasks |
| Time to first useful output on hard tasks | 12.7 s | 16 | 8.9 s–75.9 s (range) | GPT-6.1 Sol · effort high · eight hard validated tasks |
| Output tokens per call on hard tasks (Output tokens) | 335 | 16 | — | GPT-6.1 Sol · effort medium · eight hard validated tasks |
| Output tokens per call on hard tasks (Output tokens) | 436 | 16 | — | GPT-6.1 Sol · effort high · eight hard validated tasks |
| List-price cost per strict pass on hard tasks (calculation) Calculation | $0.015 | 16 | — | GPT-6.1 Sol · effort high · eight hard validated tasks |
| List-price cost per strict pass on hard tasks (calculation) Calculation | $0.026 | 16 | — | GPT-6.1 Sol · effort medium · eight hard validated tasks |
Whiskers: 95% Wilson intervalLines: fastest–slowest run (not an interval)n beside each valueMuted dots: the other configurations on the same chartHollow: list-price calculation
Codex CLI in Haiku vs Sonnet vs Opus vs Fable vs GPT-6.1 Sol on 8 hard tasks: 12 values, first Pass rate on eight hard tasks (Strict pass) 100% (16/16).
Claude Code (Sonnet 5.5, Opus 5.5) vs Codex CLI on 6 hidden-test coding tasks
4 values · open the study
| Metric | Value | n | Interval or range | Configuration |
|---|---|---|---|---|
| Coding sessions that passed every hidden check | 100% (12/12) | 12 | 76%–100% (95% CI) | GPT-6.1 Sol · effort medium · tester’s AGENTS.md · six small repository tasks with hidden tests |
| Time per coding session | 113.4 s | 12 | 78.5 s–222 s (range) | GPT-6.1 Sol · effort medium · tester’s AGENTS.md · six small repository tasks with hidden tests |
| Tool calls per coding session | 12.5 | 12 | 8–18 (range) | GPT-6.1 Sol · effort medium · tester’s AGENTS.md · six small repository tasks with hidden tests |
| List-price cost per passing coding session (calculation) Calculation | $0.098 | 12 | — | GPT-6.1 Sol · effort medium · tester’s AGENTS.md · six small repository tasks with hidden tests |
Whiskers: 95% Wilson intervalLines: fastest–slowest run (not an interval)n beside each valueMuted dots: the other configurations on the same chartHollow: list-price calculation
Codex CLI in Claude Code (Sonnet 5.5, Opus 5.5) vs Codex CLI on 6 hidden-test coding tasks: 4 values, first Coding sessions that passed every hidden check 100% (12/12).
Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks
12 values · open the study
| Metric | Value | n | Interval or range | Configuration |
|---|---|---|---|---|
| Strict pass rate by effort on eight hard tasks | 100% (16/16) | 16 | 81%–100% (95% CI) | GPT-6.1 Sol · effort low · eight hard validated tasks, effort ladder |
| Strict pass rate by effort on eight hard tasks | 100% (16/16) | 16 | 81%–100% (95% CI) | GPT-6.1 Sol · effort medium · eight hard validated tasks, effort ladder |
| Strict pass rate by effort on eight hard tasks | 100% (16/16) | 16 | 81%–100% (95% CI) | GPT-6.1 Sol · effort high · eight hard validated tasks, effort ladder |
| Total time per call by effort on hard tasks | 13.6 s | 16 | 7.9 s–44.3 s (range) | GPT-6.1 Sol · effort low · eight hard validated tasks, effort ladder |
| Total time per call by effort on hard tasks | 13.1 s | 16 | 8.5 s–61.6 s (range) | GPT-6.1 Sol · effort medium · eight hard validated tasks, effort ladder |
| Total time per call by effort on hard tasks | 18.1 s | 16 | 11.7 s–92.2 s (range) | GPT-6.1 Sol · effort high · eight hard validated tasks, effort ladder |
| Output tokens per call by effort on hard tasks (Output tokens) | 284 | 16 | — | GPT-6.1 Sol · effort low · eight hard validated tasks, effort ladder |
| Output tokens per call by effort on hard tasks (Output tokens) | 335 | 16 | — | GPT-6.1 Sol · effort medium · eight hard validated tasks, effort ladder |
| Output tokens per call by effort on hard tasks (Output tokens) | 436 | 16 | — | GPT-6.1 Sol · effort high · eight hard validated tasks, effort ladder |
| List-price cost per strict pass by effort (calculation) Calculation | $0.013 | 16 | — | GPT-6.1 Sol · effort low · eight hard validated tasks, effort ladder |
| List-price cost per strict pass by effort (calculation) Calculation | $0.026 | 16 | — | GPT-6.1 Sol · effort medium · eight hard validated tasks, effort ladder |
| List-price cost per strict pass by effort (calculation) Calculation | $0.015 | 16 | — | GPT-6.1 Sol · effort high · eight hard validated tasks, effort ladder |
Whiskers: 95% Wilson intervalLines: fastest–slowest run (not an interval)n beside each valueMuted dots: the other configurations on the same chartHollow: list-price calculation
Codex CLI in Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks: 12 values, first Strict pass rate by effort on eight hard tasks 100% (16/16).
| Metric | Value | n | Interval or range | Configuration |
|---|---|---|---|---|
| Same prompt, 10 times: strict pass rate (Exact number) | 100% (10/10) | 10 | 72%–100% (95% CI) | GPT-6.1 Sol · effort medium · same prompt repeated 10 times |
| Same prompt, 10 times: strict pass rate (JSON object) | 100% (10/10) | 10 | 72%–100% (95% CI) | GPT-6.1 Sol · effort medium · same prompt repeated 10 times |
| Same prompt, 10 times: strict pass rate (Code fix) | 100% (10/10) | 10 | 72%–100% (95% CI) | GPT-6.1 Sol · effort medium · same prompt repeated 10 times |
| Same prompt, 10 times: how many different answers (Exact number) | 1 | 10 | — | GPT-6.1 Sol · effort medium · same prompt repeated 10 times |
| Same prompt, 10 times: how many different answers (JSON object) | 1 | 10 | — | GPT-6.1 Sol · effort medium · same prompt repeated 10 times |
| Same prompt, 10 times: how many different answers (Code fix) | 6 | 10 | — | GPT-6.1 Sol · effort medium · same prompt repeated 10 times |
| Same prompt, 10 times: time per call (Exact number) | 13.4 s | 10 | 12.3 s–18 s (range) | GPT-6.1 Sol · effort medium · same prompt repeated 10 times |
| Same prompt, 10 times: time per call (JSON object) | 6.42 s | 10 | 5.3 s–8.3 s (range) | GPT-6.1 Sol · effort medium · same prompt repeated 10 times |
| Same prompt, 10 times: time per call (Code fix) | 11.3 s | 10 | 9.1 s–14.9 s (range) | GPT-6.1 Sol · effort medium · same prompt repeated 10 times |
Whiskers: 95% Wilson intervalLines: fastest–slowest run (not an interval)n beside each valueMuted dots: the other configurations on the same chart
Codex CLI in Prompt caching and run-to-run consistency in Claude Code and Codex CLI: 9 values, first Same prompt, 10 times: strict pass rate (Exact number) 100% (10/10).
| Metric | Value | n | Interval or range | Configuration |
|---|---|---|---|---|
| CLI start-up tax on a one-word answer (First output event) | 489 ms | 5 | 354 ms–1.3 s (range) | default model · CLI start-up, one-word prompt, 5 runs |
| CLI start-up tax on a one-word answer (First model output) | 5,059 ms | 5 | 4.39 s–5.48 s (range) | default model · CLI start-up, one-word prompt, 5 runs |
| CLI start-up tax on a one-word answer (Total wall time) | 5,999 ms | 5 | 5.37 s–6.51 s (range) | default model · CLI start-up, one-word prompt, 5 runs |
| Input tokens a CLI sends for a one-word answer | 17,051 | 5 | — | default model · CLI start-up, one-word prompt, 5 runs |
Lines: fastest–slowest run (not an interval)n beside each valueMuted dots: the other configurations on the same chart
Codex CLI in Routing overhead: deterministic policy vs LLM routers vs Jev: 4 values, first CLI start-up tax on a one-word answer (First output event) 489 ms.
| Metric | Value | n | Interval or range | Configuration |
|---|---|---|---|---|
| CLI vs API: time for a one-line answer (Total time) | 3.19 s | 5 | 2.9 s–3.8 s (range) | GPT-6 Luna · effort none · fixed exact reply, 5 runs |
| CLI vs API: time for a one-line answer (Total time) | 4.18 s | 5 | 3.9 s–4.5 s (range) | GPT-6.1 Sol · effort low · fixed exact reply, 5 runs |
| CLI vs API: time for a one-line answer (Total time) | 4.19 s | 5 | 3.8 s–4.7 s (range) | GPT-6.1 Sol · effort high · fixed exact reply, 5 runs |
| CLI vs API: time for a one-line answer (First useful output) | 2.79 s | 5 | 2.5 s–3.4 s (range) | GPT-6 Luna · effort none · fixed exact reply, 5 runs |
| CLI vs API: time for a one-line answer (First useful output) | 3.75 s | 5 | 3.4 s–4.1 s (range) | GPT-6.1 Sol · effort low · fixed exact reply, 5 runs |
| CLI vs API: time for a one-line answer (First useful output) | 3.79 s | 5 | 3.4 s–4.3 s (range) | GPT-6.1 Sol · effort high · fixed exact reply, 5 runs |
| CLI vs API: time for a small coding task (Total time) | 9.23 s | 3 | 9 s–11.7 s (range) | GPT-6 Luna · effort none · small coding task, 3 runs |
| CLI vs API: time for a small coding task (Total time) | 14.2 s | 3 | 13 s–14.4 s (range) | GPT-6.1 Sol · effort low · small coding task, 3 runs |
| CLI vs API: time for a small coding task (Total time) | 17.9 s | 3 | 17.7 s–22.4 s (range) | GPT-6.1 Sol · effort high · small coding task, 3 runs |
| CLI vs API: time for a small coding task (First useful output) | 8.68 s | 3 | 8.3 s–11 s (range) | GPT-6 Luna · effort none · small coding task, 3 runs |
| CLI vs API: time for a small coding task (First useful output) | 13.6 s | 3 | 12.5 s–13.8 s (range) | GPT-6.1 Sol · effort low · small coding task, 3 runs |
| CLI vs API: time for a small coding task (First useful output) | 17.3 s | 3 | 17.1 s–21.9 s (range) | GPT-6.1 Sol · effort high · small coding task, 3 runs |
| Hidden prompt: input tokens for the same one-line request | 18,859 | 5 | — | GPT-6 Luna · effort none · short fixed tasks |
| Hidden prompt: input tokens for the same one-line request | 19,551 | 5 | — | GPT-6.1 Sol · effort low · short fixed tasks |
| Hidden prompt: input tokens for the same one-line request | 19,555 | 5 | — | GPT-6.1 Sol · effort high · short fixed tasks |
| Repairing a scheduler: Claude Code vs Codex vs API (Total time) | 61.2 s | 3 | 59.9 s–69.5 s (range) | GPT-6.1 Sol · effort medium · scheduler repair, 296 checks, 3 runs |
| Repairing a scheduler: Claude Code vs Codex vs API (First useful output) | 15.6 s | 3 | 13.7 s–23 s (range) | GPT-6.1 Sol · effort medium · scheduler repair, 296 checks, 3 runs |
| Output tokens to repair the scheduler (Output tokens) | 1,181 | 3 | — | GPT-6.1 Sol · effort medium · scheduler repair, 296 checks, 3 runs |
| Codex CLI vs OpenAI API, median total time for a one-line answer | 3.5x slower | 30 | — | short fixed tasks |
Lines: fastest–slowest run (not an interval)n beside each valueMuted dots: the other configurations on the same chart
Codex CLI in Claude Code CLI vs Codex CLI vs the API: latency and tokens: 19 values, first CLI vs API: time for a one-line answer (Total time) 3.19 s.
Single call vs agent loop: does letting the model run code help? Haiku 4.5, Sonnet 5.5 and GPT-6 Luna on 8 hard tasks
26 values · open the study
Whiskers: 95% Wilson intervalLines: fastest–slowest run (not an interval)n beside each valueMuted dots: the other configurations on the same chartHollow: list-price calculation
Codex CLI in Single call vs agent loop: does letting the model run code help? Haiku 4.5, Sonnet 5.5 and GPT-6 Luna on 8 hard tasks: 26 values, first Strict pass rate: single call vs agent loop on eight hard tasks 63% (10/16).
Does a JSON schema stop format misses? Instructions vs schema mode in Claude Code and Codex CLI
16 values · open the study
100% (12/12)Calculation
100% (12/12)Calculation
100% (12/12)Calculation
100% (12/12)Calculation
Whiskers: 95% Wilson intervalLines: fastest–slowest run (not an interval)n beside each valueMuted dots: the other configurations on the same chartHollow: list-price calculation
Codex CLI in Does a JSON schema stop format misses? Instructions vs schema mode in Claude Code and Codex CLI: 16 values, first Does a JSON schema raise the pass rate? Instructions vs schema mode (Strict pass: the whole reply is the right JSON) 100% (12/12).
$0.0023Calculation
$0.0041Calculation
$0.0030Calculation
Lines: lowest–highest run (not an interval)n beside each valueMuted dots: the other configurations on the same chartHollow: list-price calculation
Codex CLI in How much of an AI bill is thinking? Reasoning tokens by model and effort: 18 values, first Reasoning share of output tokens per call on hard tasks (calculation) 57%.
| Metric | Value | n | Interval or range | Configuration |
|---|---|---|---|---|
| Time to first text: a 250-line answer, six models | 3.52 s | 4 | 2.8 s–4.4 s (range) | GPT-6.1 Sol · effort low |
| Time to first text: a 250-line answer, six models | 3.30 s | 4 | 3.2 s–3.5 s (range) | GPT-6 Luna · effort low |
| Output speed after the first text: visible tokens per second (calculation) Calculation | 80 | 4 | 72–81 (range) | GPT-6.1 Sol · effort low |
| Output speed after the first text: visible tokens per second (calculation) Calculation | 129 | 4 | 56–259 (range) | GPT-6 Luna · effort low |
| Output speed in characters per second after the first text (calculation) Calculation | 323 | 4 | 291–327 (range) | GPT-6.1 Sol · effort low |
| Output speed in characters per second after the first text (calculation) Calculation | 524 | 4 | 225–1,052 (range) | GPT-6 Luna · effort low |
| Time to first text as the prompt grows: 1k Calculation | 3.36 s | 3 | 3.4 s–4.8 s (range) | GPT-6.1 Sol · effort low |
| Time to first text as the prompt grows: 16k Calculation | 4.02 s | 3 | 3.3 s–4.3 s (range) | GPT-6.1 Sol · effort low |
| Time to first text as the prompt grows: 64k Calculation | 3.93 s | 3 | 3.4 s–4.4 s (range) | GPT-6.1 Sol · effort low |
| Total time per call by prompt size (1k prompt) | 3.43 s | 3 | 3.4 s–4.9 s (range) | GPT-6.1 Sol · effort low |
| Total time per call by prompt size (16k prompt) | 4.14 s | 3 | 4 s–4.7 s (range) | GPT-6.1 Sol · effort low |
| Total time per call by prompt size (64k prompt) | 3.96 s | 3 | 3.5 s–4.4 s (range) | GPT-6.1 Sol · effort low |
| Exact lookup answers at the 1k, 16k and 64k prompt-size targets | 100% (9/9) | 9 | 70%–100% (95% CI) | GPT-6.1 Sol · effort low |
Lines: fastest–slowest run (not an interval)Whiskers: 95% Wilson intervaln beside each valueMuted dots: the other configurations on the same chartHollow: list-price calculation
Codex CLI in Where the seconds go: first text, output speed and prompt size for 6 LLMs: 13 values, first Time to first text: a 250-line answer, six models 3.52 s.
GPT-6.1 Sol vs Claude Opus 5.5, Sonnet 5.5 and Haiku 4.5 on 4 harder tasks
10 values · open the study
| Metric | Value | n | Interval or range | Configuration |
|---|---|---|---|---|
| Pass rate on 4 harder tasks (Strict pass) | 69% (11/16) | 16 | 44%–86% (95% CI) | GPT-6.1 Sol · effort medium |
| Pass rate on 4 harder tasks (Lenient (format misses counted)) | 69% (11/16) | 16 | 44%–86% (95% CI) | GPT-6.1 Sol · effort medium |
| Calls that tried a tool although tools were off | 0% (0/16) | 16 | 0%–19% (95% CI) | GPT-6.1 Sol · effort medium |
| Strict pass rate by task: 10x10 nonogram | 75% (3/4) | 4 | 30%–95% (95% CI) | GPT-6.1 Sol · effort medium |
| Strict pass rate by task: Sudoku, 22 givens | 25% (1/4) | 4 | 4.6%–70% (95% CI) | GPT-6.1 Sol · effort medium |
| Strict pass rate by task: 6x6 Skyscrapers | 100% (4/4) | 4 | 51%–100% (95% CI) | GPT-6.1 Sol · effort medium |
| Strict pass rate by task: Seeded shuffle output | 75% (3/4) | 4 | 30%–95% (95% CI) | GPT-6.1 Sol · effort medium |
| Total time per call on harder tasks | 120.2 s | 13 | 46.2 s–273 s (range) | GPT-6.1 Sol · effort medium |
| Output tokens per call on harder tasks (Output tokens) | 4,994 | 13 | 2,099–13,413 (range) | GPT-6.1 Sol · effort medium |
| List-price cost per strict pass on harder tasks (calculation) Calculation | $0.083 | 16 | — | GPT-6.1 Sol · effort medium |
Whiskers: 95% Wilson intervalLines: fastest–slowest run (not an interval)n beside each valueMuted dots: the other configurations on the same chartHollow: list-price calculation
Codex CLI in GPT-6.1 Sol vs Claude Opus 5.5, Sonnet 5.5 and Haiku 4.5 on 4 harder tasks: 10 values, first Pass rate on 4 harder tasks (Strict pass) 69% (11/16).
Compare Codex CLI
Each bar counts the rows of one comparison: a side ahead only where its interval or range is apart, otherwise a tie or unclear.
vs coding cli
Claude Code vs Codex CLI
Claude Code ahead on 7 · Codex CLI ahead on 1 · 31 ties · 49 unclear
Watch
Live story · 211 sAgent Benchmarks, October 2026: 28 studies in one film
Agent Benchmarks, October 2026: 28 studies in one film
The headline of each of our 28 open benchmark studies, with sample sizes and intervals. Calculations labelled; failures counted.
Transcript
- October 2026 roundup · 28 studies. Agent Benchmarks: every headline. Recorded runs, 95% intervals where they exist, every failure counted. Calculations labelled.
- SWE-bench Verified: Agent resolved 25 of 33. The public panel averaged 74.1%; the intervals overlap, so no rank. Agent resolved (one attempt each): 76% (25/33) (n = 33, 95% CI 59–87%). Public panel mean, same instances: 74.1% (n = 33). Model cost per resolved instance (calculation, notional): $3.71 (n = 25). Caveat: n = 33: intervals are wide. This is a defect-finding run, not a ranking.
- SWE-bench, interim (3 of 8 pairs graded): Opus 5.5 as Agent’s brain resolved 2 of 3, Sonnet 5.5 1 of 3. Exact McNemar p = 1.0: no difference yet. Opus cost 2.6× as much, a calculation. Opus 5.5 in Agent: resolved (interim): 67% (2/3) (n = 3, 95% CI 21–94%). Sonnet 5.5 in Agent: same instances: 33% (1/3) (n = 3, 95% CI 6–79%). List-price cost, Opus vs Sonnet (calculation): 2.6× (n = 3). Caveat: Different platform builds: Opus ran on 4f6f4027; the Sonnet attempts ran on f0ac3a8a and 236c0d3f. Platform changes can move results by themselves, so a gap compares Sonnet on an older platform with Opus on the current one, not the models alone.
- Blind critics preferred the AI pull request to the merged human one on 9 of 12 tasks at the latest attempt, 6 of 12 at the first. AI preferred, latest attempt: 75% (9/12) (n = 12, 95% CI 47–91%). AI preferred, first scored attempt: 50% (6/12) (n = 12, 95% CI 25–75%). Single critic verdicts for the AI change: 69% (91/132) (n = 132, 95% CI 61–76%). Caveat: Most critics are Anthropic models, and the worker runs on an Anthropic model. Same-family preference is possible; the per-critic chart shows the OpenAI critics’ share.
- Five short tasks, nine setups: 127 of 130 calls passed, so speed and tokens separate them. Fastest median: Fable 5.1 at 1.9 s. Calls that passed their validator: 98% (127/130) (n = 130, 95% CI 93–99%). Fastest median total time (Fable 5.1): 1.9 s (n = 15). Median input tokens per call: Codex CLI vs Claude Code: 12,124 vs 2,130 (n = 130). Caveat: CLI timings include CLI start-up and the CLI’s own system prompt.
- Eight hard tasks: Sonnet 5.5, Opus 5.5, Opus 5.5 high, GPT-6.1 Sol medium, Fable 5.1 and GPT-6.1 Sol high passed every call (24/24 or 16/16). Haiku 4.5 passed 11/24. Chart: Eight hard tasks · strict pass rate · 95% intervals (n = 16–24 each). Caveat: Claude Code and Codex CLI rows pair a CLI with a model, and each CLI adds its own system prompt and start-up time. A Claude-vs-GPT row compares the route + model pairs, not the models alone.
- Coding agents on hidden-test tasks: 36 of 36 sessions passed, so time separates them. Claude Code with Sonnet 5.5: median 23.1 s; Codex CLI took 4.9× as long, with extra standing instructions. Sessions that passed every hidden check: 100% (36/36) (n = 36, 95% CI 90–100%). Median time, Sonnet 5.5 in Claude Code: 23.1 s (n = 12). Median time, Codex CLI vs Claude Code with Sonnet: 4.9× (n = 12). Caveat: Codex CLI ran with the tester’s global AGENTS.md: --ignore-user-config does not switch that file off. 12 of 12 Codex sessions read the tester’s notes, 10 wrote a WORKLOG.md and 9 reported a commit attempt. Its time, tool calls and lines changed include that work. Claude Code ran with setting sources off, and no Claude session did any of it.
- Effort ladder: 11 model and effort settings passed 176 of 176 calls strictly (16/16 each, 81%–100%). No effort level wins any of 172 rows. Calls that passed strictly, low to high effort: 100% (176/176) (n = 176, 95% CI 98–100%). Model and effort configurations: 11 (6 new, 5 reference). Effort comparison rows where one level wins: 0 (of 172 rows · 16 pairs). Caveat: Every cell passed every call, so the set has a ceiling: a perfect 16/16 has a 95% interval of 81% to 100%. This study cannot show that effort does not matter on harder work; it shows that these 8 tasks do not need more than low effort.
- Caching, a list-price calculation: 50% less on Sonnet 5.5, 53% on Opus 5.5. Same prompt 10 times: 7 of 9 cells passed every time. Sonnet 5.5 session cost saved by the cache (calculation): 50% ($0.1350 vs $0.2698). Later sessions that reused an earlier session’s cache: 0 of 4. Model-and-prompt cells that passed 10 of 10: 7 of 9. Caveat: Cross-session reuse did not happen here. The cause was not tested: the CLI may add per-process context before the user message. Do not read it as a provider property.
- Agent memory: Sonnet 5.5 followed team-only rules 40% (6/15) of the time without memory and 100% (15/15) with an 11-line file. Without the late-fee rate it asked 7/9 times. Team knowledge, no memory: 40% (6/15) (n = 15, 95% CI 20–64%). Team knowledge, 11-line file: 100% (15/15) (n = 15, 95% CI 80–100%). Asked for the missing rate: 7/9. Caveat: One small synthetic repository and one author of the facts: the curated file is an upper bound written with knowledge of the tasks.
- System One arena: on 1,085 checkable decisions, Jev 1.13 answered 76.8% right; the best open model, Clef 27B, 69.9%. Jev 1.13: 76.8% (805/1048) (n = 1048, 95% CI 74–79%). Clef 27B: 69.9% (733/1048) (n = 1048, 95% CI 67–73%). Checkable decisions: 1,085. Caveat: Local models ran quantized on one Mac; the vendors measured full-precision weights on data-centre GPUs. Latency on other hardware will differ, and Jev latency includes the network.
- Routing: Jev 74/82, Haiku 4.5 73/82, Sonnet 5.5 77/82 exact, and the intervals overlap. List-price cost is where they differ. Chart: Cost per 1,000 routing decisions (n = 82–246 each). Calculation, not a run. Caveat: The case sets and question wording were revised in fix waves against Jev answers on 2026-10-04 and 2026-10-05, so Jev has a home advantage.
- Routing overhead: an in-process policy decides in 1.42 µs; Sonnet 5.5 as a router takes 2.60 s through the CLI, about 1.8 million times longer. Chart: Time per routing decision · log scale · median to p95 (n = 82–20000 each). Caveat: The policy is timed in process and the LLM routers through a CLI: this compares the two ways of routing as deployed, not two models on equal footing. A direct API call would skip the CLI time (shown separately).
- Provider prices, reported by OpenRouter: open-weight models vary up to 12.6x across providers. Closed models: one price, plus a 5.5% credit fee. Largest price spread (DeepSeek V4 Flash 0423): 12.6x (15 providers · blended price). Gateway price equals the first-party price: 10 of 10. Endpoints in the snapshot: 265 endpoints (52 providers, 27 models). Caveat: Every price is third-party-reported by OpenRouter’s API at 2026-10-06. Prices change often; refetch before relying on them.
- Recorded SWE-bench tokens, repriced: $87.23 on Sonnet 5.5, $143.83 on Opus 5.5, $343.33 with no prompt cache. At Sonnet 5.5 prices (the model that ran): $87.23 (n = 33). At Opus 5.5 prices: $143.83 (n = 33). Sonnet 5.5 without the prompt cache: $343.33 (n = 33). Calculation, not a run. Caveat: Every repriced figure is a calculation, not a run. Only the Sonnet 5.5 row matches the model that produced the tokens.
- What a CLI adds: for a one-line answer the Codex CLI was 3.5x slower than the API and sent 19,551 input tokens. Codex CLI vs OpenAI API, one-line answer: 3.5x slower (n = 30). Input tokens the Codex CLI sends for one line: 19,551 (n = 15). Scheduler repair, median: Claude Code vs Codex CLI: 15.0 s vs 61.2 s (n = 3, models differ too). Caveat: Small samples: 3 to 5 runs per configuration. Medians with ranges, not intervals.
- Three real pull requests: the latest build verified 1 of 3, at $11.06 notional. Guardrail refusals went 26 → 19. Verified deliveries, latest build: 1 of 3 (n = 3). Notional cost, latest build, 3 tasks: $11.06 (n = 3). Guardrail refusals, first vs latest slice: 26 → 19 (n = 3). Caveat: One attempt per cell: these are defect-finding runs, not rates.
- Agent-loop strict passes: Haiku 54% (13/24); Sonnet 100% (16/16). Attempts excluded for outside reads: 2 of 56. Claude Haiku 4.5 strict pass rate, agent loop: 54% (13/24) (n = 24, 95% CI 35–72%). Claude Sonnet 5.5 strict pass rate, agent loop: 100% (16/16) (n = 16, 95% CI 81–100%). Agent-loop attempts left out for reading outside the work folder: 2 of 56 (n = 56). Caveat: The Claude single-call cells ran in another batch on 2026-10-06 (03:23 to 04:02 UTC), with the same CLI version, tasks and validators; provider load can differ by hour.
- Haiku exact routing: thinking off 87% (71/82), on 89% (73/82). Paired McNemar p = 0.754; day and account differ. Claude Haiku 4.5 (thinking off): exact routing decisions: 87% (71/82) (n = 82, 95% CI 78–92%). Claude Haiku 4.5 (thinking on): exact routing decisions: 89% (73/82) (n = 82, 95% CI 80–94%). Paired exact test, thinking off vs on (exact McNemar p): 0.754 (n = 82). Caveat: The thinking-on and Sonnet arms are the recorded routing run of 2026-10-05, reused. The thinking-off arm ran on a different day and on a different Claude subscription account, so this is a confounded comparison. Day, account, CLI version and input-token differences can affect the results. The data cannot isolate the effect of thinking.
- Format misses: instructions 35% (17/48), JSON schema 0% (0/48). The schema still gave wrong values: 13% (6/48). Format-miss rate with instructions only, all models: 35% (17/48) (n = 48, 95% CI 23–50%). Format-miss rate with a JSON schema, all models: 0% (0/48) (n = 48, 95% CI 0–7%). Wrong-values rate with a JSON schema, all models: 13% (6/48) (n = 48, 95% CI 6–25%). Caveat: Small samples: Haiku 24 calls per mode, Sonnet 12 calls per mode and GPT-6.1 Sol 12 calls per mode. A 12/12 result has a 95% interval of 76% to 100%. This is not a minimum detectable difference.
- Later Claude sessions with substantial cache reuse: new folders 0 of 2 (95% interval 0% to 66%); a fixed folder 2 of 2 (95% interval 34% to 100%). This analysis is exploratory. Later sessions with at least 50% of turn-1 input cached, A: new folder each time: 0 of 2 (95% interval 0% to 66%) (n = 2, 95% CI 0–66%). Later sessions with at least 50% of turn-1 input cached, B: fixed folder: 2 of 2 (95% interval 34% to 100%) (n = 2, 95% CI 34–100%). Caveat: The surviving protocol file was created after all counted calls. Its claimed 00:32 UTC declaration is not supported by its file birth time. Amendment 1 and 2 state 00:36 and 00:37 UTC, but separate pre-edit copies do not verify those times. The current summary was regenerated at 07:41 UTC. Treat the analysis as exploratory.
- Cache break-even, a calculation: a new prefix with a one-hour write needs 2 reuses (the 3rd request). Recorded session payback: 2 turns (6 of 6 sessions). Reuses before a 1-hour cached prefix costs less, whole prefix new (calculation): 2 reuses (the 3rd request). Turn at which a recorded Claude Code session’s total input cost with the cache first fell below its cost with no cache (calculation on recorded tokens): 2 turns (6 of 6 sessions) (n = 6). Calculation, not a run. Caveat: The retained Claude protocol file was created at 14:43:55 UTC, after the first counted session at 14:35:39 UTC on 2026-10-06. Its declaration says 14:25 UTC, but file times do not verify that claim. Treat this as a retrospective protocol record.
- Exact unseen routing decisions: Jev 82% (46/56), Haiku 79% (44/56), Sonnet 88% (49/56). The intervals overlap; no rank. Jev 1.13 (TypeSafe): exact on unseen decisions: 82% (46/56) (n = 56, 95% CI 70–90%). Claude Haiku 4.5 · Claude Code: exact on unseen decisions: 79% (44/56) (n = 56, 95% CI 66–87%). Claude Sonnet 5.5 (low) · Claude Code: exact on unseen decisions: 88% (49/56) (n = 56, 95% CI 76–94%). Caveat: The case author and two of the three routers are Claude models, so a same-family label bias is possible. A second labeller (GPT-6.1 Sol) labelled every case blind; the secondary scoring keeps only the keys where its label is inside our acceptable set.
- Reasoning cost at high vs low effort, a calculation from recorded calls: Sonnet 2.2x ($0.0043 at low, $0.0094 at high); Opus 3.6x ($0.0050 at low, $0.0180 at high). Reasoning cost per call, high ÷ low effort, Claude Sonnet 5.5 · Claude Code (calculation, means): 2.2x ($0.0043 at low, $0.0094 at high) (n = 16). Reasoning cost per call, high ÷ low effort, Claude Opus 5.5 · Claude Code (calculation, means): 3.6x ($0.0050 at low, $0.0180 at high) (n = 16). Calculation, not a run. Caveat: The hard-set protocol file was created after its first counted call. Both effort-ladder protocol files were created after their batches ended. The short-set file predates its first call, but its top-up amendment timing is unverified. Batch receipts preserve protocol text, but we cannot verify all rules were written before inference. Treat these as exploratory calculations, not preregistered tests.
- Speed anatomy, a calculation: same-text token count ratio 1.8x. Extra time to first text with the larger prompt: Haiku +0.9 s; Sonnet +1.6 s. The same 4,327-character reply: most tokens ÷ fewest tokens across models (calculation): 1.8x (n = 23). Haiku: extra time to first text at 64k vs 1k (calculation): +0.9 s (n = 6). Sonnet: extra time to first text at 64k vs 1k (calculation): +1.6 s (n = 6). Calculation, not a run. Caveat: 4 calls per model in part A and 3 per size in part B. Medians of so few calls move with one slow call, and the ranges are not confidence intervals. The 95% Wilson intervals on the lookup rates are wide.
- Cost per correct answer, a calculation: Sonnet every time $0.0132; Haiku with one retry then Sonnet $0.0566. No policy ran. Cost per correct answer, Sonnet 5.5 every time (calculation): $0.0132 (n = 24). Cost per correct answer, Haiku with one retry then Sonnet (calculation): $0.0566 (n = 48). Calculation, not a run. Caveat: Advance registration is not verified. The hard-set protocol file birth time is 2026-10-06 04:03:37 UTC; its first counted call started at 03:23:59 UTC. The ladder protocol file birth time is 14:35:11 UTC; its first Claude call started at 14:21:50 UTC. Both files claim advance declaration, but the available file times do not support that claim. Copying could explain the times; we cannot establish it.
- Strict passes on four selected harder tasks: Sol 69% (11/16); Opus 42% (5/12); Sonnet 38% (6/16). Tasks were selected with a Sonnet pilot. GPT-6.1 Sol (medium) (Codex CLI): strict pass rate on the harder tasks: 69% (11/16) (n = 16, 95% CI 44–86%). Opus 5.5 (Claude Code): strict pass rate on the harder tasks: 42% (5/12) (n = 12, 95% CI 19–68%). Sonnet 5.5 (Claude Code): strict pass rate on the harder tasks: 38% (6/16) (n = 16, 95% CI 18–61%). Caveat: Selection effect: the study picked tasks that Sonnet did not pass twice in the pilot. Sonnet passed 2 of 8 pilot calls on the kept tasks and 6 of 16 counted calls (calculation: 25% and 38%; the intervals overlap). Selection can produce this pattern, but the run does not establish its cause.
- Latency budget, a calculation: steps within an assumed 300 ms budget 3 of 14 (median: 3 of 14); within 1,500 ms 4 of 14 (median: 8 of 14), at p95 or the observed maximum. No voice turn ran. Steps that fit a 300 ms budget at the slow end (calculation): 3 of 14 (median: 3 of 14) (n = 14). Steps that fit a 1,500 ms budget at the slow end (calculation): 4 of 14 (median: 8 of 14) (n = 14). Calculation, not a run. Caveat: Timing samples omit 0 failed or untimed matched explorer calls and 0 failed or untimed Claude head-to-head calls. A failure is not a fast successful step. No pass rate is estimated here.
- Routing cost at an assumed million decisions a day, a calculation: Jev $33.70; Sonnet $7,324. No load test ran. Daily cost at 1 million decisions, Jev (calculation): $33.70 (n = 82). Daily cost at 1 million decisions, Sonnet 5.5 router (calculation): $7,324 (n = 82). Calculation, not a run. Caveat: The routing and live Jev protocols predate the first counted calls by file birth time. The overhead protocol predates its microbenchmark output. Claude ran 82 calls per router, below its 120-call cap. Jev ran 246 counted calls, below its 300-call cap. One Haiku pilot and one Jev probe are excluded. All counted calls completed without call errors. The policy microbenchmark used 5,000 warm-up iterations and 64 synthetic contexts; the minimum and individual timing samples are unavailable, so we cannot rebuild its quantiles or full range. No separate validator-control receipt or Sonnet pilot is retained.
- 28 studies, one place. Intervals, sources and every failure kept.
Live story · 48 sDoes more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on eight hard tasks
Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on eight hard tasks
All 11 configurations passed 16/16 strictly (95% interval 81%–100%): more effort bought no extra passes on this set (a ceiling). Median output tokens rose with effort; time ranges overlap. Costs are list-price calculations.
Transcript
- Effort ladder · 176 calls · 11 configurations. Does more effort buy quality? Low, medium and high effort on eight hard tasks. Sonnet 5.5 and Opus 5.5 via Claude Code, GPT-6.1 Sol via Codex CLI.
- 176 of 176 calls passed strictly, at every effort level. No format misses and no wrong answers. Calls that passed strictly, every effort: 100% (176/176) (n = 176, 95% CI 98–100%). Model and effort configurations: 11 (6 new, 5 reference) (n = 176). Format misses and wrong answers: 0 · 0 (n = 176, format · wrong). Caveat: Reference cells ran in a different batch and hour than the new cells (provider load can differ), on the same host, CLI versions, cases and flags.
- All 11 configurations passed 16/16. A 16/16 has a 95% interval of 81%–100%: the set has a ceiling, so no effort level wins. Chart: Strict pass rate by effort · 95% Wilson intervals (n = 16 each). Caveat: Every cell passed every call, so the set has a ceiling: a perfect 16/16 has a 95% interval of 81% to 100%. This study cannot show that effort does not matter on harder work; it shows that these 8 tasks do not need more than low effort.
- Median time: Sonnet 5.5 5.8 s at low, 8.8 s at high; Opus 5.5 7.5 s at low, 10.1 s at high; GPT-6.1 Sol 13.6 s at low, 18.1 s at high. Within each model the ranges overlap: no effort is ranked faster. Chart: Median total time per call · whiskers: fastest to slowest call (n = 16 each). Caveat: Each cell has only 2 calls per task. Medians of 16 calls move with a few slow calls; the ranges are wide and overlap.
- More effort writes more: Sonnet 5.5 used a median 667 / 770 / 1,192 output tokens at low / medium / high. Chart: Median output and reasoning tokens per call (n = 16 each). Caveat: Claude Code and Codex CLI rows pair a CLI with a model; each CLI adds its own system prompt and start-up time. Effort levels are not the same scale across vendors.
- List-price cost per strict pass, low → high effort, a calculation: Sonnet $0.0122 → $0.0167, Opus $0.0212 → $0.0337, GPT-6.1 Sol $0.0128 → $0.0151. Chart: List-price cost per strict pass · calculation (n = 16 each). Calculation, not a run. Caveat: List-price costs are calculations; the calls used flat subscriptions.
- 16 effort pairs, 172 comparison rows: no effort level wins any row. The rest are ties or unclear. Effort pairs compared (same model, same route): 16. Comparison rows across those pairs: 172. Rows where one effort level wins: 0 (of 172). Caveat: Every cell passed every call, so the set has a ceiling: a perfect 16/16 has a 95% interval of 81% to 100%. This study cannot show that effort does not matter on harder work; it shows that these 8 tasks do not need more than low effort.
- Start at low effort and measure. Every call, interval and cost online.
Write-ups that use this data
All postsA latency budget for voice agents: which LLM steps fit in one turn?
136.5 ms for Jev, 0.82 s for a small-model API, 2.79 s to 3.79 s for Codex CLI: which steps fit a voice agent latency budget? A thought experiment.
A voice agent latency budget, with measured times: what fits in one turn?
Rules and Jev 1.13 fit every budget we assumed; a Claude router through a CLI fits none. 14 measured steps vs 300, 800 and 1,500 ms. A thought experiment.
AI coding agent best practices: 12 rules, each backed by a measurement
12 rules for running AI coding agents, each with one measured number: validation, model choice, effort, caching, memory, routing, CLIs and sample size.