14 measured metrics · 5 calculated · 4 studies
GPT-6.1 Sol (Codex CLI)lowvshigheffort
No row separates them: 3 ties, 16 unclear.
The verdict
GPT-6.1 Sol (Codex CLI) at low effort and GPT-6.1 Sol (Codex CLI) at high effort share 14 measured metrics and 5 list-price calculations from 4 studies. Only the effort setting differs between the two sides of a row; the route and the task set are the same. No row separates them: every interval or run range overlaps, too few runs were recorded, no interval was recorded, or more is not better for that metric. The rows are 3 ties and 16 unclear; each row says why. Calculation rows are derived from list prices and recorded counts; they are not bills or runs. Some rows rest on small samples (n = 3 at the smallest).
Headline metrics
How far apart the two sides are on the headline metrics. Length is the ratio of the two values; it is not a winner.
Bar length is the ratio of the two values on a log scale, pointing to the larger one. Larger is not better for time, tokens or cost. A bar has a side’s color only when that side is ahead in the data; gray means the data does not separate them.
| Metric | GPT-6.1 Sol (Codex CLI) at low effort | GPT-6.1 Sol (Codex CLI) at high effort | n | Interval or range | Outcome | Basis | Study |
|---|---|---|---|---|---|---|---|
| Pass rate on five validated tasks | 100% (10/10)Codex CLI · effort low · five short validated tasks | 100% (15/15)Codex CLI · effort high · five short validated tasks | 10 / 15 | 95% CI: 72%–100% vs 80%–100% | Tie | The 95% intervals overlap (GPT-6.1 Sol (Codex CLI) at low effort 72% to 100%; GPT-6.1 Sol (Codex CLI) at high effort 80% to 100%), so this sample cannot separate them. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Total time per call | 6.26 sCodex CLI · effort low · five short validated tasks | 5.60 sCodex CLI · effort high · five short validated tasks | 10 / 15 | range: 4.7 s–10.5 s vs 4.1 s–19.5 s | Unclear | The run ranges (fastest to slowest) overlap (GPT-6.1 Sol (Codex CLI) at low effort 4.65 s to 10.5 s; GPT-6.1 Sol (Codex CLI) at high effort 4.05 s to 19.5 s); the medians alone do not show a reliable difference. A range is not a confidence interval. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Time to first useful output | 5.14 sCodex CLI · effort low · five short validated tasks | 5.32 sCodex CLI · effort high · five short validated tasks | 10 / 15 | range: 4 s–8.5 s vs 3.6 s–16.4 s | Unclear | The run ranges (fastest to slowest) overlap (GPT-6.1 Sol (Codex CLI) at low effort 4.02 s to 8.50 s; GPT-6.1 Sol (Codex CLI) at high effort 3.64 s to 16.4 s); the medians alone do not show a reliable difference. A range is not a confidence interval. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Input tokens per call: what the CLI sends (Cache read) | 8,064Codex CLI · effort low · five short validated tasks | 6,716Codex CLI · effort high · five short validated tasks | 10 / 15 | none recorded | Unclear | More or fewer tokens is not better or worse by itself; this row describes behaviour, not a winner. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Output tokens per call (Output tokens) | 42Codex CLI · effort low · five short validated tasks | 42Codex CLI · effort high · five short validated tasks | 10 / 15 | none recorded | Tie | Same value. More or fewer is not better by itself for this metric. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| List-price cost per passing answer (calculation)Calculation | $0.010Codex CLI · effort low · five short validated tasks | $0.013Codex CLI · effort high · five short validated tasks | 10 / 15 | none recorded | Unclear | No interval or range was recorded for either side, so the gap ($0.010 vs $0.013) is not tested against run-to-run variation. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
Marks: 95% intervals (Wilson for rates); fastest–slowest run ranges (not intervals)n is shown per side on every rowHollow marks: list-price calculations, not runs
6 headline metrics as the ratio of the two values. None of them separates the sides in the data. Widest ratio: List-price cost per passing answer (calculation), 1.3x (High effort larger).
Watch it build
A live story drawn from the same dataset in your browser. It shows the same numbers, notes and interval labels as this page.
Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on eight hard tasks
All 11 configurations passed 16/16 strictly (95% interval 81%–100%): more effort bought no extra passes on this set (a ceiling). Median output tokens rose with effort; time ranges overlap. Costs are list-price calculations.
Transcript
- Effort ladder · 176 calls · 11 configurations. Does more effort buy quality? Low, medium and high effort on eight hard tasks. Sonnet 5.5 and Opus 5.5 via Claude Code, GPT-6.1 Sol via Codex CLI.
- 176 of 176 calls passed strictly, at every effort level. No format misses and no wrong answers. Calls that passed strictly, every effort: 100% (176/176) (n = 176, 95% CI 98–100%). Model and effort configurations: 11 (6 new, 5 reference) (n = 176). Format misses and wrong answers: 0 · 0 (n = 176, format · wrong). Caveat: Reference cells ran in a different batch and hour than the new cells (provider load can differ), on the same host, CLI versions, cases and flags.
- All 11 configurations passed 16/16. A 16/16 has a 95% interval of 81%–100%: the set has a ceiling, so no effort level wins. Chart: Strict pass rate by effort · 95% Wilson intervals (n = 16 each). Caveat: Every cell passed every call, so the set has a ceiling: a perfect 16/16 has a 95% interval of 81% to 100%. This study cannot show that effort does not matter on harder work; it shows that these 8 tasks do not need more than low effort.
- Median time: Sonnet 5.5 5.8 s at low, 8.8 s at high; Opus 5.5 7.5 s at low, 10.1 s at high; GPT-6.1 Sol 13.6 s at low, 18.1 s at high. Within each model the ranges overlap: no effort is ranked faster. Chart: Median total time per call · whiskers: fastest to slowest call (n = 16 each). Caveat: Each cell has only 2 calls per task. Medians of 16 calls move with a few slow calls; the ranges are wide and overlap.
- More effort writes more: Sonnet 5.5 used a median 667 / 770 / 1,192 output tokens at low / medium / high. Chart: Median output and reasoning tokens per call (n = 16 each). Caveat: Claude Code and Codex CLI rows pair a CLI with a model; each CLI adds its own system prompt and start-up time. Effort levels are not the same scale across vendors.
- List-price cost per strict pass, low → high effort, a calculation: Sonnet $0.0122 → $0.0167, Opus $0.0212 → $0.0337, GPT-6.1 Sol $0.0128 → $0.0151. Chart: List-price cost per strict pass · calculation (n = 16 each). Calculation, not a run. Caveat: List-price costs are calculations; the calls used flat subscriptions.
- 16 effort pairs, 172 comparison rows: no effort level wins any row. The rest are ties or unclear. Effort pairs compared (same model, same route): 16. Comparison rows across those pairs: 172. Rows where one effort level wins: 0 (of 172). Caveat: Every cell passed every call, so the set has a ceiling: a perfect 16/16 has a 95% interval of 81% to 100%. This study cannot show that effort does not matter on harder work; it shows that these 8 tasks do not need more than low effort.
- Start at low effort and measure. Every call, interval and cost online.
Metric by metric
Both values of a row come from the same chart in the same study. Each row has its own axis. The shaded band is where the two intervals or ranges overlap: a side is ahead only when they do not.
- Low effort
- High effort
- 95% interval
- fastest–slowest run (not an interval)
- where the two overlap
- hollow: list-price calculation
Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head
- Pass rate on five validated tasks100% (10/10)n 10100% (15/15)n 15TiePass rate on five validated tasks: Low effort 100% (10/10) (n 10, 95% interval 72%–100%); High effort 100% (15/15) (n 15, 95% interval 80%–100%). Tie.
- Total time per call6.26 sn 105.60 sn 15UnclearTotal time per call: Low effort 6.26 s (n 10, run range 4.7 s–10.5 s); High effort 5.60 s (n 15, run range 4.1 s–19.5 s). Unclear.
- Time to first useful output5.14 sn 105.32 sn 15UnclearTime to first useful output: Low effort 5.14 s (n 10, run range 4 s–8.5 s); High effort 5.32 s (n 15, run range 3.6 s–16.4 s). Unclear.
- Input tokens per call: what the CLI sends (Cache read)8,064n 106,716n 15UnclearInput tokens per call: what the CLI sends (Cache read): Low effort 8,064 (n 10); High effort 6,716 (n 15). Unclear.
- Input tokens per call: what the CLI sends (Other input)4,059n 105,406n 15UnclearInput tokens per call: what the CLI sends (Other input): Low effort 4,059 (n 10); High effort 5,406 (n 15). Unclear.
- Output tokens per call (Output tokens)42n 1042n 15TieOutput tokens per call (Output tokens): Low effort 42 (n 10); High effort 42 (n 15). Tie.
- List-price cost per call (calculation)Calculation$0.0077n 10$0.010n 15UnclearList-price cost per call (calculation), calculation: Low effort $0.0077 (n 10, run range $0.0074–$0.026); High effort $0.010 (n 15, run range $0.0066–$0.028). Unclear.
- List-price cost per passing answer (calculation)Calculation$0.010n 10$0.013n 15UnclearList-price cost per passing answer (calculation), calculation: Low effort $0.010 (n 10); High effort $0.013 (n 15). Unclear.
Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks
- Strict pass rate by effort on eight hard tasks100% (16/16)n 16100% (16/16)n 16TieStrict pass rate by effort on eight hard tasks: Low effort 100% (16/16) (n 16, 95% interval 81%–100%); High effort 100% (16/16) (n 16, 95% interval 81%–100%). Tie.
- Total time per call by effort on hard tasks13.6 sn 1618.1 sn 16UnclearTotal time per call by effort on hard tasks: Low effort 13.6 s (n 16, run range 7.9 s–44.3 s); High effort 18.1 s (n 16, run range 11.7 s–92.2 s). Unclear.
- Output tokens per call by effort on hard tasks (Output tokens)284n 16436n 16UnclearOutput tokens per call by effort on hard tasks (Output tokens): Low effort 284 (n 16); High effort 436 (n 16). Unclear.
- List-price cost per strict pass by effort (calculation)Calculation$0.013n 16$0.015n 16UnclearList-price cost per strict pass by effort (calculation), calculation: Low effort $0.013 (n 16); High effort $0.015 (n 16). Unclear.
Claude Code CLI vs Codex CLI vs the API: latency and tokens
- CLI vs API: time for a one-line answer (Total time)4.18 sn 54.19 sn 5UnclearCLI vs API: time for a one-line answer (Total time): Low effort 4.18 s (n 5, run range 3.9 s–4.5 s); High effort 4.19 s (n 5, run range 3.8 s–4.7 s). Unclear.
- CLI vs API: time for a one-line answer (First useful output)3.75 sn 53.79 sn 5UnclearCLI vs API: time for a one-line answer (First useful output): Low effort 3.75 s (n 5, run range 3.4 s–4.1 s); High effort 3.79 s (n 5, run range 3.4 s–4.3 s). Unclear.
- CLI vs API: time for a small coding task (Total time)14.2 sn 317.9 sn 3UnclearCLI vs API: time for a small coding task (Total time): Low effort 14.2 s (n 3, run range 13 s–14.4 s); High effort 17.9 s (n 3, run range 17.7 s–22.4 s). Unclear.
- CLI vs API: time for a small coding task (First useful output)13.6 sn 317.3 sn 3UnclearCLI vs API: time for a small coding task (First useful output): Low effort 13.6 s (n 3, run range 12.5 s–13.8 s); High effort 17.3 s (n 3, run range 17.1 s–21.9 s). Unclear.
- Hidden prompt: input tokens for the same one-line request19,551n 519,555n 5UnclearHidden prompt: input tokens for the same one-line request: Low effort 19,551 (n 5); High effort 19,555 (n 5). Unclear.
How much of an AI bill is thinking? Reasoning tokens by model and effort
- Reasoning cost per strict pass by effort, with the total (calculation) (Reasoning cost per strict pass)Calculation$0.0012n 16$0.0041n 16UnclearReasoning cost per strict pass by effort, with the total (calculation) (Reasoning cost per strict pass), calculation: Low effort $0.0012 (n 16); High effort $0.0041 (n 16). Unclear.
- Reasoning cost per strict pass by effort, with the total (calculation) (Total cost per strict pass)Calculation$0.013n 16$0.015n 16UnclearReasoning cost per strict pass by effort, with the total (calculation) (Total cost per strict pass), calculation: Low effort $0.013 (n 16); High effort $0.015 (n 16). Unclear.
| Metric | GPT-6.1 Sol (Codex CLI) at low effort | GPT-6.1 Sol (Codex CLI) at high effort | n | Interval or range | Outcome | Basis | Study |
|---|---|---|---|---|---|---|---|
| Pass rate on five validated tasks | 100% (10/10)Codex CLI · effort low · five short validated tasks | 100% (15/15)Codex CLI · effort high · five short validated tasks | 10 / 15 | 95% CI: 72%–100% vs 80%–100% | Tie | The 95% intervals overlap (GPT-6.1 Sol (Codex CLI) at low effort 72% to 100%; GPT-6.1 Sol (Codex CLI) at high effort 80% to 100%), so this sample cannot separate them. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Total time per call | 6.26 sCodex CLI · effort low · five short validated tasks | 5.60 sCodex CLI · effort high · five short validated tasks | 10 / 15 | range: 4.7 s–10.5 s vs 4.1 s–19.5 s | Unclear | The run ranges (fastest to slowest) overlap (GPT-6.1 Sol (Codex CLI) at low effort 4.65 s to 10.5 s; GPT-6.1 Sol (Codex CLI) at high effort 4.05 s to 19.5 s); the medians alone do not show a reliable difference. A range is not a confidence interval. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Time to first useful output | 5.14 sCodex CLI · effort low · five short validated tasks | 5.32 sCodex CLI · effort high · five short validated tasks | 10 / 15 | range: 4 s–8.5 s vs 3.6 s–16.4 s | Unclear | The run ranges (fastest to slowest) overlap (GPT-6.1 Sol (Codex CLI) at low effort 4.02 s to 8.50 s; GPT-6.1 Sol (Codex CLI) at high effort 3.64 s to 16.4 s); the medians alone do not show a reliable difference. A range is not a confidence interval. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Input tokens per call: what the CLI sends (Cache read) | 8,064Codex CLI · effort low · five short validated tasks | 6,716Codex CLI · effort high · five short validated tasks | 10 / 15 | none recorded | Unclear | More or fewer tokens is not better or worse by itself; this row describes behaviour, not a winner. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Input tokens per call: what the CLI sends (Other input) | 4,059Codex CLI · effort low · five short validated tasks | 5,406Codex CLI · effort high · five short validated tasks | 10 / 15 | none recorded | Unclear | More or fewer tokens is not better or worse by itself; this row describes behaviour, not a winner. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Output tokens per call (Output tokens) | 42Codex CLI · effort low · five short validated tasks | 42Codex CLI · effort high · five short validated tasks | 10 / 15 | none recorded | Tie | Same value. More or fewer is not better by itself for this metric. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| List-price cost per call (calculation)Calculation | $0.0077Codex CLI · effort low · five short validated tasks | $0.010Codex CLI · effort high · five short validated tasks | 10 / 15 | range: $0.0074–$0.026 vs $0.0066–$0.028 | Unclear | The run ranges (fastest to slowest) overlap (GPT-6.1 Sol (Codex CLI) at low effort $0.0074 to $0.026; GPT-6.1 Sol (Codex CLI) at high effort $0.0066 to $0.028); the medians alone do not show a reliable difference. A range is not a confidence interval. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| List-price cost per passing answer (calculation)Calculation | $0.010Codex CLI · effort low · five short validated tasks | $0.013Codex CLI · effort high · five short validated tasks | 10 / 15 | none recorded | Unclear | No interval or range was recorded for either side, so the gap ($0.010 vs $0.013) is not tested against run-to-run variation. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Strict pass rate by effort on eight hard tasks | 100% (16/16)Codex CLI · effort low · eight hard validated tasks, effort ladder | 100% (16/16)Codex CLI · effort high · eight hard validated tasks, effort ladder | 16 | 95% CI: 81%–100% vs 81%–100% | Tie | The 95% intervals overlap (GPT-6.1 Sol (Codex CLI) at low effort 81% to 100%; GPT-6.1 Sol (Codex CLI) at high effort 81% to 100%), so this sample cannot separate them. | Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks |
| Total time per call by effort on hard tasks | 13.6 sCodex CLI · effort low · eight hard validated tasks, effort ladder | 18.1 sCodex CLI · effort high · eight hard validated tasks, effort ladder | 16 | range: 7.9 s–44.3 s vs 11.7 s–92.2 s | Unclear | The run ranges (fastest to slowest) overlap (GPT-6.1 Sol (Codex CLI) at low effort 7.94 s to 44.3 s; GPT-6.1 Sol (Codex CLI) at high effort 11.7 s to 92.2 s); the medians alone do not show a reliable difference. A range is not a confidence interval. | Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks |
| Output tokens per call by effort on hard tasks (Output tokens) | 284Codex CLI · effort low · eight hard validated tasks, effort ladder | 436Codex CLI · effort high · eight hard validated tasks, effort ladder | 16 | none recorded | Unclear | More or fewer tokens is not better or worse by itself; this row describes behaviour, not a winner. | Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks |
| List-price cost per strict pass by effort (calculation)Calculation | $0.013Codex CLI · effort low · eight hard validated tasks, effort ladder | $0.015Codex CLI · effort high · eight hard validated tasks, effort ladder | 16 | none recorded | Unclear | No interval or range was recorded for either side, so the gap ($0.013 vs $0.015) is not tested against run-to-run variation. | Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks |
| CLI vs API: time for a one-line answer (Total time) | 4.18 sCodex CLI · effort low · fixed exact reply, 5 runs | 4.19 sCodex CLI · effort high · fixed exact reply, 5 runs | 5 | range: 3.9 s–4.5 s vs 3.8 s–4.7 s | Unclear | The run ranges (fastest to slowest) overlap (GPT-6.1 Sol (Codex CLI) at low effort 3.86 s to 4.53 s; GPT-6.1 Sol (Codex CLI) at high effort 3.81 s to 4.69 s); the medians alone do not show a reliable difference. A range is not a confidence interval. | Claude Code CLI vs Codex CLI vs the API: latency and tokens |
| CLI vs API: time for a one-line answer (First useful output) | 3.75 sCodex CLI · effort low · fixed exact reply, 5 runs | 3.79 sCodex CLI · effort high · fixed exact reply, 5 runs | 5 | range: 3.4 s–4.1 s vs 3.4 s–4.3 s | Unclear | The run ranges (fastest to slowest) overlap (GPT-6.1 Sol (Codex CLI) at low effort 3.44 s to 4.10 s; GPT-6.1 Sol (Codex CLI) at high effort 3.37 s to 4.30 s); the medians alone do not show a reliable difference. A range is not a confidence interval. | Claude Code CLI vs Codex CLI vs the API: latency and tokens |
| CLI vs API: time for a small coding task (Total time) | 14.2 sCodex CLI · effort low · small coding task, 3 runs | 17.9 sCodex CLI · effort high · small coding task, 3 runs | 3 | range: 13 s–14.4 s vs 17.7 s–22.4 s | Unclear | Only 3 runs per side; the run ranges (fastest to slowest) do not overlap (GPT-6.1 Sol (Codex CLI) at low effort 13.0 s to 14.4 s; GPT-6.1 Sol (Codex CLI) at high effort 17.7 s to 22.4 s), but 3 runs cannot show a reliable difference. A range is not a confidence interval. | Claude Code CLI vs Codex CLI vs the API: latency and tokens |
| CLI vs API: time for a small coding task (First useful output) | 13.6 sCodex CLI · effort low · small coding task, 3 runs | 17.3 sCodex CLI · effort high · small coding task, 3 runs | 3 | range: 12.5 s–13.8 s vs 17.1 s–21.9 s | Unclear | Only 3 runs per side; the run ranges (fastest to slowest) do not overlap (GPT-6.1 Sol (Codex CLI) at low effort 12.5 s to 13.8 s; GPT-6.1 Sol (Codex CLI) at high effort 17.1 s to 21.9 s), but 3 runs cannot show a reliable difference. A range is not a confidence interval. | Claude Code CLI vs Codex CLI vs the API: latency and tokens |
| Hidden prompt: input tokens for the same one-line request | 19,551Codex CLI · effort low · short fixed tasks | 19,555Codex CLI · effort high · short fixed tasks | 5 | none recorded | Unclear | More or fewer tokens is not better or worse by itself; this row describes behaviour, not a winner. | Claude Code CLI vs Codex CLI vs the API: latency and tokens |
| Reasoning cost per strict pass by effort, with the total (calculation) (Reasoning cost per strict pass)Calculation | $0.0012Codex CLI · effort low | $0.0041Codex CLI · effort high | 16 | none recorded | Unclear | More or fewer usd is not better or worse by itself; this row describes behaviour, not a winner. | How much of an AI bill is thinking? Reasoning tokens by model and effort |
| Reasoning cost per strict pass by effort, with the total (calculation) (Total cost per strict pass)Calculation | $0.013Codex CLI · effort low | $0.015Codex CLI · effort high | 16 | none recorded | Unclear | More or fewer usd is not better or worse by itself; this row describes behaviour, not a winner. | How much of an AI bill is thinking? Reasoning tokens by model and effort |
Marks: 95% intervals (Wilson for rates); fastest–slowest run ranges (not intervals)n is shown per side on every rowHollow marks: list-price calculations, not runs
19 rows from 4 studies. No row separates them: 3 ties, 16 unclear.
When to pick which
Only from the rows above. A tie is not a reason to pick either side.
When to pick GPT-6.1 Sol (Codex CLI) at low effort
- Reasoning cost per strict pass by effort, with the total (calculation) (Reasoning cost per strict pass): $0.0012 vs $0.0041. A list-price calculation, not a measured difference. Calculation
When to pick GPT-6.1 Sol (Codex CLI) at high effort
No row in this data puts GPT-6.1 Sol (Codex CLI) at high effort ahead of GPT-6.1 Sol (Codex CLI) at low effort. Pick on other grounds (price, access, the tasks you run), or measure your own workload.
Side by side
The study charts, showing only GPT-6.1 Sol (Codex CLI) at each effort it ran; the two compared settings are highlighted. Open a study for every configuration.
Every interval overlaps every other: this chart does not order these rows.
| Item | Pass rate | 95% interval | n |
|---|---|---|---|
| GPT-6.1 Sol (high) · Codex CLI | 100% | 80%–100% | 15 |
| GPT-6.1 Sol (medium) · Codex CLI | 100% | 80%–100% | 15 |
| GPT-6.1 Sol (low) · Codex CLI | 100% | 72%–100% | 10 |
3 rows. All at 100%.
NotesWhiskers: 95% Wilson intervaln 10–15 per row3 of 3 at 100%: this task set cannot separate them.
Every call counts; failures and timeouts are non-passes
Whiskers are 95% Wilson intervals. The tasks are short and most configurations pass nearly all of them, so pass rate does not separate the models here.
Source: Provider head-to-head: Claude Code models vs Codex efforts
Every lane’s run range overlaps another lane’s, so the chart shows no finish order.
| Item | Total time per call | Range (lowest–highest run) | n |
|---|---|---|---|
| GPT-6.1 Sol (high) · Codex CLI | 5.6 s | 4.1 s–19.5 s | 15 |
| GPT-6.1 Sol (medium) · Codex CLI | 5.7 s | 4.1 s–25.5 s | 15 |
| GPT-6.1 Sol (low) · Codex CLI | 6.3 s | 4.7 s–10.5 s | 10 |
3 rows. Slowest GPT-6.1 Sol (low) · Codex CLI 6.3 s (range 4.7 s–10.5 s, n 10). Fastest GPT-6.1 Sol (high) · Codex CLI 5.6 s (range 4.1 s–19.5 s, n 15). All run ranges overlap.
NotesLines: fastest–slowest run (not an interval)n 10–15 per row
Median per configuration; whiskers = fastest and slowest call
One host, one network, one day. Whiskers are a range, not a confidence interval.
Source: Provider head-to-head: Claude Code models vs Codex efforts
Every lane’s run range overlaps another lane’s, so the chart shows no finish order.
| Item | Time to first useful output | Range (lowest–highest run) | n |
|---|---|---|---|
| GPT-6.1 Sol (high) · Codex CLI | 5.3 s | 3.6 s–16.4 s | 15 |
| GPT-6.1 Sol (medium) · Codex CLI | 5.1 s | 3.4 s–17.8 s | 15 |
| GPT-6.1 Sol (low) · Codex CLI | 5.1 s | 4 s–8.5 s | 10 |
3 rows. Slowest GPT-6.1 Sol (high) · Codex CLI 5.3 s (range 3.6 s–16.4 s, n 15). Fastest GPT-6.1 Sol (medium) · Codex CLI 5.1 s (range 3.4 s–17.8 s, n 15). All run ranges overlap.
NotesLines: fastest–slowest run (not an interval)n 10–15 per row
Median per configuration; whiskers = fastest and slowest call
One host, one network, one day. Whiskers are a range, not a confidence interval.
Source: Provider head-to-head: Claude Code models vs Codex efforts
- Cache read
- Other input
Totals are the sum of the parts shown. Shares are calculated from the same values.
| Item | Cache read | Other input | n |
|---|---|---|---|
| GPT-6.1 Sol (high) · Codex CLI | 6,716 | 5,406 | 15 |
| GPT-6.1 Sol (medium) · Codex CLI | 5,180 | 6,943 | 15 |
| GPT-6.1 Sol (low) · Codex CLI | 8,064 | 4,059 | 10 |
3 rows, 2 series: Cache read, Other input. Cache read: highest GPT-6.1 Sol (low) · Codex CLI 8,064 (n 10). Lowest GPT-6.1 Sol (medium) · Codex CLI 5,180 (n 15). Other input: highest GPT-6.1 Sol (medium) · Codex CLI 6,943 (n 15). Lowest GPT-6.1 Sol (low) · Codex CLI 4,059 (n 10).
Notesn 10–15 per row
Mean per call, split into prompt-cache reads and other input
The prompts are a few hundred tokens; most input is the CLI’s own system prompt and tool context. Input counts include cache reads, as the vendors report them.
Source: Provider head-to-head: Claude Code models vs Codex efforts
| Item | Output tokens | n |
|---|---|---|
| GPT-6.1 Sol (high) · Codex CLI | 42 | 15 |
| GPT-6.1 Sol (medium) · Codex CLI | 42 | 15 |
| GPT-6.1 Sol (low) · Codex CLI | 42 | 10 |
3 rows. All at 42.
Notesn 10–15 per row
Median per configuration; reasoning tokens where the CLI reports them
Vendors count reasoning differently; compare within a vendor first. A median of 0 can mean the CLI did not report reasoning for most calls.
Source: Provider head-to-head: Claude Code models vs Codex efforts
Every interval overlaps every other: this chart does not order these rows.
| Item | Cost per call | Range (lowest–highest run) | n |
|---|---|---|---|
| GPT-6.1 Sol (high) · Codex CLI | $0.01 | $0.0066–$0.028 | 15 |
| GPT-6.1 Sol (medium) · Codex CLI | $0.01 | $0.0054–$0.027 | 15 |
| GPT-6.1 Sol (low) · Codex CLI | $0.0077 | $0.0074–$0.026 | 10 |
List-price calculation, not a run. 3 rows. Highest GPT-6.1 Sol (high) · Codex CLI $0.01 (range $0.0066–$0.028, n 15). Lowest GPT-6.1 Sol (low) · Codex CLI $0.0077 (range $0.0074–$0.026, n 10). All run ranges overlap.
NotesLines: lowest–highest run (not an interval)n 10–15 per row
Reported tokens × list price; the calls ran on subscriptions
Calculation, not a bill: the calls ran on flat subscriptions. Whiskers = cheapest and most expensive call.
Sources: Provider head-to-head: Claude Code models vs Codex efforts, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices
Hover or focus a bar for its ratio to GPT-6.1 Sol (low) (the lowest value): a ratio of list-price calculations, not a measurement.
| Item | Cost per pass | n |
|---|---|---|
| GPT-6.1 Sol (low) · Codex CLI | $0.01 | 10 |
| GPT-6.1 Sol (high) · Codex CLI | $0.013 | 15 |
| GPT-6.1 Sol (medium) · Codex CLI | $0.016 | 15 |
List-price calculation, not a run. 3 rows. Highest GPT-6.1 Sol (medium) · Codex CLI $0.016 (n 15). Lowest GPT-6.1 Sol (low) · Codex CLI $0.01 (n 10).
Notesn 10–15 per row
All calls in a configuration, failures included, divided by its passes
Calculation, not a bill: reported tokens × list price; the calls ran on flat subscriptions. A failed call still costs, so a lower pass rate raises the cost per pass. Highlighted bars are on the speed/cost/quality frontier.
Sources: Provider head-to-head: Claude Code models vs Codex efforts, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices
Every interval overlaps every other: this chart does not order these rows.
| Item | Strict pass | 95% interval | n |
|---|---|---|---|
| GPT-6.1 Sol (low) · Codex CLI | 100% | 81%–100% | 16 |
| GPT-6.1 Sol (medium) · Codex CLI | 100% | 81%–100% | 16 |
| GPT-6.1 Sol (high) · Codex CLI | 100% | 81%–100% | 16 |
3 rows. All at 100%.
NotesWhiskers: 95% Wilson intervaln = 16 per row3 of 3 at 100%: this task set cannot separate them.
Every configuration: 8 tasks × 2 repetitions. Whiskers are 95% Wilson intervals
Whiskers are 95% Wilson intervals. A format miss never counts as a pass. Reference cells (efforts the hard head-to-head already ran) are reused, not rerun; Claude reference cells keep repetitions 1-2, so every cell is 8 tasks × 2 repetitions.
Source: Effort ladder: the hard task set at each effort level
Every lane’s run range overlaps another lane’s, so the chart shows no finish order.
| Item | Total time per call by effort on hard tasks | Range (lowest–highest run) | n |
|---|---|---|---|
| GPT-6.1 Sol (low) · Codex CLI | 13.6 s | 7.9 s–44.3 s | 16 |
| GPT-6.1 Sol (medium) · Codex CLI | 13.1 s | 8.5 s–61.6 s | 16 |
| GPT-6.1 Sol (high) · Codex CLI | 18.1 s | 11.7 s–92.2 s | 16 |
3 rows. Slowest GPT-6.1 Sol (high) · Codex CLI 18.1 s (range 11.7 s–92.2 s, n 16). Fastest GPT-6.1 Sol (medium) · Codex CLI 13.1 s (range 8.5 s–61.6 s, n 16). All run ranges overlap.
NotesLines: fastest–slowest run (not an interval)n = 16 per row
Median per configuration; whiskers = fastest and slowest call
Whiskers are a range (fastest and slowest call), not a confidence interval. One host, one network. Reference cells (efforts the hard head-to-head already ran) are reused, not rerun; Claude reference cells keep repetitions 1-2, so every cell is 8 tasks × 2 repetitions. The reference cells ran in a different hour.
Source: Effort ladder: the hard task set at each effort level
| Item | Output tokens | n |
|---|---|---|
| GPT-6.1 Sol (low) · Codex CLI | 284 | 16 |
| GPT-6.1 Sol (medium) · Codex CLI | 335 | 16 |
| GPT-6.1 Sol (high) · Codex CLI | 436 | 16 |
3 rows. Highest GPT-6.1 Sol (high) · Codex CLI 436 (n 16). Lowest GPT-6.1 Sol (low) · Codex CLI 284 (n 16).
Notesn = 16 per row
Median per configuration; reasoning tokens as the CLI reports them
Reasoning tokens are part of the output tokens where the CLI reports them; 0 can mean "not reported". Their content is never captured. More tokens is not better or worse by itself.
Source: Effort ladder: the hard task set at each effort level
Hover or focus a bar for its ratio to GPT-6.1 Sol (low) (the lowest value): a ratio of list-price calculations, not a measurement.
| Item | Cost per strict pass | n |
|---|---|---|
| GPT-6.1 Sol (low) · Codex CLI | $0.013 | 16 |
| GPT-6.1 Sol (medium) · Codex CLI | $0.026 | 16 |
| GPT-6.1 Sol (high) · Codex CLI | $0.015 | 16 |
List-price calculation, not a run. 3 rows. Highest GPT-6.1 Sol (medium) · Codex CLI $0.026 (n 16). Lowest GPT-6.1 Sol (low) · Codex CLI $0.013 (n 16).
Notesn = 16 per row
All calls in a configuration divided by its strict passes
Calculation, not a bill: reported tokens × list price, cache reads and writes priced as in the hard head-to-head; the calls ran on flat subscriptions. Codex input includes its own system prompt and tool schemas, so a Claude-vs-Codex gap is partly the CLI.
Sources: Effort ladder: the hard task set at each effort level, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices
- Total time
- First useful output
| Item | Total time | First useful output | Range (lowest–highest run) | n |
|---|---|---|---|---|
| Codex CLI · GPT-6.1 Sol · low | 4.2 s | 3.8 s | Total time: 3.9 s–4.5 s; First useful output: 3.4 s–4.1 s | 5 |
| Codex CLI · GPT-6.1 Sol · high | 4.2 s | 3.8 s | Total time: 3.8 s–4.7 s; First useful output: 3.4 s–4.3 s | 5 |
2 rows, 2 series: Total time, First useful output. Total time: slowest Codex CLI · GPT-6.1 Sol · high 4.2 s (range 3.8 s–4.7 s, n 5). Fastest Codex CLI · GPT-6.1 Sol · low 4.2 s (range 3.9 s–4.5 s, n 5). All run ranges overlap. First useful output: slowest Codex CLI · GPT-6.1 Sol · high 3.8 s (range 3.4 s–4.3 s, n 5). Fastest Codex CLI · GPT-6.1 Sol · low 3.8 s (range 3.4 s–4.1 s, n 5). All run ranges overlap.
NotesLines: fastest–slowest run (not an interval)n = 5 per row
Matched cohort, fixed exact reply, 5 runs per configuration
Dot = median; whiskers = fastest and slowest run (a range, not a confidence interval). Every run in these cohorts passed its validator.
Source: Provider explorer receipts: CLI vs API
- Total time
- First useful output
| Item | Total time | First useful output | Range (lowest–highest run) | n |
|---|---|---|---|---|
| Codex CLI · GPT-6.1 Sol · low | 14.2 s | 13.6 s | Total time: 13 s–14.4 s; First useful output: 12.5 s–13.8 s | 3 |
| Codex CLI · GPT-6.1 Sol · high | 17.9 s | 17.3 s | Total time: 17.7 s–22.4 s; First useful output: 17.1 s–21.9 s | 3 |
2 rows, 2 series: Total time, First useful output. Total time: slowest Codex CLI · GPT-6.1 Sol · high 17.9 s (range 17.7 s–22.4 s, n 3). Fastest Codex CLI · GPT-6.1 Sol · low 14.2 s (range 13 s–14.4 s, n 3). Not all run ranges overlap. First useful output: slowest Codex CLI · GPT-6.1 Sol · high 17.3 s (range 17.1 s–21.9 s, n 3). Fastest Codex CLI · GPT-6.1 Sol · low 13.6 s (range 12.5 s–13.8 s, n 3). Not all run ranges overlap.
NotesLines: fastest–slowest run (not an interval)n = 3 per row
Matched cohort, small coding task, 3 runs per configuration
Dot = median; whiskers = fastest and slowest run (a range, not a confidence interval). Every run in these cohorts passed its validator.
Source: Provider explorer receipts: CLI vs API
| Item | Input tokens | n |
|---|---|---|
| Codex CLI · GPT-6.1 Sol · low | 19,551 | 5 |
| Codex CLI · GPT-6.1 Sol · high | 19,555 | 5 |
2 rows. Highest Codex CLI · GPT-6.1 Sol · high 19,555 (n 5). Lowest Codex CLI · GPT-6.1 Sol · low 19,551 (n 5).
Notesn = 5 per row
Reported input tokens, matched cohort
The CLI wraps every request in its own system prompt and tool context; the bare API sends only the request. Part of the CLI input is served from cache.
Source: Provider explorer receipts: CLI vs API
- Reasoning cost per strict pass
- Total cost per strict pass (square)
Gap labels, Total cost per strict pass vs Reasoning cost per strict pass: Total cost per strict pass is x% higher (+) or lower (−) than Reasoning cost per strict pass, calculated from the two values shown (the change counted from Reasoning cost per strict pass’s value).
| Item | Reasoning cost per strict pass | Total cost per strict pass | n |
|---|---|---|---|
| GPT-6.1 Sol (low) · Codex CLI | $0.0012 | $0.013 | 16 |
| GPT-6.1 Sol (medium) · Codex CLI | $0.0023 | $0.026 | 16 |
| GPT-6.1 Sol (high) · Codex CLI | $0.0041 | $0.015 | 16 |
List-price calculation, not a run. 3 rows, 2 series: Reasoning cost per strict pass, Total cost per strict pass. Reasoning cost per strict pass: highest GPT-6.1 Sol (high) · Codex CLI $0.0041 (n 16). Lowest GPT-6.1 Sol (low) · Codex CLI $0.0012 (n 16). Total cost per strict pass: highest GPT-6.1 Sol (medium) · Codex CLI $0.026 (n 16). Lowest GPT-6.1 Sol (low) · Codex CLI $0.013 (n 16).
Notesn = 16 per row
List price ÷ strict passes; every cell is 8 tasks × 2 repetitions
Calculation, not a bill: reported tokens × list price, divided by the cell's strict passes; the calls ran on flat subscriptions. The effort-ladder cells: new calls plus reference cells reused from the hard head-to-head (Claude repetitions 1-2 only). "Default" means the effort flag was not passed. The total is the same value as the effort-ladder cost-per-pass chart. Effort levels are not the same scale across vendors, and the reference cells ran in a different batch and hour.
Sources: Reasoning token bill (calculation), Provider head-to-head, hard set: eight hard tasks with strict validators, Effort ladder: the hard task set at each effort level, Provider head-to-head: Claude Code models vs Codex efforts, Anthropic list prices (Claude models), OpenAI list prices
How a row is called
- AheadThe 95% intervals do not overlap, or the run ranges or p50–p95 bands do not overlap with at least 5 runs per side.
- TieThe values match, or both sit at the same ceiling.
- UnclearThe intervals or ranges overlap, too few runs were recorded, no interval was recorded, or more is not better (token counts are never a win).
- CalculationDerived from list prices and recorded counts. Not a bill and not a run.
The dataset compiler makes every call; this page only draws it. No composite score, no rank.
Questions
- Is GPT-6.1 Sol (Codex CLI) better at low effort or high effort?
- GPT-6.1 Sol (Codex CLI) at low effort and GPT-6.1 Sol (Codex CLI) at high effort share 14 measured metrics and 5 list-price calculations from 4 studies. Only the effort setting differs between the two sides of a row; the route and the task set are the same. No row separates them: every interval or run range overlaps, too few runs were recorded, no interval was recorded, or more is not better for that metric. The rows are 3 ties and 16 unclear; each row says why. Calculation rows are derived from list prices and recorded counts; they are not bills or runs. Some rows rest on small samples (n = 3 at the smallest).
- How were the two effort settings measured?
- They share 14 measured metrics and 5 list-price calculations from 4 public studies: Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head; Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks; Claude Code CLI vs Codex CLI vs the API: latency and tokens; How much of an AI bill is thinking? Reasoning tokens by model and effort. Only the effort setting differs between the two sides of a row.
- How does pass rate on five validated tasks change from low effort to high effort?
- GPT-6.1 Sol (Codex CLI) at low effort: 100% (10/10) (n = 10, 95% interval 72% to 100%). GPT-6.1 Sol (Codex CLI) at high effort: 100% (15/15) (n = 15, 95% interval 80% to 100%). The 95% intervals overlap (GPT-6.1 Sol (Codex CLI) at low effort 72% to 100%; GPT-6.1 Sol (Codex CLI) at high effort 80% to 100%), so this sample cannot separate them.
- How does total time per call change from low effort to high effort?
- GPT-6.1 Sol (Codex CLI) at low effort: 6.26 s (n = 10, run range 4.7 s to 10.5 s). GPT-6.1 Sol (Codex CLI) at high effort: 5.60 s (n = 15, run range 4.1 s to 19.5 s). The run ranges (fastest to slowest) overlap (GPT-6.1 Sol (Codex CLI) at low effort 4.65 s to 10.5 s; GPT-6.1 Sol (Codex CLI) at high effort 4.05 s to 19.5 s); the medians alone do not show a reliable difference. A range is not a confidence interval.
- How does time to first useful output change from low effort to high effort?
- GPT-6.1 Sol (Codex CLI) at low effort: 5.14 s (n = 10, run range 4 s to 8.5 s). GPT-6.1 Sol (Codex CLI) at high effort: 5.32 s (n = 15, run range 3.6 s to 16.4 s). The run ranges (fastest to slowest) overlap (GPT-6.1 Sol (Codex CLI) at low effort 4.02 s to 8.50 s; GPT-6.1 Sol (Codex CLI) at high effort 3.64 s to 16.4 s); the medians alone do not show a reliable difference. A range is not a confidence interval.
- How does strict pass rate by effort on eight hard tasks change from low effort to high effort?
- GPT-6.1 Sol (Codex CLI) at low effort: 100% (16/16) (n = 16, 95% interval 81% to 100%). GPT-6.1 Sol (Codex CLI) at high effort: 100% (16/16) (n = 16, 95% interval 81% to 100%). The 95% intervals overlap (GPT-6.1 Sol (Codex CLI) at low effort 81% to 100%; GPT-6.1 Sol (Codex CLI) at high effort 81% to 100%), so this sample cannot separate them.
The studies behind this page
Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head
130 timed calls on five validated tasks: Claude Haiku, Sonnet, Opus and Fable against Codex GPT-6.1 Sol. Pass rate, speed, tokens, cost per pass.
Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks
176 calls on 8 hard tasks at low, medium, high and default effort: Sonnet, Opus and GPT-6.1 Sol. Pass rate with 95% intervals, time, tokens, cost per pass.
Claude Code CLI vs Codex CLI vs the API: latency and tokens
194 timed runs: how long Claude Code, Codex CLI and the OpenAI API take to answer and to fix code, and how many hidden tokens a CLI adds.
How much of an AI bill is thinking? Reasoning tokens by model and effort
Reasoning tokens in 378 recorded calls by model and effort: share of output, list-price cost per call and per pass, and time. Calculations.