9 measured metrics · 5 calculated · 3 studies
GPT-6.1 Sol (Codex CLI)lowvsmediumeffort
No row separates them: 3 ties, 11 unclear.
The verdict
GPT-6.1 Sol (Codex CLI) at low effort and GPT-6.1 Sol (Codex CLI) at medium effort share 9 measured metrics and 5 list-price calculations from 3 studies. Only the effort setting differs between the two sides of a row; the route and the task set are the same. No row separates them: every interval or run range overlaps, too few runs were recorded, no interval was recorded, or more is not better for that metric. The rows are 3 ties and 11 unclear; each row says why. Calculation rows are derived from list prices and recorded counts; they are not bills or runs. Some rows rest on small samples (n = 10 at the smallest).
Headline metrics
How far apart the two sides are on the headline metrics. Length is the ratio of the two values; it is not a winner.
Bar length is the ratio of the two values on a log scale, pointing to the larger one. Larger is not better for time, tokens or cost. A bar has a side’s color only when that side is ahead in the data; gray means the data does not separate them.
| Metric | GPT-6.1 Sol (Codex CLI) at low effort | GPT-6.1 Sol (Codex CLI) at medium effort | n | Interval or range | Outcome | Basis | Study |
|---|---|---|---|---|---|---|---|
| Pass rate on five validated tasks | 100% (10/10)Codex CLI · effort low · five short validated tasks | 100% (15/15)Codex CLI · effort medium · five short validated tasks | 10 / 15 | 95% CI: 72%–100% vs 80%–100% | Tie | The 95% intervals overlap (GPT-6.1 Sol (Codex CLI) at low effort 72% to 100%; GPT-6.1 Sol (Codex CLI) at medium effort 80% to 100%), so this sample cannot separate them. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Total time per call | 6.26 sCodex CLI · effort low · five short validated tasks | 5.65 sCodex CLI · effort medium · five short validated tasks | 10 / 15 | range: 4.7 s–10.5 s vs 4.1 s–25.5 s | Unclear | The run ranges (fastest to slowest) overlap (GPT-6.1 Sol (Codex CLI) at low effort 4.65 s to 10.5 s; GPT-6.1 Sol (Codex CLI) at medium effort 4.10 s to 25.5 s); the medians alone do not show a reliable difference. A range is not a confidence interval. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Time to first useful output | 5.14 sCodex CLI · effort low · five short validated tasks | 5.05 sCodex CLI · effort medium · five short validated tasks | 10 / 15 | range: 4 s–8.5 s vs 3.4 s–17.8 s | Unclear | The run ranges (fastest to slowest) overlap (GPT-6.1 Sol (Codex CLI) at low effort 4.02 s to 8.50 s; GPT-6.1 Sol (Codex CLI) at medium effort 3.36 s to 17.8 s); the medians alone do not show a reliable difference. A range is not a confidence interval. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Input tokens per call: what the CLI sends (Cache read) | 8,064Codex CLI · effort low · five short validated tasks | 5,180Codex CLI · effort medium · five short validated tasks | 10 / 15 | none recorded | Unclear | More or fewer tokens is not better or worse by itself; this row describes behaviour, not a winner. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Output tokens per call (Output tokens) | 42Codex CLI · effort low · five short validated tasks | 42Codex CLI · effort medium · five short validated tasks | 10 / 15 | none recorded | Tie | Same value. More or fewer is not better by itself for this metric. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| List-price cost per passing answer (calculation)Calculation | $0.010Codex CLI · effort low · five short validated tasks | $0.016Codex CLI · effort medium · five short validated tasks | 10 / 15 | none recorded | Unclear | No interval or range was recorded for either side, so the gap ($0.010 vs $0.016) is not tested against run-to-run variation. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
Marks: 95% intervals (Wilson for rates); fastest–slowest run ranges (not intervals)n is shown per side on every rowHollow marks: list-price calculations, not runs
6 headline metrics as the ratio of the two values. None of them separates the sides in the data. Widest ratio: List-price cost per passing answer (calculation), 1.6x (Medium effort larger).
Watch it build
A live story drawn from the same dataset in your browser. It shows the same numbers, notes and interval labels as this page.
Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on eight hard tasks
All 11 configurations passed 16/16 strictly (95% interval 81%–100%): more effort bought no extra passes on this set (a ceiling). Median output tokens rose with effort; time ranges overlap. Costs are list-price calculations.
Transcript
- Effort ladder · 176 calls · 11 configurations. Does more effort buy quality? Low, medium and high effort on eight hard tasks. Sonnet 5.5 and Opus 5.5 via Claude Code, GPT-6.1 Sol via Codex CLI.
- 176 of 176 calls passed strictly, at every effort level. No format misses and no wrong answers. Calls that passed strictly, every effort: 100% (176/176) (n = 176, 95% CI 98–100%). Model and effort configurations: 11 (6 new, 5 reference) (n = 176). Format misses and wrong answers: 0 · 0 (n = 176, format · wrong). Caveat: Reference cells ran in a different batch and hour than the new cells (provider load can differ), on the same host, CLI versions, cases and flags.
- All 11 configurations passed 16/16. A 16/16 has a 95% interval of 81%–100%: the set has a ceiling, so no effort level wins. Chart: Strict pass rate by effort · 95% Wilson intervals (n = 16 each). Caveat: Every cell passed every call, so the set has a ceiling: a perfect 16/16 has a 95% interval of 81% to 100%. This study cannot show that effort does not matter on harder work; it shows that these 8 tasks do not need more than low effort.
- Median time: Sonnet 5.5 5.8 s at low, 8.8 s at high; Opus 5.5 7.5 s at low, 10.1 s at high; GPT-6.1 Sol 13.6 s at low, 18.1 s at high. Within each model the ranges overlap: no effort is ranked faster. Chart: Median total time per call · whiskers: fastest to slowest call (n = 16 each). Caveat: Each cell has only 2 calls per task. Medians of 16 calls move with a few slow calls; the ranges are wide and overlap.
- More effort writes more: Sonnet 5.5 used a median 667 / 770 / 1,192 output tokens at low / medium / high. Chart: Median output and reasoning tokens per call (n = 16 each). Caveat: Claude Code and Codex CLI rows pair a CLI with a model; each CLI adds its own system prompt and start-up time. Effort levels are not the same scale across vendors.
- List-price cost per strict pass, low → high effort, a calculation: Sonnet $0.0122 → $0.0167, Opus $0.0212 → $0.0337, GPT-6.1 Sol $0.0128 → $0.0151. Chart: List-price cost per strict pass · calculation (n = 16 each). Calculation, not a run. Caveat: List-price costs are calculations; the calls used flat subscriptions.
- 16 effort pairs, 172 comparison rows: no effort level wins any row. The rest are ties or unclear. Effort pairs compared (same model, same route): 16. Comparison rows across those pairs: 172. Rows where one effort level wins: 0 (of 172). Caveat: Every cell passed every call, so the set has a ceiling: a perfect 16/16 has a 95% interval of 81% to 100%. This study cannot show that effort does not matter on harder work; it shows that these 8 tasks do not need more than low effort.
- Start at low effort and measure. Every call, interval and cost online.
Metric by metric
Both values of a row come from the same chart in the same study. Each row has its own axis. The shaded band is where the two intervals or ranges overlap: a side is ahead only when they do not.
- Low effort
- Medium effort
- 95% interval
- fastest–slowest run (not an interval)
- where the two overlap
- hollow: list-price calculation
Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head
- Pass rate on five validated tasks100% (10/10)n 10100% (15/15)n 15TiePass rate on five validated tasks: Low effort 100% (10/10) (n 10, 95% interval 72%–100%); Medium effort 100% (15/15) (n 15, 95% interval 80%–100%). Tie.
- Total time per call6.26 sn 105.65 sn 15UnclearTotal time per call: Low effort 6.26 s (n 10, run range 4.7 s–10.5 s); Medium effort 5.65 s (n 15, run range 4.1 s–25.5 s). Unclear.
- Time to first useful output5.14 sn 105.05 sn 15UnclearTime to first useful output: Low effort 5.14 s (n 10, run range 4 s–8.5 s); Medium effort 5.05 s (n 15, run range 3.4 s–17.8 s). Unclear.
- Input tokens per call: what the CLI sends (Cache read)8,064n 105,180n 15UnclearInput tokens per call: what the CLI sends (Cache read): Low effort 8,064 (n 10); Medium effort 5,180 (n 15). Unclear.
- Input tokens per call: what the CLI sends (Other input)4,059n 106,943n 15UnclearInput tokens per call: what the CLI sends (Other input): Low effort 4,059 (n 10); Medium effort 6,943 (n 15). Unclear.
- Output tokens per call (Output tokens)42n 1042n 15TieOutput tokens per call (Output tokens): Low effort 42 (n 10); Medium effort 42 (n 15). Tie.
- List-price cost per call (calculation)Calculation$0.0077n 10$0.010n 15UnclearList-price cost per call (calculation), calculation: Low effort $0.0077 (n 10, run range $0.0074–$0.026); Medium effort $0.010 (n 15, run range $0.0054–$0.027). Unclear.
- List-price cost per passing answer (calculation)Calculation$0.010n 10$0.016n 15UnclearList-price cost per passing answer (calculation), calculation: Low effort $0.010 (n 10); Medium effort $0.016 (n 15). Unclear.
Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks
- Strict pass rate by effort on eight hard tasks100% (16/16)n 16100% (16/16)n 16TieStrict pass rate by effort on eight hard tasks: Low effort 100% (16/16) (n 16, 95% interval 81%–100%); Medium effort 100% (16/16) (n 16, 95% interval 81%–100%). Tie.
- Total time per call by effort on hard tasks13.6 sn 1613.1 sn 16UnclearTotal time per call by effort on hard tasks: Low effort 13.6 s (n 16, run range 7.9 s–44.3 s); Medium effort 13.1 s (n 16, run range 8.5 s–61.6 s). Unclear.
- Output tokens per call by effort on hard tasks (Output tokens)284n 16335n 16UnclearOutput tokens per call by effort on hard tasks (Output tokens): Low effort 284 (n 16); Medium effort 335 (n 16). Unclear.
- List-price cost per strict pass by effort (calculation)Calculation$0.013n 16$0.026n 16UnclearList-price cost per strict pass by effort (calculation), calculation: Low effort $0.013 (n 16); Medium effort $0.026 (n 16). Unclear.
How much of an AI bill is thinking? Reasoning tokens by model and effort
- Reasoning cost per strict pass by effort, with the total (calculation) (Reasoning cost per strict pass)Calculation$0.0012n 16$0.0023n 16UnclearReasoning cost per strict pass by effort, with the total (calculation) (Reasoning cost per strict pass), calculation: Low effort $0.0012 (n 16); Medium effort $0.0023 (n 16). Unclear.
- Reasoning cost per strict pass by effort, with the total (calculation) (Total cost per strict pass)Calculation$0.013n 16$0.026n 16UnclearReasoning cost per strict pass by effort, with the total (calculation) (Total cost per strict pass), calculation: Low effort $0.013 (n 16); Medium effort $0.026 (n 16). Unclear.
| Metric | GPT-6.1 Sol (Codex CLI) at low effort | GPT-6.1 Sol (Codex CLI) at medium effort | n | Interval or range | Outcome | Basis | Study |
|---|---|---|---|---|---|---|---|
| Pass rate on five validated tasks | 100% (10/10)Codex CLI · effort low · five short validated tasks | 100% (15/15)Codex CLI · effort medium · five short validated tasks | 10 / 15 | 95% CI: 72%–100% vs 80%–100% | Tie | The 95% intervals overlap (GPT-6.1 Sol (Codex CLI) at low effort 72% to 100%; GPT-6.1 Sol (Codex CLI) at medium effort 80% to 100%), so this sample cannot separate them. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Total time per call | 6.26 sCodex CLI · effort low · five short validated tasks | 5.65 sCodex CLI · effort medium · five short validated tasks | 10 / 15 | range: 4.7 s–10.5 s vs 4.1 s–25.5 s | Unclear | The run ranges (fastest to slowest) overlap (GPT-6.1 Sol (Codex CLI) at low effort 4.65 s to 10.5 s; GPT-6.1 Sol (Codex CLI) at medium effort 4.10 s to 25.5 s); the medians alone do not show a reliable difference. A range is not a confidence interval. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Time to first useful output | 5.14 sCodex CLI · effort low · five short validated tasks | 5.05 sCodex CLI · effort medium · five short validated tasks | 10 / 15 | range: 4 s–8.5 s vs 3.4 s–17.8 s | Unclear | The run ranges (fastest to slowest) overlap (GPT-6.1 Sol (Codex CLI) at low effort 4.02 s to 8.50 s; GPT-6.1 Sol (Codex CLI) at medium effort 3.36 s to 17.8 s); the medians alone do not show a reliable difference. A range is not a confidence interval. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Input tokens per call: what the CLI sends (Cache read) | 8,064Codex CLI · effort low · five short validated tasks | 5,180Codex CLI · effort medium · five short validated tasks | 10 / 15 | none recorded | Unclear | More or fewer tokens is not better or worse by itself; this row describes behaviour, not a winner. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Input tokens per call: what the CLI sends (Other input) | 4,059Codex CLI · effort low · five short validated tasks | 6,943Codex CLI · effort medium · five short validated tasks | 10 / 15 | none recorded | Unclear | More or fewer tokens is not better or worse by itself; this row describes behaviour, not a winner. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Output tokens per call (Output tokens) | 42Codex CLI · effort low · five short validated tasks | 42Codex CLI · effort medium · five short validated tasks | 10 / 15 | none recorded | Tie | Same value. More or fewer is not better by itself for this metric. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| List-price cost per call (calculation)Calculation | $0.0077Codex CLI · effort low · five short validated tasks | $0.010Codex CLI · effort medium · five short validated tasks | 10 / 15 | range: $0.0074–$0.026 vs $0.0054–$0.027 | Unclear | The run ranges (fastest to slowest) overlap (GPT-6.1 Sol (Codex CLI) at low effort $0.0074 to $0.026; GPT-6.1 Sol (Codex CLI) at medium effort $0.0054 to $0.027); the medians alone do not show a reliable difference. A range is not a confidence interval. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| List-price cost per passing answer (calculation)Calculation | $0.010Codex CLI · effort low · five short validated tasks | $0.016Codex CLI · effort medium · five short validated tasks | 10 / 15 | none recorded | Unclear | No interval or range was recorded for either side, so the gap ($0.010 vs $0.016) is not tested against run-to-run variation. | Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head |
| Strict pass rate by effort on eight hard tasks | 100% (16/16)Codex CLI · effort low · eight hard validated tasks, effort ladder | 100% (16/16)Codex CLI · effort medium · eight hard validated tasks, effort ladder | 16 | 95% CI: 81%–100% vs 81%–100% | Tie | The 95% intervals overlap (GPT-6.1 Sol (Codex CLI) at low effort 81% to 100%; GPT-6.1 Sol (Codex CLI) at medium effort 81% to 100%), so this sample cannot separate them. | Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks |
| Total time per call by effort on hard tasks | 13.6 sCodex CLI · effort low · eight hard validated tasks, effort ladder | 13.1 sCodex CLI · effort medium · eight hard validated tasks, effort ladder | 16 | range: 7.9 s–44.3 s vs 8.5 s–61.6 s | Unclear | The run ranges (fastest to slowest) overlap (GPT-6.1 Sol (Codex CLI) at low effort 7.94 s to 44.3 s; GPT-6.1 Sol (Codex CLI) at medium effort 8.54 s to 61.6 s); the medians alone do not show a reliable difference. A range is not a confidence interval. | Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks |
| Output tokens per call by effort on hard tasks (Output tokens) | 284Codex CLI · effort low · eight hard validated tasks, effort ladder | 335Codex CLI · effort medium · eight hard validated tasks, effort ladder | 16 | none recorded | Unclear | More or fewer tokens is not better or worse by itself; this row describes behaviour, not a winner. | Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks |
| List-price cost per strict pass by effort (calculation)Calculation | $0.013Codex CLI · effort low · eight hard validated tasks, effort ladder | $0.026Codex CLI · effort medium · eight hard validated tasks, effort ladder | 16 | none recorded | Unclear | No interval or range was recorded for either side, so the gap ($0.013 vs $0.026) is not tested against run-to-run variation. | Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks |
| Reasoning cost per strict pass by effort, with the total (calculation) (Reasoning cost per strict pass)Calculation | $0.0012Codex CLI · effort low | $0.0023Codex CLI · effort medium | 16 | none recorded | Unclear | More or fewer usd is not better or worse by itself; this row describes behaviour, not a winner. | How much of an AI bill is thinking? Reasoning tokens by model and effort |
| Reasoning cost per strict pass by effort, with the total (calculation) (Total cost per strict pass)Calculation | $0.013Codex CLI · effort low | $0.026Codex CLI · effort medium | 16 | none recorded | Unclear | More or fewer usd is not better or worse by itself; this row describes behaviour, not a winner. | How much of an AI bill is thinking? Reasoning tokens by model and effort |
Marks: 95% intervals (Wilson for rates); fastest–slowest run ranges (not intervals)n is shown per side on every rowHollow marks: list-price calculations, not runs
14 rows from 3 studies. No row separates them: 3 ties, 11 unclear.
When to pick which
Only from the rows above. A tie is not a reason to pick either side.
When to pick GPT-6.1 Sol (Codex CLI) at low effort
- List-price cost per passing answer (calculation): $0.010 vs $0.016. A list-price calculation, not a measured difference. Calculation
- List-price cost per strict pass by effort (calculation): $0.013 vs $0.026. A list-price calculation, not a measured difference. Calculation
- Reasoning cost per strict pass by effort, with the total (calculation) (Reasoning cost per strict pass): $0.0012 vs $0.0023. A list-price calculation, not a measured difference. Calculation
- Reasoning cost per strict pass by effort, with the total (calculation) (Total cost per strict pass): $0.013 vs $0.026. A list-price calculation, not a measured difference. Calculation
When to pick GPT-6.1 Sol (Codex CLI) at medium effort
No row in this data puts GPT-6.1 Sol (Codex CLI) at medium effort ahead of GPT-6.1 Sol (Codex CLI) at low effort. Pick on other grounds (price, access, the tasks you run), or measure your own workload.
Side by side
The study charts, showing only GPT-6.1 Sol (Codex CLI) at each effort it ran; the two compared settings are highlighted. Open a study for every configuration.
Every interval overlaps every other: this chart does not order these rows.
| Item | Pass rate | 95% interval | n |
|---|---|---|---|
| GPT-6.1 Sol (high) · Codex CLI | 100% | 80%–100% | 15 |
| GPT-6.1 Sol (medium) · Codex CLI | 100% | 80%–100% | 15 |
| GPT-6.1 Sol (low) · Codex CLI | 100% | 72%–100% | 10 |
3 rows. All at 100%.
NotesWhiskers: 95% Wilson intervaln 10–15 per row3 of 3 at 100%: this task set cannot separate them.
Every call counts; failures and timeouts are non-passes
Whiskers are 95% Wilson intervals. The tasks are short and most configurations pass nearly all of them, so pass rate does not separate the models here.
Source: Provider head-to-head: Claude Code models vs Codex efforts
Every lane’s run range overlaps another lane’s, so the chart shows no finish order.
| Item | Total time per call | Range (lowest–highest run) | n |
|---|---|---|---|
| GPT-6.1 Sol (high) · Codex CLI | 5.6 s | 4.1 s–19.5 s | 15 |
| GPT-6.1 Sol (medium) · Codex CLI | 5.7 s | 4.1 s–25.5 s | 15 |
| GPT-6.1 Sol (low) · Codex CLI | 6.3 s | 4.7 s–10.5 s | 10 |
3 rows. Slowest GPT-6.1 Sol (low) · Codex CLI 6.3 s (range 4.7 s–10.5 s, n 10). Fastest GPT-6.1 Sol (high) · Codex CLI 5.6 s (range 4.1 s–19.5 s, n 15). All run ranges overlap.
NotesLines: fastest–slowest run (not an interval)n 10–15 per row
Median per configuration; whiskers = fastest and slowest call
One host, one network, one day. Whiskers are a range, not a confidence interval.
Source: Provider head-to-head: Claude Code models vs Codex efforts
Every lane’s run range overlaps another lane’s, so the chart shows no finish order.
| Item | Time to first useful output | Range (lowest–highest run) | n |
|---|---|---|---|
| GPT-6.1 Sol (high) · Codex CLI | 5.3 s | 3.6 s–16.4 s | 15 |
| GPT-6.1 Sol (medium) · Codex CLI | 5.1 s | 3.4 s–17.8 s | 15 |
| GPT-6.1 Sol (low) · Codex CLI | 5.1 s | 4 s–8.5 s | 10 |
3 rows. Slowest GPT-6.1 Sol (high) · Codex CLI 5.3 s (range 3.6 s–16.4 s, n 15). Fastest GPT-6.1 Sol (medium) · Codex CLI 5.1 s (range 3.4 s–17.8 s, n 15). All run ranges overlap.
NotesLines: fastest–slowest run (not an interval)n 10–15 per row
Median per configuration; whiskers = fastest and slowest call
One host, one network, one day. Whiskers are a range, not a confidence interval.
Source: Provider head-to-head: Claude Code models vs Codex efforts
- Cache read
- Other input
Totals are the sum of the parts shown. Shares are calculated from the same values.
| Item | Cache read | Other input | n |
|---|---|---|---|
| GPT-6.1 Sol (high) · Codex CLI | 6,716 | 5,406 | 15 |
| GPT-6.1 Sol (medium) · Codex CLI | 5,180 | 6,943 | 15 |
| GPT-6.1 Sol (low) · Codex CLI | 8,064 | 4,059 | 10 |
3 rows, 2 series: Cache read, Other input. Cache read: highest GPT-6.1 Sol (low) · Codex CLI 8,064 (n 10). Lowest GPT-6.1 Sol (medium) · Codex CLI 5,180 (n 15). Other input: highest GPT-6.1 Sol (medium) · Codex CLI 6,943 (n 15). Lowest GPT-6.1 Sol (low) · Codex CLI 4,059 (n 10).
Notesn 10–15 per row
Mean per call, split into prompt-cache reads and other input
The prompts are a few hundred tokens; most input is the CLI’s own system prompt and tool context. Input counts include cache reads, as the vendors report them.
Source: Provider head-to-head: Claude Code models vs Codex efforts
| Item | Output tokens | n |
|---|---|---|
| GPT-6.1 Sol (high) · Codex CLI | 42 | 15 |
| GPT-6.1 Sol (medium) · Codex CLI | 42 | 15 |
| GPT-6.1 Sol (low) · Codex CLI | 42 | 10 |
3 rows. All at 42.
Notesn 10–15 per row
Median per configuration; reasoning tokens where the CLI reports them
Vendors count reasoning differently; compare within a vendor first. A median of 0 can mean the CLI did not report reasoning for most calls.
Source: Provider head-to-head: Claude Code models vs Codex efforts
Every interval overlaps every other: this chart does not order these rows.
| Item | Cost per call | Range (lowest–highest run) | n |
|---|---|---|---|
| GPT-6.1 Sol (high) · Codex CLI | $0.01 | $0.0066–$0.028 | 15 |
| GPT-6.1 Sol (medium) · Codex CLI | $0.01 | $0.0054–$0.027 | 15 |
| GPT-6.1 Sol (low) · Codex CLI | $0.0077 | $0.0074–$0.026 | 10 |
List-price calculation, not a run. 3 rows. Highest GPT-6.1 Sol (high) · Codex CLI $0.01 (range $0.0066–$0.028, n 15). Lowest GPT-6.1 Sol (low) · Codex CLI $0.0077 (range $0.0074–$0.026, n 10). All run ranges overlap.
NotesLines: lowest–highest run (not an interval)n 10–15 per row
Reported tokens × list price; the calls ran on subscriptions
Calculation, not a bill: the calls ran on flat subscriptions. Whiskers = cheapest and most expensive call.
Sources: Provider head-to-head: Claude Code models vs Codex efforts, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices
Hover or focus a bar for its ratio to GPT-6.1 Sol (low) (the lowest value): a ratio of list-price calculations, not a measurement.
| Item | Cost per pass | n |
|---|---|---|
| GPT-6.1 Sol (low) · Codex CLI | $0.01 | 10 |
| GPT-6.1 Sol (high) · Codex CLI | $0.013 | 15 |
| GPT-6.1 Sol (medium) · Codex CLI | $0.016 | 15 |
List-price calculation, not a run. 3 rows. Highest GPT-6.1 Sol (medium) · Codex CLI $0.016 (n 15). Lowest GPT-6.1 Sol (low) · Codex CLI $0.01 (n 10).
Notesn 10–15 per row
All calls in a configuration, failures included, divided by its passes
Calculation, not a bill: reported tokens × list price; the calls ran on flat subscriptions. A failed call still costs, so a lower pass rate raises the cost per pass. Highlighted bars are on the speed/cost/quality frontier.
Sources: Provider head-to-head: Claude Code models vs Codex efforts, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices
Every interval overlaps every other: this chart does not order these rows.
| Item | Strict pass | 95% interval | n |
|---|---|---|---|
| GPT-6.1 Sol (low) · Codex CLI | 100% | 81%–100% | 16 |
| GPT-6.1 Sol (medium) · Codex CLI | 100% | 81%–100% | 16 |
| GPT-6.1 Sol (high) · Codex CLI | 100% | 81%–100% | 16 |
3 rows. All at 100%.
NotesWhiskers: 95% Wilson intervaln = 16 per row3 of 3 at 100%: this task set cannot separate them.
Every configuration: 8 tasks × 2 repetitions. Whiskers are 95% Wilson intervals
Whiskers are 95% Wilson intervals. A format miss never counts as a pass. Reference cells (efforts the hard head-to-head already ran) are reused, not rerun; Claude reference cells keep repetitions 1-2, so every cell is 8 tasks × 2 repetitions.
Source: Effort ladder: the hard task set at each effort level
Every lane’s run range overlaps another lane’s, so the chart shows no finish order.
| Item | Total time per call by effort on hard tasks | Range (lowest–highest run) | n |
|---|---|---|---|
| GPT-6.1 Sol (low) · Codex CLI | 13.6 s | 7.9 s–44.3 s | 16 |
| GPT-6.1 Sol (medium) · Codex CLI | 13.1 s | 8.5 s–61.6 s | 16 |
| GPT-6.1 Sol (high) · Codex CLI | 18.1 s | 11.7 s–92.2 s | 16 |
3 rows. Slowest GPT-6.1 Sol (high) · Codex CLI 18.1 s (range 11.7 s–92.2 s, n 16). Fastest GPT-6.1 Sol (medium) · Codex CLI 13.1 s (range 8.5 s–61.6 s, n 16). All run ranges overlap.
NotesLines: fastest–slowest run (not an interval)n = 16 per row
Median per configuration; whiskers = fastest and slowest call
Whiskers are a range (fastest and slowest call), not a confidence interval. One host, one network. Reference cells (efforts the hard head-to-head already ran) are reused, not rerun; Claude reference cells keep repetitions 1-2, so every cell is 8 tasks × 2 repetitions. The reference cells ran in a different hour.
Source: Effort ladder: the hard task set at each effort level
| Item | Output tokens | n |
|---|---|---|
| GPT-6.1 Sol (low) · Codex CLI | 284 | 16 |
| GPT-6.1 Sol (medium) · Codex CLI | 335 | 16 |
| GPT-6.1 Sol (high) · Codex CLI | 436 | 16 |
3 rows. Highest GPT-6.1 Sol (high) · Codex CLI 436 (n 16). Lowest GPT-6.1 Sol (low) · Codex CLI 284 (n 16).
Notesn = 16 per row
Median per configuration; reasoning tokens as the CLI reports them
Reasoning tokens are part of the output tokens where the CLI reports them; 0 can mean "not reported". Their content is never captured. More tokens is not better or worse by itself.
Source: Effort ladder: the hard task set at each effort level
Hover or focus a bar for its ratio to GPT-6.1 Sol (low) (the lowest value): a ratio of list-price calculations, not a measurement.
| Item | Cost per strict pass | n |
|---|---|---|
| GPT-6.1 Sol (low) · Codex CLI | $0.013 | 16 |
| GPT-6.1 Sol (medium) · Codex CLI | $0.026 | 16 |
| GPT-6.1 Sol (high) · Codex CLI | $0.015 | 16 |
List-price calculation, not a run. 3 rows. Highest GPT-6.1 Sol (medium) · Codex CLI $0.026 (n 16). Lowest GPT-6.1 Sol (low) · Codex CLI $0.013 (n 16).
Notesn = 16 per row
All calls in a configuration divided by its strict passes
Calculation, not a bill: reported tokens × list price, cache reads and writes priced as in the hard head-to-head; the calls ran on flat subscriptions. Codex input includes its own system prompt and tool schemas, so a Claude-vs-Codex gap is partly the CLI.
Sources: Effort ladder: the hard task set at each effort level, Repricing calculation, Anthropic list prices (Claude models), OpenAI list prices
- Reasoning cost per strict pass
- Total cost per strict pass (square)
Gap labels, Total cost per strict pass vs Reasoning cost per strict pass: Total cost per strict pass is x% higher (+) or lower (−) than Reasoning cost per strict pass, calculated from the two values shown (the change counted from Reasoning cost per strict pass’s value).
| Item | Reasoning cost per strict pass | Total cost per strict pass | n |
|---|---|---|---|
| GPT-6.1 Sol (low) · Codex CLI | $0.0012 | $0.013 | 16 |
| GPT-6.1 Sol (medium) · Codex CLI | $0.0023 | $0.026 | 16 |
| GPT-6.1 Sol (high) · Codex CLI | $0.0041 | $0.015 | 16 |
List-price calculation, not a run. 3 rows, 2 series: Reasoning cost per strict pass, Total cost per strict pass. Reasoning cost per strict pass: highest GPT-6.1 Sol (high) · Codex CLI $0.0041 (n 16). Lowest GPT-6.1 Sol (low) · Codex CLI $0.0012 (n 16). Total cost per strict pass: highest GPT-6.1 Sol (medium) · Codex CLI $0.026 (n 16). Lowest GPT-6.1 Sol (low) · Codex CLI $0.013 (n 16).
Notesn = 16 per row
List price ÷ strict passes; every cell is 8 tasks × 2 repetitions
Calculation, not a bill: reported tokens × list price, divided by the cell's strict passes; the calls ran on flat subscriptions. The effort-ladder cells: new calls plus reference cells reused from the hard head-to-head (Claude repetitions 1-2 only). "Default" means the effort flag was not passed. The total is the same value as the effort-ladder cost-per-pass chart. Effort levels are not the same scale across vendors, and the reference cells ran in a different batch and hour.
Sources: Reasoning token bill (calculation), Provider head-to-head, hard set: eight hard tasks with strict validators, Effort ladder: the hard task set at each effort level, Provider head-to-head: Claude Code models vs Codex efforts, Anthropic list prices (Claude models), OpenAI list prices
How a row is called
- AheadThe 95% intervals do not overlap, or the run ranges or p50–p95 bands do not overlap with at least 5 runs per side.
- TieThe values match, or both sit at the same ceiling.
- UnclearThe intervals or ranges overlap, too few runs were recorded, no interval was recorded, or more is not better (token counts are never a win).
- CalculationDerived from list prices and recorded counts. Not a bill and not a run.
The dataset compiler makes every call; this page only draws it. No composite score, no rank.
Questions
- Is GPT-6.1 Sol (Codex CLI) better at low effort or medium effort?
- GPT-6.1 Sol (Codex CLI) at low effort and GPT-6.1 Sol (Codex CLI) at medium effort share 9 measured metrics and 5 list-price calculations from 3 studies. Only the effort setting differs between the two sides of a row; the route and the task set are the same. No row separates them: every interval or run range overlaps, too few runs were recorded, no interval was recorded, or more is not better for that metric. The rows are 3 ties and 11 unclear; each row says why. Calculation rows are derived from list prices and recorded counts; they are not bills or runs. Some rows rest on small samples (n = 10 at the smallest).
- How were the two effort settings measured?
- They share 9 measured metrics and 5 list-price calculations from 3 public studies: Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head; Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks; How much of an AI bill is thinking? Reasoning tokens by model and effort. Only the effort setting differs between the two sides of a row.
- How does pass rate on five validated tasks change from low effort to medium effort?
- GPT-6.1 Sol (Codex CLI) at low effort: 100% (10/10) (n = 10, 95% interval 72% to 100%). GPT-6.1 Sol (Codex CLI) at medium effort: 100% (15/15) (n = 15, 95% interval 80% to 100%). The 95% intervals overlap (GPT-6.1 Sol (Codex CLI) at low effort 72% to 100%; GPT-6.1 Sol (Codex CLI) at medium effort 80% to 100%), so this sample cannot separate them.
- How does total time per call change from low effort to medium effort?
- GPT-6.1 Sol (Codex CLI) at low effort: 6.26 s (n = 10, run range 4.7 s to 10.5 s). GPT-6.1 Sol (Codex CLI) at medium effort: 5.65 s (n = 15, run range 4.1 s to 25.5 s). The run ranges (fastest to slowest) overlap (GPT-6.1 Sol (Codex CLI) at low effort 4.65 s to 10.5 s; GPT-6.1 Sol (Codex CLI) at medium effort 4.10 s to 25.5 s); the medians alone do not show a reliable difference. A range is not a confidence interval.
- How does time to first useful output change from low effort to medium effort?
- GPT-6.1 Sol (Codex CLI) at low effort: 5.14 s (n = 10, run range 4 s to 8.5 s). GPT-6.1 Sol (Codex CLI) at medium effort: 5.05 s (n = 15, run range 3.4 s to 17.8 s). The run ranges (fastest to slowest) overlap (GPT-6.1 Sol (Codex CLI) at low effort 4.02 s to 8.50 s; GPT-6.1 Sol (Codex CLI) at medium effort 3.36 s to 17.8 s); the medians alone do not show a reliable difference. A range is not a confidence interval.
- How does strict pass rate by effort on eight hard tasks change from low effort to medium effort?
- GPT-6.1 Sol (Codex CLI) at low effort: 100% (16/16) (n = 16, 95% interval 81% to 100%). GPT-6.1 Sol (Codex CLI) at medium effort: 100% (16/16) (n = 16, 95% interval 81% to 100%). The 95% intervals overlap (GPT-6.1 Sol (Codex CLI) at low effort 81% to 100%; GPT-6.1 Sol (Codex CLI) at medium effort 81% to 100%), so this sample cannot separate them.
The studies behind this page
Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head
130 timed calls on five validated tasks: Claude Haiku, Sonnet, Opus and Fable against Codex GPT-6.1 Sol. Pass rate, speed, tokens, cost per pass.
Does more effort buy quality? Sonnet, Opus and GPT-6.1 Sol on 8 hard tasks
176 calls on 8 hard tasks at low, medium, high and default effort: Sonnet, Opus and GPT-6.1 Sol. Pass rate with 95% intervals, time, tokens, cost per pass.
How much of an AI bill is thinking? Reasoning tokens by model and effort
Reasoning tokens in 378 recorded calls by model and effort: share of output, list-price cost per call and per pass, and time. Calculations.