Haiku vs Sonnet vs Opus vs Fable vs GPT-6.1 Sol on 8 hard tasks

Live story

Drawn live in the page from the same data as the charts below. Play it, or download it as a video from the player.

Key numbers

The charts

Hover or focus a row for its exact value, interval and sample. Each chart has a Table view.

Tables

Method

1. Protocol declared before the first call. A follow-up to the five-task head-to-head (/benchmarks/model-head-to-head), where pass rate hit a ceiling.
2. 8 hard tasks, each with a deterministic validator that runs in a sandbox without network: Fix an interval-merge function (off-by-one and edge cases); Fix a time-zone day-length function (DST); Write a CSV parser (quoted newlines, strict errors); Predict JavaScript event-loop output order; Solve a multi-constraint room schedule; Write a strict SemVer 2.0.0 regex; Refactor to remove duplication, keep 20 tests green; Write a SQLite reporting query (fan-out, ties, boundaries).
3. Controls before inference: 8/8 reference answers pass, 26/26 plausible wrong answers fail, and 8/8 references wrapped in a fence or prose are flagged as format misses.
4. Configurations that reached a model: Sonnet, Opus, Opus (high), GPT-6.1 Sol (medium), Fable, GPT-6.1 Sol (high), Haiku. Claude Code: n = 24 per configuration (3 repetitions per task); Codex CLI: n = 16 per configuration (2 repetitions per task).
5. Strict pass: the whole reply, trimmed, passes the validator. Every prompt states the output format and says no code fence and no other text. This is stricter than the five-task study, which removed one wrapping fence.
6. Format miss: the strict check failed, but a lenient extractor (fenced block, outer JSON, first code line, one output line) finds an answer that passes the same validator. Reported apart from wrong answers, never as a pass.
7. Isolation: fresh empty working folder, tools off, no MCP servers, no session persistence, one turn, 300 s timeout per call. One call at a time per account.
8. Attempts: Claude Code: 120 attempts, 120 reached a model, 0 blocked; Codex CLI: 62 attempts, 32 reached a model, 30 blocked. Nothing was trimmed or retried, and no run hit a usage or rate limit.
9. One Codex batch was resumed after its process ended (declared in the protocol before the resume): the resume skipped every task, repetition and effort the batch file already held, so no recorded call was repeated or replaced.
10. Cost per strict pass: list price × reported tokens for every call in the configuration (cache reads and writes priced as in the five-task study), divided by its strict passes. A calculation.

Caveats

• 6 configurations passed every call, so the hard set still has a ceiling for them: a perfect 24/24 has a 95% interval of 86% to 100%; a perfect 16/16 has a 95% interval of 81% to 100%. Among them, only the latency, token and cost medians differ, and their per-call time ranges overlap.
• Claude Code: n = 24 per configuration (3 repetitions per task); Codex CLI: n = 16 per configuration (2 repetitions per task). Per-task cells have only 2 to 3 calls.
• Claude Code and Codex CLI rows pair a CLI with a model, and each CLI adds its own system prompt and start-up time. A Claude-vs-GPT row compares the route + model pairs, not the models alone.
• The Claude and Codex batches ran on different days on the same host, one call at a time per account. Each route used its own subscription.
• Strict format rules decide part of the result: a reply in a code fence fails. The lenient reading is shown next to it so the two can be told apart.
• CLI timings include CLI start-up and the CLI’s own system prompt. One host and network; the counted Claude and Codex batches ran hours apart. Host load was not controlled.
• Default effort means the effort flag was not passed; the CLI chose. Haiku reported a median 4,556 reasoning tokens per call, which explains much of its extra time and output.
• List-price costs are calculations; the calls used a flat subscription.

Sources

• Provider head-to-head, hard set: eight hard tasks with strict validators

  Our recorded runs · October 6, 2026

  Eight hard tasks with sandboxed deterministic validators and pre-inference controls, declared protocol, every attempt kept. Format misses are recorded apart from wrong answers.

  Raw data: provider-h2h-hard/receipts.json

• Repricing calculation

  Calculation · October 5, 2026

  Recorded token counts multiplied by the list prices in the price-list sources above. A calculation, not a run: a different model would have used a different number of tokens and reached different outcomes.

• Anthropic list prices (Claude models)

  Vendor price list · September 21, 2026

  Prices as listed by the vendor on 2026-09-21 and recorded in the product price table. Cache reads at the listed rate, one-hour cache writes at twice the input price.

• OpenAI list prices

  Vendor price list · October 3, 2026

  Token prices as listed by the vendor on 2026-10-03.

Download the data

The study as JSON (with its sources), every chart point as CSV, or the whole public dataset. Free to reuse under CC BY 4.0: credit Agent and link the study.

Cite as: Agent public benchmarks, “Haiku vs Sonnet vs Opus vs Fable vs GPT-6.1 Sol on 8 hard tasks”, updated October 6, 2026, https://agent.sasid.ai/benchmarks/hard-model-head-to-head.

Explainers that cite this study

Read the methods and terms in the context of these recorded results.

• Benchmark saturation: when every model scores 100%
• Cost per correct answer: the LLM price that counts failures
• How many runs does an LLM eval need? Sample size, with real intervals
• How to read AI benchmarks honestly
• The Pareto frontier: reading LLM cost against quality
• pass@k explained: pass@1, pass@k and pass^k with real runs
• Reasoning tokens, explained: the hidden output you pay for
• Strict grading and format misses: when a right answer fails
• Time to first token (TTFT), explained with CLI and API timings
• What is reasoning effort?
• Wilson confidence intervals for AI benchmarks

Models and comparisons in this study

• Claude Sonnet 5.5
• Claude Opus 5.5
• Claude Haiku 4.5
• Claude Fable 5.1
• GPT-6.1 Sol (Codex CLI)
• Claude Code
• Codex CLI

• Claude Sonnet 5.5 vs Claude Opus 5.5
• Claude Haiku 4.5 vs Claude Sonnet 5.5
• Claude Opus 5.5 vs Claude Fable 5.1
• Claude Sonnet 5.5 vs Claude Fable 5.1
• Claude Haiku 4.5 vs Claude Opus 5.5
• Claude Code vs Codex CLI
• Claude Sonnet 5.5 vs GPT-6.1 Sol (Codex CLI)
• Claude Opus 5.5 vs GPT-6.1 Sol (Codex CLI)
• All comparisons

Write-ups on this study