The tasks behind our benchmarks.
See the checks.

8 tasks across 7 configurations. Each cell shows recorded passes of recorded attempts. Missing cells are unknown, not failures.

6 configurations passed every call, so the hard set still has a ceiling for them: a perfect 24/24 has a 95% interval of 86% to 100%; a perfect 16/16 has a 95% interval of 81% to 100%. Among them, only the latency, token and cost medians differ, and their per-call time ranges overlap.

Claude Code: n = 24 per configuration (3 repetitions per task); Codex CLI: n = 16 per configuration (2 repetitions per task). Per-task cells have only 2 to 3 calls.

Claude Code and Codex CLI rows pair a CLI with a model, and each CLI adds its own system prompt and start-up time. A Claude-vs-GPT row compares the route + model pairs, not the models alone.

The Claude and Codex batches ran on different days on the same host, one call at a time per account. Each route used its own subscription.

Strict format rules decide part of the result: a reply in a code fence fails. The lenient reading is shown next to it so the two can be told apart.

CLI timings include CLI start-up and the CLI’s own system prompt. One host and network; the counted Claude and Codex batches ran hours apart. Host load was not controlled.

Default effort means the effort flag was not passed; the CLI chose. Haiku reported a median 4,556 reasoning tokens per call, which explains much of its extra time and output.

List-price costs are calculations; the calls used a flat subscription.

These tasks also appear in effort ladder.

Source: Haiku vs Sonnet vs Opus vs Fable vs GPT-6.1 Sol on 8 hard tasks · Pass matrix on hard tasks: configuration × task (strict passes) · Validator controls run before the first model call

6 tasks across 3 configurations. Each cell shows recorded passes of recorded attempts. Missing cells are unknown, not failures.

Codex CLI ran with the tester’s global AGENTS.md: --ignore-user-config does not switch that file off. 12 of 12 Codex sessions read the tester’s notes, 10 wrote a WORKLOG.md and 9 reported a commit attempt. Its time, tool calls and lines changed include that work. Claude Code ran with setting sources off, and no Claude session did any of it.

Every agent passed every session, so the pass rates sit at the 100% ceiling. These tasks are too easy to separate the agents on quality; only time, tool use, tokens and diff size differ.

Each run pairs a CLI with a model (Claude Code with Claude models, Codex CLI with GPT-6.1 Sol), so the results cannot separate the CLI from the model.

n = 12 sessions per agent (2 per task). Time ranges are the fastest and slowest sessions, not confidence intervals; the two lanes shared one machine.

Tokens are as each CLI reports them, with different tokenizers and context handling: compare tokens inside Claude Code (Sonnet vs Opus), not across vendors. Costs are list-price calculations on subscription sessions, not invoices.

Source: Claude Code (Sonnet 5.5, Opus 5.5) vs Codex CLI on 6 hidden-test coding tasks · Every task and agent: sessions that passed (of 2) · The six tasks and their controls

5 tasks across 9 configurations. Each cell shows recorded passes of recorded attempts. Missing cells are unknown, not failures.

The tasks are short and easy; pass rate saturates. Latency and tokens carry the signal. A harder follow-up with eight tasks and strict validators: /benchmarks/hard-model-head-to-head.

Few repetitions per cell (2 or 3 per task). Medians with ranges, not intervals.

CLI timings include CLI start-up and the CLI’s own system prompt.

One host, one network, one day.

List-price costs are calculations; the calls used flat subscriptions.

The prompt cache stayed at the provider default, so cache counters differ by route and by call order.

Haiku 4.5 reported reasoning tokens on most calls under the CLI default, which explains much of its extra time and output.

Haiku and Fable are not in the platform runner catalog, so these calls used the runner’s CLI functions directly; each receipt records the model the CLI reported.

Source: Haiku vs Sonnet vs Opus vs Fable vs Codex: a timed head-to-head · Pass matrix: configuration × task

5 tasks across 8 configurations. Each cell shows recorded passes of recorded attempts. Missing cells are unknown, not failures.

One small synthetic repository and one author of the facts: the curated file is an upper bound written with knowledge of the tasks.

The hook checks the same code rules as the grader. It shows what rules written as code can do; it cannot carry a fact such as the late-fee rate.

n = 15 per condition for Sonnet and 10 for Haiku: most full-pass intervals overlap, so most differences between conditions are not clear.

Costs are the CLI's list-price estimates for subscription sessions, not invoices.

In headless mode an agent that asks a question cannot get an answer, so asking counts as a failure here. In a live session the person would answer; the cost is the round trip.

Source: Does memory help Claude Code? 8 kinds of agent memory, tested · Full passes per task (Claude Sonnet 5.5, of 3)

3 tasks across 4 slices. 1 of 12 recorded attempts had verified delivery. One attempt per cell is not a rate.

One attempt per cell: these are defect-finding runs, not rates.

Each slice changes the platform build, and the last two also remove the caps. No two slices are a matched comparison.

Costs are list-price estimates for subscription calls, not invoices.

The f0ac3a8a h3 run was verified only after its account-route label was corrected; it counts as unverified here, as declared.

Source: Coding calibration: what broke on three real pull requests · Every attempt, failures included

The first column counts public panel models that solved an instance. The second records one Agent attempt. These columns use different denominators and are not a matched comparison.

n = 33: intervals are wide. This is a defect-finding run, not a ranking.

Systems, not models: the panel is one model in a bash-only harness; Agent is a full pipeline on one model.

Panel costs are published API costs; Agent costs are notional subscription estimates, not invoices.

Campaign 1 ended up 19 django, 3 sympy, 2 sphinx and 1 xarray after replacements. Campaign 2 covers the compiled repositories.

Verified issues are public (2015 to 2023) and likely in every model’s training data; contamination is uncontrolled for all systems.

Difficulty bands come from the panel’s own results, so a system outside the panel tends to look better than the panel on hard bands and worse on easy ones (regression to the mean). Read the band chart with that selection effect in mind.

Three campaign-1 empty patches were platform holds before delivery (missing lint tools, a too-literal plan gate, an unanswered question), not wrong fixes. They count as failures here.

Source: Agent on SWE-bench Verified vs 11 public models · Every attempt

11 tasks across 3 configurations. Each cell shows recorded passes of recorded attempts. Missing cells are unknown, not failures.

The case sets and question wording were revised in fix waves against Jev answers on 2026-10-04 and 2026-10-05, so Jev has a home advantage.

Jev’s numbers are from a live run: 3 repeats of the same 82 decisions, not 246 independent samples, so its interval is taken at n = 82. Its recorded production run of 2026-10-05 scored 74 of 82.

Claude routers ran through the Claude Code CLI; the CLI adds startup time and tool-schema tokens a direct API call would not. Haiku 4.5 ran with the CLI default extended thinking; Sonnet 5.5 at effort low, as production asks.

One sample per decision; production asks a second sample when confidence is low. Confidence-gated coverage is therefore not compared.

82 cases in four small hand-labelled sets: intervals are wide.

Clef / Clef-Flash (local): not measured (No local Clef server was running and installing a 6-20 GB model was out of scope for this run).

Economics are calculations: Same tokens, same cache-read share and same number of calls on every model; a different model would take a different path and number of turns.

Jev ran as a direct HTTPS call; the Claude routers ran through the Claude Code CLI. These are different routes, so speed and cost compare what a caller pays per decision, not one model against the other.

Jev’s latency is one 35-second window from one Mac over a home network. The API reports no server time. A caller near the API would see less.

These tasks also appear in routing overhead.

Source: Jev vs Claude as a router: accuracy and cost · Per-question accuracy by decision type (Jev: its recorded production run, one pass)

194 of 194 recorded runs passed. These counts sum different configurations; they do not compare them.

Small samples: 3 to 5 runs per configuration. Medians with ranges, not intervals.

The scheduler comparison pairs Claude Code with Sonnet 5.5 against GPT-6.1 Sol on Codex and the API; the routes and the models differ together.

The Claude CLI receipts record only the uncached remainder of the input (2 tokens), so the Claude input column is not comparable.

All runs are from one host and one network on 2026-10-03. Vendor latency changes over the day.

API costs in the raw file are list-price estimates; CLI runs are subscription calls with no per-call price.

Source: Claude Code CLI vs Codex CLI vs the API: latency and tokens · Every evaluated configuration