Where the seconds go: first text, output speed and prompt size for 6 LLMs

Key numbers

The charts

Hover or focus a row for its exact value, interval and sample. Each chart has a Table view.

Tables

Method

1. The protocol file birth precedes the first probe and counted call. Later edits have no frozen versions. Two probes checked the routes and helped calibrate ledger sizes. They stay outside all cells and charts.
2. Part A used 24 calls: 4 per model. The prompt asked for 250 numbers in words, one per line. The check compares the reply with all 250 expected lines. Models: Haiku, Sonnet, Opus, Fable, Sol (low) and Luna (low). Controls ran before the first counted call. The reference passes. All 8/8 wrapped references are format misses. All 11/11 planted wrong answers fail.
3. Part B used 36 calls: 3 per size for each model. Models: Haiku, Sonnet, Opus and Sol (low). Each call asks one exact lookup question about a synthetic stock ledger. Ledger sizes: 18 lines for 1k, 327 lines for 16k and 1,315 lines for 64k. The 1k, 16k and 64k labels are approximate targets from probe calibration calculations, using estimated Haiku prefix counts. A new seed changes each ledger. Some fixed text repeats. Cache counts do not identify cached text.
4. First text means the first non-empty text after CLI start. The time includes CLI start-up and work before the first word. The table also subtracts the CLI ready time (calculation).
5. Output speed is a calculation. Visible tokens equal output tokens minus reported reasoning tokens. When the CLI reports no reasoning count, the calculation uses zero. That does not prove the model did no reasoning. Divide visible tokens by the time from first text to call end. The plain form divides all output tokens by that time. It can overstate visible speed when reasoning comes first.
6. The harness uses the hard head-to-head machinery. It disables tools, MCP servers and saved sessions. Claude Code uses default effort; Codex CLI uses low effort. Calls run one at a time on one Mac. The timeout was 300 s. Claude Code had a 16,000-token output cap. Codex had no output-token cap. CLI versions: 2.1.286 (Claude Code) and codex-cli 0.160.0.
7. Stop rules require a stop at the first usage-limit text. The gate checks other study markers before each batch. No stored batch stopped or trimmed a cell. No counted call duplicates another. A sequence process ended at the gate. Codex part B ran later. Every counted call completed.
8. Cache-read share is a calculation: cache-read tokens divided by reported input tokens for each call. The cache table shows the median, observed range and n for each cell.

Caveats

• 4 calls per model in part A and 3 per size in part B. Medians of so few calls move with one slow call, and the ranges are not confidence intervals. The 95% Wilson intervals on the lookup rates are wide.
• First text includes CLI start-up and reasoning time. It is not the API’s time to first token; a direct API call would skip the CLI start-up (the CLI reported ready after a median 0.33 s to 0.84 s per cell).
• Claude Code ran models at their default effort; Codex CLI ran at low effort. Effort levels are not one scale, and a CLI adds its own system prompt, so a row mixes a model and its CLI.
• Output speed is a calculation from reported token counts, the length of the reply and the clock. It depends on how each CLI counts reasoning, and part A has one prompt: other text, other days or an API route can differ.
• All 36 counted ledger prompts differ. Their input counts do not compare tokenizers on the same text. The size calibration uses prefix estimates from earlier short prompts. Those calculations do not measure the prefix in each counted call.
• The tasks hit a ceiling. Part A passed 23/24. In part B, 3 of 4 models passed every lookup. See the rate stats and chart for 95% intervals. This small task set cannot rank general capability.
• One shared Mac: the gate checked other study markers before each batch. No full host-load record proves that all other work stopped.
• Prompt sizes were calibrated after two probes. The cases are synthetic and tuned to this measurement, not a sample of real workloads.
• Amendment 3 says 06:05 UTC and claims to precede Codex part B. Those calls ran at 06:03:43 to 06:04:19 UTC. The same-file amendment is retrospective; its date does not prove advance declaration.
• The controls receipt was overwritten. The surviving checks ran after the probes and before counted calls; an earlier control run cannot be audited.
• Each size used different ledger text. Prompt-size differences also include content and provider-load changes; they do not isolate a cause.
• The failed Haiku reply contained tool-shaped text and extra output. No structured tool event was recorded; it stays a strict failure and remains in timings.
• One Mac, one network, two sessions on one night. Provider load changes timings from hour to hour. The Codex CLI part B batch ran about 4 hours after the batches before it (a run script stopped while it waited for another run to finish timing), so its timings come from a later hour.

Sources

• LLM speed anatomy

  Our recorded runs · October 7, 2026

  Receipts of the speed anatomy study: one prompt that asks for 250 numbers in words (output speed, 24 calls) and a seeded synthetic ledger at three sizes with one lookup question (prompt size, 36 calls). Every attempt is kept, failures included. A new seed for every ledger call; no ledger text, prompt or model output is copied.

  Raw data: speed-anatomy/receipts.json

Download the data

The study as JSON (with its sources), every chart point as CSV, or the whole public dataset. Free to reuse under CC BY 4.0: credit Agent and link the study.

Cite as: Agent public benchmarks, “Where the seconds go: first text, output speed and prompt size for 6 LLMs”, updated October 7, 2026, https://agent.sasid.ai/benchmarks/llm-speed-anatomy.

Models and comparisons in this study

• Claude Sonnet 5.5
• Claude Opus 5.5
• Claude Haiku 4.5
• Claude Fable 5.1
• GPT-6.1 Sol (Codex CLI)
• Claude Code
• Codex CLI
• GPT-6 Luna (Codex CLI)

• Claude Sonnet 5.5 vs Claude Opus 5.5
• Claude Haiku 4.5 vs Claude Sonnet 5.5
• Claude Opus 5.5 vs Claude Fable 5.1
• Claude Sonnet 5.5 vs Claude Fable 5.1
• Claude Haiku 4.5 vs Claude Opus 5.5
• Claude Code vs Codex CLI
• Claude Sonnet 5.5 vs GPT-6.1 Sol (Codex CLI)
• Claude Opus 5.5 vs GPT-6.1 Sol (Codex CLI)
• All comparisons

Write-ups on this study