Does thinking pay for Claude Haiku 4.5? Thinking on vs off

Key numbers

The charts

Hover or focus a row for its exact value, interval and sample. Each chart has a Table view.

Tables

Method

1. Routing uses the platform’s labelled decision suites: Failure class (failure-class-2026-10-05.1, 18 cases); Message intent (frontdoor-intent-2026-10-04.2, 20 cases); Is it a rule? (memory-is-rule-2026-10-04.2, 12 cases); Context shape (context-shape-2026-10-05.1, 32 cases).
2. Production rules select the cases and leave the scored keys open. This gives 82 decisions and 194 scored questions.
3. Each decision uses one Claude Code CLI call. Calls use JSON output, tools off, no MCP and no saved session. They use the production system text, prompt and JSON schema. Calls run one at a time; temperature cannot be set.
4. The decision-eval runner scores each reply. Exact means every scored question is acceptable. Per-question accuracy counts each scored question. Errors and unanswered questions count as wrong.
5. Thinking off sets MAX_THINKING_TOKENS=0 on the claude process. Two uncounted probes check the setting, one per route. The probes and all new counted calls reported zero thinking tokens.
6. Thinking off uses new calls. Thinking on uses the CLI default in the recorded routing run. Sonnet 5.5 uses low effort in that recorded run. Both reference arms are reused; they were not rerun.
7. The exact two-sided McNemar test uses case-level pairs that differ (binomial, p = 0.5). Rate intervals are 95% Wilson intervals.
8. Timing uses wall time and the CLI’s reported API time. The same code computes nearest-rank p50 and p95 from each arm’s call log. Cost uses reported input, cache and output tokens times list price (a calculation).
9. Hard tasks use the 8 unchanged tasks and sandboxed deterministic validators from the hard head-to-head.
10. Controls ran before the first new model call, but before protocol creation. All 8 reference answers passed (8 tested). All 26 planted wrong answers failed. All 8 wrapped references were flagged as format misses.
11. Strict pass means the whole reply passes. A format miss means a lenient extractor finds a passing answer; it never counts as a strict pass. Errors and incomplete attempts count as failures.
12. Thinking off used 24 calls: 8 tasks, three repetitions each. Calls ran one at a time, with no effort flag, tools off, no MCP and one turn. The timeout was 300 s; the output-token cap was 16,000. Thinking on reuses 24 recorded default Haiku receipts.
13. Cost per strict pass divides all call costs in a cell by its strict passes. Each call cost uses list price times reported tokens (a calculation).
14. Recorded order: controls at 2026-10-06T21:27:56.463Z; protocol creation at 2026-10-06T21:29:01.607477Z; then the two probes. The first new counted call started at 2026-10-06T21:29:57.270Z. Reused reference calls predate both the controls and this protocol.
15. Declared caps: 106 new counted calls and 112 calls including probes. Observed: 106 counted calls plus 2 probes; the total stayed within both call caps.
16. Attempts: 106 counted calls, 2 uncounted probes. Nothing was trimmed or retried, and no run hit a usage limit. Every attempt is in the published extract.

Caveats

• The thinking-on and Sonnet arms are the recorded routing run of 2026-10-05, reused. The thinking-off arm ran on a different day and on a different Claude subscription account, so this is a confounded comparison. Day, account, CLI version and input-token differences can affect the results. The data cannot isolate the effect of thinking.
• 82 decisions in four small hand-labelled sets: intervals are wide, and a paired test with few discordant cases has little power. The case sets and question wording were tuned in fix waves against Jev answers (2026-10-04 to 2026-10-05).
• Thinking off here means MAX_THINKING_TOKENS=0 on the Claude Code process, checked by the CLI’s own thinking-token counters. It says nothing about the API’s thinking settings or about other models.
• Claude Code adds its own start-up time and tool-schema tokens to every call; a direct API router would skip them. CLI timings include that overhead.
• List-price costs are calculations from reported tokens; the calls used a flat subscription. Sonnet cache writes use the one-hour rate ($4 per million tokens), as recorded in its receipts. The calculation matches the CLI-reported cost.
• The Haiku routing arms have the same prompt character count for every case, but thinking-on calls report about 297 more input tokens per call (1,831 against 1,534, a calculation from the two means). The cause is unknown, and the CLI version of the thinking-on run is not recorded. At Haiku’s list price that is about $0.30 of the $5.56 cost gap per 1,000 decisions (a calculation).
• Hard tasks: 24 calls per cell (3 per task); a 4/24 result has a 95% interval of 7% to 36%. The thinking-on cell is the recorded hard head-to-head receipts (batch of 2026-10-06, effort flag not passed), reused; the thinking-off cell ran later, on another account.
• These are eight distinct hard tasks, repeated three times each. Repeats on the same task are related. Call-level Wilson intervals are descriptive; they do not measure uncertainty across unseen tasks.
• Strict format rules decide part of the hard result: a reply in a code fence fails strictly. The lenient reading is shown next to it so the two can be told apart.
• Cost per strict pass divides a cell’s cost by its strict passes (4 and 11). It has no interval, and the strict pass-rate intervals overlap, so the order of the two costs per pass is not settled.
• Both arms hit a strict-pass ceiling on Fix an interval-merge function (off-by-one and edge cases): 100% (3/3; 95% interval 44% to 100%) in each arm. Repeats on these tasks cannot establish equal accuracy on harder variants.
• Controls and reused calls predate protocol creation. The file predates the new probes and counted calls, but later amendments changed it. No frozen initial copy verifies the original wording.
• The run folder has no saved pre-batch usage-gate readings. Later corroboration does not prove that each pre-batch gate ran.
• The runs used a shared Mac. Host load was not recorded; the run does not prove isolation. Timing gaps do not establish a cause.
• Routing has a ceiling on these subsets: Message intent, 100% (20/20; 95% interval 84% to 100%) in every arm; Is it a rule?, 100% (12/12; 95% interval 76% to 100%) in every arm. These results cannot establish equal accuracy on harder cases.

Sources

• Haiku thinking on vs off

  Our recorded runs · October 6, 2026

  Receipts of the Haiku thinking study: Claude Haiku 4.5 through Claude Code with thinking off (MAX_THINKING_TOKENS=0) against the recorded thinking-on arms. Routing arms carry per-arm totals computed from each arm’s per-call log and per-case results; hard-task receipts are every attempt of the thinking-off run. Every attempt is kept, failures included. The thinking-on hard-task receipts live in the hard head-to-head extract.

  Raw data: haiku-thinking/receipts.json

• Routing runs: Jev router vs LLM routing

  Our recorded runs · October 5, 2026

  Routing decisions recorded per case and arm.

  Raw data: routing/receipts.json

• Repricing calculation

  Calculation · October 5, 2026

  Recorded token counts multiplied by the list prices in the price-list sources above. A calculation, not a run: a different model would have used a different number of tokens and reached different outcomes.

• Anthropic list prices (Claude models)

  Vendor price list · September 21, 2026

  Prices as listed by the vendor on 2026-09-21 and recorded in the product price table. Cache reads at the listed rate, one-hour cache writes at twice the input price.

• Provider head-to-head, hard set: eight hard tasks with strict validators

  Our recorded runs · October 6, 2026

  Eight hard tasks with sandboxed deterministic validators and pre-inference controls, declared protocol, every attempt kept. Format misses are recorded apart from wrong answers.

  Raw data: provider-h2h-hard/receipts.json

Download the data

The study as JSON (with its sources), every chart point as CSV, or the whole public dataset. Free to reuse under CC BY 4.0: credit Agent and link the study.

Cite as: Agent public benchmarks, “Does thinking pay for Claude Haiku 4.5? Thinking on vs off”, updated October 6, 2026, https://agent.sasid.ai/benchmarks/haiku-thinking-on-off.

Models and comparisons in this study

• Claude Sonnet 5.5
• Claude Haiku 4.5
• Claude Code

• Claude Haiku 4.5 vs Claude Sonnet 5.5
• All comparisons

Write-ups on this study