2 measured metrics · 4 calculated · 1 study
Deterministic routing policyvsClaude Haiku 4.5
Deterministic routing policy ahead on 1; 1 tie, 4 unclear. A side is ahead only where the intervals or ranges do not overlap.
The verdict
Deterministic routing policy and Claude Haiku 4.5 share 2 measured metrics and 4 list-price calculations from one study. Deterministic routing policy leads on 1 row: Time to make one routing decision, 1.42 µs vs 12,543 ms. On those rows the p50–p95 bands do not overlap; only a 95% interval is a confidence interval. The other rows are 1 tie and 4 unclear; each row says why. Calculation rows are derived from list prices and recorded counts; they are not bills or runs.
Headline metrics
How far apart the two sides are on the headline metrics. Length is the ratio of the two values; it is not a winner.
Bar length is the ratio of the two values on a log scale, pointing to the larger one. Larger is not better for time, tokens or cost. A bar has a side’s color only when that side is ahead in the data; gray means the data does not separate them.
| Metric | Deterministic routing policy | Claude Haiku 4.5 | n | Interval or range | Outcome | Basis | Study |
|---|---|---|---|---|---|---|---|
| Time to make one routing decision | 1.42 µsAgent · in process · routing overhead per decision | 12,543 msthinking on · via Claude Code · routing overhead per decision | 20000 / 82 | p50–p95: 1.42 µs–2.33 µs vs 12.54 s–34.48 s | Deterministic routing policy ahead | Claude Haiku 4.5’s median is above Deterministic routing policy’s 95th percentile (p50–p95 bands: Deterministic routing policy 1.42 µs to 2.33 µs; Claude Haiku 4.5 12,543 ms to 34,481 ms); not a confidence interval. | Routing overhead: deterministic policy vs LLM routers vs Jev |
| Routing calls that returned a decision | 100% (20000/20000)Agent · in process · routing overhead per decision | 100% (82/82)thinking on · via Claude Code · routing overhead per decision | 20000 / 82 | 95% CI: 100%–100% vs 96%–100% | Tie | The 95% intervals overlap (Deterministic routing policy 100% to 100%; Claude Haiku 4.5 96% to 100%), so this sample cannot separate them. | Routing overhead: deterministic policy vs LLM routers vs Jev |
| Added routing delay per task (calculation) (Every model call routed (49.5 per task))Calculation | 70.3 µsAgent · in process · calculation per task from recorded decision counts, decisions in line | 620.9 sthinking on · via Claude Code · calculation per task from recorded decision counts, decisions in line | — | none recorded | Unclear | No interval or range was recorded for either side, so the gap (70.3 µs vs 620.9 s, 8.8 million times) is not tested against run-to-run variation. | Routing overhead: deterministic policy vs LLM routers vs Jev |
| Added routing delay per task (calculation) (Only System One decisions (7 per task))Calculation | 9.9 µsAgent · in process · calculation per task from recorded decision counts, decisions in line | 87.8 sthinking on · via Claude Code · calculation per task from recorded decision counts, decisions in line | — | none recorded | Unclear | No interval or range was recorded for either side, so the gap (9.9 µs vs 87.8 s, 8.9 million times) is not tested against run-to-run variation. | Routing overhead: deterministic policy vs LLM routers vs Jev |
Marks: 95% intervals (Wilson for rates); median to p95 bands (not intervals)n is shown per side on every rowHollow marks: list-price calculations, not runs
4 headline metrics as the ratio of the two values. 1 of them separate the sides in the data. Widest ratio: Added routing delay per task (calculation) (Only System One decisions (7 per task)), 8.9 million x (Claude Haiku 4.5 larger).
Metric by metric
Both values of a row come from the same chart in the same study. Each row has its own axis. The shaded band is where the two intervals or ranges overlap: a side is ahead only when they do not.
- Deterministic routing policy
- Claude Haiku 4.5
- 95% interval
- median to p95 (not an interval)
- where the two overlap
- hollow: list-price calculation
Routing overhead: deterministic policy vs LLM routers vs Jev
- Time to make one routing decision1.42 µsn 2000012,543 msn 82Deterministic routing policy aheadTime to make one routing decision: Deterministic routing policy 1.42 µs (n 20000, median to p95 1.42 µs–2.33 µs); Claude Haiku 4.5 12,543 ms (n 82, median to p95 12.54 s–34.48 s). Deterministic routing policy ahead.
- Routing calls that returned a decision100% (20000/20000)n 20000100% (82/82)n 82TieRouting calls that returned a decision: Deterministic routing policy 100% (20000/20000) (n 20000, 95% interval 100%–100%); Claude Haiku 4.5 100% (82/82) (n 82, 95% interval 96%–100%). Tie.
- Added routing cost per 1,000 tasks (calculation) (Every model call routed (49.5 per task))Calculation$0.00$441.74UnclearAdded routing cost per 1,000 tasks (calculation) (Every model call routed (49.5 per task)), calculation: Deterministic routing policy $0.00; Claude Haiku 4.5 $441.74. Unclear.
- Added routing cost per 1,000 tasks (calculation) (Only System One decisions (7 per task))Calculation$0.00$62.47UnclearAdded routing cost per 1,000 tasks (calculation) (Only System One decisions (7 per task)), calculation: Deterministic routing policy $0.00; Claude Haiku 4.5 $62.47. Unclear.
- Added routing delay per task (calculation) (Every model call routed (49.5 per task))Calculation70.3 µs620.9 sUnclearAdded routing delay per task (calculation) (Every model call routed (49.5 per task)), calculation: Deterministic routing policy 70.3 µs; Claude Haiku 4.5 620.9 s. Unclear.
- Added routing delay per task (calculation) (Only System One decisions (7 per task))Calculation9.9 µs87.8 sUnclearAdded routing delay per task (calculation) (Only System One decisions (7 per task)), calculation: Deterministic routing policy 9.9 µs; Claude Haiku 4.5 87.8 s. Unclear.
| Metric | Deterministic routing policy | Claude Haiku 4.5 | n | Interval or range | Outcome | Basis | Study |
|---|---|---|---|---|---|---|---|
| Time to make one routing decision | 1.42 µsAgent · in process · routing overhead per decision | 12,543 msthinking on · via Claude Code · routing overhead per decision | 20000 / 82 | p50–p95: 1.42 µs–2.33 µs vs 12.54 s–34.48 s | Deterministic routing policy ahead | Claude Haiku 4.5’s median is above Deterministic routing policy’s 95th percentile (p50–p95 bands: Deterministic routing policy 1.42 µs to 2.33 µs; Claude Haiku 4.5 12,543 ms to 34,481 ms); not a confidence interval. | Routing overhead: deterministic policy vs LLM routers vs Jev |
| Routing calls that returned a decision | 100% (20000/20000)Agent · in process · routing overhead per decision | 100% (82/82)thinking on · via Claude Code · routing overhead per decision | 20000 / 82 | 95% CI: 100%–100% vs 96%–100% | Tie | The 95% intervals overlap (Deterministic routing policy 100% to 100%; Claude Haiku 4.5 96% to 100%), so this sample cannot separate them. | Routing overhead: deterministic policy vs LLM routers vs Jev |
| Added routing cost per 1,000 tasks (calculation) (Every model call routed (49.5 per task))Calculation | $0.00Agent · in process · calculation per 1,000 tasks from recorded decision counts | $441.74thinking on · via Claude Code · calculation per 1,000 tasks from recorded decision counts | — | none recorded | Unclear | No interval or range was recorded for either side, so the gap ($0.00 vs $441.74) is not tested against run-to-run variation. | Routing overhead: deterministic policy vs LLM routers vs Jev |
| Added routing cost per 1,000 tasks (calculation) (Only System One decisions (7 per task))Calculation | $0.00Agent · in process · calculation per 1,000 tasks from recorded decision counts | $62.47thinking on · via Claude Code · calculation per 1,000 tasks from recorded decision counts | — | none recorded | Unclear | No interval or range was recorded for either side, so the gap ($0.00 vs $62.47) is not tested against run-to-run variation. | Routing overhead: deterministic policy vs LLM routers vs Jev |
| Added routing delay per task (calculation) (Every model call routed (49.5 per task))Calculation | 70.3 µsAgent · in process · calculation per task from recorded decision counts, decisions in line | 620.9 sthinking on · via Claude Code · calculation per task from recorded decision counts, decisions in line | — | none recorded | Unclear | No interval or range was recorded for either side, so the gap (70.3 µs vs 620.9 s, 8.8 million times) is not tested against run-to-run variation. | Routing overhead: deterministic policy vs LLM routers vs Jev |
| Added routing delay per task (calculation) (Only System One decisions (7 per task))Calculation | 9.9 µsAgent · in process · calculation per task from recorded decision counts, decisions in line | 87.8 sthinking on · via Claude Code · calculation per task from recorded decision counts, decisions in line | — | none recorded | Unclear | No interval or range was recorded for either side, so the gap (9.9 µs vs 87.8 s, 8.9 million times) is not tested against run-to-run variation. | Routing overhead: deterministic policy vs LLM routers vs Jev |
Marks: 95% intervals (Wilson for rates); median to p95 bands (not intervals)n is shown per side on every rowHollow marks: list-price calculations, not runs
6 rows from 1 study. Deterministic routing policy ahead on 1; 1 tie, 4 unclear. A side is ahead only where the intervals or ranges do not overlap.
When to pick which
Only from the rows above. A tie is not a reason to pick either side.
When to pick Deterministic routing policy
- Time to make one routing decision: 1.42 µs vs 12,543 ms. Claude Haiku 4.5’s median is above Deterministic routing policy’s 95th percentile (p50–p95 bands: Deterministic routing policy 1.42 µs to 2.33 µs; Claude Haiku 4.5 12,543 ms to 34,481 ms); not a confidence interval.
- Added routing cost per 1,000 tasks (calculation) (Every model call routed (49.5 per task)): $0.00 vs $441.74. A list-price calculation, not a measured difference. Calculation
- Added routing cost per 1,000 tasks (calculation) (Only System One decisions (7 per task)): $0.00 vs $62.47. A list-price calculation, not a measured difference. Calculation
- Added routing delay per task (calculation) (Every model call routed (49.5 per task)): 70.3 µs vs 620.9 s. A list-price calculation, not a measured difference. Calculation
- Added routing delay per task (calculation) (Only System One decisions (7 per task)): 9.9 µs vs 87.8 s. A list-price calculation, not a measured difference. Calculation
When to pick Claude Haiku 4.5
No row in this data puts Claude Haiku 4.5 ahead of Deterministic routing policy. Pick on other grounds (price, access, the tasks you run), or measure your own workload.
Side by side
The study charts, showing only these two. Open a study for every configuration.
1st, 2nd …: place by median, given only to a lane whose run range overlaps no other lane’s. A lane marked ~ overlaps another lane’s range, so it gets no place.
Time per decision · log scale: each gridline is 10 times the one before
| Item | Decision time | Median to p95 | n |
|---|---|---|---|
| Deterministic routing policy (Agent, in process) | 1.42 µs | 1.42 µs–2.33 µs | 20000 |
| Claude Haiku 4.5 (thinking on, via Claude Code) | 12.54 s | 12.54 s–34.48 s | 82 |
2 rows. Slowest Claude Haiku 4.5 (thinking on, via Claude Code) 12.54 s (median to p95 12.54 s–34.48 s, n 82). Fastest Deterministic routing policy (Agent, in process) 1.42 µs (median to p95 1.42 µs–2.33 µs, n 20000). Not all run ranges overlap.
NotesLines: median to p95 (not an interval)n 82–20000 per row
Median; whiskers = median to 95th percentile
The policy is the pure in-process decision (median 1.42 µs, p95 2.33 µs), timed over 20,000 decisions after 5,000 warm-up calls. The Claude routers are wall time per call through the Claude Code CLI, one call at a time, from the recorded routing runs. Jev is wall time of a direct HTTPS call from the same Mac over a home network (246 calls in a 35-second window; its API reports no server time), so the network is inside it. A different route from the CLI, so the chart shows what a caller waits, not model compute time. The whisker is the median to the 95th percentile, not a confidence interval.
Sources: Routing overhead runs: policy microbenchmark and CLI start-up, Routing runs: Jev router vs LLM routing, Jev live run: 246 timed calls on the 82 routing decisions
| Item | Completed | 95% interval | n |
|---|---|---|---|
| Deterministic routing policy (Agent, in process) | 100% | 100%–100% | 20000 |
| Claude Haiku 4.5 (thinking on, via Claude Code) | 100% | 96%–100% | 82 |
2 rows. All at 100%.
NotesWhiskers: 95% Wilson intervaln 82–20000 per row
Completed calls ÷ calls; whiskers = 95% Wilson interval
A completed call returned a decision, right or wrong (accuracy is in the routing study). Whiskers are 95% Wilson intervals.
Sources: Routing overhead runs: policy microbenchmark and CLI start-up, Routing runs: Jev router vs LLM routing, Jev live run: 246 timed calls on the 82 routing decisions
- Every model call routed (49.5 per task)
- Only System One decisions (7 per task) (square)
Gap labels, Only System One decisions (7 per task) vs Every model call routed (49.5 per task): Only System One decisions (7 per task) is x% higher (+) or lower (−) than Every model call routed (49.5 per task), calculated from the two values shown (the change counted from Every model call routed (49.5 per task)’s value).
| Item | Every model call routed (49.5 per task) | Only System One decisions (7 per task) |
|---|---|---|
| Deterministic routing policy (Agent, in process) | $0 | $0 |
| Claude Haiku 4.5 (thinking on, via Claude Code) | $442 | $62.47 |
List-price calculation, not a run. 2 rows, 2 series: Every model call routed (49.5 per task), Only System One decisions (7 per task). Every model call routed (49.5 per task): highest Claude Haiku 4.5 (thinking on, via Claude Code) $442. Lowest Deterministic routing policy (Agent, in process) $0. Only System One decisions (7 per task): highest Claude Haiku 4.5 (thinking on, via Claude Code) $62.47. Lowest Deterministic routing policy (Agent, in process) $0.
Notes
Decisions per task from recorded runs × cost per decision
A calculation. Decisions per task: the median of 48 recorded bench runs (routing was off in them, so every model call counts as one decision a router would make). Median recorded work cost per task: $3.03. Claude router costs are list-price calculations; Jev’s is a list-price calculation too (its recorded run’s provider-reported cost is the same).
Sources: Routing overhead runs: policy microbenchmark and CLI start-up, Routing overhead per 1,000 tasks (calculation), Routing runs: Jev router vs LLM routing, Anthropic list prices (Claude models), Jev 1.13 list price, Jev live run: 246 timed calls on the 82 routing decisions
- Every model call routed (49.5 per task)
- Only System One decisions (7 per task) (square)
Gap labels, Only System One decisions (7 per task) vs Every model call routed (49.5 per task): Only System One decisions (7 per task) is x% higher (+) or lower (−) than Every model call routed (49.5 per task), calculated from the two values shown (the change counted from Every model call routed (49.5 per task)’s value).
| Item | Every model call routed (49.5 per task) | Only System One decisions (7 per task) |
|---|---|---|
| Deterministic routing policy (Agent, in process) | 70.3 µs | 9.9 µs |
| Claude Haiku 4.5 (thinking on, via Claude Code) | 621 s | 87.8 s |
List-price calculation, not a run. 2 rows, 2 series: Every model call routed (49.5 per task), Only System One decisions (7 per task). Every model call routed (49.5 per task): slowest Claude Haiku 4.5 (thinking on, via Claude Code) 621 s. Fastest Deterministic routing policy (Agent, in process) 70.3 µs. Only System One decisions (7 per task): slowest Claude Haiku 4.5 (thinking on, via Claude Code) 87.8 s. Fastest Deterministic routing policy (Agent, in process) 9.9 µs.
Notes
Decisions per task × median decision time, if every decision waits in line
A calculation and an upper bound: it assumes each decision waits for the one before. Median recorded task wall time: 10.3 min. Jev’s delay uses its live median over the API from one Mac (network included); the Claude routers’ includes the CLI.
Sources: Routing overhead runs: policy microbenchmark and CLI start-up, Routing overhead per 1,000 tasks (calculation), Routing runs: Jev router vs LLM routing, Anthropic list prices (Claude models), Jev 1.13 list price, Jev live run: 246 timed calls on the 82 routing decisions
How a row is called
- AheadThe 95% intervals do not overlap, or the run ranges or p50–p95 bands do not overlap with at least 5 runs per side.
- TieThe values match, or both sit at the same ceiling.
- UnclearThe intervals or ranges overlap, too few runs were recorded, no interval was recorded, or more is not better (token counts are never a win).
- CalculationDerived from list prices and recorded counts. Not a bill and not a run.
The dataset compiler makes every call; this page only draws it. No composite score, no rank.
Questions
- Which is better, Deterministic routing policy or Claude Haiku 4.5?
- Deterministic routing policy and Claude Haiku 4.5 share 2 measured metrics and 4 list-price calculations from one study. Deterministic routing policy leads on 1 row: Time to make one routing decision, 1.42 µs vs 12,543 ms. On those rows the p50–p95 bands do not overlap; only a 95% interval is a confidence interval. The other rows are 1 tie and 4 unclear; each row says why. Calculation rows are derived from list prices and recorded counts; they are not bills or runs.
- How were Deterministic routing policy and Claude Haiku 4.5 measured?
- They share 2 measured metrics and 4 list-price calculations from 1 public study: Routing overhead: deterministic policy vs LLM routers vs Jev. Every row names its configuration, its sample size and its interval or range.
- How do Deterministic routing policy and Claude Haiku 4.5 compare on time to make one routing decision?
- Deterministic routing policy: 1.42 µs (Agent · in process · routing overhead per decision; n = 20000; p50 to p95 1.42 µs to 2.33 µs). Claude Haiku 4.5: 12,543 ms (thinking on · via Claude Code · routing overhead per decision; n = 82; p50 to p95 12.54 s to 34.48 s). Claude Haiku 4.5’s median is above Deterministic routing policy’s 95th percentile (p50–p95 bands: Deterministic routing policy 1.42 µs to 2.33 µs; Claude Haiku 4.5 12,543 ms to 34,481 ms); not a confidence interval.
- How do Deterministic routing policy and Claude Haiku 4.5 compare on routing calls that returned a decision?
- Deterministic routing policy: 100% (20000/20000) (Agent · in process · routing overhead per decision; n = 20000; 95% interval 100% to 100%). Claude Haiku 4.5: 100% (82/82) (thinking on · via Claude Code · routing overhead per decision; n = 82; 95% interval 96% to 100%). The 95% intervals overlap (Deterministic routing policy 100% to 100%; Claude Haiku 4.5 96% to 100%), so this sample cannot separate them.
The studies behind this page
Routing overhead: deterministic policy vs LLM routers vs Jev
How much delay and cost a router adds per decision: an in-process policy, Claude routers through a CLI, and Jev. Plus CLI start-up tax and per-task totals.