The Pareto frontier: reading LLM cost against quality

How to read a frontier chart

Each point is one configuration: a model, an effort setting and the CLI that ran it. Our hard-task chart puts the strict pass rate on the vertical axis and the list-price cost per strict pass on the horizontal axis. On the hard set, the whole trimmed reply must pass the validator; a code fence or added prose fails. Upper-left is better.

All costs below are calculations, not bills: reported tokens times list price, with cache reads and writes priced separately. We sum costs over all scored calls, failures included, then divide by the passes. The calls used flat subscriptions.

The hard set has 8 tasks: 24 calls per Claude Code configuration, 16 per Codex CLI configuration. It records 30 earlier attempts blocked before any model call; those are not scored. These are CLI-and-model pairs, not isolated model tests. Both studies used one host and separate batches; timing includes CLI start-up and context.

One of seven configurations is on the frontier (a calculation from observed pass rates and costs): Claude Sonnet 5.5 through Claude Code. It passed 24 of 24 calls (95% interval 86% to 100%) at $0.0143 per strict pass. Opus 5.5 and Fable 5.1 also passed 24 of 24 (95% interval 86% to 100% each), at $0.0282 and $0.0933. Haiku 4.5 passed 11 of 24 (46%, 95% interval 28% to 65%) at $0.0672. Each has a higher calculated cost per pass than Sonnet 5.5, and none has a higher observed pass rate.

This task set has a ceiling: six of seven configurations passed every call. Their observed pass rates tie, so calculated cost decides the frontier.

Why the frontier moves with the task set and the axes

The task set. The same two axes give a different frontier on different tasks. On the hard set they give 1 point. On the five short tasks they give 2 points (a calculation from cost per pass and pass rate). These are Sonnet 5.5 (12 of 15 passed, 95% interval 55% to 93%, $0.0062) and Opus 5.5 (low). The latter passed 15 of 15 (95% interval 80% to 100%) at $0.0083.

Sonnet missed 3 short-set calls, so a costlier option with every call passed stays on that frontier. In these two studies, Opus 5.5 (low) and GPT-6.1 Sol (low) ran only on the short tasks.

The axes. The short-task frontier below uses median time per call, cost per pass and pass rate. Its note lists the pass rates. Five of nine configurations are on it (a calculation from those point estimates), all through Claude Code: Fable 5.1, Sonnet 5.5, Opus 5.5 (high), Opus 5.5 and Opus 5.5 (low).

Fable 5.1 stays on this frontier with the lowest observed median time: 1.9 s (range 1.4 s to 9.8 s, n = 15 calls). It costs $0.0205 per pass (a calculation). It passed 15 of 15 (95% interval 80% to 100%). On cost and quality alone, the cheaper Opus 5.5 (low) dominates it.

A new axis cannot remove a point from the frontier. One exception: another point ties it exactly on the old axes and wins on the new one. Here, cost and quality alone give 2 points. Adding speed gives 5 (both counts are calculations from point estimates).

Why a point near the frontier can tie with one on it

A frontier uses point estimates. Our data has spread:

• Hard set, GPT-6.1 Sol (high) through Codex CLI. It passed 16 of 16 (95% interval 81% to 100%) at $0.0151 per pass. Sonnet 5.5 passed 24 of 24 (95% interval 86% to 100%) at $0.0143. The intervals overlap; they do not show a quality gap or prove equal underlying pass rates. The $0.0008 cost gap is a calculation with no interval, so it may not hold in a new run.
• Short set, Sonnet 5.5. Its 12 of 15 passes (80%, 95% interval 55% to 93%) overlap the perfect 15 of 15 interval (80% to 100%). All 3 misses gave the expected answer plus working lines, which the exact-text validator rejects.
• Short set, speed. The Fable 5.1 range overlaps the Sonnet 5.5 range (2.2 s to 7.7 s, median 2.3 s, 15 calls each). The ranges do not separate their speed. A range is not an interval. The Sonnet 5.5 against Fable 5.1 page finds 3 ties and 11 unclear rows.

Treat the frontier as a short list, not a ranking. One gap is clear: on the hard set, the Haiku 4.5 interval (28% to 65%) does not overlap the Sonnet 5.5 interval (86% to 100%). See Wilson confidence intervals for how we draw them.

How to use a frontier

1. Pick the axes you pay for. Usually these are quality on your task, cost per correct answer, and time if a person waits.
2. Drop each option that has a rival at least as good on every axis and better on one. The rest is your short list.
3. Compare each gap with the spread. Overlapping 95% intervals do not separate the rates. Overlapping run ranges leave timing gaps unclear; neither proves equality.
4. Test the short list on your own tasks. Our hard set hit a ceiling, so it says little about your hardest work.
5. Redraw the chart when prices or models change. Our costs use list prices dated 2026-09-21 (Anthropic) and 2026-10-03 (OpenAI). The AI cost calculator prices your own volume.

Our leaderboard also keeps the axes apart, with no composite score. The checklist for reading benchmarks honestly covers the rest.

Frequently asked questions

What is a Pareto frontier in AI model selection?

It is the set of options that no rival matches or beats on every axis and beats on at least one. It depends on your task set and chosen axes.

Is the model on the frontier always the right choice?

No. The frontier describes observed point estimates. It cannot predict results on your tasks or establish a lead where intervals or ranges overlap.

Why is Sonnet on the frontier on hard tasks?

Its observed pass rate ties the other perfect configurations, at the lowest calculated cost per strict pass. The task set hit a ceiling.

Does the frontier include speed?

It can. Our short-task chart includes speed, cost and quality. The hard-task chart uses only cost and quality.