pass@k explained: pass@1, pass@k and pass^k with real runs

pass@1 vs pass@k vs pass^k

Each metric answers a different question:

• pass@1: if I ask once, how often is the answer right?
• pass@k: if I ask k times, how often is at least one answer right?
• pass^k: if I ask k times, how often are all k answers right?

A task gets n tries (n at least k), and c of them pass. C(a, b) is the number of ways to choose b items from a:

pass@1 = c / n
pass@k = 1 − C(n − c, k) / C(n, k)
pass^k = C(c, k) / C(n, k)

The pass@k formula is the unbiased estimator from the HumanEval paper (Chen et al. 2021). The pass^k metric comes from the τ-bench paper (Yao et al. 2024).

Compute each value per task, then take the mean over the tasks. At k = 1 all three are the same number. As k grows, pass@k never falls and pass^k never rises.

A worked example from our hard set

In the hard head-to-head, seven configurations ran 8 hard tasks with strict validators. Claude Haiku 4.5 through Claude Code had 3 tries per task. One try is one call. It passed 11 of 24 calls (46%, 95% interval 28% to 65%):

Per task, the passes in 3 tries were 3, 1, 2, 0, 0, 3, 2 and 0. Take the "Fix a time-zone day-length function (DST)" task by hand: Haiku passed 1 of 3 tries (c = 1, n = 3). Calculation:

• pass@1 = 1/3
• pass@2 = 1 − C(2, 2) / C(3, 2) = 1 − 1/3 = 2/3
• pass@3 = 1 − C(2, 3) / C(3, 3) = 1 − 0 = 1
• pass^3 = C(1, 3) / C(3, 3) = 0

Over all 8 tasks, the same three metrics are below (calculation from the pass matrix). With n = k = 3, a task scores 1 on pass@3 if any try passed, and on pass^3 if all three did:

Eight tasks is a very small n, and the pass@3 and pass^3 intervals overlap. So the example shows how the metrics behave, not a precise rate for Haiku. The 11/24 interval also treats tries on one task as independent, so the real uncertainty can be wider.

The other six configurations passed every call. On Claude Code that is 24/24 (95% interval 86% to 100%). On Codex CLI it is 16/16 (81% to 100%), with 2 tries per task. Every task cell is full, so every metric is 100% there. This is a ceiling: no metric separates those six.

When pass@k fits and when pass^k fits

Use pass@k when you can test every sample and keep one that passes. Examples are code with unit tests, output with a validator, or a person who picks from k drafts. The cost is k calls per task.

Use pass^k when nothing checks the output. An unattended agent, a scheduled job or a pipeline step must pass every time.

Use pass@1 when the user gets one answer and no retry.

On Haiku's hard tasks, the gap between pass@3 (5/8) and pass^3 (2/8) shows inconsistency. It solved 5 tasks at least once, but only 2 of them on every try.

Why more samples do not fix a consistent error

pass@k helps only when the tries differ. When a model makes the same mistake each time, c is 0 and pass@k is 0 for every k. The consistency study asked the same prompt 10 times:

• Exact number: Haiku 4.5 passed 0/10 (95% interval 0% to 28%). It gave the same wrong number every time: 289 (expected: 282). Consistent is not the same as correct, and pass@10 is 0 (calculation).
• JSON object: Haiku passed 1/10 strictly (95% interval 2% to 40%), and 9 more replies were correct but in the wrong format. Because one try passed, pass@10 is 100%, but pass^10 is 0% (calculation). A strict parser with no retry rejects 9 of 10.

A format rule fails in the same way. In the five-task head-to-head, Sonnet 5.5 through Claude Code passed 0/3 tries (95% interval 0% to 56%, a calculation) on "Multi-step shift arithmetic". Every reply ended on the expected answer (2292) but added working lines, which the exact-text validator rejects. So pass@3 is 0 (calculation): every try broke the same rule.

On the hard set, 5 of Haiku's 13 non-passes were format misses and 8 were wrong answers:

A lenient reading accepts those format misses. Then 7 of 8 tasks had at least one correct answer in 3 tries (calculation; 95% interval 53% to 98%). Strictly, 5 of 8 did.

How to report pass@k

1. Name the metric and k: pass@1, pass@3 or pass^3.
2. Give the tries per task (at least k) and the number of tasks.
3. Use the unbiased estimator.
4. Give a 95% interval. A count of tasks takes a Wilson interval.
5. Say if the check is strict or lenient, and report format misses apart from wrong answers.
6. Say when a set hits a ceiling: when every cell is n of n, no metric separates the models.

Frequently asked questions

What is the difference between pass@1 and pass@k?

The chance that one try passes is pass@1. The chance that at least one of k tries passes is pass@k. On our 8 hard tasks, Claude Haiku 4.5 passed 11/24 calls (pass@1 46%, 95% interval 28% to 65%). It passed 5/8 tasks at least once in 3 tries (pass@3 63%, a calculation, 95% interval 31% to 86%).

Is a higher k always better?

No. A higher k never lowers pass@k, but the model stays the same. Each extra try costs a call and needs a check. A high k can also hide inconsistency: Haiku passed the JSON prompt 1/10 times strictly, yet its pass@10 is 100% (calculation).

What is pass^k?

pass^k is the chance that all k tries pass. It measures reliability for work that nobody checks. For Haiku on the hard set, pass^3 was 2/8 tasks (25%, a calculation, 95% interval 7% to 59%), against 5/8 for pass@3.

How many samples do I need for pass@k?

You need at least k tries per task. We did not measure how the estimate changes with the number of tries. The number of tasks sets the width of the interval. With 8 tasks, a pass@3 of 5/8 still has a 95% interval of 31% to 86%.