The McNemar test: compare two models on the same cases

Why compare the models case by case

A hard case can lower both models’ scores. Separate Wilson intervals ignore that link. A paired test keeps it.

Our routing study gives the example. Three routers answered the same 82 typed decisions. We pair Jev’s recorded production pass from 2026-10-05 with one Claude Code CLI pass per Claude router. An exact decision has every scored question acceptable. Counts with 95% Wilson intervals:

• Jev 1.13: 74 of 82 (81.9% to 95.0%)
• Claude Haiku 4.5: 73 of 82 (80.4% to 94.1%)
• Claude Sonnet 5.5: 77 of 82 (86.5% to 97.4%)

This chart uses a later Jev live run: 221 of 246 calls across three repeats of the same 82 decisions. Its case-level 95% interval is 81.9% to 95.0%; repeats are not independent. The paired table uses the earlier pass.

All intervals overlap: no router is ahead. Does one win more disagreements?

The 2 by 2 table and a worked example

Sort cases into four cells: both right, only the first right, only the second right, both wrong. The middle cells are discordant pairs. The study’s "Same cases, two routers" table and public routing receipts give these counts for Jev’s recorded pass. The p values are calculations from the discordant cells.

Each row holds 82 cases. Haiku is right on 70 + 3 = 73 and Jev on 70 + 4 = 74. They differ by one case but disagree on 7; accuracies hide that.

Both routers were right in 70 and 73 of the 82 cases; those cases cannot inform this test. Two of four suites hit a ceiling. All three routers got every message-intent case (20 of 20, 84% to 100%) and every is-it-a-rule case (12 of 12, 76% to 100%) exactly right. Those 32 cases (calculation: 20 + 12) are both-right in every pair and cannot separate them.

How the exact test works

The null hypothesis treats disagreements as independent coin flips: either router is equally likely to be the only one right. Let b count only-first-right cases, c only-second-right cases, and d = b + c. Then b follows a binomial distribution: d trials, probability 0.5. The two-sided p value is p = min(1, 2 × P(X ≤ min(b, c))), where X has that distribution.

Both table values follow (calculations):

• Sonnet vs Jev: b = 4, c = 1, d = 5. P(X ≤ 1) = (1 + 5) / 32. So p = 2 × 6/32 = 0.375.
• Haiku vs Jev: b = 3, c = 4, d = 7. P(X ≤ 3) = (1 + 7 + 21 + 35) / 128 = 0.5. So p = 2 × 0.5 = 1.

Both match the table. All disagreements on one side give the smallest p: p = min(1, 2 × 0.5^d) (calculation). With 5 disagreements that is 2/32 = 0.0625, above 0.05. With 6 it is 2/64 = 0.03125, below 0.05. So the test needs at least 6 disagreements, and exactly 6 must all fall on one side.

The chi-square form approximates many disagreements; with few, use the exact form. In Python, scipy.stats.binomtest(min(b, c), b + c, 0.5, alternative="two-sided").pvalue gives the same p for disagreements. With none, report p = 1. Both pairs were checked.

How to read p

• p measures surprise. Under equal chances, p = 0.375 means a 4 to 1 or more lopsided split, either way, occurs 37.5% of the time (calculation: 12/32). That is common; it is not the probability that the routers are equal.
• No difference found is not "the same". The Sonnet pair has 5 disagreements, so no split could give p below 0.05. The data cannot separate them or show they are equal.
• p is not an effect size. Read the gap from the counts. Sonnet minus Jev is (4 − 1) / 82 = +3.7 points. Haiku minus Jev is (3 − 4) / 82 = −1.2 points. Both are calculations.

When to use it, and where it stops

Use it for right-or-wrong results on the same items. For independent case sets, use an unpaired test and show intervals; difficulty can differ. For numbers such as cost, use a paired test on differences.

Limits in our study:

• One sample per item. One pass per router, including Jev’s recorded production pass. Production asks for a second sample at low confidence; we did not compare that.
• Several pairs. Two pairs add chances of a false alarm. Neither passed; a correction could only raise the p values.
• A home advantage. We revised the case sets against Jev answers on 2026-10-04 and 2026-10-05. A paired test cannot undo a case set that favours one side.
• Few cases. 82 cases give wide intervals and only 7 and 5 disagreements. The test assumes independent cases; related cases can weaken that assumption.
• Report the table, not only p. Show n, the four cells and each interval. Then anyone can redo the test.

Frequently asked questions

When should I use a McNemar test instead of comparing accuracies?

Use it for right-or-wrong answers to the same cases. Accuracies hide disagreements: Claude Haiku 4.5 and Jev 1.13 differ by one exact decision (73 and 74 of 82), but disagree on 7, split 3 to 4.

What are discordant pairs?

They are the cases where exactly one of the two models was right. Claude Haiku 4.5 and Jev 1.13 had 7 (3 + 4). Claude Sonnet 5.5 and Jev 1.13 had 5 (4 + 1).

Why can overlapping intervals still hide a difference?

Separate intervals omit the case-by-case link. A paired test can detect a difference despite overlap. Here both methods find no difference. We have no example where pairing separated models that intervals did not.

How many test cases do I need for a McNemar test?

Disagreements are the count that matters. The exact test needs at least 6 to reach p below 0.05; at that minimum, all must favour one side (calculation). Five disagreements never pass, not even a 5 to 0 split (p = 0.0625). Our 82 cases gave 7 and 5 disagreements. Without a power study, we cannot say how many cases a real gap needs.