Strict grading and format misses: when a right answer fails

What a format miss is

A format miss holds the right answer in the wrong shape: code fences or extra working lines in our two studies.

Our hard-task study puts every call in one of three outcomes:

• Strict pass: the whole reply, trimmed, passes the validator.
• Format miss: the strict check fails, but an extractor finds an answer in the reply that passes the same validator. The extractor looks for a fenced block, outer JSON, a first code line or one output line.
• Wrong answer: neither check passes.

Every hard-set prompt required output only. Some explicitly banned code fences; the strict check rejected them. All 5 hard-set format misses passed after extraction from a code fence. A format miss never counts as a strict pass.

The five-task study had 3 non-passes in 130 calls, all from Claude Sonnet 5.5 on one task. Each ended on the right number, 2292, after working lines. The exact-text validator rejects that.

Strict or lenient: the reader decides

Ask who reads the reply.

• Code reads it. Raw JSON or exact-value checks can reject fences. Grade that requirement. The strict rate describes this harness, not every production pipeline.
• A person reads it, or your harness strips wrappers. Grade leniently, or report both readings.

This design rule is not a measurement. We did not measure real parser failures on fenced replies.

Declare the reading before the first call. Use one validator: extraction changes where you look, not what counts as right.

What the reading changed in our runs

The rates below use 95% Wilson intervals. Across all 7 configurations, 139 of 152 calls passed strictly (91%, 95% interval 85.9% to 94.9%). 144 of 152 had a correct answer (95%, 90.0% to 97.3%). The gap is 5 calls, or 3.3 points (a calculation). The intervals overlap, so neither reading is ahead under our conservative interval rule. This is not a test of the paired gap.

Of 13 non-passes, 5 were format misses and 8 wrong answers. All 13 came from Claude Haiku 4.5. The other 6 configurations had neither; pooling hid this.

Haiku passed 11 of 24 strictly (46%, 28% to 65%). It gave a correct answer in 16 of 24 (67%, 47% to 82%). Those intervals overlap, so neither reading is ahead under that rule. The observed gap remains 5 calls. Both Haiku intervals sit below Claude Sonnet 5.5's 24 of 24 (86% to 100%). Counting format misses still leaves a gap (Haiku vs Sonnet).

In the repeated-prompt study, grading changes a verdict. Haiku ran each prompt 10 times. On the JSON prompt it had 1 strict pass, 9 format misses and 0 wrong answers: 1/10 strict (2% to 40%) and 10/10 lenient (72% to 100%; a calculation, 1 + 9 of 10). Sonnet 5.5 and GPT-6.1 Sol (medium) passed 10/10 (72% to 100%). Strictly, they are ahead of Haiku: the intervals do not overlap. Read leniently, all three have 10/10 (95% interval 72% to 100% each): an observed tie, not proof of equal capability.

On the exact-number prompt, Haiku had 10 wrong answers and 0 format misses (0/10, 0% to 28%): an answer problem, not a format problem.

How to report both readings

1. Report three counts: strict passes, format misses and wrong answers.
2. Give k of n and a 95% interval for each reading. Keep the series separate.
3. State the observed gap. Under our interval rule, neither side is ahead when intervals overlap. This does not prove equality.
4. Split by configuration and by task. Our pooled 91% hid that Haiku held all 13 non-passes.
5. Report zeros. "0 format misses" is a result.
6. Test the validator first. Ours passed 8 of 8 reference answers, rejected 26 of 26 plausible wrong answers and flagged 8 of 8 wrapped references as format misses.

The counts are small: 13 non-passes and 5 format misses. The two readings use the same calls, so they are not independent samples. These intervals use calls as n; repeated tasks do not provide that many independent tests of new-task performance. The study separately reports 30 earlier Codex attempts blocked before any model call, excluded from the 152 scored calls.

Claude rows used CLI default effort; GPT-6.1 Sol used medium or high through Codex CLI. Each CLI adds its own context, so comparisons across routes describe model and CLI pairs. Six configurations passed every call, so the set has a ceiling for them (24/24 is 86% to 100%; 16/16 is 81% to 100%).

How to cut format misses

• State the format. Our prompts did; Haiku still had 5 format misses in 24 calls. Rules do not guarantee compliance.
• Validate and retry. Ask again after a format failure. We retried nothing, so retry recovery is unknown. Count the extra call.
• Parse leniently where safe. For a human reader, strip one wrapping fence. Keep the strict score to show rule failures.
• Use a judge for open answers. A rule cannot grade an essay. See LLM-as-judge, and how to read AI benchmarks honestly for the rules on n and intervals.

Frequently asked questions

What is a format miss?

A reply with the right answer in the wrong format, for example inside a code fence although the prompt said not to. On our hard set it was 5 of 13 non-passes in 152 calls. The other 8 were wrong answers.

Should my eval accept answers in a code fence?

Only if the reader accepts it. Raw JSON or exact-value checks must reject fences unless the pipeline removes them. For human readers, report both readings. On our repeated JSON prompt, Haiku had 9 format misses in 10 calls. All 9 were fenced replies. We did not test why the model added fences.

Does strict grading make a model look worse?

Our shared validator keeps every strict pass in the lenient count. The strict rate cannot exceed the lenient rate. On the hard set, 6 configurations had 0 format misses, so both readings gave them the same rate. Haiku 4.5 went from 46% strict (11/24, 95% interval 28% to 65%) to 67% lenient (16/24, 47% to 82%). The intervals overlap.

How do I report strict and lenient results?

Show both rates with k of n and a 95% interval, plus the three counts. Our example: Haiku 4.5 had 11/24 strict (28% to 65%), 16/24 lenient (47% to 82%), 5 format misses and 8 wrong answers. State the observed gap and whether the intervals overlap.