{"i":17,"slug":"haiku-thinking-on-off","chart":{"id":"haiku-thinking-hard-pass","title":"Haiku thinking study: pass rate on eight hard tasks","subtitle":"Claude Haiku 4.5 in Claude Code; strict and lenient","kind":"dot-range","unit":"rate","polarity":"higher","yLabel":"Passed","series":[{"name":"Strict pass","points":[{"label":"Claude Haiku 4.5 (thinking off) · Claude Code","value":0.1667,"lo":0.0668,"hi":0.3585,"n":24,"highlight":true},{"label":"Claude Haiku 4.5 (thinking on) · Claude Code","value":0.4583,"lo":0.2789,"hi":0.6493,"n":24,"highlight":false}]},{"name":"Lenient (format misses counted)","points":[{"label":"Claude Haiku 4.5 (thinking off) · Claude Code","value":0.1667,"lo":0.0668,"hi":0.3585,"n":24,"highlight":true},{"label":"Claude Haiku 4.5 (thinking on) · Claude Code","value":0.6667,"lo":0.4671,"hi":0.8203,"n":24,"highlight":false}]}],"note":"Whiskers are 95% Wilson intervals over calls. Three repeats per task are related, so these intervals do not measure uncertainty across unseen tasks. A format miss (right answer in the wrong wrapping) never counts as a strict pass. The thinking-on cell is the 24 recorded Haiku receipts of the hard head-to-head, reused; the thinking-off cell ran later on another account.","whisker":"ci95","factContext":"eight hard validated tasks, thinking on vs off","sourceIds":["agent-haiku-thinking","agent-provider-h2h-hard"]}}