{ "schema": "agent-public-bench@1", "generatedAt": "2026-10-07T00:00:00.000Z", "url": "https://agent.sasid.ai/benchmarks/agent-memory", "study": { "slug": "agent-memory", "title": "Does memory help Claude Code? 8 kinds of agent memory, tested", "seoTitle": "Does CLAUDE.md help? Agent memory tested on Claude Code", "description": "200 graded Claude Code sessions: no memory, /init, curated, raw notes, dreamed notes, a long handbook, a Stop hook. What helped and what it cost.", "question": "Does project memory make Claude Code do better work, and which kind of memory?", "answer": "Claude Sonnet 5.5 followed almost every rule it could see without any memory (95% of code rules, 100% of folder rules), but only 6/15 of the team-knowledge checks; with an 11-line curated file it passed 15/15. When a memory file held the late-fee rate, Sonnet used it 15/15 times, even from raw notes that also held a stale rate; without it, Sonnet stopped and asked 7 of 9 times, while Haiku invented a rate and flagged it in 0/6 sessions. The smaller Haiku 4.5 was misled by messy memory: with the raw notes it ran the stale test command in 10/10 sessions, and after one dreaming pass in 1/10. A Stop hook enforced every code rule but could not carry a fact, and it used 1.6× the input tokens of no memory. Full-pass intervals overlap for most pairs (n = 15 per condition).", "date": "2026-10-06", "updated": "2026-10-06", "tags": [ "claude-code", "agent-memory", "claude-md", "hooks", "context-engineering" ], "method": [ "One small Node.js repository with real conventions: money in integer cents, coded errors, a clock helper, a log helper, append-only migrations, a generated report, a changelog rule and a late-fee rate that exists only in memory.", "Five tasks: refunds, a report formatting bug, an optional phone field, a late fee \"at our standard rate\", and a control task where memory does not matter.", "Eight conditions: no memory, the real /init output, a curated 11-line file, 56 raw dated notes (repeats, one-off events and notes that a later note contradicts), the same notes after one \"dreaming\" pass, the curated facts inside a 210-line handbook, a Stop hook that blocks finishing while a rule is broken, and curated plus hook.", "Claude Code 2.1.286 headless; Sonnet 3 repetitions per cell, Haiku 2; tools Bash, Read, Edit, Write, Glob, Grep; OS sandbox; the tester's own instruction files excluded and auto memory off (both checked with a probe).", "Grading after the session: hidden tests run in a sandbox, plus deterministic checks on the lines the agent added. Full pass = tests and every applicable check. Every attempt counts.", "The protocol was declared before the first counted session. One amendment, made before any Haiku session, added the Haiku lane. One erratum: the raw notes file holds 56 notes, not 58 as first written. The frozen handbook has 210 lines, not 211 as the protocol first stated. Measurements did not change." ], "caveats": [ "One small synthetic repository and one author of the facts: the curated file is an upper bound written with knowledge of the tasks.", "The hook checks the same code rules as the grader. It shows what rules written as code can do; it cannot carry a fact such as the late-fee rate.", "n = 15 per condition for Sonnet and 10 for Haiku: most full-pass intervals overlap, so most differences between conditions are not clear.", "Costs are the CLI's list-price estimates for subscription sessions, not invoices.", "In headless mode an agent that asks a question cannot get an answer, so asking counts as a failure here. In a live session the person would answer; the cost is the round trip." ], "sourceIds": [ "agent-memory-study" ], "stats": [ { "id": "memory-sessions", "label": "Claude Code sessions, every one graded (120 Sonnet 5.5, 80 Haiku 4.5)", "value": 200, "unit": "count", "display": "200", "n": 200, "note": "0 planned sessions not run." }, { "id": "memory-team-knowledge-none", "label": "Team-knowledge checks passed with no memory (Sonnet 5.5)", "value": 0.4, "unit": "rate", "display": "40% (6/15)", "n": 15, "ci": [ 0.1982, 0.6425 ] }, { "id": "memory-team-knowledge-curated", "label": "Team-knowledge checks passed with an 11-line curated file (Sonnet 5.5)", "value": 1, "unit": "rate", "display": "100% (15/15)", "n": 15, "ci": [ 0.7961, 1 ] }, { "id": "memory-late-fee-known", "label": "Late fee right when any memory file held the rate (Sonnet 5.5)", "value": 1, "unit": "rate", "display": "15/15", "n": 15, "ci": [ 0.7961, 1 ] }, { "id": "memory-late-fee-asked", "label": "Without the rate, sessions that asked for it and wrote no code (Sonnet 5.5)", "value": 0.7778, "unit": "rate", "display": "7/9", "n": 9, "note": "The other 2 guessed a rate and said it was a placeholder." }, { "id": "memory-late-fee-flagged-sonnet", "label": "Without the rate: sessions whose final message said the rate was unknown or a guess (Sonnet 5.5)", "value": 1, "unit": "rate", "display": "9/9", "n": 9 }, { "id": "memory-late-fee-flagged-haiku", "label": "Without the rate: sessions whose final message said the rate was unknown or a guess (Haiku 4.5)", "value": 0, "unit": "rate", "display": "0/6", "n": 6, "note": "Every other Haiku session invented a rate and reported the task done." }, { "id": "memory-init-broken-command", "label": "Sessions with the /init file that ran the broken test command it copied from the README (Sonnet 5.5)", "value": 0.8667, "unit": "rate", "display": "13/15", "n": 15 }, { "id": "memory-hook-input-ratio", "label": "Median input tokens per session, Stop hook only vs no memory (calculation, Sonnet 5.5)", "value": 1.6, "unit": "ratio", "display": "1.6×", "n": 15 }, { "id": "memory-haiku-raw-broken-command", "label": "Haiku 4.5 with the raw notes: sessions that ran the stale test command", "value": 1, "unit": "rate", "display": "10/10", "n": 10 }, { "id": "memory-haiku-dreamed-broken-command", "label": "Haiku 4.5 with the dreamed notes: sessions that ran the stale test command", "value": 0.1, "unit": "rate", "display": "1/10", "n": 10 }, { "id": "memory-total-cost", "label": "List-price estimate of every session (calculation, not an invoice)", "value": 17.49, "unit": "usd", "display": "$17.49", "n": 200 } ], "charts": [ { "id": "memory-full-pass", "title": "Full pass rate by kind of memory", "subtitle": "Hidden tests pass and every convention check passes · 95% Wilson intervals", "kind": "dot-range", "unit": "rate", "whisker": "ci95", "series": [ { "name": "Claude Sonnet 5.5", "points": [ { "label": "No memory", "value": 0.6, "lo": 0.3575, "hi": 0.8018, "n": 15 }, { "label": "/init CLAUDE.md", "value": 0.6, "lo": 0.3575, "hi": 0.8018, "n": 15 }, { "label": "Curated, 11 lines", "value": 1, "lo": 0.7961, "hi": 1, "n": 15, "highlight": true }, { "label": "Raw notes, 60 lines", "value": 0.9333, "lo": 0.7018, "hi": 0.9881, "n": 15 }, { "label": "Dreamed notes", "value": 1, "lo": 0.7961, "hi": 1, "n": 15 }, { "label": "Handbook, 210 lines", "value": 1, "lo": 0.7961, "hi": 1, "n": 15 }, { "label": "Stop hook only", "value": 0.8, "lo": 0.5481, "hi": 0.9295, "n": 15 }, { "label": "Curated + hook", "value": 1, "lo": 0.7961, "hi": 1, "n": 15 } ] }, { "name": "Claude Haiku 4.5", "points": [ { "label": "No memory", "value": 0.2, "lo": 0.0567, "hi": 0.5098, "n": 10 }, { "label": "/init CLAUDE.md", "value": 0.2, "lo": 0.0567, "hi": 0.5098, "n": 10 }, { "label": "Curated, 11 lines", "value": 0.7, "lo": 0.3968, "hi": 0.8922, "n": 10 }, { "label": "Raw notes, 60 lines", "value": 0.6, "lo": 0.3127, "hi": 0.8318, "n": 10 }, { "label": "Dreamed notes", "value": 0.7, "lo": 0.3968, "hi": 0.8922, "n": 10 }, { "label": "Handbook, 210 lines", "value": 0.3, "lo": 0.1078, "hi": 0.6032, "n": 10 }, { "label": "Stop hook only", "value": 0.8, "lo": 0.4902, "hi": 0.9433, "n": 10 }, { "label": "Curated + hook", "value": 0.9, "lo": 0.5958, "hi": 0.9821, "n": 10 } ] } ], "note": "Claude Code 2.1.286, 5 tasks in one small repository. Sonnet: 3 repetitions per cell (n = 15 per condition); Haiku: 2 (n = 10). A condition is better only when its interval does not overlap the other's.", "sourceIds": [ "agent-memory-study" ] }, { "id": "memory-knowledge-class", "title": "Where memory helps: what the repo shows vs what only the team knows", "subtitle": "Share of convention checks passed, pooled by kind of knowledge (Claude Sonnet 5.5)", "kind": "grouped-bar", "unit": "rate", "series": [ { "name": "Rules the code already shows", "points": [ { "label": "No memory", "value": 0.95, "lo": 0.863, "hi": 0.9829, "n": 60 }, { "label": "/init CLAUDE.md", "value": 0.95, "lo": 0.863, "hi": 0.9829, "n": 60 }, { "label": "Curated, 11 lines", "value": 1, "lo": 0.9398, "hi": 1, "n": 60 }, { "label": "Raw notes, 60 lines", "value": 1, "lo": 0.9398, "hi": 1, "n": 60 }, { "label": "Dreamed notes", "value": 1, "lo": 0.9398, "hi": 1, "n": 60 }, { "label": "Handbook, 210 lines", "value": 1, "lo": 0.9398, "hi": 1, "n": 60 }, { "label": "Stop hook only", "value": 1, "lo": 0.9398, "hi": 1, "n": 60 }, { "label": "Curated + hook", "value": 1, "lo": 0.9398, "hi": 1, "n": 60 } ] }, { "name": "Rules the folders hint at", "points": [ { "label": "No memory", "value": 1, "lo": 0.8454, "hi": 1, "n": 21 }, { "label": "/init CLAUDE.md", "value": 1, "lo": 0.8454, "hi": 1, "n": 21 }, { "label": "Curated, 11 lines", "value": 1, "lo": 0.8454, "hi": 1, "n": 21 }, { "label": "Raw notes, 60 lines", "value": 1, "lo": 0.8454, "hi": 1, "n": 21 }, { "label": "Dreamed notes", "value": 1, "lo": 0.8454, "hi": 1, "n": 21 }, { "label": "Handbook, 210 lines", "value": 1, "lo": 0.8454, "hi": 1, "n": 21 }, { "label": "Stop hook only", "value": 1, "lo": 0.8454, "hi": 1, "n": 21 }, { "label": "Curated + hook", "value": 1, "lo": 0.8454, "hi": 1, "n": 21 } ] }, { "name": "Team knowledge only", "points": [ { "label": "No memory", "value": 0.4, "lo": 0.1982, "hi": 0.6425, "n": 15 }, { "label": "/init CLAUDE.md", "value": 0.6667, "lo": 0.4171, "hi": 0.8482, "n": 15 }, { "label": "Curated, 11 lines", "value": 1, "lo": 0.7961, "hi": 1, "n": 15 }, { "label": "Raw notes, 60 lines", "value": 1, "lo": 0.7961, "hi": 1, "n": 15 }, { "label": "Dreamed notes", "value": 1, "lo": 0.7961, "hi": 1, "n": 15 }, { "label": "Handbook, 210 lines", "value": 1, "lo": 0.7961, "hi": 1, "n": 15 }, { "label": "Stop hook only", "value": 0.6667, "lo": 0.4171, "hi": 0.8482, "n": 15 }, { "label": "Curated + hook", "value": 1, "lo": 0.7961, "hi": 1, "n": 15 } ] } ], "note": "Code shows: money in cents, coded errors, the clock helper, the log helper. Folders hint: append-only migrations, a new migration, the generated report. Team knowledge only: the changelog rule and the late-fee rate.", "sourceIds": [ "agent-memory-study" ] }, { "id": "memory-team-knowledge-by-model", "title": "Team knowledge followed, Sonnet vs Haiku", "subtitle": "Changelog rule and late-fee rate, pooled · 95% Wilson intervals", "kind": "dot-range", "unit": "rate", "whisker": "ci95", "series": [ { "name": "Claude Sonnet 5.5", "points": [ { "label": "No memory", "value": 0.4, "lo": 0.1982, "hi": 0.6425, "n": 15 }, { "label": "/init CLAUDE.md", "value": 0.6667, "lo": 0.4171, "hi": 0.8482, "n": 15 }, { "label": "Curated, 11 lines", "value": 1, "lo": 0.7961, "hi": 1, "n": 15 }, { "label": "Raw notes, 60 lines", "value": 1, "lo": 0.7961, "hi": 1, "n": 15 }, { "label": "Dreamed notes", "value": 1, "lo": 0.7961, "hi": 1, "n": 15 }, { "label": "Handbook, 210 lines", "value": 1, "lo": 0.7961, "hi": 1, "n": 15 }, { "label": "Stop hook only", "value": 0.6667, "lo": 0.4171, "hi": 0.8482, "n": 15 }, { "label": "Curated + hook", "value": 1, "lo": 0.7961, "hi": 1, "n": 15 } ] }, { "name": "Claude Haiku 4.5", "points": [ { "label": "No memory", "value": 0, "lo": 0, "hi": 0.2775, "n": 10 }, { "label": "/init CLAUDE.md", "value": 0.1, "lo": 0.0179, "hi": 0.4042, "n": 10 }, { "label": "Curated, 11 lines", "value": 0.8, "lo": 0.4902, "hi": 0.9433, "n": 10 }, { "label": "Raw notes, 60 lines", "value": 0.6, "lo": 0.3127, "hi": 0.8318, "n": 10 }, { "label": "Dreamed notes", "value": 0.8, "lo": 0.4902, "hi": 0.9433, "n": 10 }, { "label": "Handbook, 210 lines", "value": 0.3, "lo": 0.1078, "hi": 0.6032, "n": 10 }, { "label": "Stop hook only", "value": 0.8, "lo": 0.4902, "hi": 0.9433, "n": 10 }, { "label": "Curated + hook", "value": 1, "lo": 0.7225, "hi": 1, "n": 10 } ] } ], "note": "Both models had the same memory files. The smaller model followed the team rules less often when the facts sat in long or messy files.", "sourceIds": [ "agent-memory-study" ] }, { "id": "memory-late-fee", "title": "\"Charge our standard late fee\": what Sonnet 5.5 did", "subtitle": "The rate (1.25%, decided 2026-09-15) is in no file of the repository; the raw notes also hold a stale 2%", "kind": "stacked-bar", "unit": "count", "series": [ { "name": "Used the current rate (1.25%)", "points": [ { "label": "No memory", "value": 0, "n": 3 }, { "label": "/init CLAUDE.md", "value": 0, "n": 3 }, { "label": "Curated, 11 lines", "value": 3, "n": 3, "highlight": true }, { "label": "Raw notes, 60 lines", "value": 3, "n": 3, "highlight": true }, { "label": "Dreamed notes", "value": 3, "n": 3, "highlight": true }, { "label": "Handbook, 210 lines", "value": 3, "n": 3, "highlight": true }, { "label": "Stop hook only", "value": 0, "n": 3 }, { "label": "Curated + hook", "value": 3, "n": 3, "highlight": true } ] }, { "name": "Asked for the rate, wrote no code", "points": [ { "label": "No memory", "value": 3, "n": 3 }, { "label": "/init CLAUDE.md", "value": 2, "n": 3 }, { "label": "Curated, 11 lines", "value": 0, "n": 3 }, { "label": "Raw notes, 60 lines", "value": 0, "n": 3 }, { "label": "Dreamed notes", "value": 0, "n": 3 }, { "label": "Handbook, 210 lines", "value": 0, "n": 3 }, { "label": "Stop hook only", "value": 2, "n": 3 }, { "label": "Curated + hook", "value": 0, "n": 3 } ] }, { "name": "Guessed another rate", "points": [ { "label": "No memory", "value": 0, "n": 3 }, { "label": "/init CLAUDE.md", "value": 1, "n": 3 }, { "label": "Curated, 11 lines", "value": 0, "n": 3 }, { "label": "Raw notes, 60 lines", "value": 0, "n": 3 }, { "label": "Dreamed notes", "value": 0, "n": 3 }, { "label": "Handbook, 210 lines", "value": 0, "n": 3 }, { "label": "Stop hook only", "value": 1, "n": 3 }, { "label": "Curated + hook", "value": 0, "n": 3 } ] } ], "note": "Late-fee task, 3 sessions per condition. \"Asked\" means the agent searched the repository, found no rate and stopped with a question instead of code.", "sourceIds": [ "agent-memory-study" ] }, { "id": "memory-late-fee-haiku", "title": "\"Charge our standard late fee\": what Haiku 4.5 did", "subtitle": "The rate (1.25%, decided 2026-09-15) is in no file of the repository; the raw notes also hold a stale 2%", "kind": "stacked-bar", "unit": "count", "series": [ { "name": "Used the current rate (1.25%)", "points": [ { "label": "No memory", "value": 0, "n": 2 }, { "label": "/init CLAUDE.md", "value": 0, "n": 2 }, { "label": "Curated, 11 lines", "value": 2, "n": 2, "highlight": true }, { "label": "Raw notes, 60 lines", "value": 2, "n": 2, "highlight": true }, { "label": "Dreamed notes", "value": 2, "n": 2, "highlight": true }, { "label": "Handbook, 210 lines", "value": 2, "n": 2, "highlight": true }, { "label": "Stop hook only", "value": 0, "n": 2 }, { "label": "Curated + hook", "value": 2, "n": 2, "highlight": true } ] }, { "name": "Guessed another rate", "points": [ { "label": "No memory", "value": 2, "n": 2 }, { "label": "/init CLAUDE.md", "value": 2, "n": 2 }, { "label": "Curated, 11 lines", "value": 0, "n": 2 }, { "label": "Raw notes, 60 lines", "value": 0, "n": 2 }, { "label": "Dreamed notes", "value": 0, "n": 2 }, { "label": "Handbook, 210 lines", "value": 0, "n": 2 }, { "label": "Stop hook only", "value": 2, "n": 2 }, { "label": "Curated + hook", "value": 0, "n": 2 } ] } ], "note": "Late-fee task, 2 sessions per condition. \"Asked\" means the agent searched the repository, found no rate and stopped with a question instead of code.", "sourceIds": [ "agent-memory-study" ] }, { "id": "memory-broken-test-command", "title": "A stale README command: who still ran it?", "polarity": "lower", "subtitle": "Sessions that ran the README test command, which fails on Node 25 · 95% Wilson intervals", "kind": "dot-range", "unit": "rate", "whisker": "ci95", "series": [ { "name": "Claude Sonnet 5.5", "points": [ { "label": "No memory", "value": 0.8, "lo": 0.5481, "hi": 0.9295, "n": 15 }, { "label": "/init CLAUDE.md", "value": 0.8667, "lo": 0.6212, "hi": 0.9626, "n": 15 }, { "label": "Curated, 11 lines", "value": 0, "lo": 0, "hi": 0.2039, "n": 15 }, { "label": "Raw notes, 60 lines", "value": 0, "lo": 0, "hi": 0.2039, "n": 15 }, { "label": "Dreamed notes", "value": 0, "lo": 0, "hi": 0.2039, "n": 15 }, { "label": "Handbook, 210 lines", "value": 0, "lo": 0, "hi": 0.2039, "n": 15 }, { "label": "Stop hook only", "value": 0.6, "lo": 0.3575, "hi": 0.8018, "n": 15 }, { "label": "Curated + hook", "value": 0, "lo": 0, "hi": 0.2039, "n": 15 } ] }, { "name": "Claude Haiku 4.5", "points": [ { "label": "No memory", "value": 1, "lo": 0.7225, "hi": 1, "n": 10 }, { "label": "/init CLAUDE.md", "value": 1, "lo": 0.7225, "hi": 1, "n": 10 }, { "label": "Curated, 11 lines", "value": 0, "lo": 0, "hi": 0.2775, "n": 10 }, { "label": "Raw notes, 60 lines", "value": 1, "lo": 0.7225, "hi": 1, "n": 10 }, { "label": "Dreamed notes", "value": 0.1, "lo": 0.0179, "hi": 0.4042, "n": 10 }, { "label": "Handbook, 210 lines", "value": 0, "lo": 0, "hi": 0.2775, "n": 10 }, { "label": "Stop hook only", "value": 1, "lo": 0.7225, "hi": 1, "n": 10 }, { "label": "Curated + hook", "value": 0, "lo": 0, "hi": 0.2775, "n": 10 } ] } ], "note": "The README and npm test give a command that fails on Node 25. The curated, dreamed and handbook files name the right one. The raw notes hold both: an old \"run npm test\" note and a later correction. The /init file repeats the README.", "sourceIds": [ "agent-memory-study" ] }, { "id": "memory-input-tokens", "title": "What memory costs in context", "subtitle": "Median input tokens per session, cache reads included (Claude Sonnet 5.5)", "kind": "bar", "unit": "tokens", "series": [ { "name": "Median input tokens per session", "points": [ { "label": "No memory", "value": 78455, "n": 15 }, { "label": "/init CLAUDE.md", "value": 82262, "n": 15 }, { "label": "Curated, 11 lines", "value": 65045, "n": 15, "highlight": true }, { "label": "Raw notes, 60 lines", "value": 76963, "n": 15 }, { "label": "Dreamed notes", "value": 68218, "n": 15 }, { "label": "Handbook, 210 lines", "value": 85223, "n": 15 }, { "label": "Stop hook only", "value": 125674, "n": 15 }, { "label": "Curated + hook", "value": 69224, "n": 15 } ] } ], "note": "Input = uncached input + cache reads + cache writes over every turn, as the CLI reports it. Most of it is read from the prompt cache. The hook adds turns: each block sends the agent back to work.", "sourceIds": [ "agent-memory-study" ] }, { "id": "memory-cost-per-full-pass", "title": "List-price cost per fully correct result (calculation)", "subtitle": "Sum of the CLI's cost estimates for a condition, divided by its full passes", "kind": "grouped-bar", "unit": "usd", "series": [ { "name": "Claude Sonnet 5.5", "points": [ { "label": "No memory", "value": 0.1386, "n": 9 }, { "label": "/init CLAUDE.md", "value": 0.1359, "n": 9 }, { "label": "Curated, 11 lines", "value": 0.0818, "n": 15 }, { "label": "Raw notes, 60 lines", "value": 0.1009, "n": 14 }, { "label": "Dreamed notes", "value": 0.0896, "n": 15 }, { "label": "Handbook, 210 lines", "value": 0.1006, "n": 15 }, { "label": "Stop hook only", "value": 0.1278, "n": 12 }, { "label": "Curated + hook", "value": 0.0843, "n": 15 } ] }, { "name": "Claude Haiku 4.5", "points": [ { "label": "No memory", "value": 0.3786, "n": 2 }, { "label": "/init CLAUDE.md", "value": 0.4317, "n": 2 }, { "label": "Curated, 11 lines", "value": 0.1095, "n": 7 }, { "label": "Raw notes, 60 lines", "value": 0.1255, "n": 6 }, { "label": "Dreamed notes", "value": 0.1153, "n": 7 }, { "label": "Handbook, 210 lines", "value": 0.2615, "n": 3 }, { "label": "Stop hook only", "value": 0.1419, "n": 8 }, { "label": "Curated + hook", "value": 0.096, "n": 9 } ] } ], "note": "Sessions ran on a subscription; these are the CLI's list-price estimates, not bills. A failed session still costs money, so cost per correct result falls when fewer sessions fail.", "sourceIds": [ "agent-memory-study" ] }, { "id": "memory-wall-time", "title": "Time per session", "subtitle": "Median wall time in seconds", "kind": "grouped-bar", "unit": "seconds", "series": [ { "name": "Claude Sonnet 5.5", "points": [ { "label": "No memory", "value": 18, "n": 15 }, { "label": "/init CLAUDE.md", "value": 19, "n": 15 }, { "label": "Curated, 11 lines", "value": 21.9, "n": 15 }, { "label": "Raw notes, 60 lines", "value": 26.8, "n": 15 }, { "label": "Dreamed notes", "value": 27.2, "n": 15 }, { "label": "Handbook, 210 lines", "value": 23.6, "n": 15 }, { "label": "Stop hook only", "value": 27.3, "n": 15 }, { "label": "Curated + hook", "value": 22, "n": 15 } ] }, { "name": "Claude Haiku 4.5", "points": [ { "label": "No memory", "value": 54.2, "n": 10 }, { "label": "/init CLAUDE.md", "value": 52.9, "n": 10 }, { "label": "Curated, 11 lines", "value": 51.7, "n": 10 }, { "label": "Raw notes, 60 lines", "value": 51, "n": 10 }, { "label": "Dreamed notes", "value": 51.8, "n": 10 }, { "label": "Handbook, 210 lines", "value": 49.9, "n": 10 }, { "label": "Stop hook only", "value": 68.5, "n": 10 }, { "label": "Curated + hook", "value": 52.8, "n": 10 } ] } ], "note": "Up to four sessions ran at a time on one machine. Sessions without memory were often shorter because they stopped to ask or skipped the changelog.", "sourceIds": [ "agent-memory-study" ] }, { "id": "memory-dreaming-scorecard", "title": "Dreaming: what one consolidation pass kept and dropped", "subtitle": "Fixed-pattern checks on each memory file: 10 current facts, 5 stale notes, 9 transient notes", "kind": "grouped-bar", "unit": "count", "series": [ { "name": "Current facts kept (of 10)", "points": [ { "label": "Raw notes", "value": 10 }, { "label": "Dream 1", "value": 10 }, { "label": "Dream 2", "value": 10 }, { "label": "Dream 3", "value": 10 }, { "label": "Curated", "value": 10 }, { "label": "/init", "value": 5 } ] }, { "name": "Stale notes left (of 5)", "points": [ { "label": "Raw notes", "value": 5 }, { "label": "Dream 1", "value": 0 }, { "label": "Dream 2", "value": 0 }, { "label": "Dream 3", "value": 0 }, { "label": "Curated", "value": 0 }, { "label": "/init", "value": 1 } ] }, { "name": "Transient notes left (of 9)", "points": [ { "label": "Raw notes", "value": 9 }, { "label": "Dream 1", "value": 1 }, { "label": "Dream 2", "value": 1 }, { "label": "Dream 3", "value": 1 }, { "label": "Curated", "value": 0 }, { "label": "/init", "value": 0 } ] } ], "note": "Each dream is one Claude Sonnet call with no tools over the 56 raw notes. A stale note that the new file marks as replaced does not count as left. The one transient note every dream kept is the current dev-server port.", "sourceIds": [ "agent-memory-study" ] } ], "tables": [ { "id": "memory-by-condition", "title": "Every condition, Claude Sonnet 5.5", "columns": [ { "key": "condition", "label": "Memory", "unit": "text" }, { "key": "full", "label": "Full pass", "unit": "text" }, { "key": "functional", "label": "Hidden tests pass", "unit": "text" }, { "key": "code", "label": "Code-visible rules", "unit": "text" }, { "key": "structure", "label": "Folder-visible rules", "unit": "text" }, { "key": "tribal", "label": "Team knowledge", "unit": "text" }, { "key": "broken", "label": "Ran the stale test command", "unit": "text" }, { "key": "input", "label": "Median input tokens", "unit": "tokens" }, { "key": "wall", "label": "Median time (s)", "unit": "seconds" }, { "key": "hook", "label": "Hook blocks", "unit": "count" } ], "rows": [ { "condition": "No memory", "full": "9/15", "functional": "12/15", "code": "57/60", "structure": "21/21", "tribal": "6/15", "broken": "12/15", "input": 78455, "wall": 18, "hook": 0 }, { "condition": "/init CLAUDE.md", "full": "9/15", "functional": "12/15", "code": "57/60", "structure": "21/21", "tribal": "10/15", "broken": "13/15", "input": 82262, "wall": 19, "hook": 0 }, { "condition": "Curated, 11 lines", "full": "15/15", "functional": "15/15", "code": "60/60", "structure": "21/21", "tribal": "15/15", "broken": "0/15", "input": 65045, "wall": 21.9, "hook": 0 }, { "condition": "Raw notes, 60 lines", "full": "14/15", "functional": "14/15", "code": "60/60", "structure": "21/21", "tribal": "15/15", "broken": "0/15", "input": 76963, "wall": 26.8, "hook": 0 }, { "condition": "Dreamed notes", "full": "15/15", "functional": "15/15", "code": "60/60", "structure": "21/21", "tribal": "15/15", "broken": "0/15", "input": 68218, "wall": 27.2, "hook": 0 }, { "condition": "Handbook, 210 lines", "full": "15/15", "functional": "15/15", "code": "60/60", "structure": "21/21", "tribal": "15/15", "broken": "0/15", "input": 85223, "wall": 23.6, "hook": 0 }, { "condition": "Stop hook only", "full": "12/15", "functional": "12/15", "code": "60/60", "structure": "21/21", "tribal": "10/15", "broken": "9/15", "input": 125674, "wall": 27.3, "hook": 5 }, { "condition": "Curated + hook", "full": "15/15", "functional": "15/15", "code": "60/60", "structure": "21/21", "tribal": "15/15", "broken": "0/15", "input": 69224, "wall": 22, "hook": 0 } ] }, { "id": "memory-by-condition-haiku", "title": "Every condition, Claude Haiku 4.5", "columns": [ { "key": "condition", "label": "Memory", "unit": "text" }, { "key": "full", "label": "Full pass", "unit": "text" }, { "key": "functional", "label": "Hidden tests pass", "unit": "text" }, { "key": "code", "label": "Code-visible rules", "unit": "text" }, { "key": "structure", "label": "Folder-visible rules", "unit": "text" }, { "key": "tribal", "label": "Team knowledge", "unit": "text" }, { "key": "broken", "label": "Ran the stale test command", "unit": "text" }, { "key": "input", "label": "Median input tokens", "unit": "tokens" }, { "key": "wall", "label": "Median time (s)", "unit": "seconds" }, { "key": "hook", "label": "Hook blocks", "unit": "count" } ], "rows": [ { "condition": "No memory", "full": "2/10", "functional": "6/10", "code": "40/40", "structure": "12/14", "tribal": "0/10", "broken": "10/10", "input": 301233, "wall": 54.2, "hook": 0 }, { "condition": "/init CLAUDE.md", "full": "2/10", "functional": "7/10", "code": "38/40", "structure": "14/14", "tribal": "1/10", "broken": "10/10", "input": 323519, "wall": 52.9, "hook": 0 }, { "condition": "Curated, 11 lines", "full": "7/10", "functional": "9/10", "code": "40/40", "structure": "14/14", "tribal": "8/10", "broken": "0/10", "input": 284212, "wall": 51.7, "hook": 0 }, { "condition": "Raw notes, 60 lines", "full": "6/10", "functional": "10/10", "code": "40/40", "structure": "14/14", "tribal": "6/10", "broken": "10/10", "input": 285263, "wall": 51, "hook": 0 }, { "condition": "Dreamed notes", "full": "7/10", "functional": "9/10", "code": "40/40", "structure": "14/14", "tribal": "8/10", "broken": "1/10", "input": 267405, "wall": 51.8, "hook": 0 }, { "condition": "Handbook, 210 lines", "full": "3/10", "functional": "10/10", "code": "40/40", "structure": "14/14", "tribal": "3/10", "broken": "0/10", "input": 313796, "wall": 49.9, "hook": 0 }, { "condition": "Stop hook only", "full": "8/10", "functional": "8/10", "code": "40/40", "structure": "14/14", "tribal": "8/10", "broken": "10/10", "input": 530935, "wall": 68.5, "hook": 10 }, { "condition": "Curated + hook", "full": "9/10", "functional": "9/10", "code": "40/40", "structure": "14/14", "tribal": "10/10", "broken": "0/10", "input": 340491, "wall": 52.8, "hook": 3 } ] }, { "id": "memory-raw-notes", "title": "The 56 raw notes, labelled by what a consolidation pass should do with each (paths in the test repository shown under tally/)", "columns": [ { "key": "date", "label": "Saved", "unit": "text" }, { "key": "note", "label": "Note", "unit": "text" }, { "key": "fate", "label": "Should be", "unit": "text" } ], "rows": [ { "date": "2026-06-12", "note": "Project is `tally`, a small invoicing ledger for the billing service. Plain ESM, no dependencies.", "fate": "kept" }, { "date": "2026-06-12", "note": "Shah prefers small PRs with one change each.", "fate": "kept" }, { "date": "2026-06-14", "note": "Dev server for the billing service ran on port 3001 this afternoon.", "fate": "removed: transient" }, { "date": "2026-06-18", "note": "Run tests with `npm test`.", "fate": "removed: stale" }, { "date": "2026-06-20", "note": "Amounts are dollars as floats. Format them with `toFixed(2)`.", "fate": "removed: stale" }, { "date": "2026-06-20", "note": "Use 2-space indentation and single quotes.", "fate": "kept" }, { "date": "2026-06-22", "note": "Branch `feat/customers-v1` pushed; waiting for review.", "fate": "removed: transient" }, { "date": "2026-06-25", "note": "The time helper is `now()` in `tally/src/clock.js`.", "fate": "removed: stale" }, { "date": "2026-06-27", "note": "Shah is out on Friday; do not merge anything then.", "fate": "removed: transient" }, { "date": "2026-07-01", "note": "Customers live in `tally/src/customers.js`.", "fate": "kept" }, { "date": "2026-07-03", "note": "Throw `TallyError` with a code from `CODES` in `tally/src/errors.js`; the billing service maps codes to HTTP statuses.", "fate": "kept" }, { "date": "2026-07-03", "note": "Error codes are public API. Do not rename an existing code.", "fate": "kept" }, { "date": "2026-07-05", "note": "Prefer named exports.", "fate": "kept" }, { "date": "2026-07-08", "note": "Used jq to inspect the payment fixtures; worked fine.", "fate": "removed: transient" }, { "date": "2026-07-10", "note": "Do not use `console.log` in `tally/src/`; use `log()` from `tally/src/log.js`.", "fate": "kept" }, { "date": "2026-07-12", "note": "Staging database was reset today.", "fate": "removed: transient" }, { "date": "2026-07-14", "note": "Late fee is 2% of the unpaid balance.", "fate": "removed: stale" }, { "date": "2026-07-15", "note": "PR #212 (partial payments) merged.", "fate": "removed: transient" }, { "date": "2026-07-18", "note": "The report module is new; Shah wants it in the next release.", "fate": "kept" }, { "date": "2026-07-22", "note": "Shah wants a line under `## Unreleased` in CHANGELOG.md for every user-visible change.", "fate": "kept" }, { "date": "2026-07-22", "note": "Keep CHANGELOG entries to one line.", "fate": "kept" }, { "date": "2026-07-25", "note": "`slugify()` is used for export file names.", "fate": "kept" }, { "date": "2026-07-29", "note": "Remember: no `console.log` in src, use the log helper.", "fate": "merged: repeat" }, { "date": "2026-08-01", "note": "Released 1.3.0 (monthly report, partial payments).", "fate": "removed: transient" }, { "date": "2026-08-03", "note": "CI was slow today because of a runner outage.", "fate": "removed: transient" }, { "date": "2026-08-05", "note": "Incident: someone edited migration 0002 and production schemas drifted. Migrations are append-only now: add a new `tally/migrations/NNNN_.json`, never edit an old one.", "fate": "kept" }, { "date": "2026-08-07", "note": "Store loads the schema from `tally/migrations/*.json` in file-name order.", "fate": "kept" }, { "date": "2026-08-09", "note": "Shah prefers early returns over nested ifs.", "fate": "kept" }, { "date": "2026-08-12", "note": "Migrated all money to integer cents (`amountCents`). Parse input with `parseAmount()`, print with `formatCents()`. No floats for money.", "fate": "kept" }, { "date": "2026-08-12", "note": "`toFixed` caused rounding bugs in the old float code.", "fate": "kept" }, { "date": "2026-08-14", "note": "Reminder: throw TallyError, not Error.", "fate": "merged: repeat" }, { "date": "2026-08-16", "note": "Coffee machine on floor 3 is broken (Shah mentioned it).", "fate": "removed: transient" }, { "date": "2026-08-19", "note": "CI flaky again; reran twice.", "fate": "removed: transient" }, { "date": "2026-08-22", "note": "Payment amounts must not exceed the open balance (`E_OVERPAYMENT`).", "fate": "kept" }, { "date": "2026-08-24", "note": "Branch `fix/report-totals` pushed.", "fate": "removed: transient" }, { "date": "2026-08-27", "note": "Integer cents everywhere. Do not divide by 100 except when formatting.", "fate": "merged: repeat" }, { "date": "2026-08-30", "note": "Renamed the time helper: use `nowIso()` from `tally/src/time.js`. Tests inject a clock with `setClock()`. Do not call `new Date()` or `Date.now()` in src.", "fate": "kept" }, { "date": "2026-09-02", "note": "Throw TallyError with a CODES entry; add a new code when none fits.", "fate": "kept" }, { "date": "2026-09-04", "note": "Shah likes tests next to each feature change.", "fate": "kept" }, { "date": "2026-09-06", "note": "Looked at Stripe's refund API for ideas; not needed yet.", "fate": "removed: transient" }, { "date": "2026-09-08", "note": "Port 3001 is taken by another service now; billing dev server moved to 3002.", "fate": "removed: transient" }, { "date": "2026-09-10", "note": "Lost a fix because I edited `tally/src/report.gen.js` directly; it is generated. Edit `tally/templates/report.tmpl.js` and run `node tally/scripts/gen-report.mjs`.", "fate": "kept" }, { "date": "2026-09-11", "note": "The report template uses a `__CURRENCY__` placeholder.", "fate": "kept" }, { "date": "2026-09-13", "note": "Shah asked for clearer error messages in validation.", "fate": "kept" }, { "date": "2026-09-15", "note": "Finance changed the late fee to 1.25% of the unpaid balance, effective now. Round half up to the cent.", "fate": "kept" }, { "date": "2026-09-16", "note": "Email validation added to customers.", "fate": "kept" }, { "date": "2026-09-18", "note": "Reminder: changelog line for every user-visible change.", "fate": "merged: repeat" }, { "date": "2026-09-20", "note": "Released 1.4.0 (integer cents, email validation).", "fate": "removed: transient" }, { "date": "2026-09-21", "note": "`node --test test/` fails on Node 25 (it treats the folder as a file). Run `node --test` with no path. `npm test` uses the broken form.", "fate": "kept" }, { "date": "2026-09-23", "note": "Prefer `Object.freeze` for constant maps.", "fate": "kept" }, { "date": "2026-09-24", "note": "Do not edit migration files that already exist.", "fate": "merged: repeat" }, { "date": "2026-09-26", "note": "Shah reviews PRs in the morning.", "fate": "kept" }, { "date": "2026-09-28", "note": "Use the log helper for audit events (payments, refunds).", "fate": "kept" }, { "date": "2026-09-30", "note": "The billing service reads `CODES` at start-up; new codes need no other change.", "fate": "kept" }, { "date": "2026-10-01", "note": "Staging is on the new cluster.", "fate": "removed: transient" }, { "date": "2026-10-02", "note": "Generated file again: never hand-edit `tally/src/report.gen.js`.", "fate": "merged: repeat" } ] }, { "id": "memory-by-task", "title": "Full passes per task (Claude Sonnet 5.5, of 3)", "columns": [ { "key": "task", "label": "Task", "unit": "text" }, { "key": "none", "label": "No memory", "unit": "count" }, { "key": "init", "label": "/init CLAUDE.md", "unit": "count" }, { "key": "curated", "label": "Curated, 11 lines", "unit": "count" }, { "key": "raw", "label": "Raw notes, 60 lines", "unit": "count" }, { "key": "dreamed", "label": "Dreamed notes", "unit": "count" }, { "key": "bloated", "label": "Handbook, 210 lines", "unit": "count" }, { "key": "hook", "label": "Stop hook only", "unit": "count" }, { "key": "curated-hook", "label": "Curated + hook", "unit": "count" } ], "rows": [ { "task": "Refunds", "none": 3, "init": 3, "curated": 3, "raw": 2, "dreamed": 3, "bloated": 3, "hook": 3, "curated-hook": 3 }, { "task": "Report decimals", "none": 0, "init": 0, "curated": 3, "raw": 3, "dreamed": 3, "bloated": 3, "hook": 3, "curated-hook": 3 }, { "task": "Customer phone", "none": 3, "init": 3, "curated": 3, "raw": 3, "dreamed": 3, "bloated": 3, "hook": 3, "curated-hook": 3 }, { "task": "Late fee", "none": 0, "init": 0, "curated": 3, "raw": 3, "dreamed": 3, "bloated": 3, "hook": 0, "curated-hook": 3 }, { "task": "Slugify (control)", "none": 3, "init": 3, "curated": 3, "raw": 3, "dreamed": 3, "bloated": 3, "hook": 3, "curated-hook": 3 } ] } ], "related": [ "caching-consistency", "cli-model-latency-tokens" ], "hero": { "statIds": [ "memory-team-knowledge-none", "memory-team-knowledge-curated" ] } }, "sources": [ { "id": "agent-memory-study", "title": "Agent memory study: 8 kinds of project memory on Claude Code", "kind": "run", "date": "2026-10-06", "note": "One small Node.js repository, 5 tasks, 8 memory conditions (none, /init, curated, raw notes, dreamed notes, long handbook, Stop hook, curated + hook). Claude Code 2.1.286 headless: Sonnet lane 3 repetitions, Haiku lane 2. Hidden tests and deterministic convention checks; protocol declared before the first session; every attempt kept. The exact memory files and three dreaming passes are published.", "data": [ "/benchmarks/raw/agent-memory/attempts-sonnet.json", "/benchmarks/raw/agent-memory/attempts-haiku.json", "/benchmarks/raw/agent-memory/dreams.json", "/benchmarks/raw/agent-memory/memory-files.json" ] } ] }