Does letting an AI run code help? A single call vs an agent loop on 8 hard tasks

TL;DR

• Question: does an agent loop that can write and run code pass more hard tasks than one call with tools off? We ran the same 8 hard tasks, with the same strict validators, both ways.
• Answer: not clearly. No model's loop was ahead of its single call, because every pair of 95% intervals overlaps. Claude Haiku 4.5: 11 of 24 as one call, 13 of 24 as a loop (28% to 65% against 35% to 72%). Claude Sonnet 5.5: 24 of 24 and 16 of 16 (86% to 100% against 81% to 100%), a ceiling. GPT-6 Luna in the Codex CLI: 10 of 16 and 12 of 14 (39% to 82% against 60% to 96%).
• The model, not the harness, decided whether to run code. Haiku ran code in 24 of 24 loop attempts. Sonnet did in 3 of 16. GPT-6 Luna did in 2 of 14. For the last two, the "loop" was mostly a single call with tools in reach.
• The loop used more tokens; time ranges overlap. For Haiku, median total tokens went from 9,038 to 78,432 (8.7 times, a calculation). Median time went from 39.0 s to 56.8 s (the ranges overlap). List-price cost per strict pass went from $0.067 to $0.142 (a calculation).
• Code ran on the largest observed task change. Haiku's event-loop prediction went from 0 of 3 to 3 of 3. It wrote the program and ran it. Its strict count fell on the money refactor (2 of 3 to 0 of 3). Three attempts per task is too few to rank tasks.
• No outside edits landed. 56 loop attempts, 0 edits landed outside the work folder. Two attempts read a notes folder outside it and are left out of every rate.

Full study, charts and data: the single call vs agent loop study.

What we tested

A common claim is that an agent harness beats a plain model call. A harness lets the model write code, run it and fix its answer. We wanted a number for that claim.

We used the eight hard tasks of our hard model head-to-head. Each task has a sandboxed validator. The tasks are an interval-merge fix, a DST day-length fix, a CSV parser and an event-loop output prediction. The other four are a room schedule, a strict SemVer regex, a money refactor and a SQL report.

• Single call. The task prompt, tools off, one turn. This is our stand-in for an API call. We used no paid API key, so the CLI with tools off is the "API call" arm.
• Agent loop. The same prompt plus one paragraph: "You may create and run files in the current folder to test your answer. Your final message must be only the answer, in the format stated above." We graded the final message exactly like a single call.
• Six cells. Claude Haiku 4.5 and Claude Sonnet 5.5 in Claude Code, and GPT-6 Luna in the Codex CLI at medium effort. Each model ran both ways. The two Claude single-call cells are reference cells from the hard head-to-head. We reused them and did not rerun them.

Did the loop raise the pass rate? Not clearly

A strict pass needs the whole final message to pass the validator as written. A right answer in the wrong wrapping is a format miss. Examples are a code fence or a sentence before the code. A format miss does not count.

Our rule: a side is ahead only when the 95% Wilson intervals do not overlap. None did here. For Haiku, the gain is 2 attempts out of 24 (+8 points, a calculation). The overlapping intervals do not establish a gain.

The lenient reading counts format misses as right answers. Haiku went from 16 of 24 (47% to 82%) as a single call to 20 of 24 (64% to 93%) as a loop. Those intervals also overlap. Haiku had 8 wrong answers as a single call and 4 as a loop. Its format misses went from 5 to 7. The final message was not "only the answer" in those 7 attempts. Read these as counts from small samples.

Tool use: the model chose whether to run code

The prompt allowed code. It did not require it. The models chose very differently.

• Haiku 4.5 ran at least one tool in 24 of 24 attempts. The median was 3 tool calls (2 to 18). It made 49 shell calls and 45 Write calls; 2 Write calls were refused.
• Sonnet 5.5 ran a tool in 3 of 16 attempts: the DST fix once and the event-loop prediction twice. It answered the other 13 directly.
• GPT-6 Luna ran one shell command in 2 of 14 scored attempts, the event-loop prediction and the room schedule, both in repetition 2.

So the Sonnet and Luna loop cells are mostly single calls that had tools in reach. This matters for Luna in particular. Its pass rate rose from 10 of 16 to 12 of 14, and 12 of the 14 loop attempts ran no code at all. We cannot credit the rise to code. Luna's scored loop covers seven tasks; the single-call cell covers all eight. Both excluded loop attempts were on the DST task. The two Luna routes also differ in other ways. The single call used our runner's route. The loop used codex exec. The CLI prompt and its length differ too.

Where pass counts rose, and where they fell

Each bar rests on 2 or 3 attempts, so every whisker is wide. For Haiku, the 95% Wilson intervals are 0–56% for 0/3, 6–79% for 1/3, 21–94% for 2/3 and 44–100% for 3/3. Every task pair overlaps. Use this chart to find failures, not to rank tasks. For Haiku:

• Up: event-loop prediction 0/3 to 3/3. In all three attempts it wrote the program into a file, ran it with node and copied the output. Running the program turns a prediction into a measurement. This was the largest observed strict-pass change. The intervals still overlap.
• Up: room schedule 0/3 to 2/3, and DST fix 1/3 to 2/3. For the schedule, Haiku wrote a script and ran it in all three attempts.
• Down: money refactor 2/3 to 0/3, interval merge 3/3 to 2/3 and SemVer regex 3/3 to 2/3. Most of these were right answers with prose around them.
• Same: CSV parser 2/3 both ways. SQL report 0/3 both ways, each with 2 right answers in the wrong wrapping.

In one DST attempt, Haiku used 19 turns and 18 tool calls. It returned prose followed by code. Strict grading rejected the leading prose, and lenient extraction found no passing answer. This attempt failed under both readings.

What the loop costs

All ranges below show the smallest and largest scored attempt, not a confidence interval.

Median total time per attempt, single call to loop: Haiku 39.0 s to 56.8 s (1.5 times, a calculation). Sonnet 7.7 s to 7.4 s (about the same). GPT-6 Luna 5.2 s to 9.3 s (1.8 times, a calculation). All three pairs of fastest-to-slowest ranges overlap. The median does not show the full spread. Haiku's slowest loop attempt took 223.7 s. Its 95th percentile went from 68.5 s to 176.3 s (n = 24 each; calculation by linear interpolation).

Tokens are where the loop is clearly different. A loop sends its growing context again on every turn. Median total tokens (input with cache reads, plus output), single call to loop: Haiku 9,038 to 78,432. Sonnet 3,380 to 10,483. GPT-6 Luna 11,954 to 16,058. Part of the rise has nothing to do with running code. Sonnet ran no tool in 13 of 16 attempts, yet its median input rose from 2,281 to 9,550 tokens (ranges 2,234–2,669 and 9,398–33,040). The CLI prompt and tool definitions may explain part of this rise. We did not test the cause. A token count is not good or bad by itself. It is the price of the loop.

At list price, cost per strict pass, single call to loop, is a calculation. It divides the cost of every attempt in the cell, failures included, by its strict passes:

For Haiku, 24 loop attempts cost about $1.85 at list price and 24 single calls about $0.74. The calls ran on flat subscriptions, so none of this is a bill. The costs have no interval, so the Luna change is an untested difference.

Did the sandbox hold?

We ran the loops with write access to one empty folder and no network. Before each CLI's loop batch, we ran one uncounted sandbox probe. Both CLIs blocked the network command, refused a write to the parent folder and created a file in the work folder.

Then we audited all 56 loop transcripts for file edits and for paths in shell commands.

• 0 edits outside the work folder ran. In 2 Haiku attempts the model tried to write a test file under a temp path outside the folder. Claude Code or the sandbox refused every one of those writes, and no file landed.
• 2 attempts are contaminated and left out. GPT-6 Luna searched a notes folder outside the work folder on the DST task, in both repetitions. Both attempts passed. The Codex CLI loads the account's own instruction file, but we did not test whether those instructions caused the search. We kept them in the data and left them out of all scored metrics, including time, tokens and cost. The protocol states this exclusion rule, but its file times do not prove when the rule was written.
• Reference answers were locked. The file that holds the planted answers had no read access while any agent ran.

If a loop reads outside its folder, its score may not be its own work. That is why we audit.

Questions people ask

Does tool use improve LLM accuracy? In this test, not clearly. Haiku gained 2 attempts out of 24 with overlapping intervals. Sonnet was already at 24 of 24 as a single call (95% interval 86–100%). The largest observed task change was on program-output prediction; its task intervals still overlap. It did not fix the final message: the loop raised Haiku's format misses from 5 to 7.

What is the difference between an agent harness and a single call? A single call sends one prompt and takes one reply. A harness adds tools, a loop and a way to read the results. The model can then test and revise. It can add turns, tokens and time. Here, the time ranges overlap.

Will my model use the tools I give it? It depends on the model. Haiku 4.5 used them every time. Sonnet 5.5 and GPT-6 Luna used them rarely and only on some tasks. Check the tool-call count in your own runs.

Is the loop worth the cost? For Haiku here, the loop cost about 2.1 times as much per strict pass (a calculation). The data cannot separate its gain from noise. For Sonnet, which already passed, it cost about 1.9 times as much per strict pass (a calculation). Both Sonnet cells passed every attempt.

What to do

1. Measure pass, time and tokens together. A loop that adds 2 passes and 8.7 times the median tokens (a calculation) is a trade, not a free gain.
2. Use a loop where a check can run. Predicting program output and searching a small space were the two tasks where Haiku moved most. Check your own tasks the same way.
3. Fix the last message. Tell the model to end with only the answer. If you strip prose, report that as lenient grading; keep the strict score. 7 of 24 Haiku loop attempts had the right answer in the wrong wrapping.
4. Check whether the loop adds value when a single call already passes. Sonnet passed every attempt both ways on this set. The loop used more tokens and cost more per pass (calculation). This ceiling does not show that loops have no value on other tasks.
5. Sandbox it and audit the transcripts. Give the loop one folder, no network and a list of the paths it may touch. Then read what it did.

How we measured

• Protocol timing. The file was created at 21:30 UTC on 2026-10-06, before the new calls started at 21:41 UTC. The reused Claude reference calls ran earlier. The file was modified after the runs and includes dated amendments and an outcome log. Its file times do not prove that all current text existed before inference. 40 Claude sessions and 1 probe; 32 Codex calls and 2 probes. Nothing was retried or trimmed, and no usage limit was hit.
• Tasks and grading. The same prompts, validators, strict grading and lenient extractor as the hard head-to-head. Controls ran before the new inference batch. 8 of 8 reference answers pass. 26 of 26 planted wrong answers fail. 8 of 8 wrapped references are flagged as format misses.
• Claude Code loop. Tools: shell, read, edit, write, glob and grep. The Claude Code sandbox was on, with no network and no unsandboxed commands. No MCP servers, no user settings, at most 30 turns and the same 16,000-token output cap per response as the single calls.
• Codex CLI loop. codex exec with the workspace-write sandbox, network off and approvals never. 10 minutes per loop session in both CLIs; single calls had a 300-second timeout.
• No answer is a fail. An error, a timeout or a turn limit counts as a failed attempt. None happened.
• Cost is reported tokens times list price. A calculation.
• Intervals are 95% Wilson intervals for rates and fastest-to-slowest ranges for times, tokens and tool calls. A range is not a confidence interval.

Public receipts: new calls and audit, reference calls, and study data.

Caveats

• Shared host. Other local jobs may affect wall time. These ranges describe the observed runs, not isolated model speed.
• Different hours. The Claude single-call cells ran earlier on 2026-10-06 (03:23 to 04:02 UTC), on the same Claude Code version, tasks and validators. Provider load can differ by hour.
• CLI and model pairs. Claude Code and the Codex CLI add their own system prompts and tool schemas. A gap between Claude and GPT-6 Luna rows is partly the CLI.
• Two Luna routes. The single call and the loop used different Codex routes, so the Luna comparison mixes tools with route.
• Small samples. 2 or 3 attempts per task and cell. Wilson intervals treat attempts as independent trials. Repeated attempts on eight fixed tasks do not measure success across coding work. Read the intervals.
• A ceiling. Sonnet passed everything both ways. This set cannot show whether a loop helps a model that already passes.
• Two attempts left out. GPT-6 Luna has no scored DST result in the loop cell, so that cell has n = 14. Its single-call cell includes both DST attempts. The scored task mix differs. All scored metrics exclude the two contaminated attempts.
• Costs are calculations, not bills.
• Disclosure. I build Agent, a product that runs agent loops, so I have an interest in loops looking good. The result does not show that, and we publish it as measured. The receipts and the method are public.

What to read next

• Hard tasks: Claude Haiku vs Sonnet vs Opus vs Fable
• Harness vs model: where do AI coding agent gains come from?
• Format misses vs wrong answers in LLM evals
• Claude Haiku vs Claude Sonnet: every measured row

See what your agent loop does

Agent records every step of a loop with its model, its cost and its time. Try Agent and compare a single call with a loop on your own work.