The Harness Tax
We gave the same model the same coding jobs inside two agent harnesses, Claude Code and pi, and metered every run. We call the cost of resending a harness's setup text on every step the harness tax. Claude Code's is far bigger than pi's.
Two harnesses, one model
A coding agent is a model plus a harness. The harness is the program around the model. It writes the system prompt and defines the tools, such as running a shell command or editing a file. It runs the loop and decides what the model sees on each step. One step is one model call. We held the model fixed at Claude Opus 5.5 with high reasoning effort, and swapped only the harness. There is one catch. pi 0.73.1 came out before Opus 5.5. So it gave the model a fixed thinking budget, instead of letting the model decide how long to think.
We ran six Python tasks, three times per harness. Three are real bug fixes from open-source projects (markdown-it-py, inflect, boltons), so each has the fix the project merged. The other three are synthetic: a Claude agent wrote them for this study.
Claude Code is Anthropic's own agent. It ships a long system prompt and a big tool kit: a todo list, sub-agents, web fetch, careful rules about git. pi, by Mario Zechner, goes the other way. Its prompt fits on a page and it has four tools. We measured how much text each one sends on its first model call, before the model has done any work:
Tokens of context on the first model call, median over each harness's 18 main runs. That counts the system prompt, tool definitions, the task prompt and any instruction file in the repo. The task prompt and repo are the same for both. One repo (markdown-it-py) has a long AGENTS.md that adds about 6k to both sides.
The model remembers nothing between calls. On every step the harness sends the whole conversation again, starting with that first-call context. Prompt caching makes the repeat cheap: a cached reread costs $0.20 per million tokens, a twentieth of a fresh one. But an agent takes many steps: model calls in a typical run here. And the context grows on each one. Resending the setup text on all those steps is the harness tax. The question is how much it adds to the bill.
We are not the first to ask. In April, siddharth (@sambharia) wrote The Harness Tax, which traces where Claude Code's tokens go on a small job. In September, a UC Berkeley and Arena team published HarnessTax, a much bigger study. It ran seven models in three harnesses (Claude Code, Codex CLI and pi) on SWE-bench Lite and Terminal-Bench 2.0. They found that the harness barely changed how often tasks were solved. On SWE-bench Lite, Claude Code cost about twice as much as pi. Our study is far smaller. What it adds is a blind grade of the code itself, not just of whether the tests pass. It also lets you replay every run.
The scorecard
Medians per task for cost and time, mean for the grade. "Solved" means every hidden test passed, including tests that already passed before the change. So a fix that breaks something else does not count. "Grade" is the blind grader's score out of 10. Underline marks the cheaper, faster or higher-graded side. CC is Claude Code.
Watch two runs side by side
Each replay below is a real pair of runs on the same task, one per harness, started together. The meters show money spent so far, computed from each harness's own token log. The tape shows each tool call as it happened, and how the run ended. A few pi calls name tools that don't exist, such as pelican_write; pi answers those with an error. Press Replay, or drag the slider to scrub.
Two things stand out once you watch a few. First, the meters climb in steps, one per model call. The big steps are calls where the model writes a lot. Claude Code's first call is a big step too, when its prompt isn't cached yet. Second, the two runs rarely take the same path, even on the same task. Those choices move the bill too, as the next section shows.
Why the tax should fade on long jobs
The harness tax is a fixed charge on every step, so it adds up in a straight line. But each step also rereads everything said so far, and that part grows with the length of the job. On a short job the tax is a big share of the bill. On a long job the growing conversation and the model's own output take over, and the tax's share shrinks. Drag the slider to see where our runs sit.
A toy model fitted to our main runs. Both harnesses get the same context growth per call. They also get the same output and uncached input per call, averaged over the runs. The only difference is the starting context. Dots are real runs.
The run that sat waiting
Every tick is one tool call; calls a few seconds apart are stacked in one column. The shaded stretch is one shell command that was still running when the limit hit.
What the tests couldn't tell us
After the launch, Gagan (@gagu_sh) suggested two checks that cost less than a model grader. One counts the lines each diff changes outside the code the project's own fix touched. The other is property tests. These throw thousands of random inputs at the code and check that a rule always holds. A Python library, Hypothesis, runs them. The line count needs a project's own fix to compare against, so it ran on markdown-it-py and boltons. The property tests ran on boltons and the template engine. Each covered 12 diffs.
The graders had marked down 5 diffs on those tasks for correctness. 4 of them failed the property tests, and the tests found no bug the graders had missed. The fifth diff caps how deeply a template can nest, and our tests never tried deep nesting. We wrote the tests after reading the graders' notes, so they looked where the graders had looked. What they added was a concrete failing input for each of the four diffs. On the boltons task, for example: write aéa, seek back one character, ask for the length, then read one character. The read comes back empty. They also caught bugs in two of the answer keys: the fix boltons merged, and the solution we wrote for the template engine task. The boltons fix still lets the length call move the read position. Our template engine solution crashes on a huge list index too.
When a property test fails, Hypothesis cuts the input down to the smallest one that still fails. Gagan then suggested keeping each of those small inputs as a plain test. A grader can read it, and any later fix has to pass it. We did, and found one more crash. On two of the boltons diffs, a slightly different input makes that empty read crash instead. The fifth diff gives up at about 3,300 nested parentheses with an error of its own. The task doesn't allow that, and the graders marked it down. We added a test with 3,500 nested parentheses so their note has a concrete input. No real template nests that deep. Our own solution breaks far sooner, at about 140. With these tests, all five diffs the graders marked down fail. But we wrote every test after seeing the failures, so they repeat what we found rather than check it.
We expected diffs that stray far from the project's fix to be riskier. A high count would then be a warning sign. It went the other way, and the count found nothing the graders missed. On both tasks, the diffs that changed the most beyond the project's fix were among those the graders rated best. On markdown-it-py the extra lines also fix link destinations, a case the project's fix misses. On boltons they rewrite the length call so a read right after it no longer loses a character.
Limits
- A small sample: six Python tasks, three runs each per harness. Three are real bug fixes from open-source projects, merged after the model's training cutoff but still public. The other three are synthetic tasks, written by a Claude agent. Each per-task median rests on three runs.
- The replay also has three small pilot tasks, run before the main round. They are not in the scorecard or the numbers above.
- The runs shared one
/tmpfolder. On the markdown-it-py task, four later runs (two per harness) reused a JavaScript library the first run had installed there. So those four may be a little cheap and quick. - Some runs used the network. All six inflect runs read the maintainer's blog page on project layout. One Claude Code run on the markdown-it-py task installed the JavaScript markdown-it library. On the boltons task, one run installed pytest and another pulled it in through uv. Runs on both sides pulled dev tools such as ruff, tox and the mypy type checker through uvx. The logs show mypy downloads in one Claude Code run and two pi runs. No run fetched the project's own fix.
- One model, and a pi version older than it. A different model, or a newer pi, could change the picture.
- The main graders ran on the agents' own model, Opus 5.5, inside Claude Code. The outside re-grades by GPT and Gemini cover only half the diffs. Each of those models graded each task in one call.
- Everything ran on one machine through one API route, at most two runs at a time, with launch order alternated.
How we ran it
Before any paid run of the main round, we wrote down the plan. It set the tasks, the metrics and the bar for calling a winner. One change came after the first two runs, when we had seen only their cost. They cost more than expected. So we changed the rule for how many runs each task gets, to fit the budget, and added a hard stop at $37. Both harnesses ran headless with stock settings and all permissions allowed. Each run got a fresh copy of the repo, the same prompt and a 30-minute limit. Nobody nudged them. We recomputed cost from each harness's own token log with one price table. Per million tokens, input costs $4, output $20, a cache read $0.20 and a cache write $5. Claude Code's own cost report matched ours. pi prices tokens with the table we gave it, and its totals matched too.
Changes since launch
- Nov 9, 2026: GPT-6.1 Sol Pro re-graded the diffs we still had.
- Nov 9: credited earlier work on the same question.
- Nov 9: made the page easier to read. Credited Gagan, who suggested two checks that cost less than a model grader.
- Nov 9: added Gemini 4 Argon as a third grader.
- Nov 10: ran both of Gagan's checks, property tests and a count of lines changed beyond the project's own fix. Results are in What the tests couldn't tell us.
- Nov 10: saved the smallest failing inputs as plain tests, as Gagan suggested next.
- Nov 10: a second review found we had overstated how well the property tests matched the graders. We had also left out that runs shared a prompt cache (see the limits). Both are fixed. We also made the page easier to read and gave the comics real punch lines.