The Harness Tax

Mon, 09 Nov 2026 03:00:00 +0000

Updated Nov 10, 2026: the changes since launch are listed at the end.

We ran Claude Code and pi, a minimal open-source coding agent, on the same model and the same six coding jobs. pi’s median run cost 29% less and was a bit faster. Claude Code’s code graded slightly better. None of these gaps met the bar we set before running.

The two differ only in the harness: the system prompt, the tools and the loop around the model. On a typical run, Claude Code’s first model call sends about 27k tokens, and pi’s sends about 3k. We call the cost of resending a harness’s setup text on every step the harness tax. Claude Code’s is far bigger than pi’s. A prompt cache cuts the price of rereading that text to a twentieth. But an agent takes many steps. We ran each harness three times on every task, to see how big the tax is. Three of the six tasks are real bug fixes from open-source projects. The other three are synthetic: a Claude agent wrote them for this study.

Evals on Swapnil Sharma

The Harness Tax