The Harness Tax: Benchmarking Agents on the Exact Same Model

2026-09-03

by Uri Walevski

Most popular agent frameworks like Pi, OpenClaw, and Hermes share an unstated assumption. They assume an agent is a long-running process in a private VM or terminal, running shell commands, modifying local file trees, and talking to a single developer.

The prompt2bot harness was built for a different reality.

It does not assume a virtual machine. It runs directly inside real communication channels like WhatsApp, Telegram, and team group chats. It natively understands messaging primitives that terminal agents never have to think about, such as quoted replies, emoji reactions for approvals, voice memos, and multi-user participation where three different people might speak in the same thread.

It is provider-agnostic. The exact same typed loop drives Gemini, Claude, and Kimi without framework lock-in. Because it operates in live production messaging where every turn counts, token efficiency is not an afterthought. It uses a three-tier compaction system: decaying historical summaries, scratchpad spills for large tool outputs, and atomic tool-call pairing that preserves cache prefixes instead of blowing up context windows.

When people benchmark agents, they almost always compare models. You see Claude against GPT against Gemini. The harness is treated as invisible plumbing, as if every framework calling Gemini 3.8 Flash gets the same result at the same cost.

That assumption is wrong. The harness is not neutral plumbing. The harness defines the tool schemas, controls the system prompt, formats the history, and determines how much conversational filler the model generates.

To see what a harness actually costs, you have to hold the model constant.

The Head-to-Head Test

We ran @earendil-works/pi-coding-agent and our harness on identical task repositories, using the exact same model, gemini-3.8-flash, on both sides.

Pi is one of the cleanest terminal agent implementations in the ecosystem. It has proper tool calling, git integration, and a dedicated evals package. If any framework represents the modern standard for coding agents, it is Pi.

Here is what happened when both ran on a multi-file repository bug (t2-config-loader, testing configuration precedence and environment variable parsing):

Metric	prompt2bot harness	Pi (pi-coding-agent)	Delta
Model	gemini-3.8-flash	gemini-3.8-flash	Identical
Test Result	Passed (2/2 tests)	Passed (2/2 tests)	Both solved
Total LLM Calls	16	36	55.5% fewer calls
Output Tokens	944	8,711	89.2% fewer output tokens
Input Tokens	113,067	240,044	52.9% fewer input tokens
Total Tokens	120,857	248,755	51.4% fewer total tokens

Both harnesses solved the bug. Both ran the test suite and confirmed it was green before stopping.

Yet Pi burned over 248,000 tokens to do it. Our harness did it in 120,000. We cut the total token bill in half on the exact same model.

On a separate algorithmic discount bug (t1-bugfix-discount), the pattern repeated. Pi solved the task in 29 calls and 7,600 output tokens. Our harness solved it in 20 calls and 1,239 output tokens, using 83% fewer completion tokens.

Where does this massive difference come from?

The Startup Tax

The first leak happens before the user even finishes typing.

On Turn 1, Pi injected 4,239 tokens into the context window for a single hello. It loaded its default instructions, session state conventions, extension mechanisms, and tool declarations into prompt space right out of the gate.

Our harness started with ~800 tokens.

When every turn appends to history, a 4,000-token starting penalty compounds across every subsequent step. By Turn 20, that initial overhead has been re-read and billed twenty times over.

The Output Verbosity Trap

The second leak is completion bloat.

Output tokens are roughly four times more expensive than input tokens, and they are the primary driver of end-to-end latency. In both tasks, Pi generated between 7,600 and 8,700 output tokens. It narrated its thought process at length, repeated file contents in markdown blocks, and added conversational pleasantries between tool steps.

Our harness emitted 944 output tokens on the config task and 1,239 on the discount task.

This is not prompt engineering telling the model to "be concise." Telling a model to be concise in a prompt rarely works for long trajectories. It comes from the schema design. When tools return structured data and the event loop cleanly separates internal execution from outward communication, the model stops treating every tool turn like an essay. It calls the tool, inspects the result, and moves on.

Cache Stability and the Conversational Benchmark

We observed the same dynamic on τ-bench, the industry benchmark for multi-turn conversational tool agents.

Testing identical customer service exchange flows on the same model, the standard reference harness emitted 3,406 completion tokens of conversational padding. Our harness completed the exact same database updates and customer notifications in 1,137 completion tokens, a 66% reduction in generated text.

Cache hit rate showed an even larger gap.

Because our harness preserves deterministic history prefixes, Google GenAI hit a 55.2% prompt cache rate across multi-turn sessions. The reference harness mutated prompt structures and timestamps between turns, dropping its cache hit rate to 0%. Even when input counts look similar on paper, a 55% cache hit rate drops real API bills significantly.

The Limit-Cycle Basin

There is another failure mode that benchmarks rarely talk about: the infinite tool loop.

During initial tests with Gemini thinking models, we hit a subtle bug in our test adapter. The adapter was returning tool results without properly appending the model's preceding tool call and its thoughtSignature back into the history array.

The model entered what dynamical systems call a trapped limit-cycle basin. It called find_user_id_by_name_zip, got the answer, forgot it had called the tool because the call was missing from its own perspective, and immediately called find_user_id_by_name_zip again. It repeated this 25 times until hitting the maximum turn limit.

Many agent frameworks try to fix this by adding scolding prompts: "Do not repeat tool calls," or "Review your previous steps carefully."

Prompts do not fix structural history corruption. The fix was purely architectural: ensuring the model's emitted events, including provider-specific metadata, are preserved as first-class history events before tool results are fed back. Once the history contract was intact, the loop vanished completely without adding a single instructional token to the prompt.

What a Harness Should Be

There is a growing trend to treat agent harnesses like operating systems. Frameworks keep adding middleware layers, plugin registries, background daemons, and nested hook architectures.

Every layer you add between the model and the actual interface extracts a tax. It costs tokens at startup, it costs tokens during execution, and it creates failure modes that prompt engineers spend weeks trying to patch with more words.

An agent harness does not need to be an operating system. It needs to be a lean, typed bridge. If you build the plumbing correctly, you do not need to wait for the next frontier model to cut your token bill in half. You can do it right now by fixing the runtime.

← All posts