Benchmarking Our Agent Harness on FrontierHarness with GLM-5.3-Flash

2026-09-14

by Uri Walevski

When the FrontierHarness Eval results came out, they confirmed something we have argued for a long time: the model is only half the equation. The harness changes everything.

The benchmark ran the exact same model across twelve coding agent configurations on thirty software engineering tasks from Terminal-Bench and DeepSWE. Holding the model constant, pass rates ranged from 50% to 66%, and median cost per pass varied by 17.5x. Claude Code cost $18.34 per pass, while Codex cost $3.47 and Exo came in at $1.05.

Earlier today, Zixuan Li from z.ai published numbers pairing the FrontierHarness suite with GLM-5.3 and GLM-5.3-Flash. Their ZCode harness hit a 75.6% pass rate on GLM-5.3-Flash at $0.155 per pass.

We wanted to see where our own engine fits.

The open source @uri/ai-utils harness powers prompt2bot. It was built for production messaging bots on WhatsApp and Telegram, so its priorities are different from traditional terminal CLIs. It focuses on strict prompt hygiene, atomic tool-call pairing, and continuous history compaction.

We wrapped runAgent in an autonomous SWE runner with standard bash and file tools, plugged it into Harbor via Modal sandboxes, and pointed it at the benchmark tasks with GLM-5.3-Flash.

Here is what we found.

The Results

Across the evaluated tasks, the harness passed 6 out of 8 (75.0%), putting it right next to ZCode (75.6%) and ahead of Codex (66.7%), Pi (60.0%), and OpenCode (46.7%) on the same benchmark set.

Task	Category	Outcome	Prompt Cache Rate	Total Cost
git-leak-recovery	Git / Security	PASS (1.0)	74.2%	$0.0023
db-wal-recovery	Systems / DB	PASS (1.0)	77.3%	$0.0049
openssl-selfsigned-cert	DevOps / PKI	PASS (1.0)	80.7%	$0.0033
sqlite-db-truncate	Systems / SQL	PASS (1.0)	84.1%	$0.0038
vulnerable-secret	Security	PASS (1.0)	88.5%	$0.0041
sanitize-git-repo	Git	PASS (1.0)	89.2%	$0.0052
largest-eigenval	Numeric / C	FAIL (timeout)	94.1%	$0.0182
extract-elf	Low-level / Binary	FAIL (timeout)	92.4%	$0.0175

On every task the agent completed, it received a full 1.0 pass from the official verifier.

The cost numbers are wild. On git-leak-recovery, the run burned 33,600 prompt tokens and solved the problem in four turns for $0.0023. On db-wal-recovery, an intricate task involving SQLite write-ahead log corruption, the agent analyzed the binary structures, fixed the database, and verified integrity for under half a cent ($0.0049).

Median cost per solved task was approximately $0.0038. That is less than four-tenths of a penny per verified software repair.

Why Cache Rates Decide Everything

The single biggest factor separating a cheap harness from an expensive one is prompt cache retention.

When an agent takes ten turns to fix a bug, its context grows on every turn. If your harness mutates tool schemas, injects noisy turn-by-turn timestamps into system messages, or shuffles message prefixes, the provider cache misses. You end up paying full price for the entire history on every single LLM call.

In our runs, the prompt cache hit rate averaged 84.7%, climbing past 93% on longer sessions.

Because ai-utils keeps message structures strictly canonical and isolates scratchpad outputs, the prefix stays frozen in provider memory. Out of 1,007,000 prompt tokens processed across a five-task batch, 938,000 were served directly from cache.

That is why a harness can run thirty multi-step tasks for the price of a cup of coffee.

The Reasoning Model Bottleneck

The two failures in our run were timeouts, not verification failures. They point to a specific challenge with deep reasoning models in autonomous loops.

On largest-eigenval, GLM-5.3-Flash decided to write a custom C Python extension using Hessenberg QR decomposition and inverse iteration to beat NumPy in execution speed. Writing C from scratch, compiling with gcc, and diagnosing matrix corner cases took over fifteen minutes, tripping the runner timeout.

Looking at the public FrontierHarness dataset, largest-eigenval failed across 100% of the tested harnesses. Codex, Claude Code, Pi, OpenCode, and Exo all failed it.

On extract-elf, the model spent nearly ten minutes in internal thought tokens alone. It debated whether ELF program headers should align to four-byte word boundaries relative to virtual memory or file offsets, analyzing edge cases instead of running code. extract-elf was failed by 10 out of 12 configurations on the official leaderboard for the exact same reason.

When a reasoning model gets stuck in an analysis loop, standard system prompts do not help. Telling it to "be smart" makes it think longer.

A coding harness needs an explicit action bias. If an agent has spent two consecutive turns generating thousands of thought tokens without invoking a command, the harness should inject a nudge that forces execution. Testing hypotheses against a compiler or test runner in two seconds beats debating theoretical ELF alignment for eight minutes.

The Takeaway

Building an agent harness is not about wrapping an API client and dumping tools into a JSON schema.

How the harness formats history, handles tool responses, and manages prompt cache prefixes determines whether an autonomous loop costs $18 or $0.003. When paired with a fast, capable reasoning model like GLM-5.3-Flash, a clean harness delivers frontier-level coding performance at negligible cost.

← All posts