A coding agent is more than the language model at its center. It also includes the system prompt, tools and context-management rules that guide the model through a repository. A new study finds that changing this surrounding harness can alter inference costs dramatically, while often leaving benchmark success rates indistinguishable from ordinary run-to-run variation.
A controlled comparison
In “What Does a Harness Buy? Tokens, Mostly,” Shandong University researchers Yangze Liu and Zhongyi Han ask a specific question: if the model and coding tasks stay fixed, how much does the agent harness itself change performance? The preprint was submitted to arXiv on October 3, 2026. The Decoder covered it on October 24; VentureBeat published its account on October 23. The work is a research preprint, not a peer-reviewed journal result. The Decoder and VentureBeat both reported the central cost-versus-score finding.
A harness is the operating layer around a model: its instructions, available tools, handling of returned output and approach to a growing conversation. Liu and Han compared three production harnesses—Claude Code, mini-SWE-agent and OpenCode. The first two are at opposite ends of the study’s description of harness weight: Claude Code supplies an expansive toolset and context-management features, while mini-SWE-agent offers Bash and comparatively little additional machinery. OpenCode is another full-featured agent environment. These are configurations under test, not a universal ranking of the products.
The researchers ran five models in all three harnesses: Qwen3.6-35B-A3B, Qwen3.8-27B, DeepSeek-V4-Flash, GLM-5.3-Flash and HY4-Preview. They also ran Claude Opus 5 inside Claude Code as a reference point, not as a model compared across all three harnesses. The benchmark was SWE-bench Verified, a set of software issues whose proposed fixes can be checked against tests. Eight tasks requiring a public web service were excluded, leaving 492. The hardest 45 formed one set; the remaining 447 made up a larger pool, on which the researchers tested the two Qwen models.
What changed—and what did not
The central methodological move was to compare harness swaps with repeated runs of an unchanged setup. Coding-agent outcomes can vary: a model may make a different sequence of edits or fail at a different point even when nothing in the configuration changes. Earlier one-run comparisons cannot cleanly assign every score difference to the harness. Liu and Han conducted reruns to estimate that background variation rather than treating the first observed score as definitive.
On the 447-task pool, Claude Code and mini-SWE-agent were equivalent within a five-percentage-point margin for both Qwen models, using the study’s statistical equivalence test. That does not mean their scores were exactly identical, nor does equivalence prove that no smaller difference exists. It means the results fit within the specified band under the test. The reported intervals for Claude Code minus mini-SWE-agent were −4.7 to +1.1 points for Qwen3.6-35B-A3B and −3.7 to +1.0 for Qwen3.8-27B. The authors’ conclusion is deliberately bounded to those models, tasks and margin.
On the 45 hardest tasks, the researchers found that swapping harnesses changed task-level pass/fail outcomes at a median rate of 13 percent. Rerunning the same harness also produced a median 13 percent flip rate. In other words, harness changes altered which individual tasks passed, but the number of flips did not exceed the observed noise from another run. The authors additionally report that a harness’s apparent wins did not consistently recur on the same tasks. This is a finding about these paired trials; it is not evidence that harness engineering can never improve an agent.
One exception stands out. OpenCode lagged by as much as nine percentage points on the large pool, the only reported harness effect that cleared the study’s noise threshold. The paper links roughly half of one Qwen3.8 gap to an output cap: OpenCode could stop a response at 32,000 output tokens, including in the middle of reasoning, and such trials often failed. Claude Code encountered the cap too, but could prompt the model to continue. That explanation is partial, not a complete account of OpenCode’s deficit.
Different harnesses can fail in different ways
The detailed failure analysis adds important context to a single pass-rate figure. The authors report that some mini-SWE-agent runs ran into context limits because it lacks the same kind of context management as its competitors. In one OpenCode configuration, its search utility needed an offline setup adjustment, which the researchers made for the reported arm. Time limits were also not perfectly matched in some locally served model runs. The paper explicitly flags this asymmetry as an uncontrolled factor, so the measured gap cannot be read as a clean causal verdict on OpenCode’s design alone.
The token bill starts before the work
The clearest difference was cost. Holding the model, tasks and price list constant, the per-task cost varied by up to three times among harnesses. The researchers attribute much of the gap to the fixed system prompt and tool schemas sent at the start of an interaction and repeated with each model step. Their measured initial context was about 16,581 tokens for Claude Code, 7,025 for OpenCode and 829 for mini-SWE-agent. Since agents make multiple calls, a large preamble is paid repeatedly in token use, even when cached input receives a lower unit price.
The effect depends on how many steps a model takes. In the paper’s GLM-5.3-Flash comparison, a trial cost 1.81 Chinese yuan in Claude Code and 0.56 yuan in OpenCode, while mini-SWE-agent’s cost sat between them. OpenCode’s lower price should not be mistaken for a free efficiency gain: it also stopped earlier in some runs and achieved worse results in the affected large-pool comparison. On DeepSeek-V4-Flash, by contrast, mini-SWE-agent used more calls than Claude Code and cost slightly more—about 15.2 cents versus 14.6 cents per trial. There is no single harness price order that applies regardless of model and behavior.
These are estimates computed from token counts and vendor list prices for API-served models, not a universal invoice. The paper’s locally hosted Qwen runs have token counts but no comparable per-token bill, and Claude Opus 5 was used through a subscription, so its cost figures are not actual per-token charges. Cached-input pricing reduced costs but did not reorder the harnesses within the authors’ pricing analysis. The data therefore make a strong case that the prompt and tool surface affect token consumption, while actual spending will still depend on model, provider rates, caching and usage pattern.
Where the result has limits
The benchmark’s sample size affects what it can establish. In the authors’ power analysis, 45 tasks could detect a 13-point difference only about half the time at the observed task-level discordance; at that sample size, no difference reached 80 percent power. The larger set could resolve differences around five points. That leaves smaller advertised gains—such as a vendor’s claimed 2.4-point improvement—outside what these tests can confidently sort from noise. The study does not disprove every such claim; it demonstrates why a small, single-run score gap is weak evidence.
There is also a benchmark-integrity lesson. In early experiments with web access, the researchers found that 312 of 439 DeepSeek and Qwen trials had retrieved the upstream fix from GitHub. They discarded those results and blocked network access for the final evaluation. That is a research design decision, not an accusation that a model was trained on the benchmark answers. The incident illustrates how agents with unrestricted browsing can retrieve task-specific solutions during evaluation, changing what a benchmark score measures.
Finally, three harnesses and one benchmark cannot represent every coding workflow. SWE-bench Verified tests repository issue resolution in an automated setup; it does not measure all dimensions developers might care about, such as ease of supervision, reliability across private codebases, security, user experience or setup friction. The models and harness versions are pinned to the study’s runs. Products evolve, and the authors do not establish that later releases will behave the same way. The paper’s conclusion should be read as an empirical snapshot, not an enduring law of agent design.
What benchmark readers should take away
The practical message is not “simpler is always better.” Rather, extra harness machinery needs to be evaluated against both its task outcomes and its repeated token overhead. When a proposed harness update claims a modest accuracy gain, a convincing test should rerun a baseline, pair results on the same tasks and report uncertainty—not just publish the best score from one attempt. A cost comparison should likewise disclose prompt sizes, number of calls, cache assumptions and whether failed or cut-short attempts are counted.
Liu and Han’s study offers a concrete reason to demand that fuller accounting. On their test, heavier scaffolding did not deliver a robust pass-rate advantage over a minimal setup within the tested five-point band, but the fixed instructions and tool definitions could raise token costs substantially. OpenCode’s observed underperformance also cautions against turning the result into a blanket endorsement of minimalism: harness-specific stopping rules, runtime constraints and recovery behavior can matter, sometimes in ways that confound the comparison.
For developers choosing an agent, the study suggests measuring the task mix they actually care about and tracking cost per completed task, not simply cost per attempt. For readers of vendor benchmarks, its deeper contribution is a method: measure run-to-run variation alongside harness differences. A leaderboard that blends models and scaffolds without repeated trials may conflate model capability, harness choices and random outcomes. The paper does not settle the broader question of how best to build coding agents. It makes the narrower question—what did this layer buy, and at what token cost—harder to ignore.
FAQ
What is a coding-agent harness?
It is the software and configuration around a language model that lets it act on a code repository, including its system instructions, tools, context handling and rules for continuing or ending a task.
Did every harness have the same pass rate?
No. The paper found Claude Code and mini-SWE-agent equivalent within a five-point margin on the 447-task pool for two Qwen models. OpenCode trailed by up to nine points in that pool. On the 45 hardest tasks, harness swaps and unchanged reruns produced similar task-flip rates.
Why could costs differ so much?
Each model step can include the harness’s system prompt and tool descriptions again. A larger fixed prompt repeated over many calls can accumulate substantial input-token use. The number of steps, model pricing and cache treatment also affect the final bill.
Does the paper prove lightweight agents are better?
No. It reports limited comparisons on SWE-bench Verified under the studied configurations. A minimal harness was not uniformly cheaper for every model, and the OpenCode result shows that output caps and runtime behavior can also alter both cost and success.
Is the paper peer reviewed?
The cited version is an arXiv preprint submitted October 3, 2026. The sources checked here do not identify it as a peer-reviewed publication.