Towards Standardization of the Agent Harness

The past few years have moved language models from answering questions to acting: reading files, running commands, browsing the web, and editing code. These agentic systems are evaluated on a growing number of benchmarks, from software engineering to web navigation to general assistance. Yet the field has paid far more attention to the models and the benchmarks than to the thin layer of software that connects them. That layer is the agent harness.

An agent harness is the program that runs a model against a benchmark. It sets up the environment, exposes the tools, drives the observation loop, and decides what counts as a pass. It is, in effect, the referee of every agent evaluation. And unlike models and benchmarks, it has no shared standard. Every research group builds its own, and the differences are not cosmetic.

This paper makes a simple argument: the agent harness is the least standardized and least reported component of the evaluation stack, and it is the one most likely to make otherwise comparable results diverge. We propose a minimal standard for agent harnesses, define its core components, specify a common execution interface, and describe a reporting format designed for reproducibility. We ground the proposal in a case study on SWE-bench Mobile, where two plausible harness configurations produce a 2% and a 12% success rate on the same task set [3]. Our aim is not to dictate a single implementation, but to give the community a shared contract that makes scores comparable, reproducible, and trustworthy.

The paper is organized as follows. Section 2 defines what an agent harness is and the five jobs it performs. Section 3 describes where harnesses diverge and why that divergence breaks comparisons. Section 4 proposes the standard itself: a set of core components, a common execution interface, and a reporting format. Section 5 presents the SWE-bench Mobile case study that motivates the design. Section 6 situates the proposal against related evaluation infrastructure, and Section 7 concludes with the immediate, low-cost step we recommend the community adopt.

A benchmark for agents is not a single file. It is a contract between a task suite and the software that will run it. The harness is the part that honors that contract. It performs five jobs, and every one of them is a place where results can quietly diverge.

First, the harness instantiates the environment. For a software engineering benchmark this means checking out a repository, installing dependencies, and applying a starting state. For a web benchmark it means launching a browser and a server. Second, it exposes tools to the model: a shell, a file editor, a search index, a browser. The set of tools and their signatures define what the agent is allowed to do.

Third, the harness runs the loop. It sends the model an observation, receives an action, executes it, and returns the result. This loop is where timing and error handling live: how long the agent may think, how long a tool may run, what happens when a tool throws, and whether the agent may retry. Fourth, it manages state across turns, deciding what persists between steps and what is reset. Fifth, it scores the outcome, translating the agent's final state into a pass or a fail.

None of these five jobs has a widely agreed specification. The result is that two groups can report results on the same benchmark and the same model and mean different things. The harness, not the model, is often the difference.

To make this concrete, trace a single task through a harness. The model receives an initial observation, perhaps a repository listing and a failing test report. It responds with an action, say a command to edit a file. The harness runs that command in the environment and returns the new state. The model reads the result and acts again. After some number of steps, the harness stops the agent and grades the final state. Every step in that description is underspecified: what the initial observation contains, how long the model may take, what happens when the command fails, and how the final state is judged. The harness is the collection of decisions that fill in those blanks, and there is no agreement about what the right defaults are.

Fragmentation shows up in small, mundane choices that are easy to overlook and hard to compare. Consider how a harness formats an observation. One harness sends the model a full file listing with line numbers; another sends only a diff; a third sends a truncated preview. Each is a different input distribution, and models are sensitive to that distribution. A change that looks like an implementation detail can shift a score by several points.

Timeouts are another axis. A harness that gives an agent thirty seconds per tool call and one that gives five minutes are running different evaluations, even when the task set is identical. Retry policies compound this: some harnesses let a failed tool call be retried indefinitely, others cap it at one. Error handling matters too. A harness that surfaces a raw traceback to the model gives it information that one that collapses errors into a single message withholds.

The scoring rule is the most consequential divergence of all. Pass criteria range from exact output matching, to unit-test execution, to human judgment. On software benchmarks, whether a patch is graded by running the hidden test suite, by a subset of tests, or by a text-similarity heuristic changes the difficulty of the task substantially [2]. When the scoring rule differs, the leaderboard is not comparing agents; it is comparing graders.

The cumulative effect is that reported scores are not portable. A number published by one group cannot be reproduced by another without the harness, and the harness is rarely published in enough detail to rebuild. This is not a failure of any single group. It is a structural gap in how the field reports agent evaluation, and it is the gap this paper addresses.

The consequences go beyond inconvenience. When harnesses differ, a leaderboard can rank models by the harness rather than by the agent. Two groups comparing the same pair of models might reach opposite conclusions if their harnesses penalize different behaviors. A model that is robust to truncated observations will look strong under one harness and ordinary under another. This is not a hypothetical concern; it is the natural result of treating the harness as an invisible constant when it is, in fact, a variable.

The problem is compounded by the fact that harness differences are rarely visible in a paper. The prose may say that a model was given access to a shell and a file editor, but it will not say how observations were truncated, how long a step could run, or how a partial edit was graded. These details are the substance of the evaluation, and their absence makes the reported number unverifiable. The fix is not more careful prose; it is a structured way to record the harness so that it travels with the result.

We propose a standard that is deliberately minimal. It does not tell anyone which framework to use or which tools to expose. It defines the few things every harness must make explicit so that results can be compared. We organize the standard around three parts: a fixed set of core components, a common execution interface, and a reporting format.

To show how much the harness matters, we ran a controlled experiment on SWE-bench Mobile, a benchmark of real mobile app issues. We held the model and the task set fixed and varied only the harness configuration [3]. The two configurations were both reasonable choices, the kind a group would make without thinking hard about it.

The first configuration used a permissive setup: generous tool timeouts, unlimited retries on tool errors, and full error output returned to the model. The second used a stricter setup: short timeouts, a single retry, and collapsed error messages. On the same tasks and the same model, the permissive harness achieved a 12% success rate while the strict harness achieved 2% [3]. A sixfold gap, produced by configuration choices that most papers never report.

The mechanism is not mysterious. The stricter harness cut off agents mid-edit when a build command ran long, and it hid the traceback that would have told the agent what to fix. Neither behavior is wrong on its own; both are defensible engineering decisions. But they change the evaluation, and without a manifest the reader cannot know which evaluation was run.

This single example is not evidence that every benchmark behaves this way. It is evidence that the harness is a real variable, large enough to dominate the differences between many models. A field that reports scores without reporting the harness is reporting an uncontrolled experiment.

The case study is deliberately narrow, and its limits are worth stating plainly. It uses one model, one benchmark, and two configurations, so it cannot tell us how large harness effects are in general or which benchmarks are most sensitive. What it can do is demonstrate that the effect exists, that it is large, and that it is produced by choices a paper would normally leave out. If the field adopted the manifest we propose, this experiment would be the first line of a reproducibility audit rather than a one-off observation.

A reader might reasonably ask which configuration is correct. Our answer is that neither is wrong, and that is precisely the point. The strict configuration is a defensible choice for a group that wants to bound compute costs, and the permissive configuration is defensible for a group that wants to give the agent every chance. Both groups are entitled to their settings. They are not entitled to report the result as if the settings did not matter. The manifest does not choose a default for them; it makes the choice visible.

The most direct antecedent to this paper is the call for shared evaluation infrastructure for agents. Kapoor and colleagues argue that the missing piece in agent evaluation is not better models or better benchmarks but better infrastructure: a common way to run, track, and compare agents so that results accumulate rather than fragment[1]. Their proposal for a holistic agent leaderboard is complementary to ours. Where they focus on the platform that hosts and aggregates evaluations, we focus on the harness contract that makes those evaluations comparable in the first place. A leaderboard is only as trustworthy as the harnesses feeding it, and a shared harness standard is the foundation a holistic leaderboard needs.

On the benchmark side, SWE-bench established the template for real-world software engineering evaluation, and its design choices around grading are precisely the kind of thing a harness must make explicit [2]. AgentBench and GAIA broadened evaluation to general agentic behavior across environments [4] [5]. Each of these benchmarks ships with its own harness conventions, and none of them is interchangeable with another. That is not a criticism of the benchmarks; it is the motivation for a common layer above them.

On the engineering side, the ecosystem of agent frameworks has grown quickly, from general tooling for language-model applications to dedicated agent libraries [8] [6]. These frameworks make it easy to stand up a harness, which is a benefit, but they also make it easy to stand up many different harnesses with no shared contract. Evaluation tooling has begun to address this, with libraries that standardize how models are scored [7]. Our proposal sits between the frameworks and the benchmarks: a thin, framework-agnostic contract that any harness can implement.

The impulse to standardize is not new, and adjacent fields offer both a warning and a template. Software benchmarking learned long ago that a benchmark is only as meaningful as the harness that runs it, and reproducibility initiatives in machine learning have pushed for structured reporting of experimental conditions. Our proposal draws on that tradition but narrows it to the agent setting, where the harness is unusually consequential because it directly shapes the model's input and its opportunities to act. We believe the agent harness deserves the same kind of structured attention that model cards and dataset documentation have already received.

Agent evaluation is only as trustworthy as the harness that runs it, and the harness is currently the least standardized part of the stack. We have argued that this is a fixable problem, and we have proposed a minimal standard: five core components, a three-operation execution interface, and a machine-readable reporting manifest. The case study on SWE-bench Mobile shows why the standard matters, with a sixfold difference in measured performance coming from harness configuration alone.

We do not expect a single standard to be adopted overnight, and we are not proposing one that competes with existing frameworks. We are proposing a contract that frameworks can implement and papers can report. The immediate, low-cost step is for every published result to include a harness manifest. If the field adopts that one habit, agent scores will become comparable, reproducible, and trustworthy, and the leaderboards built on them will finally mean what they claim.

Several questions remain open and are natural directions for future work. We have not specified the exact schema for the manifest, and we expect it to evolve as groups adopt it. We have not measured how sensitive each major benchmark is to harness configuration, which would help the community know where the standard matters most. And we have not addressed the incentives: a group with a strong result has little reason to volunteer the settings that produced it. Addressing that last point, whether through community norms, reviewer requirements, or platform policy, may matter more than any technical detail. The standard gives the field the vocabulary it needs; building the habit is the work that remains.