Doc2Harness: Benchmarking how well AI builds agent harnesses

Researchers Yuyao Ge, Shenghua Liu, Peizhong Gao, Chendi Ge, Chenxu Liu, Yiwei Wang, Lingrui Mei, Baolong Bi, Jiayu Yao, Jiafeng Guo, and Xueqi Cheng released Doc2Harness on October 14, 2026, introducing a two-stage deterministic benchmark designed to evaluate how large language models build scaffolding and execution harnesses for autonomous coding agents. The benchmark paper, detailed in arxiv.org's report, establishes an evaluation framework that tasks build models with generating an entire candidate agent harness from documentation and interface contracts rather than merely solving isolated downstream tasks within static runtime environments.

This release covers the complete evaluation protocol, containerized testing architecture, and comparative performance metrics across nineteen frontier systems tested on long-horizon software engineering workloads. It is aimed at machine learning engineers, systems architects, and infrastructure teams seeking deterministic methods to measure how artificial intelligence models structure complex software repositories, split modular interfaces, and enforce execution contracts across isolated operational environments.

01Two-stage containerized evaluation architecture

Doc2Harness separates the construction of agent scaffolding from runtime verification across two distinct container phases. In Stage 1, a build model operates inside a fixed builder harness where it receives an interface-only contract skeleton alongside a unified development document that consolidates behavioral specifications gathered from twelve frontier agent harnesses. Working within an isolated container, the build model interprets these specifications to construct a fully functioning candidate agent harness.

In Stage 2, the newly constructed candidate harness is frozen and deployed into separate containers running dedicated test instances. Each test scenario executes independently across these isolated runtime environments to ensure zero cross-test contamination. To verify execution fidelity without relying on nondeterministic language outputs, each test run is driven end to end by a scripted mock model designed to trigger specific agent workflows.

Verification relies on four deterministic, observable runtime channels that capture real system state during execution. Doc2Harness scores candidate harnesses by measuring requests transmitted to the language model, file mutations written to disk, operating system process exit codes, and standard command output streams. The framework incorporates positive controls across all scoring routines to detect and eliminate vacuous passes where an incomplete harness returns successful return codes without satisfying functional requirements.

02Frontier model benchmarks across 328 test cases

Across nineteen evaluated frontier models assessed with three independent construction runs on 328 test cases, systems demonstrated significant variance in their capacity to generate compliant software architecture. GPT-6 Astra achieved the highest overall pass rate across the evaluation suite at 53.05 percent, while the lowest-performing model recorded a pass rate of 27.44 percent, establishing an empirical performance spread of approximately twenty-six percentage points across evaluated architectures.

Even the leading system failed nearly half of the 328 benchmark cases, demonstrating that end-to-end scaffolding engineering presents persistent hurdles for frontier reasoning models. Benchmark error distributions revealed that failures consistently concentrated in how cross-cutting system functionality is divided across modular boundaries, specifically when synchronizing file mutations, output streaming, and API request schemas across separate components.

03Command execution and model invocation workflows

Engineers evaluating candidate harnesses or driving automated evaluation pipelines can invoke the testing harness directly through command-line tools. Standard package management commands handle updates across testing environments:

pip install --upgrade doc2harness

When driving candidate harnesses against specific model architectures, command lines pass the target build model directly into evaluation scripts:

doc2harness-eval --model gpt-6-astra --build-runs 3

Within automated developer agent terminals, teams configure the underlying model alias or switch default providers directly using operational configuration commands:

/model gpt-6-astra

04Modular boundary failures and test constraints

The primary technical bottleneck identified in Doc2Harness centers on module decomposition and boundary enforcement. While models can construct isolated script elements, they routinely fail when tasked with maintaining contract fidelity across interconnected files, resulting in schema mismatches between API request formatters and local disk write handlers.

Evaluation reliability depends strictly on executing candidate harnesses inside isolated container environments. Attempting to run Stage 2 tests within shared filesystems invalidates disk observation channels, as residual file artifacts alter test baselines and trigger false positives across positive control checks.

Deterministic scoring also relies entirely on the scripted mock model driver during Stage 2 validation runs. Substituting the scripted mock driver with a dynamic language model introduces nondeterministic token outputs, breaking the four-channel scoring pipeline and preventing exact reproduction of process exit codes and command outputs.

05Automated harness validation in software pipelines

Doc2Harness establishes a standard for shifting agent evaluation away from downstream code completion towards foundational systems scaffolding. As autonomous coding agents take on multi-step software tasks in enterprise environments, engineering teams require reproducible benchmarks that measure an agent's architectural discipline and systems programming reliability.

Subsequent research will track whether frontier models can overcome the 53 percent ceiling by improving modular reasoning and cross-component state synchronization. Teams deploying internal coding agents are integrating multi-channel container scoring into continuous integration suites to validate harness contracts before deploying autonomous agents to production repositories.