Training Agent Harness Like Training a ML Model

I'll pull the source article so the paragraphs stick to real details rather than inventing numbers or claims.Henry Pan published work on treating an AI agent harness as trainable weights rather than a hand-tuned wrapper. The harness is the system around a model—Claude Code, Codex, or a ChatGPT-style interface—that decides what the model sees, which tools it can call, and how environment output is fed back. On a thin baseline with only Run and Submit actions, training on 39 SWE-bench tasks lifted solves from 8/39 to 14/39 over 29 experiment runs (8 promoted, 20 rejected). On 38 Terminal Bench 2.0 tasks, after a determinism reset and re-baseline at 16/38, promoted changes pushed the trained harness to 23/38. The project repository is github.com/workofart/harness-training, and the write-up is dated July 18, 2026.

The loop is framed as agent-guided discrete program search with a terminal reward and persistent search memory, mapped onto a PyTorch-like API. The harness file is the “weights”; an improvement agent (GPT-5.5 high via Codex CLI) acts as a gradient estimator and proposes one bounded edit per epoch; the task LLM stays frozen as a backbone (Qwen 3.6 35B A3B FP4 on SGLang); a criterion compares candidate vs baseline solves; an optimizer promotes via git fast-forward or rejects while keeping a git ref; and learning.md holds cross-epoch memory like optimizer state. Training used rented 5090 GPUs, seed 12345, temperature 1.0, top-p 0.95, 32k context, and roughly one 30–40 task experiment every ~45 minutes. Promoted mechanisms were concrete: step-100 submit reminders, git-state guards against agents wiping their own fixes, pipefail rewrites so tail exit codes stop masking failed tests, length-cutoff repair that disables thinking and forces a tool call, and write/replace tools that base64-persist edits. Holding the Terminal Bench–trained harness fixed and swapping models (DeepSeek V3.2, MiniMax M2.5, GPT-OSS 20B/120B, MiMo V2.5, GPT-5.5) still beat the official Terminus 2 harness on the full 89-task Terminal Bench 2.0 set; a SWE-bench–trained harness also transferred gains to Terminal Bench.

For engineers building agents, the result is that large score moves can come from harness code, not only from a new model checkpoint. Failure modes the trained harness fixed include writing fixes only in ephemeral shell output, burning the full output budget on reasoning with no tool call, wiping on-disk edits during “cleanup,” and invalid tool schemas that used to kill the trial. Evaluation decomposed wins into failed-verification to solved (+23), timeout to solved (+24, e.g. GPT-5.5 headless-terminal 1755s to 221s; DeepSeek compile-compcert 1538s to 165s), and repeated missing tool calls to solved (+6), with first-try valid tool-call recovery and validation-error recovery rising sharply. Token economics diverged by model: GPT-OSS 120B shrank trajectories and cut tokens per solve; GPT-OSS 20B survived longer and spent more input tokens on failures that still did not resolve. That means harness edits redistribute which models look “good” on the same benchmark.

Competitive context is the same model-plus-harness stack that already separates Claude Code, Codex, and custom agents on Terminal Bench and SWE-bench. Official Terminus 2 is the comparison baseline for Terminal Bench 2.0; Pan’s trained harness, once frozen, lifted several open and closed models without retraining any of them. The earlier pre-determinism campaign of 1,000+ experiments showed that non-deterministic logits and environment noise made promotion look like coin flips (identical prompts could emit list_dir vs search_text on the same step). Hardening inference (SGLang deterministic mode, concurrency cap ≤10) and scrubbing environment entropy (memory addresses, temp paths, TZ, hash seeds, apt mirrors) raised the candidate promotion rate from 7.9% to 25.2% under the same promotion criteria. The market implication is straightforward: teams that only swap model APIs leave performance on the table relative to teams that version and train the harness as a first-class artifact.

Practical takeaway: treat harness.py like a model checkpoint—initialize thin enough to learn, measure on a fixed task panel, promote only on reproducible lifts, and keep a learning memo of rejected ideas. Watch whether multi-seed deterministic promotion becomes the default regularization story, whether order of promoted commits still matters after later edits, and where the harness ceiling ends and the frozen model ceiling begins. Builders shipping coding agents should instrument the same failure ledger (verification fail, timeout, missing/invalid tool calls) and ask whether the next gain is a new model or a one-line policy change that keeps the fix on disk and the agent inside the step budget.