OpenBench – A benchmark for comparing coding-agent harnesses

I'll pull the source post and HN thread so the write-up sticks to real names and numbers only.Matthew Lam released OpenBench v1, an open benchmark for comparing coding-agent harnesses rather than models alone. The project is public at github.com/minghinmatthewlam/openbench under the MIT license and was posted to Hacker News with 1 point and 0 comments at the time of listing. The framing is fixed: hold the model and task constant, vary the wrapper (CLI run loop, tools, and permissions), and score outcomes on correctness, wall-clock speed, and token cost.

Technically, OpenBench runs each harness headlessly against self-contained tasks. A checker script is the only judge of success (exit 0 means solved; an optional SCORE line allows partial credit), not the harness’s own claim. Track A pins every compatible harness to the same canonical model, gpt-5.5-medium, so gaps come from scaffolding and tooling. Integrated adapters cover codex, pi, opencode, cursor, and devin, plus open-model claude; the v1 announcement also lists grok build and runs against models such as gpt 5.6-sol, GLM 5.2, Kimi K3, and Grok 4.5. Tasks span repo-authored core work, Exercism imports, and a Terminal-Bench frontier tier. Early matrices report that clean frontier harnesses often hit a correctness ceiling on synthetic core tasks, while efficiency still spreads—up to about 4× on wall-clock and about 8× on token tax—with pi repeatedly the fastest and leanest. A committed open-model matrix (pi and opencode × four open models × three hard tasks × three trials) cost about $1.02 in first-party API spend. A local Terminal-Bench run of 45 cells landed at 12/15 per harness (mean score 0.80), below the synthetic ceiling. Codex showed unusually high token use in Lam’s early panels.

For engineers shipping agents in production, the useful claim is operational: once models clear day-to-day tasks, cost per solve and latency dominate the bill. OpenBench is built to rank harness-plus-model stacks on that efficiency surface, including private-codebase evals so teams can test against their own slices without publishing transcripts. The runner is resumable, supports a null negative control, Docker isolation, and Wilson 95% intervals so small trial counts are not over-read as leaderboards.

Market context is crowded on model leaderboards and thin on harness comparison. Routing layers (OpenRouter, Cloudflare, Databricks, Vercel, Ramp) optimize which model serves a request; OpenBench sits one layer down, asking which agent stack uses tokens and time well on real work. That matches the same shift Lam cites at enterprises such as DoorDash’s codebase and PR evals: judge agents on your code and workflows, not only public suite scores.

Practical takeaway: treat harness choice as a measured variable. Reproduce a cheap open-model panel with obench, pin one model across candidates, and rank on tokens-per-solve and seconds-per-solve once correctness saturates. Watch next for wider harness coverage, more Terminal-Bench and private-repo tasks, open-model panels across GLM, Kimi, and DeepSeek, and whether Codex’s high token footprint holds under larger, multi-trial runs.