OpenBench – A benchmark for comparing coding-agent harnesses

I'll pull the source post and thread so the paragraphs stick to real details only—no invented numbers or dates.Matthew Lam published OpenBench v1, an open framework for comparing coding-agent harnesses rather than models alone. The project lives at minghinmatthewlam/openbench on GitHub under an MIT license, with live results on the project GitHub Pages site. The framing is same model, different wrapper: pin a shared model and task set, then measure how much the CLI harness—run loop, tool set, permission policy—changes outcomes on correctness, wall-clock speed, and token cost. On Hacker News the linked thread had 1 point and 0 comments at capture time.

Mechanically, OpenBench runs each harness headlessly against self-contained tasks and grades with an external checker script (exit 0 equals solved; optional SCORE lines allow partial credit), never the harness’s own success claim. Track A pins compatible harnesses to the same canonical model, gpt-5.5-medium, so gaps are attributed to scaffolding. Separate open-model panels run models such as glm-5.2, deepseek-v4-flash, kimi-k2.7-code, and glm-4.7-flash through adapters. Task tiers include core synthetic tasks in tasks/, Exercism imports, and a Terminal-Bench frontier tier. Adapters cover codex, pi, opencode, cursor, devin, and open-model claude, plus a null negative-control adapter that edits nothing and should fail every checker. The installable CLI is obench (run, report, doctor, validate, and related commands); results append as JSONL with Wilson 95% intervals on success rates.

For engineers building or buying agent stacks, the useful signal is that harness choice is a first-class variable, not a thin wrapper around model quality. Headline findings so far say correctness can saturate on repo-authored synthetic tasks for clean frontier harnesses, so leaderboards on those tiers stop discriminating on pass rate. Efficiency still separates: wall-clock spread up to about 4× and token tax up to about 8× on early matrices, with pi repeatedly the fastest and leanest in measured panels. Open models have looked surprisingly close in some matrices; a committed 72-run open-model matrix cost about $1.02 in first-party API spend. That matters when teams are optimizing spend and latency, not only raw solve rate.

The competitive context is the crowded coding-agent CLI market—codex, pi, opencode, cursor, devin, and open-model claude among the integrated harnesses—where product comparisons usually mix model, prompt policy, tools, and UX into one score. Lam’s announcement also places OpenBench next to the rise of model routers (OpenRouter, Cloudflare, Databricks, Vercel, Ramp) and internal codebase evals such as DoorDash-style PR/codebase work: companies need harness-plus-model evaluation on their own tasks, not only public model leaderboards. Cursor has trended well on correctness in Lam’s updates; token-heavy behavior can still make a high-solve harness expensive per solve, which is the efficiency axis OpenBench is built to surface.

Practical takeaway: treat harness selection as an measurable product decision—run same-model, multi-trial cells, read Wilson intervals instead of single-run percentages, and watch efficiency metrics when correctness plateaus. What to watch next is expansion of harder long-horizon tasks (Lam notes correctness needs more difficult work), promotion of Terminal-Bench frontier runs into committed datasets (local TB work had been below the synthetic-task ceiling at 12/15 per harness / mean score 0.80 in the reported local panel), private-repo evals via obench init and local-only transcripts, and whether custom harness variants and ablations (for example codex stock vs ablated variants) become standard in team eval pipelines.