OpenBench – A benchmark for comparing coding-agent harnesses

Gathering the source material so the paragraphs stay factual and specific.Matthew Lam introduced OpenBench v1 as an open framework for measuring AI performance and efficiency against a real codebase and use case. The announcement framed the problem as companies learning they cannot simply maximize token spend, and instead need ways to measure how agents actually perform. On Hacker News AI Agents the item still sat at 1 point with 0 comments, with the primary write-up living on Lam’s X post.

Technically, OpenBench targets the tradeoff between correctness and efficiency, scoring correctness percentage alongside token use (input, output, and cache) and latency. It is built so people can extend the task set, run many harness-plus-model combinations, and plug in custom harness variants—for example codex ablations versus stock codex. Lam also described tooling to discover and add reliable tasks from a repo. Integrated harnesses named at launch include codex, claude, cursor, devin, grok build, and pi, with early evals across models such as gpt 5.6-sol, GLM 5.2, Kimi K3, and Grok 4.5.

For engineers and builders, the useful cut is not “which model is best” in isolation but which harness-plus-model stack wins on their own tasks under fixed cost and latency budgets. Harness choice—tool wiring, prompts, scaffolding, caching behavior—shows up in the same scorecard as the model. That gives teams a path to A/B harness changes the way they already A/B model versions, with metrics that map to PR quality, spend, and wall-clock time rather than leaderboard vibes.

Market context sits next to the rise of model routers such as OpenRouter, Cloudflare, Databricks, Vercel, and Ramp. Routers decide where traffic goes; OpenBench is aimed at whether the agent stack that receives that traffic is correct and efficient on production-like work. Lam pointed to DoorDash’s codebase and PR evals as the pattern companies are already inventing in-house. OpenBench tries to productize that cross-harness, cross-model evaluation layer so teams do not have to rebuild it from scratch.

Practical takeaway: if you run coding agents in production or near it, treat OpenBench-style evals as a first-class loop—define tasks from your repo, freeze harness and model pairs, and score correctness against tokens and latency before you change defaults. Watch whether the public task set grows beyond Lam’s initial integrations, how custom harness variants are shared, and whether cache-aware token accounting becomes the standard way people compare agent stacks rather than raw completion scores alone.