Ship Harness Bench – Same prompt, same model, different harnesses

What happened

A developer named grigio published Ship Harness Bench, an open-source benchmark that sends the same prompt to the same underlying language model through nine different coding harnesses and publishes all the resulting simulators side by side for direct comparison. The project lives at grigio.github.io/ship-harness-bench and its source is on GitHub, and it earned its early discussion on Hacker News under the AI Agents thread.

This article breaks down what Ship Harness Bench actually measures, how the test was constructed, why harness choice matters as much as model choice, and which developers and teams stand to learn most from the results. If you build, evaluate, or choose AI coding tools for any production workflow, the findings here are worth understanding before you pick your next harness.

A developer published Ship Harness Bench, a public benchmark designed to isolate the effect of the coding harness on AI output quality while holding the model and the prompt constant. The benchmark gave nine harnesses — Codex, DSH, Jcode, Jcode FX, Jcode 2, O-0x, Muse, OpenCode, and Pi — the same task and let each one produce a live, playable ship simulator. Every result is hosted as a standalone demo so anyone can launch each simulation in a browser and judge the difference themselves. The project is open source on GitHub under the grigio account, and the benchmark page itself is a single navigable site that lists every harness output with its name, a short description of the simulation it produced, and a direct link.

How it works

The names attached to each output reveal something about how each harness interpreted or scaffolded the task. Codex produced a simulator called Ship Sim using import maps and modular Three.js with a day and night cycle and wave controls. DSH produced Open Sea, a self-contained single HTML file with an inline Three.js bundle and keyboard sailing controls. Those two outputs alone show a meaningful divergence in how the harnesses handled dependency management, bundling strategy, and feature scope from an identical starting prompt.

The benchmark's design is deliberately minimal: one prompt, one model, nine harnesses, nine outputs. Each harness receives the task of building an interactive ship simulation, and the harness is responsible for scaffolding the environment, managing dependencies, running verification steps if any, and delivering a final artifact. The resulting files range from Jcode's clean vanilla JavaScript with verification scripts, to Jcode FX's enhanced variant built with bun and polished visual effects, to Jcode 2's second iteration using static JavaScript with no build step at all. O-0x vendored Three.js and used a minimal vanilla implementation with orbit controls. Muse used Vite with custom fonts, a time-of-day slider, and ambient sound. OpenCode vendored Three.js with OrbitControls for a straightforward implementation. Pi used TypeScript and Vite and added seagulls, splashes, wave audio, and ambient soundscapes.

Why it matters

The range of outputs from those nine runs shows that the harness controls far more than the developer might expect. Tool selection for browser interaction, whether the harness runs verification scripts, whether it chooses a build tool like Vite or bun or no build step at all, and whether it bundles dependencies inline or links them via import maps — all of those decisions belong to the harness rather than the model, and all of them produced distinct, observable results in the final simulation.

The central claim of Ship Harness Bench is that the harness is a worthy optimization alongside model selection. When every other variable is fixed, the harness still produces outputs that differ in feature completeness, dependency strategy, visual fidelity, and interactivity. That is a significant finding for anyone who currently treats the model as the primary lever for improving AI coding output. The benchmark provides a concrete, reproducible way to test that claim: run the same task yourself through two different harnesses and compare what comes out.

The project also highlights two specific capabilities that the benchmark page identifies as differentiating factors: browser-use and memory. A harness that can see, interact with, and verify output in a real browser is described as providing one of the most useful skills in the benchmark context. Memory and other auxiliary tools are identified as shaping output quality alongside raw model capability. These are not abstract claims — the nine live simulations exist as direct evidence, and anyone can open them and draw their own conclusions about which harnesses best exercised those capabilities.

Who is affected

Engineering teams evaluating AI coding agents for internal adoption are the most immediate audience. If a team is choosing between Codex, OpenCode, or any of the other harnesses represented in the benchmark, Ship Harness Bench gives them a concrete output to inspect rather than a marketing claim to read. The task — building a Three.js-based ship simulator — is representative of a class of real coding work: multi-file projects with graphics dependencies, build tooling decisions, and interactive runtime requirements. Any team that ships web applications with similar characteristics can use the benchmark outputs as a proxy for how each harness might handle their own tasks.

Independent developers who build or maintain harnesses are equally affected. The benchmark makes the harness's internal choices visible in a way that a simple pass-or-fail test would not. The difference between Pi's TypeScript and Vite setup with ambient soundscapes and O-0x's minimal vendored Three.js implementation with orbit controls is not a difference in prompt or model — it is a difference in how each harness scaffolded and extended the work. Harness developers now have a published reference point for how their tool compares against nine peers on the same task.

What to watch next

The benchmark is a snapshot from a single run of a single prompt. The most useful next development would be additional prompts covering different task categories — a data processing script, a multi-file API client, a test suite — to determine whether harness rankings are consistent across task types or whether some harnesses are specialized for certain kinds of work. The current nine harnesses also reflect a particular moment in the tooling landscape, and adding newer or updated harnesses as they ship would make the benchmark more useful as a longitudinal reference.

Builders who want to verify the claims themselves can visit grigio.github.io/ship-harness-bench, launch each simulator directly, and inspect the source through the GitHub repository. The questions worth tracking are whether the harnesses with browser-use capabilities consistently produce more complete outputs, whether memory-augmented harnesses outperform stateless ones on tasks requiring multi-step reasoning, and whether the harness that added ambient sound and seagulls did so because it had richer tooling or simply because the model steered it that way — a distinction the current benchmark design does not resolve, but which a follow-up controlled study could.