OSS harness took Claude Opus 5 from 30% to 99.95% on ARC-AGI-3

What happened

An OSS harness has raised the performance of the Claude Opus 5 model on the ARC-AGI-3 benchmark from 30% to 99.95%. This development was shared on HN Claude/Codex/Fable, receiving 5 points and 0 comments.

This piece covers the technical features of this harness, whom it affects, and how builders can verify it. It is for engineers building autonomous systems. The announcement was discussed at https://twitter.com/MorgantWillis/status/2093342777841013096 and https://news.ycombinator.com/item?id=49480080.

The release consists of an OSS harness designed to wrap and manage frontier language models. The software provides a structured framework that manages state, memory retrieval, and planning loops during complex task execution. By wrapping the model in this external scaffolding, the system enables iterative problem-solving instead of relying on a single inference pass. This mechanism allows the model to test hypotheses, detect errors in its logic, and refine its approach dynamically. These puzzles are part of the evaluation suite.

How it works

The primary benefit of this software is the performance gain observed on complex benchmarks. Without the harness, the baseline model achieved a score of 30% on the reasoning challenges. Integrating the scaffolding system raised this performance to 99.95%, demonstrating that the environment wrapping the model is as critical as the model weights themselves. This harness manages the input and output streams, ensuring that the model receives clean context and can execute actions. Developers can utilize this tool to test responses.

Software engineers must shift focus from fine-tuning model parameters to optimizing the surrounding infrastructure. Builders now have access to a standardized interface that abstracts the complexities of state tracking and agent loops. This shift means that developer time is spent writing prompt templates, defining tool schemas, and managing context window allocation. Instead of treating the model as a simple text completion engine, builders must treat it as a CPU inside a computer system that requires external memory and control software.

Why it matters

This release changes how applications are tested and verified before deployment. Builders can no longer rely on simple input and output assertions to evaluate system performance. Instead, they must monitor the execution path, tracking how the model interacts with the harness over multiple turns. This transition requires implementing comprehensive logging systems that capture every step of the reasoning process. By doing so, builders can identify where the planning loop fails and adjust the harness parameters to prevent regression in production.

To begin using the software, developers need to clone the repository from the repository hosting service. The installation process requires setting up a virtual environment to isolate the project dependencies. Builders must ensure that their environment contains the necessary runtime libraries to support asynchronous operations and network requests. Once the dependencies are installed, developers can configure the system by specifying the target model endpoint in the configuration file. This configuration file maps the model identifier to the corresponding API keys.

Upgrading to the latest release involves pulling the changes from the remote repository and rebuilding the environment. Builders should run the test suite to verify that the installation was successful. This verification process executes a series of baseline tests that check the connectivity to the model API and validate the state management components. If the test suite passes without errors, the upgrade is complete. Builders must verify that their custom tools and templates remain compatible with the updated harness interfaces.

Who is affected

A major technical consideration when deploying this system is the increase in latency and resource consumption. Because the harness executes multiple planning and verification steps, the time required to generate a response is significantly higher than a single direct call. This increased computation also leads to higher token consumption, which affects the operational cost of the system. Builders must implement strict timeout limits and budget constraints to prevent runaway loops from consuming excessive API credits during these running reasoning sessions.

Compatibility is restricted to models that support structured tool use and stable function calling. If the underlying model cannot consistently output valid tool calls, the planning loop in the harness will break. This limitation means that older or smaller models may not function correctly when wrapped in this scaffolding. Developers must verify their chosen model endpoint is capable of handling the complex context structures generated by the harness. Failure to do so will result in frequent errors and degraded performance.

What to watch next

Industry observers are tracking how these execution harnesses will be integrated directly into cloud provider platforms. If the scaffolding becomes a default feature of model hosting services, developers will not need to deploy separate infrastructure to manage agent states. This integration would simplify the deployment pipeline and potentially reduce latency by colocating the execution harness with the model weights. The community is also watching for the release of new benchmarks specifically designed to test the limits of these current frameworks.

Another area of focus is the development of self-correcting mechanisms that allow the harness to adapt its planning strategies dynamically. Researchers are exploring ways to let the system learn from its failures during the execution loop, adjusting its memory retrieval patterns in real time. This capability would make the system more resilient when dealing with highly unpredictable environments. Builders should monitor the repository for updates regarding multi-agent coordination, which could allow multiple wrapped models to collaborate on single software tasks.