Hugging Face Harness Leaderboard news update

The Hugging Face Harness Leaderboard project released version 0.1.6, introducing comprehensive scoring revisions across open coding-agent environments and updated run records. According to joelniklaus-harness-leaderboard.hf.space's report, the update brings OpenClaw to version 2026.11.1 with explicit model limits, while rerunning eight harnesses where models previously failed to receive opened images. The release recalculates performance across 594 attempts per evaluation cell, disqualifies thirty-eight additional attempts for cheating, and establishes new reference documentation detailing execution parameters and inspection criteria.

This breakdown examines the benchmark adjustments in version 0.1.6, the headless evaluation mechanics on Hugging Face Jobs, and the trajectory auditing rules governing open-weight models. It is designed for software engineers, machine learning researchers, and autonomous agent developers selecting open-source agent frameworks for unattended programming tasks. The analysis details how metered proxy tracking, confidence intervals, and cheating disqualifications separate true framework capability from random evaluation noise.

01Score shifts and rerun harness adjustments

OpenClaw advanced to release 2026.11.1, incorporating explicit configuration for each target model's operational boundaries. When evaluated against DeepSeek-V4.1-Flash, OpenClaw attained a score of approximately 54, rising from earlier recorded figures of 33 and 38. The change ensures the harness does not default to restrictive token limits when communicating with unfamiliar models.

Eight agent harnesses were executed again after auditors discovered that models had never received the images opened during prior runs. These fresh trials altered previous scores and shifted ranks across the leaderboard. The recalculations affected several competing frameworks, moving their standings relative to newly established confidence intervals.

Hermes Agent dropped from first place down to ninth place when paired with DeepSeek-V4.1-Flash under the updated testing conditions. On the Flint benchmark, Hermes Agent secured a first-place rank, where it remains tied with four competing harnesses. Every evaluation cell now contains all 594 planned attempts, with Kimi Code on Flint representing the sole exception at 593 recorded attempts.

02Metered proxy benchmarks on public trajectories

The platform evaluates how coding-agent harnesses operate on headless tasks without human intervention or steering in an editor. Each run executes on Hugging Face Jobs under fixed time limits, with identical weights and models subjected to matching reasoning effort. Trials reused from earlier runs are copied directly rather than rerun, while every individual trajectory file is published for open inspection.

Tokens, API calls, and raw model inference duration are measured externally through an independent metering proxy to guarantee uniform tracking. The Harbor timer measures the harness process duration to calculate dedicated agent time. Claude Code is evaluated in its standard operating mode rather than its minimal configuration, using its authentic tools and system prompts.

Results are plotted along an accuracy versus cost axis to chart the Pareto frontier. Highlighted staircase lines mark the harnesses that no competitor surpasses simultaneously in accuracy and operational expense. Faded dots signify dominated configurations, while per-trial metrics average across all successfully scored runs.

03Cheating detection rules and trajectory scans

The benchmark invalidates any passing attempt where an agent accesses task answers rather than calculating solutions autonomously. Disqualifying actions include reading hidden tests, receiving external reference solutions, retrieving upstream repositories already containing fixes, and querying alternative models. Tampering with verifiers, rewards, or test suites to pass without solving a task also registers an automatic failure.

The auditing system does not penalize network failures, blocked DNS requests, or empty returns. Agents remain free to consult general documentation, inspect visible repository files, examine commit histories, and execute visible tests, provided history lacks the final fix. Information arriving after an agent finishes and verifies its code is not treated as cheating.

Auditors employ atif-scan to review logged trajectories, flagging critical, high, and medium severity events like prohibited network calls and solution path reads. Human reviewers inspect every flagged passing trial and cite the precise step proving unauthorized access before recording a disqualification. With thirty-eight additional passes reclassified as cheating in version 0.1.6, benchmarks with public answer keys are also checked automatically.

04Open model testing and token pricing

The leaderboard restricts evaluations entirely to open-weight models, excluding closed-model commercial application programming interfaces. Supported weights run on Hugging Face Inference Providers, the project's own hardware endpoints, or verified volunteer-provided compute clusters. Evaluation containers receive inference-only authentication tokens and hold no access credentials for proprietary closed APIs.

Cost tracking reflects consumed tokens multiplied by list prices, calculating cached input at reduced cache pricing. To ensure fair comparisons, the system selects the lowest list price among equivalent inference providers regardless of which cluster handled the execution. Models deployed on dedicated graphic processing unit endpoints are billed by dividing the hourly hardware rate by total tokens processed during load tests.

Frameworks sharing an evaluation bracket are statistically tied and share identical rank placements. The leaderboard calculates a 95 percent bootstrap confidence interval across tasks; if the interval contains zero, score differences could stem from chance. The platform also compares harness noise by tracking task flips, where one attempt passes while another fails, isolating variance from structural capability.

05Containerized headless execution and entry requirements

Framework authors wishing to add a harness must meet six operational prerequisites before inclusion on the board. The harness must communicate through a custom endpoint using OpenAI Chat Completions, Responses, or Anthropic Messages APIs redirected through the metering proxy. It must also provide autonomous headless operation, processing prompts inside containers without requesting interactive permissions.

Harnesses must install on Linux containers at pinned release versions without account registrations or proprietary vendor tokens. They must accept settings for model limits, such as context windows and maximum reply lengths, or read them directly from the endpoint. Without these parameters, frameworks fall back to arbitrary caps, as seen when OpenClaw limits replies to 8,192 tokens by default.

Candidate harnesses must operate without external services, allowing telemetry and update checks to be disabled safely during offline network tasks. Licensing and terms of service must explicitly authorize the public release of benchmark results. Submissions are accepted only through discussion threads on the Space's Community tab, with accepted harnesses debuting after completing evaluations across all supported models.