Harness-evals. compare agent designs with the model held fixed
Developer Slava Dubrov released harness-evals v0.5.1 to evaluate AI agent harnesses with models held fixed using standard library Python 3.12 code.
By Dillip Chowdary • Nov 16, 2026 • Source: github.com

At a glance
- Product
- harness-evals
- Version
- v0.5.1
- Runtime
- Python 3.12 or later
- Dependencies
- Python standard library only
- Price
- Free open source
- Availability
- GitHub public repository
In this post
Developer Slava Dubrov released harness-evals version v0.5.1, an open-source Python library designed to evaluate AI agent harnesses and automated workflows while holding the underlying language model fixed. Detailed in github.com's report, the tool tracks repeated-run performance across metrics including success rate, critical violations, execution cost, latency percentiles, and token consumption without relying on third-party dependencies outside the standard library.
This technical analysis breaks down the framework architecture, execution adapters, custom validation checks, and bootstrap confidence intervals used to compare competing agent harness designs. It is written for software engineers, machine learning platform developers, and system architects who need to determine whether prompt adjustments, routing workflows, or autonomous agent loops provide measurable reliability before promoting code.
01Standard library installation and offline simulation
Engineers can install the companion package from the Recursive Harness Engineering series directly using pip or uv targeting the v0.5.1 release tag. The repository includes an offline demonstration in examples/refund_desk.py that executes within one second using simulated agents with seeded errors.
- Clone repository fetch the codebase using git clone with branch v0.5.1 and depth one from GitHub.
- Run simulation execute uv run python examples/refund_desk.py out to generate episode records and graphical figures.
- Summarize logs run the harness-evals summarize command against out/episodes.jsonl specifying the refund desk use case.
- Compare candidates execute the harness-evals compare command comparing the baseline agent against the candidate workflow.
Note: The software requires Python 3.12 or later and operates with zero external dependencies outside the Python standard library. It does not invoke language models or execute external harnesses directly, relying entirely on user adapters to run tasks and return observations.
02Fixed model methodology and system architecture
Standard model benchmarks hold harness logic static while swapping foundational models, but harness evaluation reverses this experimental setup completely. The underlying model, provider endpoint, temperature settings, evaluation tasks, and test checks remain frozen so observed metric shifts isolate the harness design.
The system structure divides evaluation into distinct modules across usecase.py, core.py, runner.py, and metrics.py. A UseCase specifies the tasks, quality gates, and optional custom metrics, while Task defines input prompts alongside required obligations, and Episode records recorded actions, total cost, latency, and model calls.
Evaluation runners rotate harness execution order across multiple trials and generate manifest records documenting frozen parameters. Result modules calculate Pareto frontiers to identify designs no competitor beats across selected trade-offs, alongside failure map SVG diagrams that plot task failures against check categories.
03Task contracts and custom adapter execution
Advertisement
Tech Pulse Daily
Get tomorrow's pulse first
Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.
A developer defines task contracts using built-in assertions including did, never, at_most, says, never_says, ended, state, and judged. Assertions must return strict boolean values, because returning None or unhandled exceptions signals missing evidence and causes the evaluator to record an undecided status.
Checks categorized under never and at_most operate as critical rules by default, meaning any breach logs a violation that disqualifies candidate harnesses from passing quality gates. Engineers can wrap custom Python functions inside a Check object, but string returns like lowercase false are rejected as non-passing.
Adapters receive isolated task copies to ensure independent execution, and the runner creates frozen snapshots of each returned episode. Missing episodes or raised adapter exceptions trigger infrastructure failures that mark every task in the batch as failed, keeping the error message in Episode.error rather than masking software bugs as low scores.
04Simulated refund desk benchmark comparison
In the packaged refund desk simulation, an autonomous agent design competed against a deterministic workflow across five support tasks evaluated over ten trials. The trial simulation tracked success rates, critical violations, dollar costs, latency percentiles, and token footprints under frozen model parameters.
| Harness | Success | pass^k (k=10) | Violations | Cost per task | p50 latency | p95 latency | Calls per task | Tokens per task |
|---|---|---|---|---|---|---|---|---|
| agent | 0.90 (0.78–1.00) | 0.60 | 2 | $0.00022 | 5.52 s | 8.42 s | 3.62 | 3475 |
| workflow | 0.80 (0.40–1.00) | 0.80 | 0 | $0.00006 | 1.92 s | 2.17 s | 1.00 | 960 |
Although the autonomous agent posted a higher raw success rate of 0.90 compared to 0.80 for the workflow, the workflow logged zero rule violations. The candidate workflow also reduced p95 latency by 6.25 seconds, required 2.62 fewer model calls, and consumed 2,515 fewer tokens per task.
05Bootstrap statistical intervals and decision gates
The library calculates a ninety-five percent bootstrap confidence interval for every reported metric by resampling whole tasks across 2,000 iterations. Resampling operates at the task level rather than pooling individual runs in isolation, preserving the principle that repeated trials on a single task do not provide independent evidence across distinct tasks.
Automated decision logic evaluates candidates against predefined quality gates and operational guardrails to output recommendations to adopt, reject, or continue investigating. In the refund desk comparison, the system issued an adopt decision for the workflow because the baseline agent recorded two critical violations against a gate requiring zero.
Summary reports track specialized metrics including pass to the k, cost per success, and calibration agreement measuring how often automated graders match human labels. These quantitative outputs ensure that engineering teams adopt architectural changes based on verifiable performance bounds rather than uncalibrated spot checks.
Frequently asked questions
What runtime dependencies does harness-evals require?
The library requires Python 3.12 or later and uses only the Python standard library with zero external dependencies.
How does harness evaluation differ from standard model benchmarking?
Standard benchmarks hold the harness constant while changing models, whereas harness-evals freezes the model, endpoint, temperature, and tasks while varying harness orchestration.
Why did the simulated comparison adopt the workflow over the agent?
The baseline agent committed two critical rule violations when the quality gate allowed zero, while the workflow achieved zero violations, lower costs, and faster latency.
How does the framework compute statistical intervals?
The library generates 95 percent bootstrap confidence intervals by resampling whole evaluation tasks with their runs 2,000 times.
Sources
Author
Dillip Chowdary
Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.
Related on Tech Bytes
Advertisement