Home / Blog / Harness-evals. compare agent designs with the model held…
Tech News

Harness-evals. compare agent designs with the model held fixed

Developer Slava Dubrov released harness-evals v0.5.1 to evaluate AI agent harnesses with models held fixed using standard library Python 3.12 code.

By Dillip Chowdary • Nov 16, 2026 • Source: github.com

Harness-evals. compare agent designs with the model held fixed
Image: github.com

At a glance

Product
harness-evals
Version
v0.5.1
Runtime
Python 3.12 or later
Dependencies
Python standard library only
Price
Free open source
Availability
GitHub public repository

Developer Slava Dubrov released harness-evals version v0.5.1, an open-source Python library designed to evaluate AI agent harnesses and automated workflows while holding the underlying language model fixed. Detailed in github.com's report, the tool tracks repeated-run performance across metrics including success rate, critical violations, execution cost, latency percentiles, and token consumption without relying on third-party dependencies outside the standard library.

This technical analysis breaks down the framework architecture, execution adapters, custom validation checks, and bootstrap confidence intervals used to compare competing agent harness designs. It is written for software engineers, machine learning platform developers, and system architects who need to determine whether prompt adjustments, routing workflows, or autonomous agent loops provide measurable reliability before promoting code.

01Standard library installation and offline simulation

Engineers can install the companion package from the Recursive Harness Engineering series directly using pip or uv targeting the v0.5.1 release tag. The repository includes an offline demonstration in examples/refund_desk.py that executes within one second using simulated agents with seeded errors.

  1. Clone repository fetch the codebase using git clone with branch v0.5.1 and depth one from GitHub.
  2. Run simulation execute uv run python examples/refund_desk.py out to generate episode records and graphical figures.
  3. Summarize logs run the harness-evals summarize command against out/episodes.jsonl specifying the refund desk use case.
  4. Compare candidates execute the harness-evals compare command comparing the baseline agent against the candidate workflow.

Note: The software requires Python 3.12 or later and operates with zero external dependencies outside the Python standard library. It does not invoke language models or execute external harnesses directly, relying entirely on user adapters to run tasks and return observations.

02Fixed model methodology and system architecture

Standard model benchmarks hold harness logic static while swapping foundational models, but harness evaluation reverses this experimental setup completely. The underlying model, provider endpoint, temperature settings, evaluation tasks, and test checks remain frozen so observed metric shifts isolate the harness design.

The system structure divides evaluation into distinct modules across usecase.py, core.py, runner.py, and metrics.py. A UseCase specifies the tasks, quality gates, and optional custom metrics, while Task defines input prompts alongside required obligations, and Episode records recorded actions, total cost, latency, and model calls.

Evaluation runners rotate harness execution order across multiple trials and generate manifest records documenting frozen parameters. Result modules calculate Pareto frontiers to identify designs no competitor beats across selected trade-offs, alongside failure map SVG diagrams that plot task failures against check categories.

03Task contracts and custom adapter execution

Advertisement

Tech Pulse Daily

Get tomorrow's pulse first

Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.

A developer defines task contracts using built-in assertions including did, never, at_most, says, never_says, ended, state, and judged. Assertions must return strict boolean values, because returning None or unhandled exceptions signals missing evidence and causes the evaluator to record an undecided status.

Checks categorized under never and at_most operate as critical rules by default, meaning any breach logs a violation that disqualifies candidate harnesses from passing quality gates. Engineers can wrap custom Python functions inside a Check object, but string returns like lowercase false are rejected as non-passing.

Adapters receive isolated task copies to ensure independent execution, and the runner creates frozen snapshots of each returned episode. Missing episodes or raised adapter exceptions trigger infrastructure failures that mark every task in the batch as failed, keeping the error message in Episode.error rather than masking software bugs as low scores.

04Simulated refund desk benchmark comparison

In the packaged refund desk simulation, an autonomous agent design competed against a deterministic workflow across five support tasks evaluated over ten trials. The trial simulation tracked success rates, critical violations, dollar costs, latency percentiles, and token footprints under frozen model parameters.

HarnessSuccesspass^k (k=10)ViolationsCost per taskp50 latencyp95 latencyCalls per taskTokens per task
agent0.90 (0.78–1.00)0.602$0.000225.52 s8.42 s3.623475
workflow0.80 (0.40–1.00)0.800$0.000061.92 s2.17 s1.00960

Although the autonomous agent posted a higher raw success rate of 0.90 compared to 0.80 for the workflow, the workflow logged zero rule violations. The candidate workflow also reduced p95 latency by 6.25 seconds, required 2.62 fewer model calls, and consumed 2,515 fewer tokens per task.

05Bootstrap statistical intervals and decision gates

The library calculates a ninety-five percent bootstrap confidence interval for every reported metric by resampling whole tasks across 2,000 iterations. Resampling operates at the task level rather than pooling individual runs in isolation, preserving the principle that repeated trials on a single task do not provide independent evidence across distinct tasks.

Automated decision logic evaluates candidates against predefined quality gates and operational guardrails to output recommendations to adopt, reject, or continue investigating. In the refund desk comparison, the system issued an adopt decision for the workflow because the baseline agent recorded two critical violations against a gate requiring zero.

Summary reports track specialized metrics including pass to the k, cost per success, and calibration agreement measuring how often automated graders match human labels. These quantitative outputs ensure that engineering teams adopt architectural changes based on verifiable performance bounds rather than uncalibrated spot checks.

Frequently asked questions

What runtime dependencies does harness-evals require?

The library requires Python 3.12 or later and uses only the Python standard library with zero external dependencies.

How does harness evaluation differ from standard model benchmarking?

Standard benchmarks hold the harness constant while changing models, whereas harness-evals freezes the model, endpoint, temperature, and tasks while varying harness orchestration.

Why did the simulated comparison adopt the workflow over the agent?

The baseline agent committed two critical rule violations when the quality gate allowed zero, while the workflow achieved zero violations, lower costs, and faster latency.

How does the framework compute statistical intervals?

The library generates 95 percent bootstrap confidence intervals by resampling whole evaluation tasks with their runs 2,000 times.

Sources

Dillip Chowdary

Author

Dillip Chowdary

Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.

Related on Tech Bytes

Advertisement

5-min tech signal

Weekday briefing for engineers who skip the noise.

No spam · Unsubscribe anytime

Advertisement

✈️ CareerPilot

Your AI job-search copilot

Match your resume against live Ashby, Greenhouse & Lever openings — fit scores, job-specific resume optimization and email alerts.

Find matching jobs →

Free Tools

Browse all tools →