Veo 3 launched at Google I/O just a few weeks ago, and since then we’ve seen countless videos go viral, delighting millions of people and demonstratin...
What infrastructure noise means for agentic coding evals
Agentic coding evaluations measure how well a system can plan, edit, run tools, and recover from mistakes across multi-step tasks. Those results are only useful if the score mostly reflects the agent’s behavior. Infrastructure noise is everything else that moves the score: flaky sandboxes, variable latency, tool timeouts, cache hits that change what the agent “sees,” non-deterministic file system state, and harness differences between runs. When that noise is large, a higher score can mean a luckier environment rather than a better agent.
Anthropic Engineering’s focus on this problem is practical. Agentic coding is not a single prompt-and-response benchmark. It is a loop over tools, code, tests, and sometimes network-backed services. Each hop is a place where timing, retries, and environment drift can dominate model quality. Treating the eval as “just the model” hides the harness that actually decides pass or fail.
Where the noise usually comes from
Most noise is not mysterious. It shows up in a few recurring layers. The agent runtime may restart processes, reuse containers, or leave artifacts from a previous attempt. Tool APIs can rate-limit, throttle, or return partial results under load. Build and test commands can depend on cold versus warm caches. Parallel workers can contend for CPU, disk, or ports, so the same agent looks slower or more error-prone on a busy host than on an idle one.
- Environment drift: OS packages, language runtimes, and default configs that differ across machines or days.
- Timing effects: timeouts that fire only under load, or races that depend on how fast tools respond.
- State leakage: leftover files, env vars, or ports from earlier trials that change later behavior.
- Harness variance: different prompt wrappers, retry policies, or tool schemas between “eval A” and “eval B.”
If two runs of the same agent disagree, the first question should be whether the infrastructure was fixed—not whether the model “got worse overnight.”
How to design evals that keep noise in check
Start by freezing the execution surface. Pin images, dependency locks, and tool versions. Reset the workspace to a known tree before every trial. Prefer hermetic sandboxes over shared hosts when you need comparable scores. Capture structured logs for every tool call: start time, end time, exit code, and truncated stderr. Without that trail, you cannot separate agent mistakes from infrastructure failures.
Then separate “agent error” from “infra failure” in the scoring rules. A timed-out network call is not the same as a wrong edit. A flaky test suite is not the same as a bad patch. Report both rates. If infra failures rise, freeze model comparison until the harness is stable again. Also run multiple trials and report spread, not only the best run. High variance is a signal that the eval is measuring the lab as much as the agent.
Reading scores with infrastructure in mind
When you compare agents, demand the same harness, the same machine class, and the same retry policy. A score gain that disappears after normalizing timeouts or clearing caches is not progress. Prefer metrics that are robust to small timing shifts: task completion under a fixed budget, edit correctness checked by deterministic tests, and recovery after a controlled injected fault. Soft metrics that depend on wall-clock speed need explicit load controls or they will reward whoever ran on a quieter box.
Infrastructure noise will never reach zero in real agentic systems. The goal is to keep it small enough that ranking and regressions stay trustworthy. Build the eval like production tooling: reproducible environments, clear failure classes, and logs you can audit. That discipline is what makes agentic coding evals useful for engineering decisions instead of noise-driven leaderboard churn.