How to Design an Agent Evaluation That Doesn't Lie to You

What happened

A GitHub repository called researchops-agent, published by cedRiC874 at commit 6457358d74cc07106dfb7a348ac143cdaa87e459, contains an article titled "How to Design an Agent Evaluation That Doesn't Lie to You." The piece surfaced on Hacker News with one point and no comments. Its author draws on a background in clinical research methodology and applies that skepticism to the agent-evaluation practices now standard across the machine learning industry.

This article is for engineers who build, test, or sign off on agent systems and who suspect their internal evaluations may be telling them a more flattering story than the data supports. It unpacks the structural problems that make agent evals especially prone to self-deception, maps them onto analogous controls that clinical trials developed over decades, and describes the concrete checks a builder should run before trusting a result well enough to put it in a status report or a README.

The author was integrating a second model provider into the ResearchOps Agent, an agent for scientific data analysis built around deterministic statistics, evidence binding, and human approval. A separate validation pipeline called LongiEye is also part of the public work described. During integration, the Kimi path failed across two independent, separately authorized attempts at the very first response: validation failed and the system refused to continue rather than improvise, a fail-closed design. The logged artifact carried status=failed, error_code=kimi_chat_response_invalid, and causal_root_cause=undetermined_without_raw_provider_payload. The closest natural summary would have attributed the failure to provider incompatibility, but the author stopped short and built an offline probe instead.

How it works

That probe required neither network access nor authorization. It took a single golden SSE fixture, a standard streaming response known to pass the real v2 parser, then transformed it into 11 variants that a real provider might plausibly return. Eight of the 11 variants failed. Five produced exactly the same kimi_chat_response_invalid error that appeared online. Triggering factors included an extra system_fingerprint field, an omitted empty finish_reason in a non-terminal chunk, a repeated role, a dated model ID, and an additional refusal:null in the delta. The parser rejected any response containing an unrecognized key because it used an exact-key check. Without the probe, the author writes, the record would permanently read "the provider was incompatible" when the correct reading was "my parser was too strict."

The article identifies three structural properties that make agent evaluations systematically prone to self-deception. First, the denominator is hidden. The same Kimi run can be reported as 0 out of 3 scenarios completed or as 1 out of 8 model requests executed, with 0 trusted tool calls, 0 tool executions, and 0 usage observations. Each framing is accurate, but only the second tells the reader that the failure happened at response validation, before the model ever had a chance to attempt the task. Reporting only the most favorable denominator misleads without altering any data point.

Why it matters

Second, success can be manufactured silently. An LLM can wrap an error in an apology, a retry, and fluent prose until the output resembles a correct answer, and if scorer feedback is fed back into the agent it may learn to pass the scorer. Third, most agent evaluations lack a non-agent baseline. Comparing model A with model B is common; asking whether a fixed workflow, a single retrieval step, or a deterministic script could already complete the task is rare. Without that counterfactual, an evaluation can only show that an agent can do something, not that it is worth using for that thing.

The article maps these three failure modes onto established clinical trial controls. Pre-specifying the primary endpoint corresponds to not waiting until the run is over and then choosing the best-looking metric. Measuring success rate, tool correctness, citations, latency, cost, and formatting, then reporting only the largest gain, is winner-picking. Multiplicity correction addresses the temptation to try many prompt variants and present the winner as though only one comparison was made. Protocol locking prevents changing the evaluation set while developing against it: run, inspect, revise, rerun, and call the second round the evaluation result, and those tasks have already become a development set even if the filename still says test.

Blinding maps directly to LLM-as-judge evaluation. Visible source labels leave the evaluation unblinded, and presenting candidates in a fixed order introduces position bias. The article suggests anonymizing the source, randomly swapping order within each pair, and checking flip consistency between A/B and B/A presentations. High-stakes conclusions should also receive independent, blinded human review. The most transferable idea, the author argues, is denominator discipline borrowed from intention-to-treat, per-protocol, and available-case analyses in clinical research. In a simulated ResearchOps study, the design requested an intention-to-treat population; the raw dataset contained 240 participants, but 28 had missing follow-up outcomes, so the statistical analysis included only 212.

Who is affected

Any engineer who writes evaluation code, reviews evaluation reports, or decides whether an agent is ready to ship is exposed to these failure modes. The article is grounded in two public repositories, so the affected work is open to inspection. The ResearchOps Agent is described as an agent for scientific data analysis; evaluation errors in that context carry direct consequences for the reliability of conclusions drawn from data. The LongiEye validation pipeline is a companion system for a public longitudinal cohort, where denominator errors translate into incorrect population statistics.

More broadly, anyone using LLM-as-judge evaluation to compare model outputs is affected by the position bias and source-visibility problems described. Teams that run multiple prompt variants before selecting one for comparison are running into the multiplicity problem whether they recognize it or not. Organizations that develop agent prompts and evaluation sets in parallel, then declare the evaluation clean when scores improve, are using what the article calls a development set, not a test set, even if they have not changed the filename.

What to watch next

The verification paths promised in the article point to the public ResearchOps Agent repository and to offline probe output that any reader can reproduce. A builder who wants to apply these controls should start by checking whether their current evaluation pre-specifies a primary metric before any run begins, whether they report all denominators including tasks that crashed or never started, and whether their LLM-as-judge setup randomizes presentation order and checks flip consistency.

The article does not propose a framework or a new tool; it proposes discipline. That means the next practical step is internal: audit the last three evaluation reports your team published and ask whether the denominator shown was the most flattering available one, whether prompt selection was performed before or after looking at scores, and whether any causal claim about a provider, a model, or a configuration was actually supported by evidence about your own code. The commit hash at the top of the source repository is public, so any future changes to the article will be visible in the diff history.