What the Specification Did Not Say: Gaps in Test-Graded Coding-Agent

Researcher Fei Ding released a thirteen-page study, static analysis suite, and benchmark correction dataset detailing specification-oracle gaps in test-graded coding-agent evaluations on October 22, 2026. The findings, detailed in arxiv.org's report under identifier arXiv:2610.30635 in computer science computation and language, demonstrate that common autonomous programming benchmarks measure whether an agent duplicates internal upstream names rather than whether it solves requested software specifications. Across ninety-eight graded milestones and thirty-five published runs, benchmarks routinely grade code against tests requiring arbitrary constants and paths omitted from prompts.

This breakdown examines the newly quantified failure rates across software engineering evaluation harnesses, the static audit classifier built across five programming languages, and the benchmark addenda released for open harness corrections. The release is designed for model builders, benchmark maintainers, and evaluation engineers tracking agentic performance on complex multi-step software tasks. It details why popular benchmark suites reward agents trained on exact repository layouts while unfairly penalizing functionally correct independent implementations that lack access to unstated internal names.

What Specification Did Not shipped

Author Fei Ding shipped a formalization of the specification-oracle gap alongside a multi-language static audit tool and corrected filter lists for software engineering agent evaluations. The release formalizes how natural language prompts omit exact upstream implementation details while automated test suites grade against those exact unmentioned names, file paths, and literal strings. To detect these hidden dependencies automatically, the author developed a static audit suite supporting five programming languages that classifies every dependency in scored test suites. On held-out validation tests, this static classifier achieves eighty-nine point two percent precision and ninety-two point nine percent recall across name, path, and literal requirements.

Alongside the diagnostic parser, the release delivers specification addenda and benchmark filter lists that identify unstated requirements across ninety-eight graded milestones in SWE-Milestone. In twenty-five of those milestones, scoring tests directly depend on unstated, non-inferable names or paths that natural language specifications never provide to autonomous agents. The toolkit isolates instances where chained dependent tasks compound these inconsistencies across complex multi-file engineering problems. By releasing standardized addenda and filters, the release enables evaluation teams to screen existing benchmarks, remove distorted test requirements, and score coding systems strictly against observable behavioral criteria rather than repository memorization.

What changed for builders in Specification Did Not

The research provides quantitative evidence explaining why leading coding agents experience steep drop-offs during test-graded multi-step development tasks. On thirty-five published runs across SWE-Milestone, build units containing an unstated dependency suffer an eighty point six percent failure rate, compared to a forty-eight point four percent failure rate for comparable units without unstated dependencies. Controlled ablation tests revealed that withholding a single hidden name from the reference implementation causes compilation failures in one hundred twenty-six out of one hundred twenty-eight cases. This gap severely distorts repository-level scores, unfairly favoring agents that reproduce historical upstream naming conventions over clean alternative architectures.

Across four additional benchmarks analyzed in the study, gap prevalence directly reflects task construction methodology rather than underlying model capabilities. Tasks built from release notes or language model summaries exhibit specification-oracle gaps in one-third or more of their task sets, creating widespread scoring anomalies. Conversely, benchmark suites where human authors screened or authored tasks specifically for their evaluation tests show virtually no unstated naming gaps. The findings demonstrate that benchmark rankings remain relatively robust at the top level when removing tainted tests, but individual repository scores shift substantially once evaluation harnesses evaluate true specification compliance.

How to install or upgrade Specification Did Not

Evaluation teams and agent builders can integrate the static audit and updated benchmark filters into existing development workflows to cleanse agent evaluation harnesses. Engineers updating their local coding agent CLI tooling can pull the latest framework updates directly through standard package distribution channels before running test suites. Keeping development harnesses updated prevents stale test oracles from masking model regressions during local integration testing. Builders can run the vendor update commands to refresh their agent harnesses before executing benchmark suites across modified repositories:

npm install -g @anthropic-ai/claude-code
claude update

When evaluating models against the corrected benchmark addenda, developers configure specific model endpoints directly within interactive sessions or through startup parameters. Operators can switch model targets interactively during an active session using the in-app switch command, or launch the command line tool pinned directly to a dedicated evaluation model version for consistent test runs. To ensure long-term consistency across automated benchmarking pipelines, teams can configure the default model target in their persistent configuration settings:

/model claude-fable-5-1
claude --model fable
export ANTHROPIC_MODEL=claude-fable-5-1

Gotchas and compatibility in Specification Did Not

A primary gotcha in uncorrected test suites involves chained dependent tasks, where an unstated name in an early milestone cascades through subsequent milestones. When an autonomous agent chooses a reasonable, idiomatic class or module name that differs from the upstream repository, every subsequent test relying on that upstream path breaks immediately. This cascading breakdown explains why build unit failure rates jump from forty-eight point four percent to eighty point six percent on affected milestones. Harnesses that require strict compilation before running functional assertions penalize systems disproportionately, turning minor naming discrepancies into total benchmark task failures.

Compatibility varies significantly depending on how benchmark suites curate their natural language specifications and grading test oracles. In harnesses derived from release notes or synthetic summaries, over thirty-three percent of tasks suffer from unstated oracle dependencies, making raw scores unreliable indicators of functional software engineering ability. However, because overall leaderboard rank orders remain largely stable when stripping corrupted tests, teams may mistakenly assume their test suites are sound. Builders must review repository-level metric shifts, as models that merely memorize upstream naming patterns artificially outperform innovative models on contaminated tasks.

What to watch after Specification Did Not

Following the release of the static audit tool and corrected filter lists, evaluation maintainers across the software engineering ecosystem face growing pressure to sanitize legacy datasets. The release provides specification addenda for SWE-Milestone and establishes an automated audit framework capable of classifying name, path, and literal dependencies across five programming languages. Teams managing benchmark leaderboards should watch for public adoption of the provided filter lists, which isolate twenty-five flawed milestones without requiring complete dataset retirement. Adopting these filters enables clean comparisons across published runs without penalizing models for legitimate architectural divergence.

Future benchmark construction is shifting away from automated scraping of release notes and uncurated issue descriptions toward verified specification-oracle alignment. Maintainers of the four additional benchmarks evaluated in the research are reviewing whether grading harnesses evaluate behavioral outputs or enforce upstream implementation artifacts. As autonomous coding agents tackle longer task sequences, the audit methodology offers a reproducible mechanism to verify that scoring tests assert only behaviors explicitly stated in initial task prompts. Tracking how harness maintainers integrate these specification addenda will indicate whether industry leaderboards reflect true software development autonomy.