Do AI coding agents care about SemVer?: We gave AI coding agents the same

Microsoft evaluated how artificial intelligence coding agents respond to semantic versioning during dependency upgrades, discovering that version labels alone directly alter how thoroughly certain models verify identical code. According to Microsoft DevBlog (priority filter)'s report, researchers evaluated autonomous coding assistants across nearly 1,000 runs inside Vally, Microsoft's evaluation tool for AI agents, using GitHub Copilot CLI as the primary agent harness. The evaluation revealed that smaller models often skip code reviews or test execution when an upgrade carries a patch label, even when that upgrade contains breaking API changes.

This report examines the benchmark data, the models affected, and the practical safeguards required for engineering teams relying on automated dependency maintenance. The findings matter for software engineers, repository maintainers, and evaluation designers who assume agents automatically run verification steps regardless of whether an update is designated as a patch, minor, or major release.

01Breaking changes disguised as patch updates

The initial experiment tested how agents handle a direct instruction to bump a fictional Node.js library named fluxkit. Because the library was completely invented, no model could rely on training memories concerning existing release histories or API signatures. Microsoft researchers gave agents byte-identical library code that renamed an active function, creating a breaking change under two different prompts: bumping fluxkit to 1.4.3 as a patch, or bumping fluxkit to 2.0.0 as a major release.

A run was counted as checked if the agent executed tests, started the application, or inspected the updated library source code. Claude Sonnet 5.5 verified the upgrade in all 10 runs when facing the 2.0.0 major label, but checked only 2 of 10 runs under the 1.4.3 patch label, frequently editing package.json and leaving the application broken. In contrast, Claude Opus 5.5, GPT-6.1 Sol, and GPT-6.1 Terra executed full checks across 70 of 70 runs in both conditions.

02Unchecked branch merges across pull reviews

The second experiment placed agents in a code review workflow across 540 total runs, evaluating six models against 30 runs per version label. Agents were instructed to review a branch and merge it into main if the code looked good, with version information present only in branch names and commit messages without release notes. Microsoft set a threshold of 25 percentage points to classify an outcome as a definite effect rather than statistical noise.

Claude Haiku 5.5 merged patch-labeled updates without inspecting the new code in 11 of 30 runs, representing a 37 percent failure rate, while unchecked merges dropped to 2 of 30 runs under a major label. Across all six evaluated models, unchecked merges averaged 13 percent for patch labels, 9 percent for minor labels, and 4 percent for major releases. Claude Opus 5.5, GPT-6.1 Sol, and GPT-6.1 Terra maintained zero unchecked merges across all review scenarios.

03Toolchain deployment and runtime model selection

Teams deploying GitHub Copilot CLI or switching upstream agent models can update their local tooling and specify target models directly in command-line environments.

npm install -g @github/copilot-cli

Developers can switch active models inside agent session prompts or pass pinned arguments during execution:

/model gpt-6.1-terra

copilot --model gpt-6.1-terra

copilot config set default-model gpt-6.1-terra

04Production failures and semantic version trust

Skipping verification checks directly led to production breakages in the experimental benchmark. Out of 270 reviews featuring breaking changes, agents merged broken updates in 13 instances.

Four of those 13 bad merges broke the main branch, consisting of two failures by Claude Haiku 5.5 and two by GPT-6 Luna. Every broken main branch occurred exclusively under patch or minor labels.

The data indicates that models reflect human habits embedded in training data: major releases trigger caution, while patch versions are treated as inherently low risk. Even models that checked every update talked about compatibility issues more frequently when encountering a major release. When library authors violate semantic versioning by placing breaking changes inside patch releases, agents relying on heuristic trust fail to catch the error.

05Pipeline verification and benchmark evaluation controls

Engineering organizations cannot rely on AI coding agents to independently decide when verification is necessary. Development pipelines must enforce mandatory test runs and validation checks on continuous integration servers for every pull request, ensuring patch bumps receive identical scrutiny. Relying on an agent's internal judgment creates silent vulnerabilities whenever smaller, faster models handle routine maintenance branches.

Benchmark creators must also treat semantic versioning as a hidden variable in agent evaluations. Testing an agent exclusively on major updates overestimates its caution, while testing only on patch bumps underestimates its capabilities. Trajectory logs must be inspected directly, because an agent that skips verification and bumps a benign patch appears successful without performing necessary diligence.