Not all model upgrades are upgrades — Tech Bytes analysis

A new model ships with lower per-token pricing and stronger benchmark scores. Teams switch. A week later someone asks why the agent is burning 12x more tokens on the same task and returning worse output. That pattern is the subject of Microsoft Dev Blogs’ post Not all model upgrades are upgrades, which reports on agent behavior under Claude Sonnet 4.6 versus Claude Sonnet 5.

The evaluation used GitHub Copilot and ran 150 agent tasks across 15 scenarios on those two models. The setup isolates the upgrade decision engineers actually make: same agent surface, same task set, different model. The headline result is not that Sonnet 5 always fails. It is that lower unit price and better leaderboard numbers can still produce higher total token spend and weaker task quality when the model is used as an agent.

For builders, the gap that matters is between per-token cost and end-to-end cost. Agents loop: plan, tool call, observe, retry. A model that is cheaper per token but more verbose, more tool-heavy, or less decisive can multiply total tokens. In this write-up, that multiplier reaches 12x on the same task while quality drops. Benchmark wins that look like free upgrades stop looking free once agent trajectories, not single-shot completions, are the unit of work.

That finding cuts against the market habit of treating each new Claude Sonnet release as an automatic default. Pricing and public scores still drive migration decisions. Agent workloads on products like GitHub Copilot are sensitive to how a model explores, recovers, and terminates. A cheaper, higher-scoring model can lose on total cost and outcome quality if those behaviors change for the worse under multi-step tool use.

The practical takeaway is to treat model upgrades as workload experiments, not table stakes. Before flipping defaults, re-run the agent scenarios that matter—tool-heavy flows, long trajectories, recovery after failed calls—and measure total tokens and task success, not only list price and benchmark deltas. Watch next for whether teams keep dual-model setups, route by scenario, or gate upgrades on agent-level evals instead of leaderboard and pricing alone.