Not All Model Upgrades Are Upgrades [Evals]
Dillip Chowdary
July 21, 2026 · 7 min read · Source: Microsoft Developer Blog
Bottom Line
A newer model with a cheaper rate card can still be more expensive. Microsoft measured Claude Sonnet 5 vs 4.6 on agent tasks and saw median token multipliers up to ~12× on architecture work — enough to erase a 33% per-token discount.
Key Takeaways
- ›Sonnet 5 rate card in the study: $2/$10 vs Sonnet 4.6 $3/$15 (input/output per 1M) — Sonnet 5 cheaper per token.
- ›Architecture tasks: ~12× more tokens at the median for Sonnet 5; some scenarios much higher.
- ›Code upgrade tasks: ~10× token gap; example cost $2.01 vs $0.55 per run in the write-up.
- ›Evaluate upgrades with fixed harness + fixed tasks; never with “feels smarter.”
- ›Gate production model flips on cost-per-success, not benchmark screenshots.
Microsoft’s Agent Experience team published “Not all model upgrades are upgrades” with hard numbers from Claude Sonnet 4.6 vs Sonnet 5 agent runs.
Rate card vs bill
| Metric | Sonnet 4.6 | Sonnet 5 |
|---|---|---|
| Input / 1M | $3.00 | $2.00 |
| Cached input | $0.30 | $0.20 |
| Output / 1M | $15.00 | $10.00 |
| Architecture tokens (median) | baseline | ~12× higher |
| Example code-upgrade $ / run | $0.55 | $2.01 |
Method (copy this)
- Fixed product surface (GitHub Copilot Chat in VS Code)
- Two task classes: architecture/design grounded in docs, and SPFx project upgrades
- Many repeats per scenario to capture variance
- Compare tokens and dollars, not only pass rates
When to choose each
- Choose the newer model when: quality or autonomy gains outweigh measured $/success and you have budget headroom.
- Stay on the older model when: token multipliers destroy unit economics or the new model thrashes tools without better outcomes.
Harness controls that reduce upgrade risk
- Cap max tool calls and max dollars per task
- Force plan → execute → verify phases
- Cache prompts and shared context aggressively
- Run dual-model shadow traffic for a week before flipping defaults
Primary source:
Microsoft Developer Blog →
Verify claims against the original before changing production systems.