Test Agent UX Changes Before You Ship [2026]
Dillip Chowdary
July 21, 2026 · 5 min read · Source: Microsoft Developer Blog
Bottom Line
Do not ship “obvious” agent UX fixes first. Microsoft’s SPFx work shows agents often ignore advisory docs after forming a plan from training data — only evals reveal whether a change works.
Key Takeaways
- ›Every improvement is a hypothesis; LLMs produce counterintuitive results regularly.
- ›In SPFx upgrade scenarios, a docs tip recommending a CLI was ignored in every run.
- ›Agents often lock a plan from pretraining; later docs confirm package names but ignore advice.
- ›Test doc/CLI format changes offline with evals before production ship.
- ›Connect this to dual-model cost evals and multi-turn agent eval design.
Microsoft’s AX post warns: after you build an eval baseline, do not jump straight to shipping hypotheses.
The SPFx story
SharePoint Framework upgrades are incremental. Agents needed to discover a CLI from documentation that included a tip recommending the CLI. The agent ignored it in every run. It had already formed a plan from training data; the tip was treated as optional noise.
Design implication
- Prefer hard constraints (tool availability, failing tests) over soft advice in docs
- Put critical commands in machine-readable surfaces agents must fetch (schemas, CLI --help JSON)
- Measure before/after with multi-turn evals, not anecdote
Recommended workflow
- Write the hypothesis (“docs callout → CLI usage”)
- Encode success in automated graders
- Run N trials offline
- Ship only if the metric moves without collateral regressions
Primary source: Microsoft Developer Blog →
Verify claims against the original before changing production systems.