Foundry Agent Optimizer scores hosted agents, rewrites prompts, creates skills, and compares models before deployment. Read the rollout guide now.
What the optimizer is for
Foundry Agent Optimizer is a pre-deployment layer for hosted AI agents. Instead of shipping a prompt and model pair on intuition alone, it scores the agent, rewrites prompts, creates skills the agent can call, and compares candidate models so you can see tradeoffs before anything hits production. The point is not a prettier dashboard. It is a repeatable path from “this agent works in a demo” to “this agent is ready to host.”
Hosted agents fail in ways chat demos hide: brittle instructions, missing skills, and model choices that look fine on a single example but break under real tool use. The optimizer treats those as first-class inputs. You bring the agent definition and the behaviors you care about; it returns scores, revised prompts, generated skills, and a model comparison you can act on before deployment.
Scoring hosted agents before they go live
Scoring works best when it maps to outcomes you would actually reject in production. Frame checks around task completion, instruction following, safe refusal when tools or data are out of scope, and consistent use of the skills you expect the agent to invoke. A single composite score is useful for ranking candidates; breakdowns matter more for diagnosis—if the agent scores poorly on tool selection but well on plain Q&A, the fix is not “pick a smarter model” by default.
Run the same score set against every revision you care about: the original agent, the prompt-rewritten variant, and each model under comparison. That keeps the comparison honest. If scores only improve after you change the evaluation criteria mid-flight, you did not improve the agent—you moved the goalposts. Lock the score definitions early, then let the optimizer rewrite and re-score against that fixed bar.
Prompt rewrites, skills, and model comparison
Prompt rewriting should tighten role, constraints, and tool policy without smuggling in hidden requirements the score suite never tests. Prefer short, explicit rules over long narrative instructions. Call out when the agent must use a skill, when it must ask for missing inputs, and when it must refuse. After a rewrite, re-score immediately; if the score does not move, the rewrite was style, not substance.
Skill creation fills the gap between “the model knows how in principle” and “the agent can do this reliably under host constraints.” Skills should be small, named capabilities with clear inputs and outputs—things like structured lookup, validation steps, or multi-step workflows the base prompt alone keeps getting wrong. Model comparison then sits on top of the same prompts and skills: hold the agent definition fixed and swap models so differences reflect the model, not a drifting prompt. Rank by score and by operational fit—latency, cost envelope, and how cleanly the model follows skill boundaries—not by a single demo transcript.
- Score first against fixed criteria tied to production failures.
- Rewrite prompts only when re-scores improve on those criteria.
- Add skills for repeated, structured gaps the prompt cannot stabilize.
- Compare models with prompts and skills held constant.
- Deploy only after the chosen combination clears your score bar.
How to roll this out without thrash
Treat the rollout guide as the sequence of record: define score criteria, baseline the current hosted agent, run rewrites and skill generation, compare models, then promote the winner. Keep one staging host for optimizer output and one production host that only receives versions that passed the score bar. Document which prompt version, which skills, and which model won the comparison so the next change has a clear parent to regress against.
Avoid optimizing forever in a loop. Pick a score threshold that matches the risk of the agent’s tools and data access, run a small number of rewrite and skill iterations, choose a model from the comparison, and deploy. After go-live, feed real failure modes back into the score suite rather than inventing new criteria on every incident. That keeps Foundry Agent Optimizer a gate before deployment, not an endless workshop after it.