Microsoft Foundry Agent Runtime Architecture [2026]

Microsoft Foundry Agent Service is no longer just a place to wrap a chat model with a few function calls. As of June 03, 2026, its runtime is a managed control plane for stateful agents: it hosts prompt and hosted agents, persists conversation context, brokers tool authentication, emits traces, and ties the whole system back to Microsoft Entra, Azure RBAC, content filters, and regional data controls.

The Lead

The runtime's architecture matters because agent systems fail differently from ordinary request-response apps. A service can return HTTP 200 while still leaking context, calling the wrong tool, accumulating stale memory, or hiding a slow retrieval step inside a polished answer. Foundry tries to make those moving pieces explicit.

Microsoft's runtime documentation describes three core runtime components: agents, conversations, and responses. An agent defines the model, instructions, and tools. A conversation persists history across turns. A response is the generated output and the record of what happened during processing. Older Azure AI Agent Service material used threads, runs, and messages; the newer Foundry docs shift the mental model toward conversation and response objects.

For engineering teams, the practical question is not whether Foundry can produce a demo. The question is whether its runtime boundaries help you build agents that survive production requirements:

• State must be inspectable, deletable, and regionally governed.
• Tools need scoped authentication instead of shared secrets in prompts.
• Memory needs product rules, not magical recall.
• Tracing must attribute latency and errors to planning, retrieval, tool calls, and final generation.
• Governance has to be enforced before publication, not retrofitted after adoption.

Architecture & Implementation

Runtime Objects

A Foundry agent starts with a definition: model, instructions, tool configuration, and optional safety controls. The runtime hosts both prompt agents and hosted agents, which means teams can start from declarative portal-built agents or bring code-based agents built with frameworks such as Agent Framework or LangGraph.

The operational loop is straightforward, but the data model is richer than a chat transcript:

1. Create or version an agent with instructions, model selection, and tools.
2. Create or reuse a conversation when multi-turn continuity is required.
3. Generate a response from user input, prior context, and available tools.
4. Persist items such as messages, tool calls, tool outputs, and final output content.
5. Inspect traces and metrics to understand cost, latency, and decision flow.

This is the correct abstraction for enterprise agents because the runtime can treat tool outputs and assistant messages as first-class records. That gives later turns a durable context trail without forcing the application to rebuild every prompt from raw logs.

# Conceptual response flow, not a complete SDK sample
conversation = create_conversation(user_id)
response = create_response(
    agent='support-triage-agent',
    conversation=conversation.id,
    input='Investigate this failed deployment'
)

for item in response.items:
    record_trace_item(item.type, item.id)

Memory Is Long-Term State, Not Bigger Context

Microsoft positions Foundry memory as a managed, long-term memory capability in preview. That distinction is important. Conversation history is short-term continuity. Memory stores are designed to retain distilled knowledge across sessions, devices, and workflows.

A good implementation treats memory as a product data store with rules:

• Store durable preferences, profile facts, workflow state, or domain-specific facts that have clear future value.
• Avoid storing raw secrets, regulated identifiers, or transient debugging details as memory.
• Define deletion and correction paths before enabling recall in customer-facing flows.
• Use memory search as a tool with permissions, observability, and review gates.

Memory is powerful because it reduces repetitive user onboarding and lets agents adapt over time. It is risky for the same reason. Teams should run sample prompts, tool arguments, and remembered values through privacy review; for quick local checks before sharing telemetry examples, TechBytes' Data Masking Tool is a useful companion.

Tools And Authentication

Foundry's tool catalog is the runtime's main extension surface. Current Microsoft docs list built-in options such as web search, file search, memory, code interpreter, MCP servers, custom functions, Azure AI Search, Azure Functions, and other connectors. The architectural win is managed authentication.

The tool design choices break down into three buckets:

• Built-in tools: services such as Code Interpreter and File Search can authenticate through Foundry Agent Service without extra application plumbing.
• Project connections: external data sources such as Azure AI Search or SharePoint use configured Foundry project connections.
• MCP servers: Model Context Protocol tools can use API keys, Microsoft Entra authentication with managed identity, or OAuth user-level identity passthrough.

That last point is strategically important. A serious enterprise agent cannot run every action as a single backend superuser. OAuth On-Behalf-Of patterns let the runtime preserve user-level authorization when an agent calls downstream tools, while managed identity supports service-scoped automation.

Benchmarks & Metrics

Microsoft does not publish a single universal benchmark that says Foundry Agent Service adds a fixed amount of latency or supports a fixed throughput across all agents. That would be misleading anyway. Agent performance depends on model choice, token volume, tool count, retrieval latency, memory lookup behavior, streaming strategy, network placement, and downstream API reliability.

The useful benchmark is a workload-specific scorecard. For a Foundry deployment, measure these before and after adding memory or tools:

• End-to-end latency: p50, p95, and p99 response time from user request to usable output.
• Time to first token: especially for streaming interfaces where perceived speed matters.
• Tool-call count: average and worst-case number of tool invocations per completed task.
• Tool latency: p95 latency per tool, with separate buckets for retrieval, code execution, and external APIs.
• Token consumption: input, output, and retrieved-context tokens per successful workflow.
• Memory hit quality: percentage of recalled memories that are relevant, stale, contradictory, or privacy-sensitive.
• Task success rate: completed workflow percentage, not just model completion percentage.
• Escalation and rollback rate: how often humans or compensating actions are required.

Foundry tracing, currently documented as preview, is aimed directly at this measurement problem. The trace model captures user inputs and agent outputs, tool calls and results, token consumption, and time signals such as duration and latency. Traces can be inspected in the Foundry portal and stored in Azure Monitor Application Insights.

For engineers, the most important tracing question is attribution. You need to know whether a slow answer was caused by model generation, memory search, an MCP server, Azure AI Search, code execution, or agent planning. Without that breakdown, teams tend to over-tune prompts when the real issue is a slow tool or a bloated context window.

# Minimal metric shape for an agent response
agent.response.duration_ms
agent.response.input_tokens
agent.response.output_tokens
agent.tool.calls.count
agent.tool.duration_ms{tool_name}
agent.memory.search.count
agent.memory.accepted_items.count
agent.workflow.success

Strategic Impact

Foundry Agent Service is Microsoft's attempt to move agent adoption from experimentation into managed enterprise deployment. The center of gravity is not the model alone. It is the runtime contract around state, identity, tools, observability, publication, and lifecycle control.

That changes how platform teams should think about build-versus-buy decisions:

• Use Foundry when governance, Microsoft Entra integration, regional data controls, and Azure-native observability are more valuable than owning every orchestration primitive.
• Use a lighter custom runtime when the agent is narrow, stateless, and has few privileged tool calls.
• Use framework-hosted agents when your team already has agent logic in code but wants Foundry for deployment, tracing, identity, and publication.
• Keep model selection abstract where possible, because Foundry's promise includes swapping models from the catalog without rewriting agent code.

The governance angle is especially strong. Foundry Agent Service documentation highlights Microsoft Entra identity, Azure RBAC, content filters, virtual network isolation, agent identity, and publication through Microsoft 365 Copilot, Teams, and the Entra Agent Registry. Those are not cosmetic features. They are the controls that determine whether a legal, finance, support, or security team can approve an agent for real users.

There is also an organizational impact. Agents blend application logic, data access, model behavior, and operations. That means ownership cannot sit only with an AI prototyping team. Production Foundry agents need a cross-functional contract:

• Engineering owns runtime behavior, integration tests, deployment, and rollback.
• Security owns identity scopes, network boundaries, and tool permissions.
• Data teams own retrieval indexes, memory policy, and retention rules.
• Product owns user experience, escalation paths, and acceptable automation boundaries.
• Compliance owns evidence, audit trails, and release gates for regulated workflows.

Road Ahead

The next phase for Foundry Agent Service will be judged less by demo breadth and more by operational maturity. The documentation already points to several areas that will define adoption in 2026: memory moving beyond preview, richer tracing for multi-agent workflows, stronger MCP governance, and standardized publishing into Microsoft 365 surfaces.

Teams evaluating Foundry should run a staged rollout rather than a broad platform mandate:

1. Start with a low-risk internal agent that has one or two read-only tools.
2. Add tracing and build a latency, token, and tool-error baseline before expanding scope.
3. Introduce memory only after defining retention, deletion, correction, and review rules.
4. Move from read-only tools to write-capable tools behind explicit approval or policy checks.
5. Publish to business users only after RBAC, content filters, network controls, and audit evidence are in place.

The runtime is promising because it recognizes that agent engineering is no longer just prompt engineering. The durable systems work is around state boundaries, tool contracts, identity propagation, trace semantics, and governance loops. Foundry gives Azure-centered teams a managed place to put those controls. The hard part is still yours: deciding what an agent is allowed to remember, what it is allowed to do, and how you will prove it behaved correctly when the workflow matters.