Microsoft Foundry Agent Runtime Architecture [2026]

Microsoft Foundry Agent Service is no longer just a portal experience for building chatbots. As of July 01, 2026, it is best understood as a managed runtime for agents that need state, tools, identity, observability, and enterprise controls. The design question for engineering teams is not whether an agent can call a model. It is whether the runtime can remember safely, call tools predictably, expose failures, and fit into an Azure governance model.

The Lead

The platform gives developers three operating patterns. Prompt agents are authored in the portal, SDKs, or REST and run on Microsoft-managed infrastructure. Hosted agents let teams package their own orchestration code as containers while Foundry handles endpoint hosting, scaling, identity, session state, and observability. Existing applications can also call the Responses API directly to use Foundry models and platform tools without moving the whole agent process.

That split matters because agent architecture has two layers: the reasoning loop and the operating envelope. The reasoning loop chooses a model, instructions, context, and tools. The operating envelope determines where state lives, which identity calls downstream systems, how traces are stored, who can publish an agent, and what happens when memory or tool output is wrong.

• Prompt agents minimize custom runtime work for teams that can express behavior through instructions and tool configuration.
• Hosted agents preserve framework choice for teams using Microsoft Agent Framework, Semantic Kernel, LangGraph, OpenAI Agents SDK, or custom code.
• Responses API gives external apps a project-scoped entry point into models and Foundry platform tools.

Architecture & Implementation

Runtime primitives: agents, conversations, responses

Foundry Agent Service uses three core runtime components: agents, conversations, and responses. An agent is the persisted orchestration definition: model, instructions, tools, parameters, code, and optional safety or governance settings. A conversation persists history across turns. A response is the generated output for an input, optionally involving tool calls, retrieval, streaming, or background execution.

The loop is simple but consequential. User input and optional conversation history feed response generation. The model may call tools. Tool results flow back into the context. The response can append items to the conversation, and those items become part of future turns. In production, every part of that loop needs controls because a tool call can mutate business state and a remembered fact can influence future decisions.

Memory as managed long-term state

Memory is currently a preview capability, and teams should treat it as production-sensitive state even when they are still prototyping. Microsoft describes Foundry memory as long-term memory that extracts information from conversations, consolidates it into durable knowledge, and retrieves relevant memories in later sessions. It is not the same as simply replaying chat history into a larger context window.

• User profile memory stores durable preferences and personal context, such as language, product defaults, or accessibility needs.
• Chat summary memory stores distilled summaries of prior conversation topics and threads.
• Procedural memory stores reusable routines and operating patterns inferred from prior interactions.
• Item-level CRUD lets developers create, read, update, list, and delete individual memory records.
• Store-level TTL sets default retention for newly created memory entries.

The key engineering risk is memory corruption. Prompt injection can produce one bad answer, but corrupted memory can bias future answers across sessions. Foundry memory guidance explicitly calls out prompt injection and memory corruption, so memory writes should be treated like privileged state mutation. For test fixtures and trace samples, teams should remove identifiers before sharing them; TechBytes’ Data Masking Tool is useful when preparing non-production examples that still need realistic structure.

Tools and Toolbox

Tools are the action surface of the runtime. Foundry supports built-in tools such as Web Search, Code Interpreter, File Search, Azure AI Search, Azure Functions, and function calling. It also supports custom tool patterns including Model Context Protocol, OpenAPI 3.0, OpenAPI 3.1, Toolbox, and Agent-to-Agent communication in preview.

The important implementation pattern is centralization. A toolbox bundles tools into a single MCP-compatible endpoint, supports versioning, and centralizes authentication. That lets platform teams manage credentials, token refresh, and policy enforcement once instead of duplicating tool wiring across every agent definition.

• Use tool_choice set to auto when the model should decide whether a tool is needed.
• Use tool_choice set to required when the response must be grounded in tool output.
• Use tool_choice set to none when deterministic no-tool behavior is required.
• Keep tool instructions specific, especially when File Search, Web Search, and internal APIs overlap.
• Treat tool outputs as untrusted input before they drive writes, approvals, or external actions.

Identity and governance boundary

Agent identity is where Foundry becomes an enterprise runtime instead of a developer demo. Foundry can provision Microsoft Entra agent identities so agents authenticate to downstream systems without embedding secrets in prompts, tool arguments, or connection strings. At runtime, Agent Service handles token exchange and downstream services still enforce RBAC or their own authorization policies.

Foundry’s resource model also separates governance concerns. A top-level Foundry resource scopes management operations, while projects isolate development assets such as files, agents, evaluations, and access control. This gives platform teams a workable hierarchy for policy, networking, model deployment, and project-level experimentation.

Benchmarks & Metrics

There is no single meaningful benchmark for an agent runtime because agent workloads vary by model, tool latency, memory usage, retrieval quality, and human approval flow. The right measurement approach is a scorecard that separates runtime health from task quality.

• Trace freshness: Microsoft says traces for framework integrations typically appear in the Foundry portal within 2-5 minutes after execution.
• Task adherence: Microsoft’s evaluation guidance gives 85% task adherence as an example release threshold, not a universal target.
• Tool success rate: Track successful tool invocations, empty results, schema errors, retries, and authorization failures separately.
• Latency distribution: Measure p50, p95, and p99 response latency, plus per-tool latency when actions fan out.
• Cost per completed task: Count model tokens, embedding calls, Code Interpreter sessions, search calls, and Azure Monitor retention costs.
• Memory precision: Audit remembered facts for correctness, staleness, user scope, retention status, and delete compliance.

Observability is strongest when traces, evaluations, and replay data are connected. Foundry tracing captures model calls, tool invocations, agent decision flow, inputs, outputs, retries, latency, and cost signals. For popular frameworks, Microsoft provides OpenTelemetry-based integrations, with native behavior for Microsoft Agent Framework and Semantic Kernel when tracing is enabled on the project.

scorecard:
  runtime:
    - p95_response_latency
    - tool_call_success_rate
    - retry_rate
    - trace_ingestion_delay
  quality:
    - task_adherence
    - groundedness
    - safety
    - memory_accuracy
  governance:
    - agent_identity_coverage
    - privileged_tool_calls
    - retained_trace_content
    - expired_memory_deletes

Security teams should pay close attention to content recording. Microsoft’s tracing guidance warns that traces can capture sensitive user inputs, model outputs, tool arguments, and tool results. Content recording is useful during debugging, but it should be disabled or tightly governed in production environments where prompts or tools carry regulated data.

Strategic Impact

Foundry Agent Service shifts the build-versus-buy line for agent infrastructure. In 2024 and 2025, many teams built custom wrappers around LLM APIs to manage conversation state, vector search, tool calls, identity, retry behavior, logging, and dashboards. By mid-2026, Microsoft is packaging much of that runtime surface into Azure-native infrastructure.

That has practical consequences for platform teams:

• Standardization: Teams can converge on shared agent runtime primitives without forcing one orchestration framework.
• Auditability: Agent identities, traces, and project scopes make agent activity easier to inventory and review.
• Reuse: Toolboxes and private tool catalogs let organizations publish approved capabilities once and reuse them across agents.
• Distribution: Published agents can reach custom apps, Microsoft 365 Copilot, Teams, and Entra Agent Registry depending on configuration.
• Risk reduction: RBAC, private networking, content safety, evaluations, and observability give security reviewers concrete controls to inspect.

The main tradeoff is platform coupling. Teams that adopt Foundry deeply gain Azure-native governance and managed operations, but they also need to design for model-region availability, tool-region support, Azure Monitor costs, preview feature limits, and Microsoft’s release cadence. For regulated organizations already standardized on Azure, that coupling may be an advantage. For multi-cloud agent platforms, the better pattern may be to use Foundry where Azure governance is required and keep orchestration abstractions portable at the application layer.

Road Ahead

The runtime is moving toward a clearer separation between agent logic and governed capability surfaces. Memory, Toolbox, OpenTelemetry tracing, Entra agent identities, and publishing protocols all point in the same direction: agents are becoming managed software actors, not just prompt templates.

The next engineering frontier is policy at the point of action. It is not enough to know that an agent called a tool. Enterprises need to know whether the call was permitted for that user, tenant, data class, environment, and workflow state. Expect more investment in AI gateways, MCP governance, approval policies, replayable traces, and evaluation gates that run before deployment and continuously after release.

For teams adopting Foundry now, the pragmatic path is staged. Start with a narrow prompt or hosted agent, enable tracing from day one, attach one or two governed tools, and define an evaluation set before expanding memory or distribution. Add memory only when personalization or workflow continuity justifies the added state risk. Treat every external tool as a data boundary. Use agent identities for downstream access instead of shared secrets.

The platform’s direction is clear: Microsoft wants Foundry to be the control plane where enterprise agents are built, observed, governed, and distributed. The winning implementations will be the ones that treat that control plane as architecture, not decoration.