Microsoft Three-Layer LLM Routing Architecture for AI Agents on AKS

Microsoft has released a reference architecture for routing traffic from AI agents on Azure Kubernetes Service. Covered by Claudio Masolo on InfoQ, the design frames agent inference as three linked decisions rather than a single gateway hop: which model answers a call, how that call is managed end to end, and which GPU replica actually runs it.

The three-layer split separates model selection from request control and from GPU placement. Model routing decides which LLM should handle a given agent call. Call management covers how the request is admitted, tracked, and completed across the path. Replica routing chooses which GPU-backed instance serves the work on AKS. Keeping those concerns apart lets each layer change without forcing a full rewrite of the path from agent to hardware.

For engineers building multi-model agents on Kubernetes, that separation matches how production systems already fail and scale. Model choice, policy and retries, and GPU capacity do not move on the same schedule. A reference layout that makes those seams explicit reduces ad hoc sidecars and one-off proxies when teams add models, tighten control over calls, or grow GPU pools on AKS.

The design sits in a market where agent stacks increasingly mix models, tools, and cluster-backed inference instead of a single static endpoint. A published AKS-oriented pattern from Microsoft gives platform and ML teams a shared vocabulary for the same three decisions vendors and open-source gateways solve in different packages. It also makes Azure’s agent-on-Kubernetes story easier to compare against other cloud and self-managed routing approaches that collapse model, control, and placement into fewer layers.

What to watch next is how teams adopt the three choices in real agent services: whether model routing stays policy-driven at the edge, how call management is owned relative to the agent runtime, and whether GPU replica selection stays inside the cluster scheduler or moves into a dedicated inference router. Operators should map existing traffic paths to those three layers before adding more models or GPU capacity so each new route has a clear owner.