Route coding agent requests to the right model on Amazon Bedrock

Amazon Web Services published a reference architecture and deployment sample demonstrating how to route coding agent requests across foundation models on Amazon Bedrock using LiteLLM and NVIDIA NeMo Switchyard. According to AWS Machine Learning Blog's report, the implementation pairs switchyard-server 0.3.0 from crates.io with the LiteLLM v1.102.0 image inside an Amazon ECS task to measure cost per solved task.

This guide examines the AWS Cloud Development Kit infrastructure, network safeguards, dual-model configurations, and benchmark baselines for engineering teams building autonomous developer tooling with Claude Code or OpenAI Codex CLI. It details how cloud architects can direct agent requests between capable reasoning engines and cost-efficient models without introducing judge model latency.

01Cost per solved task and routing strategies

Coding agents run iterative task loops where a single developer request triggers file inspection, code edits, and test runs across dozens of model invocations. Evaluating agent infrastructure purely on price per token ignores the expense of failed attempts, replayed context histories, and retries that never pass verification. Cost per solved task divides total infrastructure spend across all runs by the count of tasks whose tests pass.

NVIDIA NeMo Switchyard is an open source library that helps agents select which model handles each request under administrator-defined strategies. Its LiteLLM integration supports stage routing, which reads recent tool activity and result histories before picking one of two candidate models. The library also supports weighted random routing for traffic splits, while LiteLLM recently introduced an auto-router strategy.

The integration excludes classifier-backed options such as task and composite routing because those mechanisms require intermediate judge model calls. Stage routing relies directly on the conversation history sent by the client agent without adding secondary evaluation latency. This approach provides rapid decisions between a capable model and a lower-cost efficient alternative.

02Private subnet container and load balancer pathing

The architecture deploys LiteLLM alongside the Switchyard plugin inside a single container image hosted on Amazon Elastic Container Service on AWS Fargate. Tasks run inside private subnets across two Availability Zones, while an Application Load Balancer terminates HTTPS using an AWS Certificate Manager certificate. Route 53 directs domain traffic to the load balancer, which targets container port 4000.

An AWS WAF Web Access Control List inspects all incoming traffic using IP reputation lists, known bad input rules, and per-IP rate limits. The load balancer enforces a path allowlist restricted to chat completions, model listing, and health liveliness endpoints. Any requests targeting the LiteLLM admin dashboard, documentation, or internal key management routes receive an automatic 404 response.

Agents send OpenAI-compatible payloads with the model field set to switchyard and authenticate using a shared gateway key. AWS Secrets Manager stores the master key, and container startup scripts prepend the standard sk prefix during initialization. Security groups isolate containers to load balancer traffic, while an egress NAT gateway manages container image pulls.

03Network security and Bedrock runtime VPC endpoints

LiteLLM validates incoming client credentials before passing execution context to the embedded Switchyard plugin to select a destination model. The container task uses an AWS Identity and Access Management role to access Amazon Bedrock Converse and ConverseStream APIs. All model inference requests travel through an Amazon Bedrock Runtime interface endpoint in the Amazon Virtual Private Cloud.

Routing inference traffic through private VPC interface endpoints prevents model payloads and generated code tokens from traversing the public internet. Streaming responses return through LiteLLM to the client with the model field retained as switchyard. The gateway injects an x-litellm-model-name response header that explicitly identifies which underlying foundation model answered the request.

Container operational logs stream to a dedicated Amazon CloudWatch log group retained for thirty days, while Container Insights collects execution metrics. Deployment profile text passes into the task definition through environment variables, which the startup script writes to temporary storage. This design isolates sensitive configuration details while maintaining visibility into agent workloads.

04Deployment prerequisites and AWS CDK tooling

Deploying the sample requires an active non-production AWS account provisioned with model access for Anthropic Claude Opus 5.5 and NVIDIA Nemotron 3 Super 120B on Amazon Bedrock. AWS verified this dual-model pairing specifically within the US East N.

Virginia region. Teams targeting other regions must verify local model availability and tool-calling support before initiating deployments.

Local administrative environments require Node.js 20 or later, npm, Python 3.11 or later, and Docker configured with BuildKit. Administrators must install the AWS Command Line Interface version 2 configured with credentials and an explicit target region bootstrapped for the AWS CDK. Public endpoints require an owned hostname configured with an ACM certificate or Route 53 zone.

The AWS sample repository contains Cloud Development Kit templates, deployment profiles, and benchmark harnesses to evaluate routed agent operations. The implementation installs switchyard-server 0.3.0 from crates.io and executes the published LiteLLM v1.102.0 container image. Because NVIDIA designates Switchyard server 0.3.0 as a demo and evaluation component, organizations must review compliance policies prior to production adoption.

05LiteLLM model groups and baseline configurations

Deployment profiles organize gateway settings by pairing litellm.yaml with switchyard.toml inside dedicated profile directories. The configuration registers the switchyard group with Claude Opus 5.5 listed as the capable candidate and Nemotron 3 Super 120B as the efficient candidate. Stage routing reads candidate order directly, making configuration sequence critical for assigning model roles.

The profile also defines standalone baseline groups for Claude Opus 5.5 and Nemotron 3 Super 120B. Because the native Stage plugin rejects groups that lack two candidates, the image registers a custom wrapper plugin. This wrapper applies Switchyard routing to multi-candidate groups while passing single-model requests through directly to enable comparative benchmarking.

Gateway parameters enforce a 600-second request timeout, two automatic retries, and parameter dropping to strip unsupported fields before calling Bedrock. The benchmark harness measures cost per solved task across the routed switchyard deployment and each single-model baseline group. These comparative metrics reveal whether stage routing lowers overall agent expenses without degrading coding task completion rates.