Home / Blog / How Salesforce is advancing global-scale model serving with…
Tech News

How Salesforce is advancing global-scale model serving with cross-Region

Salesforce deployed SageMaker AI cross-Region inference routing across 30+ global Regions and hundreds of AWS accounts to absorb heavy model spikes.

By Dillip Chowdary • Nov 18, 2026 • Source: AWS Machine Learning Blog

How Salesforce is advancing global-scale model serving with cross-Region
Image: AWS Machine Learning Blog

At a glance

Product
SageMaker AI inference router
Customer
Salesforce Agentforce Foundation
Deployment scale
30+ global Regions, hundreds of accounts
Failover latency
Begins within ~1 minute
Retry limit
1 automatic retry per request
Telemetry fields
InvokedRoutedResourceArn, RoutingReason

Salesforce deployed cross-Region inference routing in Amazon SageMaker AI to manage large-scale model serving across more than thirty global Regions and hundreds of AWS accounts. The Agentforce Foundation team adopted the capability to eliminate capacity shortfalls for internal and open source models, as detailed in AWS Machine Learning Blog's report.

This architectural report explains how Salesforce transitioned from custom gateway routing to SageMaker AI's capacity-aware infrastructure while strictly preserving regional data residency. The analysis outlines practical deployment steps, cross-account security policies, and operational telemetry for engineering leads, cloud architects, and machine learning platform operators managing global inference fleets.

01Capacity fragmentation across thirty global regions

The Agentforce Foundation team serves generative models across thirty global Regions, where localized customer traffic surges frequently exhausted computing capacity while other data centers sat idle. Provisioning every regional cluster for maximum overlapping peaks proved prohibitively expensive. When traffic spikes created local capacity shortfalls, the team faced direct availability risks for Agentforce and the Salesforce Trust Layer.

Salesforce previously routed overflow traffic through its Internal Gateway using basic health checks and custom failover logic. That initial architecture lacked real-time visibility into remote endpoint capacity, leaving the gateway unable to verify if receiving clusters could handle added volume. Cross-Region requests also exited private networks through NAT and internet gateways, adding security overhead and creating risks of cascading failures during outages.

02Managed routing targets and health signals

Amazon SageMaker AI provides the inference router as a regional resource that replaces direct endpoint invocation calls across client applications. Callers invoke the router by supplying a logical routing target name, such as a toxicity classifier, which maps to specific endpoint Amazon Resource Names and inference components. Each router isolates routing targets so that an operational issue affecting one model never disrupts traffic for other services.

The router favors local endpoints but continuously evaluates target health using platform faults, container 5xx errors, throttles, and timeouts. When errors persist, the router shifts traffic toward healthy endpoints within about a minute, though lower-traffic destinations can take several minutes. After an endpoint recovers, traffic returns gradually over several minutes rather than shifting abruptly, preventing destructive oscillations between Regions.

Note: Client errors like invalid inputs and model latency do not trigger traffic shifts. The router considers execution delays representative of long prompt contexts rather than infrastructure failures, and it does not deploy models or scale instances.

Advertisement

Tech Pulse Daily

Get tomorrow's pulse first

Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.

03PrivateLink integration and data residency guards

Salesforce integrated the router into its Internal Gateway egress path, inserting the invocation call immediately after data residency checks and before response processing. All client traffic flows through AWS PrivateLink VPC endpoints, eliminating previous reliance on public internet routing. By embedding the router directly into the existing gateway pipeline, the team adopted managed routing without altering client application interfaces.

The Internal Gateway retains full authority over data residency enforcement before dispatching any inference payload. When a request carries a mandatory data residency requirement, the gateway pins the call to a local-only routing target that lacks remote destinations. This architectural boundary ensures that regulated customer data never leaves its designated home Region under any failover scenario.

04Cross-account authorization and destination policies

Operating inference routers across distinct AWS accounts requires explicit resource policies on every destination endpoint. The account owning the destination attaches an endpoint policy granting invocation permissions to the account hosting the router. To prevent unauthorized access, the policy enforces an Amazon Resource Name condition restricting invocation rights to the specific designated router instance.

In a production test configuration, Salesforce deployed a router in account 111122223333 in us-west-2 and routed requests to endpoint example-toxicity-use2 in account 444455556666 in us-east-2. The policy on the destination endpoint permitted sagemaker:InvokeEndpoint actions while verifying that the SourceInferenceRouterArn matched example-agent-router. This verified that managed routers reliably dispatch inference calls across both account and geographic perimeters.

  1. Authorize destination Attach a resource-based policy to the destination endpoint granting InvokeEndpoint permissions to the primary router account.
  2. Enforce source condition Add an ArnEquals condition matching the SourceInferenceRouterArn to restrict access to the designated router.
  3. Register target destination Add the destination endpoint Amazon Resource Name to the calling account's logical routing target configuration.

05Response telemetry and failover observations

Salesforce captures routing telemetry directly from response headers, treating routing metadata as first-class operational signals. The Internal Gateway records InvokedRoutedResourceArn and RoutingReason on the same structured log line as existing request and trace identifiers. These attributes feed application performance monitoring tools and join with Amazon CloudWatch metrics during diagnostic investigations.

The RoutingReason field informs operators why an execution path was chosen, logging values such as local-preferred, failover, capacity-available, or only-healthy. Because the returned ARN explicitly names the responding account and Region, engineers can immediately identify where each payload ran. Senior Director Amit Gupta and software engineer Mark Wang confirmed that this telemetry provides definitive visibility during operational shifts.

Frequently asked questions

How does the SageMaker AI inference router protect customer data residency?

Salesforce evaluates data residency flags in its Internal Gateway before sending requests to the router. Requests requiring regional residency are pinned to an in-Region routing target with local endpoints, ensuring traffic never crosses geographic borders.

What errors trigger cross-Region traffic shifts in SageMaker AI?

The router diverts traffic when an endpoint experiences platform faults, container 5xx errors, throttles, or timeouts. Client errors like invalid inputs and model latency do not trigger failovers, as latency often reflects prompt length rather than infrastructure failure.

How does an endpoint authorize a router from another AWS account?

The account owning the destination endpoint attaches a resource-based policy granting InvokeEndpoint access to the router account. The policy enforces an ArnEquals condition that restricts access exclusively to the specific SourceInferenceRouterArn.

Sources

Dillip Chowdary

Author

Dillip Chowdary

Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.

Related on Tech Bytes

Advertisement

5-min tech signal

Weekday briefing for engineers who skip the noise.

No spam · Unsubscribe anytime

Advertisement

✈️ CareerPilot

Your AI job-search copilot

Match your resume against live Ashby, Greenhouse & Lever openings — fit scores, job-specific resume optimization and email alerts.

Find matching jobs →

Free Tools

Browse all tools →