Build a Physical AI model factory with NVIDIA Cosmos 3 on SageMaker

What happened

AWS and NVIDIA have detailed a reference architecture for building Physical AI systems on Amazon SageMaker HyperPod, showing how teams can run the full model factory — synthetic data generation, post-training, and closed-loop evaluation — using NVIDIA Cosmos 3 on a persistent cluster backed by Amazon EKS. The announcement comes from the AWS Machine Learning Blog and targets ML platform engineers and research teams who are moving Physical AI workloads from experimental runs into production-grade pipelines.

This piece breaks down what the pipeline actually involves, why a persistent cluster changes the economics of Physical AI training, and what builders should verify before committing GPU hours to a continuous evaluation loop. Whether you are running robotics simulations, autonomous vehicle perception models, or any other embodied-intelligence workload, the architecture described here is designed to keep GPUs working rather than idle between stages.

AWS published a reference post showing how to chain the three core stages of a Physical AI model factory — synthetic data generation, post-training, and closed-loop evaluation — on Amazon SageMaker HyperPod running on Amazon EKS. The workload uses NVIDIA Cosmos 3 as the world model at the center of that pipeline. Rather than treating each stage as an isolated job submitted to a cluster that spins up and tears down, the approach anchors everything to a persistent, resilient HyperPod cluster so that the infrastructure stays live across all three stages of the factory.

How it works

The design philosophy treats GPU goodput — the fraction of GPU time spent on productive compute rather than on restarts, checkpointing delays, or node failures — as the primary operational metric. SageMaker HyperPod is positioned to protect goodput through its resilience features, which detect node failures and attempt automatic recovery without requiring the full training job to restart from scratch.

The pipeline opens with synthetic data generation, where NVIDIA Cosmos 3 produces simulation data that Physical AI models need in large volumes because real-world sensor data is expensive and slow to collect. That synthetic data feeds a post-training stage where a base model is fine-tuned or aligned using the generated scenarios. Closed-loop evaluation then places the model back inside the simulator, measures whether it performs correctly, and feeds results back into the data generation stage to close the loop.

Why it matters

Running this on SageMaker HyperPod on EKS means the orchestration layer is Kubernetes, giving teams access to standard job scheduling while HyperPod adds health monitoring and automatic node replacement on top. The persistent cluster design matters because spinning down and relaunching GPU clusters between pipeline stages introduces latency and provisioning uncertainty; keeping the cluster live lets each stage hand off directly to the next without cold-start overhead.

Physical AI is distinct from language or vision AI in one important way: the model must interact with a simulated or real physical environment, which means evaluation is not a static benchmark but a dynamic loop. Closed-loop evaluation with a world model like Cosmos 3 makes that loop tractable at scale, but it is computationally expensive. The ability to sustain that loop on resilient infrastructure without losing hours to node failures or manual restarts directly changes what iteration speed is possible.

GPU goodput is the right metric to optimize here because GPU clusters for Physical AI workloads are among the most expensive line items a team will run. A cluster that is provisioned but idle between stages, or that loses partial progress to a node failure and must re-run data generation, burns budget without advancing the model. Surfacing goodput as a first-class concern rather than an afterthought pushes teams to measure and protect it systematically.

Who is affected

The primary audience is ML platform teams building or maintaining infrastructure for Physical AI workloads — robotics, autonomous vehicles, industrial automation, and any domain where a model must reason about or act within a physical environment. These teams are often caught between research teams that want fast iteration and finance teams watching GPU spend, and a goodput-oriented architecture speaks directly to both concerns.

Secondary stakeholders are the ML engineers doing the actual post-training and evaluation work. A persistent cluster with automatic recovery means fewer manual interventions when a node goes down mid-training, which reduces the operational burden on individuals who would otherwise need to monitor jobs and manually requeue failed runs. Teams already using SageMaker HyperPod for large language model training will find the EKS integration familiar but may need to adapt their job definitions to handle the three-stage handoff.

What to watch next

Builders evaluating this architecture should verify that their Cosmos 3 integration supports the synthetic data formats their downstream post-training stage expects, since format mismatches between generation and training are a common failure point in multi-stage pipelines. They should also establish a baseline goodput measurement before moving to persistent clusters so the improvement is quantifiable rather than assumed.

On the infrastructure side, the behavior of HyperPod automatic recovery under the specific failure modes common to long-running simulation workloads — GPU memory errors, network fabric interruptions, storage throughput saturation — is worth stress-testing in a staging environment. AWS documentation for HyperPod on EKS and NVIDIA documentation for Cosmos 3 are the authoritative sources to track as both products continue to develop; any changes to the Cosmos 3 world model's API surface or HyperPod's recovery semantics will affect how the three stages hand off to each other.