Build a model improvement flywheel with Amazon SageMaker HyperPod Data

Amazon Web Services introduced Data Capture for Amazon SageMaker HyperPod, a built-in feature that records live model inputs and outputs to Amazon S3. According to AWS Machine Learning Blog's report, the mechanism logs real-world inference payloads across three tiers while preserving model availability.

This walkthrough explains how engineering teams capture production traffic, isolate blind spots with Amazon Athena, and fine-tune models using the HyperPod training operator on a single cluster. The architecture targets machine learning engineers operating foundation models on Amazon Elastic Kubernetes Service who require automated retraining pipelines.

01Three tier data capture architecture

HyperPod Data Capture operates across three collection layers that can be activated together in a single configuration block. Tier 1 captures full request and response payloads through native SageMaker endpoints for compliance audits. Tier 2 uses Application Load Balancer access logs to record client IP addresses, user agents, timings, and status codes.

Tier 3 operates at the model pod level, capturing complete inference payloads along with enriched event metadata. The system manages this tier using dedicated sidecars, including an NGINX proxy, an OpenTelemetry collector, and an S3 uploader. Because Tiers 1 and 3 both store full payloads, Tier 3 serves as the primary tier for training collection.

Production payloads frequently contain sensitive details, including order numbers, names, and mailing addresses. Operators running Tier 3 must implement security controls prior to ingesting payloads into retraining pipelines. Protection strategies include custom sampling percentages, AWS Key Management Service encryption keys, S3 lifecycle rules, and redaction through Amazon Comprehend.

02Cluster prerequisites and dual GPU configuration

Running the complete improvement pipeline on a single SageMaker HyperPod cluster requires Amazon Elastic Kubernetes Service alongside two specialized operators. Administrators must install the HyperPod inference operator to manage deployments and the training operator add-on to schedule retraining workloads. Cluster health can be verified using kubectl by checking the custom resource definitions.

Hardware provisioning requires two distinct NVIDIA graphics processing units running across the cluster nodes. The inference endpoint deploys on an ml.g6e.8xlarge instance to serve incoming traffic, while the fine-tuning workload executes on an ml.g6e.xlarge instance. Both node types provide an NVIDIA L40S GPU equipped with 48 gigabytes of memory.

Rigid GPU allocation rules govern scheduling across the underlying Kubernetes infrastructure. Because the inference pod claims its dedicated L40S GPU for the entire lifetime of the active endpoint, a single-GPU cluster cannot run retraining jobs. Without a second available GPU, the training job enters a pending state.

03Five step continuous improvement flywheel

The operational workflow moves through five synchronized stages that run across the shared HyperPod environment. The pipeline begins by staging the Qwen2.5-3B-Instruct base model in Amazon S3, selected because the model fits on an L40S GPU and fine-tunes quickly. The inference operator serves the model while recording production traffic directly into S3 prefixes.

Inference traffic flows through an NGINX proxy container that copies each transaction to an OpenTelemetry collector sidecar. The collector aggregates records into batches and delivers the files to an Amazon S3 bucket mounted through a Mountpoint-S3 volume. Amazon Athena then queries the captured records directly in place without data movement.

Once Athena queries identify classification failures on newly released products, a SQL rule automatically assigns supervisory labels to the stored requests. The training operator launches a HyperPodPyTorchJob on the second GPU node to fine-tune the model against the new dataset. The updated model then deploys to a secondary endpoint alongside active capture.

04Identity permissions and driver access roles

The end-to-end data pipeline relies on four coordinated IAM identities to enforce least-privilege boundaries between serving, storage, and training. The Mountpoint-S3 Container Storage Interface driver operates under its own IAM roles for service accounts identity rather than pod credentials. This driver role requires s3:PutObject, s3:AbortMultipartUpload, and s3:DeleteObject permissions on the capture bucket.

Missing driver permissions cause silent storage failures where inference endpoints process live requests normally while no captured records reach Amazon S3. The inference operator execution role also requires s3:PutObject permissions alongside KMS decryption and data key generation rights. Administrators investigating missing capture files must verify the CSI driver role first.

The remaining pipeline identities handle model retraining and analysis queries within the environment. The training pod authenticates through an EKS Pod Identity service account, holding read privileges on base weights and capture data. Amazon Athena requires read access to the capture prefix and dedicated write access for query results.

05Model pod deployment and buffer parameters

Engineers configure Tier 3 capture by embedding a dedicated dataCapture block inside standard vLLM InferenceEndpointConfig manifests. The deployment executes the vLLM container image over port 8000, passing runtime parameters including a 2048 token context limit and a 90 percent memory utilization ceiling. Pod resources require two CPU cores, 16 gigabytes of memory, and one GPU.

The dataCapture configuration defines an S3 destination path and enables Tier 3 collection with an initial sampling percentage set to 100 percent. The buffer configuration batches records to manage upload overhead, specifying a threshold of 50 records and a 60-second flush interval. Payloads up to 1024 kilobytes are accepted to preserve multi-turn prompt transcripts.

Operators can tune collection parameters directly in the manifest to match production throughput and storage budgets. Sampling can be reduced to 50 percent, and buffer sizes can be expanded to 100 records to lower S3 request volume during traffic spikes. The configuration also supports restricting captured content types and assigning specific KMS keys.