Multi-Turn Reinforcement Learning on Amazon SageMaker AI: A hands-on guide

Amazon Web Services introduced Multi-Turn Reinforcement Learning on Amazon SageMaker AI, a serverless training capability designed to optimize language models across multi-step agentic workflows. As detailed in AWS Machine Learning Blog's report, the technique trains models using the final outcome of an entire multi-turn interaction rather than intermediate human-labeled steps. By integrating SageMaker AI training jobs with an agent runtime hosted on Amazon Bedrock AgentCore or custom compute, the framework credits decisions made across long sequences of actions. In an experimental demonstration using Wordle, the workflow elevated an open model named gpt-oss-20b from a 61 percent solve rate to an 84 percent solve rate on held-out words at its default reasoning effort.

This hands-on walkthrough examines the complete operational pipeline required to implement Multi-Turn Reinforcement Learning, covering everything from agent deployment to token economics. Machine learning engineers and enterprise developers looking to customize cost-effective models for multi-step internal tasks can use this architecture to teach complex policies without manual labeling. The workflow establishes a split structure where SageMaker AI manages the underlying policy model while Amazon Bedrock AgentCore hosts the interaction environment, tools, and scoring logic. Datasets, agents, and fine-tuned weights remain entirely within the customer's AWS account and are never used to train base foundation models.

A hands-on guide: what actually changed

Traditional supervised fine-tuning requires step-by-step target demonstrations, which fails when agentic tasks offer many valid decision paths toward a final objective. Standard single-turn reinforcement learning also struggles because it cannot assign credit to a specific action executed several steps before a successful or failed outcome. Multi-Turn Reinforcement Learning changes this by allowing the agent to interact freely over multiple turns, scoring the entire sequence only after the episode concludes. Group Relative Policy Optimization calculates updates across several parallel rollouts of the same prompt, adjusting a Low-Rank Adaptation adapter on the policy model based on reward differences.

To demonstrate the capability, the authors implemented a Wordle player where a model receives feedback over six turns to guess a five-letter word. The environment accepts guesses through a guess_word tool, provides colored feedback, and calculates a final outcome-dominant reward. A successful solve awards between 1.05 and 1.55 depending on speed, while a loss incurs negative 0.5, and failing to generate a valid guess yields negative 1.5. In a 100-step training run spanning 105,486 rollouts and 12 hours 40 minutes, the gpt-oss-20b model improved its solve rate from 15 percent to 69 percent at low reasoning effort, proving that small models can master multi-turn tasks cheaply.

A hands-on guide: how it works

The operational architecture splits responsibilities between two systems connected via an OpenAI-compatible endpoint. SageMaker AI coordinates the distributed training job and owns the base model, while Amazon Bedrock AgentCore hosts the agent code, environment rules, and scoring functions in a single Python file. During each rollout, SageMaker passes a training row containing the secret target word to the AgentCore runtime. A Strands agent inside the runtime executes turns against the policy model, feeding game feedback and known letter states back into the conversation context until the episode terminates and a single scalar reward is returned.

The training framework runs on serverless infrastructure, removing the need to manage instance types or manual clustering. Cloud resource bills split across three token categories alongside AgentCore vCPU and memory consumption. In the Wordle demonstration, training consumed 491,616,806 prefill tokens at $0.12 per million, 159,737,098 sample tokens at $0.30 per million, and 643,308,554 train tokens at $0.36 per million. This brought the total SageMaker token cost to $338.50, while AgentCore runtime compute added roughly $79 based on Amazon CloudWatch metrics in the us-east-1 region.

A hands-on guide: why it matters now

Frontier foundation models already handle common puzzles, but deploying high-parameter proprietary models for every enterprise agent creates unsustainable inference latency and API expenses. Multi-Turn Reinforcement Learning allows organizations to take smaller, inexpensive open-weights models and specialize them for unique internal workflows that public models never encountered. By training directly against execution outcomes rather than artificial supervised dialogues, the model discovers its own chain-of-thought strategies and tool-calling patterns without requiring thousands of manually reviewed transcripts.

Controlling reasoning intensity directly influences both token bills and final inference performance. The AgentCore environment supports a REASONING_EFFORT configuration variable that was set to low during the Wordle training run, which cut sample token usage by 63 to 83 percent compared to medium effort. Despite training strictly under low reasoning constraints, the fine-tuned gpt-oss-20b LoRA adapter generalized effectively when evaluated at higher reasoning settings, demonstrating that businesses can keep fine-tuning budgets tight while preserving high downstream task accuracy during production serving.

A hands-on guide: who is affected

This release directly impacts artificial intelligence practitioners and cloud infrastructure teams building tool-augmented agents across AWS environments. Supported base models for serverless MTRL currently include Amazon Nova 2 Lite and GPT-OSS-20B in both US East (N. Virginia) and US West (Oregon), alongside Gemma 4 31B instruction-tuned and Qwen 3.6 27B available in US West (Oregon). Teams can alternatively route SageMaker AI prompts away from AgentCore to custom infrastructure running on Amazon Elastic Kubernetes Service, Amazon Elastic Compute Cloud, or AWS Fargate via an AWS Lambda forwarder.

Developers setting up these jobs must configure specific local prerequisites and Identity and Access Management policies. Setup requires Python 3.10 or later, uv, Node 20 or later, botocore 1.43.37 or later for the AgentRFT CreateJob API, and the AgentCore command-line interface. The training role must explicitly trust job.sagemaker.amazonaws.com with sts:AssumeRole and sts:TagSession actions and attach the AmazonSageMakerJobFullAccess policy. Using the generic sagemaker.amazonaws.com principal causes immediate job assumption errors. Meanwhile, the AgentCore runtime execution role requires the AmazonSageMakerJobRuntimeAccess policy to invoke the policy model.

A hands-on guide: what to watch

Teams deploying the sample code from the aws-samples repository should pay close attention to regional resource alignment and dataset layout. The training job and the Amazon Simple Storage Service data bucket must reside in the exact same AWS Region as the AgentCore runtime, because CreateJob rejects cross-region runtime ARNs. The training dataset itself only needs to supply task seeds, such as the secret word built from open-source SCOWL lists, while all domain validation and vocabulary logic remains bundled within the agent deployment package.

Deployment mechanics also present specific operational friction points to monitor. The AgentCore CLI compiles roughly 700 MB of dependencies for Linux ARM64 environments and orchestrates role creation through AWS CloudFormation, avoiding packaging errors that frequently occur when developers upload archives manually through the web console. As AWS expands model coverage and tooling hooks, engineers should monitor MLflow training tracking dashboards to ensure reward distributions do not collapse and that backward pass training tokens remain balanced against episode lengths.