Amazon Web Services has published a hands-on guide for Multi-Turn Reinforcement Learning (MTRL), a new capability on Amazon SageMaker AI aimed at a specific gap: tuning models for tasks that take several dependent steps, where there is no single correct answer to imitate and the only reliable signal is whether the whole task eventually succeeded.

What is new

MTRL lets a team train a model against an agent that acts over multiple turns in a real or simulated environment, then scores the entire episode with one reward at the end. SageMaker AI runs the training job and holds the model being trained; the agent itself, along with the task environment and the scoring logic, is hosted separately on Amazon Bedrock AgentCore (or on a customer's own infrastructure such as Amazon EKS, EC2, or Fargate). The service supports four base models at launch, each trained through a lightweight adapter rather than retraining the whole network, and it is currently available only in specific AWS regions.

  • Supported base models: Amazon Nova 2 Lite, GPT-OSS-20B, Gemma 4 31B (instruction-tuned), Qwen 3.6 27B
  • Regions: US East (N. Virginia) and US West (Oregon), with Gemma and Qwen currently limited to Oregon
  • Pricing is per million tokens across three meters: prefill, sample, and train tokens, priced per base model
  • In AWS's worked example, training GPT-OSS-20B for 100 steps used about 1.29 billion tokens across roughly 105,000 rollouts over 12 hours 40 minutes, costing about $339 in training tokens plus about $79 in AgentCore runtime usage
  • Result: the model's solve rate on held-out Wordle words rose from 61 to 84 percent at its default reasoning effort, and from 15 to 69 percent at the lower reasoning effort it was actually trained at

Why single-turn fine-tuning falls short here

To understand why this matters, it helps to recall how models are normally adapted to a task. Supervised fine-tuning shows a model example inputs paired with correct outputs and nudges it to reproduce them. That works well when there is a single right answer per prompt. Reinforcement learning instead scores a model's output and adjusts it to produce higher-scoring outputs more often, which suits tasks where many different outputs could be acceptable, but single-turn RL still judges one response at a time.

Agentic tasks break both assumptions. An agent that guesses a word, triages a support ticket, or works through an internal process takes a sequence of actions, each shaped by what happened in the step before. There is rarely one correct sequence to copy, which rules out supervised learning, and a single-turn reward cannot assign credit to a decision made several steps before the final outcome becomes known. This is the classic reinforcement-learning credit-assignment problem, and it is what MTRL is built to address by training on the result of the full episode rather than any one step in isolation.

How it works

For every training example, SageMaker AI sends a prompt (in AWS's example, one containing a hidden five-letter word) to the agent running on AgentCore. The agent calls the model under training through an OpenAI-compatible endpoint, plays out the task using a tool call, and receives structured feedback after each move, exactly as a Wordle player sees letters turn green, yellow, or grey. When the episode ends, win or lose, the agent reports a single number back to SageMaker as the reward.

The learning algorithm used is Group Relative Policy Optimization (GRPO), a reinforcement-learning method that plays out several attempts at the same task and compares their rewards against each other, rather than against a fixed baseline, to decide which behaviours to reinforce. The updates are applied through a LoRA (Low-Rank Adaptation) adapter, a small set of additional trainable parameters layered onto the frozen base model, which keeps training cheaper than updating every weight in a 20-billion-parameter model.

Three structural choices stand out. First, the agent, the environment, and the reward function all live in one file deployed to AgentCore, separate from the training job. Second, the training dataset itself carries almost nothing but the task identifier (the secret word, in this case); the policy model never sees the answer. Third, the reward is deliberately outcome-dominant: in AWS's design a solve pays between 1.05 and 1.55 depending on how quickly it happened, a loss after using all guesses pays -0.5, and failing to ever make a valid guess pays -1.5, with a small bounded bonus for good intermediate strategy layered on top.

Why it matters

Wordle is explicitly described as a teaching example rather than the point of the exercise; large frontier models already play it competently. The intended use case is different: making a small, cheaper model competent at a multi-step workflow specific to one organisation, such as an internal process that public models were never exposed to during their original training. Because MTRL only needs a way to score whether a task succeeded, not a labelled set of correct trajectories, it opens up tuning for tasks where writing out the right sequence of actions would be impractical or where many different sequences could count as success. AWS also notes that customer datasets, agents, and trained models remain in the customer's own account and are not used to train the underlying base models.

Limits and open questions

The guide is candid about rough edges. Reward design took a revision: an initial per-guess scoring scheme, summed naively over a whole game, produced the perverse outcome that a losing game could score higher than winning one, and a fast solve could score lower than a slow one, because bonus terms piled up over more turns. The fix separates a dominant outcome score from a small, capped shaping term that only breaks ties within the same outcome tier.

Several training failures are also documented: authentication tokens breaking under concurrent rollouts, a rollout that never reaches the model failing the entire job rather than just that episode, and the model occasionally drifting from using its tool correctly into writing plain text when the learning rate was too high, which caused every rollout to collect the same low reward and stalled learning. Training reward in the Wordle run plateaued after about 15 steps, and validation scored during training (at temperature 0, deterministic decoding) reported a 37 percent solve rate at the end of the run, substantially lower than the 84 percent measured when the trained model was evaluated at the temperature and reasoning effort it would actually be served at. AWS flags this gap explicitly: the metrics inside the training job are not the same as performance in production settings, and teams should evaluate separately at serving-time settings before judging a run's success. The post does not report whether additional training steps beyond 100, or changes to the number of rollouts per task, would raise the plateau, saying only that this was left untested.

How to try it

AWS has released a sample repository covering the full pipeline: creating the two required IAM roles (one of which needs a non-obvious trust policy for job.sagemaker.amazonaws.com), deploying the agent to AgentCore with its CLI, verifying the deployment with a smoke-test payload before spending any GPU time, preparing a dataset that holds only task identifiers rather than labelled answers, launching the training job through a MultiTurnRLTrainer class, monitoring reward and validation metrics in MLflow, and finally running separate evaluation jobs at the temperature and reasoning-effort settings the model will actually be served at. The guide recommends starting with an outcome-based reward, adding any shaping terms sparingly, and checking reward values against real recorded trajectories before trusting a training run.

Sources