GPT-OSS-20B is an open-weight model that can produce a visible chain-of-thought before answering, which is useful for checking how it reached a conclusion. The problem, according to an AWS engineering walkthrough, is that this chain-of-thought comes out in English almost no matter what language you ask for. Tell it to reason in French and it will say yes, then think in English anyway.

What is new

The post describes a concrete fine-tuning project run on Amazon SageMaker HyperPod, a managed cluster service for distributed training. The team built a 50-example test (10 math and logic problems in English, French, German, Spanish and Italian) and measured two things: whether the model's internal reasoning came out in the requested language, and whether its final answer did too.

The base model scored 22 percent on reasoning-language correctness across the 50 cases. Broken down by language, that 22 percent was almost entirely English examples, which it got right 100 percent of the time, while French and Spanish scored 0 percent, Italian 0 percent, and German only 10 percent. In short, the instruction to reason in a given language was essentially ignored unless that language happened to be English.

How it works

To fix this, the team used two different training techniques and compared them directly. Supervised fine-tuning (SFT) shows a model worked examples of the behavior you want, in this case chain-of-thought reasoning and answers already written in the target language, and adjusts the model's weights to imitate that pattern. Reinforcement learning with Group Relative Policy Optimization (GRPO) works differently: it generates several candidate responses to the same prompt, scores them with a reward function, and nudges the model toward whichever responses scored above average for that batch. The key limitation the post stresses is that GRPO can only amplify a behavior the model already produces at least occasionally; if the desired behavior never shows up in the model's output, there is nothing to reinforce.

The training data was HuggingFaceH4's Multilingual-Thinking dataset, a chain-of-thought set translated into English, French, German, Spanish and Italian, with roughly 949 training examples and 50 held out for evaluation. SFT ran with LoRA (Low-Rank Adaptation), a parameter-efficient method that trains a small add-on set of weights rather than the full 20-billion-parameter model, for 1,000 steps on four ml.g6e.12xlarge instances (16 L40S GPUs). After SFT, GRPO was applied on top of the resulting checkpoint, using eight sampled generations per prompt and a reward function that scored answer-language correctness, reasoning-language correctness and conciseness.

  • Base GPT-OSS-20B: 22 percent reasoning-language correct (English 100 percent, German 10 percent, French/Spanish/Italian 0 percent)
  • After SFT: reasoning-language correct 100 percent, but answer-language correct only 62 percent
  • After SFT plus GRPO: answer-language correct rose to 70 percent, an 8-point gain, while reasoning-language stayed at 100 percent
  • On a separate GSM8K math benchmark where the base model already scored 90.5 percent, GRPO produced no improvement at all

Why it matters

SFT closed almost the entire gap for reasoning in the correct language, taking it from 22 percent to 100 percent, which the post calls the harder problem since the model had essentially no ability to do this at all beforehand. But SFT alone left a second gap: even when the reasoning was correctly in French or German, the final short answer often reverted to English, landing at only 62 percent accuracy. That is the part GRPO moved the needle on, adding roughly 8 percentage points by reinforcing the cases where the model already sometimes got the answer language right.

This produces a practical rule for teams fine-tuning open-weight models: use SFT when the model cannot do something at all, and reach for GRPO only to polish a behavior that already shows up some of the time. The team tested this rule on a second, unrelated model, Qwen2.5-7B-Instruct, across two tasks. On the same multilingual reasoning task, where there was room to improve, SFT plus GRPO beat SFT alone (44 percent versus 38 percent on getting both reasoning and answer language correct). On GSM8K, where the base model was already near its ceiling at 90.5 percent accuracy, GRPO added nothing, because when nearly all sampled responses in a batch are already correct, the reward signal that GRPO relies on collapses to zero and there is no gradient to learn from.

Limits and open questions

Per-language results after GRPO were uneven: answer-language accuracy reached 80 percent for German and 70 percent for French, but only 60 percent for Spanish and 40 percent for Italian, on a test set of just 50 examples. That is a small evaluation set, so these specific percentages should be read as directional rather than statistically precise. The post also notes that GRPO's gains plateaued quickly in this setup and that the best checkpoint had to be picked by watching the answer-language metric during training rather than simply running to a fixed step count. The Qwen control experiment described as producing the GSM8K comparison is described as an internal test whose code has not yet been released publicly, so it cannot currently be reproduced independently.

How to try it

AWS published the full GPT-OSS-20B workflow in the awsome-distributed-ai GitHub repository, including Kubernetes manifests to run SFT, GRPO, and side-by-side inference comparisons on an Amazon SageMaker HyperPod cluster using Amazon EKS orchestration, the TRL training library, LoRA adapters, and Amazon FSx for Lustre for shared checkpoint storage. The post walks through building a training container image, launching the SFT job, converting checkpoints to a format GRPO and inference scripts can read, running the GRPO stage on top of the SFT checkpoint, and deploying three separate inference pods (base, SFT-only, and SFT+GRPO) that can be queried with the same prompt to compare outputs directly. It requires GPU instances with EFA networking support, a HuggingFace access token for the GPT-OSS-20B model and the Multilingual-Thinking dataset, and standard tools such as the AWS CLI, kubectl and Docker.

Sources