Fine-tuning GPT-OSS-20B for multilingual reasoning on Amazon SageMaker

The draft ran long. I’m tightening each section to the 80–160 word band and keeping the full piece inside 800–1,000 words.All the source figures are in, but the piece is about 165 words over the cap. I’m cutting repeated phrasing while keeping every name and number.Amazon Web Services fine-tuned GPT-OSS-20B, an open-weight reasoning model that thinks in English, for multilingual reasoning on Amazon SageMaker HyperPod. AWS Machine Learning Blog's report puts the base model at 22 percent reasoning-language correct, 11 of 50 cases. Asked for French, German, Spanish, or Italian, the model acknowledged the instruction and wrote its chain of thought in English.

This piece covers HuggingFaceH4/Multilingual-Thinking, the supervised fine-tuning run, and when Group Relative Policy Optimization is worth adding. It is for teams serving reasoning models to non-English users, and for teams choosing SFT, GRPO, or both on an open-weight model. A Qwen2.5-7B control on two tasks with different capability gaps repeats that choice. SFT does most of the work, and GRPO refines only where headroom already exists.

Fine-tuning GPT-OSS-20B for multilingual: what actually changed

On a 50-case eval of ten math and logic problems across English, French, German, Spanish, and Italian, the base openai/gpt-oss-20b scored 22 percent, 11 of 50. The per-language split was English 100 percent, French 0 percent, German 10 percent, Spanish 0 percent, and Italian 0 percent, almost entirely the English items. SFT for 1,000 steps raised reasoning-language accuracy to 100 percent, 50 of 50, on those same items. The system prompt names the reasoning language, and the chain of thought stays in that language.

The gain came in two stages. Around step 200, reasoning-language correct hit 100 percent. Around step 1000, it also began giving the answer in the target language. At step 1000, answer-language correct was 62 percent, 31 of 50, and both-correct hit 62 percent, 31 of 50. Answer accuracy by language was French 60 percent, German 60 percent, Spanish 30 percent, Italian 60 percent, and English 100 percent. With non-English reasoning near zero at the start, this SFT pass is not optional.

Fine-tuning GPT-OSS-20B for multilingual: how it works

HuggingFaceH4/Multilingual-Thinking is a chain-of-thought dataset translated into French, Spanish, German, and Italian alongside English, with roughly 949 training examples and 50 validation examples. The prompt sets a reasoning language the model must reason and answer in. Reasoning-language correct scores the chain of thought, and answer-language correct scores the final answer. SFT teaches a behavior the model does not have. GRPO samples a group of responses per prompt and pushes the policy toward the above-average ones. A zero-probability behavior leaves GRPO very little to reinforce.

GPT-OSS SFT and GRPO used PyTorch TRL on SageMaker HyperPod with Amazon EKS. The elastic HyperPodPyTorchJob managed by Kueue, launched with hyperpodrun, ran on four ml.g6e.12xlarge instances, 16 L40S 48GB GPUs, with LoRA rank 8, alpha 16, FSDP full-shard, bf16, and gradient checkpointing. Qwen2.5-7B also used LoRA, with veRL, Ray, KubeRay, and vLLM. A Ray head and worker pods mount Amazon FSx for Lustre for the base model, training data, and checkpoints. Elastic Fabric Adapter carries collectives. vLLM rollouts return to veRL, which writes the LoRA adapter back to FSx.

Fine-tuning GPT-OSS-20B for multilingual: why it matters now

A user who asks in French expects the reasoning and the answer in French, not an acknowledgment followed by English. German, Spanish, and Italian questions fail the same way. The choice is SFT, reinforcement learning with GRPO, or both. French, Spanish, and Italian began at 0 percent reasoning-language correct, and German at 10 percent, so the desired chain of thought almost never appeared. GRPO can up-weight only samples that a group of responses already contains. English, at 100 percent, had no reasoning-language gap to close.

SFT installed the missing behavior. Reasoning language reached 50 of 50 near step 200. Answer language caught up near step 1000 and stopped at 31 of 50. Spanish answers at 30 percent trail French, German, and Italian at 60 percent, so some target languages still vary. That remainder is headroom for GRPO, because a target-language answer already happens sometimes. Qwen2.5-7B tests the same choice on two tasks with different capability gaps, on two g6e.48xlarge instances with 16 L40S GPUs. The method has to match the gap.

Fine-tuning GPT-OSS-20B for multilingual: who is affected

Teams that already run SageMaker HyperPod on Amazon EKS and need GPT-OSS-20B to reason outside English follow this setup. Multi-node SFT needs EFA-enabled instances, NVIDIA NCCL, and aws-ofi-nccl. Amazon FSx for Lustre mounts with PersistentVolumeClaim fsx-claim at /fsx. The image goes to a registry such as Amazon Elastic Container Registry. A Hugging Face token must allow openai/gpt-oss-20b and HuggingFaceH4/Multilingual-Thinking. The workstation needs the AWS CLI, kubectl, docker, and envsubst. GPT-OSS SFT uses four ml.g6e.12xlarge nodes. The Qwen control uses two g6e.48xlarge nodes.

Files live in awslabs/awsome-distributed-ai under 3.test_cases/pytorch/trl/gpt-oss-lora-grpo/. Operators copy env_vars.example to env_vars, set the ECR registry, image name, tag, and Hugging Face token, then source env_vars. artifacts/build_push.sh bundles PyTorch, TRL, Hugging Face Transformers, PEFT, and the train and inference scripts. kubectl targets the cluster in us-east-2. kubernetes/fsx-storage-manager.yaml supplies a shell for checkpoint conversion and eval logs. SFT applies hyperpod-eks/train-lora-hyperpod-elastic-g6e.yaml. Pods are labeled app=gpt-oss-lora-elastic, and kubectl logs -f follows that label during training. Runs start from Amazon SageMaker Studio or the AWS CLI.

Fine-tuning GPT-OSS-20B for multilingual: what to watch

At step 1000, reasoning language is already 100 percent and answer language is not. Both-correct remains 62 percent, 31 of 50, so a correct chain of thought can still end in the wrong language. Spanish answers sit at 30 percent, with French, German, and Italian at 60 percent and English at 100 percent. Record the two metrics separately before serving the adapter. convert_fsdp_checkpoint.py, run in the fsx-storage-manager pod, turns /fsx/checkpoints/checkpoint-1000 into the PEFT adapter /fsx/checkpoints/converted-peft/lora-checkpoint-1000-peft that TRL's GRPO trainer and the inference scripts load.

Keep GRPO for the metric that still moves. Reasoning language went from 11 of 50 to 50 of 50, so that score has no weak samples left. Answer language at 31 of 50 still varies, and GRPO prefers the above-average answers in each group. A healthy SFT log showed loss 1.035, grad_norm 0.163, learning_rate 0.0002, and mean_token_accuracy 0.716 at epoch 3.25, then loss 1.003, grad_norm 0.171, and acc 0.722 at epoch 3.32, then loss 0.976, grad_norm 0.159, and acc 0.726 at epoch 3.57.