Three-stage curriculum fine-tuning reduces early training instability and improves domain recall on SLMs. Build the pipeline fast. Read now.
Why curriculum order matters for SLMs
Small language models have less capacity to absorb noisy or conflicting gradients early in fine-tuning. When hard, long, or out-of-distribution examples appear first, loss spikes, attention patterns scramble, and the model often locks onto shallow surface cues. A curriculum reorders the same data so the model first masters simple, high-confidence patterns, then steps into denser domain material. That ordering does not add new labels; it changes when each example exerts pressure on the weights, which is enough to cut early instability and leave more headroom for domain-specific recall later.
Treat the curriculum as a training schedule, not a separate architecture. You still use standard supervised fine-tuning or instruction tuning. The pipeline’s job is to score, bucket, and feed examples in stages so each stage starts from a model that already handles the previous stage’s distribution without thrashing.
Three stages that keep training stable
Stage one is foundation: short prompts, clear targets, low ambiguity, and light domain jargon. The goal is clean next-token behavior and basic instruction following on your format, not deep expertise. Stage two is structured domain work: medium-length examples with real terminology, multi-step answers, and the schema or style you want in production. Stage three is hard cases: edge conditions, long context, rare entities, and near-miss negatives that force the model to separate related concepts. Move to the next stage only when validation loss and a small held-out checklist of domain prompts stop improving on the current stage.
- Stage 1 — format fluency and easy positives
- Stage 2 — core domain recall and multi-step answers
- Stage 3 — hard edges, long inputs, and confusable pairs
Keep learning rate and batch size conservative when you enter a harder stage; a sudden jump in difficulty is often what reintroduces the instability you just avoided. Prefer more epochs on cleaner stages over one long mixed pass when data volume is limited.
Build the pipeline quickly
Start from a single labeled file and add a difficulty score per row. Useful signals that need no external models include token length, number of domain keywords, presence of multi-hop structure, and whether the target answer is short and unique. Sort ascending by score, split into three contiguous buckets, and write three train shards. Wire your trainer to load stage one, train, evaluate, then swap the dataset path for stage two and three without resetting the optimizer if you want continuity—or reset it if stage transitions still show sharp loss jumps. Log stage id, step, and a fixed eval set so you can see when domain recall rises without early train loss blowing up.
Ship a minimal eval set before you tune hyperparameters: a handful of easy format checks, a middle band of typical domain questions, and a few adversarial or long-form items. If middle-band accuracy stalls while stage-three loss falls, your hard bucket is dominating and you should rebalance or shorten stage three.
Practical guardrails
Do not hide poor labels behind curriculum order; easy-stage noise still teaches wrong habits. Keep a small random holdout outside all stages so you measure true generalization, not memorization of the hard tail. When you retrain, freeze the scoring rules that define the stages so runs stay comparable. The payoff is operational: fewer aborted runs from early spikes, clearer stage-level ablations, and stronger domain recall on an SLM without changing the base model or inventing a heavier stack.