QLoRA can fine-tune a 65B model on one 48GB GPU. This guide shows how to adapt small domain SLMs, measure quality, and ship safely. Read now.
Why domain SLMs and QLoRA belong together
Small language models (SLMs) are often the right fit for a single domain: they are cheaper to run, easier to constrain, and simpler to evaluate than a general-purpose giant. Fine-tuning lets you push vocabulary, tone, and task structure toward your data without rebuilding a model from scratch. QLoRA makes that practical at scale: by combining quantization with low-rank adapters, you can fine-tune a 65B model on one 48GB GPU, so capacity and domain focus no longer force a multi-GPU cluster for every experiment.
The goal is not to maximize parameter count. It is to keep a model small enough to serve under your latency and cost budget while still absorbing the patterns that matter in your domain—forms of language, decision rules, and edge cases that generic pretraining never saw often enough.
How to adapt a small domain model
Start with a clear task definition and a curated training set. Prefer high-signal examples over volume: paired inputs and expected outputs, short rationales when the task is ambiguous, and explicit negative cases for failure modes you refuse to learn. Split data so validation never leaks into training, and keep a held-out slice that mirrors production traffic, not only easy textbook samples.
With QLoRA, you freeze the quantized base weights and train low-rank adapters. That keeps memory down on a single 48GB GPU even for large bases, and it makes rollback simple: you can swap adapters without redeploying the full stack of base weights. Tune rank, learning rate, and steps conservatively; over-fitting a small domain set is common. Track loss on both train and validation, and stop when validation stops improving rather than when train loss bottoms out.
Measure quality before you ship
Offline metrics alone are not enough. Combine automated checks with domain review:
- Task accuracy or exact-match on structured outputs, plus semantic similarity where answers can vary in wording.
- Constraint tests: required fields present, forbidden topics refused, format schemas valid.
- Regression suite against known good answers from the current production path, so fine-tuning does not fix one case and break five others.
- Human review on a stratified sample—easy, hard, and adversarial prompts from real users.
Score the model against the baseline you actually run today, not against an abstract “smartness” bar. A domain SLM wins when it is more reliable on your tasks at a cost and latency you can sustain, even if a larger general model still wins open-ended chat.
Ship safely
Treat adapters as versioned artifacts with pinned base weights, data snapshot IDs, and eval reports. Roll out behind a canary: shadow traffic first if you can, then a small share of live traffic with clear fallback to the previous model. Log inputs, outputs, and policy flags so you can spot drift, toxicity, or format breakage early. Keep a kill switch that drops adapters and reverts to the prior path without a full redeploy drama.
Revisit the domain set on a schedule. Fine-tuning is not a one-shot win; product language, regulations, and failure modes shift. When quality slips, retrain from the same disciplined pipeline—data hygiene, QLoRA adapters, measured evals, controlled release—rather than patching prompts alone. That loop is how domain SLMs stay useful after the first demo.