QLoRA can finetune a 65B model on one 48GB GPU, while FSDP+QLoRA spreads 70B training over 2x24GB cards with sharding and 4-bit adapters. Read now.
Why Full Finetuning Breaks on Modest Hardware
Training every weight of a large language model needs enough VRAM for parameters, gradients, optimizer state, and activations at once. That stack grows roughly with model size and quickly outruns a single consumer or mid-range data-center card. Parameter-efficient finetuning (PEFT) flips the problem: freeze the base model and train only a small set of extra parameters. LoRA is the most common form of that idea—low-rank matrices injected into linear layers that capture the task-specific update while the original weights stay fixed.
QLoRA takes the same adapter idea further by storing the frozen base model in 4-bit precision and training the LoRA adapters in higher precision. The summary case is the practical proof point: a 65B model can be finetuned on one 48GB GPU. You still pay for a full forward and backward pass through the base network, but you no longer materialize full-precision copies of every weight or optimizer tensors for them. That is what makes “cheap GPU” finetuning of very large models realistic instead of purely theoretical.
When One Card Is Not Enough: FSDP Plus QLoRA
Some models still do not fit even with 4-bit weights and adapters—for example a 70B model that exceeds a single 48GB envelope, or a workload that needs larger batches and longer sequences. Fully Sharded Data Parallel (FSDP) addresses that by splitting parameters, gradients, and optimizer state across devices. Each GPU holds only a shard of the model, gathers what it needs for the current compute step, then frees or reshards memory. Pairing FSDP with QLoRA means the shards are 4-bit base weights plus trainable adapters, not a full-precision replica of the entire network on every card.
The concrete multi-GPU pattern in the summary is instructive: spread 70B training over two 24GB cards with sharding and 4-bit adapters. Two smaller cards often cost less and are easier to obtain than one large card, and FSDP turns that split into a usable training setup rather than a half-loaded machine that OOMs on the first step. Communication cost rises because shards must be gathered and reduced, but memory per device drops enough that the run becomes possible at all.
How to Choose Between Single-GPU QLoRA and Multi-GPU Sharding
- Prefer single-GPU QLoRA when the 4-bit base model plus adapters, optimizer state for adapters only, and your activation footprint fit with headroom for the longest sequence you care about.
- Move to FSDP+QLoRA when peak VRAM still spikes—long context, large batch, or a model near the edge of one card’s capacity.
- Keep adapter rank and target modules modest at first; rank drives trainable parameter count and therefore optimizer memory, which is the part that still scales with PEFT.
- Watch host RAM and disk for offload or checkpoint paths; “fits on GPU” is not the only bottleneck when shards and checkpoints bounce through the CPU.
Treat memory budgeting as a checklist, not a guess. Measure base-model footprint in 4-bit, adapter size for the layers you actually train, gradient and optimizer tensors for those adapters, and activation memory under your real sequence length. If any of those exceed a single card after reasonable batch-size and gradient-accumulation tradeoffs, sharding is the next lever—not more aggressive hyperparameter hope.
Practical Training Habits That Matter More Than Fancy Stacks
On cheap GPUs, stability often matters more than peak theoretical throughput. Use gradient accumulation to emulate larger batches without inflating VRAM. Checkpoint adapters frequently; they are small, so saving them is cheap insurance against a flaky multi-hour job. Validate that frozen base weights stay frozen and that only adapter tensors receive gradients—silent full-model updates will blow memory and defeat PEFT. When you scale from one card to two, start with the same effective batch size and learning-rate regime you already trust, then adjust only after the sharded run is numerically healthy.
Distributed training with PEFT and LoRA is not about matching the largest cluster. It is about matching model size, precision, and parallel strategy to the cards you can actually rent or buy. QLoRA on one 48GB GPU and FSDP+QLoRA across two 24GB cards are two ends of the same idea: keep the heavy base model compressed and shared, train only the thin adapter surface, and spend memory where the update actually lives.