Run a fine-tuned Llama-4-3B model locally with zero cloud dependency. Step-by-step QLoRA setup, quantization, and edge deployment. Full breakdown.
Why Fine-Tune a Small Model for the Edge
A 3B-parameter model is small enough to run on a single consumer GPU, a modern laptop, or even some phones, which makes it a practical base for applications that never call out to a cloud API. The tradeoff is that a small general model rarely matches your task out of the box. Fine-tuning closes that gap: instead of relying on a large hosted model to handle your domain, you teach a compact local one to do a narrow job well.
The privacy argument is the main reason to accept that tradeoff. When the model runs entirely on the device, user inputs, prompts, and generated output never leave the machine. That removes a class of compliance and data-handling problems, and it lets the app keep working with no network at all.
QLoRA: Fine-Tuning Without a Datacenter
QLoRA makes fine-tuning feasible on modest hardware by combining two ideas. The base model is loaded in a quantized, low-precision form so its weights take far less memory, and training happens through small low-rank adapter matrices rather than by updating the full network. The frozen base stays fixed; you only learn the adapters, which are a tiny fraction of the total parameters.
Practically, this means the bulk of your effort goes into the dataset, not the compute. Curate examples that reflect exactly how the model will be used — the same input format, the same expected output style — and keep them clean and consistent. A few well-chosen hyperparameters matter most:
- Adapter rank: higher rank gives the model more capacity to adapt but costs more memory and can overfit small datasets.
- Learning rate and epochs: too many passes over a small dataset memorizes it; watch a held-out validation set.
- Target modules: which layers get adapters, typically the attention and projection weights.
Quantization and Packaging for Deployment
After training, you merge the adapters back into the base weights (or keep them separate to swap tasks) and quantize the result for inference. Quantization shrinks the weights to lower precision so the model fits in less memory and runs faster on the target device. There is a real quality-versus-size curve here: more aggressive quantization frees up memory and speeds generation but can degrade output, so test each level against your own examples rather than assuming the smallest file is acceptable.
Package the quantized model in a format your target runtime understands, then confirm it loads and generates within the memory and latency budget of the actual device — not just your training machine.
Running It Locally
On the device, a local inference runtime loads the quantized weights and serves generation without any outbound calls. Budget for the model's memory footprint plus the working memory that grows with context length, and measure tokens-per-second on real hardware so the interface can be designed around the true speed. Streaming output token by token keeps a slower local model feeling responsive.
Treat the fine-tuned model as something you version and re-evaluate. As you gather more examples of how people actually use the app, fold them into the dataset and retrain the adapters — the loop stays cheap precisely because QLoRA and quantization keep each pass small.