Run large language models at home, BitTorrent‑style

Petals lets people run large language models at home in a BitTorrent-style swarm. Supported models include Llama 3.1 up to 405B, Mixtral 8x22B, Falcon at 40B and above, and BLOOM at 176B. Users can generate text and fine-tune those models for their own tasks on a consumer-grade GPU or Google Colab.

The mechanics are distributed inference. You load only part of the model on your machine, then join a network of other participants who host the remaining parts. Single-batch inference reaches up to 6 tokens per second for Llama 2 70B and up to 4 tokens per second for Falcon 180B, which the project presents as enough for chatbots and interactive apps.

For engineers and builders, the draw is access past a fixed black-box API. Petals claims you can use any fine-tuning and sampling methods, run custom paths through the model, and inspect hidden states. The pitch is API-level convenience with the flexibility of PyTorch and Hugging Face Transformers.

That puts Petals next to classic hosted LLM APIs rather than as a drop-in replacement for them. Hosted APIs abstract away model weights and hardware; Petals keeps the model open and splits the cost of serving across consumer GPUs and Colab instances. The tradeoff is clear from the stated numbers: usable interactive latency on very large models, not frontier throughput.

Practical next steps are concrete. Try the Colab notebook, read the GitHub docs, and decide whether partial local loading plus a peer network fits your workload. If you have spare GPU capacity, you can contribute it to the swarm. Follow progress on Discord or the project email list for updates on network health, supported models, and fine-tuning workflows.