Petals: Running Large AI Models BitTorrent-Style

Petals is an open source system for running large language models at home in a way the project describes as “BitTorrent-style.” Instead of requiring one machine powerful enough to hold an entire model, Petals lets each participant load only a small part of the model and connect to a network where other people serve the remaining layers. The project is MIT-licensed and came out of the BigScience research workshop. Its repository and documentation live at https://github.com/bigscience-workshop/petals.

How The Swarm Works

The core idea is pooling. As the project puts it, a participant loads a small part of the model, then joins a network of people serving the other parts. The system uses pipeline parallelism and tensor parallelism so that a single very large model can be split across many separate computers rather than packed onto one.

This design means no single contributor needs to hold the full set of weights. A person running a server hosts a portion of the layers, and inference requests pass through the chain of peers that collectively make up the model. Because it relies on volunteers sharing GPU capacity, Petals is a community-run network rather than a centrally operated service.

Supported Models

The project highlights distributed support for several large models, including Llama 3.1 at sizes up to 405B parameters, Mixtral 8x22B, Falcon at 40B and larger, and BLOOM at 176B. Examples and benchmarks also reference Llama 2 70B, Llama 65B, and Falcon 180B. In principle, models available from the Hugging Face Model Hub can be hosted through the system.

Some of these are gated models. For weights such as Llama 3.1, a contributor first needs to request access and authenticate with huggingface-cli login before running a server.

Performance And Speed

Petals reports that fine-tuning and inference can run up to 10 times faster than offloading, the common alternative for fitting a model that does not fit in GPU memory. For single-batch inference, the project cites speeds of up to 6 tokens per second for Llama 2 70B and up to 4 tokens per second for Falcon 180B. The documentation describes these rates as fast enough for chatbots and interactive applications.

Performance depends on the network and the peers currently serving each part of the model, since the work is spread across volunteer machines rather than dedicated hardware.

Running A Server Or Client

For inference, Petals keeps a familiar interface. A client uses AutoDistributedModelForCausalLM.from_pretrained(model_name) together with a standard tokenizer, then calls model.generate(...). The project notes that users keep the comforts of an API with the flexibility of PyTorch and Hugging Face Transformers, which allows custom sampling methods, custom model paths, and access to hidden states.

Hosting a GPU to contribute to a swarm is supported on Linux through Anaconda, on Windows through WSL, through Docker, and on macOS with Apple M1 and M2 chips. A contributor starts a server with python -m petals.cli.run_server followed by a model name.

Privacy And Background

The project states that hosting a server does not let other people run custom code on the host machine. For sensitive data, Petals supports private swarms, where a closed group runs its own network rather than joining the public one. The system is backed by two academic papers, published at ACL 2023 and NeurIPS 2023.