Running 120B AI Models on Networked Mini PCs

python -m petals.cli.run_server meta-llama/Llama-2-120b-chat \
  --num_blocks 8 --torch_dtype float16 \
  --initial_peers /ip4/159.89.214.152/tcp/31337/p2p/QmNLei78zWmzUdbeRB3CiUfAizuHuybmBZgzYBW1RwDbtW

This command launches a Petals server that contributes computational resources to run a portion of a 120-billion parameter language model. Rather than loading the entire model onto one machine, the system distributes different transformer blocks across multiple consumer-grade devices connected over the internet.

Distributed Inference Architecture

Petals implements a peer-to-peer network where each participating computer hosts several transformer blocks from a large language model. When a user sends a prompt, the input passes sequentially through blocks hosted on different machines across the network. Each node processes its assigned layers and forwards the intermediate activations to the next peer in the chain.

The architecture resembles pipeline parallelism but operates across geographically distributed devices with varying network latencies. A typical setup might have 8-12 mini PCs, each equipped with 16GB RAM and a mid-range GPU like an RTX 3060, collectively hosting a model that would normally require 240GB of VRAM on a single system.

Network bandwidth becomes the primary bottleneck. Activations between layers typically measure 10-50MB depending on sequence length and batch size. On a gigabit connection, transferring these tensors adds 80-400ms latency per hop. The system uses compression and caching strategies to minimize data movement, but multi-hop inference still operates at 1-3 tokens per second compared to 20-50 tokens per second on dedicated hardware.

Performance Characteristics

Benchmark tests using networked RTX 3060 systems running BLOOM-176B show throughput of approximately 1.2 tokens per second with 4-6 participating nodes. Quality remains identical to centralized inference since the same weights and operations execute, just distributed across machines.

Latency varies significantly based on network topology. Local area networks with sub-5ms ping times achieve 2-3 tokens per second, while internet-connected peers spanning continents may drop to 0.5 tokens per second. The system automatically discovers and prefers nearby peers to optimize routing.

Memory efficiency improves dramatically compared to running quantized models locally. A single mini PC with 24GB RAM can only load heavily quantized 30B models, sacrificing quality. That same device contributes to running full-precision 120B models when networked, maintaining output fidelity while sharing the memory burden.

Home Lab Implementation

Setting up a distributed inference cluster requires identical model versions across all nodes. The Petals client automatically handles peer discovery through a distributed hash table, similar to BitTorrent. Each server specifies which transformer blocks to host, typically allocating based on available VRAM.

from petals import AutoDistributedModelForCausalLM
model = AutoDistributedModelForCausalLM.from_pretrained(
    "bigscience/bloom-120b",
    max_retries=3,
    timeout=60
)

This code connects to the public Petals swarm, automatically finding available peers hosting BLOOM blocks. Private clusters can operate behind firewalls using VPN tunnels or Tailscale mesh networks, ensuring data never leaves trusted infrastructure.

Configuration requires balancing block allocation across nodes. Hosting too many blocks on one device creates bottlenecks, while spreading blocks too thinly increases network hops. Most deployments assign 6-10 consecutive blocks per node, minimizing inter-node communication while maintaining reasonable memory usage.

Practical Limitations

The approach works best for applications tolerating high latency. Interactive chatbots feel sluggish at 1-2 tokens per second, but batch processing tasks like document summarization or dataset annotation remain viable. Research workflows benefit significantly since they prioritize model capability over response speed.

Reliability depends on peer availability. If a node hosting critical blocks goes offline, inference fails until the system reroutes through alternative peers or those blocks come back online. Production deployments typically require redundancy, with multiple nodes hosting duplicate blocks.

Network costs can accumulate quickly. Processing a single 2,000-token conversation transfers roughly 500MB of activation data across the cluster. Organizations with metered bandwidth or data caps may find costs prohibitive compared to API services.

The technology democratizes access to frontier models for researchers and hobbyists who can pool resources. A group of five people, each contributing a $800 mini PC, can collectively run models that would otherwise require $15,000 in specialized hardware.