Hardware-First Guide to Selecting Open-Source LLMs

Open-source language models now consist of at least a couple billion parameters, and that parameter count translates directly into a memory bill that has to be paid in GPU VRAM. Before picking a model for a local coding assistant or a document workflow, it helps to work out what the available hardware can actually load. Hugging Face’s guide on optimizing LLMs for speed and memory lays out the arithmetic and the techniques that change it, available at https://huggingface.co/docs/transformers/main/en/llm_tutorial_optimization.

Estimating the Memory a Model Needs

Each parameter is a decimal number stored in a numeric format, usually float32, bfloat16, or float16. Loading the weights of a model with X billion parameters takes roughly 4 * X GB of VRAM in float32 precision. Because models are rarely run in full float32 anymore, the more useful rule of thumb is about 2 * X GB in bfloat16 or float16.

The Hugging Face guide works through several concrete examples in bfloat16: GPT3 needs about 350 GB, Bloom about 352 GB, Llama-2-70b about 140 GB, Falcon-40b about 80 GB, MPT-30b about 60 GB, and StarCoder about 31 GB. For short inputs, under roughly 1024 tokens, the memory needed for inference is dominated by the memory needed to load the weights, so these figures double as a starting estimate for running the model.

At the time the guide was written, the largest GPU chips on the market, the A100 and H100, offered 80 GB of VRAM. Most of the models above exceed that just to load, which means they require tensor parallelism or pipeline parallelism to spread layers across multiple devices. In Transformers, loading with device_map="auto" distributes layers across the available GPUs automatically.

How Quantization Lowers the Bar

When a GPU cannot hold a model in bfloat16, quantization reduces the precision of the weights while trying to keep inference results close to the original. The guide notes that weights can be quantized to 8-bit or 4-bit without a significant loss in performance, and even to 3 or 2 bits with an acceptable loss.

The practical effect is shown with the bigcode/octocoder model, a model of more than 15 billion parameters. In bfloat16 it consumed about 32 GB of VRAM. Loading it in 8-bit precision with the load_in_8bit=True flag brought peak usage down to a little over 15 GB, low enough to run on a consumer card like the 4090. Switching to 4-bit with load_in_4bit=True reduced it further to just over 9 GB, which puts the model within reach of GPUs such as the RTX 3090, V100, and T4.

There is a trade-off. Quantization dynamically dequantizes weights back to bfloat16 during the matrix multiplications, so inference time is often not reduced and can increase, with the more aggressive 4-bit scheme running slower than 8-bit. If GPU memory is not a constraint, the guide notes there is often no need to quantize at all.

Memory Costs That Scale With Context

Model weights are only part of the budget. The default self-attention computation grows quadratically in memory with the sequence length, which becomes a serious problem around 16000 input tokens. Flash Attention produces numerically identical outputs while keeping the memory cost growing only linearly with sequence length, and the guide recommends always using it when available.

Auto-regressive generation also relies on a key-value cache to avoid recomputing earlier tokens. That cache itself can grow large. For octocoder at a hypothetical sequence length of 16000, storing the cache in float16 takes around 15 GB, roughly half the size of the model weights. Architectures using Multi-Query Attention or Grouped-Query Attention shrink that cache substantially, which is why models intended for chat and other long-context tasks are often built with them.

Taken together, the choice of model is less about a single specification and more about matching parameter count, precision, context length, and attention design to the VRAM on hand.