Optimizing LLM Inference Speed in Transformers

Running large language models for inference is difficult because the models hold billions of parameters that have to be stored and processed. According to the Hugging Face Transformers documentation, loading a 70B parameter Llama 2 model requires 256GB of memory for full precision weights and 128GB for half-precision weights, while the most powerful current GPUs, the A100 and H100, hold only 80GB. Generation is also slow because the model is called repeatedly to produce each new token, and the input sequence grows longer as generation progresses. The Transformers optimization guide at https://huggingface.co/docs/transformers/main/en/llm_optims describes several techniques to reduce both problems.

Caching and compilation

LLMs compute key-value (kv) pairs for each input token, and because generated output becomes part of the next input, the same computation gets repeated. A kv-cache stores past keys and values instead of recomputing them. The default kv-cache is dynamic and grows with each generation step, which prevents the use of torch.compile, an optimization that fuses PyTorch code into optimized kernels.

The static kv-cache addresses this by pre-allocating the cache to a maximum size, which allows it to be combined with torch.compile for up to a 4x speed up. The documentation notes that the actual speed up varies with model size, where larger models see a smaller gain, and with hardware. For basic use, the cache implementation is set to “static” in a model’s generation configuration; more advanced cases can initialize a StaticCache object directly or compile the entire generate() call into a single graph.

Faster decoding strategies

Speculative decoding uses a second, smaller assistant model to generate candidate tokens that the larger model then verifies in a single forward pass. When the candidates are correct, the main model effectively gets those tokens without generating them itself. The documentation states there is no accuracy loss because the verification pass produces the same output the larger model would have. For the largest gain, the assistant should be much smaller than the main model and share the same tokenizer.

Prompt lookup decoding is a related variant that works well for input-grounded tasks such as summarization, where the prompt and output often share words. Overlapping n-grams from the prompt are used as candidate tokens. Both strategies are compatible with greedy search and sampling.

Attention and memory

Self-attention grows quadratically in compute and memory as the number of input tokens increases, which becomes a larger problem for the long sequences LLMs handle. The guide recommends FlashAttention-2 or PyTorch’s scaled dot product attention (SDPA) as more memory-efficient implementations. FlashAttention-2 breaks attention into smaller chunks and reduces read and write operations to GPU memory, improving on the original FlashAttention by parallelizing over the sequence length dimension. It is enabled by setting the attention implementation to “flash_attention_2” when loading a model. SDPA is enabled automatically in PyTorch 2.0 and selects the most performant attention algorithm available on a CUDA backend.

Quantization stores model weights at lower precision to reduce memory usage. The documentation gives concrete figures for Mistral-7B: loading it in half precision with bfloat16 requires 13.74GB of memory, while loading it in 8-bit requires 6.87GB. The guide also cautions that quantization can slightly increase latency, except for AWQ and fused AWQ modules, because of the extra step needed to quantize and dequantize weights. For systems that are not limited by GPU memory, quantization is not strictly necessary.

These techniques can be combined, and the Transformers documentation remains the reference for the exact parameters and supported models.