Optimizing llama-server Speed with Batch Tweaks

What It Is

Llama-server performance tuning focuses on adjusting how the inference engine processes tokens and manages memory. Recent experiments demonstrate that batch-related parameters can dramatically affect throughput without requiring hardware upgrades or model changes.

The core optimization involves three interconnected settings. Batch size (--batch-size 2048) controls how many tokens the server processes in a single forward pass through the model. Microbatch size (--ubatch-size 1024) breaks that larger batch into smaller chunks for actual computation, balancing memory usage against processing efficiency. Flash attention (--flash-attn on) implements a memory-efficient attention mechanism that reduces the quadratic memory complexity of standard transformer attention.

RAM cache allocation (--cache-ram 61440) reserves 60GB of system memory for KV cache storage, keeping previously computed attention keys and values readily accessible. This prevents redundant calculations when processing long contexts or maintaining conversation history across multiple requests.

Why It Matters

Inference speed directly impacts user experience and operational costs. A configuration that doubles throughput effectively halves the hardware requirements for serving the same number of requests. For teams running local deployments, these optimizations mean fewer GPUs needed and lower electricity bills.

The batch size adjustments particularly benefit scenarios with longer prompts or multi-turn conversations. Processing 2048 tokens per batch instead of smaller defaults means fewer round trips through the model, reducing overhead from memory transfers and kernel launches. Organizations handling document analysis, code generation, or extended dialogues see the most dramatic improvements.

Flash attention matters because standard attention mechanisms consume memory proportional to sequence length squared. At 200,000 token context windows, this becomes prohibitive. Flash attention maintains the same mathematical operations while restructuring memory access patterns, enabling longer contexts without running out of VRAM.

The RAM cache strategy shifts bottlenecks from GPU memory to system RAM, which typically costs less per gigabyte. This architectural choice works well for multi-GPU setups where PCIe bandwidth between host and device can become a limiting factor.

Getting Started

Start by identifying current bottlenecks. Run llama-server with default settings and monitor GPU utilization, memory consumption, and tokens per second. If GPU utilization stays below 80%, batch parameters likely need adjustment.

Test this configuration as a baseline:

 --host 0.0.0.0 \
 -m /path/to/model.gguf \
 --ctx-size 8192 \
 --batch-size 2048 \
 --ubatch-size 512 \
 --flash-attn on

Increase --ubatch-size gradually while monitoring VRAM usage. The optimal value depends on available GPU memory and model size. For 24GB cards, values between 512-1024 typically work well. Larger cards can push higher.

Add RAM caching once batch parameters stabilize. Calculate available system RAM, subtract OS overhead and other application needs, then allocate 70-80% of the remainder:

--cache-ram 49152 # 48GB for systems with 64GB RAM

Enable multi-GPU support by setting CUDA_VISIBLE_DEVICES=0,1,2 before the command. Llama-server automatically distributes model layers across available devices.

Monitor performance with the built-in metrics endpoint at https://localhost:8080/metrics or by tracking response times in application logs.

Context

Alternative approaches include quantization (reducing model precision to 4-bit or 8-bit), which trades some quality for speed, or switching to vLLM, which implements continuous batching for higher throughput under concurrent requests. TensorRT-LLM offers even faster inference but requires more complex setup and NVIDIA hardware.

Batch optimization works best for single-user scenarios or sequential processing. High-concurrency deployments benefit more from parallel request handling, where frameworks like vLLM or Text Generation Inference excel. The --parallel 1 setting in the example configuration confirms this single-stream focus.

Self-speculative decoding (https://www.reddit.com/r/LocalLLaMA/comments/1qpjc4a/add_selfspeculative_decoding_no_draft_model/) represents the next frontier, where models predict multiple tokens simultaneously and verify them in parallel. This technique can multiply effective throughput without additional hardware, though implementation remains experimental in llama.cpp.

Hardware limitations still apply. No amount of batch tuning overcomes insufficient VRAM for model weights. Context length remains constrained by total available memory across all optimization strategies.