Tuning llama-server Batch and Parallel Settings

The llama-server tool that ships with llama.cpp serves models over an HTTP API and supports running multiple requests at once. A handful of command-line options control how the server batches work and how many requests it handles in parallel. The official documentation at https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md lists these options along with their defaults, which is the place to confirm exact behavior before changing a running deployment.

The Batch and Parallel Options

Three options come up most often when adjusting how the server processes requests.

The -np or --parallel flag sets the number of server slots. Its default is -1, which the documentation describes as automatic. Each slot is a unit that processes a request, and the server reports the configured count through the total_slots field, described as “the total number of slots for process requests (defined by --parallel option).”

The -b or --batch-size flag sets what the documentation calls the “logical maximum batch size,” with a default of 2048. The -ub or --ubatch-size flag sets the “physical maximum batch size,” with a default of 512. The logical batch is the larger grouping the server works with, while the physical batch is the smaller chunk actually handed to the model in a single step.

Context size is set separately with -c or --ctx-size. Its default is 0, which the documentation notes means the value is loaded from the model. Total context usage is given by the formula prompt_n + cache_n + predicted_n.

Continuous Batching

The server enables continuous batching by default. The relevant flag, -cb / --cont-batching (with -nocb / --no-cont-batching to disable it), controls “whether to enable continuous batching (a.k.a dynamic batching).” The documentation also lists “Continuous batching” and “Parallel decoding with multi-user support” among the server’s features, which is how a single instance serves several users at the same time rather than handling requests strictly one after another.

Because the documentation does not publish tuning tables or per-hardware recommendations, the practical approach is to change one option at a time and measure the result rather than copying numbers from elsewhere.

Reading Throughput Metrics

The server can expose a Prometheus-compatible metrics endpoint, but it is off by default. The --metrics flag enables it, and the documentation states the endpoint “is only accessible if --metrics is set.”

Once enabled, the endpoint reports several values useful for tuning. llamacpp:prompt_tokens_seconds is described as “Average prompt throughput in tokens/s,” and llamacpp:predicted_tokens_seconds as “Average generation throughput in tokens/s.” Counters track totals such as llamacpp:prompt_tokens_total (“Number of prompt tokens processed”) and llamacpp:tokens_predicted_total (“Number of generation tokens processed”).

Other gauges describe how busy the server is. llamacpp:requests_processing reports the “Number of requests processing,” llamacpp:requests_deferred the “Number of requests deferred,” and llamacpp:n_busy_slots_per_decode the “Average number of busy slots per llama_decode() call.” The /slots endpoint returns per-slot details including “speed, processed tokens, sampling parameters, etc.”

Watching these figures while adjusting the parallel and batch options shows whether a change actually moved throughput or slot utilization, instead of relying on assumptions about how the settings interact.