ik_llama.cpp Adds Graph Split Mode for Multi-GPU

ik_llama.cpp is a fork of llama.cpp focused on better CPU and hybrid GPU/CPU performance, described in its README as a “llama.cpp fork with additional SOTA quants and improved performance.” Maintained by GitHub user ikawrakow, the project recently gained a new multi-GPU split mode aimed at running large mixture-of-experts models more efficiently across several CUDA cards.

A New Way to Split Work Across GPUs

The feature arrived through pull request 1022 at https://github.com/ikawrakow/ik_llama.cpp/pull/1022, which the author describes as “a very rough around the edges POC for tensor parallelism (TP) for MoE models.” It was first written for GLM-4.5 and GLM-4.6 style models. During development the option was renamed from “split mode row” to “split mode graph,” reflecting how it works.

Standard layer mode distributes a model’s layers across GPUs, so each card owns a consecutive block of the network. Graph mode instead splits the computation graph itself, allowing work to run in parallel across GPUs up to synchronization points. A core difficulty was that the scheduler tended to create splits where one GPU’s output became another GPU’s input, which forced the cards to run sequentially. The solution involved arranging two splits that can run in parallel and then combining their results in a separate step.

What the Benchmarks Show

The pull request reports results on a system with two RTX 3090 GPUs running GLM-4.5-AIR. For prompt processing, graph mode beat layer mode at every context length tested, reaching roughly 60 percent faster at 30k tokens. Token generation behaved differently: at zero context it was actually slower than layer mode, and it only pulled ahead at longer context lengths, around 20k tokens.

A run using a Q8_0 KV cache at 55k tokens showed prompt processing about 66 percent faster and token generation about 42 percent faster than layer mode. The author also noted that results on an L3-70B model beat mainline llama.cpp’s existing “split mode row” on both prompt processing and token generation.

Limitations and Status

The split mode graph started as a proof of concept and carried several caveats. Token generation under hybrid or partial offload was weaker and needed a sufficiently long context to come out ahead of layer mode. Full GPU offload was required at first because tensor overrides were not yet implemented, and CPU plus GPU hybrid inference was not initially supported. Early bugs affected interleaved and row-interleaved quants and were later fixed, and at least one user hit a CUDA graph execution error that was traced back to a flag mistake rather than the feature itself.

The pull request was merged into the project’s main branch on December 1, 2025. According to the repository README, ik_llama.cpp treats CPU (AVX2 or better, ARM NEON or better) and CUDA (Turing or newer) as the fully functional and performant backends, while other backends are not covered for issue resolution. For people running large MoE models locally on multiple NVIDIA cards, the graph split mode offers another option to compare against the existing layer and row modes depending on context length and offload setup.