Running LLMs on AMD Strix Halo with llama.cpp

AMD’s Strix Halo platform, sold under the Ryzen AI Max+ branding, pairs an integrated GPU with a large pool of shared system memory. That combination has drawn attention from people running language models locally, because the integrated GPU can address far more memory than a typical discrete consumer card. The llama.cpp project, a widely used inference engine for running models on consumer hardware, maintains a community benchmark thread for AMD’s ROCm (HIP) backend at https://github.com/ggml-org/llama.cpp/discussions/15021, and Strix Halo results have been contributed there.

What the Benchmarks Measure

The llama.cpp ROCm thread uses a standard test so results stay comparable across machines: Llama 2 7B with Q4_0 quantization. The maintainers chose this model because it is simple to compute and small enough to fit on a 4GB GPU. Two figures are reported for each system. The “pp512” number measures prompt processing throughput, and the “tg128” number measures token generation throughput, both in tokens per second. Results are listed twice, once with flash attention disabled and once with it enabled.

It is worth being clear about scope: the published numbers describe the 7B test model, not the largest models the hardware can hold. The thread is a comparison scoreboard rather than a guide to a specific model size.

Strix Halo Results

A contributor reported an AMD Ryzen AI Max+ 395 system with 128GB of LPDDR5 memory. Without flash attention, it reached 911.36 tokens per second on prompt processing and 50.01 tokens per second on generation. With flash attention enabled, prompt processing rose to 1003.53 tokens per second while generation held steady at 49.87 tokens per second. The test model occupied roughly 3.56 GiB.

A separate entry for the Radeon 8060S, the integrated GPU used in the Strix Halo platform, ran in a system-shared DDR5 configuration. It reached 351.36 tokens per second on prompt processing and 47.97 tokens per second on generation without flash attention, and 366.08 and 48.97 tokens per second respectively with it enabled.

The gap between the two entries points to a practical detail the thread calls out directly: for integrated graphics, memory speed and the number of memory channels greatly affect inference speed. The faster LPDDR5 configuration on the AI Max+ 395 processed prompts well over twice as fast as the shared DDR5 setup, even though token generation rates were similar.

Why the Shared Memory Pool Matters

The notable feature of Strix Halo for local inference is the size of the memory the integrated GPU can use. A 128GB pool is far larger than the memory on most discrete consumer GPUs, which means larger models and longer context windows can stay in memory that the GPU addresses directly. The 7B benchmark in the thread does not stress that capacity, but the same hardware that posts those numbers is what holds bigger models.

For people experimenting with local inference, the takeaway from the published data is concrete rather than speculative. Flash attention improved prompt processing on Strix Halo without hurting generation speed, and memory bandwidth was the clearest differentiator between configurations. Anyone benchmarking their own setup can add results to the same thread, since it stays open for new contributions.