Running LLMs Locally on AMD Radeon GPUs With ROCm

AMD has published a practical guide describing how to run large language models locally on Radeon GPUs. The walkthrough, available on the company’s ROCm blog at https://rocm.blogs.amd.com/artificial-intelligence/language-models-locally/README.html, covers the tools, runtimes, and setup steps needed to move from a downloaded model to a working local inference setup on AMD hardware.

Hardware and Software Foundation

The guide is written for systems with an AMD Radeon GPU, and it distinguishes between integrated GPUs and discrete GPUs rather than targeting a single specific card. It references several RDNA architecture build targets, including gfx1100 for the RDNA 3 architecture, gfx115x for RDNA 3.5, and gfx12xx for RDNA 4. These identifiers matter during the build process because the compiled software needs to be matched to the GPU architecture in use.

All of the recommended tools are built on ROCm, AMD’s software platform for GPU computing. The guide notes that ROCm is recommended for best performance, with Vulkan available as an alternative backend. Supported operating systems are Windows and Linux, each with separate instructions. There are no instructions for macOS.

Tools and Runtimes

The guide highlights four options for running models locally. Lemonade is described as a launcher built for AMD acceleration that supports GGUF and ONNX formats across CPU, GPU, and NPU. LM Studio is presented as a desktop environment for downloading, serving, and interacting with models. Ollama is described as a command-line and graphical interface tool. llama.cpp is characterized as the low-level, highly optimized foundation that powers many local AI solutions. Python bindings are also available through llama-cpp-python.

The example model used throughout the walkthrough is Phi-3.5-mini-instruct, a small model with 33 layers. According to the guide, this model needs roughly 2GB of VRAM plus additional memory for the context cache. The size of that cache depends on the chosen context length: a 4096-token context uses about 1.5GB of VRAM for the key-value cache, while an 8192-token context uses about 3GB. The guide suggests a 2048-token context when GPU memory is limited.

Setup and Quantization

The setup process begins with prerequisites including Git, Git LFS, a Conda or Miniconda environment, Python 3.10 or newer, CMake, Ninja, and Clang. The workflow then moves through converting a model, building llama.cpp, optionally quantizing the weights, and running inference.

Model conversion involves creating a Conda environment, cloning llama.cpp, downloading a model from Hugging Face, and running a conversion script to produce a GGUF file. Building llama.cpp uses ROCm flags such as enabling HIP and setting the AMD GPU target to match the card’s architecture, or alternatively building with Vulkan support.

Quantization reduces model size at the cost of some quality. The guide lists Q4_K_M as a recommended balance of size and quality, Q4_K_S for smaller size with slightly more quality loss, Q5_K_M for better quality at a larger file size, and Q8_0 for the highest quality. Once a model is prepared, it can be run through llama-cli, Lemonade, LM Studio, Ollama, or directly from Python. For systems with more than one AMD GPU, the HIP_VISIBLE_DEVICES setting selects which device to use, with device 0 used by default.

Practical Takeaway

The guide gives AMD GPU owners a documented path to running language models on their own machines without depending on cloud services. By covering both Windows and Linux, multiple front-end tools, and quantization choices, it aims to make local inference approachable for users who prefer to keep models and data on local hardware.