GLM-4-9B-Chat Converted to GGUF for llama.cpp

THUDM’s GLM-4-9B-Chat model has been quantized to the GGUF format and published on Hugging Face, making the bilingual model available to run locally through llama.cpp. The conversion was carried out by the community contributor legraphista, who lists the work as a static quantization of the original THUDM/glm-4-9b-chat release. The full set of files is hosted at https://huggingface.co/legraphista/glm-4-9b-chat-GGUF.

GGUF is the file format used by llama.cpp to store models and their quantized weights. Converting a model to GGUF lets it be loaded by llama.cpp and the tools built on top of it, rather than requiring the original framework the model was trained in.

Quantization Options and File Sizes

The original model uses the BF16 (bfloat16) data type, and the GGUF repository offers a range of quantized versions so the model can fit different amounts of memory. The listed options and their file sizes include:

• BF16: 18.81GB
• Q8_0: 9.99GB
• Q6_K: 8.26GB
• Q5_K: 7.14GB
• Q4_K: 6.25GB
• IQ4_NL: 5.51GB
• Q3_K: 5.06GB
• Q2_K: 3.99GB

Smaller quantization levels produce smaller files at the cost of precision, while the larger files stay closer to the original weights. The repository describes the files as static quantizations, and the page also documents an importance-matrix (IMatrix) approach used during the conversion. The quantization itself was produced through the llama.cpp project, with the repository pointing to pull request 6999 on the llama.cpp GitHub for the conversion support.

The model is published with two listed languages, Chinese (zh) and English (en), matching the bilingual nature of the original GLM-4-9B-Chat release.

Downloading and Running the Model

The repository documents downloading individual files with the Hugging Face command line tool. After installing it with pip install -U "huggingface_hub[cli]", a single quantized file can be pulled down:

huggingface-cli download legraphista/glm-4-9b-chat-GGUF \
  --include "glm-4-9b-chat.Q8_0.gguf" --local-dir ./

Once a file is downloaded, the repository shows running it through llama.cpp:

llama.cpp/main -m glm-4-9b-chat.Q8_0.gguf --color -i \
  -p "prompt here (according to the chat template)"

The page notes that prompts should follow the model’s chat template. It provides both a simple template and a version that includes a system prompt, using GLM’s special tokens such as [gMASK], <sop>, <|system|>, <|user|>, and <|assistant|>. Larger files that have been split into multiple parts can be reassembled following the merge instructions in the repository’s FAQ.

Why Local GGUF Conversions Appear

Community GGUF conversions like this one make a model usable without the original training framework or a hosted API. Running the quantized files locally through llama.cpp keeps inference on a user’s own hardware, and the choice of quantization level lets the same model target machines with different memory budgets. For GLM-4-9B-Chat specifically, the conversion brings a bilingual Chinese and English model into the llama.cpp ecosystem, where it can be loaded by any tool that reads GGUF files.

For the complete list of quant types, download steps, chat templates, and FAQ entries, the repository page at https://huggingface.co/legraphista/glm-4-9b-chat-GGUF is the authoritative reference.