GLM-5: 744B Parameters with 40B Sparse Activation

GLM-5 represents a significant architectural shift in large language models, deploying 744 billion total parameters while activating only 40 billion per forward pass through mixture-of-experts routing.

Zhipu AI released GLM-5 in early 2025 as their flagship model, building on the GLM-4 architecture with a dramatically scaled mixture-of-experts (MoE) design. The model divides its massive parameter count across specialized expert networks, selectively engaging roughly 5% of total capacity for each token prediction. This sparse activation strategy allows the model to maintain computational efficiency comparable to dense models one-eighteenth its size while theoretically accessing knowledge distributed across the full 744B parameter space.

Training Approach

The training methodology combines standard transformer pre-training with MoE-specific optimizations. GLM-5 employs a gating network that routes each token to a subset of expert modules, typically activating 8-12 experts from a pool of several hundred. This routing happens at multiple layers throughout the model’s depth.

Zhipu AI trained GLM-5 on a multilingual corpus exceeding 10 trillion tokens, with particular emphasis on Chinese and English language pairs. The training process incorporated several stability techniques essential for MoE models at this scale. Load balancing losses prevent the gating network from routing all tokens to a small subset of popular experts, which would waste the model’s capacity. Expert dropout during training encourages robust routing decisions that generalize beyond the training distribution.

The model uses a modified attention mechanism called GLM’s bidirectional attention, which processes prefix tokens bidirectionally while maintaining causal masking for generated tokens. This hybrid approach aims to improve context understanding without sacrificing autoregressive generation quality.

Notable Results

GLM-5 demonstrates competitive performance across standard benchmarks while showing particular strength in multilingual and reasoning tasks. On MMLU (Massive Multitask Language Understanding), the model achieves scores in the mid-80s, placing it alongside other frontier models released in the same period. Chinese language benchmarks show stronger results, with GLM-5 outperforming comparably sized Western models by 5-10 percentage points on tasks like C-Eval.

The sparse activation architecture provides measurable efficiency gains. Inference latency for GLM-5 approximates that of dense 40-50B parameter models despite the 18x larger total parameter count. Memory bandwidth becomes the primary bottleneck rather than computation, as the model must load expert parameters from memory even when not computing with them.

Code generation represents another area of strength. On HumanEval, GLM-5 scores above 75% pass@1, handling both Python and Chinese-language programming tasks. The model demonstrates improved instruction following compared to GLM-4, particularly for multi-step reasoning problems requiring chain-of-thought decomposition.

Running Locally

Deploying GLM-5 locally presents substantial hardware requirements. The full model requires approximately 1.5TB of GPU memory when loaded in 16-bit precision, necessitating a cluster of high-end GPUs. A typical deployment might use 12-16 NVIDIA A100 80GB GPUs or equivalent hardware.

Quantization offers a practical path to local deployment. 4-bit quantization reduces memory requirements to roughly 400GB, making the model accessible on 6-8 consumer GPUs. Zhipu AI provides official quantized checkpoints through their model hub:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "THUDM/glm-5-744b-4bit",
    device_map="auto",
    trust_remote_code=True,
    load_in_4bit=True
)

tokenizer = AutoTokenizer.from_pretrained(
    "THUDM/glm-5-744b-4bit",
    trust_remote_code=True
)

The MoE architecture creates unique deployment challenges. Unlike dense models where tensor parallelism cleanly distributes computation, expert parallelism requires careful orchestration to minimize inter-GPU communication. Most practitioners rely on inference frameworks like vLLM or TGI that handle expert routing and load balancing automatically.

Trade-offs

The sparse activation design creates a fundamental tension between capacity and efficiency. While GLM-5 contains 744B parameters worth of learned knowledge, each inference pass accesses only a fraction of that capacity. This means the model may underperform dense models of equivalent active parameters on tasks requiring integration of diverse knowledge within a single forward pass.

Expert specialization introduces another consideration. The gating network learns to route certain types of inputs to specific experts, which can improve efficiency but may also create brittleness. Inputs that fall outside the training distribution might route to poorly suited experts, degrading performance more sharply than dense models would.

Memory costs remain substantial despite sparse activation. The entire parameter set must remain accessible during inference, even if most parameters stay inactive. This creates a higher baseline memory requirement compared to dense alternatives, though inference computation stays proportional to active parameters.

The model’s Chinese language focus makes it particularly valuable for bilingual applications but potentially less optimal for English-only deployments where models like GPT-4 or Claude might offer better performance per dollar of infrastructure cost.