llama.cpp Adds Step-3.5-Flash & Kimi-Linear-48B

llama.cpp has integrated support for two significant language models: DeepSeek’s Step-3.5-Flash and Moonshot AI’s Kimi-Linear-48B. The additions expand the inference engine’s compatibility with cutting-edge architectures, bringing high-performance reasoning models and extended context capabilities to local deployments.

The Announcement

The llama.cpp project merged pull requests enabling native support for both models in recent commits. Step-3.5-Flash represents DeepSeek’s latest iteration in their reasoning model family, while Kimi-Linear-48B introduces Moonshot AI’s linear attention architecture with a 48-billion parameter configuration. Both models can now run on consumer hardware through llama.cpp’s optimized inference pipeline.

DeepSeek’s Step-3.5-Flash builds on their R1 reasoning framework but emphasizes faster inference speeds compared to previous versions. The model maintains chain-of-thought capabilities while reducing computational overhead during the reasoning process. Kimi-Linear-48B tackles a different challenge: processing extremely long contexts efficiently through linear attention mechanisms rather than traditional quadratic attention.

The integration required architecture-specific modifications to llama.cpp’s model loading and tensor operation systems. Contributors implemented custom attention patterns and layer normalization schemes to accommodate each model’s unique requirements.

Under the Hood

Step-3.5-Flash introduces a modified attention mechanism that splits reasoning into discrete steps while maintaining coherence across the inference chain. The implementation in llama.cpp handles this through specialized KV cache management:

// Simplified example of step-based cache handling
struct step_cache {
    int32_t n_steps;
    int32_t current_step;
    struct ggml_tensor * step_embeddings;
    struct ggml_tensor * reasoning_state;
};

The model’s architecture separates fast-path inference for straightforward queries from multi-step reasoning for complex problems. llama.cpp’s implementation detects reasoning triggers and allocates additional compute resources dynamically.

Kimi-Linear-48B’s integration proved more complex due to its linear attention design. Traditional transformer attention scales quadratically with sequence length (O(n²)), making long contexts prohibitively expensive. Linear attention reduces this to O(n) through kernel-based approximations:

# Conceptual linear attention vs standard attention
# Standard: Q @ K.T @ V requires O(n²) memory
# Linear: (Q @ K_features) @ V requires O(n) memory

The llama.cpp implementation maps these operations to GGML’s tensor primitives, enabling hardware acceleration through CUDA, Metal, and CPU SIMD instructions. The model maintains competitive performance on contexts exceeding 100,000 tokens while using substantially less memory than equivalent quadratic-attention models.

Both integrations support quantization through llama.cpp’s standard GGUF format. Users can run 4-bit, 5-bit, or 8-bit quantized versions depending on available hardware and accuracy requirements. The quantization process preserves model capabilities while reducing memory footprint by 4-8x compared to full precision weights.

Who This Affects

Developers building reasoning-intensive applications gain access to Step-3.5-Flash’s improved inference speed. The model handles mathematical problem-solving, code generation with explanation, and multi-step logical deduction without requiring cloud API access. Applications in education, research assistance, and technical documentation benefit from local deployment options.

Organizations processing long documents find value in Kimi-Linear-48B’s extended context window. Legal document analysis, scientific literature review, and codebase understanding become feasible on local infrastructure. The linear attention architecture makes these workloads practical on hardware that would struggle with traditional long-context models.

Research teams experimenting with reasoning architectures now have reference implementations for both step-based reasoning and linear attention patterns. The open-source integration provides insights into production-ready optimization techniques for novel architectures.

Hardware enthusiasts running models on consumer GPUs benefit from llama.cpp’s continued optimization work. Both models run on 24GB VRAM configurations when properly quantized, bringing capabilities previously limited to high-end server hardware within reach of prosumer setups.

Perspective

The rapid integration of new architectures into llama.cpp demonstrates the project’s maturity as infrastructure for local AI deployment. Supporting Step-3.5-Flash and Kimi-Linear-48B within weeks of their release shows active collaboration between model developers and the inference optimization community.

These additions highlight diverging approaches to model capability expansion. Step-3.5-Flash prioritizes reasoning quality and speed through architectural refinement, while Kimi-Linear-48B tackles the fundamental attention scaling problem. Both represent meaningful progress beyond simply increasing parameter counts.

The availability of these models through llama.cpp accelerates experimentation with advanced capabilities outside centralized API services. Developers can iterate on applications requiring reasoning or long-context understanding without per-token costs or rate limits. This accessibility shifts advanced model capabilities from research demonstrations to practical deployment scenarios.

As model architectures continue diversifying beyond standard transformers, inference engines like llama.cpp serve as critical translation layers between innovation and application. The project’s ability to rapidly adopt new designs ensures that architectural advances reach users without waiting for vendor-specific implementations.