llama.cpp Adds Full MCP Support with Tools & UI

./llama-server --mcp-server filesystem --mcp-server-arg path=/home/user/docs

This command launches llama.cpp’s server with Model Context Protocol (MCP) support, connecting the language model to a filesystem server that provides file access capabilities. The recent integration brings standardized tool use to one of the most popular local LLM inference engines.

Overview

llama.cpp now implements the Model Context Protocol, an open standard developed by Anthropic for connecting AI models to external tools and data sources. The integration transforms llama.cpp from a pure inference engine into a platform capable of executing function calls, accessing databases, reading files, and interacting with APIs through a standardized interface.

MCP defines how models discover available tools, construct function calls, and receive structured responses. Instead of each application implementing custom tool-calling logic, llama.cpp can now work with any MCP-compliant server. The protocol handles the communication layer between the model and external resources, while llama.cpp manages the inference and function call generation.

The implementation supports both the server and client sides of MCP. As a server, llama.cpp exposes model capabilities to MCP clients. As a client, it connects to MCP servers that provide tools like file systems, databases, or web search. This bidirectional support makes llama.cpp a versatile component in agent-based workflows.

Installation and Configuration

Building llama.cpp with MCP support requires enabling the feature during compilation:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DLLAMA_MCP=ON
cmake --build build --config Release

The MCP feature adds dependencies for JSON-RPC communication and WebSocket support. Once compiled, the llama-server binary includes MCP endpoints alongside the existing HTTP API.

Configuration happens through command-line arguments or a JSON config file. Multiple MCP servers can run simultaneously, each providing different tool sets. A typical setup might include filesystem access, SQLite database queries, and HTTP request capabilities:

{
  "mcp_servers": [
    {
      "name": "filesystem",
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/workspace"]
    },
    {
      "name": "sqlite",
      "command": "mcp-server-sqlite",
      "args": ["--db-path", "./data.db"]
    }
  ]
}

The web UI automatically detects available MCP tools and displays them in the interface. Models can then invoke these tools during conversation without additional configuration.

Usage Examples

Function calling works through the standard chat completion API. Models trained for tool use (like Llama 3.1 or Mistral variants) generate structured function calls when appropriate:

import requests

response = requests.post('http://localhost:8080/v1/chat/completions', json={
    "model": "llama-3.1-8b-instruct",
    "messages": [
        {"role": "user", "content": "What files are in the current directory?"}
    ],
    "tools": "auto"
})

# Model generates a function call to list_directory
# llama.cpp executes it via the filesystem MCP server
# Returns results in the next message

The web interface provides visual feedback during tool execution. When a model requests file access or database queries, the UI shows the function call, execution status, and returned data. This transparency helps debug agent behaviors and understand decision-making processes.

Multi-step workflows combine multiple tool calls. A model might read a configuration file, query a database based on its contents, then write results to a new file. llama.cpp handles the orchestration, executing each function call and feeding results back to the model for the next decision.

Limitations and Considerations

MCP support requires models specifically trained for function calling. Base models or instruction-tuned variants without tool-use training produce unreliable results. The model must generate properly formatted function calls matching the JSON schema provided by MCP servers.

Performance overhead exists for tool-heavy workflows. Each function call adds latency as llama.cpp communicates with external MCP servers, waits for execution, and processes results. Complex multi-step tasks can take significantly longer than pure text generation.

The implementation currently supports stdio and SSE (Server-Sent Events) transports for MCP communication. WebSocket support remains experimental. Some MCP servers may not work correctly depending on their transport requirements.

Security considerations matter when exposing filesystem or database access to language models. MCP servers should run with minimal permissions, and production deployments need careful sandboxing. The filesystem server in particular requires explicit path restrictions to prevent unauthorized access.

Documentation for the MCP integration remains sparse compared to core llama.cpp features. Developers need familiarity with both the MCP specification (https://spec.modelcontextprotocol.io) and llama.cpp’s architecture to troubleshoot issues or extend functionality.