llama-swap: On-Demand Local Model Swapping

Running several local language models on one machine usually means starting and stopping servers by hand or keeping every model resident in memory at once. llama-swap takes a different path. It is a Go-based proxy that sits in front of local model servers and switches between them automatically based on which model a request asks for. The project is published under the MIT license at https://github.com/mostlygeek/llama-swap and describes itself as having zero dependencies, requiring only a single binary and one configuration file.

How The Swap Works

llama-swap exposes an OpenAI and Anthropic API compatible endpoint. When a request arrives, the proxy reads the model value from the request body and loads the matching server configuration. If a different upstream server is already running, llama-swap replaces it with the correct one. That replacement is the swap the project is named for.

By default the proxy handles one model at a time, which keeps memory use predictable on a single machine. A matrix configuration changes that behavior, allowing multiple models to load concurrently with custom swap logic defined through a small DSL. Models can also unload themselves automatically after a period of inactivity through a ttl setting.

Configuration

Setup centers on a config.yaml file. A minimal configuration defines a models section, a model ID used in API calls, and a cmd value that holds the command to start the underlying server. A ${PORT} placeholder is filled in with an automatically assigned port number when llama-swap launches the process.

Beyond the basics, the file supports aliases, custom environment variables through env, a cmdStop command for graceful Docker or Podman shutdown, reusable snippets via macros, and hooks that run tasks on startup, including preloading a model. Request filtering is available through stripParams, setParams, and setParamsByID, and an API key option can restrict access to the endpoints.

Supported Backends

Any OpenAI compatible server can run behind llama-swap. The documentation names llama.cpp’s llama-server, vLLM, tabbyAPI, and stable-diffusion.cpp, and notes that llama-server is the best supported. Python servers such as vLLM or tabbyAPI are recommended to run through Podman or Docker so that environment isolation and proper SIGTERM handling are in place.

The proxy serves a broad set of endpoints. On the OpenAI side these include chat completions, completions, responses, embeddings, model listing, audio speech and transcription, and image generation and edits. Anthropic compatible routes cover messages and token counting. llama-server specific endpoints such as rerank, infill, and completion are also passed through.

Built-In Tooling

llama-swap ships with its own management surface in addition to the model endpoints. A /health route returns a simple status, a /running route reports active models, and a /metrics route exposes Prometheus formatted data. A web UI served at /ui provides a playground for testing models, token metrics, request and response inspection, manual model loading and unloading, and live log streaming.

Installation is available through Docker, Homebrew, MacPorts, WinGet, pre-built binaries for Linux, macOS, Windows, and FreeBSD, or by building from source with Go and Node.js. For anyone juggling multiple local models on limited hardware, the project turns model selection into a property of each request rather than a manual operations task.