Running Multimodal Gemma Locally on Mac with MLX-VLM

MLX-VLM is a Python package for inference and fine-tuning of vision-language models and omni models on a Mac using Apple’s MLX framework. According to the project repository at https://github.com/Blaizzy/mlx-vlm, it handles text, image, audio, and video inputs locally, and its supported models include Gemma multimodal variants that run on Apple Silicon.

What MLX-VLM Does

The package is described as a tool for inference and fine-tuning of vision language models (VLMs) and omni models, which the project defines as VLMs with audio and video support. Installation is a single pip command, pip install -U mlx-vlm, after which the package can generate output through several interfaces.

Those interfaces include a command-line tool such as mlx_vlm.generate, a Python API built around load and generate functions, a FastAPI server with OpenAI-compatible endpoints, and a Gradio-based chat UI. The server exposes endpoints for chat completions, responses, text-to-speech, and speech-to-text, among others.

Input handling spans more than static images. The repository documents single and multi-image analysis for visual reasoning tasks, audio support, and video understanding for captioning and summarization with select models. It also documents combined image and audio inputs for multimodal use.

Gemma and Other Supported Models

The repository references several Gemma multimodal models, including a gemma-3n variant used in audio and image-plus-audio examples, along with larger Gemma instruction-tuned models and assistant drafter variants used for speculative decoding. For certain quantized formats, the README notes that on Apple Silicon using Metal these models work without an extra flag.

Beyond Gemma, the project lists dedicated documentation for a range of vision and OCR models, such as DeepSeek-OCR, ERNIE 4.5 VL, Phi-4 Multimodal, MiniCPM-o, Granite Vision, and Moondream3. For video understanding specifically, the supported models listed are Qwen2-VL, Qwen2.5-VL, Idefics3, and LLaVA.

Performance and Fine-tuning Features

The repository documents several features aimed at running these models efficiently on local hardware. It supports KV cache quantization in both uniform and TurboQuant forms, and activation quantization for the CUDA path. For fine-tuning, the project supports LoRA and QLoRA.

Other documented features include speculative decoding with drafter families such as EAGLE-3 and MTP, continuous batching for higher throughput with concurrent requests, automatic prefix caching that reuses key/value cache state across requests with shared prefixes, and vision feature caching that stores encoded image features. The README also lists structured outputs through OpenAI-compatible JSON schema constraints, per-token log probabilities, and a thinking budget for reasoning models.

Distributed inference is supported across multiple computers by sharding the language model rather than the vision tower. The package is primarily designed for Mac using MLX, with a separate CUDA path available for NVIDIA GPUs that requires activation quantization for certain models. The project is released under the MIT license.

Where It Fits

For developers who want to run multimodal models without sending data to a remote service, MLX-VLM provides a documented path on Apple Silicon. The combination of a Python API, a command-line tool, and an OpenAI-compatible server means existing tooling can target a locally hosted model. The full list of supported models, installation steps, and feature documentation lives in the project repository.