Qwen2-VL 2B Vision Model Runs In The Browser

A community ONNX conversion of Qwen2-VL-2B-Instruct makes it possible to run a vision-language model directly inside a web browser using Transformers.js, the JavaScript library that brings Hugging Face models to client-side environments. The model card for this build is published at https://huggingface.co/onnx-community/Qwen2-VL-2B-Instruct.

Qwen2-VL is a multimodal model that accepts both images and text and produces text in response. The 2-billion-parameter instruction-tuned variant is the smallest of the line, and this repository packages it with ONNX weights so it can be loaded and executed by Transformers.js rather than a Python backend. The model is released under the Apache 2.0 license.

How The Browser Build Is Packaged

The repository is a conversion of the original Qwen/Qwen2-VL-2B-Instruct model into ONNX format. According to the model card, the weights are provided “to be compatible with Transformers.js.” The conversion script referenced on the page splits the model into separate ONNX files for the text embeddings, the text decoder, and the vision encoder, using the transformers, optimum, and onnxslim tooling. Splitting the model this way lets the runtime load each component as needed.

Because Transformers.js can run ONNX models in the browser, the build does not require a hosted inference server. The model card notes that no inference provider currently deploys this repository, which is consistent with its intended use as a client-side model.

Running It With Transformers.js

The model card lists two ways to use the build. The first is the high-level pipeline API:

import { pipeline } from '@huggingface/transformers';

const pipe = await pipeline(
  'image-text-to-text',
  'onnx-community/Qwen2-VL-2B-Instruct'
);

The second is a more detailed example that loads the processor and model directly, reads and resizes an image, builds a conversation with an image and a text instruction, and generates a response:

import { AutoProcessor, Qwen2VLForConditionalGeneration, RawImage }
  from "@huggingface/transformers";

const model_id = "onnx-community/Qwen2-VL-2B-Instruct";
const processor = await AutoProcessor.from_pretrained(model_id);
const model = await Qwen2VLForConditionalGeneration.from_pretrained(model_id);

const image = await (await RawImage.read(url)).resize(448, 448);
const conversation = [
  { role: "user", content: [
    { type: "image" },
    { type: "text", text: "Describe this image." },
  ]},
];
const text = processor.apply_chat_template(conversation, { add_generation_prompt: true });
const inputs = await processor(text, image);
const outputs = await model.generate({ ...inputs, max_new_tokens: 128 });

The example on the model card uses the task type image-text-to-text and demonstrates the model describing the contents of a photograph in response to a “Describe this image” prompt. The package is installed with npm i @huggingface/transformers.

What This Means For Developers

Running a vision-language model in the browser keeps image data on the user’s device, since inference happens locally rather than being sent to an API. The model card positions this as a conversational, instruction-tuned image-text-to-text model, so descriptive and question-answering use cases over images fit its design.

The model card itself focuses on packaging and usage rather than benchmark numbers, so claims about accuracy, throughput, or hardware requirements should be measured against the specific deployment rather than assumed. As the smallest model in the Qwen2-VL family, the 2B variant trades some capability for a footprint that is more practical to download and run on a client device.