20B Parameter Model Runs Locally in Browser

What It Is

A 20 billion parameter language model now runs entirely within a web browser, processing everything on the client machine without any server communication. The implementation relies on WebGPU for hardware acceleration, combined with Transformers.js v4 (currently in preview) and ONNX Runtime Web for model execution.

The model, GPT-OSS-20B, has been converted to ONNX format and optimized for browser deployment at https://huggingface.co/onnx-community/gpt-oss-20b-ONNX. WebGPU provides the computational muscle needed to handle inference for a model of this scale, accessing GPU resources directly from the browser environment. This represents a significant leap from earlier browser-based AI demos that typically maxed out at models under 1 billion parameters.

The technical stack demonstrates how far browser capabilities have evolved. WebGPU exposes low-level graphics and compute capabilities previously unavailable to web applications, while ONNX Runtime Web handles the actual model execution with optimizations specific to browser environments.

Why It Matters

Privacy-conscious applications gain a powerful new deployment option. Medical professionals could analyze patient data without transmitting sensitive information to external servers. Legal teams might review confidential documents using AI assistance while maintaining complete data isolation. Financial analysts could process proprietary information without exposure risks inherent in cloud-based solutions.

Offline functionality becomes genuinely practical for sophisticated AI tasks. Researchers working in remote locations, journalists in areas with unreliable connectivity, or anyone facing network restrictions can access capable language models without internet dependency. The model loads once, then operates independently.

Development workflows shift when models run locally. Prototyping becomes faster since developers can iterate without managing server infrastructure or API rate limits. Small teams and individual developers gain access to capabilities previously requiring substantial backend resources. The barrier to experimenting with large language models drops considerably.

Edge computing scenarios expand beyond mobile apps and IoT devices to include standard web browsers. Organizations can distribute AI capabilities to users without scaling server infrastructure proportionally. This architectural change could reduce operational costs while improving response times since inference happens on local hardware.

Getting Started

Access the live demonstration at https://huggingface.co/spaces/webml-community/GPT-OSS-WebGPU to test the model directly. The interface loads the model files and runs inference entirely within the browser tab.

For implementation, the source code in the demo repository shows the integration pattern:

const generator = await pipeline('text-generation', 
 'onnx-community/gpt-oss-20b-ONNX', 
 { device: 'webgpu' }
);

const output = await generator('Your prompt here', {
 max_new_tokens: 100
});

Browser requirements include WebGPU support, currently available in Chrome 113+ and Edge 113+ with the feature enabled. GPU memory matters significantly - systems with 8GB+ VRAM handle the model more comfortably, though it can run on less with reduced performance.

Initial model loading takes several minutes as the browser downloads and caches the ONNX files. Subsequent sessions load faster from the browser cache. Inference speed varies dramatically based on GPU capabilities, ranging from several seconds per token on integrated graphics to near-real-time on discrete GPUs.

Context

Traditional browser-based models like BERT or DistilBERT typically contain 100-400 million parameters. Running a 20B parameter model represents roughly a 50x increase in scale. Previous attempts at browser-based inference focused on quantized models under 7B parameters, making this implementation a notable expansion of what’s feasible.

WebAssembly-based approaches like llama.cpp compiled to WASM offer an alternative for browser-based inference, generally providing better CPU performance but lacking GPU acceleration. WebGPU’s advantage lies in parallel processing capabilities essential for transformer architectures.

Limitations remain substantial. Model loading consumes significant bandwidth - the ONNX files total several gigabytes. Memory constraints restrict which devices can run the model effectively. Performance lags far behind server-grade GPUs or specialized inference hardware. Quantization techniques could reduce resource requirements but aren’t yet implemented in this demonstration.

The approach suits specific use cases rather than replacing server-based inference entirely. Applications requiring guaranteed response times, serving many concurrent users, or needing the absolute latest models still benefit from traditional API-based architectures. Browser-based inference excels where privacy, offline access, or eliminating server costs outweigh performance considerations.