browser-inference

作者将上周的 MoE 专家按需从磁盘流式加载的方案接入 llama.cpp 的 WebGPU 后端，使 80B MoE 模型（Qwen3-Next，48GB GGUF）也能在 Chrome 标签页中运行，无需为每个模型手写引擎。该方案新增了 ggml-webgpu 的缓冲类型、slot 映射的 MUL_MAT_ID 着色器和驻留逻辑，实测在 M4 Pro MacBook 上以 2.9 tok/s 解码 80B 模型，较小模型可达 22 tok/s。

LocalMind is updated to stream mixture-of-experts weights from disk while generating in a browser tab via WebGPU, letting a 24 GB Mac run a 14.4 GB Gemma 4 26B-A4B and even a 37 GB Qwen3.6 model, with output that matches llama.cpp on the same GGUF files.

Mentria.ai's browser-based inference engine achieves 25-30 tokens per second with a 1-bit 27B parameter model on a 6 GB RTX 3060 laptop GPU using WebGPU, enabling efficient AI inference without server installation.

Hugging Face releases 207 WebGPU kernels for local AI inference in browsers, along with a JavaScript loader library and a benchmarking tool called Fleet.

parakeet.wgsl enables fast, accurate NVIDIA Parakeet TDT 0.6B V2 speech transcription entirely in the browser using raw WebGPU compute shaders and SIMD WASM, with a live demo and open-source library.

Gemma 4 is demonstrated running in-browser via WebGPU at 255 tokens per second, using kernels generated by Fable 5, showcasing efficient on-device inference.

PrintGuard 2.0 is a major rewrite of a few-shot FDM fault detector using a ShuffleNetV2 backbone and prototypical network, now with a single Python engine that runs unmodified on both CPython and Pyodide in the browser via a platform abstraction layer, enabling per-printer sensitivity tuning and fair inference scheduling.

Trained a prompt injection classifier using ml-intern and DeepSeek V4 Flash, achieving 99% F1 with DistilBERT, optimized to ONNX int8 (~65MB) and deployable in the browser via Transformers.js v3.

The article highlights the new WebGPU backend in llama.cpp/ggml, enabling GPU-accelerated local AI model inference in browsers, developed by Reese Levine and team at USCS over the past year and a half.

A developer demonstrates running the Qwen3.6-27B AI model entirely on WebGPU in a browser, though speed is not optimal.