glance-vlm speedlab is now open source!

read: https://glance.yohei.me/speed/ try: https://github.com/yoheinakajima/glance-speedlab…

turn any webcam into multiple live AI detectors (emotion, count, object), running locally

⚡ Recorded live on an Apple M5, one-question loop: PyTorch/MPS FP16: ~210 ms p50 MLX 8-bit: ~160 ms p50

🧪 Controlled nine-question, fresh-frame benchmark: 358.5 → 259.6 ms p50 27.6% lower latency 84/84 decisions matched

21 experiments, reproducible benchmarks, paper, and failures

Glance Speedlab — a local VLM latency study

Source: https://glance.yohei.me/speed/

Abstract

Live camera VLM systems are often described by model throughput, but their useful latency is a product of capture, transport, vision encoding, repeated text work, decoding and scheduling. We built a local-first lab around Glance and ran 21 preregistered experiments to separate those costs.

Native multi-question batching produced a 2.405× speedup. A smaller 2B model was 2.994× faster than the 4B reference but failed the quality guardrail. The accepted 8-bit MLX direct scorer reduced fresh-frame p50 from 358.5 to 259.6 ms (1.381×), matched 84/84 fixed-suite decisions and kept maximum probability drift to 0.039.

The main lesson is methodological: optimize realized model work, not configuration labels or proxy counts. Several attractive changes reduced nominal tokens or precision without producing a safe end-to-end win.

1. System and method

Glance answers typed questions about a frame by reading the probability of constrained answer tokens directly. It does not generate prose. The live system wraps that scorer with camera capture, a scheduler and a same-origin local gateway.

camera → crop / resize → local gateway → vision encoder
       → shared-prefix question batch → answer-token probabilities
       → scheduler / temporal policy → browser telemetry

Each experiment declared a hypothesis, primary metric, quality guardrail, hardware, version and stopping rule before implementation. Accepted speedups had to survive paired measurement and preserve decisions within the declared tolerance. The fixed suite used nine statements across reusable test images; raw frames and model weights are excluded from the public artifact.

**Apparatus.**Apple M5, 32 GB unified memory. Reference: Qwen3-VL-2B-Instruct via Glance, PyTorch MPS FP16. Candidate: Qwen3-VL-2B-Instruct via MLX, 8-bit.

2. Results

InterventionMeasured resultDecisionNative multi-question request2.405× faster; exact decisions within toleranceKeep8-bit MLX direct scoring358.5 → 259.6 ms p50; 84/84 decisions; max drift 0.039Ship experimentally2B instead of 4B2.994× faster; 83.3% agreementFast tier only4-bit MLX direct scoringSpeed held; maximum probability drift reached 0.361RejectFixed-shape suffix compilation0.6% p50 improvementBelow materialityUniform vision-token reductionUp to 15.6% faster; one decision changed, drift reached 0.411RejectUntrained early decoder exits6–22% faster; no tested depth passed both gatesRejectBrowser payload workBase64 and JSON cost at most 0.1 ms p95 at 320 pxDeprioritize The MLX result was not accepted on latency alone. It preserved every fixed-suite decision in the paired run, replicated at 1.334×, and was then integrated into Glance behind an explicit backend flag. PyTorch remains the compatibility default.

Native batching was the clearest architectural win: one shared visual prefix and one multi-question request avoided repeated setup. By contrast, browser serialization was measurable but too small to explain user-visible latency.

3. What did not work

• Quality fail**4-bit quantization.**It retained speed but moved probabilities too far for a dependable scorer.
• No material win**Compilation.**Fixed-shape suffix compilation improved p50 by only 0.6%.
• Quality fail**Uniform token cuts.**Lowering visual resolution removed useful evidence along with compute.
• Quality fail**Raw early exits.**Shallower decoder states were faster, but were not trained to be calibrated decision heads.
• Wrong bottleneck**Transport micro-optimization.**At the tested frame size, model compute dominated the browser-to-loopback path.

Keeping these results matters. They narrow the search space and expose a repeated failure mode: a cheaper representation is not automatically a cheaper or equivalent computation.

4. Interpretation

The successful changes reduced work the hardware actually performed: shared-prefix batching removed duplicate model passes, while MLX paired an Apple-native runtime with accepted 8-bit weights and direct answer-token scoring. The rejected changes mostly optimized a proxy—declared precision, token count or graph form—without securing both realized latency and decision stability.

The evidence points toward conditional compute rather than one globally smaller model:

cheap temporal gate
    ↓ changed enough
fast 2B / MLX scorer
    ↓ uncertain or fine-detail
full reference path

This cascade is still a hypothesis. The synthetic temporal gate reduced triggers by 95.1%, but requires labeled real-camera sequences before it can support a public efficiency claim.

5. Limitations

• The primary measurements come from one Apple M5 machine with 32 GB unified memory.
• The fixed suite is deliberately small and measures decision preservation, not broad semantic capability.
• Thermal state, other Apple chips and lower-memory machines were not characterized.
• The temporal result is synthetic; no real-camera quality or staleness study has been completed.
• This is a working technical report and open artifact, not peer-reviewed evidence.

6. Next hypotheses

1. A trained intermediate decision head can retain full-depth margins while recovering the 11–22% compute exposed by raw truncation.
2. Learned or task-conditioned vision-token selection can recover the measured prefix opportunity without uniform-resizing quality loss.
3. Model-shaped fused 8-bit Metal kernels can improve the accepted MLX path beyond generic eager execution.
4. A calibrated 2B→4B cascade can retain most of the 2B speedup while escalating uncertain or fine-detail frames.
5. Task-aware temporal reuse can lower effective compute on labeled camera streams while bounding staleness.

7. Reproducibility and availability

The stable typed-question protocol, direct probability readout, shared-prefix batching and optional MLX backend live in Glance. Speedlab contains the live A/B interface, client telemetry, benchmark scripts, failed variants and full research record.

Yohei (@yoheinakajima): all three at a time sub 0.5 second latency!

• expression
• object detection
• finger count

switched to qwen 8B via glance i think direct on MLX