Vendor-agnostic ML inference on production edge devices [R]

Reddit r/MachineLearning Tools
vendor-agnostic ml-inference edge-devices vulkan ncnn face-detection embedding

Summary

Describes using ncnn's Vulkan backend for vendor-agnostic ML inference on production edge devices, achieving 10x speedup over CPU ONNX for face detection and embedding models.

I work on PostSlate, a video editing tool, and this comes out of our own work. We run ML models on-device, face detection and embedding among other things, which means we can't assume anything about the user's GPU. NVIDIA discrete, AMD, Intel integrated, Apple Silicon, all of it. That rules out CUDA immediately, we needed one backend that runs everywhere. We landed on ncnn's Vulkan backend. Numbers on a 4070, fp16: ArcFace R50 (face embedding): 30 ms on ONNX CPU → 3 ms on ncnn Vulkan SCRFD (face detection): 25 ms → 2.5 ms Model size: ArcFace 174 MB (ONNX fp32) → 87 MB (ncnn fp16 weight storage) Of course the real speedup comes from offloading compute to the GPU, but this wouldn't be possible without the power of Vulkan. The speed wasn't even the deciding factor, it's that Vulkan drivers already exist on every machine we ship to. This means that we don't have to force the user to download a specific runtime and no vendor-specific installs. Full writeup with the rest of the numbers: https://getpostslate.com/blog/faster-local-inference
Original Article

Similar Articles

Rewriting model inference with CUDA kernels: the bottleneck was not just GEMM [P]

Reddit r/MachineLearning

Author describes building FlashRT, a CUDA-first inference runtime that rewrites model inference paths with C++/CUDA kernels to address bottlenecks beyond GEMM for small-batch/realtime workloads, achieving significant latency improvements on Jetson Thor and RTX 5090. The article discusses lessons on precision (FP8 helpful, FP4 mixed) and the need to bypass generic runtimes for realtime inference.