@DanKornas: Need to run a neural network on-device without adding a stack of runtime dependencies? ncnn is a neural network inferen…

X AI KOLs Timeline Tools
neural-network inference-framework open-source mobile embedded vulkan on-device

Summary

ncnn is a lightweight, open-source neural network inference framework for deploying PyTorch and ONNX models to mobile, embedded, and desktop targets without heavy dependencies, supporting CPU and Vulkan GPU backends.

Need to run a neural network on-device without adding a stack of runtime dependencies? ncnn is a neural network inference framework for AI/ML engineers deploying models to mobile, embedded, and desktop targets. It helps you move PyTorch and ONNX models onto devices by converting them with pnnx, then running inference through CPU or Vulkan GPU backends. Key features: • Model conversion – pnnx converts PyTorch and ONNX models into ncnn format. • Lean runtime – no third-party runtime dependencies or BLAS/NNPACK requirement. • CPU execution – supports ARM NEON optimizations and multi-core scheduling. • GPU acceleration – provides a Vulkan backend on supported platforms. • Deployment options – offers a C++ core, C API, Python binding, examples, and build guides for mobile, desktop, browser, and edge targets. It’s open-source (BSD 3-Clause license; bundled third-party components retain their own licenses). Link in the reply
Original Article
View Cached Full Text

Cached at: 07/27/26, 07:56 PM

Need to run a neural network on-device without adding a stack of runtime dependencies?

ncnn is a neural network inference framework for AI/ML engineers deploying models to mobile, embedded, and desktop targets.

It helps you move PyTorch and ONNX models onto devices by converting them with pnnx, then running inference through CPU or Vulkan GPU backends.

Key features: • Model conversion – pnnx converts PyTorch and ONNX models into ncnn format. • Lean runtime – no third-party runtime dependencies or BLAS/NNPACK requirement. • CPU execution – supports ARM NEON optimizations and multi-core scheduling. • GPU acceleration – provides a Vulkan backend on supported platforms. • Deployment options – offers a C++ core, C API, Python binding, examples, and build guides for mobile, desktop, browser, and edge targets.

It’s open-source (BSD 3-Clause license; bundled third-party components retain their own licenses).

Link in the reply

Similar Articles

Rewriting model inference with CUDA kernels: the bottleneck was not just GEMM [P]

Reddit r/MachineLearning

Author describes building FlashRT, a CUDA-first inference runtime that rewrites model inference paths with C++/CUDA kernels to address bottlenecks beyond GEMM for small-batch/realtime workloads, achieving significant latency improvements on Jetson Thor and RTX 5090. The article discusses lessons on precision (FP8 helpful, FP4 mixed) and the need to bypass generic runtimes for realtime inference.

On-Device Neural Architecture Search

arXiv cs.LG

Proposes a lightweight neural architecture search performed directly on the deployment device for near-sensor computing, validated on sEMG sign language and fault diagnosis datasets, achieving improved accuracy and reduced RAM occupancy.