I added OpenVINO support to Laya: 40 ms per question on CPU, 3.4x faster than PyTorch
Summary
Added OpenVINO support to Laya, achieving 40 ms per question on CPU, which is 3.4 times faster than PyTorch.
Similar Articles
Laya (OS Jev) on Mac M4 CoreML Offline (45 decisions per second)
The Laya model runs offline on Apple's M4 chip using CoreML, achieving 45 decisions per second in inference.
Laya model playing Flappy Bird on a CPU using OpenVINO INT8 inference
A 421M-parameter Laya model is demonstrated playing Flappy Bird on a desktop CPU using OpenVINO INT8 quantization, showcasing efficient AI inference on consumer hardware.
convaiinnovations/laya
Laya is an open-source non-autoregressive decision model that provides typed answers with calibrated probabilities, designed for tasks like email triage and conversational AI, showing significant performance improvements over existing models.
laya.cpp: Optimized laya near-instant decision making
laya.cpp is an optimized C++ implementation for near-instant decision making, built on ggml with custom CUDA kernels, demonstrating significant speed improvements over Python for the Laya model.
@LinQ444: jev 和laya的对比 https://github.com/mizorewww/laya-mlx…
Laya-MLX is an open-weight tool for running typed decision AI models locally on Apple Silicon with low latency, providing native inference without cloud APIs. It includes benchmarks showing fast performance on devices like M3 Max.