Tag
Mistral 发布 Timestral 时间序列基础模型家族，包含 5M 到 209M 参数四个版本，其 209M 模型在 GIFT-Eval、TIME 和 fev-bench 三大基准上均排名第一，支持零样本概率多变量预测且可在 CPU 上推理。
Xolotl 是一个基于 Strata 的推理引擎，可在单张 RTX 4090 消费级 PC 上以约 27 tok/s 运行 552B 的 DeepSeek-V4.1-Flash，并即将支持新的 1017 量化版本。
A tinkerer jokingly rebuts Dario Amodei's critique of open-weight models by running MiMo-2.6-Pro (FP8, ~566GB) on a Jetson AGX Thor with 128GB memory at 0.646 tokens/sec — painfully slow, but fully traced and replayable step by step.
该论文提出 Stepped MoE，一个将弹性结构与稀疏门控路由统一的框架，使单一 LLM 能同时适应部署约束和任务需求，在 1B-4B 参数规模间灵活切换并优于稠密模型 2-5%。
This paper presents the first end-to-end benchmark of a voice-first offline AI stack for African language communities, evaluating three instruction-tuned 2–5B models across four quantization formats on NVIDIA Jetson Orin NX and Raspberry Pi 5 hardware, finding that Q4_K_M offers the best size-to-quality trade-off.
A dual-DGX-Spark user reports that a new GLM 5.3 Flash recipe delivers a 50-90% decode speed boost, making it faster than DeepSeek v4.0 Flash while offering better reasoning quality on local hardware.
论文对比了 Jev-1.13.0 等决策模型与大型语言模型在 6G Open RAN RIC 中解释网络意图并生成 A1 策略的性能：LLM 因延迟过高（中位 0.286–2.35 秒）几乎无法满足 1 秒近实时预算，而 Jev 在 99.8% 的调用中达标，且未显示出优于 Jev 的 SLA 改善。
The paper introduces Decode-Latency Feedback Prefill (DLFP), a model-free proportional controller implemented in vLLM that adapts chunked prefill sizing based on observed decode latency, reducing P99 inter-token latency by ~28% on Qwen3-0.6B on an A100, while also documenting a clear generalization failure on larger models and multi-GPU setups.
This paper presents a neurosymbolic router that uses grammatical inference (L* algorithm) to learn a DFA-based dispatcher, sending structured queries to deterministic symbolic solvers and reserving small language models for open-ended problems on edge hardware. Evaluated on a Raspberry Pi 4B, the router achieves 98.3% overall accuracy and runs 8.8x faster than Program-of-Thought baselines.
Thomas Cottenier from Arm will present at PyTorch Conference North America on how AI agents can synthesize customized runtimes using PyTorch components for optimized inference on edge hardware.
RunAnywhere released a comprehensive suite of eight cross-platform SDKs that enable developers to run multimodal AI models locally on any device, automatically routing inference to the best available hardware acceleration including Apple Neural Engine and Snapdragon Hexagon NPU.
The article compares the routes of two AI model architecture camps, Mamba and RWKV: Mamba targets cloud-side enterprise AI platforms, while RWKV targets on-device and edge devices, each adapting to different hardware and inference scenarios.
Liquid AI released LFM2.5-2.6B, a 2.69B parameter model with 128K context and tool calling, optimized for multi-step agent workflows and capable of running at 30 tok/s on a phone with a 1.67GB Q4_K_M GGUF, though coding and knowledge-heavy tasks remain weak compared to larger models.
Creator of bigedgeonmoe open-source codebase enables running massive MoE models (up to 120B parameters) on mobile devices and consumer PCs, achieving 6 tokens/s for Qwen 35B on a mid-range phone.
Microsoft releases VibeVoice-ASR-BitNet, a compressed multilingual ASR model for real-time CPU inference. It achieves 1.6-2.3x faster inference than Whisper.cpp with real-time capability on as few as 3 CPU threads.
StickyMoE proposes a differentiable routing consistency loss that encourages adjacent tokens to activate the same experts in MoE models, reducing expert-swapping overhead and cache misses during inference on edge devices by up to 3.92× while improving perplexity.
BaseRT is a native Metal inference runtime for LLMs on Apple Silicon, achieving up to 1.56x higher decode throughput than llama.cpp and 1.35x higher than MLX across tested models.
This paper presents a cascaded multi-granularity pruning framework for deploying LLMs on Industrial IoT edge devices, achieving up to 13.8x compression with minimal accuracy loss on MHA+GELU architectures while exposing a collapse on GQA+SwiGLU designs.
Liquid AI released LFM2.5-230M, a compact 230M-parameter hybrid model optimized for on-device deployment with fast edge inference speeds (213 tok/s on Galaxy S25 Ultra) and built for agentic tasks via reinforcement learning.
Using TurboQuant, the user achieved 20 tokens per second on a Qwen 3.6 35B MoE model running on a GTX1060 3GB, showcasing impressive performance on outdated hardware.