Tag
vLLM's nightly build now supports PLE disk offload, helping offload KV-cache or memory pressure to disk for LLM inference serving. The update is available on vLLM's nightly channel.
A creator is building a work-in-progress YouTube playlist/course covering LLM training and inference topics such as distributed training, fine-tuning, RL, OPD, serving, speculative decoding, and optimizers, and is asking for suggestions to make it more complete.
Modal 宣布达成 10 亿美元年化营收并完成 D 轮融资，同时继续推广其面向 AI 工作负载的容器化云基础设施平台，涵盖推理、训练、沙盒与函数计算。
Hugging Face's Open TTS Leaderboard compares not only TTS models but also inference implementations, benchmarking streaming performance (TTFA), inference speed, speech quality (WER/CER, speaker similarity), and GPU/CPU results—useful for anyone building local voice agents.
@NVIDIARTXSpark shares three techniques to accelerate Mixture-of-Experts (MoE) inference on the NVIDIA DGX Spark, covering optimized kernels, FP8 quantization, and multi-token prediction to boost tokens per second.
OpenRouter shipped its biggest provider router update since founding, shifting from price-based prioritization to rewarding the best quality inference per cost so requests stay cheap while being fast and reliable, with providers earning more traffic by delivering better value.
Vercel 宣布与 Particle 合作，通过 Vercel AI Gateway 提供 Particle 的模型与推理服务，旨在让开发者和企业更便捷地获取智能能力。
Modal CEO Erik Bernhardsson 在播客中讨论算力市场仍高度非标准化、NeoCloud 与 NeoLab 的商业模式风险、推理市场走向商品化，以及 GPU 价格未来 12 个月可能因大买家持续囤积而上涨。
DeepSeek open-sourced DeepInfer, its inference stack that reuses KV cache from SSDs via 3FS to deliver first tokens up to 26x faster, enabled by MLA's 93% KV cache compression.
作者发布了开源项目 DecisionFormer，可将普通 LLM 转换为直接预测选择而非生成文本的决策模型，适用于决策模型实验与推理方式探索。
DiffuSpace 发布 Clover-35B-A3B-Preview，首个开源块扩散语言模型，采用 35B 参数 MoE 骨干（约 3B 激活参数）和 128K 上下文窗口，宣称在发布时达到开源扩散语言模型的最先进性能。
新分析指出，最便宜的 LLM 并不总是完成任务的最便宜方式：在会话中途切换模型可能因丢失上下文缓存而抵消成本节省，智能路由应优化整个工作流而非仅看模型价格。
论文研究了循环语言模型中深度可复用性的来源，发现保持循环步数可外推的关键不是循环边界归一化，而是训练与推理时循环拓扑的一致性（agreement）；移除 Ouro 模型的边界归一化会在训练预算内将准确率降至零，但从头训练的小模型显示不加归一化反而外推更好。
Ubicloud explains how vLLM's new constrained decoding support (server-side flag in vLLM 0.31) fixes tool-call schema accuracy, boosting it to 100% for newer Kimi and DeepSeek models after only 92% compliance with vLLM 0.28.
This arXiv paper reformulates structured generation as an ambiguity-aware inference problem, replacing single-reference supervision with a multi-valid learning framework that represents candidate outputs as graphs and selects among them via graph-based reasoning over semantic alignment and relational consistency, yielding improved accuracy and robustness under ambiguity.
Hugging Face 发布了一个 torch-neuronx 模型支持查询 Space，可查看哪些 Hub 上的模型能在 AWS Trainium 上直接运行，并提供可复现的运行脚本和 NEFF 缓存以跳过编译。目前仅覆盖 text-generation 与 image-text-to-text 类 Transformers 模型，未来将扩展到 diffusers 和 sentence transformers。
作者基于 Artificial Analysis 的评分绘制了 11 个开放权重模型的参数量-得分对比图：816B 的 Meta Muse Spark Open 得分 39，而仅 180B 的 Qwen3.8-Flash-Next 得分 40；图表还包含 MiMo-V2.6-Pro、DeepSeek V4.1 Flash 等模型及量化体积、本地部署建议等信息。
OpenAI 正式向所有用户开放其 Decisions API，基于 ChatGPT Luna 重新配置以实现超低延迟与低成本推理，直接与 TypeSafe 的 Jev 等决策模型展开竞争。
A llama.cpp pull request discussion about improving strata decode performance on an AMD R9700 GPU, referencing the ggml/llama.cpp C/C++ LLM inference project.
MIT Technology Review examines the overlooked data storage costs of AI inference, noting that a single 8-second AI-generated video can leave behind 500MB of back-end data (130x the user-visible file) and citing IDC/McKinsey forecasts of 35% CAGR inference demand growth through 2030.