Tag
该论文研究了扩散大语言模型作为投机解码草稿模型时的两个关键参数——扩散努力程度与接受准则宽松度——的联合变化，发现二者协同调整能够超越单步扩散草稿方法的速度-质量帕累托前沿。
A user shared a helpful guide on LLM inference by @lamhot_ai that covers the high-level details of inference, and added annotations highlighting the most important parts.
Cloudflare disclosed a tenant isolation flaw in its GLM-5.3 Workers AI model where cross-request KV cache exposure leaked other users' data in 155 long-context responses, caused by a SGLang misconfiguration and bug, now remediated with hardened deployments.
SPECMEM提出了一种在部署期间持续维护的工具调用记忆系统，通过在线写回已验证的调用记录并配合LRU淘汰和副作用门控机制，在多会话多用户场景下实现最高1.74倍的墙钟加速，同时避免了不可逆操作的错误。
Dwarkesh Patel shared a set of flashcards on semiconductor manufacturing created with his tutor Will Michaels, along with additional flashcards for other technical podcast episodes and a curated list of further reading resources.
A user asks whether upgrading to 128GB DDR4 RAM alongside dual RTX 3090s (48GB VRAM) would let them run models like DeepSeek V4.1 and GLM 5.3 Flash at low quantizations with ~20 tokens/s, or whether a DDR5 system would be needed.
开发者发布免费工具 ModelAnalysis.si，可跨 450+ 模型计算 API 调用与自托管（RunPod 租用 GPU）之间的成本盈亏平衡点，支持输入/输出 token 量与缓存命中率调节、VRAM 量化规格估算，以及基于实时定价和 ELO 排名的帕累托前沿分析。
该论文系统比较了 2:4 半结构化稀疏化与量化在 LLM 推理中的取舍，发现在仍有更低精度可用时，加入 2:4 稀疏化既更慢又更不准确；只有在达到硬件量化下限（如 FP8/NVFP4）后，2:4 稀疏化才展现出 1.3–1.7× 的线性层加速价值，且需通过重训练恢复质量。
A hobbyist researcher proposes a speculative decoding scheme where a 2-bit model copy on GPU drafts tokens while the full residual (6-8 bit) in system RAM verifies them, only exchanging activations over PCIe. Early measurements suggest 2-bit + 8-bit residual outperforms Q8_0 quality and the 2-bit drafter beats Gemma 4's official drafter on acceptance rate.
Redis promoted three capabilities for reducing AI application costs: semantic caching via LangCache, persistent agent memory, and RAM-to-SSD storage tiering with Redis Flex, framed as a sponsored post by @svpino.
DStill 是一个两阶段轨迹一致性蒸馏框架，将块扩散投机解码中多步精修的 drafter 蒸馏为单步 drafting，在三个模型族上相较单步 DFlash 最高提升 28% 接受长度和 25% 加速，较自回归解码最高达 6.45 倍加速。
llm-d 是一个开源、Kubernetes 原生的分布式 LLM 推理栈，支持 vLLM、SGLang 等引擎，可在 GPU、TPU、XPU 等异构硬件上提供生产级服务，并宣称大幅提升吞吐量与降低 TTFT。
A developer released open_rknpu2_lm_infer, an open-source LLM inference engine for the Rockchip RK3588 NPU written in Rust and C++ that bypasses proprietary vendor libraries, running Qwen3.5-4B at roughly twice the speed of Rockchip's RKLLM SDK with multimodal input and speculative decoding support.
PyTorch Conference North America in San Jose (Oct 20-21) will feature a poster by Bongwoo Bak of Moreh on KV cache block placement, covering vLLM offloading strategies between GPU, CPU, and shared storage. The post also promotes conference registration tiers and pricing.
PyTorch 2.15 brings major MPS backend improvements for Mac users, including FP8 support, 5-6.5× faster MoE model decoding, 8× faster softmax, and dramatic memory reductions for video VAE decoding, driven by Hugging Face's PyTorch team and community contributors.
AWS introduces a push-based NIXL connector in vLLM that moves KV cache transfer coordination off the critical path for disaggregated prefill-decode inference, delivering up to 3x better TTFT and 30% better TPOT on EC2 P5en instances with EFA.
llama.cpp 新增对 Cohere 系列模型的支持，继续扩展其 C/C++ 本地 LLM 推理框架的模型兼容性。
ATEST: Adaptive Resource-Aware Test-Time Scaling for LLM Inference 提出了一种资源感知的测试时缩放框架，联合考虑额外采样的预期价值与当前 GPU 成本，在 GPU 负载变化下动态决定采样何时停止，从而在满足能耗预算的同时保持准确率，并显著减少 GPU 时间与能耗。
作者分享在Strata上从IQ2_XS迁移到Unsloth UD-IQ4_XS时的经验：0.1.45之前缺少sampling配置会默认贪心解码导致模型循环，以及默认STRATA_ROUTE_TAIL_SKIP=7会牺牲4-bit量化质量，并给出配置与校准建议。
A hobbyist built a distributed inference system chaining two gaming PCs, an M4 Pro MacBook, and two iPhone 17 Pro Maxes to run Qwen 3.8 Flash Next (a 48-layer MoE model with 512 experts), achieving 37-39 tok/s at 60k context via pipelined speculative decoding across the device chain.