Tag
Halocline is an open-source MIT-licensed inference engine that runs Qwen mixture-of-experts and dense models on a single RTX 4090 (sm_89), featuring sparse and dense paths with competitive quality and speed against llama.cpp and NInfer.
The paper proposes a holistic optimization pipeline for generative recommendation inference, using Tree Attention and other compiler-level optimizations to eliminate redundant KV cache accesses and host-side overhead, achieving up to 25× lower end-to-end latency than vLLM on Ascend NPUs without custom kernels.
Meta SuperIntelligence Lab researchers present Fuse-All MoE, a fused Pallas TPU kernel that co-optimizes expert computation, output communication, and load balancing for MoE inference. The approach improves serving throughput by up to 34.5% over the Megablox EP16 baseline, with up to 21.3% lower decode kernel latency and substantial prefill load-balancing gains.
Basalt 0.4.0 adds multi-model and router support (DeepSeek V4.1 Flash), video and file support, and its first dense model Qwen 3.8 27B NVFP4, reaching up to 776 tok/s structured output on an RTX 5090 with 92.8% top-1 fidelity vs BF16.
Baseten open-sourced a from-scratch inference engine for DeepSeek V4.1-Flash built in Rust and CUDA, reporting ~1,740 TPS decode on 4x B300 GPUs, ~2,060 TPS on 8x B300, and 1M-token prefill in about 12 seconds.
提出 Evolving Infra 一种面向智能体工作负载的自适应 LLM 服务框架,解耦计算分配与 KV 缓存管理,实验显示在 GLM-5.2 和 KimiK2.6 上端到端延迟及 P95 TTFT 较 NVIDIA Dynamo 最多降低 4.1x 和 14x。
A new Cohere paper shows speculative decoding works well on fine-grained MoE models even at batch size 1, since adjacent tokens share 20%–45% of experts (vs 4%–12% expected), making verification far cheaper than prior work predicted across 10 MoE models tested on H100s.
YSK post revealing that Strata/PolyStrata silently caps model thinking at 75% of max_tokens via an 'answer_room' feature and injects a wrap-up nudge in the model's own voice, with no error, flag, or finish reason surfaced to the client.
A lawyer and PLI Labs founder publishes full benchmark data for running DeepSeek-V4.1-Flash-1017 (2.75 bpw EXL3) across two DGX Sparks with a 3090 box handling RDMA-served tables and a speculative drafter, hitting 2,600+ tok/s prefill to 256K context and 322 tok/s structured output. All code, launch scripts, and raw JSON are released publicly, with the author openly noting where his build loses to competing 2x Spark setups.
vLLM's nightly build now supports PLE disk offload, helping offload KV-cache or memory pressure to disk for LLM inference serving. The update is available on vLLM's nightly channel.
MAServeBench 是一个基准测试框架,将多智能体协作作为可配置的负载维度来评估 LLM 服务系统与 GPU/NPU 硬件性能,通过层级语义轨迹实现可复现的依赖追踪与任务级测量。
Decagon 介绍了 AutoResearch 系统:由 AI agents 自动诊断推理瓶颈、在生产流量回放环境中验证优化(量化、调度、引擎补丁、投机解码等),报告端到端吞吐提升最高达 2.5x,且生产变更仍需人工审批。
DeepSeek open-sourced DeepInfer, its inference stack that reuses KV cache from SSDs via 3FS to deliver first tokens up to 26x faster, enabled by MLA's 93% KV cache compression.
该预印本提出 BIEvolve,通过契约引导的静态依赖分析与跨批次数值一致性动态验证,为 LLM 内核生成代理加入批不变性(BI)保障,将 BI 通过率从 67.6% 提升至 91.9%,同时保留 91.8% 的基线加速收益。
WAKMan is a workload-aware KV cache manager for LLM servers that automatically selects its eviction policy from live traffic, improving hit rate over vLLM's LRU by 5 percentage points on average across mixed single-turn, multi-turn, and agentic request workloads.
NVIDIA 的这篇技术博客介绍了在 DGX Spark 上加速 Ling-3.0-flash MoE 模型推理的三种优化技术:Humming MXFP4 MoE 内核、LM head 的在线 FP8 量化,以及基于 NEXTN 的多 token 预测投机解码,并分析了组合配置下的集成问题。
TAO 提出一个工具感知的联合调度框架,用于基于 LLM 的智能体服务,通过动态规划在容量约束下选择和放置会话,相比最优基线最高降低 26.8% 延迟、36.3% TTFT 并提升 29.5% 解码吞吐。
The paper shows that MoE channel selectors need no offline calibration data when using top-k selection instead of calibrated thresholds, and introduces a weight-derived reader (ternary sketch + low-rank map) that retains 98.1% accuracy at 17.8% of routed-expert reads while speeding up MoE blocks 1.60-2.69x over optimized vLLM.
Ubicloud explains how vLLM's new constrained decoding support (server-side flag in vLLM 0.31) fixes tool-call schema accuracy, boosting it to 100% for newer Kimi and DeepSeek models after only 92% compliance with vLLM 0.28.
对 Qwen3.8-Flash-Next NVFP4 模型在单张 RTX 5090 上的三个推理引擎(basalt、capy-flash、ninfer-ext)进行全 262k 上下文基准测试,basalt 在所有上下文长度和两种解码模式下均领先(prefill 8.1k-9.2k tok/s,greedy decode 176-265 tok/s),解码排名与 MTP 草稿接受率一致。