Tag
An open-source LLM engineer's practical handbook called time-to-first-token, offering a 10-week, 50-lesson roadmap to build a production-ready, OpenAI-compatible inference service from scratch — covering vLLM source code, PagedAttention, monitoring, 1000+ concurrency load testing, and quantization optimization.
llama.cpp v0.7.0 released with support for Qwen3.8-Flash-Next GGUF models via MTP-based speculative decoding on 128GB Mac/Spark setups, showcasing efficient local LLM serving.
A survey paper that reframes long-context compression as a memory-management problem for LLMs, unifying prompt compression, KV-cache management, retrieval augmentation, and agent memory into a single information-lifecycle framework with a budgeted cost model.
DLite提出一种无历史上下文的投机解码框架，通过复用目标模型的隐状态进行局部预测并省去历史KV缓存维护，在agentic工作负载下将接受长度提升至2-6，较SOTA方法DSpark提升最高70%。
苹果发布新版 MLX 框架，展示如何在单台 Mac Studio 上运行 Muse Spark Open（近万亿参数）等前沿模型，并通过 RDMA over Thunderbolt 驱动在多台 Mac 间实现分布式张量/流水线并行推理，在四台 M5 Ultra 集群上达到每秒 36 tokens 以上的解码速度。
A community benchmark evaluated 21 Qwen3.8 27B model/KV quantization combinations (189 benchmarks) that fit in 16GB VRAM, finding q5_1/q5_1 is usually optimal and that upping model quant at q4_0/q4_0 beats lower quants at full f16 KV cache.
论文提出 REWIND，一种基于 token 熵动态的训练-free 控制器，用于在测试时对 LLM 推理轨迹进行定点后缀重新生成修复，在三个模型系列和五个基准上超越所有基线。
论文提出联合子空间稀疏注意力（JSSA），通过学习共享低维查询-键子空间并设计免gather的索引注意力核，在128K上下文下实现最高6.43倍解码吞吐提升，同时保持接近全KV的精度。
论文提出 Spectral-MoE，一种在稠密模型转 MoE 时为不同层动态分配专家利用率（utilization budget）的方法，包括静态和动态两种分配器，在 Llama-2-7B、Llama-3-8B、Qwen2.5-7B 上显著降低困惑度并提升下游任务准确率，确立了逐层专家利用率这一新的优化方向。
Oxford researchers directly measure the inference energy efficiency of 86 open-weight LLM checkpoints across seven benchmarks and six hardware platforms, finding large energy disparities among similarly capable models and arguing benchmark scores should be reported alongside measured energy consumption.
提出PN-Mix，一种面向RoPE解码器LLM的混合精度键缓存量化方法，通过查询-键对范数识别显著通道对，在约2.26位平均精度下达到或超越现有方法在2.32–2.39位下的精度，且可优雅降级至1.5位。
ReaDy introduces a speculative decoding method that uses hybrid linear attention recurrent states as dynamic context for drafting, achieving a 3.16x decoding speedup over autoregressive generation with Qwen3.8-27B on an NVIDIA B200 while improving throughput and acceptance length over DSpark across seven benchmarks.
RTX 3090 Founders Edition owners running local LLMs may be cooking their VRAM: benchmarks show stock fan/power curves let GDDR6X hit 112°C within seconds of token generation, far above the 96°C continuous rating. The poster recommends capping the card at 270W with fans at 100% (or 240W at 70% fan) at roughly a 30% throughput cost, and notes undervolting doesn't help because Nvidia only applies clock offsets above 300W.
该推文总结了优化 LLM 推理的 12 种 KV 缓存技术，包括滑动窗口、跨层共享、MLA、量化、前缀复用、分页、驱逐和 GQA/MQA 等，帮助开发者根据具体瓶颈选择合适的组合。
Introduces Constrained Decoding Models (CDMs), a new AI model architecture family (Classify, Extract, Summarize) that skips token-by-token generation to deliver faster, cheaper results — claiming 200x speed and 467x cost improvements over frontier LLM APIs, now live on OpenRouter, Vercel, LiteLLM, and LangChain.
SonicBoom v0.0.1 is a native C++/CUDA tensor runtime with a graph compiler that achieves 52 tok/s decode on a laptop RTX 3050 vs llama.cpp's 45 tok/s using the same ggml-derived quantized kernels, showing that gains now come from the compiler layer rather than hand-written kernels.
Janas 是一个开源工具，通过从 SSD 流式加载专家模型（mixture-of-experts），让内存小于模型体积的笔记本也能运行 LLM。在 32GB 内存的笔记本上，Qwen3-Next-80B（48GB）推理速度达 37 tok/s，显著超越 llama.cpp 的 2.8 tok/s。
PyTorch Conference North America (San Jose, Oct 20-21) will feature Carlos Costa of IBM Research and co-founder of llm-d sharing news on how the llm-d project has matured to deliver state-of-the-art performance for generative AI serving, including multi-tenant QoS and zero-downtime operations.
strata-mlx update: interrupted long conversations no longer get reread — time to first character for a 10K-token session drops from 19.7s to 0.26s (M4 Max), and memory management on 32 GB Macs is optimized to avoid swap. The project is an Apple Silicon MLX inference engine for running 125B MoE models like Qwen3.8-Flash-Next, with throughput significantly better than llama.cpp and LM Studio.
PreSTO 提出一种预测性子树预取方法加速 PowerMH 功率采样器，通过批量预取未来的 MH 提案并保持转移核不变，在七个数据集和五个 LLM 上实现 1.71×–2.33× 的中位加速。