llm-serving

Tag

Cards List
#llm-serving

To Keep or Not to Keep: Learning KV Cache Retention in Disaggregated LLM Serving Systems

arXiv cs.LG ↗ · 2026-10-22 Cached

提出 KVLearn——一种面向分离式 LLM 服务架构的基于学习的 KV 缓存保留框架,通过前缀复用预测、成本感知保留分数和自适应阈值控制,端到端首 token 延迟最多降低 56%,节点间 KV 传输量最多减少 53%。

0 favorites 0 likes
#llm-serving

@rohanpaul_ai: Some of the most useful inference work right now is happening in the speculative decoding component. It has become one …

X AI KOLs Following ↗ · 2026-10-21 Cached

EAGLE-4 发布,作为投机解码加速组件在 Qwen3.8-27B 上达到 961.6 tokens/s,单请求下比 DFlash 2 快 1.28 倍、比 DSpark 快 1.40 倍,通过 draft-and-verify 循环实现约 3 倍加速且输出完全一致。

0 favorites 0 likes
#llm-serving

Eagle-4 Released: Faster than DFlash2 & DSpark

Reddit r/LocalLLaMA ↗ · 2026-10-20 Cached

EAGLE-4 发布:新一代投机解码草稿模型与 eagle-harness 工具包,可跨批量大小、硬件和工作负载自适应加速 LLM 推理,速度超过 DFlash2 与 DSpark,支持 SGLang、vLLM 和 TokenSpeed 引擎。

0 favorites 0 likes
#llm-serving

24GB is plenty if your engines take turns

Reddit r/LocalLLaMA ↗ · 2026-10-20

Berth is a system service that automatically swaps and loads AI engines on a single GPU, managing VRAM so tools like ComfyUI, llama-swap, and OpenWebUI don't squander memory or collide. It also exposes engines via a unified API endpoint for seamless switching across workloads, including gaming.

0 favorites 0 likes
#llm-serving

@svpino: November 3rd — Forge '26 — San Francisco. This is probably one of the biggest AI events in the world. NVIDIA's Jensen H…

X AI KOLs Timeline ↗ · 2026-10-19 Cached

Forge '26, hosted by Fireworks AI on November 3rd in San Francisco, is billed as one of the biggest AI events of the year, featuring fireside chats with NVIDIA's Jensen Huang and DHH plus a keynote from Fireworks CEO Lin Qiao. The free, application-only event focuses on how top teams choose models, design evals, serve them, and route requests in production.

0 favorites 0 likes
#llm-serving

Where Prefill Expert Reduction Hurts MoE Models: A Few Prompt Tokens Carry Most of the Short-Answer F1 Loss

arXiv cs.CL ↗ · 2026-10-19 Cached

该论文研究在 MoE 模型 prefill 阶段减少路由专家对短答案问答 F1 的影响,发现大部分损失来自提示词首尾少数敏感 token,并提出将约 4% 的 token 保持全专家以几乎无损地降低 TTFT。

0 favorites 0 likes
#llm-serving

Efficient CPU-GPU collaborative MoE inference via Adaptive Expert Assignment

arXiv cs.AI ↗ · 2026-10-19 Cached

The paper presents Apex, a CPU-GPU collaborative MoE inference system that adaptively assigns GPU-missed experts to either CPU or GPU based on token load, improving prefill speed by up to 51.9% and decode throughput by up to 39.5% over the strongest baseline.

0 favorites 0 likes
#llm-serving

Intel LLM-Scaler-vLLM PV 1.5 brings big improvements

Reddit r/LocalLLaMA ↗ · 2026-10-16 Cached

Intel released LLM-Scaler-vLLM PV 1.5, a major update to its containerized vLLM stack for Arc Graphics hardware, jumping from vLLM 0.14 to 0.26, adding new Qwen and Gemma model support, FP8/INT4 quantization, and performance improvements.

0 favorites 0 likes
#llm-serving

ModelSphere launches as an open-source LLM deployment platform for mixed GPU fleets

Reddit r/artificial ↗ · 2026-10-16

Phancy Group and RiseUnion launched ModelSphere, an open-source LLM deployment platform on GitHub under Apache 2.0, targeting production serving on mixed GPU fleets with Kubernetes, Helm, SGLang, and vLLM integration. Reception is still early, and its listed features like autoscaling and shared KV cache remain unverified project claims.

0 favorites 0 likes
#llm-serving

@jerryjliu0: congrats on the launch!!

X AI KOLs Timeline ↗ · 2026-10-16 Cached

Zakaria 介绍 overshoot:一款优化 AI 团队推理栈的软件,号称可将 GPU 成本减半且无质量损失,用户累计已节省超 500 万美元。Jerry Liu 转发祝贺其发布。

0 favorites 0 likes
#llm-serving

AbridgeKV: Compressing Long Reasoning into the KV Cache of Learned Pause Tokens

arXiv cs.CL ↗ · 2026-10-16 Cached

AbridgeKV trains models to compress completed blocks of generated chain-of-thought text into the KV entries of a few learned pause tokens, achieving a 3.5× smaller cache and 3.2× faster decoding on a saturated H100 with no accuracy loss at a quarter of the full KV budget.

0 favorites 0 likes
#llm-serving

When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill

Hugging Face Daily Papers ↗ · 2026-10-15 Cached

该论文研究长上下文 LLM 服务中远程 KV 缓存层级的性能瓶颈,提出按并行 rank 分片 MLA KV 存储、异步加载等优化,将 p99 延迟从 83.32 秒降至 0.67 秒,并显著提升预填充吞吐。

0 favorites 0 likes
#llm-serving

Moving a running model's layers between GPUs mid-reply (8 ms, no re-prefill)

Reddit r/LocalLLaMA ↗ · 2026-10-14

开发者展示开源推理引擎 jitllm,借助 JIT 内核生成与分页 KV 缓存,可在推理过程中把 Llama-3.2 模型各层在不同 GPU/CPU 之间动态迁移(最短 8 毫秒暂停、无需重预填充)。

0 favorites 0 likes
#llm-serving

Instead of waiting for an M5 Ultra 512 GB: pairing a new M5 Ultra 256 GB with my M3 Ultra 512 GB for disaggregated prefill/decode (GLM-5.3-Flash, MLX, Thunderbolt 5 RDMA)

Reddit r/LocalLLaMA ↗ · 2026-10-13

A detailed technical writeup on pairing an M5 Ultra 256 GB for prefill with an M3 Ultra 512 GB for decode, using disaggregated inference over Thunderbolt 5 RDMA with GLM-5.3-Flash on MLX, cutting time-to-first-token on long prompts from ~40s to ~18.5s.

0 favorites 0 likes
#llm-serving

Flama: 8 years building a Python framework where ML models and LLMs are first-class, now written up on arXiv [P]

Reddit r/MachineLearning ↗ · 2026-10-13

作者介绍 Flama——一个将 ML 模型和 LLM 作为一等公民的 Python 框架,八年间从模型部署发展到 2.0 支持 LLM,并与 Miguel Durán-Olivencia 一起将整体架构写成一篇 83 页的 arXiv 论文。模型被打包为单一 .flm 文件,支持 vLLM/MLX 后端以及 OpenAI、Anthropic、Ollama 等多种协议。

0 favorites 0 likes
#llm-serving

Lecture: Building a GPU Compiler From Scratch. Lowering PyTorch Graphs to CUDA via Multi-Level IRs (GPU_MODE video + article)

Reddit r/LocalLLaMA ↗ · 2026-10-12

作者从零构建了一个 GPU 编译器 emmy,展示如何通过多级中间表示(IR)将 PyTorch 计算图逐步 lowering 为 CUDA 内核,并发布了讲座视频、文章和开源代码。在多个基准测试中其生成的内核超越 Torch Inductor 和 cuBLAS,且支持 vLLM 端到端模型推理服务。

0 favorites 0 likes
#llm-serving

KV-Repurpose: A Testbed for Evaluating Non-Prefix KV-Cache Reuse

arXiv cs.CL ↗ · 2026-10-12 Cached

本文提出 KV-Repurpose,一个基于 vLLM 的开源测试平台,用于评估超越精确前缀匹配的分段 KV-cache 复用;在 Zipfian RAG 负载下,其 token 级缓存命中率达到 63.9%,远高于前缀缓存的 3.3%,并显著降低 TTFT。

0 favorites 0 likes
#llm-serving

@rohanpaul_ai: New Tsinghua paper builds TokenRouter, a serving system that runs per-token small-and-large model routing at upto 64.15…

X AI KOLs Timeline ↗ · 2026-10-10 Cached

清华团队发表 TokenRouter 论文,提出一种按 token 级别在大小模型间路由的 LLM 服务系统,通过独立服务器协作与 KV cache 传递,在 5 种路由方法上实现相比现有最强框架 2.01 到 64.15 倍的吞吐提升。

0 favorites 0 likes
#llm-serving

PR merged: `/v1/systemone` support in vllm

Reddit r/LocalLLaMA ↗ · 2026-10-10 Cached

vLLM 合并了对 `/v1/systemone` 的支持,作为其开源 LLM 推理与服务库的最新功能更新。

0 favorites 0 likes
#llm-serving

RaReCache: Bridging the Gap in Cross-Model KV Cache Reuse via Rank disagreement-based Selective Recomputation

arXiv cs.AI ↗ · 2026-10-09 Cached

RaReCache 提出一种基于秩分歧(rank disagreement)的选择性重计算框架,使大型模型能够直接复用小型模型预填充的 KV 缓存,在大幅缩小参数差距的同时保留 95-99% 的目标精度,并实现最高 3.04 倍的预填充加速。

0 favorites 0 likes
← Previous
Next →
← Back to home

Submit Feedback