llm-serving

Tag

Cards List
#llm-serving

GitHub - T-Crypt/halocline: Sparse + Dense. Qwen4 family on one RTX 4090 (sm_89 only)

Reddit r/LocalLLaMA ↗ · 18h ago Cached

Halocline is an open-source MIT-licensed inference engine that runs Qwen mixture-of-experts and dense models on a single RTX 4090 (sm_89), featuring sparse and dense paths with competitive quality and speed against llama.cpp and NInfer.

0 favorites 0 likes
#llm-serving

Optimize the Cob, Then the Kernel: Optimizing Beam Search for Generative Recommendation Inference

arXiv cs.LG ↗ · yesterday Cached

The paper proposes a holistic optimization pipeline for generative recommendation inference, using Tree Attention and other compiler-level optimizations to eliminate redundant KV cache accesses and host-side overhead, achieving up to 25× lower end-to-end latency than vLLM on Ascend NPUs without custom kernels.

0 favorites 0 likes
#llm-serving

Fuse-All MoE Inference Kernel with Load Balancing on TPUs

arXiv cs.LG ↗ · 4d ago Cached

Meta SuperIntelligence Lab researchers present Fuse-All MoE, a fused Pallas TPU kernel that co-optimizes expert computation, output communication, and load balancing for MoE inference. The approach improves serving throughput by up to 34.5% over the Megablox EP16 baseline, with up to 21.3% lower decode kernel latency and substantial prefill load-balancing gains.

0 favorites 0 likes
#llm-serving

Basalt 0.4.0: More models, still as fast - Qwen 27B up to 770 tok/s on a 5090

Reddit r/LocalLLaMA ↗ · 4d ago

Basalt 0.4.0 adds multi-model and router support (DeepSeek V4.1 Flash), video and file support, and its first dense model Qwen 3.8 27B NVFP4, reaching up to 776 tok/s structured output on an RTX 5090 with 92.8% top-1 fidelity vs BF16.

0 favorites 0 likes
#llm-serving

@baseten: We built an inference engine for DeepSeek V4.1-Flash from scratch in Rust and CUDA. 4xB300: ~1,740 TPS decode 8xB300: ~…

X AI KOLs Following ↗ · 6d ago Cached

Baseten open-sourced a from-scratch inference engine for DeepSeek V4.1-Flash built in Rust and CUDA, reporting ~1,740 TPS decode on 4x B300 GPUs, ~2,060 TPS on 8x B300, and 1M-token prefill in about 12 seconds.

0 favorites 0 likes
#llm-serving

Evolving Infra: Agent-Adaptive LLM Serving at Scale

Papers with Code Trending ↗ · 2026-11-09 Cached

提出 Evolving Infra 一种面向智能体工作负载的自适应 LLM 服务框架,解耦计算分配与 KV 缓存管理,实验显示在 GLM-5.2 和 KimiK2.6 上端到端延迟及 P95 TTFT 较 NVIDIA Dynamo 最多降低 4.1x 和 14x。

0 favorites 0 likes
#llm-serving

@rohanpaul_ai: New @cohere paper finds that speculative decoding works well on fine-grained MoE models even at batch size 1, where pri…

X AI KOLs Timeline ↗ · 2026-11-02 Cached

A new Cohere paper shows speculative decoding works well on fine-grained MoE models even at batch size 1, since adjacent tokens share 20%–45% of experts (vs 4%–12% expected), making verification far cheaper than prior work predicted across 10 MoE models tested on H100s.

0 favorites 0 likes
#llm-serving

YSK: Strata and polystrata have this functionality called answer_room

Reddit r/LocalLLaMA ↗ · 2026-11-02

YSK post revealing that Strata/PolyStrata silently caps model thinking at 75% of max_tokens via an 'answer_room' feature and injects a wrap-up nudge in the model's own voice, with no error, flag, or finish reason surfaced to the client.

0 favorites 0 likes
#llm-serving

Lawyer with too many GPUs, part 3: DeepSeek-V4.1-Flash on 2x DGX Spark + an old 3090 box. 2,600+ tok/s prefill to 256K, 322 tok/s structured output, full stack and raw data published

Reddit r/LocalLLaMA ↗ · 2026-10-31

A lawyer and PLI Labs founder publishes full benchmark data for running DeepSeek-V4.1-Flash-1017 (2.75 bpw EXL3) across two DGX Sparks with a 3090 box handling RDMA-served tables and a speculative drafter, hitting 2,600+ tok/s prefill to 256K context and 322 tok/s structured output. All code, launch scripts, and raw JSON are released publicly, with the author openly noting where his build loses to competing 2x Spark setups.

0 favorites 0 likes
#llm-serving

vLLM now has PLE disk offload support on nightly

Reddit r/LocalLLaMA ↗ · 2026-10-30 Cached

vLLM's nightly build now supports PLE disk offload, helping offload KV-cache or memory pressure to disk for LLM inference serving. The update is available on vLLM's nightly channel.

0 favorites 0 likes
#llm-serving

MAServeBench: Benchmarking Multi-Agent LLM Serving Systems with Hierarchical Collaboration Workloads

Hugging Face Daily Papers ↗ · 2026-10-29 Cached

MAServeBench 是一个基准测试框架,将多智能体协作作为可配置的负载维度来评估 LLM 服务系统与 GPU/NPU 硬件性能,通过层级语义轨迹实现可复现的依赖追踪与任务级测量。

0 favorites 0 likes
#llm-serving

Building an Inference Stack That Improves Itself (20 minute read)

TLDR AI ↗ · 2026-10-29 Cached

Decagon 介绍了 AutoResearch 系统:由 AI agents 自动诊断推理瓶颈、在生产流量回放环境中验证优化(量化、调度、引擎补丁、投机解码等),报告端到端吞吐提升最高达 2.5x,且生产变更仍需人工审批。

0 favorites 0 likes
#llm-serving

@_avichawla: DeepSeek did it again! They just open-sourced DeepInfer, the inference stack behind cache hits that return the first to…

X AI KOLs Timeline ↗ · 2026-10-28 Cached

DeepSeek open-sourced DeepInfer, its inference stack that reuses KV cache from SSDs via 3FS to deliver first tokens up to 26x faster, enabled by MLA's 93% KV cache compression.

0 favorites 0 likes
#llm-serving

God Does Not Play Dice: Enforcing Batch Invariance in Agentic Kernel Generation

arXiv cs.AI ↗ · 2026-10-28 Cached

该预印本提出 BIEvolve,通过契约引导的静态依赖分析与跨批次数值一致性动态验证,为 LLM 内核生成代理加入批不变性(BI)保障,将 BI 通过率从 67.6% 提升至 91.9%,同时保留 91.8% 的基线加速收益。

0 favorites 0 likes
#llm-serving

WAKMan: Workload Aware KVCache Manager

arXiv cs.AI ↗ · 2026-10-28 Cached

WAKMan is a workload-aware KV cache manager for LLM servers that automatically selects its eviction policy from live traffic, improving hit rate over vLLM's LRU by 5 percentage points on average across mixed single-turn, multi-turn, and agentic request workloads.

0 favorites 0 likes
#llm-serving

3 Ways to Boost MoE Inference on NVIDIA DGX Spark

Reddit r/LocalLLaMA ↗ · 2026-10-28 Cached

NVIDIA 的这篇技术博客介绍了在 DGX Spark 上加速 Ling-3.0-flash MoE 模型推理的三种优化技术:Humming MXFP4 MoE 内核、LM head 的在线 FP8 量化,以及基于 NEXTN 的多 token 预测投机解码,并分析了组合配置下的集成问题。

0 favorites 0 likes
#llm-serving

TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving

Hugging Face Daily Papers ↗ · 2026-10-27 Cached

TAO 提出一个工具感知的联合调度框架,用于基于 LLM 的智能体服务,通过动态规划在容量约束下选择和放置会话,相比最优基线最高降低 26.8% 延迟、36.3% TTFT 并提升 29.5% 解码吞吐。

0 favorites 0 likes
#llm-serving

Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data

Hugging Face Daily Papers ↗ · 2026-10-27 Cached

The paper shows that MoE channel selectors need no offline calibration data when using top-k selection instead of calibrated thresholds, and introduces a weight-derived reader (ternary sketch + low-rank map) that retains 98.1% accuracy at 17.8% of routed-expert reads while speeding up MoE blocks 1.60-2.69x over optimized vLLM.

0 favorites 0 likes
#llm-serving

Fixing Tool-Call Accuracy in vLLM

Hacker News Top ↗ · 2026-10-26 Cached

Ubicloud explains how vLLM's new constrained decoding support (server-side flag in vLLM 0.31) fixes tool-call schema accuracy, boosting it to 100% for newer Kimi and DeepSeek models after only 92% compliance with vLLM 0.28.

0 favorites 0 likes
#llm-serving

Qwen3.8-FN at NVFP4 ( Basalt vs cachy-flash vs ninfer-ext)

Reddit r/LocalLLaMA ↗ · 2026-10-22

对 Qwen3.8-Flash-Next NVFP4 模型在单张 RTX 5090 上的三个推理引擎(basalt、capy-flash、ninfer-ext)进行全 262k 上下文基准测试,basalt 在所有上下文长度和两种解码模式下均领先(prefill 8.1k-9.2k tok/s,greedy decode 176-265 tok/s),解码排名与 MTP 草稿接受率一致。

0 favorites 0 likes
Next →
← Back to home

Submit Feedback