Tag
提出DSpine，一种通过网络深度注入相邻因果条件的并行推测解码drafter，在SGLang中实现高效并行执行，较DFlash在Qwen3-8B上平均接受长度提升27.8%、吞吐量提升23.3%。
Koa-action is a Salesforce AI framework that formulates low-latency atomic decision tasks (classification, scoring, endpointing) as single-token constrained generation via atomic label tokens and supervised fine-tuning, achieving competitive accuracy with several-fold faster inference than frontier models like GPT-5 and Claude-4.8-Opus.
This paper presents the first systematic study of zero-knowledge-friendly quantization for LLMs, formalizing the concept and evaluating nine models (including Qwen2.5-14B and Qwen3-30B-A3B) across weight, activation, and nonlinear lookup table precision. It finds that activation precision and RMSNorm inverse-square-root lookups dominate ZK proving costs and that conventional low-bit quantization heuristics do not translate to proportional proving savings.
TORQUE 提出一种在旋转前后联合优化保留哪些坐标以高精度的量化框架，在固定比特预算下证明 top-k 保留可最小化误差上界，并在 KV 缓存压缩、激活压缩和近邻检索上取得更好的精度-存储权衡。
This post promotes vLLM, a high-throughput inference engine for LLMs, and details its sessions and activities at PyTorchCon North America.
Forlinx Embedded has launched an M.2 AI accelerator card based on Rockchip's RK1820 and RK1828 processors, providing 20 TOPS of INT8 performance for local LLM inference on embedded Linux and Android systems.
Benchmarking distributed LLM inference on Kubernetes reveals unauthenticated control-plane ports and security exposures, with mitigations like NetworkPolicy and WireGuard impacting performance.
The paper introduces Parallel Power Tempering (PPT), an inference-time sampling method that uses multiple parallel tempering chains at different sharpening levels to resolve the exploration-exploitation trade-off in power-sharpened LLM sampling, outperforming RL post-training and approaching frontier-model reasoning performance.
This paper identifies 'phase sensitivity' in models using chunked KV-cache compression, showing that retrieval accuracy oscillates in cycles matching the compression stride and can flip answers or shift long-context accuracy by up to 40 points at 128K tokens, an effect confirmed via controlled 0.6B pretraining experiments to stem from the compression itself.
This paper explores approximating softmax in pretrained LLMs for kernel acceleration, demonstrating performance gains like up to 25.8% speedup on Blackwell B200 with minimal perplexity impact.
The article explains why replacing a dense feed-forward layer with a top-2 Mixture of Experts (MoE) in LLMs can increase inference latency due to communication overheads in multi-GPU setups, despite reducing FLOPs per token.
DwarfStar is a specialized inference engine designed to run large language models like DeepSeek and GLM efficiently on consumer hardware, supporting multiple platforms and advanced features like SSD streaming.
At PyTorch Conference North America, Maajid Khan from Fujitsu Research India will present a talk on optimizing Mixture of Experts LLM inference on ARM CPUs using vLLM and OpenVINO.
User shares performance results and setup details for running the Qwen3.8-Flash-Next model with Llama.cpp on a system featuring an NVIDIA 5090 GPU and 64GB RAM, noting unexpected low RAM usage during inference.
Quail is an open source AI-SQL engine that integrates query planning with LLM inference to achieve over 1 billion input tokens per minute on an H100 GPU.
The blog post details the implementation of watermarking in vLLM for establishing text provenance in AI-generated content, leveraging randomness in sampling to balance non-distortion, robustness, and speed.
This paper synthesizes research on efficient distributed LLM serving, connecting vLLM and llm-d as complementary layers and proposing an Inference Execution Planner for future scheduler development.
The paper proposes a Context-to-Answer-Aligned Memory Compression (CMC) framework that compresses long input contexts into compact memory embeddings to reduce LLM inference costs without modifying decoder weights, achieving significant performance and efficiency gains.
stuntd is a local proxy server that learns from your traffic to serve typed LLM decisions locally using the Laya model, compatible with the Jev protocol and requiring no API key.
The paper introduces PAGE, a partition-aware gated KV-cache eviction method that uses a scalar metric to predict input classes and apply eviction only when safe, reducing accuracy degradation in large language models.