inference-efficiency

The paper proposes TSS, a target-side sparsification framework for speculative decoding in domain-specific LLMs, which improves inference efficiency and performance by skipping selected target layers.

AgentRouter is a lightweight classifier for routing steps in agentic workflows to different model tiers, achieving 72% cost reduction with minimal quality degradation compared to frontier-only models.

StepKV is a step-aware KV cache compression method for LLM agents that retains reasoning steps to maintain accuracy under low memory budgets, addressing reasoning continuity disruption in multi-step inference.

Reviser is a novel decoder-only Transformer model that enables revision-capable text generation via autoregressive cursor actions, achieving competitive performance with lower inference compute compared to baselines.

The paper proposes OBC-Prune, a calibration method for pruning large reasoning models that identifies causally important reasoning circuits to improve accuracy and reduce inference overhead on benchmarks like MATH500 and LiveCodeBench.

This paper revisits Kashin-decomposition-based weight quantization for large language models and proposes an improved algorithm using structured orthogonal transforms, reducing computational cost and ensuring numerical stability compared to methods like OPTQ and QuIP.

FlexComp is a method-agnostic framework that decouples compression ratios from training and deployment, allowing a single model to compress LLM context at any ratio using Matryoshka-style training and per-input budget selection for efficient inference.

KVMem virtualizes long agent workspaces by paging KV state across GPU memory and storage, improving inference efficiency and task success on consumer hardware up to 1M tokens.

This survey examines inference-efficiency techniques for video large language models, analyzing cost reductions across frame sampling, encoding, token compression, and language model stages while identifying evaluation gaps.

BeaconKV is a training-free method that uses beacon queries to compress key-value cache for efficient inference in Large Reasoning Models, reducing memory usage and improving throughput without sacrificing accuracy.

This megathread compiles a list of the latest open-source projects, research papers, and hardware innovations focused on optimizing inference, efficiency, and accessibility for open-source LLMs and related technologies.

This paper proposes LAC (LightActor, deepCritic), an offline reinforcement learning method that allocates model capacity to the critic for improved inference efficiency, matching performance of complex generative actors with lower latency.

Z.ai revealed it was behind the viral Ox Alpha model and released GLM-5.3-Flash, a multimodal model designed for extreme efficiency and low-cost inference, performing on par with leading models at a fraction of the cost.

OpenAI has published lab results for its custom ASIC Jalapeño, showing it outperforms NVIDIA's GB200 and GB300 chips and rivals Vera Rubin in inference efficiency.

The paper proposes Attention-Aware Transform Coding (AATC) for compressing KV caches in large language models, achieving near-lossless accuracy at around 5.8x compression by minimizing distortion through attention mechanisms.

The paper introduces Internalized Visual Thinking (IVT), a post-training framework that trains multimodal models to predict future frame embeddings, enabling direct answer generation without synthesizing intermediate images, thereby reducing latency over 5x for proactive video reasoning.

This paper introduces Dual-Flow Transformers, an architecture that decouples prefill and decode computation by adding an auxiliary flow for continuation prediction while sharing weights and the primary KV cache, enabling phase-specific compute allocation and improved efficiency.

This paper introduces LorExperts and BTExperts, router-preserving compression methods for Mixture-of-Experts LLMs that cluster experts and represent non-dominant members as low-rank corrections, improving compression quality over prior methods like D2-MoE.

This paper proposes ReCo, a reward-coordinated compression framework that uses a process-reward estimator to adaptively compress KV-cache, control reflection tokens, and enable early stopping, cutting generated tokens by 37–65% and latency by ~2x across reasoning models while preserving accuracy.

ResKV proposes a KV cache compression method that divides a fixed budget into an exact main cache and a compact residual cache to reconstruct omitted attention contributions, improving performance on LongBench and RULER across multiple backbones.