Tag
RestoreKV introduces a learned restoration mechanism that complements query-agnostic KV cache eviction, recovering full-cache behavior under aggressive budgets by generating a compact context-conditioned restore cache via a single LoRA-adapted pass, improving performance on four long-context benchmarks.
Masked distillation is a knowledge-distillation framework that trains a student LLM to predict only solution tokens while a reasoning teacher provides feedback, aiming to internalize chain-of-thought computation into model parameters. The method shows task-dependent success, working on GSM8K but requiring small scaffolds for harder tasks like Countdown.
Introduces ADSD, a prompt-suffix attack that causes acceptance collapse in speculative decoding by forcing the draft model to propose tokens the target model is unlikely to accept, increasing inference time while preserving task quality.
MiniCache is a program caching framework that reuses computation across similar requests by parameterizing Program-of-Thought programs, using small models for semantic variable extraction and speculative drafting to improve LLM inference efficiency.
Echo is a system that achieves performance comparable to the Fable model at one-third the cost by efficiently allocating inference across open-weight models. It provides free credits and requires no credit card.
SelKV is a training-free framework for KV cache compression that uses a soft cosine gate for selective merging and an attention-ratio compensation mechanism to correct softmax imbalance, achieving near-lossless generation at 25% cache size and 3.3x decoding speedup on LongBench.
Proposes AdaLook, an adaptive multi-step lookahead decoding framework for masked diffusion language models that dynamically determines rollout depth and branch expansion based on candidate-score variance, achieving better accuracy-decoding steps trade-off compared to existing one-step lookahead decoding methods.
This paper proposes EcoSpec, a cost-aware speculative decoding framework for Mixture-of-Experts models that accounts for expert activation costs during draft selection. It achieves up to 1.62× speedup on large-scale MoE models like DeepSeek-V3.1, Qwen3-235B-A22B, and GPT-OSS-120B by reducing expert footprint without modifying the target model's verification rule.
This paper proposes a recall-controlled abort cascade that uses lightweight probes on LLM agent internal representations to detect and abort doomed episodes early, saving up to 47% inference compute while maintaining high recall of successful episodes.
This paper identifies evaluation inconsistencies in diffusion LLM decoding methods, showing that prompt template choice significantly impacts rankings, and proposes guidelines for reliable evaluation.
BlockPilot proposes an instance-adaptive policy that predicts the optimal block size for diffusion-based speculative decoding, achieving significant speedup with minimal overhead.
This paper introduces LatentMAS, a training-free framework for multi-agent systems that enables large language model agents to collaborate directly in continuous latent space via shared latent working memory, achieving up to 14.6% higher accuracy and 4x faster inference while reducing token usage by over 70%.
Proposes ReLAR, a reinforcement-guided latent refinement framework that iteratively updates hidden representations in LLMs before decoding, improving reasoning reliability and efficiency compared to chain-of-thought methods.
This paper proposes that the KV cache in transformers acts as a notebook of memoized conclusions, enabling surgical editing and composition without full recomputation. The method achieves significant latency reductions while preserving decision equivalence across model scales.
PolyKV is a layer-wise KV cache compression framework that assigns heterogeneous eviction policies and non-uniform budgets per layer, significantly improving over uniform baselines on LongBench with LLaMA-3.1-8B and Qwen3-8B.
Introduces Code2LoRA, a hypernetwork that converts an entire code repository into a LoRA adapter for code LLMs, eliminating inference-time token overhead for repository-level knowledge.
TimeRouter introduces an efficient routing framework for time-series foundation models that uses lightweight discriminative routing and selective gating to adaptively select the best expert model without LLM overhead, achieving state-of-the-art on the GIFT-EVAL leaderboard.
A tweet criticizes current LLM architecture for wasteful recomputation due to order-dependent context, and proposes encoding context units separately to enable order-invariant, efficient caching and generation.
IDPR is a framework for response-conditioned inhibitory deliberation that first generates a fast intuitive answer, then uses an inhibition controller to decide whether to invoke slow reasoning, achieving efficiency gains while maintaining accuracy.
LazyAttention introduces a novel attention mechanism that defers positional encoding to enable zero-copy, position-agnostic KV cache reuse across multiple requests. The approach reduces time-to-first-token by 1.37× and increases throughput by 1.40× compared to Block-Attention in RAG settings with skewed document distributions.