Tag
Today, we release an experimental DSpark draft model for our vision-language model (VLM) LFM2.5-VL-3B, which adds a speculative decoding path for faster inference with minimal memory cost.
This paper introduces KITE, a KV-invariant transformer expansion method that efficiently scales LLMs by reducing inference costs while maintaining performance. It presents the SST model that achieves lower training loss and reduced inference cost compared to baselines.
This paper introduces Planned Test-Time Scaling (PTTS), a method that coordinates reasoning branches to enhance performance on challenging tasks, achieving significant gains over repeated sampling in mathematical reasoning benchmarks.
The paper proposes a Context-to-Answer-Aligned Memory Compression (CMC) framework that compresses long input contexts into compact memory embeddings to reduce LLM inference costs without modifying decoder weights, achieving significant performance and efficiency gains.
This paper introduces Multi-Split Boundary Decision (MSBD) to reduce inference costs in zero-shot page stream segmentation using large language models, demonstrating improved efficiency while maintaining accuracy for appropriate window sizes.
InstinctFlash is a high-performance serving framework for robotics models that enables real-time inference of 5B world-action models on Jetson Thor, with reported speedups of up to 33.78×.
PRQuant is a training-free and low-overhead framework for quantizing linear layers in large language models, using permutation and residual compensation to reduce inference latency while improving accuracy over baselines like MXFP4.
TreeSpark introduces calibrated, load-adaptive draft trees for semi-autoregressive speculative decoding in language models, increasing draft token acceptance by 15-25% and speeding up decoding by 8-14%.
Model Grafting technique modifies Qwen3.5-4B into a causal encoder-decoder, creating variants with up to 3.7x speedup in prompt processing and minimal accuracy loss.
This tweet recommends a Stanford course on efficient generative language models, covering techniques from pre-training to inference to balance performance and cost with limited compute.
This paper proposes L0-MoE, a lightweight Mixture-of-Experts approach using L0-regularization to accelerate dense Large Language Models with up to 2.5x speedup while maintaining competitive performance.
Elastic Threshold Attention (ETA) is a trainable sparse attention architecture that improves long-context decoding speed without quality degradation by using dynamic thresholds predicted from query representations.
Baseten's 'Inference Engineering' is a systematic book that explains AI inference optimization techniques from CUDA to production deployment, helping engineers efficiently run open-source models in production environments.
This article describes a fork of llama.cpp called focus-llama that implements Declarative Attention from a recent paper, allowing models to declare needed context chunks during inference to optimize KV cache usage and reduce decode time.
An unofficial fork of llama.cpp introduces a persistent expert pool for MoE models, optimized to reduce expert re-copies over PCIe on 16GB AMD gfx906 GPUs, thereby improving decode throughput for large context lengths.
The user has renamed a GitHub repository to HyperQwen to focus on optimizing Qwen model inference speeds on local hardware and is seeking testers with 4090s and 5090s GPUs for both Windows and Linux.
The paper investigates budgeted repair methods for stale KV caches in LLM systems after document edits, demonstrating that contiguous edit-local windows efficiently recover performance and are faster than full re-prefill.
This paper introduces a calibrated instrument for measuring how inference optimizations impact the output quality of AI models.
VideoMM introduces an adaptive macro-micro inference framework that reduces visual token overhead in video MLLMs, achieving a 6.13× speedup and 7.4% accuracy gain for efficient long-form video understanding.
The paper proposes 'Early-Bird Decoding,' a framework to accelerate diffusion large language models by using learnable block sizes and parallel sampling, achieving significant throughput improvements without modifying pretrained weights.