decoding

Fable 5 demonstrates impressive ability to decode shorthand where users type the first letter of each word, accurately reconstructing the intended message.

Set Diffusion introduces a new class of language models that interpolates between autoregressive and diffusion models by factorizing token generation over flexible-position, flexible-length token sets. This enables faster decoding and flexible token ordering, achieving better speed-quality tradeoffs on reasoning, summarization, and unconditional generation tasks.

Proposes Dual-Confidence Contrastive Decoding (DCCD), a training-free method for retrieval-augmented generation that handles intra-context conflicts in multi-document settings by combining document-level and token-level confidence signals, and introduces the DRQA benchmark for factual-conflict QA.

Dr. Julie Elie of UC Berkeley won the 2026 Coller-Dolittle Prize for decoding the core vocabulary of zebra finches, identifying 11 distinct calls by combining observation, machine learning, and behavioral experiments. Her work marks a significant step in interspecies communication.

This paper proposes FADE, a training-free method that mitigates hallucinations in Large Vision-Language Models by attenuating FFN outputs at critical layers to reduce language-prior dominance, demonstrating effectiveness across multiple benchmarks.

This paper reinterprets masked diffusion language model decoding as continuous clean-state prediction, introducing a flow-based framework where tokens are updated continuously and asynchronously based on confidence, achieving 97% of LLaDA's performance with 25% of the decoding budget.

This paper proposes Multi-Block Diffusion Language Models (MBD-LMs), extending single-block diffusion to concurrent multi-block decoding with improved training strategies like Multi-block Teacher Forcing and an optimized Block Buffer decoding algorithm. Experiments show increased tokens per forward pass and improved accuracy on benchmarks.

A tweet listing 16 inference optimization techniques for achieving sub-second LLM responses, including KV-caching, speculative decoding, FlashAttention, and various parallelism methods.

Introduces Speculative Refinement (SpecRef), a training-free hybrid decoding strategy that warm-starts a masked diffusion language model from an autoregressive draft using entropy-guided selective masking. Evaluated across six benchmarks, it reveals that code benchmarks conflate structural discovery with logical correctness, identifies a refinement tension phenomenon, and shows that evaluation protocols can produce different model rankings.

huff12 is a 12-stream Huffman decoder optimized for Apple Silicon processors, aiming to improve decoding performance through parallel stream processing.

A tweet highlights the value of understanding decoding and sampler mechanisms in LLMs for gaining an edge.

This paper proposes TIE, a knowledge fusion framework for masked diffusion language models that tracks confidence dynamics to identify reliable decoding trajectories and iteratively transfers partially denoised sequences between models, improving generation quality on reasoning tasks.

Announced release of Luce KVFlash, a tool to run models inside Lucebox at 256k context without KVCache OOM, achieving up to 2.9x faster decoding at long context using speculative prefill and dynamic offloading.

LatticeBridge proposes a twisted sequential Monte Carlo decoder for structured sequence generation that improves constraint satisfaction by treating the problem as rare-event inference, outperforming greedy and beam baselines on CommonGen, E2E NLG, and WikiBio.

This paper introduces ADAS, a training-free reranking rule for parallel masked diffusion decoding that uses attention to discount tokens that strongly attend to uncertain positions, improving low-NFE performance on reasoning and code tasks with minimal runtime overhead.

Ahmad Osman teases an upcoming article on X that simplifies LLM decoding.

This paper proposes AXON, a training-free module that improves the quality-latency trade-off of discrete diffusion language model decoding by intelligently selecting 'anchor' tokens to reveal first, using attention, uncertainty, and confidence signals to support subsequent denoising steps. Experiments on reasoning and code-generation benchmarks show AXON reduces function evaluations while maintaining or improving accuracy.

COFT is a training-free decoding method that applies token-level fairness control and conformal calibration to reduce bias in chain-of-thought reasoning of large language models, achieving 30-55% bias reduction with minimal computational overhead.

This paper systematically investigates when and which parts of the prompt KV cache become dispensable during LLM decoding, showing that redundancy primarily involves chat template scaffolding rather than task content, and replacement with neutral filler preserves accuracy.

Based on the SGLang Omni team's internal decision-making article, this post introduces the operating principles of LLM inference systems in an accessible way, starting from basic concepts such as autoregressive decoding, KV cache, and continuous batching.