attention

This paper proposes Energy-Gated Attention (EGA) and Morlet Positional Encoding (MoPE) to address missing inductive biases in transformer attention: token salience and scale-adaptive locality. Experiments on TinyShakespeare show superadditive gains when combined, highlighting complementarity.

NestedKV is a training-free KV cache compression method that uses nested memory routing with multi-time-scale anomaly scoring to improve long-context language model efficiency, achieving significant gains on benchmarks like RULER and LongBench.

This paper explores how an exponentially decaying memory module from RAT+ can improve query-aware sparse inference methods for long-context language models, demonstrating consistent accuracy gains across various sparse budgets on needle-in-a-haystack tasks.

A paper explores letting language model agents 'sleep' to reset internal state and improve performance on long-horizon tasks, addressing context length scaling issues.

Meta open-sources TLX Block Attention, a warp-specialized Triton kernel that achieves 2.3x speedup for block-diagonal self-attention on NVIDIA Blackwell GPUs, with up to 3.5x speedup when fused with rotary embeddings.

This paper argues that extreme context sparsity is a principled and feasible foundation for LLM inference, showing that current models tolerate up to 100× sparsity without quality loss and that sparse decode kernels can accelerate processing by 10× on existing hardware.

This paper theoretically characterizes the representational capacity of Neural Process (NP) architectures, proving a strict hierarchy among Conditional, Attentive, Convolutional, and Transformer NPs, and showing that finite-dimensional latent variables do not expand representational capacity beyond the encoder.

A 12-chapter interactive textbook that teaches how to build a GPT-like language model from absolute scratch, with fully annotated code and beginner-friendly explanations.

Delta Attention Residuals is a drop-in upgrade to residual connections that routes over deltas instead of cumulative hidden states, achieving sharper cross-layer routing and 1.7-8.2% lower perplexity at scales up to 7.6B parameters, and enabling fine-tuning of pretrained models like Qwen3-0.6B with negligible overhead.

ThriftAttention proposes a selective mixed-precision attention method that computes a small fraction of query-key blocks in FP16 and the rest in FP4, achieving near-FP16 quality with FP4 efficiency for long-context inference.

Tensor Cache introduces a two-level caching mechanism that compresses evicted key-value pairs from sliding-window attention into a fixed-size associative memory, improving long-context language modeling without unbounded memory growth.

A thread explaining the mathematical foundations behind key transformer concepts including attention, scaling factor, backpropagation, gradient descent, cross-entropy loss, RoPE, and RMSNorm.

DeepSeek's KV cache compression innovations, including MLA and CSA/HCA, reduce KV cache size by 93%, enabling efficient long-context inference and SSD-based caching, as demonstrated by antirez's ds4.c project.

An educational thread explaining 11 key lessons for understanding and building LLM architectures from scratch, covering tokens, embeddings, attention, positional encoding, data quality, and common misconceptions.

EntmaxKV introduces a support-aware sparse decoding framework for entmax attention that reduces KV-cache memory traffic by exploiting sparsity before loading pages, achieving significant speedups on long-context benchmarks while maintaining output quality.

SEGA is a training-free method that improves high-resolution text-to-image generation by adaptively scaling attention across RoPE components based on spatial-frequency structure during denoising steps.

This paper introduces Exact Linear Attention (ELA), a mechanism that achieves linear computational complexity for Transformer attention without approximation error by leveraging kernel decomposition, and addresses gradient explosion and token dilution through constrained kernel functions. It also presents engineering innovations including Hyper Link, Memory Lobe, and a routing bias for Mixture of Experts.

The paper decomposes the attention interaction matrix into routing (skew-symmetric) and filtering (symmetric) components, introducing S-D attention to disentangle them. It reveals a spectral cascade in routing that predicts where attention can be simplified, achieving significant parameter reduction with minimal perplexity loss.

This paper identifies imbalanced attention head groups in MLLMs that drive or resist modality-conflict hallucination, and proposes MACI, a causal intervention that suppresses hallucination-driving heads only when conflict is detected, achieving large hallucination reduction across five models.

Lighthouse Attention is a selection-based hierarchical attention mechanism that accelerates long-context pretraining by running forward+backward passes ~17× faster at 512K context and delivering 1.4–1.7× end-to-end speedup at 98K context, validated with Llama-3 530M on 50B tokens.