attention

This survey presents a systematic taxonomy of memory mechanisms in large language models, classifying along axes of representation, update dynamics, and persistence, and formalizing the underlying mechanistic components.

MM-ShiftKV is a training-free method that improves KV cache selection for multimodal LLMs by approximating decoding-time query behavior during prefilling, reducing memory footprint while preserving performance.

This paper presents the first mechanistic interpretability study of Multi-head Latent Attention (MLA), analyzing how its low-rank bottleneck separates content from positional information and reshapes transformer circuits.

A novel patient-independent multimodal framework for automatic depression detection integrates BiLSTM with intra- and cross-modal attention and domain-adversarial training to improve generalization across speakers. It achieves state-of-the-art accuracy of 93.2% on the Androids-Corpus dataset.

Proposes RED-PIM, an algorithm-architecture co-design that reduces inter-bank data movement from O(N^2) to O(N) and shrinks attention matrices, achieving significant inference time reductions (16% to 99.99%) for transformer models.

This technical report explores adding lightweight depthwise convolution to the query/key/value projections in Transformer blocks for LLMs, providing local inductive bias that improves downstream accuracy with negligible parameter cost.

CacheBlend, a EuroSys 2025 Best Paper, solves the problem that 90% of KV cache is never reused due to rigid prefix-matching in prompt caching. By selectively recomputing only boundary tokens between documents, it achieves 2-4x faster multi-document processing without quality loss, implemented in the open-source LMCache layer.

Introduces Looped Latent Attention (LLA), a post-training codec that compresses KV cache in looped transformers by exploiting low-rank structure across recurrence steps, achieving significant compression ratios while maintaining performance.

This paper applies Wilsonian renormalization group theory to analyze Transformer attention as a perturbation of the MLP residual-stack fixed point, determining whether attention is relevant or irrelevant based on data correlation length. Experiments on synthetic Markov chains confirm that attention's relevance depends on the spectral structure of the data-generating process, with the first-layer head dominating the transition.

Presents a continuous geometric framework modeling Transformer operations as integro-differential equations on a semantic fiber bundle, validated across multiple architectures.

Introduces Support Vector Attention (SV-Attention), a trainable max-margin memory that provides certified selection of tokens with zero weight and exact unlearning via a reversible incremental solver. It achieves improved rare-item recall and patient-record deletion capabilities.

UnisonAI is a zero-parameter, exact-fractional geometric engine that derives intelligence from mathematical law rather than training, based on the Smithian Fold Theory, and claims to outperform trained models like GPT on held-out text.

This paper proposes a learnable Dirichlet-process cache that allocates memory slots only for novel inputs, enabling efficient associative recall with a cache size proportional to distinct items rather than token count. It combines DP-means clustering with recurrent backbones, demonstrating effectiveness on associative recall benchmarks and real-world streams.

Recent analysis reveals that retrieval-based tool selection for LLM agents caps out at recovering ~23% of failures, while readout-side interventions addressing attention biases recover 59-91% of failures, indicating that the real bottleneck is in the model's output processing rather than input filtering.

This blog post from PyTorch presents novel kernel fusion techniques for normalization ops like LayerNorm and RMSNorm, achieving significant speedups by reducing memory-IO overhead. Techniques include Lazy Pre-Norm and Multi-CTA Norm Fusion, hiding up to 90% of normalization latency when fused with GEMMs, and the FlashNormAttention algorithm achieving up to 35% kernel speedup.

Meta introduces techniques like Lazy Pre-Norm, Multi-CTA Norm Fusion, and FlashNormAttention to fuse normalization operations with GEMM and Attention kernels, hiding up to 90% of normalization latency on NVIDIA B200 hardware and achieving up to 35% latency reduction in attention blocks.

KQ-SVD is a new method for KV-cache compression that directly approximates the attention matrix via optimal low-rank decomposition, achieving 5-10x lower error than key-only SVD on LLaMA and Mistral models.

This paper proposes a cross-modal generative framework that synthesizes fetal Doppler ultrasound waveforms from fetal-maternal electrocardiograms, using cross-modal attention and dilated convolutions, achieving improved synthesis quality and quantifying the influence of maternal-fetal coupling.

This tutorial demonstrates how to profile various attention implementations in PyTorch, from naive attention to scaled dot-product attention with different backends, using the PyTorch profiler on an NVIDIA A100 GPU.

This paper introduces FourierQK, a method that applies FFT-based frequency-domain preprocessing to learned query and key projections in transformer attention, achieving significant validation loss reductions on character-level language modelling. The approach preserves the full attention score structure and demonstrates reproducible gains over standard dot-product attention.