Tag
AoH is a data-free method that identifies retrieval and streaming heads from the spectral geometry of query-key projections, enabling sparse attention without runtime attention scores. At 50% sparsity it retains 96.5% of full-attention performance while reducing prefill/decode latency and KV-cache memory.
TaskSense introduces a task-centric world modeling framework that uses stochastic spatial attention conditioned on previous latent states and an auxiliary inverse-dynamics objective to focus on control-relevant regions, improving robustness to visual distractions compared to DreamerV3.
This paper introduces Matrix Zonotopic Attention (MZAttn), a context-adaptive value projection for set transformers, and provides a theoretical analysis showing it can represent target operators that standard attention requires depth to approximate. Experiments demonstrate performance gains on high-rank combinatorial set-prediction tasks.
The paper introduces DUD (Decoupled Update Dynamics), a framework that separates Feed-Forward Network and Attention contributions via causal interventions to improve uncertainty quantification and calibration in large language models, outperforming state-of-the-art baselines.
This paper applies graph signal processing to analyze how LLMs internally represent numerical sequences during in-context learning, finding that attention-induced token graphs and hidden-state signals show systematic, context-dependent signatures related to input complexity.
ATFlash introduces a per-RoPE-wavelength distance window that prunes query-key inner-product terms proportional to each frequency pair's wavelength, cutting 37-48% of attention compute with minimal quality loss and up to 1.31x speedups on long-context LLM inference.
AttnLink is a research paper presenting an attention-based framework for schema linking in Text-to-SQL, converting LLM internal attention into continuous relevance scores for schema items. Experiments on Spider, BIRD, and Spider2-SQLite show high mAP scores and millisecond-scale latency.
This paper identifies a numerical underflow failure mode in ALiBi positional encodings that can zero out attention weights, analyzes its impact, and evaluates mitigation strategies for training models with ALiBi.
A tweet highlights an arxiv paper by Michel Fabrice Serret that introduces Transformers and attention mechanisms from an applied mathematics perspective, covering vectorization, multi-head attention, and methods to reduce attention costs like KV caching and latent attention.
ReTopK is a training-free method that accelerates dynamic Top-K sparse attention for long-context LLMs by reusing historical query–support pairs, avoiding full-context scoring and global Top-K selection. It achieves up to 3.07x speedup over Exact Top-K with only 0.50% perplexity increase at 128K context.
Kimi K3 by Moonshot is an open-weight model ranking fourth among 580 models, featuring innovations like Kimi Delta Attention to reduce KV cache memory, Quantile Balancing for expert load balancing, and AgentENV for efficient RL training sandboxing.
The tweet highlights an explainer of Moonshot AI's Kimi K3, a 2.8-trillion-parameter open model using novel Kimi Delta Attention to cut memory growth and speed up long-context inference, with strong performance on agentic and long-horizon tasks at low caching cost.
This blog post derives Kimi Delta Attention step by step from standard softmax attention through linear attention and DeltaNet variants, explaining the state update equations used by recent Qwen and Kimi models.
Pulsar Attention replaces the static anchor in Star Attention with content-aware summaries and attention sinks, reducing FLOPs by 3.3x while outperforming dense attention on long-context benchmarks.
This paper introduces CvAdamW, an AdamW variant that monitors attention specific heat to detect grokking phase transitions and dynamically scales weight decay, achieving grokking on modular arithmetic where baseline fails.
This paper introduces a novel continual learning framework for time series forecasting that uses attention-guided experience replay to enable models to adapt to new data distributions while avoiding catastrophic forgetting, evaluated on benchmarks and real-world piezometric data.
Proposes Topologically Regularized Side-Path (TRSP) to mitigate representation collapse in LLMs by balancing spectral trade-offs between mixing efficiency and information capacity, achieving significant gains on long-context benchmarks.
Baidu's open-source Unlimited OCR model proposes the Reference Sliding Window Attention (R-SWA) mechanism, achieving continuous parsing of dozens of pages with 3 billion parameters, gaining high attention on GitHub and HuggingFace.
E-SpecFormer introduces LiTAN, a Softmax- and LayerNorm-free attention mechanism, enabling efficient end-to-end automatic modulation and covert channel recognition on edge devices with under 10k parameters and FPGA/CPU co-execution speeds of 92μs per frame.
This paper demonstrates that naive train/test splitting on sliding-window sequences can severely inflate or deflate performance metrics in multi-task learning for predictive maintenance, and proposes a leakage-robust evaluation protocol.