Tag
An analysis of 847 AI agent runs reveals that larger context windows cause performance drops due to attention cliffs, and Synap is presented as a tool to efficiently manage context and reduce token usage.
This paper introduces Attention-Aware Routing (AAR), a method that enhances Mixture-of-Experts language models by incorporating attention weights into the router, improving mathematical reasoning performance and revealing coupled dynamics between routing and attention.
This paper introduces Complex KDA, an enhanced version of Kimi Delta Attention that combines a delta-rule transformation with a reflection to achieve greater expressivity, outperforming Transformers in some tasks while maintaining efficiency, with open-source code and models available.
This tweet explains how core AI operations like attention and convolution are based on foundational mathematics such as linear algebra and probability, emphasizing the value of understanding these operations over model names.
The article introduces a visualization tool for the attention mechanism in Transformers, allowing users to customize input matrices and see step-by-step computations of attention scores. It is an initial version with plans for future updates based on user feedback.
This article explores two possible architectural hypotheses for the underlying base model of TypeSafe's Jev API: a bidirectional encoder or a modified causal decoder, and analyzes the relevant evidence and implications.
MetaRTL is a two-stage framework for relational table learning that uses meta-path attention to efficiently capture relational semantics, demonstrated effective across multiple datasets.
Meta Engineering extends FlashAttention-4 with MXFP8 support for NVIDIA Blackwell, achieving up to 2.85 PFLOP/s forward performance and integrating into production training workflows like GEM.
The article describes an optimization for the DeepSeek-V4.1-Flash AI model on Metal hardware, where the attention kernel was improved to only launch necessary tiles, reducing compute waste and enhancing inference speed.
This paper proposes an attention-based finite-volume scheme for shock transport, using a CFL-conditioned flux to enable efficient and accurate simulations with larger time steps.
Proposes CRFCAN, a complex-valued residual network for joint channel and phase noise estimation in sub-THz OFDM systems, which achieves end-to-end recovery with physics-inspired structure and outperforms conventional and state-of-the-art deep learning models.
This article explains in detail why the KV cache stores K and V but not Q in large language models, and discusses prefill, prefix cache, and architectural innovations in models like DeepSeek-V4.1-Flash.
Recurrent Looped Transformer (RLT) is a novel architecture combining a causal encoder with a recurrent decoder to achieve latent reasoning with unbounded temporal depth, model-hardware co-design, and model-RL algorithm co-design.
The paper introduces declarative attention, a technique where LLMs explicitly declare which context segments to attend to, reducing token usage by up to 52% with minimal accuracy drops.
The article speculates that the next major breakthrough after the attention mechanism may involve AI architectures with input-dependent weights, potentially building on DeepSeek's Engram mechanism.
Grouped Value Attention reduces transformer KV cache size by storing grouped values and reconstructing keys via a learned linear map, achieving near-GQA accuracy with a smaller persistent cache.
Random Attention presents a signal-free KV-cache eviction policy for reasoning models that matches or exceeds the performance of learned methods on benchmarks like MATH-500 and LiveCodeBench, while being faster in inference.
RenderFormer-V2 is a transformer-based neural rendering model that handles diverse light-transport effects using a two-stage sequence-to-sequence architecture with improved attention mechanisms and support for heterogeneous scene primitives.
This paper introduces Declarative Attention, a protocol that allows language models to declare where to attend in their chain-of-thought, reducing attended tokens by 52% on Gemma-4-31B and 31.1% on Qwen-3.6-27B with minimal accuracy drops.
This article provides a detailed explanation of the Transformer architecture, covering attention mechanisms, QKV, and residual connections, while tracing its historical development from N-gram to LSTM.