Tag
This paper introduces a diversity-aware, layer-wise scoring method for KV cache eviction in large language models, incorporating attention dispersion and redundancy to improve performance on LongBench datasets.
MM-ABC提出一个面向通用移动操作的基础模型，通过稀疏多层VLM特征实现空间感知、以世界想象与几何意图为监督的训练分支，以及MM-APT中用掩码联合注意力协调操作与移动流。模型在5000+小时、17种本体的数据上预训练，在RoboCasa365、LIBERO等基准及真实移动操作任务上取得强性能。
The paper investigates quantizing softmax in transformer attention during pretraining, comparing calibration and rounding strategies to reduce validation loss gaps. Experiments at 124M parameters show minimal loss degradation with specific techniques.
A tweet describing an anime opening-style video that summarizes nine years of AI development from the Attention paper to Opus 5.5, featuring music videos and a reference mini-site.
The paper explores CoWindow Attention (CoWA) and MassAlloc Attention (MALA) as optimized attention mechanisms for AI models, achieving substantial speedups in training and inference while reducing computational costs and maintaining performance.
This paper proposes an explainable hate speech detection framework integrating DistilBERT embeddings, BiLSTM, and an attention mechanism, achieving high F1-scores on benchmark datasets for both binary and multi-class classification.
Quartet introduces a graph transformer architecture with quad-branch cross-attention and a causal random walk sampler to enhance performance on relational graph tasks, outperforming state-of-the-art baselines like RelGT and HGT.
An analysis of 847 AI agent runs reveals that larger context windows cause performance drops due to attention cliffs, and Synap is presented as a tool to efficiently manage context and reduce token usage.
This paper introduces Attention-Aware Routing (AAR), a method that enhances Mixture-of-Experts language models by incorporating attention weights into the router, improving mathematical reasoning performance and revealing coupled dynamics between routing and attention.
This paper introduces Complex KDA, an enhanced version of Kimi Delta Attention that combines a delta-rule transformation with a reflection to achieve greater expressivity, outperforming Transformers in some tasks while maintaining efficiency, with open-source code and models available.
This tweet explains how core AI operations like attention and convolution are based on foundational mathematics such as linear algebra and probability, emphasizing the value of understanding these operations over model names.
The article introduces a visualization tool for the attention mechanism in Transformers, allowing users to customize input matrices and see step-by-step computations of attention scores. It is an initial version with plans for future updates based on user feedback.
This article explores two possible architectural hypotheses for the underlying base model of TypeSafe's Jev API: a bidirectional encoder or a modified causal decoder, and analyzes the relevant evidence and implications.
MetaRTL is a two-stage framework for relational table learning that uses meta-path attention to efficiently capture relational semantics, demonstrated effective across multiple datasets.
Meta Engineering extends FlashAttention-4 with MXFP8 support for NVIDIA Blackwell, achieving up to 2.85 PFLOP/s forward performance and integrating into production training workflows like GEM.
The article describes an optimization for the DeepSeek-V4.1-Flash AI model on Metal hardware, where the attention kernel was improved to only launch necessary tiles, reducing compute waste and enhancing inference speed.
This paper proposes an attention-based finite-volume scheme for shock transport, using a CFL-conditioned flux to enable efficient and accurate simulations with larger time steps.
Proposes CRFCAN, a complex-valued residual network for joint channel and phase noise estimation in sub-THz OFDM systems, which achieves end-to-end recovery with physics-inspired structure and outperforms conventional and state-of-the-art deep learning models.
This article explains in detail why the KV cache stores K and V but not Q in large language models, and discusses prefill, prefix cache, and architectural innovations in models like DeepSeek-V4.1-Flash.
Recurrent Looped Transformer (RLT) is a novel architecture combining a causal encoder with a recurrent decoder to achieve latent reasoning with unbounded temporal depth, model-hardware co-design, and model-RL algorithm co-design.