Tag
FreeSolo introduces a native sparse attention mechanism combining short-range sliding window and long-range keypoint-based branches, achieving linear-time complexity and improved zero-shot length extrapolation to 256k+ tokens without costly long-context fine-tuning.
The paper introduces NoteBank, a training-free sparse episodic memory that augments linear attention models by archiving selected text and replaying it through native computation, boosting Gated DeltaNet's long-context retrieval accuracy on RULER from 1.7% to 70.3% at one million tokens with negligible FLOPs overhead.
作者介绍 ALHR（Adaptive Learnable Hierarchical Routing），一种基于静态二叉树与可学习函数的稀疏注意力系统，在 MQAR 测试中每查询仅读取约 30 个键（相比稠密注意力的 512 个），实现约 35.3 倍 KV 压缩，同时保持接近的准确率，并将推理复杂度降至 NlogN、显存线性扩展。
Edge0 开源一个月后展示了 35B 模型在 iPhone 飞行模式下以 1.8GB 峰值内存运行的能力，无需云端或服务器，已扩展到耳机、智能眼镜、机器人等设备并拥有超过 8 万用户。
介绍 Jag，一个 0.8B 视觉定位模型，通过单次前向传播直接预测边界框而非自回归生成坐标，在保持与 LocateAnything 相当精度的同时，速度提升 3.62 倍、峰值 GPU 显存降低 73.88%，并开源了模型权重、代码与在线演示。
本文提出SPIN（Shadow Predictive Indexer），一种免训练的方法，利用历史解码迭代中的indexer分数预测重要KV块，从而跳过部分上下文评分，实现30-40%稀疏度，在vLLM端到端服务中吞吐量提升最高14.9%、中位ITL降低13.2%。
The paper proposes KITE, a training-free framework that shifts latent multi-agent KV communication from sender-side state fidelity to receiver-side task sufficiency, transmitting only a task-effective key layer. Experiments show 28-36× communication reduction, up to 3× inference speedup, and accuracy gains of up to 23.3 percentage points across seven benchmarks.
LatentGRM 提出了一种基于语义分块与压缩重建的潜在评估框架，通过在连续隐空间中完成 rubric 引导的成对偏好判断，将评估轨迹压缩 8.9–9.2 倍并把推理时间减少 6.1–7.0 倍，同时保持与显式 SFT 评委相当的偏好准确率。
REMORY introduces a neural memory network that supplements textual summaries with a bounded sequence of soft memory tokens for context compaction in long-horizon agents, enabling a frozen LLM to approach full-context performance using only 5.2% of input positions with consistent gains on Qwen3.8-27B and GLM-5.3-Flash.
LiFT introduces a family of looped generative models that scale computation by repeatedly applying a shared Diffusion Transformer (DiT) core, training each loop step on a continuous-depth flow-matching regression target that allows inference far beyond training depth. On ImageNet 256x256, LiFT-L/2 beats a dense DiT-XL/2 baseline by 3.34 FID points while using ~60% fewer parameters and 52% fewer inference FLOPs.
MoRA proposes a structured MoE expert pruning framework using learnable router biases optimized with LM loss and a routing-diversity regularizer, plus a post-pruning expert approximation mechanism, outperforming state-of-the-art pruning methods on Qwen3-30B-A3B, DeepSeek-V2-Lite, and Moonlight-16B-A3B with 25-50% expert removal.
The post highlights Andrej Karpathy's argument that small 3B models can replace frontier-model calls, and introduces TwIL-LM3-Pro, a 3.6B-parameter open-source reasoning model from webAI that matches Qwen3-8B on formal logic tasks while running locally on CPU or 4GB VRAM.
Peking University researchers introduce NAMOH, an architecture-native sparse attention mechanism that activates K of H heads per token, so head routing jointly determines active parameters and available context. It enables parameter scaling to directly support efficient long-context scaling, outperforming fully activated models with the same total parameters while remaining compatible with GQA and existing sparse attention methods.
提出了一种多语言命名实体标记器用于隐私脱敏，通过提示前沿LLM提供监督并结合人工金标准数据微调XLM-R，在35种语言上实现88.8的红action F1，同时在CPU吞吐上比GLiNER2快4.9倍，并发布代码、提示与训练配方。
Sieve and Sage 是一个两阶段 RALM 拒答框架，用轻量级 Sieve 模块在调用昂贵 LLM (Sage) 之前过滤干扰性检索证据，并区分不可答与被干扰两种状态，相比单阶段基线最高提升 69.4%p 准确率、55.2%p Macro-F1，并实现 1.99 倍加速。
HyperZip proposes an efficient LLM-based lossless data compression framework that uses diffusion-based LLMs with Multi-Token Prediction and a hypernetwork to adapt models to target data without fine-tuning, achieving a superior compression-rate/throughput trade-off over autoregressive baselines.
SMat-Attention introduces structured causal masks with tunable VC-dimension to bridge softmax attention and linear attention, achieving subquadratic prefill, constant-time decoding with O(T^{1-1/d}) cached states, and improved recall over Mamba-2 and Gated DeltaNet backbones.
This survey analyzes the evolution of attention mechanisms in large language models through a five-dimensional lens of contextual memory (representation, update, access, readout, integration), drawing on 59 release-level records across 14 model lineages, and proposes a stateful multidimensional memory-routing hypothesis for future sequence architecture design.
FocusVTC 通过自适应分辨率打破了视觉文本压缩中的压缩率-性能权衡，用 REL-SFT 和 GRPO 训练模型在低 DPI 全局视图基础上选择性增强关键区域，在 RULER、LongBench、MRCR 等长上下文基准上大幅超越 Glyph 等基线，并实现 2.79 倍在线端到端加速。
Draft-KV is a method for latent communication between language models that sends key-value states instead of decoded text, improving accuracy on benchmarks like MMLU with minimal parameters while keeping models frozen.