Tag
The paper proposes Parallel Pretraining (PPT), which concatenates documents with their translations in the same training sequence, and finds that a two-stage PPT-then-standard-pretraining approach consistently improves general language understanding and cross-lingual performance over standard pretraining at matched token budgets.
该论文提出因果潜在工作区 Transformer（causal latent workspace Transformer），通过在共享深度的循环块之间加入可读写的前缀专属潜在槽位，提升了循环 Transformer 的语言建模困惑度，并在三个字节级语料上的多数设置中优于五个基线对照。
该论文将离散扩散的前向过程重新设计为可调的"信任轴"，通过区域非均匀的腐蚀核使模型能够检测并纠正被污染的条件上下文，在170M参数掩码扩散模型上以低于5小时的微调取得了显著的上下文腐蚀检测与infilling提升。
The author shares slides from their CoLM 2026 talk on "Context Beyond the Window," posting a few slides that look directionally promising since a recording may not exist.
提出 First Passage Attention（FPA），用路径极值替代单调衰减累积，从而在压缩 KV 缓存的同时实现完美的长程检索；在 110M 语言模型上以约 25% 缓存达到 8 倍训练长度的 passkey 检索。
提出了 Continuum Attention，一种用连续分段线性记忆曲线上的积分替代离散键值查找的因果 Transformer 注意力算子，在语言建模困惑度上优于标准 softmax 与 focal attention。
该论文提出让谐振序列模型（ResonatorLM）的读出相位依赖当前 token，在保持 FFT 卷积训练路径的同时引入输入依赖性，使在 256 token 训练、1024 token 测试时困惑度从 12.81 降至 3.76，显著改善长度泛化。
该论文研究无分词器的字节级语言模型，发现配合 token-superposition 训练与哈希嵌入后，扁平 Transformer 随规模扩大能持续超越子词模型，并能自发学习出类似分词器的局部文本抽象，据此可将投机解码的接受率提升 3.4 倍。
OpenESM 提出能量引导模型（ESM），将能量引导的潜状态优化与 token 生成统一到单一架构中，在大规模语料上进行能量引导预训练，表现出良好的扩展性并在多数下游任务上超越 GPT，同时完全开源模型权重与训练基础设施。
This paper extends prior-fitted networks (TabPFN) to natural language by training a 300M-parameter byte-level transformer entirely on synthetic sequences drawn from randomly sampled recurrent causal models — a 'prior over languages'. With frozen weights, the model learns to predict real languages in context across six languages, and can also count, compare numbers, add approximately, and predict deterministic sequences like primes and Kolakoski, showing in-context language learning can emerge from a synthetic non-linguistic prior.
The paper introduces Coffee, a plug-and-play framework that guides discrete diffusion models at inference time using compiled finite-state objectives, avoiding exponential enumeration of token completions while supporting hard constraints and learned soft objectives across symbolic, language, and biological benchmarks.
The paper introduces triadic linear attention, which generalizes linear attention by writing a triadic outer product of two keys and a value into a third-order tensor state, enabling an E-fold state-size increase with only two extra projections. Applied to Gated DeltaNet and scalar-gated linear attention, it improves long-context language modeling and recall, outperforming alternative state-enlargement methods.
FRAC是基于分数阶动力学的新型选择性状态空间模型架构，用幂律长记忆替代传统SSM的指数遗忘，通过log间距指数模式和近似实现高效循环模块，并在1.3B参数语言模型上显著提升长上下文性能。
Stanford University's CS336 course 'Building Language Models from Scratch' has released 18 complete lecture videos, open-source assignments, and course materials, comprehensively covering the entire workflow of large language model engineering, suitable for self-learners to systematically study LLM development.
This paper introduces Complex KDA, an enhancement to Kimi Delta Attention that improves the expressivity of linear RNNs by enabling 2D rotations and state-tracking capabilities, achieving superior length extrapolation and language modeling performance.
IntBMoE is a novel Mixture-of-Experts method that decouples participation, execution, and materialization costs through block-level conditioning and sparse execution, demonstrating improvements in image classification, language modeling, and recommendation systems, with real-world deployment in AMap's generative recommendation system.
This paper introduces Complex KDA, an enhanced version of Kimi Delta Attention that combines a delta-rule transformation with a reflection to achieve greater expressivity, outperforming Transformers in some tasks while maintaining efficiency, with open-source code and models available.
This paper introduces a method using rotating sparse wiring on a hypercube to replace attention in neural networks, achieving comparable or better performance with significantly reduced computational cost and parameters.
DantinoX is an open-source JAX/Flax library that provides a unified framework for training and comparing autoregressive, masked diffusion, and flow-matching language models, enabling controlled experiments and benchmarking.
This tweet discusses the challenge of using probabilistic methods in AI writing models to replicate individual language decisions, potentially advancing AI-assisted writing.