Tag
This arXiv paper introduces TEMPER, a tensorized parameterization for hyper-connection residual routing that reduces parameter growth while preserving expressive, manifold-constrained routing. Experiments show it matches or outperforms existing methods with substantially fewer additional parameters.
A new transformer variant, the full-bandwidth transformer, feeds back top-layer hidden states through a gated linear unit to improve reasoning and efficiency without altering the core architecture. Trained up to 400B tokens, it matches standard transformers trained with 1.5x more data while producing shorter reasoning traces.
This paper from the Kathleen series shows that an attention-free, byte-level model with ~0.5M parameters can beat a parameter-matched transformer on WikiText-103 language modeling and generation, introduces a non-parametric 'Form Distance' metric for evaluating text realism, and demonstrates that retrieval-augmented decoding from the model's own training corpus improves generation quality.
This paper investigates whether Arabic character form-function relationships are arbitrary from an NLP perspective, showing that random character remappings to reduced rasm sets can achieve competitive performance across various NLP tasks.
PreDiff-LM proposes a hybrid attention mechanism that preserves causal attention for prompt tokens and bidirectional attention for masked target tokens, enabling adaptation of pretrained autoregressive models for discrete masked diffusion language modeling, achieving improvements in perplexity and downstream tasks over prior diffusion baselines.
This paper analyzes why deterministic JEPA-style latent prediction works for images but not for text, attributing the failure to high conditional variance in language where masked contexts admit multiple valid completions whose representations lack a coherent center.
This paper proposes a parameter-free adaptive sparse attention method that uses gzip compression ratios to dynamically select non-redundant blocks for long-range attention, achieving significant perplexity improvements over fixed and learned sparse attention baselines on PG-19 language modeling.
A personal project where a 0.5M parameter language model was trained on 1 billion tokens from the Fineweb-edu dataset.
Gigatoken is a drop-in replacement tokenizer claiming up to 1000x speedup over HuggingFace's tokenizers, supporting many common tokenizers and CPUs.
This paper analyzes off-policy distillation for LLM pre-training, characterizing how training objectives shape token-level supervision and downstream capabilities, and proposes adaptive objective routing that applies different objectives to different data domains, reframing pre-training as a data-conditional supervision design problem.
This paper proposes Multi-level Context Fusion MOE (MCF-MOE), a framework that improves routing consistency in Mixture-of-Experts models by integrating cross-layer semantic aggregation and local token-level interactions, outperforming strong baselines on language modeling and understanding benchmarks.
Researchers propose Latent Thought Flows, compressing 256 text tokens into 8 continuous latents for one-step generation, achieving a better Pareto frontier than autoregressive baselines.
A tweet distinguishes two AI career paths—API Caller vs. Architect—and recommends Stanford's free CS336 course for those wanting to become architects.
A discussion thread for the upcoming COLM 2026 decision notifications.
ResonatorLM introduces a physics-derived causal resonant field mixing mechanism to replace self-attention in transformers, achieving significant speedups and improved accuracy on long-context language modeling tasks.
The paper proposes the state-prediction separation hypothesis, suggesting that separating state prediction from token prediction in Transformers improves language modeling performance and efficiency across scales, with experiments showing consistent gains.
Flexformer proposes a flexible linear Transformer with fully learnable attention kernels using random Fourier features, achieving linear complexity while matching or exceeding softmax attention performance on language modeling and sequence classification tasks.
Stanford's CS336 course on language modeling from scratch is announced, featuring intensive hands-on assignments covering tokenizers, transformers, data, and alignment.
Proposes Erase-then-Delta Attention (EDA), a memory update rule for linear attention that decouples erase and write addresses to selectively suppress stale information before writing new content. Experiments on 2.5B dense and 25B MoE models demonstrate consistent gains in standard and long-context evaluations.
Stanford CS336 aims to teach students how to build language models from scratch, with deep understanding of the full-stack design of data, systems, and models. The course videos are publicly available and suitable for AI beginners.