language-modeling

This tweet recommends Stanford's CS336 course and a series of learning resources as a preparation path for joining OpenAI.

Alisa Liu announces she is joining OpenAI and shares a blog post about her job search experience, including insights from Stanford's Language Modeling from Scratch course.

This article explores using the gzip compression algorithm as a language model, demonstrating that compression algorithms can generate text by scoring candidate continuations based on compressed length, using beam search to produce output.

The tweet describes how any compression tool, including gzip, can be adapted for language modeling, and that gzip can generate text that resembles Shakespeare. A write-up is linked.

Proposes ac-gpt, a simple modification to causal Transformers that enables evaluating and sampling from arbitrary conditionals (past, future, mixed) in a single forward pass while preserving left-to-right ordering and next-token prediction, allowing existing LLMs to be fine-tuned for arbitrary conditioning.

This paper theoretically analyzes diffusion language models through a bias-variance lens, identifying trade-offs between masking and uniform diffusion kernels. It proposes SemDLM+, which adds a global transition and semantic-frequency penalty to overcome the semantic basin problem, achieving competitive generation quality on LM1B and OpenWebText benchmarks.

Introduces Kuramoto attention, a self-attention layer where hidden states are phase angles on a torus, enabling synchronization through gated cosine similarity and circular mean updates. The layer performs comparably to standard transformers on character-level language modeling.

This paper investigates whether Transformers need separate key and value projections, finding that sharing them can reduce KV cache by 50% with only 3.1% higher perplexity, and further cuts when combined with GQA and MQA.

Stanford is offering a comprehensive course, CS336, where students build a language model from scratch, covering data collection, transformer construction, training, and evaluation.

The Cognitive Categorical Transformer (CCT) augments GPT-2 Small with category-theoretic components, achieving a 12% relative perplexity reduction on WikiText-103 under matched training conditions, with simplicial message passing responsible for 84% of the improvement.

Introduces Parallax, a parameterized local linear attention mechanism with hardware-aware optimization that improves LLM pretraining efficiency and performance, achieving Pareto improvements at 0.6B and 1.7B scales.

This paper introduces on-policy distillation, which trains a student model on its own trajectories with teacher token-level KL supervision to fix train-inference mismatch, unifying forward-KL, reverse-KL, and JSD losses, with reverse-KL favored for smaller students.

NVIDIA introduces Gated DeltaNet-2, a method for editing compressed model memory without catastrophic forgetting, using independent gates for erase and write operations. It outperforms existing models like Mamba-2 and Mamba-3 on language modeling and long-context tasks.

HRM-Text introduces a Hierarchical Recurrent Model that decouples computation into slow and fast layers, enabling efficient pretraining from scratch on only 40 billion tokens and a $1,500 budget, achieving competitive performance with larger models.

FlowLM introduces a flow matching language model derived from pre-trained diffusion models via efficient fine-tuning, enabling high-quality few-step text generation that rivals 2,000-step diffusion sampling with far fewer training epochs.

Gated DeltaNet-2 introduces separate erase and write gates for linear attention, achieving superior performance in long-context language modeling and retrieval tasks.

This paper introduces S-FLM, a novel flow-based language model that operates in a hyperspherical latent space to address the computational costs and semantic limitations of existing discrete diffusion and continuous flow models.

This paper introduces Attractor Models, which use fixed-point solving and implicit differentiation for efficient iterative refinement, achieving superior language modeling and reasoning performance with reduced computational costs compared to traditional transformers.

This paper introduces a diffusion language model that treats text as a continuous process over binary bitstreams, using entropy-gated stochastic sampling to close the performance gap with autoregressive models. It achieves state-of-the-art results on LM1B and OWT benchmarks while reducing memory footprint.

ELF proposes a continuous diffusion model for language that uses embedding space and flow matching, outperforming existing discrete and continuous diffusion language models with fewer sampling steps.