Tag
This paper introduces Graded Large Language Models (GLLMs), an algebraic framework that imposes a hierarchical grading on transformer representations, theoretically improving sample efficiency for language hierarchies while preserving inference complexity. It provides geometric and information-theoretic justifications, and outlines a grade-selection procedure validated in a companion manuscript.
The Prism Transformer replaces uniform multi-head attention with a progressive head schedule that increases head count across layers, enabling a local-to-global hierarchy without extra parameters or FLOPs. It consistently outperforms standard Transformers on language modeling and zero-shot benchmarks at 124M, 354M, and 757M scales.
Proposes a hierarchical attention mechanism using overlapping Schwarz domain decomposition to replace dense global low-rank attention with a two-level additive structure of local and coarse blocks, showing faster training and better accuracy with fewer parameters.
NousResearch releases Lighthouse Attention, a selection-based hierarchical attention that achieves 1.4-1.7x wall-clock speedup at 98K context and ~17x faster forward/backward pass than standard attention at 512K context on a single B200, validated on 530M-parameter Llama-3 models across 50B tokens.
Lighthouse Attention is a training-only hierarchical selection-based attention algorithm that reduces computational complexity for long sequence training of causal transformers, enabling faster pre-training with competitive final loss after a recovery phase.