Tag
A YouTube creator returns after a hiatus to teach building a distributed training framework from first principles, focusing on advanced AI topics like DeepSeek, MoE, and MLA, with an emphasis on developing problem-solving skills and self-confidence.
RecurrentGPT introduces a recurrent depth transformer that uses gated modulation to iteratively reuse shared layers, achieving competitive accuracy with fewer parameters and improved memory efficiency compared to standard transformers.
This paper presents Wiola, a 13M parameter decoder-only language model with novel components like Spiral Rotary Positional Encoding and Gated Spiral Attention to enhance parameter efficiency for small-scale, on-device language models.
The paper introduces full-bandwidth transformers, which use latent feedback to enhance autoregressive models by allowing non-verbalized computation to re-enter the stack, improving performance with negligible decoding overhead.
FLARE++ is a low-rank attention architecture that replaces static learned queries with input-conditioned dynamic routing, improving on FLARE across PDE surrogate benchmarks and Long Range Arena while preserving linear complexity.
This paper shows that four seemingly minor architectural choices—normalization, GQA, pretraining context length, and sliding window attention—have a compoundingly negative effect on long context extensibility, dropping performance by up to 47% when combined. The authors release OlmPool, a set of 26 comparable 7B models, after 170,000 GPU hours of controlled ablations.
TenStrip/10Eros-Max is an experimental AI model that modifies the MiniMax H3 base model by transferring learned patterns from LTX 2.3, Wan 2.2, and Krea 2 video and image diffusion models using orthogonal projection, enhancing aesthetic and motion character while preserving core video and audio capabilities.
Looped transformers reuse the same layers across multiple passes to trade parameter count for compute, achieving better reasoning with fewer weights. The article traces the idea back to the Universal Transformer (2018) and explains why it initially failed due to scaling laws and timing.
Introduces Multi-Head Attention Residuals (MHAR), which reshapes the routing query into per-subspace heads so each feature subspace reads depth history via its own softmax. Trained from scratch on a Nemotron-based corpus, MHAR consistently improves validation loss over standard Transformers from 100M to 1B scale and also boosts mid-training downstream accuracy.
This paper introduces Bifocal Attention, which decouples positional encoding into geometric (standard RoPE) and spectral (learnable harmonic operators) modalities to address the 'Spectral Rigidity' of fixed RoPE, improving algorithmic generalization beyond the training window.
Presents a continuous geometric framework modeling Transformer operations as integro-differential equations on a semantic fiber bundle, validated across multiple architectures.
DeepLoop introduces a residual scaling method for looped Transformers that adjusts for parameter visits, improving stability and performance when physical blocks are reused across multiple rounds.
This course note summarizes the architectural evolution from the original Transformer to modern LLMs, focusing on convergent developments such as pre-normalization, RMS normalization, and RoPE, and provides hyperparameter selection recommendations.
GigaWorld-Policy-0.5 is an enhanced World Action Model for robot control that improves training and inference efficiency through a Mixed Action-Conditioned World Modeling strategy and a Mixture-of-Transformers architecture, achieving 85ms latency on a local RTX 4090.
OpenMythos is an open-source project that theoretically reproduces the Claude Mythos architecture, fully implementing the Recurrent-Depth Transformer (RDT), supporting MLA/GQA attention mechanisms and sparse MoE, providing preset configurations from 1B to 1T parameters, and is installable via pip.
This chapter reviews current understanding of Large Language Models, discussing their Transformer architecture, emergent capabilities resembling human cognition, and debates about whether LLMs genuinely understand or merely simulate understanding.
The article argues that open-weight AI models are catching up to closed ones not via distillation but due to the modularisation of the AI stack—stable interfaces (Transformer architecture, OpenAI-compatible APIs, agentic harnesses) allow innovations to diffuse rapidly across the ecosystem, shrinking the capability gap while keeping a massive price advantage, potentially leading to a commoditisation of frontier AI.
A comprehensive 15-part series covering LLM internals from tokenization to serving, grounded in Gemma 4 12B's actual config.
OneRank proposes a Transformer-native multi-task ranking framework that integrates feature encoding and prediction to reduce inter-task interference and improve ranking performance in recommender systems.
This article provides a visual guide to the Transformer architecture in Large Language Models, covering self-attention, causal self-attention, masked multi-head attention, and the output layer with step-by-step explanations and examples.