@XAMTO_AI: Major Open Source! OpenMythos —— A theoretical reproduction project of the Claude Mythos architecture! Built by KyeGomezB from first principles, fully implementing the Recurrent-Depth Transformer (RDT): • Prelude (prelude layer)…

Major Open Source Release! OpenMythos – A Theoretical Reconstruction of the Claude Mythos Architecture! Built from first principles by KyeGomezB, fully implements the Recurrent-Depth Transformer (RDT): • Prelude layer → Recurrent Block (can loop multiple times) → Coda layer • Each loop injects the original input signal (Input Injection) to maintain stability and prevent drift • Supports switching between MLA (Multi-Latent Attention) or GQA • Sparse MoE (routed experts + shared experts), extremely low activation rate but massive parameter count • Increasing loop count during inference = implicit deeper reasoning (silent CoT in latent space) Pre-configured parameter presets from 1B to 1T are provided (mythos_1b to mythos_1t), along with training scripts and full documentation. pip install open-mythos to start experimenting! GitHub (14.6k stars): https://github.com/kyegomez/OpenMythos … This might be one of the closest open-source implementations to the “next-generation reasoning architecture”!

Training

The training script for the 3B model on FineWeb-Edu is at training/3b_fine_web_edu.py. Single GPU: bash python training/3b_fine_web_edu.py Multi-GPU (auto-detects GPU count): bash torchrun --nproc_per_node=$(python -c "import torch; print(torch.cuda.device_count())") training/3b_fine_web_edu.py Key design choices:

The Central Hypothesis

Claude Mythos is suspected to be a Recurrent-Depth Transformer (RDT) — also called a Looped Transformer (LT). Rather than stacking hundreds of unique layers, a subset of layers is recycled and run through multiple times per forward pass. Same weights. More loops. Deeper thinking. This is not chain-of-thought. There is no intermediate token output. All of this reasoning happens silently, inside a single forward pass, in continuous latent space.

Architecture

A looped transformer divides its layers into three functional blocks: Input ↓ [Prelude P] — standard transformer layers, run once ↓ [Recurrent Block R] — looped T times ↑_______↓ (hidden state h updated each loop with input injection e) ↓ [Coda C] — standard transformer layers, run once ↓ Output The recurrent block update rule at each loop step t: h_{t+1} = A·h_t + B·e + Transformer(h_t, e) Where:

• h_t is the hidden state after loop t
• e is the encoded input (from the Prelude), injected at every loop
• A and B are learned injection parameters
• The Transformer blocks apply attention and MLP as usual The injection of e at every step is what prevents the model from drifting — it keeps the original input signal alive throughout the entire recurrence depth. The full implementation is in open_mythos/main.py. See the OpenMythos class reference for a detailed API walkthrough, configuration options, and usage examples.

Attention Implementations

The attention layer is switchable via cfg.attn_type:

The Stability Problem (and How It Was Likely Solved)

Training looped models is notoriously unstable. Two failure modes dominate:

• Residual explosion — the hidden state h_t grows unboundedly across loops
• Loss spikes — training diverges suddenly due to large spectral norms in injection parameters

The Dynamical Systems View

Recast looping as a discrete linear time-invariant (LTI) dynamical system over the residual stream. Ignoring the nonlinear Transformer contribution, the recurrence becomes: h_{t+1} = A·h_t + B·e For this LTI system, stability is governed entirely by the spectral radius of A:

• ρ(A) < 1 → stable, convergent
• ρ(A) ≥ 1 → unstable, divergent Empirically, every divergent training run learns ρ(A) ≥ 1. Every convergent run maintains ρ(A) < 1.

The Fix

Constrain the injection parameters so that stability is guaranteed by construction:

1. Parameterize A as a continuous negative diagonal matrix
2. Discretize using ZOH/Euler schemes: A_discrete = exp(Δt · A_continuous)
3. Enforce negativity via A := Diag(-exp(log_A)) with a learned scalar Δt
4. This ensures ρ(A) < 1 always holds, regardless of learning rate or batch noise The result: the looped model becomes significantly more robust to hyperparameter selection and trains cleanly even at high learning rates. This is the Parcae architecture (Prairie et al., 2026), and it represents the most likely class of solution Anthropic used to make Mythos trainable.

The Loop Index Embedding Hypothesis

A key open question is whether the looped block behaves identically on every iteration, or whether it can learn to do different things at different loop depths. Without any positional signal across loops, the same weights must handle both early-stage pattern matching and late-stage refinement — a tight constraint. A RoPE-like embedding of the loop index injected alongside the input at each step would allow the same parameters to implement functionally distinct operations across iterations, much like how RoPE allows the same attention heads to behave differently at different sequence positions. If Mythos uses this technique, each loop is not a repetition — it is a distinct computational phase, all sharing weights but operating in different representational regimes. This would substantially increase the expressiveness of the recurrent block without increasing parameter count.

The Overthinking Problem

More loops is not always better. Beyond a certain depth, excessive recurrence degrades predictions — the hidden state drifts past the solution and into noise. This is the “overthinking” failure mode. The original Universal Transformer (Dehghani et al., 2018) addressed this with an Adaptive Computation Time (ACT) halting mechanism: a learned scalar per position that dynamically decides when to stop looping. Positions that are harder to process receive more computation; simple tokens halt early. Mythos almost certainly has some version of this. The model cannot naively run the maximum number of loops on every input — it needs a learned signal for when the answer has converged. The ACT mechanism also makes the model Turing-complete under certain assumptions, which has theoretical implications for the class of problems it can solve.

Mixture of Experts — Suspected for Large Parameter Counts

The looped transformer explains the depth of Mythos’s reasoning, but not the breadth. Handling wildly different domains — code, math, literature, science, law — with the same weights requires Mixture of Experts (MoE). The suspected design replaces every FFN in the Recurrent Block with a fine-grained MoE layer: each FFN is split into many small experts (1/m the normal size), a router selects the top-mK of them per token via learned affinity scores, and a small number of shared experts are always activated regardless of routing to absorb common cross-domain knowledge — syntax, basic reasoning, general context — that would otherwise be redundantly learned by every routed expert. Routing collapse is prevented through a bias term on the router logits adjusted dynamically during training, keeping load balanced across experts without distorting the loss signal. As the hidden state h_t evolves across loop iterations, the router may select different expert subsets at each depth, making every loop computationally distinct despite shared weights. MoE provides breadth; looping provides depth. If the activation ratio is ~5%, Mythos could hold hundreds of billions of total parameters while activating only a small fraction per token — the true parameter count, if ever disclosed, would be a storage number, not a compute number.

The Memorization-Reasoning Tradeoff

Looped models exhibit an interesting dichotomy: looping improves reasoning but can hurt memorization. The recurrent structure is optimized for iterative composition — running a reasoning chain forward — but does not inherently improve the storage of rote facts. This maps to an observable characteristic of Mythos: it reasons exceptionally well about novel problems it has never seen, but its factual recall can be inconsistent. The architecture is structurally biased toward composition over memorization. Looping-based regularization (Saunshi et al., 2025) can be used to balance this tradeoff during training — applying stronger looping constraints for reasoning tasks while relaxing them for retrieval tasks.

Parameter Reuse via LoRA Adaptation

A complementary approach from Relaxed Recursive Transformers (Bae et al., 2024): rather than requiring fully identical weights at every loop, add a small depth-wise LoRA module at each iteration.