Tripstoph/RRT-Foundation · Hugging Face

Source: https://huggingface.co/Tripstoph/RRT-Foundation

https://huggingface.co/Tripstoph/RRT-Foundation#rrt-355m–softmax-free-attention-at-gpt-2-medium-scaleRRT-355M — softmax-free attention at GPT-2 Medium scale

Headline result:a GPT-2 Medium–shaped checkpoint (~354 Mparameters) trainedfrom scratch without softmax, evaluated on astandardized 22-task in-context learning benchmark, withopen kernelswhere sparse inference isbit-identical to denseon this checkpoint.

This Hugging Face repo shipsweights, config, and substrate constantsonly.Inference requires the RRT engineon GitHub (RRT-LLM-FOUNDATION, AGPL-3.0). StocktransformersGPT-2 will produce incorrect outputs.

Training iscomplete. No additional checkpoints are planned from this repository.

https://huggingface.co/Tripstoph/RRT-Foundation#capability-evaluation-22-task-coreCapability evaluation (22-task CORE)

ModelCORENotesGPT-2 124M0.1211floor reference, same harnessGPT-2 medium0.1770dense softmax foil, matched scaleRRT-355M****0.1558softmax-free, this checkpointPythia 410M0.1895modern baseline, same harness CORE= mean centered accuracy across 22 in-context learning tasks (DCLM protocol, Karpathy nanochateval\_bundle). RRT-355M is0.021 belowthe GPT-2 medium foil and0.035 abovethe GPT-2 124M floor — a measurable tradeoff, not a capability collapse.

**Task asymmetry (RRT − GPT-2 medium, centered score):**gains on multiple-choice reasoning (arc\_easy+0.12,agi\_eval\_lsat\_ar+0.09,openbook\_qa+0.07); largest regressions on continuation tasks (lambada\_openai−0.16,coqa−0.13,squad−0.07).

**Not evaluated:**MMLU, GSM8K, HumanEval, chat/instruction benchmarks, or fine-tuned downstream tasks. Details:eval/eval\_summary\.jsonon this repo; full write-up onGitHubdocs/EVALUATION\.md.

https://huggingface.co/Tripstoph/RRT-Foundation#mechanism-and-trainingMechanism and training

MetricValueNotesStructural edge sparsity99.66 %fidelity gate; training measurementTraining dataFineWeb-Edu11.534 B tokens, 4× H100, 22k itersBest val loss (ckpt)2.8001iteration 21 000Weight file~1011 MB bf16model\.safetensors Three metrics — do not conflate:(1) structural sparsity during training, (2) coarse-tile skip at inference (34–55%, long context), (3) CORE behavioral score above.

Each attention edge applies frictionln\(max\(i−j, 1\)\)and gateμ = η / \(1 \+ η^n\)^\(1/n\)withn = 1\.25. INT8 pre-pass skips tiles with no active edges;bit-identicalto dense on this checkpoint. v2 kernel:21/22 CORE tasks identicalto v1 (Δ CORE −0.0016).

https://huggingface.co/Tripstoph/RRT-Foundation#systems-notes-secondarySystems notes (secondary)

MetricValueCaveatINT8 tile skip @ T=2048 / 819234% / 55%layer-12 micro-bench, H100Kernel vs SDPA @ T=204811.5×not end-to-end generationPeak attention VRAM @ T=163845.5 GBGPT-2 XLreference forward, RTX 3070

https://huggingface.co/Tripstoph/RRT-Foundation#files-in-this-repoFiles in this repo

FilePurposemodel\.safetensorsbf16 weightsconfig\.jsonarchitecture metadatarrt\_substrate\_constants\.jsoninference requiresn\_backbone,C\_maxonlyeval/CORE summary JSON, comparison CSV, parity notesfigures/key charts from benchmark reporttokenizer\_pointer\.txt``openai\-community/gpt2BPE

https://huggingface.co/Tripstoph/RRT-Foundation#reproduceReproduce

git clone https://github.com/tripstoph/RRT-LLM-FOUNDATION.git
cd RRT-LLM-FOUNDATION
pip install -e .
python eval/run_core_eval.py --model rrt:_state/ckpt.pt --snapshot-dir engine --seed 1337
# Quick smoke (~minutes): python eval/smoke_core.py --model rrt:_state/ckpt.pt --snapshot-dir engine

Expected full CORE:0.1558. Claims ↔ evidence:GitHubdocs/CLAIMS\.md.

https://huggingface.co/Tripstoph/RRT-Foundation#scopeScope

RRT-355M validates the attention mechanism in isolation. Broader pipeline work is explored separately underRelational Autopoietic Substrate (RAS); no timeline or additional model releases are committed from this repository.

https://huggingface.co/Tripstoph/RRT-Foundation#limitationsLimitations

• Custom Triton engine (Hopper sm_90); notAutoModelForCausalLM
• CORE below dense GPT-2 medium at matched scale
• Single checkpoint; no scale-up from this repo
• Speed/memory figures are kernel benchmarks with stated context

https://huggingface.co/Tripstoph/RRT-Foundation#citationCitation

@misc{rrt-355m-2026,
  author       = {Tripstoph},
  title        = {RRT-355M: Softmax-free attention at GPT-2 Medium scale},
  year         = {2026},
  publisher    = {HuggingFace},
  howpublished = {\url{https://huggingface.co/Tripstoph/RRT-Foundation}},
  note         = {Proof-of-mechanism weights; engine at GitHub under AGPL-3.0.},
}

Last updated: 2026-06-21