Modular TTT: Rethinking Test-Time Training as Composable Modules

Hugging Face Daily Papers Papers
test-time-training modular-framework sequence-modeling online-learning fast-weights research

Summary

This paper introduces Modular TTT, a framework that represents test-time training inner learners as directed acyclic graphs, enabling systematic ablation and composition of components. The authors train 410M and 1.45B parameter models on 100B tokens, achieving performance comparable to GatedDeltaNet.

Test-time training (TTT) views sequence modeling as an online learning problem in which fast weights are updated by an internal learning rule. Despite the growing number of TTT variants, existing approaches typically hard-code each variant separately, which makes it difficult to design new TTT methods and to isolate the role of each component. To address this, we propose Modular TTT, a framework that represents the inner learner as a directed acyclic graph and exposes the fast-weight network, loss function, learning rate, weight decay, and normalization as explicit design dimensions. Modular TTT automatically composes primitive-level train-view forward, train-view backward, and causal query-view rules into the full graph-level TTT computation, including the fast-weight state transition. Using Modular TTT, we systematically ablate the components of TTT and find that small learning-rate initialization, weight decay, and a single-layer nonlinearity improve performance, while MSE and inner-product losses perform similarly. Deeper fast-weight networks and normalization tend to hurt performance because they induce excessively large activations, while residual connections and gating provide little measurable benefit. Guided by these findings, we train the best resulting variant as 410M- and 1.45B-parameter models on 100B tokens, and observe training loss and benchmark performance comparable to Gated DeltaNet.
Original Article
View Cached Full Text

Cached at: 08/10/26, 06:14 AM

Paper page - Modular TTT: Rethinking Test-Time Training as Composable Modules

Source: https://huggingface.co/papers/2608.07110

Abstract

Test-timetraining(TTT)viewssequencemodelingasanonlinelearningprobleminwhichfastweightsareupdatedbyaninternallearningrule.DespitethegrowingnumberofTTTvariants,existingapproachestypicallyhard-codeeachvariantseparately,whichmakesitdifficulttodesignnewTTTmethodsandtoisolatetheroleofeachcomponent.Toaddressthis,weproposeModularTTT,aframeworkthatrepresentstheinnerlearnerasadirectedacyclicgraphandexposesthefast-weightnetwork,lossfunction,learningrate,weightdecay,andnormalizationasexplicitdesigndimensions.ModularTTTautomaticallycomposesprimitive-leveltrain-viewforward,train-viewbackward,andcausalquery-viewrulesintothefullgraph-levelTTTcomputation,includingthefast-weightstatetransition.UsingModularTTT,wesystematicallyablatethecomponentsofTTTandfindthatsmalllearning-rateinitialization,weightdecay,andasingle-layernonlinearityimproveperformance,whileMSEandinner-productlossesperformsimilarly.Deeperfast-weightnetworksandnormalizationtendtohurtperformancebecausetheyinduceexcessivelylargeactivations,whileresidualconnectionsandgatingprovidelittlemeasurablebenefit.Guidedbythesefindings,wetrainthebestresultingvariantas410M-and1.45B-parametermodelson100Btokens,andobservetraininglossandbenchmarkperformancecomparabletoGatedDeltaNet.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2608\.07110

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.07110 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.07110 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.07110 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

Test-Time Training Undermines Safety Guardrails

arXiv cs.LG

This paper identifies three threat models for test-time training (TTT) that adversaries can exploit to bypass safety filters in LLMs, achieving high attack success rates. The findings reveal that TTT introduces new vulnerabilities that undermine existing safety guardrails.

Self-Guided Test-Time Training for Long-Context LLMs

Hugging Face Daily Papers

Proposes Self-Guided Test-Time Training (S-TTT), where the model identifies relevant evidence spans in long contexts for adaptation, achieving up to 15% relative improvement on long-context reasoning benchmarks.

Linearizing Vision Transformer with Test-Time Training

Hugging Face Daily Papers

This paper proposes a method to convert pretrained Softmax attention models into linear-complexity Test-Time Training (TTT) architectures, achieving comparable text-to-image quality to fine-tuned Softmax models while significantly accelerating inference. The approach is validated by linearizing Stable Diffusion 3.5, resulting in SD3.5-T^5 with 1.32x speedup at 1K resolution.

U-TTT: Towards Generalizable PET Image Denoising via Test-Time Training

Hugging Face Daily Papers

This paper introduces U-TTT, a U-shaped deep learning model with test-time training layers and dual-domain adaptation for robust PET image denoising under distribution shifts, achieving state-of-the-art performance across different dose levels and scanner types.