Self-Guided Test-Time Training for Long-Context LLMs

Hugging Face Daily Papers Papers
long-context test-time-training self-guided llm reasoning

Summary

Proposes Self-Guided Test-Time Training (S-TTT), where the model identifies relevant evidence spans in long contexts for adaptation, achieving up to 15% relative improvement on long-context reasoning benchmarks.

Long-context processing has become increasingly important for large language models (LLMs), but simply extending the context window does not guarantee effective utilization of long inputs. As input length grows, accuracy often degrades, indicating that models still struggle to identify and use the evidence most relevant to a question. A promising way to improve long-context utilization is test-time training (TTT), which treats the test context as a training example for instance-specific parameter adaptation. However, applying TTT to the entire long context is prohibitively expensive, while adapting on randomly sampled spans introduces severe noise. Because most spans in a long context are irrelevant to the specific question, training on them may even degrade the base model's performance. Our preliminary study shows that TTT is highly sensitive to training-span quality: on LongBench-v2, TTT on randomly sampled spans hurts performance, whereas TTT on oracle spans substantially improves it. Motivated by this, we propose a simple method, Self-Guided TTT (S-TTT): before adaptation, the model identifies the evidence spans it should learn from, and the standard language-modeling training objective is applied only to those selected spans. On two challenging long-context reasoning benchmarks, LongBench-v2 and LongBench-Pro, S-TTT improves accuracy for both Qwen3-4B-Thinking-2507 and Llama-3.1-8B-Instruct, achieving up to a 15% relative improvement.
Original Article
View Cached Full Text

Cached at: 07/13/26, 07:50 AM

Paper page - Self-Guided Test-Time Training for Long-Context LLMs

Source: https://huggingface.co/papers/2607.09415 Authors:

,

,

,

,

,

,

,

,

,

,

,

Abstract

Long-contextprocessinghasbecomeincreasinglyimportantforlargelanguagemodels(LLMs),butsimplyextendingthecontextwindowdoesnotguaranteeeffectiveutilizationoflonginputs.Asinputlengthgrows,accuracyoftendegrades,indicatingthatmodelsstillstruggletoidentifyandusetheevidencemostrelevanttoaquestion.Apromisingwaytoimprovelong-contextutilizationistest-timetraining(TTT),whichtreatsthetestcontextasatrainingexampleforinstance-specificparameteradaptation.However,applyingTTTtotheentirelongcontextisprohibitivelyexpensive,whileadaptingonrandomlysampledspansintroducesseverenoise.Becausemostspansinalongcontextareirrelevanttothespecificquestion,trainingonthemmayevendegradethebasemodel’sperformance.OurpreliminarystudyshowsthatTTTishighlysensitivetotraining-spanquality:onLongBench-v2,TTTonrandomlysampledspanshurtsperformance,whereasTTTonoraclespanssubstantiallyimprovesit.Motivatedbythis,weproposeasimplemethod,Self-GuidedTTT(S-TTT):beforeadaptation,themodelidentifiestheevidencespansitshouldlearnfrom,andthestandardlanguage-modelingtrainingobjectiveisappliedonlytothoseselectedspans.Ontwochallenginglong-contextreasoningbenchmarks,LongBench-v2andLongBench-Pro,S-TTTimprovesaccuracyforbothQwen3-4B-Thinking-2507andLlama-3.1-8B-Instruct,achievinguptoa15%relativeimprovement.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.09415

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.09415 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.09415 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.09415 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling

Hugging Face Daily Papers

This paper introduces AutoTTS, an environment-driven framework that automates the discovery of test-time scaling strategies for LLMs by formulating it as controller synthesis. It demonstrates improved accuracy-cost tradeoffs on mathematical reasoning benchmarks with minimal computational overhead.

Modular TTT: Rethinking Test-Time Training as Composable Modules

Hugging Face Daily Papers

This paper introduces Modular TTT, a framework that represents test-time training inner learners as directed acyclic graphs, enabling systematic ablation and composition of components. The authors train 410M and 1.45B parameter models on 100B tokens, achieving performance comparable to GatedDeltaNet.

Positional Failures in Long-Context LLMs: A Blind Spot in Reasoning Benchmarks

arXiv cs.CL

This paper identifies a blind spot in long-context LLM reasoning benchmarks: they fail to control task position within the context, allowing positional failures to go undetected. The authors propose Context Rot Evaluation (CRE) to systematically vary task position, filler content, and context length, revealing severe accuracy drops for some models when reasoning tasks are placed in the middle of long contexts.