TTPO: Test-Time Policy Optimization

Hugging Face Daily Papers Papers
test-time-training mathematical-reasoning large-language-models reinforcement-learning self-distillation

Summary

TTPO introduces a label-free method for test-time training in mathematical reasoning that matches supervised performance by asymmetrically handling agreeing and disagreeing rollouts.

Recent prominent post-training methods, such as Reinforcement Learning (RL) and On-Policy Self-Distillation (OPSD), have driven rapid progress in mathematical reasoning for large language models, yet their reliance on ground-truth labels precludes test-time training (TTT). Replacing ground truth with majority-vote pseudo-labels is a natural alternative, yet it is fragile: an incorrect vote corrupts the teacher and misleads every token. We observe that this failure mode is asymmetric: rollouts that disagree with the pseudo-label are typically wrong regardless of whether the vote itself is correct. Building on this observation, we propose Test-Time Policy Optimization (TTPO), an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts with Grouped RL. Token-level selection further refines both branches: distillation down-weights already-converged positions, while RL penalizes only confident errors. Both updates remain well-grounded even under frequent pseudo-label errors, and majority-vote routing yields tighter self-supervision as the model improves. Without any labels, TTPO matches label-supervised OPSD on five competition-level benchmarks, raises Qwen3-1.7B from 38.0% to 45.2% in TTT, yields +25.2% to +36.4% without thinking, and shows strong cross-task generalization.
Original Article
View Cached Full Text

Cached at: 08/28/26, 07:24 AM

Paper page - TTPO: Test-Time Policy Optimization

Source: https://huggingface.co/papers/2608.27448

Abstract

Test-Time Policy Optimization enables label-free test-time training for mathematical reasoning by asymmetrically distilling agreeing rollouts and penalizing disagreeing ones, matching supervised performance.

Recent prominent post-training methods, such asReinforcement Learning(RL) andOn-Policy Self-Distillation(OPSD), have driven rapid progress in mathematical reasoning for large language models, yet their reliance on ground-truth labels precludestest-time training(TTT). Replacing ground truth withmajority-vote pseudo-labelsis a natural alternative, yet it is fragile: an incorrect vote corrupts the teacher and misleads every token. We observe that this failure mode is asymmetric: rollouts that disagree with the pseudo-label are typically wrong regardless of whether the vote itself is correct. Building on this observation, we proposeTest-Time Policy Optimization(TTPO), an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts withGrouped RL.Token-level selectionfurther refines both branches: distillation down-weights already-converged positions, while RL penalizes only confident errors. Both updates remain well-grounded even under frequent pseudo-label errors, and majority-vote routing yields tighter self-supervision as the model improves. Without any labels, TTPO matches label-supervised OPSD on five competition-level benchmarks, raises Qwen3-1.7B from 38.0% to 45.2% in TTT, yields +25.2% to +36.4% without thinking, and shows strong cross-task generalization.

View arXiv pageView PDFProject pageGitHub8Add to collection

Get this paper in your agent:

hf papers read 2608\.27448

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.27448 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.27448 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.27448 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Planned Test-Time Scaling with Coordinated Reasoning Paths

arXiv cs.CL

This paper introduces Planned Test-Time Scaling (PTTS), a method that coordinates reasoning branches to enhance performance on challenging tasks, achieving significant gains over repeated sampling in mathematical reasoning benchmarks.

TPPO: Tracking-Value PPO For Language Model Reasoning

Hugging Face Daily Papers

该论文提出 Tracking-Value PPO（TPPO），通过在对数几率空间对前缀值进行贝叶斯分解，构建累积值参数化以更好地跟踪语言模型推理中的值演化。TPPO 在 Qwen3-4B-Base 和 Qwen3-8B-Base 上分别比 PPO 提升 7.0 和 8.4 个数学推理分数点，并改善了分布外泛化能力。

Modular TTT: Rethinking Test-Time Training as Composable Modules

Hugging Face Daily Papers

This paper introduces Modular TTT, a framework that represents test-time training inner learners as directed acyclic graphs, enabling systematic ablation and composition of components. The authors train 410M and 1.45B parameter models on 100B tokens, achieving performance comparable to GatedDeltaNet.