@QGallouedec: multi-turn RL and the "tito" problem keeps coming up. we've been working on it for a while, and the takeaway is that it…
Summary
A developer shares that addressing the 'tito' problem in multi-turn reinforcement learning is simpler than commonly believed, requiring only one implementation rule and a chat-template property that models already support.
View Cached Full Text
Cached at: 05/29/26, 11:45 AM
multi-turn RL and the “tito” problem keeps coming up. we’ve been working on it for a while, and the takeaway is that it’s much easier than people are making it.
it takes 1 implementation rule, and 1 chat-template property that all models already comply with.
**that’s all you https://t.co/O7BeRiPi5Y
Similar Articles
Agentic RL: Token-In, Token-Out Done Right (16 minute read)
This article explains the 'Token-In, Token-Out' (TITO) invariant in reinforcement learning for LLMs, highlighting a common error when training multi-turn agents with tool calls. It presents two solutions: using per-model renderers or designing training to avoid re-encoding decoded tokens, emphasizing prefix-preserving chat templates.
Multi-Turn Reasoning When Context Arrives in Pieces: Scalable Sharding and Memory-Augmented RL
This paper addresses the 'Lost in Conversation' problem where LLMs struggle with information revealed across multiple turns. It proposes a scalable sharding pipeline to create multi-turn training data from single-turn QA datasets and uses reinforcement learning with verifiable rewards to train a memory-augmented policy that maintains a compact rolling memory, improving multi-turn reasoning accuracy and generalizing zero-shot to harder tasks.
UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning
UI-TARS-2 is a native GUI-centered agent model that addresses data scalability, multi-turn RL, and environment stability challenges, achieving state-of-the-art results on GUI benchmarks (88.2 on Online-Mind2Web, 47.5 on OSWorld, 50.6 on WindowsAgentArena,73.3 on AndroidWorld) and outperforming Claude and OpenAI agents.
RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training
The paper proposes Reverse-Turn Policy Optimization (RTPO) to stabilize multi-turn agentic reinforcement learning training by organizing rollouts as reverse trees and performing turn-level updates in reverse order, showing improvements over baselines.
SocialRL: Refining LLMs' Social Intelligence through Multi-turn Reinforcement Learning and Reward Design
The paper introduces SocialRL, a multi-turn reinforcement learning framework that enhances the social intelligence of large language models through delayed reward propagation and fine-grained process rewards, achieving notable improvements in goal completion for dialogue systems.