Multi-Turn Reasoning When Context Arrives in Pieces: Scalable Sharding and Memory-Augmented RL

# Multi-Turn Reasoning When Context Arrives in Pieces: Scalable Sharding and Memory-Augmented RL
Source: [https://arxiv.org/html/2606.12941](https://arxiv.org/html/2606.12941)
Wenqin LiuRui Liu The University of Melbourne \{shutong2,wenqinl,ruiliu2\}@student\.unimelb\.edu\.auMingming Gong The University of Melbourne mingming\.gong@unimelb\.edu\.au &Jiaxian Guo Google Research Australia jeffguo@google\.com

###### Abstract

When a user reveals task\-critical information across several conversation turns, LLM accuracy drops by up to 65% despite full context availability\. We show that thisLost in Conversationdegradation can be substantially mitigated by training models to maintain a compact rolling memory instead of attending to a growing history\. To make such training scalable, we introduce a low\-cost sharding pipeline that converts single\-turn QA datasets into multi\-turn fragmented\-information episodes, eliminating the need for hours of manual annotation\. Training only on sharded GSM8K, our memory\-augmented policy significantly improves multi\-turn accuracy and generalises zero\-shot to harder math and out\-of\-domain long\-context QA\. Moreover, memory\-trained models outperform full\-history baselines even when given the full history at test time, suggesting that learning to compress induces more robust incremental reasoning than full\-context exposure alone\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2606.12941v1/x1.png)Figure 1:Overview of our approach\. We shard single\-turn QA problems into sharded multi\-turn episodes and use RLVR to train a memory\-augmented policy via multi\-turn DAPO\. Models trained exclusively on sharded GSM8K generalise well to in\-domain and OOD long\-context datasets\.Consider a user interacting with a travel\-planning assistant\. Rather than providing all requirements upfront, the user incrementally reveals constraints such as including the destination, budget, and travel dates across multiple turns\. Successfully completing such task requires the model to continuously accumulate and reconcile information distributed throughout the conversation\. Yet, frontier LLMs struggle substantially in such settings, with average task accuracy degrading by 39% on the benchmark ofLabanet al\.\([2026](https://arxiv.org/html/2606.12941#bib.bib1)\)\. This phenomenon is termedLost in Conversation\(LiC\)\. Since incremental context is fundamental to both human\-AI dialogue and agentic workflows, LiC represents a major bottleneck for real\-world LLM applications\.

Reinforcement Learning with Verifiable Rewards \(RLVR\) has driven recent progress on reasoning\(DeepSeek\-AI,[2025](https://arxiv.org/html/2606.12941#bib.bib17); Yuet al\.,[2025b](https://arxiv.org/html/2606.12941#bib.bib12)\), but two obstacles block its direct application to LiC\. First, training data is expensive: the LiC benchmark pipeline requires 1 to 4 hours of manual inspection per task and 30 LLM simulations per instruction, yielding only 90 to 120 validated examples\(Labanet al\.,[2026](https://arxiv.org/html/2606.12941#bib.bib1)\), insufficient for RLVR at scale\. Second, models default to reasoning over the full chat history, and this strategy degrades with length\(Liuet al\.,[2023](https://arxiv.org/html/2606.12941#bib.bib7)\), leaving no obvious mechanism for robust cross\-turn accumulation\.

We address both obstacles with a single recipe\. The model maintains a bounded natural\-language memory buffer that it rewrites at every turn, trained via RLVR on cheaply sharded multi\-turn data\. Only the buffer, not the raw history, is carried forward\. While such memory buffers have been shown to compress single\-turn long documents effectively\(Yuet al\.,[2025a](https://arxiv.org/html/2606.12941#bib.bib2)\), whether the same mechanism extends to incremental dialogue, where information is partial at every step and must be integrated under reward, remains unknown\. We answer this question affirmatively, and find that bounded memory is not merely a viable substitute for full history but a stronger training signal\.

Our contributions are threefold: \(1\) A cheap, scalable sharding pipeline that converts any single\-turn QA dataset into multi\-turn fragmented episodes using only 1 to 3 few\-shot examples, with noise\-augmented variants for ablation; \(2\) A memory\-augmented, multi\-turn RL recipe that recovers up to 60 points of LiC degradation on GSM8KCobbeet al\.\([2021](https://arxiv.org/html/2606.12941#bib.bib3)\)and surpasses full\-history training under all evaluation conditions; \(3\) Evidence that this capability is domain\-general: models trained only on GSM8K generalise zero\-shot to MATH500Hendryckset al\.\([2021](https://arxiv.org/html/2606.12941#bib.bib4)\); Lightmanet al\.\([2023](https://arxiv.org/html/2606.12941#bib.bib5)\)and LongBenchBaiet al\.\([2024](https://arxiv.org/html/2606.12941#bib.bib6)\)\.

## 2Related Work

#### The Lost in Conversation problem\.

While most multi\-turn benchmarks treat conversations episodically\(Zhenget al\.,[2023](https://arxiv.org/html/2606.12941#bib.bib13); Kwanet al\.,[2024](https://arxiv.org/html/2606.12941#bib.bib14)\),Labanet al\.\([2026](https://arxiv.org/html/2606.12941#bib.bib1)\)show that frontier model performance degrades by 39% on average under incremental context revelation, a setting that remains largely underexplored for training\. Concurrent work\(Liet al\.,[2026](https://arxiv.org/html/2606.12941#bib.bib15)\)addresses solvability detection from incomplete shard sequences via RLVR, yet scalable data construction and accumulation reliability across complete sequences remain open\.

#### Memory\-augmented language models\.

As context grows, model performance degrades due to the lost\-in\-the\-middle phenomenon\(Liuet al\.,[2023](https://arxiv.org/html/2606.12941#bib.bib7)\), motivating memory mechanisms that avoid relying on the full history\. Retrieval\-based approaches\(Lewiset al\.,[2021](https://arxiv.org/html/2606.12941#bib.bib18); Gaoet al\.,[2024](https://arxiv.org/html/2606.12941#bib.bib19)\)address this but require explicit retrieval infrastructure; an appealing alternative is an in\-context memory buffer that the model itself maintains\.Yuet al\.\([2025a](https://arxiv.org/html/2606.12941#bib.bib2)\)demonstrates the viability by training a model to iteratively compress document chunks into a bounded memory buffer, outperforming full\-context approaches on long\-context QA\.

#### RLVR\.

RLVR has emerged as a powerful training paradigm for eliciting reasoning capability in LLMs\(DeepSeek\-AI,[2025](https://arxiv.org/html/2606.12941#bib.bib17); Wenet al\.,[2025](https://arxiv.org/html/2606.12941#bib.bib21)\), with subsequent work focusing on stabilising policy optimisation\(Yuet al\.,[2025b](https://arxiv.org/html/2606.12941#bib.bib12); Yueet al\.,[2025](https://arxiv.org/html/2606.12941#bib.bib20); Liuet al\.,[2025](https://arxiv.org/html/2606.12941#bib.bib16)\)\. MemAgent\(Yuet al\.,[2025a](https://arxiv.org/html/2606.12941#bib.bib2)\)extends this to sequences of context\-independent model calls, propagating a single outcome reward across all intermediate memory update steps, but operates over document chunks and has not been extended to genuinely multi\-turn dialogue, where information is partial and must be integrated incrementally\.

## 3Methodology

### 3\.1Sharded Dataset Construction

We construct multi\-turn sharded datasets from GSM8K and MATH500 via LLM\-based sharding, which preserves the logical structure of each problem by decomposing it into semantically complete units\. Each problem is processed through a three\-step prompting pipeline: \(1\) the problem statement is segmented into its minimal logical units; \(2\) the segmentation is then verified for completeness and non\-redundancy; \(3\) the core question is extracted and withheld from the shard sequence, instead presented at every turn alongside the current shard\. The pipeline requires only 1–3 manually annotated few\-shot examples as reference in the prompt, which significantly reduces human effort\.

For long\-context datasets, which lack the discrete reasoning structure that motivates LLM\-based segmentation, the LongBench QA documents are instead chunked into fixed\-size token segments aligned to sentence boundaries, averaging 10 to 14 shards per document\.

To ablate what training structure best induces memory extraction, we train three dataset variants: \(1\)Sharded, following the pipeline above; \(2\)Full\-question, where one randomly\-positioned shard contains the full question and remaining shards contain randomly sampled Wikitext passagesMerityet al\.\([2016](https://arxiv.org/html/2606.12941#bib.bib24)\), forcing the model to distinguish relevant content from noise; and \(3\)Mixed, interleavingShardedwith Wikitext noise shards to combine logical decomposition with distraction\. We additionally construct noise\-augmented sets from MATH500 by injectingn∈\{2,4\}n\\in\\\{2,4\\\}noise shards between each turn to assess test\-time robustness\.

### 3\.2RL with In\-Context Memory

Training a model to accumulate information across fragmented turns introduces a credit assignment problem: the policy must learn to write useful memory states at every intermediate turn, yet the only training signal is whether the final answer is correct\. We extend the multi\-conv DAPO framework\(Yuet al\.,[2025a](https://arxiv.org/html/2606.12941#bib.bib2)\)from single\-turn native inference to a genuinely multi\-turn setting, applying it to discrete reasoning shards derived from single\-turn problems rather than chunks of a single long document\.

#### Memory Mechanism\.

At each shard turnt∈\{1,…,K\}t\\in\\\{1,\\dots,K\\\}, the policy receives the questionqq, the current memory statemtm\_\{t\}, and the incoming shardsts\_\{t\}, and generates a rewritten memorymt\+1∼πθ\(⋅∣q,mt,st\)m\_\{t\+1\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid q,m\_\{t\},s\_\{t\}\)\. The memory is a free\-form natural language buffer capped atLm=256L\_\{m\}=256tokens, deliberately constrained to prevent the model from copying all previously seen evidence and to make selective retention non\-trivial\. At the final turn, the policy conditions only onqqand the terminal memorymKm\_\{K\}to produce an answera∼πθ\(⋅∣q,mK\)a\\sim\\pi\_\{\\theta\}\(\\cdot\\mid q,m\_\{K\}\)\. The memory representation receives no direct supervision; its structure emerges from the downstream reward signal alone\.

#### Training Objective\.

For each sample,GGtrajectories are sampled fromπθold\\pi\_\{\\theta\_\{\\text\{old\}\}\}, each yielding a scalar rewardR\(g\)=𝒱​\(a\(g\),y⋆\)R^\{\(g\)\}=\\mathcal\{V\}\(a^\{\(g\)\},y^\{\\star\}\)from a rule\-based verifier\. Group\-relative advantages are computed as:

A^\(g\)=R\(g\)−1G​∑h=1GR\(h\)\.\\hat\{A\}^\{\(g\)\}=R^\{\(g\)\}\-\\frac\{1\}\{G\}\\sum\_\{h=1\}^\{G\}R^\{\(h\)\}\.\(1\)This advantage is propagated uniformly to all turns in trajectorygg\. As determining what a memory stateshouldcontain at any given turn is non\-deterministic, we avoid intermediate credit assignment\. Following DAPO\(Yuet al\.,[2025b](https://arxiv.org/html/2606.12941#bib.bib12)\), the actor is updated with a dual\-clipped surrogate with token\-mean aggregation across all turns, ensuring longer memory updates do not disproportionately dominate the gradient relative to the final answer turn\. A KL penalty is applied as a separate loss term to constrain policy drift:

ℒ​\(θ\)=ℒclip​\(θ\)\+β​DKL​\(πθ∥πref\)\.\\mathcal\{L\}\(\\theta\)=\\mathcal\{L\}\_\{\\text\{clip\}\}\(\\theta\)\+\\beta\\,D\_\{\\text\{KL\}\}\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\text\{ref\}\}\)\.\(2\)

#### Full\-History Baseline\.

We train a separatefull\-historymodel where the policy receives the concatenated chat history of all previous turns, rather than a compressed memory state\. At inference, all models are evaluated under both thememoryandfull\-historycondition, isolating whether performance differences are driven by memory compression or the underlying shard structure\.

## 4Experimental Setup

We train two model scales, Qwen2\.5\-Math\-1\.5B\-InstructYanget al\.\([2024](https://arxiv.org/html/2606.12941#bib.bib25)\)and Qwen3\-4B\-ThinkingYanget al\.\([2025](https://arxiv.org/html/2606.12941#bib.bib26)\), withGG=4 trajectories per prompt on 2×A100 GPUs using veRL\+vLLMShenget al\.\([2025](https://arxiv.org/html/2606.12941#bib.bib23)\); Kwonet al\.\([2023](https://arxiv.org/html/2606.12941#bib.bib22)\)\. The actor uses a dual\-clipped PPO surrogateSchulmanet al\.\([2017](https://arxiv.org/html/2606.12941#bib.bib27)\); Yuet al\.\([2025b](https://arxiv.org/html/2606.12941#bib.bib12)\), DrGRPO\-style advantage normalisationLiuet al\.\([2025](https://arxiv.org/html/2606.12941#bib.bib16)\); Shaoet al\.\([2024](https://arxiv.org/html/2606.12941#bib.bib28)\), KL penalty \(β=0\.001\\beta=0\.001\), and learning rate1×10−61\\times 10^\{\-6\}\.

We evaluate under two inference conditions:memory, where the model receives only the question, current memory state, and incoming shard at each turn, andfull\-history, where all previous shards are provided as context\. OOD evaluation covers five LongBench subsets: 2WikiMQAHoet al\.\([2020](https://arxiv.org/html/2606.12941#bib.bib8)\), HotpotQAYanget al\.\([2018](https://arxiv.org/html/2606.12941#bib.bib9)\), MultifieldQA\(Baiet al\.,[2024](https://arxiv.org/html/2606.12941#bib.bib6)\), QasperDasigiet al\.\([2021](https://arxiv.org/html/2606.12941#bib.bib10)\), and TriviaQAJoshiet al\.\([2017](https://arxiv.org/html/2606.12941#bib.bib11)\)\. Training is conducted exclusively on GSM8K, evaluating on the other datasets\. For brevity, variant comparisons and LongBench results are 4B\-only, where the larger model provides a cleaner signal\.

## 5Results

![Refer to caption](https://arxiv.org/html/2606.12941v1/x2.png)Figure 2:Accuracy under full\-history inference across datasets and model scales \(a–b\), and MATH500 noise robustness under shard injection \(c–d\), also evaluated with full conversation history available\. Memory RL consistently outperforms both baselines, recovering LiC degradation and maintaining an advantage under noise\.ModelMethodGSM8KMATH500Noisy \(nn=2\)Noisy \(nn=4\)Qwen2\.5\-1\.5BBase0\.1990\.3340\.2900\.284Memory RL \(Sharded\)0\.7990\.5500\.2900\.290Qwen3\-4B \(rr=512\)Base0\.2650\.0500\.0640\.050Memory RL \(Sharded\)0\.8250\.5160\.1600\.048Qwen3\-4B \(rr=1024\)Base0\.3700\.2600\.2320\.196Memory RL \(Sharded\)0\.8770\.6380\.5380\.438Memory RL \(Full\-Question\)0\.8230\.6820\.6380\.516Memory RL \(Mixed\)0\.8490\.6920\.6300\.570Table 1:Accuracy under the rolling\-memory framework\. Memory RL models consistently outperform base models, generalising to unseen harder datasets\. Truncation atr=512r\{=\}512collapses thinking model generalisation\.#### Memory RL vs\. Full\-History Training

[Figure˜2](https://arxiv.org/html/2606.12941#S5.F2)reveals a substantialLiCeffect between single\-turn and sharded multi\-turn base models \(19\.4–64\.9pp drop\)\. Full\-history RL partially recovers this, but Memory RL outperforms it by 15\.2–35\.6pp across both datasets and model scales, gains that persist under identical full\-history conditions without the memory mechanism, suggesting the memory training objective instils more robust representations rather than merely exploiting compression at inference time\. Injecting noise shards further exposes this gap; Full\-history RL degrades sharply while Memory RL remains more resilient atn=2n\{=\}2, consistent with selective retention implicitly encouraging noise\-filtering\.

#### Memory\-Augmented Evaluation

[Table˜1](https://arxiv.org/html/2606.12941#S5.T1)shows that these gains extend to the rolling\-memory setting, where Memory RL produces consistent improvements across both scales, with the 1\.5B and 4B \(r=1024r=1024\) models improving by 60% and 51% on GSM8K respectively, and both generalising to the unseen, harder MATH500 dataset, supporting that training induces general incremental reasoning rather than dataset\-specific pattern matching\.

Ther=512r=512ablation reveals that the 4B thinking model requires sufficient response length to externalise its reasoning chain, with truncation collapsing MATH500 accuracy to 0\.050 and suppressing noise robustness; both recover atr=1024r=1024\. Among these variants,Shardedachieves the highest in\-distribution accuracy but degrades most under noise, whileMixedoffers the best overall balance, as interleaving context shards with noise during training encourages the model to both decompose problems logically and filter irrelevant content, improving resilience when noisy shards are encountered at inference time\.

#### Generalisation to Long\-Context QA

[Table˜2](https://arxiv.org/html/2606.12941#S5.T2)shows that all Memory RL variants generalise to every LongBench QA subset without task\-specific training, improving average F1 by 31\.7–43\.4% over the base model\.Full\-QuestionandMixedoutperformShardedon every subset, suggesting that training on noisy contexts builds distractor\-filtering skills that transfer more readily to open\-domain QA thanSharded’s sequential reasoning\. Overall, these results indicate that memory\-based RL induces a general\-purpose information extraction capability that transfers broadly across domains\.

Model2WikiHotpotMultiQasperTriviaBase0\.4700\.4070\.2600\.2560\.691Sharded0\.6820\.5880\.3920\.3070\.775Full\-Q0\.7050\.6160\.4130\.3750\.850Mixed0\.7130\.6270\.4060\.3900\.852Table 2:LongBench F1 score for training variants \(4B\)

## 6Conclusion

We presented a scalable sharding pipeline for constructing multi\-turn training episodes from single\-turn QA datasets, and showed that training a memory\-augmented policy via multi\-turn DAPO substantially recoversLiCdegradation\. Memory\-trained models outperform full\-context baselines even under full\-context evaluation, and generalise zero\-shot to harder mathematical reasoning and out\-of\-domain long\-context QA, suggesting that memory compression induces a domain\-agnostic incremental reasoning capability\. We aim for this work to provide a practical and reproducible foundation for future research on LLM reliability\.

## Limitations

All datasets and models used are publicly available for research purposes\. GSM8K and both Qwen models are released under permissive open\-source licenses \(MIT and Apache 2\.0 respectively\); sharded derivatives produced in this work are likewise intended for research use only\.

Due to resource constraints, experiments are conducted on two scales within a single model family \(Qwen\)\. While the lost\-in\-conversation phenomenon has been demonstrated consistently across all tested model families\(Labanet al\.,[2026](https://arxiv.org/html/2606.12941#bib.bib1)\), and we therefore expect our findings to generalise, cross\-family validation remains future work\.

The memory buffer is intentionally capped at 256 tokens, stricter than the shard content itself, to force the model to learn selective retention rather than verbatim copying; larger buffer sizes may relax this inductive bias at the cost of less compact representations\.

Finally, while our models generalise zero\-shot to LongBench QA subsets, these are evaluated without task\-specific training, meaning performance may not reflect the full potential of the memory mechanism under domain\-matched conditions\.

## References

- Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou, Y\. Dong, J\. Tang, and J\. Li \(2024\)LongBench: a bilingual, multitask benchmark for long context understanding\.External Links:2308\.14508,[Link](https://arxiv.org/abs/2308.14508)Cited by:[§1](https://arxiv.org/html/2606.12941#S1.p4.1),[§4](https://arxiv.org/html/2606.12941#S4.p2.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.External Links:2110\.14168,[Link](https://arxiv.org/abs/2110.14168)Cited by:[§1](https://arxiv.org/html/2606.12941#S1.p4.1)\.
- P\. Dasigi, K\. Lo, I\. Beltagy, A\. Cohan, N\. A\. Smith, and M\. Gardner \(2021\)A dataset of information\-seeking questions and answers anchored in research papers\.External Links:2105\.03011,[Link](https://arxiv.org/abs/2105.03011)Cited by:[§4](https://arxiv.org/html/2606.12941#S4.p2.1)\.
- DeepSeek\-AI \(2025\)DeepSeek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.External Links:2501\.12948,[Link](https://arxiv.org/abs/2501.12948)Cited by:[§1](https://arxiv.org/html/2606.12941#S1.p2.1),[§2](https://arxiv.org/html/2606.12941#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Gao, Y\. Xiong, X\. Gao, K\. Jia, J\. Pan, Y\. Bi, Y\. Dai, J\. Sun, M\. Wang, and H\. Wang \(2024\)Retrieval\-augmented generation for large language models: a survey\.External Links:2312\.10997,[Link](https://arxiv.org/abs/2312.10997)Cited by:[§2](https://arxiv.org/html/2606.12941#S2.SS0.SSS0.Px2.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021\)Measuring mathematical problem solving with the math dataset\.NeurIPS\.Cited by:[§1](https://arxiv.org/html/2606.12941#S1.p4.1)\.
- X\. Ho, A\. D\. Nguyen, S\. Sugawara, and A\. Aizawa \(2020\)Constructing a multi\-hop qa dataset for comprehensive evaluation of reasoning steps\.External Links:2011\.01060,[Link](https://arxiv.org/abs/2011.01060)Cited by:[§4](https://arxiv.org/html/2606.12941#S4.p2.1)\.
- M\. Joshi, E\. Choi, D\. S\. Weld, and L\. Zettlemoyer \(2017\)TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension\.External Links:1705\.03551,[Link](https://arxiv.org/abs/1705.03551)Cited by:[§4](https://arxiv.org/html/2606.12941#S4.p2.1)\.
- W\. Kwan, X\. Zeng, Y\. Jiang, Y\. Wang, L\. Li, L\. Shang, X\. Jiang, Q\. Liu, and K\. Wong \(2024\)MT\-eval: a multi\-turn capabilities evaluation benchmark for large language models\.External Links:2401\.16745,[Link](https://arxiv.org/abs/2401.16745)Cited by:[§2](https://arxiv.org/html/2606.12941#S2.SS0.SSS0.Px1.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.External Links:2309\.06180,[Link](https://arxiv.org/abs/2309.06180)Cited by:[§4](https://arxiv.org/html/2606.12941#S4.p1.3)\.
- P\. Laban, H\. Hayashi, Y\. Zhou, and J\. Neville \(2026\)LLMs get lost in multi\-turn conversation\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2606.12941#S1.p1.1),[§1](https://arxiv.org/html/2606.12941#S1.p2.1),[§2](https://arxiv.org/html/2606.12941#S2.SS0.SSS0.Px1.p1.1),[Limitations](https://arxiv.org/html/2606.12941#Sx1.p2.1)\.
- P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. Kiela \(2021\)Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.External Links:2005\.11401,[Link](https://arxiv.org/abs/2005.11401)Cited by:[§2](https://arxiv.org/html/2606.12941#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Li, P\. Chen, Z\. Zhang, T\. Yang, X\. Zhang, H\. Li, T\. Cao, M\. Zeng, Z\. Wu, M\. Jiang, H\. Li, L\. Li, and B\. Yin \(2026\)Mitigating lost in multi\-turn conversation via curriculum rl with verifiable accuracy and abstention rewards\.External Links:2510\.18731,[Link](https://arxiv.org/abs/2510.18731)Cited by:[§2](https://arxiv.org/html/2606.12941#S2.SS0.SSS0.Px1.p1.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2023\)Let’s verify step by step\.arXiv preprint arXiv:2305\.20050\.Cited by:[§1](https://arxiv.org/html/2606.12941#S1.p4.1)\.
- N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. Liang \(2023\)Lost in the middle: how language models use long contexts\.External Links:2307\.03172,[Link](https://arxiv.org/abs/2307.03172)Cited by:[§1](https://arxiv.org/html/2606.12941#S1.p2.1),[§2](https://arxiv.org/html/2606.12941#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. Lin \(2025\)Understanding r1\-zero\-like training: a critical perspective\.External Links:2503\.20783,[Link](https://arxiv.org/abs/2503.20783)Cited by:[§2](https://arxiv.org/html/2606.12941#S2.SS0.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2606.12941#S4.p1.3)\.
- S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher \(2016\)Pointer sentinel mixture models\.External Links:1609\.07843Cited by:[§3\.1](https://arxiv.org/html/2606.12941#S3.SS1.p3.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.External Links:1707\.06347,[Link](https://arxiv.org/abs/1707.06347)Cited by:[§4](https://arxiv.org/html/2606.12941#S4.p1.3)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo \(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.External Links:2402\.03300,[Link](https://arxiv.org/abs/2402.03300)Cited by:[§4](https://arxiv.org/html/2606.12941#S4.p1.3)\.
- G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. Wu \(2025\)HybridFlow: a flexible and efficient rlhf framework\.InProceedings of the Twentieth European Conference on Computer Systems,EuroSys ’25,pp\. 1279–1297\.External Links:[Link](http://dx.doi.org/10.1145/3689031.3696075),[Document](https://dx.doi.org/10.1145/3689031.3696075)Cited by:[§4](https://arxiv.org/html/2606.12941#S4.p1.3)\.
- X\. Wen, Z\. Liu, S\. Zheng, S\. Ye, Z\. Wu, Y\. Wang, Z\. Xu, X\. Liang, J\. Li, Z\. Miao, J\. Bian, and M\. Yang \(2025\)Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms\.External Links:2506\.14245,[Link](https://arxiv.org/abs/2506.14245)Cited by:[§2](https://arxiv.org/html/2606.12941#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§4](https://arxiv.org/html/2606.12941#S4.p1.3)\.
- A\. Yang, B\. Zhang, B\. Hui, B\. Gao, B\. Yu, C\. Li, D\. Liu, J\. Tu, J\. Zhou, J\. Lin, K\. Lu, M\. Xue, R\. Lin, T\. Liu, X\. Ren, and Z\. Zhang \(2024\)Qwen2\.5\-math technical report: toward mathematical expert model via self\-improvement\.External Links:2409\.12122,[Link](https://arxiv.org/abs/2409.12122)Cited by:[§4](https://arxiv.org/html/2606.12941#S4.p1.3)\.
- Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning \(2018\)HotpotQA: a dataset for diverse, explainable multi\-hop question answering\.External Links:1809\.09600,[Link](https://arxiv.org/abs/1809.09600)Cited by:[§4](https://arxiv.org/html/2606.12941#S4.p2.1)\.
- H\. Yu, T\. Chen, J\. Feng, J\. Chen, W\. Dai, Q\. Yu, Y\. Zhang, W\. Ma, J\. Liu, M\. Wang, and H\. Zhou \(2025a\)MemAgent: reshaping long\-context llm with multi\-conv rl\-based memory agent\.External Links:2507\.02259,[Link](https://arxiv.org/abs/2507.02259)Cited by:[§1](https://arxiv.org/html/2606.12941#S1.p3.1),[§2](https://arxiv.org/html/2606.12941#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2606.12941#S2.SS0.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2606.12941#S3.SS2.p1.1)\.
- Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, T\. Fan, G\. Liu, L\. Liu, X\. Liu, H\. Lin, Z\. Lin, B\. Ma, G\. Sheng, Y\. Tong, C\. Zhang, M\. Zhang, W\. Zhang, H\. Zhu, J\. Zhu, J\. Chen, J\. Chen, C\. Wang, H\. Yu, W\. Dai, Y\. Song, X\. Wei, H\. Zhou, J\. Liu, W\. Ma, Y\. Zhang, L\. Yan, M\. Qiao, Y\. Wu, and M\. Wang \(2025b\)DAPO: an open\-source llm reinforcement learning system at scale\.External Links:2503\.14476,[Link](https://arxiv.org/abs/2503.14476)Cited by:[§1](https://arxiv.org/html/2606.12941#S1.p2.1),[§2](https://arxiv.org/html/2606.12941#S2.SS0.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2606.12941#S3.SS2.SSS0.Px2.p1.4),[§4](https://arxiv.org/html/2606.12941#S4.p1.3)\.
- Y\. Yue, Y\. Yuan, Q\. Yu, X\. Zuo, R\. Zhu, W\. Xu, J\. Chen, C\. Wang, T\. Fan, Z\. Du, X\. Wei, X\. Yu, G\. Liu, J\. Liu, L\. Liu, H\. Lin, Z\. Lin, B\. Ma, C\. Zhang, M\. Zhang, W\. Zhang, H\. Zhu, R\. Zhang, X\. Liu, M\. Wang, Y\. Wu, and L\. Yan \(2025\)VAPO: efficient and reliable reinforcement learning for advanced reasoning tasks\.External Links:2504\.05118,[Link](https://arxiv.org/abs/2504.05118)Cited by:[§2](https://arxiv.org/html/2606.12941#S2.SS0.SSS0.Px3.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.External Links:2306\.05685,[Link](https://arxiv.org/abs/2306.05685)Cited by:[§2](https://arxiv.org/html/2606.12941#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix AAppendix

### A\.1Training Hyperparameters

All models are trained for 10 epochs on sharded GSM8K with a batch size of 64 on 2×\\timesA100 GPUs, taking an average of approximately 35 hours per run until convergence\. Sequence lengths are deliberately constrained during training\. The memory buffer is capped at 256 tokens to force selective retention rather than verbatim copying, and the maximum number of shards per episode is capped at 35 to bound trajectory length\.

ParameterValueTraining epochs10Train batch size64Max shard tokens1,024 tokensMax memory bufferLmL\_\{m\}256 tokensMax shards per episode35Max response length512 / 1,024 tokensLR warmup steps10Wall\-clock time∼\\sim30 hours / runTable 3:RL training configuration\. Max response length 512/1,024 corresponds to 1\.5B/4B models respectively\.The response length budget differs by model scale: the 1\.5B model is a math\-specialised model and produces compact answers, sor=512r\{=\}512suffices\. The 4B model is a general\-purpose thinking model that expends a substantial portion of its response budget on chain\-of\-thought reasoning before producing an answer; atr=512r\{=\}512this reasoning is truncated, collapsing MATH500 accuracy to 0\.050\. Settingr=1024r\{=\}1024restores meaningful performance, as reported in[Table˜1](https://arxiv.org/html/2606.12941#S5.T1)\.

### A\.2Sharding Prompt

Sharding PromptYou are given a math problem\. Rewrite it as a natural multi\-turn conversation\. Each turn should restate part of the original problem in conversational form, preserving ALL information exactly\.
Rules:
1\. You may rephrase, but preserve all numbers, expressions, symbols, constraints, and formatting\.
2\. Only describe what is explicitly stated\. Do not add reasoning, methods, hints, or interpretations\.
3\. Do not add or remove facts\.
4\. Output only JSON with key shards\.
Example 1:
Q: Betty is saving money for a new wallet which costs $100\. Betty has only half of the money she needs\. Her parents decided to give her $15 for that purpose, and her grandparents twice as much as her parents\. How much more money does Betty need to buy the wallet?\{"shards": \[\{"shard": "How much more money does Betty need to buy the wallet?"\}, \{"shard": "The wallet costs $100"\}, \{"shard": "Betty has only half of the money she needs"\}, \{"shard": "Her parents gave her $15"\}, \{"shard": "Her grandparents gave twice as much as her parents"\}\]\}Example 2:
Q: What is the value of34×323^\{4\}\\times 3^\{2\}?\{"shards": \[\{"shard": "An expression is given:34×323^\{4\}\\times 3^\{2\}\."\},\{"shard": "The question asks for its value\."\}\]\}Example 3:
Q: Convert the point\(0,3\)\(0,3\)in rectangular coordinates to polar coordinates\. Enter your answer in the form\(r,θ\)\(r,\\theta\), wherer\>0r\>0and0≤θ<2​π0\\leq\\theta<2\\pi\.\{"shards": \[\{"shard": "A point is given:\(0,3\)\(0,3\)in rectangular coordinates\."\}, \{"shard": "It should be converted to polar coordinates\."\}, \{"shard": "The answer format is\(r,θ\)\(r,\\theta\), wherer\>0r\>0and0≤θ<2​π0\\leq\\theta<2\\pi\."\}\]\}Q: \[\[QUESTION\]\]Figure 3:Sharding prompt used to construct multi\-turn episodes from single\-turn problems\.\[\[QUESTION\]\]is replaced with the problem text at runtime\.The segmentation prompt is shown in[Figure˜3](https://arxiv.org/html/2606.12941#A1.F3), representative of the validation and question extraction prompts which follow the same format\. It includes explicit rules instructing the model to retain all numerical and symbolic content verbatim while avoiding added reasoning, hints, or interpretations, with output constrained to JSON for reliable programmatic parsing\. Three few\-shot examples spanning simple arithmetic, symbolic expressions, and multi\-constraint geometry encourage generalisation across diverse problem types, and the approach can readily extend to other domains by substituting the few\-shot examples accordingly\.

### A\.3Memory Mechanism

Prompt templates for intermediate memory update turns and the final answer turn are shown in[Figure˜4](https://arxiv.org/html/2606.12941#A1.F4)\. The final answer turn requires the model to produce its answer in`\\boxed\{\}`format, enabling rule\-based match verification against the ground\-truth solution without any learned reward model\.

Intermediate TurnYou are given a question and additional information\. Please read the new information carefully and update your understanding to help answer the question\. Retain all relevant details from your previous understanding while incorporating the new information\. ⟨question⟩ \{question\} ⟨/question⟩ ⟨memory⟩ \{memory\} ⟨/memory⟩ ⟨segment⟩ \{segment\} ⟨/segment⟩ Updated memory:

Final TurnYou are presented with a problem and a previous memory\. Please answer the problem based on the previous memory and put the answer in \\boxed\{\}\. ⟨question⟩ \{question\} ⟨/question⟩ ⟨memory⟩ \{memory\} ⟨/memory⟩ Your final answer:

Figure 4:Prompt templates for intermediate memory update turns and the final answer turn\. At intermediate turns the model rewrites its memory given a new shard; at the final turn it answers using only the terminal memory state, with no access to the shard history\.