Procedural Memory Distillation: Online Reflection for Self-Improving Language Models

arXiv cs.AI Papers
self-improving language-models distillation reinforcement-learning procedural-memory online-reflection reward-verification

Summary

Procedural Memory Distillation (PMD) converts cross-episode signals from reinforcement learning rollouts into reusable procedural memory that is distilled into the policy weights during training, enabling self-improving language models without memory at inference. Experiments show PMD outperforms SDPO by 3.8-5.5% on SCIKNOWEVAL and 7.9-13.6% on LIVECODEBENCH.

arXiv:2607.01480v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR), along with recent selfdistillation variants such as SDPO, evaluates each rollout against a verifier and updates the policy from that episode-level signal. However, the richer procedural information in the rollout is rarely retained or reused. Across episodes and epochs, the model repeatedly encounters related problems under a changing policy, producing cross-episode signals that episode-local updates cannot capture: which strategies consistently pass verification, which failure modes persist, which patterns recur. We propose Procedural Memory Distillation (PMD), which converts these crossepisode signals into reusable procedural memory and distills it into the policy's weights during training. This memory functions as a training scaffold, absorbed into the policy itself, yielding a memory-free model at inference. PMD organizes the memory at three levels of abstraction: raw trajectories, self-reflected strategies and lessons, and higher-level behavioral patterns that recur across problems, all extracted online from the model's own trajectories. A memory-conditioned self-teacher draws on the accumulated experience to supervise the student on its own rollouts, enabling student to progressively internalize procedural knowledge within its parameters. The central design principle is co-evolution: the policy generates rollouts that update the memory, and memory shapes the supervision that updates the policy. Empirically, across Qwen3-8B and OLMo3-Instruct-7B, PMD improves over SDPO by 3.8-5.5% on SCIKNOWEVAL and 7.9-13.6% on LIVECODEBENCH. Co-evolution powers these gains: freezing either the memory or the policy trails PMD by more than 10% across SCIKNOWEVAL domains.
Original Article
View Cached Full Text

Cached at: 07/03/26, 05:44 AM 

# Procedural Memory Distillation: Online Reflection for Self-Improving Language Models
Source: [https://arxiv.org/abs/2607.01480](https://arxiv.org/abs/2607.01480)
[View PDF](https://arxiv.org/pdf/2607.01480)

> Abstract:Reinforcement learning with verifiable rewards \(RLVR\), along with recent selfdistillation variants such as SDPO, evaluates each rollout against a verifier and updates the policy from that episode\-level signal\. However, the richer procedural information in the rollout is rarely retained or reused\. Across episodes and epochs, the model repeatedly encounters related problems under a changing policy, producing cross\-episode signals that episode\-local updates cannot capture: which strategies consistently pass verification, which failure modes persist, which patterns recur\. We propose Procedural Memory Distillation \(PMD\), which converts these crossepisode signals into reusable procedural memory and distills it into the policy's weights during training\. This memory functions as a training scaffold, absorbed into the policy itself, yielding a memory\-free model at inference\. PMD organizes the memory at three levels of abstraction: raw trajectories, self\-reflected strategies and lessons, and higher\-level behavioral patterns that recur across problems, all extracted online from the model's own trajectories\. A memory\-conditioned self\-teacher draws on the accumulated experience to supervise the student on its own rollouts, enabling student to progressively internalize procedural knowledge within its parameters\. The central design principle is co\-evolution: the policy generates rollouts that update the memory, and memory shapes the supervision that updates the policy\. Empirically, across Qwen3\-8B and OLMo3\-Instruct\-7B, PMD improves over SDPO by 3\.8\-5\.5% on SCIKNOWEVAL and 7\.9\-13\.6% on LIVECODEBENCH\. Co\-evolution powers these gains: freezing either the memory or the policy trails PMD by more than 10% across SCIKNOWEVAL domains\.

## Submission history

From: Semih Yavuz \[[view email](https://arxiv.org/show-email/9c696e1f/2607.01480)\] **\[v1\]**Wed, 1 Jul 2026 21:20:57 UTC \(696 KB\)

Similar Articles

OPRD: On-Policy Representation Distillation

Hugging Face Daily Papers

OPRD proposes a new knowledge distillation method that aligns student and teacher hidden states across layers during on-policy rollouts, eliminating sampling variance from token-space KL estimation. Empirically, OPRD outperforms output-space baselines on math reasoning benchmarks (AIME 2024/2025, AIMO) while being 1.44x faster and using 54% less memory.

@michalwols: Amazing talk on on policy self distillation on @yacinelearning 's channel

X AI KOLs Timeline

This talk introduces self-distillation methods (SDPO and SDFT) that use model-generated rollouts and rich environment feedback (e.g., compiler errors) to create dense token-level learning signals, achieving 6× faster convergence than GRPO and 11× shorter reasoning traces, while also enabling sequential skill learning without forgetting.

OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning

Hugging Face Daily Papers

OPID proposes an on-policy skill distillation framework that extracts dense hindsight supervision from completed trajectories, combining outcome-based RL with token-level self-distillation to improve language agent training efficiency and performance on multi-turn tasks.