Prompt-Driven Exploration

# Prompt-Driven Exploration
Source: [https://arxiv.org/html/2607.08837](https://arxiv.org/html/2607.08837)
Sunshine Jiang1,3,John Marangola1,3,David Zhang1,3,Raghuram Kowdeed3,Ruiyang Luo1,3,Nitish Dashora1,3,Richard Li1,3,Pulkit Agrawal1,3,Zhang\-Wei Hong1,2,3 Massachusetts Institute of Technology1, MIT\-IBM Computing Research Lab2, Improbable AI Lab3

###### Abstract

Exploration is essential to RL since a policy cannot improve by repeatedly sampling the behaviors it already prefers\. Standard methods inject stochasticity in the action space, but such jitter only yields rollouts close to the original\. Escaping a weak policy often requires global perturbations that action noise cannot produce\. Large language models \(LLMs\) and vision\-language\-action \(VLA\) models offer a pathway: they condition the policy on a natural language prompt, and since the rollout follows from it, modifying the prompt induces global changes\. The challenge is finding prompts that induce useful global changes\. With a weak policy that rarely succeeds, reward is too sparse to select on\. Our idea is to refine prompts from the rollouts themselves: a vision\-language model \(VLM\) reasons over the rollout video, diagnoses how the policy responded, and rewrites the prompt to elicit better behavior next time\. This procedure realizes posterior sampling, a classical RL exploration framework, at the level of prompts: the VLM maintains an implicit distribution over useful prompts and updates it from observed rollouts\. We call this strategy Prompt\-Driven Exploration \(PDE\)\. Across manipulation and reasoning tasks, PDE enables RL to learn successful policies even from zero\-reward starts, and improves sample efficiency more broadly\. Our website is available at[https://xinyunsunshine\.github\.io/prompt\-rl](https://xinyunsunshine.github.io/prompt-rl)\.

## 1Introduction

Reinforcement learning \(RL\)\([kaelbling1996reinforcement,](https://arxiv.org/html/2607.08837#bib.bib20)\)has become a dominant post\-training paradigm for foundation models because it enables scalable self\-improvement beyond supervised learning\. RL fine\-tuning has unlocked strong reasoning and mathematical capabilities in large language models \(LLMs\)\([jaech2024openai,](https://arxiv.org/html/2607.08837#bib.bib18);[Guo2025DeepSeekR1,](https://arxiv.org/html/2607.08837#bib.bib14)\)and enabled direct alignment with human preferences in large diffusion\-based text\-to\-image models\([black2024training,](https://arxiv.org/html/2607.08837#bib.bib6)\)\. However, self\-improvement is bottlenecked by exploration: a policy can only surpass its current behavior by producing rollouts different from those it already favors, so that RL can reinforce the higher\-reward ones\.

Standard practice perturbs the policy in action space by sampling actions stochastically rather than greedily\([williams1992simple,](https://arxiv.org/html/2607.08837#bib.bib46);[schulman2017ppo,](https://arxiv.org/html/2607.08837#bib.bib36);[haarnoja2018soft,](https://arxiv.org/html/2607.08837#bib.bib15)\)\. But such action\-level noise only jitters individual actions and yields rollouts close to the original; it induces only local exploration\([osband2016deep,](https://arxiv.org/html/2607.08837#bib.bib32);[ecoffet2021first,](https://arxiv.org/html/2607.08837#bib.bib13)\), and the set of rollouts reachable by step\-wise noise shrinks rapidly with the horizon and action dimension\. Escaping a weak policy often requires global perturbations that alter behavior across the entire rollout, which action noise cannot produce\. This limitation is especially pronounced in settings without a strong warm start, particularly vision\-language\-action \(VLA\) model fine\-tuning on manipulation, where state\-of\-the\-art models often start at near\-zero success rates\([kim2024openvla,](https://arxiv.org/html/2607.08837#bib.bib22);[black2024pi0,](https://arxiv.org/html/2607.08837#bib.bib5)\)\.

If action noise only produces local exploration, how can we perturb the policy globally? Foundation models offer a pathway\. LLMs\([brown2020language,](https://arxiv.org/html/2607.08837#bib.bib7);[achiam2023gpt,](https://arxiv.org/html/2607.08837#bib.bib1)\)and VLAs\([kim2024openvla,](https://arxiv.org/html/2607.08837#bib.bib22);[black2024pi0,](https://arxiv.org/html/2607.08837#bib.bib5)\)are conditioned on a natural language prompt, and since the entire rollout follows from it, modifying the prompt induces global changes\. Figure[1](https://arxiv.org/html/2607.08837#S1.F1)illustrates this behavior: given the prompt put the green container on the bottom rack,” the policy picks up the container but fails to place it fully on the rack\. Action noise merely jitters the arm without changing the strategy\. Rephrasing the prompt as put the green container completely on the bottom rack” redirects the policy to the correct contact point and succeeds, without any weight updates\. Prior work on context engineering and prompt optimization has shown that prompt phrasing can significantly shape model behavior\([brown2020language,](https://arxiv.org/html/2607.08837#bib.bib7);[opsahl2024optimizing,](https://arxiv.org/html/2607.08837#bib.bib31);[karnik2024embodied,](https://arxiv.org/html/2607.08837#bib.bib21);[hong2024curiosity,](https://arxiv.org/html/2607.08837#bib.bib16)\), but prompts have not, to our knowledge, been used as an axis for exploration in RL\.

The open question is how to find prompts that induce useful global changes\. A natural answer is to try candidates and keep those that yield higher reward, but with a weak policy that rarely succeeds, reward is too sparse to select on\. We instead refine prompts from the rollouts themselves: a vision\-language model \(VLM\) reasons over the rollout video, diagnoses how the policy responded, and rewrites the prompt to elicit better behavior next time\. This procedure mirrors posterior\-sampling RL at the prompt level\([strens2000bayesian,](https://arxiv.org/html/2607.08837#bib.bib39);[osband2013posterior,](https://arxiv.org/html/2607.08837#bib.bib33);[daniel2018tutorial,](https://arxiv.org/html/2607.08837#bib.bib10);[osband2016deep,](https://arxiv.org/html/2607.08837#bib.bib32)\)\. A VLA defines a family of prompt\-conditioned policies, so a distribution over prompts implicitly induces a distribution over policies, structured by the pretrained language prior\. The VLM acts as an amortized posterior update: it samples plausible prompts from its language prior\([yang2023large,](https://arxiv.org/html/2607.08837#bib.bib52);[zhou2022large,](https://arxiv.org/html/2607.08837#bib.bib56);[soylu2024fine,](https://arxiv.org/html/2607.08837#bib.bib38)\)and refines this distribution by reasoning over observed trajectories\([qwen2025vl,](https://arxiv.org/html/2607.08837#bib.bib47);[openai2024gpt4o,](https://arxiv.org/html/2607.08837#bib.bib17);[gemini2024,](https://arxiv.org/html/2607.08837#bib.bib42)\), without gradient training\. This aligns with evidence that in\-context learning can behave as implicit Bayesian inference\([xie2021explanation,](https://arxiv.org/html/2607.08837#bib.bib49);[wang2023large,](https://arxiv.org/html/2607.08837#bib.bib44);[akyurek2022learning,](https://arxiv.org/html/2607.08837#bib.bib3)\)\. We call the resulting algorithm Prompt\-Driven Exploration \(PDE\): a VLM iteratively updates a prompt distribution from observed trajectories, and each rollout is generated by sampling a prompt from this distribution\.

Our contribution is a simple exploration strategy that enables RL to escape weak initial policies by refining prompts from policy rollouts\. We evaluate PDE on LIBERO\([liu2023libero,](https://arxiv.org/html/2607.08837#bib.bib26)\)and LIBERO\-PRO\([zhou2025liberopro,](https://arxiv.org/html/2607.08837#bib.bib55)\), using VLAs trained on only a fraction of the demonstrations\. In this regime, the policy has no successful rollouts to reinforce, and action\-space perturbations rarely discover success by chance\. Across tasks of varying difficulty, PDE achieves higher success rates with far fewer environment interactions and solves tasks where action\-space exploration fails\. Our analysis shows that VLM updates reliably shift the prompt distribution toward prompts that elicit success, supporting the posterior\-sampling view behind PDE\. To show the generality of PDE beyond VLA control, we further evaluate it on challenging LLM coding tasks, where it also improves sample efficiency\.

![Refer to caption](https://arxiv.org/html/2607.08837v1/x1.png)Figure 1:Left:Given the prompt “put the green container on the bottom rack,” the VLA policy picks up the container but fails to place it fully on the rack, causing it to fall off the edge \(top row\)\. Rephrasing the prompt as “put the green container completely on the bottom rack” leads the same policy to succeed, without any weight updates \(bottom row\)\.Right:RL with our prompt\-driven exploration \(PDE\) improves the success rate from10%10\\%to50%50\\%, whereas PPO with standard action\-noise exploration\([schulman2017ppo,](https://arxiv.org/html/2607.08837#bib.bib36)\)remains near zero\.
## 2Related works

RL post\-training for VLAs\.Fine\-tuning pretrained VLAs under sparse binary rewards is challenging at scale\([tan2025interactive,](https://arxiv.org/html/2607.08837#bib.bib40);[liu2026what,](https://arxiv.org/html/2607.08837#bib.bib27);[li2026simplevlarl,](https://arxiv.org/html/2607.08837#bib.bib24)\)\. Recent work reduces what RL must update: freezing the backbone and training a small module on top, such as an RL token with actor\-critic head\([xu2025rlt,](https://arxiv.org/html/2607.08837#bib.bib50)\)or residual off\-policy actors distilled back into the base policy\([xiao2026selfimproving,](https://arxiv.org/html/2607.08837#bib.bib48)\); or optimizing in the latent space of a pretrained diffusion policy\([wagenmaker2025steering,](https://arxiv.org/html/2607.08837#bib.bib43)\)\. These change*how*the policy is updated; we change*what*it is asked to do\.

Foundation reward models\.A complementary line densifies the reward by training text\-conditioned progress models, either as goal\-reaching value functions\([ma2022vip,](https://arxiv.org/html/2607.08837#bib.bib30);[ma2023liv,](https://arxiv.org/html/2607.08837#bib.bib29)\), from interpolated progress labels on demonstrations\([ma2024vision,](https://arxiv.org/html/2607.08837#bib.bib28);[liang2026robometer,](https://arxiv.org/html/2607.08837#bib.bib25);[zhai2025vision,](https://arxiv.org/html/2607.08837#bib.bib53)\), or from preferences\([wang2024rl,](https://arxiv.org/html/2607.08837#bib.bib45)\)\. These reshape the reward for a given rollout distribution; PDE changes the rollout distribution itself, and the two are compatible\.

Prompt optimization\.LLMs can serve as prompt optimizers that iteratively refine prompts\([yang2023large,](https://arxiv.org/html/2607.08837#bib.bib52);[zhou2022large,](https://arxiv.org/html/2607.08837#bib.bib56)\), and analogous methods exist for VLMs through visually grounded discrete optimization\([du2024ipo,](https://arxiv.org/html/2607.08837#bib.bib12)\)and continuous context embeddings\([zhou2022learning,](https://arxiv.org/html/2607.08837#bib.bib54)\)\. Closest to our setting, CoVer\-VLA\([kwok2026scalingverificationeffectivescaling,](https://arxiv.org/html/2607.08837#bib.bib23)\)uses a trained verifier to optimize VLA instructions at test time\. We instead use prompt diversity as an exploration mechanism during training\.

RL exploration strategies\.Classical alternatives to action noise include count\-based bonuses\([bellemare2016unifying,](https://arxiv.org/html/2607.08837#bib.bib4)\), curiosity\-driven intrinsic motivation\([pathak2017curiosity,](https://arxiv.org/html/2607.08837#bib.bib34)\), entropy regularization\([haarnoja2018soft,](https://arxiv.org/html/2607.08837#bib.bib15)\), and parameter\-space noise\([plappert2018parameter,](https://arxiv.org/html/2607.08837#bib.bib35)\)\. These see limited adoption in VLA post\-training, where auxiliary density or forward models over high\-dimensional visual inputs are costly to train alongside billion\-parameter policies\. PDE explores in the space of task specifications instead, leveraging the language conditioning already present in pretrained VLAs\.

## 3Preliminaries

We fine\-tune a vision\-language\-action \(VLA\) policy\([black2024pi0,](https://arxiv.org/html/2607.08837#bib.bib5);[kim2024openvla,](https://arxiv.org/html/2607.08837#bib.bib22)\)with reinforcement learning \(RL\)\([kaelbling1996reinforcement,](https://arxiv.org/html/2607.08837#bib.bib20)\)in a multi\-task setting\. The standard RL pipeline trains a policyπθ\\pi\_\{\\theta\}on a distribution of hand\-crafted tasks\. Each taskggis paired with a canonical promptpgp\_\{g\}\. At the start of each rollout, a taskggis sampled and a promptppis given toπθ\\pi\_\{\\theta\}\. At each timesteptt, the policy receives the environmental observationoto\_\{t\}\(e\.g\., text, RGB images, etc\) and outputs an actionata\_\{t\}until horizonTT\. We denote the resulting rollout byτp=\(o0,a0,…,oT−1,aT−1,oT\)\\tau\_\{p\}=\(o\_\{0\},a\_\{0\},\\dots,o\_\{T\-1\},a\_\{T\-1\},o\_\{T\}\)\. The policy receives a terminal reward

R​\(τp,g\)=\{1,if​τp​completes task​g,0,otherwise\.R\(\\tau\_\{p\},g\)=\\begin\{cases\}1,&\\text\{if \}\\tau\_\{p\}\\text\{ completes task \}g,\\\\ 0,&\\text\{otherwise\}\.\\end\{cases\}
Reward depends on the task, not the prompt\.The rewardRRis a function of the taskggand rolloutτp\\tau\_\{p\}, but not of the promptpp\. For example, ifggis “open the drawer,” any rollout that opens the drawer receives reward11, whether the prompt is the canonical promptpg=p\_\{g\}=“open the drawer” or an alternative such as “pull the handle toward you\.” Thus, a prompt need not faithfully describeggto be useful\. It only needs to conditionπθ\\pi\_\{\\theta\}toward a rollout that completesgg\. Section[4](https://arxiv.org/html/2607.08837#S4)shows how our method exploits this flexibility for exploration\. The RL objective is to maximize expected success:

J​\(πθ\)=𝔼g∼p𝒢,p∼ρ\(⋅∣g\),τp∼πθ\(⋅∣p\)​\[R​\(τp,g\)\]\.\\displaystyle J\(\\pi\_\{\\theta\}\)=\\mathbb\{E\}\_\{\\,g\\sim p\_\{\\mathcal\{G\}\},\\;p\\sim\\rho\(\\cdot\\mid g\),\\;\\tau\_\{p\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid p\)\}\\\!\\left\[\\,R\(\\tau\_\{p\},g\)\\,\\right\]\.\(1\)

## 4Method: Prompt\-Driven Exploration \(PDE\)

Standard RL explores by sampling actions stochastically fromπθ\(⋅∣ot,pg\)\\pi\_\{\\theta\}\(\\cdot\\mid o\_\{t\},p\_\{g\}\)around the canonical promptpgp\_\{g\}, perturbing behavior locally at each timestep\. PDE introduces a complementary axis of exploration: the promptppitself\. Recall from Section[3](https://arxiv.org/html/2607.08837#S3)that the rewardR​\(τp,g\)R\(\\tau\_\{p\},g\)depends only on the taskggand the resulting trajectory, not on the prompt that produced it\. Any prompt that drivesπθ\\pi\_\{\\theta\}to completeggtherefore yields a successful rollout, regardless of whether its wording matchespgp\_\{g\}\. Becauseπθ\\pi\_\{\\theta\}conditions onppat every timestep, replacingpgp\_\{g\}with an alternative shifts the policy’s action distribution globally, exploring trajectories that step\-wise action noise may not reach\.

Overview\.The challenge in exploration in the prompt space is selecting informative prompts: ones whose rollouts reveal how the policy responds and suggest how to refine the prompt next time\. PDE addresses this in two stages\. First, we search for useful prompts by rolling out the initial policy under candidates drawn from a broad distributionρ\\rhoand using a VLM to analyze the resulting trajectories—diagnosing how the policy responded to each prompt and rewriting candidates accordingly—to updateρ\\rhotoward prompts that better elicit the intended behavior \(Update Prompt Posterior, Fig\.[2](https://arxiv.org/html/2607.08837#S4.F2)\)\. Onceρ\\rhohas concentrated on such prompts, we fine\-tuneπθ\\pi\_\{\\theta\}on rollouts sampled underp∼ρp\\sim\\rhovia RL \(RL Update, Fig\.[2](https://arxiv.org/html/2607.08837#S4.F2)\)\. Algorithm[1](https://arxiv.org/html/2607.08837#alg1)summarizes the procedure\.

![Refer to caption](https://arxiv.org/html/2607.08837v1/x2.png)Figure 2:Overview of our method\. A VLM defines a prompt samplerρ\(⋅∣g,ℋ\)\\rho\(\\cdot\\mid g,\\mathcal\{H\}\)over candidate prompts for taskgg, given its canonical promptpgp\_\{g\}and previous rollout feedback\. Prompts sampled fromρ\\rhoare executed by the VLA policyπθ\\pi\_\{\\theta\}\. The resulting trajectories and rewards are added to the history, allowingρ\\rhoto propose prompts that better elicit rollouts completing the taskgg\(green arrow, “Update Prompt Posterior”\)\. Finally,πθ\\pi\_\{\\theta\}is fine\-tuned with RL on the collected rollouts \(blue arrow, “RL Update”\)\.### 4\.1Posterior Sampling over Prompts

From PSRL to prompt\-space exploration\.Posterior sampling for reinforcement learning \(PSRL\)\([strens2000bayesian,](https://arxiv.org/html/2607.08837#bib.bib39);[osband2013posterior,](https://arxiv.org/html/2607.08837#bib.bib33)\)maintains a posterior over policies\. At iterationii, PSRL samples one policy, executes it for one episode—from the initial state to termination or horizonTT—and updates the posterior using the observed trajectory and reward\. This produces consistent exploration: all actions in the episode are chosen by the same sampled policy, rather than being perturbed independently at each timestep\. The challenge is scalability\. For modern VLAs, the policy is a large neural networkπθ\\pi\_\{\\theta\}, making a Bayesian posterior over parameters intractable to maintain and sample from\.

Prompts induce a tractable policy posterior\.PDE recovers the core idea of PSRL through the language interface of a pretrained VLA\. For fixed VLA parametersθ\\theta, each promptppinduces a policy

πp\(⋅∣o\):=πθ\(⋅∣o,p\)\.\\pi\_\{p\}\(\\cdot\\mid o\):=\\pi\_\{\\theta\}\(\\cdot\\mid o,p\)\.Thus, a distribution over prompts induces a distribution over policies\. In this induced policy class, drawing a policy hypothesis amounts to choosing a prompt\. Because the prompt is fixed throughout the rollout, it can shift the behavior attempted in the episode while preserving temporal coherence\. The uncertainty is no longer over independent low\-level action perturbations, but over which prompt will elicit successful behavior from the VLA\.

A VLM as an implicit prompt posterior\.PDE represents this prompt distribution with a VLM prompt samplerρ\\rho\. Let

ℋi=\{\(gj,pj,τj,R​\(τj,gj\)\)\}j<i\\mathcal\{H\}\_\{i\}=\\\{\(g\_\{j\},p\_\{j\},\\tau\_\{j\},R\(\\tau\_\{j\},g\_\{j\}\)\)\\\}\_\{j<i\}denote the interaction history before iterationii\. PDE samplespi∼ρ\(⋅∣gi,ℋi\)p\_\{i\}\\sim\\rho\(\\cdot\\mid g\_\{i\},\\mathcal\{H\}\_\{i\}\), whereρ\\rhoconditions on the task, previous prompts, trajectories, and rewards\. The prompt posterior is updated implicitly by appending each new rollout toℋi\\mathcal\{H\}\_\{i\}, as shown in Algorithm[1](https://arxiv.org/html/2607.08837#alg1)\.

Unlike a classical Bayesian posterior,ρ\(⋅∣g,ℋi\)\\rho\(\\cdot\\mid g,\\mathcal\{H\}\_\{i\}\)has no explicit density over natural language\. It is instead queried as an implicit sampler: given the task and rollout feedback, it proposes the next prompt to try\. This lets PDE use the VLM’s language and vision priors to propose coherent, task\-relevant prompts, while trajectory feedback adapts those proposals to the current VLA\.

### 4\.2Combining PDE with Policy Optimization

We implement the RL update in Algorithm[1](https://arxiv.org/html/2607.08837#alg1)with Proximal Policy Optimization \(PPO\)\([schulman2017ppo,](https://arxiv.org/html/2607.08837#bib.bib36)\)\. For a rolloutτ=\{\(ot,at\)\}t=0T−1\\tau=\\\{\(o\_\{t\},a\_\{t\}\)\\\}\_\{t=0\}^\{T\-1\}collected under promptpp, PPO minimizes

ℒPPO​\(θ\)\\displaystyle\\mathcal\{L\}^\{\\mathrm\{PPO\}\}\(\\theta\)=−𝔼t​\[min⁡\(wt​\(θ\)​A^t,clip⁡\(wt​\(θ\),1−ϵ,1\+ϵ\)​A^t\)\],wt​\(θ\)=πθ​\(at∣ot,p\)πold​\(at∣ot,p\),\\displaystyle=\-\\mathbb\{E\}\_\{t\}\\\!\\left\[\\min\\\!\\left\(w\_\{t\}\(\\theta\)\\hat\{A\}\_\{t\},\\operatorname\{clip\}\(w\_\{t\}\(\\theta\),1\-\\epsilon,1\+\\epsilon\)\\hat\{A\}\_\{t\}\\right\)\\right\],w\_\{t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(a\_\{t\}\\mid o\_\{t\},p\)\}\{\\pi\_\{\\mathrm\{old\}\}\(a\_\{t\}\\mid o\_\{t\},p\)\},\(2\)whereA^t\\hat\{A\}\_\{t\}is the advantage estimate\. The main issue is that PDE may collect data under exploratory promptsp∼ρ\(⋅∣g,ℋ\)p\\sim\\rho\(\\cdot\\mid g,\\mathcal\{H\}\), while evaluation uses the canonical promptpgp\_\{g\}\. We use two modifications to make exploratory rollouts improve the canonical\-prompt policy\.

Mixture sampling\.During data collection, we samplep∼αδpg\+\(1−α\)ρ\(⋅∣g,ℋ\)p\\sim\\alpha\\delta\_\{p\_\{g\}\}\+\(1\-\\alpha\)\\rho\(\\cdot\\mid g,\\mathcal\{H\}\)\. Prompts fromρ\\rhoencourage exploration, while prompts frompgp\_\{g\}keep training aligned with evaluation\. We increaseα\\alphaas success underpgp\_\{g\}improves, shifting training toward the canonical promptpgp\_\{g\}\.

Mixed backpropagation\.A rollout collected under an exploratory promptppdirectly trainsπθ\(⋅∣o,p\)\\pi\_\{\\theta\}\(\\cdot\\mid o,p\), but may not improveπθ\(⋅∣o,pg\)\\pi\_\{\\theta\}\(\\cdot\\mid o,p\_\{g\}\)\. To couple the two prompts, we replace the current\-policy log\-probability in the PPO ratio with the average log\-probability under both prompts:

ℓt​\(θ\)\\displaystyle\\ell\_\{t\}\(\\theta\)=12​log⁡πθ​\(at∣ot,pg\)\+12​log⁡πθ​\(at∣ot,p\),\\displaystyle=\\frac\{1\}\{2\}\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid o\_\{t\},p\_\{g\}\)\+\\frac\{1\}\{2\}\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid o\_\{t\},p\),\(3\)wt​\(θ\)\\displaystyle w\_\{t\}\(\\theta\)=exp⁡\(ℓt​\(θ\)−log⁡πold​\(at∣ot,p\)\)\.\\displaystyle=\\exp\\\!\\left\(\\ell\_\{t\}\(\\theta\)\-\\log\\pi\_\{\\mathrm\{old\}\}\(a\_\{t\}\\mid o\_\{t\},p\)\\right\)\.We then use this modifiedwt​\(θ\)w\_\{t\}\(\\theta\)in the PPO loss\. Thus, an action receives a strong update only when it is likely under both the exploratory promptppand the canonical promptpgp\_\{g\}\. The mixture schedule, trajectory summarization protocol, and other hyperparameters are given in Appendix[A](https://arxiv.org/html/2607.08837#A1)\.

Algorithm 1Prompt\-Driven Exploration \(PDE\)\.1:VLA policy

πθ\\pi\_\{\\theta\}; task distribution

𝒢\\mathcal\{G\}; VLM prompt sampler

ρ\\rho; mixture schedule

\{αi\}i=1N\\\{\\alpha\_\{i\}\\\}\_\{i=1\}^\{N\}; iterations

NN
2:Initialize history

ℋ1←∅\\mathcal\{H\}\_\{1\}\\leftarrow\\emptyset
3:for

i=1,…,Ni=1,\\dots,Ndo

4:Sample a task

gi∼𝒢g\_\{i\}\\sim\\mathcal\{G\}with canonical prompt

pgip\_\{g\_\{i\}\}
5:Sample a prompt

pi∼αiδpgi\+\(1−αi\)ρ\(⋅∣gi,ℋi\)p\_\{i\}\\sim\\alpha\_\{i\}\\delta\_\{p\_\{g\_\{i\}\}\}\+\(1\-\\alpha\_\{i\}\)\\rho\(\\cdot\\mid g\_\{i\},\\mathcal\{H\}\_\{i\}\)
6:Roll out

πθ\(⋅∣o,pi\)\\pi\_\{\\theta\}\(\\cdot\\mid o,p\_\{i\}\); observe

τi\\tau\_\{i\}and

R​\(τi,gi\)R\(\\tau\_\{i\},g\_\{i\}\)
7:Update history

ℋi\+1←ℋi∪\{\(gi,pi,τi,R​\(τi,gi\)\)\}\\mathcal\{H\}\_\{i\+1\}\\leftarrow\\mathcal\{H\}\_\{i\}\\cup\\\{\(g\_\{i\},p\_\{i\},\\tau\_\{i\},R\(\\tau\_\{i\},g\_\{i\}\)\)\\\}
8:Update

θ\\thetawith PPO on

τi\\tau\_\{i\}using mixed backpropagation between

pip\_\{i\}and

pgip\_\{g\_\{i\}\}
9:endfor

10:return

πθ\\pi\_\{\\theta\}

## 5Experiments

We structure the experiments around three questions: \(1\) we test whether PDE can find prompts that make a weak policy achieve non\-zero reward and whether these prompts induce global behavior changes rather than local action noise \(Section[5\.2](https://arxiv.org/html/2607.08837#S5.SS2)\), \(2\) we evaluate whether PDE enables RL fine\-tuning from weak VLA initializations with near\-zero success rates on LIBERO\-PRO and ManiSkill \(Sections[5\.3](https://arxiv.org/html/2607.08837#S5.SS3)and[5\.4](https://arxiv.org/html/2607.08837#S5.SS4)\), with ablations in Section[B\.3](https://arxiv.org/html/2607.08837#A2.SS3), and \(3\) we demonstrate that PDE also applies beyond robotics by using it on challenging LLM coding tasks\.

### 5\.1Setup

Environment and Metric\.We evaluate on LIBERO\([liu2023libero,](https://arxiv.org/html/2607.08837#bib.bib26)\)and ManiSkill\([tao2024maniskill3,](https://arxiv.org/html/2607.08837#bib.bib41)\)\. We reportsuccess\_once, the binary episode\-level success metric used by the RLinf evaluation suite111[https://rlinf\.readthedocs\.io/](https://rlinf.readthedocs.io/), averaged over250250evaluation environments per task\. We start RL training with an initial policy from a weak Pi05 SFT checkpoint\([chen2025pirl,](https://arxiv.org/html/2607.08837#bib.bib9)\)trained with a limited number of demonstrations from four LIBERO task suites, excluding LIBERO\-90, unless specified\.

Baselines\.We compare PDE against six baselines covering three standard ways to improve exploration or learning in PPO: adding noise, adding auxiliary exploration bonuses, and using dense reward\-model feedback\. All methods start from the same supervised fine\-tuning \(SFT\) checkpoint, use the canonical task promptpgp\_\{g\}for training, and are trained with the same PPO\([schulman2017ppo,](https://arxiv.org/html/2607.08837#bib.bib36)\)optimizer settings and rollout budget\. Checkpoint details and implementation hyperparameters are provided in the corresponding result sections and Appendix[B](https://arxiv.org/html/2607.08837#A2)\.

- •Noise:Action Noiseis standard PPO with stochastic action sampling\.Parameter Noiseapplies Gaussian perturbations\([plappert2018parameter,](https://arxiv.org/html/2607.08837#bib.bib35)\)to the policy weights at the beginning of each rollout\.
- •Bonus: Random Network Distillation \(RND\)\([burda2018exploration,](https://arxiv.org/html/2607.08837#bib.bib8)\)adds intrinsic reward, computed as the prediction error of a learned predictor against a fixed random target network\.
- •Dense reward:Robometer\([liang2026robometer,](https://arxiv.org/html/2607.08837#bib.bib25)\)provides dense task\-progress rewards from egocentric images and the task description\.

### 5\.2Illustrative Experiment: Close the Microwave

Setup\.We first examine a single LIBERO\-90 task with a visually clear failure mode:g=g=*“close the microwave”*\(Figure[3](https://arxiv.org/html/2607.08837#S5.F3), left\)\. The canonical prompt ispg=gp\_\{g\}=g\. We start with an initial policy from a weak Pi05 SFT checkpoint\([chen2025pirl,](https://arxiv.org/html/2607.08837#bib.bib9)\)trained with a limited number of demonstrations from four LIBERO task suites, excluding LIBERO\-90\. As shown in Figure[3](https://arxiv.org/html/2607.08837#S5.F3)\(right\), PPO with standard action\-space exploration remains near zero and reaches only∼10%\\sim\\\!10\\%success by step100100, indicating that local action perturbations are inefficient for escaping this initial policy\.

![Refer to caption](https://arxiv.org/html/2607.08837v1/x3.png)Figure 3:Left:The SFT checkpoint fails under the original prompt “close the microwave”—the robot grasps the mug instead of pushing the door \(top row\)\. Our prompt optimization discovers an alternative prompt, “push on the microwave door until it shuts”, that redirects the policy to the correct contact point and action, achieving nonzero success without any weight updates \(bottom row\)\.Right:RL with our prompt driven exploration \(PDE\) bootstraps from0%0\\%to∼98%\\sim\\\!98\\%success on the*original*prompt, while standard action\-noise exploration remains low success rate throughout 100\-step training\.Why does the initial policy fail?The checkpoint exhibits a consistent failure mode across rollouts \(Figure[3](https://arxiv.org/html/2607.08837#S5.F3), top\): the arm grasps the yellow\-handled mug, moves it toward the microwave, and stalls without contacting the microwave door\. This behavior is consistent with a visually similar training task,*“put the yellow and white mug in the microwave and close it\.”*The shared scene and the word*“microwave”*appear to bias the policy toward a pick\-and\-place trajectory, although the target task only requires closing the door\. Thus, the canonical prompt elicits the wrong motor program\. Action noise perturbs this behavior only locally, so most sampled rollouts still manipulate the mug and fail\.

Can PDE discover successful prompts?We next run the prompt\-posterior update stage of PDE on the fixed initial policy, before any policy weight updates\. Within8585cumulative rollouts, PDE discovers1010–1212unique prompts with nonzero success across runs \(Figure[10](https://arxiv.org/html/2607.08837#A2.F10)\)\. The successful prompts fall into three interpretable categories:

- •Explicit contact and action: prompts such as*“push the microwave door closed,”**“push the microwave door until it clicks,”*and*“swing the microwave door shut”*specify the desired physical interaction, shifting the policy from pick\-and\-place behavior to push\-and\-close behavior\.
- •Spatial or generic reference: prompts such as*“close the black appliance door on the left,”**“close the kitchen appliance door,”*and*“shut the open appliance door”*identify the target by appearance or location rather than by the object name*“microwave,”*reducing reliance on the misleading object\-name cue\.
- •Distractor exclusion: prompts such as*“do not close the cabinet – close microwave”*and*“first move to the microwave, then close the door”*explicitly redirect the policy away from distractors and toward the intended contact point\.

These prompts show that the VLM can use rollout feedback to infer how the VLA interprets previous prompts and propose alternatives that better elicit the desired behavior\. However, prompt optimization alone is insufficient: the best discovered prompts remain below40%40\\%success\. Language can reveal a more useful behavior mode, but RL is needed to turn it into a successful policy\.

Can PDE bootstrap RL from zero success?We then fine\-tune the policy with PPO using PDE rollouts\. Because PDE samples from the learned prompt distributionρ\(⋅∣g,ℋ\)\\rho\(\\cdot\\mid g,\\mathcal\{H\}\), training receives nonzero\-reward trajectories even though the canonical promptpgp\_\{g\}initially has0%0\\%success\. As shown in Figure[1](https://arxiv.org/html/2607.08837#S1.F1)\(right\), PDE starts at∼20%\\sim\\\!20\\%success, reflecting the successful exploratory prompts, and reaches near100%100\\%success by step100100under the canonical promptpgp\_\{g\}\. In contrast, Action Noise barely escapes zero, RND fails to make progress, and Parameter Noise improves only partially with high variance\.

Summary\.This case study illustrates the core mechanism of PDE: prompt\-space exploration discovers globally different rollouts that action\-space perturbations rarely reach, and PPO transfers these successful behaviors back to the policy under the canonical promptspgp\_\{g\}\.

### 5\.3Benchmark Results on LIBERO\-PRO

![Refer to caption](https://arxiv.org/html/2607.08837v1/x4.png)Figure 4:Aggregated training curves grouped by task difficulty based on initial success rate\.Left:Hard tasks \(initial Success Rate<0\.1<0\.1, 47 tasks\)\.Middle:Medium tasks \(0\.1≤0\.1\\leqinitial Success Rate<0\.8<0\.8, 35 tasks\)\.Right:Easy tasks \(initial Success Rate≥0\.8\\geq 0\.8, 38 tasks\)\. Shaded regions indicate standard error of the mean across tasks\. The larger standard error on hard tasks reflects high variance between tasks that remain near zero throughout training and those that eventually improve\.![Refer to caption](https://arxiv.org/html/2607.08837v1/x5.png)Figure 5:Aggregated GR00T training curves grouped by task difficulty based on initial success rate\.Left:Hard tasks \(initial Success Rate=0=0,49tasks\)\.Middle:Medium tasks \(0<0<initial Success Rate≤0\.4\\leq 0\.4,31tasks\)\.Right:Easy tasks \(initial Success Rate\>0\.4\>0\.4,40tasks\)\.![Refer to caption](https://arxiv.org/html/2607.08837v1/x6.png)Figure 6:Aggregated Pi0 training curves grouped by task difficulty based on initial success rate\.Left:Hard tasks \(initial Success Rate=0=0,49tasks\)\.Middle:Medium tasks \(0<0<initial Success Rate≤0\.4\\leq 0\.4,24tasks\)\.Right:Easy tasks \(initial Success Rate\>0\.4\>0\.4,27tasks\)\.Setup\.We evaluate PDE on LIBERO\-PRO\([zhou2025liberopro,](https://arxiv.org/html/2607.08837#bib.bib55)\), a benchmark that modifies LIBERO environments and tasks to create challenging generalization settings where current VLA models often struggle\. This makes LIBERO\-PRO well suited for testing whether an exploration method can fine\-tune a policy from a weak initialization\. We evaluate on120120tasks spanning four LIBERO task suites\. The benchmark includes many tasks where the initial policy has near\-zero success, while also covering easier tasks where action\-space exploration can make progress\. This range lets us compare exploration methods across different levels of initial policy competence\. We exclude*language perturbation*because PDE also modifies prompts, which would confound the comparison, and*environment perturbation*because its BDDL files are not publicly released\. We train separate policies for each benchmark group; details are provided in Appendix[B](https://arxiv.org/html/2607.08837#A2)\.

Does PDE improve over standard action noise?Since our method is designed primarily to help when the base policy provides little or no reward signal, we group the 120 tasks into three difficulty tiers by the initial policy’s success rate to test whether the gains concentrate on the hardest tasks as predicted:*hard*\(init SR<10%<10\\%; 47 tasks\),*medium*\(1010–80%80\\%; 36 tasks\), and*easy*\(≥80%\\geq 80\\%; 37 tasks\)\. Figure[4](https://arxiv.org/html/2607.08837#S5.F4)and Table[10](https://arxiv.org/html/2607.08837#A2.T10)report success rates aggregated by tier\. Our method outperforms all the baselines across every difficulty tier, with the largest gap on the*hard*tier \(60%60\\%relative improvement\), where many tasks have zero initial success and action\-noise methods receive no learning signal\. This is consistent with the microwave case study \(Section[5\.2](https://arxiv.org/html/2607.08837#S5.SS2)\): PDE discovers language groundings that unlock nonzero performance on tasks where the initial policy has overfit to a wrong motor program, precisely the regime in which action\-space perturbations are ineffective\.

Does PDE generalize across base models?We repeat the difficulty\-aggregated benchmark on GR00T and Pi0 \(Table[11](https://arxiv.org/html/2607.08837#A2.T11), Figures[5](https://arxiv.org/html/2607.08837#S5.F5)–[6](https://arxiv.org/html/2607.08837#S5.F6)\)\. PDE consistently improves over Action Noise across both backbones\. On GR00T, PDE improves learning across all difficulty groups, with the largest gains on hard and medium tasks\. On Pi0, PDE also improves hard and medium tasks, while the easy\-task curves are closer because both methods already start from high success\. For Pi0 hard tasks, the benefit is most visible in the learning curve: PDE maintains a consistent advantage throughout training, although the absolute success rates remain low, making aggregate table gains appear smaller\. We use the same PDE procedure for both models, but choose the backpropagation variant separately\.

### 5\.4Benchmark Results on ManiSkill

Setup\.Following\([liu2026what,](https://arxiv.org/html/2607.08837#bib.bib27)\), we evaluate generalization on ManiSkillPutOnPlateInScene25\([tao2024maniskill3,](https://arxiv.org/html/2607.08837#bib.bib41)\), a tabletop pick\-and\-place benchmark where the policy must place one of2525objects onto a receptacle in a kitchen scene\. The Pi0\.5 SFT checkpoint is trained only on the in\-distribution split, which contains1616objects and16,38416\{,\}384episodes synthesized with MPLib222[https://github\.com/haosulab/mplib](https://github.com/haosulab/mplib)\. We fine\-tune one policy per OOD variant with PPO, evaluating1212variants across three generalization scenarios:

- •Vision\(55variants\): novel backgrounds and textures induced by image overlays of varying intensity\.
- •Semantics\(44variants\): unseen objects, varied instructions, and distractors such as extra objects\.
- •Execution\(33variants\): varied initial states, unseen robot poses, and dynamic disturbances\.

Does PDE improve over action\-space exploration?We aggregate results in two ways: by initial difficulty, defined by the initial policy’s success rate \(*hard*:<25%<25\\%,*medium*:2525–50%50\\%,*easy*:≥50%\\geq 50\\%\), and by generalization scenario\. Table[1](https://arxiv.org/html/2607.08837#S5.T1)reports both views\. PDE outperforms Action Noise in every difficulty tier and generalization scenario\. Although Action Noise already reaches nontrivial success on ManiSkill hard tasks, PDE consistently improves performance across the full benchmark, showing that prompt\-space exploration remains useful beyond settings where action\-space exploration completely fails\.

Table 1:Aggregated success rate \(%\) on ManiSkill, by difficulty and by generalization axis\.DifficultyGeneralizationHardMediumEasyVisualSemanticExecutionAllInitial policy12\.17±0\.8712\.17\{\\scriptstyle\\pm 0\.87\}35\.16±1\.0835\.16\{\\scriptstyle\\pm 1\.08\}64\.96±1\.4864\.96\{\\scriptstyle\\pm 1\.48\}48\.16±2\.5948\.16\{\\scriptstyle\\pm 2\.59\}29\.58±3\.1729\.58\{\\scriptstyle\\pm 3\.17\}28\.31±2\.5428\.31\{\\scriptstyle\\pm 2\.54\}37\.59±1\.7637\.59\{\\scriptstyle\\pm 1\.76\}Action Noise51\.65±2\.5651\.65\{\\scriptstyle\\pm 2\.56\}63\.39±2\.5763\.39\{\\scriptstyle\\pm 2\.57\}71\.73±2\.9571\.73\{\\scriptstyle\\pm 2\.95\}72\.12±2\.2472\.12\{\\scriptstyle\\pm 2\.24\}54\.64±3\.2154\.64\{\\scriptstyle\\pm 3\.21\}53\.96±2\.7353\.96\{\\scriptstyle\\pm 2\.73\}62\.31±1\.6762\.31\{\\scriptstyle\\pm 1\.67\}\\rowcolorourskyblue PDE \(Ours\)58\.77±2\.73\\mathbf\{58\.77\}\{\\scriptstyle\\pm 2\.73\}76\.68±2\.58\\mathbf\{76\.68\}\{\\scriptstyle\\pm 2\.58\}84\.33±2\.14\\mathbf\{84\.33\}\{\\scriptstyle\\pm 2\.14\}85\.12±1\.86\\mathbf\{85\.12\}\{\\scriptstyle\\pm 1\.86\}63\.65±3\.29\\mathbf\{63\.65\}\{\\scriptstyle\\pm 3\.29\}63\.75±2\.70\\mathbf\{63\.75\}\{\\scriptstyle\\pm 2\.70\}73\.32±1\.64\\mathbf\{73\.32\}\{\\scriptstyle\\pm 1\.64\}

### 5\.5Analysis

Table 2:Success rate \(%\) on object\-task perturbation for tasks 0, 5, 2 \(Regimes 1–3\) at steps 0, 30, 120\.Table[2](https://arxiv.org/html/2607.08837#S5.T2)shows three regimes where PDE improves over standard exploration and dense\-reward baselines:

Faster learning when action noise can eventually succeed\.In Regime 1, Action Noise eventually reaches high success \(93\.5%93\.5\\%at step120120\), but PDE learns much faster: it starts with nonzero success from exploratory prompts \(24\.4%24\.4\\%at step0\) and reaches99\.0%99\.0\\%by step3030, compared with65\.2%65\.2\\%for Action Noise\.

Bootstrapping when action noise fails\.In Regime 2, Action Noise, Robometer, and RND remain at0%0\\%throughout training\. PDE, however, obtains nonzero success from prompt\-space exploration \(4\.3%4\.3\\%at step0\) and improves to77\.7%77\.7\\%by step120120, showing that alternative prompts can provide the successful rollouts needed for RL to begin improving\.

Transfer when prompt discovery alone is insufficient\.In Regime 3, PDE has0%0\\%success at steps0and3030, indicating that prompt discovery alone does not immediately solve the task\. Nevertheless, after RL training, PDE reaches81\.8%81\.8\\%success by step120120, while Action Noise, Robometer, and RND remain at0%0\\%\. This suggests that successful exploratory rollouts from related tasks can train reusable skills that transfer to harder tasks\.

### 5\.6Real\-World Manipulation

Setup\.The simulation results in Section[5\.3](https://arxiv.org/html/2607.08837#S5.SS3)show that PDE is most useful when the base policy receives sparse or no reward signal\. This is precisely the regime in which real\-world robot learning is most costly: each rollout requires physical execution time, and collecting many zero\-reward trajectories is prohibitively inefficient\. We therefore evaluate whether PDE provides the same sample\-efficiency advantage on a physical manipulation platform\.

![Refer to caption](https://arxiv.org/html/2607.08837v1/x7.png)Figure 7:Real\-world experiment setup\.Our hardware setup consists of a Franka FR3 arm, a set of manipulable tabletop objects, two drawers, and a two\-level rack, as shown in Figure[7](https://arxiv.org/html/2607.08837#S5.F7)\. We first perform supervised fine\-tuning \(SFT\) ofπ0\.5\\pi\_\{0\.5\}on 10 language\-conditioned manipulation tasks that cover a range of difficulty and horizon lengths, from short\-horizon pick\-and\-place tasks to longer multi\-stage tasks involving drawers and spatial constraints\. Starting from the SFT checkpoint, we then continue training with either PPO using action noise or PPO augmented with PDE\. Additional implementation details are provided in Appendix[B\.2](https://arxiv.org/html/2607.08837#A2.SS2)\.

Results\.Figure[8](https://arxiv.org/html/2607.08837#S5.F8)shows that PDE substantially improves real\-world sample efficiency\. Averaged over three real\-world tasks, PPO with action noise improves only modestly as more rollouts are collected, increasing from roughly 13% success before online training to about 20% after 128 rollouts\. In contrast, PPO\+PDE improves much faster, reaching about 35% success after 64 rollouts and about 45% after 128 rollouts\. Thus, after the same number of real\-world rollouts—approximately one hour of robot data—PPO\+PDE achieves more than twice the success rate of PPO with action noise\.

The qualitative examples in Figure[8\(b\)](https://arxiv.org/html/2607.08837#S5.F8.sf2)help explain this improvement\. The original task prompts often under\-specify the intended behavior, which leads to systematic failures\. For example, the instruction “put the green container in the bowl” can cause the policy to select the wrong target object when multiple relevant objects are present\. PDE resolves this ambiguity by producing a more specific instruction, such as “put the green container in the green bowl,” which makes the target relation explicit\. Similarly, the instruction “open the top drawer and put the tape inside” can cause the robot to collide with the drawer because the prompt does not sufficiently specify the required spatial motion\. The optimized prompt adds the missing spatial constraint, encouraging the robot to move higher before completing the placement\.

![Refer to caption](https://arxiv.org/html/2607.08837v1/x8.png)\(a\)
![Refer to caption](https://arxiv.org/html/2607.08837v1/x9.png)\(b\)

Figure 8:Real\-world manipulation results\.\(a\)Average success rate over three real\-world tasks\. PPO\+PDE improves substantially faster than PPO with action noise, achieving more than twice the success rate after 128 rollouts\.\(b\)Failure modes of the original prompts and corresponding PDE\-optimized prompts\. The original prompts under\-specify important task details, such as the target object or the required spatial motion, while the optimized prompts make these details explicit and lead to successful execution\.
### 5\.7Applications to Language Model Tasks

We also show the generality of PDE beyond robotics by applying it to language\-model RL tasks: competitive programming on LiveCodeBench\([jain2025livecodebench,](https://arxiv.org/html/2607.08837#bib.bib19)\)and mathematical reasoning on AIME 2026\([dekoninck2026matharena,](https://arxiv.org/html/2607.08837#bib.bib11)\)\. In both settings, each problemgghas a canonical promptpgp\_\{g\}, PDE samples an alternative promptp∼ρ\(⋅∣g,ℋ\)p\\sim\\rho\(\\cdot\\mid g,\\mathcal\{H\}\)during training, and evaluation uses onlypgp\_\{g\}\. Full details and learning curves are provided in Appendix[C](https://arxiv.org/html/2607.08837#A3)\.

LiveCodeBench\.We evaluate PDE with RLOO\([ahmadian2024back,](https://arxiv.org/html/2607.08837#bib.bib2)\)using600600training problems and held\-out evaluation under canonical prompts\. PDE improves early sample efficiency: PDE\+RLOO reaches50%50\\%held\-out accuracy by the fourth training step, while RLOO reaches this level at the eighth step\. PDE\+RLOO also achieves larger early gains, including42\.4%42\.4\\%vs\.29\.9%29\.9\\%at step11,53\.1%53\.1\\%vs\.36\.0%36\.0\\%at step33, and54\.6%54\.6\\%vs\.41\.3%41\.3\\%at step44\.

AIME 2026\.We fine\-tune Qwen3\-4B\([yang2025qwen3,](https://arxiv.org/html/2607.08837#bib.bib51)\)with GRPO\([shao2024deepseekmath,](https://arxiv.org/html/2607.08837#bib.bib37)\)on all3030AIME 2026 problems\. PDE matches GRPO in final accuracy while modestly improving early learning: after the first iteration, GRPO\+PDE reaches48\.9%48\.9\\%accuracy compared to45\.8%45\.8\\%for GRPO, and both methods reach53\.3%53\.3\\%after three iterations\.

Together, these results show that prompt\-space exploration is not specific to VLA control: it can also expose useful solution modes in language\-model RL and improve early training efficiency under the same optimization budget\.

## 6Conclusion

We introduced Prompt\-Driven Exploration \(PDE\), which performs posterior sampling over prompts to explore globally different behaviors\. By updating a prompt distribution from rollout feedback, PDE improves sample efficiency on LIBERO and LIBERO\-PRO, solves tasks where action\-space exploration fails, and extends to language\-model RL\. More broadly, PDE shows that for foundation\-model policies, effective exploration can come from the context that conditions the policy, not only from perturbing its actions\.

## References

- \[1\]J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat, et al\.Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774, 2023\.
- \[2\]A\. Ahmadian, C\. Cremer, M\. Gallé, M\. Fadaee, J\. Kreutzer, O\. Pietquin, A\. Üstün, and S\. Hooker\.Back to basics: Revisiting reinforce\-style optimization for learning from human feedback in llms\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), pages 12248–12267, 2024\.
- \[3\]E\. Akyürek, D\. Schuurmans, J\. Andreas, T\. Ma, and D\. Zhou\.What learning algorithm is in\-context learning? investigations with linear models\.arXiv preprint arXiv:2211\.15661, 2022\.
- \[4\]M\. Bellemare, S\. Srinivasan, G\. Ostrovski, T\. Schaul, D\. Saxton, and R\. Munos\.Unifying count\-based exploration and intrinsic motivation\.Advances in neural information processing systems, 29, 2016\.
- \[5\]K\. Black, N\. Brown, D\. Driess, A\. Esmail, M\. Equi, C\. Finn, N\. Fusai, L\. Groom, K\. Hausman, B\. Ichter, et al\.π​\_​0\\pi\\\_0: A vision\-language\-action flow model for general robot control\.arXiv preprint arXiv:2410\.24164, 2024\.
- \[6\]K\. Black, M\. Janner, Y\. Du, I\. Kostrikov, and S\. Levine\.Training diffusion models with reinforcement learning\.InThe Twelfth International Conference on Learning Representations, 2024\.
- \[7\]T\. Brown et al\.Language models are few\-shot learners\.Advances in Neural Information Processing Systems, 33, 2020\.
- \[8\]Y\. Burda, H\. Edwards, A\. Storkey, and O\. Klimov\.Exploration by random network distillation\.arXiv preprint arXiv:1810\.12894, 2018\.
- \[9\]K\. Chen, Z\. Liu, T\. Zhang, Z\. Guo, S\. Xu, H\. Lin, H\. Zang, Q\. Zhang, Z\. Yu, G\. Fan, et al\.π\\pirl: Online rl fine\-tuning for flow\-based vision\-language\-action models\.arXiv preprint arXiv:2510\.25889, 2025\.
- \[10\]J\. R\. Daniel, V\. R\. Benjamin, K\. Abbas, O\. Ian, and W\. Zheng\.A tutorial on thompson sampling\.Foundations and Trends® in Machine Learning, 11\(1\):1–99, 2018\.
- \[11\]J\. Dekoninck, N\. Jovanović, T\. Gehrunger, K\. Rögnvalddson, I\. Petrov, C\. Sun, and M\. Vechev\.Beyond benchmarks: Matharena as an evaluation platform for mathematics with llms\.2026\.
- \[12\]Y\. Du, W\. Sun, and C\. G\. Snoek\.Ipo: Interpretable prompt optimization for vision\-language models\.Advances in Neural Information Processing Systems, 37:126725–126766, 2024\.
- \[13\]A\. Ecoffet, J\. Huizinga, J\. Lehman, K\. O\. Stanley, and J\. Clune\.First return, then explore\.Nature, 590\(7847\):580–586, 2021\.
- \[14\]D\. Guo, D\. Yang, Z\. Zhang, et al\.Deepseek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature, 645\(8081\), 2025\.
- \[15\]T\. Haarnoja, A\. Zhou, P\. Abbeel, and S\. Levine\.Soft actor\-critic: Off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.InInternational conference on machine learning, pages 1861–1870\. Pmlr, 2018\.
- \[16\]Z\.\-W\. Hong, I\. Shenfeld, T\.\-H\. Wang, Y\.\-S\. Chuang, A\. Pareja, J\. Glass, A\. Srivastava, and P\. Agrawal\.Curiosity\-driven red\-teaming for large language models\.arXiv preprint arXiv:2402\.19464, 2024\.
- \[17\]A\. Hurst, A\. Lerer, A\. P\. Goucher, A\. Perelman, A\. Ramesh, A\. Clark, A\. Ostrow, A\. Welihinda, A\. Hayes, A\. Radford, et al\.Gpt\-4o system card\.arXiv preprint arXiv:2410\.21276, 2024\.
- \[18\]A\. Jaech, A\. Kalai, A\. Lerer, A\. Richardson, A\. El\-Kishky, A\. Low, A\. Helyar, A\. Madry, A\. Beutel, A\. Carney, et al\.Openai o1 system card\.arXiv preprint arXiv:2412\.16720, 2024\.
- \[19\]N\. Jain, K\. Han, A\. Gu, W\.\-D\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. Stoica\.Livecodebench: Holistic and contamination free evaluation of large language models for code\.arXiv preprint arXiv:2403\.07974, 2024\.
- \[20\]L\. P\. Kaelbling, M\. L\. Littman, and A\. W\. Moore\.Reinforcement learning: A survey\.Journal of artificial intelligence research, 4:237–285, 1996\.
- \[21\]S\. Karnik, Z\.\-W\. Hong, N\. Abhangi, Y\.\-C\. Lin, T\.\-H\. Wang, C\. Dupuy, R\. Gupta, and P\. Agrawal\.Embodied red teaming for auditing robotic foundation models\.arXiv preprint arXiv:2411\.18676, 2024\.
- \[22\]M\. J\. Kim, K\. Pertsch, S\. Karamcheti, T\. Xiao, A\. Balakrishna, S\. Nair, R\. Rafailov, E\. Foster, G\. Lam, P\. Sanketi, et al\.Openvla: An open\-source vision\-language\-action model\.arXiv preprint arXiv:2406\.09246, 2024\.
- \[23\]J\. Kwok, X\. Zhang, M\. Xu, Y\. Liu, A\. Mirhoseini, C\. Finn, and M\. Pavone\.Scaling verification can be more effective than scaling policy learning for vision\-language\-action alignment, 2026\.
- \[24\]H\. Li, Y\. Zuo, J\. Yu, Y\. Zhang, Y\. Zhaohui, K\. Zhang, X\. Zhu, Y\. Zhang, T\. Chen, G\. Cui, D\. Wang, D\. Luo, Y\. Fan, Y\. Sun, J\. Zeng, J\. Pang, S\. Zhang, Y\. Wang, Y\. Mu, B\. Zhou, and N\. Ding\.SimpleVLA\-RL: Scaling VLA training via reinforcement learning\.InThe Fourteenth International Conference on Learning Representations, 2026\.
- \[25\]A\. Liang, Y\. Korkmaz, J\. Zhang, M\. Hwang, A\. Anwar, S\. Kaushik, A\. Shah, A\. S\. Huang, L\. Zettlemoyer, D\. Fox, et al\.Robometer: Scaling general\-purpose robotic reward models via trajectory comparisons\.arXiv preprint arXiv:2603\.02115, 2026\.
- \[26\]B\. Liu, Y\. Zhu, C\. Gao, Y\. Feng, Q\. Liu, Y\. Zhu, and P\. Stone\.Libero: Benchmarking knowledge transfer for lifelong robot learning\.arXiv preprint arXiv:2306\.03310, 2023\.
- \[27\]J\. Liu, F\. Gao, B\. Wei, X\. Chen, Q\. Liao, Y\. Wu, C\. Yu, and Y\. Wang\.What can RL bring to VLA generalization? an empirical study\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems, 2025\.
- \[28\]Y\. J\. Ma, J\. Hejna, C\. Fu, D\. Shah, J\. Liang, Z\. Xu, S\. Kirmani, P\. Xu, D\. Driess, T\. Xiao, et al\.Vision language models are in\-context value learners\.InThe Thirteenth International Conference on Learning Representations, 2025\.
- \[29\]Y\. J\. Ma, V\. Kumar, A\. Zhang, O\. Bastani, and D\. Jayaraman\.Liv: Language\-image representations and rewards for robotic control\.InInternational Conference on Machine Learning, pages 23301–23320\. PMLR, 2023\.
- \[30\]Y\. J\. Ma, S\. Sodhani, D\. Jayaraman, O\. Bastani, V\. Kumar, and A\. Zhang\.Vip: Towards universal visual reward and representation via value\-implicit pre\-training\.arXiv preprint arXiv:2210\.00030, 2022\.
- \[31\]K\. Opsahl\-Ong, M\. J\. Ryan, J\. Purtell, D\. Broman, C\. Potts, M\. Zaharia, and O\. Khattab\.Optimizing instructions and demonstrations for multi\-stage language model programs\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9340–9366, 2024\.
- \[32\]I\. Osband, C\. Blundell, A\. Pritzel, and B\. Van Roy\.Deep exploration via bootstrapped dqn\.Advances in neural information processing systems, 29, 2016\.
- \[33\]I\. Osband, D\. Russo, and B\. V\. Roy\.\(more\) efficient reinforcement learning via posterior sampling\.InAdvances in Neural Information Processing Systems \(NeurIPS\), pages 3003–3011, 2013\.
- \[34\]D\. Pathak, P\. Agrawal, A\. A\. Efros, and T\. Darrell\.Curiosity\-driven exploration by self\-supervised prediction\.InInternational conference on machine learning, pages 2778–2787\. PMLR, 2017\.
- \[35\]M\. Plappert, R\. Houthooft, P\. Dhariwal, S\. Sidor, R\. Y\. Chen, X\. Chen, T\. Asfour, P\. Abbeel, and M\. Andrychowicz\.Parameter space noise for exploration\.InInternational Conference on Learning Representations \(ICLR\), 2018\.
- \[36\]J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov\.Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347, 2017\.
- \[37\]Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu, et al\.Deepseekmath: Pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300, 2024\.
- \[38\]D\. Soylu, C\. Potts, and O\. Khattab\.Fine\-tuning and prompt optimization: Two great steps that work better together\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10696–10710, 2024\.
- \[39\]M\. Strens\.A bayesian framework for reinforcement learning\.InInternational Conference on Machine Learning \(ICML\), pages 943–950, 2000\.
- \[40\]S\. Tan, K\. Dou, Y\. Zhao, and P\. Krähenbühl\.Interactive post\-training for vision\-language\-action models\.arXiv preprint arXiv:2505\.17016, 2025\.
- \[41\]S\. Tao, F\. Xiang, A\. Shukla, Y\. Qin, X\. Hinrichsen, X\. Yuan, C\. Bao, X\. Lin, Y\. Liu, T\.\-k\. Chan, et al\.Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai\.arXiv preprint arXiv:2410\.00425, 2024\.
- \[42\]G\. Team, P\. Georgiev, V\. I\. Lei, R\. Burnell, L\. Bai, A\. Gulati, G\. Tanzer, D\. Vincent, Z\. Pan, S\. Wang, et al\.Gemini 1\.5: Unlocking multimodal understanding across millions of tokens of context\.arXiv preprint arXiv:2403\.05530, 2024\.
- \[43\]A\. Wagenmaker, M\. Nakamoto, Y\. Zhang, S\. Park, W\. Yagoub, A\. Nagabandi, A\. Gupta, and S\. Levine\.Steering your diffusion policy with latent space reinforcement learning\.arXiv preprint arXiv:2506\.15799, 2025\.
- \[44\]X\. Wang, W\. Zhu, M\. Saxon, M\. Steyvers, and W\. Y\. Wang\.Large language models are latent variable models: Explaining and finding good demonstrations for in\-context learning\.Advances in Neural Information Processing Systems, 36:15614–15638, 2023\.
- \[45\]Y\. Wang, Z\. Sun, J\. Zhang, Z\. Xian, E\. Biyik, D\. Held, and Z\. Erickson\.Rl\-vlm\-f: Reinforcement learning from vision language foundation model feedback\.arXiv preprint arXiv:2402\.03681, 2024\.
- \[46\]R\. J\. Williams\.Simple statistical gradient\-following algorithms for connectionist reinforcement learning\.Machine learning, 8\(3\):229–256, 1992\.
- \[47\]C\. Wu, J\. Li, J\. Zhou, J\. Lin, K\. Gao, K\. Yan, S\.\-m\. Yin, S\. Bai, X\. Xu, Y\. Chen, et al\.Qwen\-image technical report\.arXiv preprint arXiv:2508\.02324, 2025\.
- \[48\]W\. Xiao, H\. Lin, A\. Peng, H\. Xue, T\. He, Z\. Luo, Y\. Xie, F\. Hu, L\. Fan, G\. Shi, and Y\. Zhu\.Self\-improving vision\-language\-action models with data generation via residual RL\.InThe Fourteenth International Conference on Learning Representations, 2026\.
- \[49\]S\. M\. Xie, A\. Raghunathan, P\. Liang, and T\. Ma\.An explanation of in\-context learning as implicit bayesian inference\.arXiv preprint arXiv:2111\.02080, 2021\.
- \[50\]C\. Xu, J\. T\. Springenberg, M\. Equi, A\. Amin, A\. Esmail, S\. Levine, and L\. Ke\.Rl token: Bootstrapping online rl with vision\-language\-action models\.2025\.Physical Intelligence\.
- \[51\]A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388, 2025\.
- \[52\]C\. Yang, X\. Wang, Y\. Lu, H\. Liu, Q\. V\. Le, D\. Zhou, and X\. Chen\.Large language models as optimizers\.InThe Twelfth International Conference on Learning Representations, 2024\.
- \[53\]S\. Zhai, Q\. Zhang, T\. Zhang, F\. Huang, H\. Zhang, M\. Zhou, S\. Zhang, L\. Liu, S\. Lin, and J\. Pang\.A vision\-language\-action\-critic model for robotic real\-world reinforcement learning\.arXiv preprint arXiv:2509\.15937, 2025\.
- \[54\]K\. Zhou, J\. Yang, C\. C\. Loy, and Z\. Liu\.Learning to prompt for vision\-language models\.International journal of computer vision, 130\(9\):2337–2348, 2022\.
- \[55\]X\. Zhou, Y\. Xu, G\. Tie, Y\. Chen, G\. Zhang, D\. Chu, P\. Zhou, and L\. Sun\.Libero\-pro: Towards robust and fair evaluation of vision\-language\-action models beyond memorization\.\[arXiv preprint arXiv:2510\.03827\], 2025\.
- \[56\]Y\. Zhou, A\. I\. Muresanu, Z\. Han, K\. Paster, S\. Pitis, H\. Chan, and J\. Ba\.Large language models are human\-level prompt engineers\.InThe eleventh international conference on learning representations, 2023\.

## Appendix AImplementation Details

### A\.1VLM Prompt Sampler as an Implicit Posterior

PDE represents the prompt distributionρ\(⋅∣g,ℋ\)\\rho\(\\cdot\\mid g,\\mathcal\{H\}\)with a VLM supervisor𝒱\\mathcal\{V\}\. As in Section[4\.1](https://arxiv.org/html/2607.08837#S4.SS1), the interaction history before iterationttis

ℋt=\{\(gi,pi,τi,R​\(τi,gi\)\)\}i<t,\\mathcal\{H\}\_\{t\}=\\\{\(g\_\{i\},p\_\{i\},\\tau\_\{i\},R\(\\tau\_\{i\},g\_\{i\}\)\)\\\}\_\{i<t\},\(4\)wheregig\_\{i\}is the task,pip\_\{i\}is the prompt used for rollout,τi\\tau\_\{i\}is the resulting trajectory, andR​\(τi,gi\)R\(\\tau\_\{i\},g\_\{i\}\)is the task reward\.

In implementation, we store a compact task\-specific history to fit the VLM context window\. For each taskgg, we define

ℋ~tg=\{\(pi,ci,si\):gi=g,i<t\},\\widetilde\{\\mathcal\{H\}\}\_\{t\}^\{g\}=\\\{\(p\_\{i\},c\_\{i\},s\_\{i\}\)\\;:\\;g\_\{i\}=g,\\;i<t\\\},\(5\)wheresi∈\[0,1\]s\_\{i\}\\in\[0,1\]is the empirical success rate of promptpip\_\{i\}overNNrollouts, andcic\_\{i\}is a one\-sentence VLM summary of the corresponding rollout behavior\. Thus,ℋ~tg\\widetilde\{\\mathcal\{H\}\}\_\{t\}^\{g\}is a compressed representation of the full historyℋt\\mathcal\{H\}\_\{t\}restricted to taskgg\.

We also maintain a task\-specific prompt pool

𝒫tg⊆\{pi:\(pi,ci,si\)∈ℋ~tg\},\\mathcal\{P\}\_\{t\}^\{g\}\\subseteq\\\{p\_\{i\}:\(p\_\{i\},c\_\{i\},s\_\{i\}\)\\in\\widetilde\{\\mathcal\{H\}\}\_\{t\}^\{g\}\\\},\(6\)containing prompts whose empirical success exceeds a thresholdη\\eta\. At prompt\-update iterationtt, we query the VLM as

p∼ρt\(⋅∣g,ℋt\)≈𝒱\(⋅∣pg,𝒫tg,ℋ~tg\),p\\sim\\rho\_\{t\}\(\\cdot\\mid g,\\mathcal\{H\}\_\{t\}\)\\;\\approx\\;\\mathcal\{V\}\(\\cdot\\mid p\_\{g\},\\mathcal\{P\}\_\{t\}^\{g\},\\widetilde\{\\mathcal\{H\}\}\_\{t\}^\{g\}\),\(7\)wherepgp\_\{g\}is the canonical prompt for taskgg\. Sampling from the prompt posterior therefore reduces to asking the VLM for candidate prompts conditioned on the canonical prompt, the successful prompt pool, and compressed rollout feedback\. Updating the posterior is implemented by appending the latest rollout summaries and success rates toℋ~tg\\widetilde\{\\mathcal\{H\}\}\_\{t\}^\{g\}and re\-querying the VLM\.

##### Rollout summaries\.

After evaluating a prompt forNNrollouts, we query𝒱\\mathcal\{V\}once on the rollout videos and ask for a single\-sentence description of what the policy attempted and where it failed\. We store only this sentencecic\_\{i\}, not the videos, so the context grows with the number of evaluated prompts rather than the number of frames\. The exact summary prompt is given in Appendix[A\.5](https://arxiv.org/html/2607.08837#A1.SS5)\.

##### Pool admission\.

At each prompt\-update iteration, we evaluateKKcandidate prompts\. A candidateppis added to𝒫tg\\mathcal\{P\}\_\{t\}^\{g\}if its empirical success rate satisfiess≥ηs\\geq\\eta\. Prompts withs<ηs<\\etaare still stored inℋ~tg\\widetilde\{\\mathcal\{H\}\}\_\{t\}^\{g\}as negative feedback\. Prompts with perfect success are kept in the pool without further refinement\.

### A\.2Prompt Discovery and Policy Optimization Schedule

In the VLA experiments, we separate prompt discovery from policy optimization so that the prompt sampler is updated against a fixed initial policy\. This avoids a moving target: ifθ\\thetachanges while prompts are being evaluated, the same prompt may appear good or bad for reasons unrelated to the prompt update\.

We use a two\-stage schedule:

- •Prompt\-posterior update\.We freezeθ\\theta\. For each taskgg, PDE samples candidate prompts fromρt\(⋅∣g,ℋt\)\\rho\_\{t\}\(\\cdot\\mid g,\\mathcal\{H\}\_\{t\}\), evaluates each prompt forNNrollouts, and updatesℋ~tg\\widetilde\{\\mathcal\{H\}\}\_\{t\}^\{g\}and𝒫tg\\mathcal\{P\}\_\{t\}^\{g\}\.
- •RL update\.We freeze the discovered prompt pools and optimizeθ\\thetawith PPO\. Rollouts are collected from a mixture of the canonical promptpgp\_\{g\}and the discovered prompt pool𝒫g\\mathcal\{P\}^\{g\}, as described below\.

We enter the RL stage once the prompt pool contains at leastMminM\_\{\\min\}prompts, or after a maximum prompt\-search budget is reached\.

### A\.3Anchoring Policy Updates to the Canonical Prompt

During RL, exploratory prompts provide successful rollouts, but evaluation is always under the canonical promptpgp\_\{g\}\. We therefore use two mechanisms to transfer improvements from exploratory prompts back topgp\_\{g\}\.

##### Mixture sampling\.

For taskgg, each rollout prompt is sampled as

p∼αt​δpg\+\(1−αt\)​Uniform​\(𝒫g\),p\\sim\\alpha\_\{t\}\\delta\_\{p\_\{g\}\}\+\(1\-\\alpha\_\{t\}\)\\mathrm\{Uniform\}\(\\mathcal\{P\}^\{g\}\),\(8\)where𝒫g\\mathcal\{P\}^\{g\}is the frozen prompt pool found during prompt discovery\. Prompts from𝒫g\\mathcal\{P\}^\{g\}provide reward signal early in training, while rollouts underpgp\_\{g\}keep optimization aligned with evaluation\.

##### Adaptive mixture coefficient\.

Letst\(pg\)s\_\{t\}^\{\(p\_\{g\}\)\}be the empirical success rate of rollouts collected under the canonical prompt at training steptt\. We track an exponential moving average

s¯t\\displaystyle\\bar\{s\}\_\{t\}=β​st\(pg\)\+\(1−β\)​s¯t−1,\\displaystyle=\\beta s\_\{t\}^\{\(p\_\{g\}\)\}\+\(1\-\\beta\)\\bar\{s\}\_\{t\-1\},\(9\)αt\\displaystyle\\alpha\_\{t\}=clip​\(s¯tc,αmin,1\),\\displaystyle=\\mathrm\{clip\}\\\!\\left\(\\frac\{\\bar\{s\}\_\{t\}\}\{c\},\\alpha\_\{\\min\},1\\right\),\(10\)whereβ\\betais the smoothing factor,ccis the consolidation target, andαmin\>0\\alpha\_\{\\min\}\>0ensures that the canonical prompt is always sampled\. When canonical\-prompt success is low, most rollouts come from the discovered prompt pool\. As canonical\-prompt success improves,αt\\alpha\_\{t\}increases and training shifts towardpgp\_\{g\}\.

##### Mixed backpropagation\.

A rollout collected under an exploratory promptppdirectly updatesπθ\(⋅∣o,p\)\\pi\_\{\\theta\}\(\\cdot\\mid o,p\), but evaluation usesπθ\(⋅∣o,pg\)\\pi\_\{\\theta\}\(\\cdot\\mid o,p\_\{g\}\)\. To couple these two prompt\-conditioned policies, we replace the current\-policy log\-probability in the PPO ratio with

ℓt​\(θ\)=12​log⁡πθ​\(at∣ot,pg\)\+12​log⁡πθ​\(at∣ot,p\)\.\\ell\_\{t\}\(\\theta\)=\\frac\{1\}\{2\}\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid o\_\{t\},p\_\{g\}\)\+\\frac\{1\}\{2\}\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid o\_\{t\},p\)\.\(11\)The PPO ratio becomes

wt​\(θ\)=exp⁡\(ℓt​\(θ\)−log⁡πold​\(at∣ot,p\)\),w\_\{t\}\(\\theta\)=\\exp\\\!\\left\(\\ell\_\{t\}\(\\theta\)\-\\log\\pi\_\{\\mathrm\{old\}\}\(a\_\{t\}\\mid o\_\{t\},p\)\\right\),\(12\)where the denominator uses the old log\-probability recorded under the rollout promptpp\. Gradients flow through both forward passes\. This encourages actions that are likely under both the exploratory prompt and the canonical prompt, helping transfer successful exploratory behavior back to the policy evaluated underpgp\_\{g\}\.

### A\.4Hyperparameters

Table[3](https://arxiv.org/html/2607.08837#A1.T3)lists all hyperparameters used in our experiments\. Unless stated otherwise, values are shared across LIBERO and LIBERO\-PRO\.

Table 3:Hyperparameters for PDE\. Values in the top two blocks are specific to PDE; the bottom block lists the standard PPO hyperparameters used for policy\-gradient updates in Regime B\.SymbolDescriptionValuePosterior\-sampling loopT0T\_\{0\}Pool\-building iterations10KKCandidate prompts per iteration5NNRollouts per candidate10η\\etaPool admission threshold\>0\>0Deployment\-prompt anchoringαmin\\alpha\_\{\\min\}Floor on deployment\-prompt mixture weight0\.05β\\betaEMA smoothing factor0\.3ccConsolidation target0\.5PPOϵ\\epsilonClip range0\.2ηlr\\eta\_\{\\text\{lr\}\}Learning rate5×10−65\\times 10^\{\-6\}BBBatch size2048λ\\lambdaGAEλ\\lambda0\.95γ\\gammaDiscount factor0\.99EEEpochs per update4
### A\.5Supervisor Prompt Templates

We reproduce the three prompt templates used to query𝒱\\mathcal\{V\}: \(i\) the candidate\-generation template, which takes\(p0g,𝒫t,ℋt\)\(p\_\{0\}^\{g\},\\mathcal\{P\}\_\{t\},\\mathcal\{H\}\_\{t\}\)as input and returnsKKproposed prompts; \(ii\) the rollout\-summarization template, which takesNNrollout videos of a single prompt as input and returns a one\-sentence summarycc; \(iii\) the system prompt that frames the supervisor’s role\. Each template is given verbatim below, with placeholders in\{curly braces\}\.

##### \(i\) Candidate generation and rollout summarization

> \#\# Training Step \{step\} \-\-\- Prompt Pool \*\*GOAL: \{task\_description\}\*\* \*\*Original prompt\*\*: "\{task\_description\}" \| EMA success: \{orig\_rate\} \[window trend: \{trend\}\] \#\#\# Current Pool Prompts and Their Rollout Videos \*\*Prompt\*\*: "\{prompt\_i\}" \| EMA success: \{ema\_i\} \{rollout video or frames for prompt\_i\} … \(repeated for each prompt in the pool\) \#\#\# Your Task 1\. \*\*Analyze each video\*\*: What is the robot doing under each prompt? Which prompts produce the best behavior toward the goal? 2\. \*\*Summarize each prompt’s effect\*\* in one line\. 3\. \*\*Generate 1 NEW prompt\*\* to add to the pool: \- Address failure modes you observed \- Be semantically diverse from existing prompts \- Try different phrasings, action verbs, spatial references Return ONLY valid JSON: \{ "new\_prompts": \["new prompt"\], "summaries": \{"exact prompt text": "one\-line behavior summary", …\}, "analysis": "brief reasoning about what to try next" \}

##### \(ii\) System prompt

> You are an expert in robotic manipulation and prompt engineering for vision\-language\-action models\. Each prompt is accompanied by \{frames\_per\_video\} frames from a robot rollout video, shown in temporal order \(Frame 1 = start, Frame \{frames\_per\_video\} = end\)\. Trace the robot’s movement across frames to understand what it does over time\. Your tasks: 1\. Analyze the rollout videos \-\-\- what is the robot doing? How close to the goal? 2\. Identify failure modes \(wrong object, wrong location, failed grasp, etc\.\) 3\. Analyze how different prompts affect the robot’s behavior 4\. Generate 1 NEW prompt to add to the pool The GOAL is: "\{task\_description\}" The robot is controlled by a vision\-language\-action model \(pi0\.5\)\. Guidelines for new prompts: \- Be clear and specific about the object and target location \- Use natural language similar to LIBERO benchmark conventions \- Try different levels of specificity, action verbs, spatial references \- Keep prompts concise \(5\-\-15 words typically\)

## Appendix BExperiment Details

### B\.1Compute

For each experiment, we used one node \(8\) of H200s\.

### B\.2Real\-World Experiment Details

#### B\.2\.1SFT Task Suite

Theπ0\.5\\pi\_\{0\.5\}checkpoint used as the starting point for all RL fine\-tuning experiments is trained on the ten language\-conditioned manipulation tasks listed in Table[4](https://arxiv.org/html/2607.08837#A2.T4)\. The tasks share a fixed tabletop scene containing four manipulable objects \(tape, bowl, green container, and a small object placed on a marked region of the workspace\), two stacked drawers \(top and bottom\), and a two\-level rack\. Together they span four object–receptacle relationships—placing an object*in*a bowl \(task 1\),*on*a rack \(tasks 2, 3, 10\),*in front of*the camera \(tasks 4, 5\), and*inside*a drawer \(tasks 6–9\)—over three target objects \(tape, bowl, green container\) and four receptacle locations \(bowl, top rack, bottom rack, top drawer, bottom drawer, front of the camera\)\.

Each task is collected via teleoperated demonstrations on the Franka FR3 platform described in Section[5\.6](https://arxiv.org/html/2607.08837#S5.SS6), with 40 demonstrations per task and object positions randomized within a fixed workspace region across demonstrations\. We hold the demonstration budget fixed across tasks rather than scaling to per\-task difficulty, so the SFT checkpoint’s per\-task competence reflects the intrinsic difficulty of each task under uniform supervision\.

Table 4:SFT task suite\.The ten tasks theπ0\.5\\pi\_\{0\.5\}SFT checkpoint is trained on, with 40 teleoperated demonstrations per task\.
#### B\.2\.2Detailed Results

Table[5](https://arxiv.org/html/2607.08837#A2.T5)reports per\-task success rates on the three real\-world manipulation tasks\. All methods start from the same SFT checkpoint and are given an identical online\-interaction budget of 128 real\-world rollouts \(two iterations of 64 rollouts each\); every entry is measured over 20 evaluation rollouts\. Under this small budget, vanilla PPO yields only modest gains and degrades one task, while PPO\+PDE improves every task by \+20\.0 to \+40\.0 points, tripling the average success rate of the SFT policy \(15\.0% to 45\.0%\) and more than doubling that of vanilla PPO\.

Table 5:Per\-task success rate \(%\) on real\-world tasks\.SFT: frozen checkpoint before any RL fine\-tuning;PPO,PPO\+PDE: final success rates after RL fine\-tuning with a budget of 128 real\-world rollouts \(2 iterations of 64\), over 20 eval rollouts per task; \(\+X\) is the gain in percentage points over SFT\. Vanilla PPO yields only modest gains and even degrades one task; adding PDE improves every task and attains the highest final success rate across the board, tripling the average success rate of the SFT policy\.

### B\.3Ablation Studies

Is prompt optimization necessary, or do random paraphrases suffice?A natural concern is whether PDE’s gains come simply from exposing the policy to*diverse*phrasings of the goal, rather than from*optimizing*which phrasings enter the curriculum\. To isolate this, we replace PDE’s discovered prompt pool with an un\-optimized one: for each task we query an VLM \(Qwen3\-VL\-235B\) for five random paraphrases of the canonical instructionpgp\_\{g\}that describe the same task, using*no*rollout feedback or success\-rate filtering\. Everything else—the EMA mixed\-backpropagation curriculum, the consolidation floor, and all hyperparameters—is held fixed\. We compare this*Random Prompts*variant against PDE \(optimized pool\) and a plain PPO baseline trained only onpgp\_\{g\}, on the Object suite with task perturbation\.

Figure[9](https://arxiv.org/html/2607.08837#A2.F9)shows that random paraphrases are not enough\. At step120120\(≈15\\approx\\\!15M env steps\), PDE reaches62%62\\%success while Random Prompts reaches only42%42\\%and plain PPO35%35\\%\(mean across1010tasks\)\. Random paraphrasing yields only a marginal gain over PPO and tracks the PPO baseline for most of training, falling roughly2020points short of PDE\. The reason is visible at the level of individual prompts: evaluated on the base SFT policy, the random paraphrases attain a mean success rate of just6\.8%6\.8\\%on the tasks where PDE’s optimized prompts reach29\.2%29\.2\\%, and0%0\\%on99of the1010tasks\. Un\-optimized rephrasings rarely unlock reward on the hard perturbed tasks; PDE’s search is what surfaces the spatially and visually grounded phrasings that move the policy off zero\. Optimizing*which*prompts enter the curriculum—not merely diversifying them—is therefore essential\.

![Refer to caption](https://arxiv.org/html/2607.08837v1/x10.png)Figure 9:Ablation on prompt*optimization*\(Object suite, task perturbation\)\. PDE \(optimized prompt pool\) vs\. Random Prompts \(five un\-optimized LLM paraphrases per task\) vs\. PPO \(canonical prompt only\)\. Curves are mean success rate across the1010tasks; shaded regions are±1\\pm 1SE across tasks\. Random paraphrases improve only marginally over PPO and fall well short of PDE, showing that optimizing which prompts enter the curriculum—not merely diversifying them—is what drives the gain\.How does the number of discovered prompts scale with the search budget?Prompt discovery is both consistent across runs and scales reliably with the search budget: the cumulative number of nonzero prompts grows approximately linearly with rollouts, with each independent run finding99–1212nonzero prompts after8484rollouts \(Figure[10](https://arxiv.org/html/2607.08837#A2.F10)\)\. This suggests that allocating more rollouts directly translates to a larger and more diverse curriculum pool\.

Does PDE produce a paraphrase\-robust policy?PDE trains under a learned distribution over prompts, so the resulting policy is exposed to many surface realizations of the same goal during fine\-tuning\. We construct two prompt sets of size55:*\(i\) IN\-Distribution Train Prompts*—the canonical instructionpg=p\_\{g\}\{=\}*“close the microwave”*together with the four exploratory phrasings PDE drew fromρ\(⋅∣g,ℋ\)\\rho\(\\cdot\\mid g,\\mathcal\{H\}\)during training;*\(ii\) Out\-Of\-Distribution Paraphrases*—five LLM\-generated rephrasings that the policy never saw\. As shown in Table[7](https://arxiv.org/html/2607.08837#A2.T7), PDE generalizes across the surface form of the instruction, with no drop on unseen prompts\.

![Refer to caption](https://arxiv.org/html/2607.08837v1/x11.png)
Figure 10:Cumulative unique nonzero prompts discovered vs\. cumulative rollouts across three independent runs on the microwave task\.Table 7:Success Rate \(%\) on the Microwave task across1010prompts \(55IND,55OOD\)\.
Table 8:Eval SR on original prompt\.Is a single best prompt enough?PDE mixes rollouts from a discovered prompt pool and gradually shifts training toward the canonical promptpgp\_\{g\}\. A natural alternative is to train on only the single highest\-success prompt, or to converge to the best discovered prompt rather thanpgp\_\{g\}\. Evaluated under each method’s own training prompt, the single\-best\-prompt variant scores highest \(65\.4%65\.4\\%vs\.59\.6%59\.6\\%for PDE, Table[9](https://arxiv.org/html/2607.08837#A2.T9)\), since the policy specializes entirely on one instruction\. However, this advantage disappears at deployment: when evaluated under the*original*instructionpgp\_\{g\}, training on a single discovered prompt drops to39\.6%39\.6\\%SR while PDE reaches63\.8%63\.8\\%\(Table[8](https://arxiv.org/html/2607.08837#A2.T8)\)—a gap of over2424points\. The discovered prompt acts as a scaffold that unlocks early reward, but converging back topgp\_\{g\}during training is what transfers that improvement to the deployment instruction\.

Does mixed backpropagation help?Backpropagating on the original prompt alone receives sparse reward early in training and learns slowly; backpropagating on the curriculum prompt alone learns faster but plateaus, since gradient updates are never conditioned on the deployment prompt\. Mixed backpropagation achieves the highest final success rate by combining the dense gradient signal of the curriculum with direct optimization on the deployment instruction \(Table[9](https://arxiv.org/html/2607.08837#A2.T9)\)\.

How sensitive is the curriculum to the consolidation targetcc?Table[9](https://arxiv.org/html/2607.08837#A2.T9)shows that all three settings \(c∈\{0\.25,0\.5,0\.75\}c\\in\\\{0\.25,0\.5,0\.75\\\}\) converge to comparable final success rates within per\-task SE, indicating that the method is not sensitive to this hyperparameter\. We usec=0\.5c\{=\}0\.5as the default\.

Table 9:Ablations on Object/task perturbation at step 120 \(250 envs, mean across 10 tasks\)\. Default variant shaded\.AblationVariantSR \(%\)Training promptsSingle best prompt65\.4\\mathbf\{65\.4\}Converge to Best63\.763\.7\\cellcolorourskyblueFull pool \(Ours\)\\cellcolorourskyblue59\.659\.6BackpropagationOriginal\-only49\.649\.6Curriculum\-only45\.045\.0\\cellcolorourskyblueMixed \(Ours\)\\cellcolorourskyblue59\.6\\mathbf\{59\.6\}Consolidationccc=0\.25c\{=\}0\.2558\.858\.8\\cellcolorourskybluec=0\.5c\{=\}0\.5\(Ours\)\\cellcolorourskyblue59\.659\.6c=0\.75c\{=\}0\.7563\.7\\mathbf\{63\.7\}
### B\.4Detailed results

Table 10:Pi0\.5 aggregated success rate \(%\) on LIBERO\-PRO, by task difficulty and by perturbation axis\.Table 11:GR00T and Pi0 aggregated success rate \(%\) on LIBERO\-PRO by task difficulty\.Table 12:Per\-task success rate \(%\) on LIBERO\-PRO with GR00T, sorted by difficulty bucket and PPO init SR\.Diff\.SuitePert\.TaskInitPPOOurseasygoalobject7100\.0096\.0096\.00easy10swap183\.6796\.0096\.00easyobjectswap679\.25100\.0092\.00easy10swap578\.4396\.0092\.00easyobjectobject878\.3396\.00100\.00easyobjectswap878\.33100\.00100\.00easy10object177\.5592\.00100\.00easyobjectobject476\.7996\.00100\.00easy10object375\.5696\.00100\.00easyobjectswap375\.56100\.00100\.00easyobjectswap475\.00100\.00100\.00easyspatialswap169\.3996\.0096\.00easy10swap667\.92100\.00100\.00easyobjectobject667\.92100\.00100\.00easy10object467\.8680\.0096\.00easyobjectswap566\.67100\.00100\.00easyobjectobject163\.27100\.00100\.00easyobjectswap163\.27100\.00100\.00easyobjectobject762\.79100\.00100\.00easy10object262\.7596\.00100\.00easy10swap460\.7188\.0096\.00easyobjecttask159\.18100\.00100\.00easy10swap258\.8296\.00100\.00easyspatialswap558\.8284\.00100\.00easyspatialobject157\.1484\.00100\.00easyobjectobject956\.36100\.00100\.00easyobjectswap755\.81100\.0096\.00easyspatialswap055\.10100\.00100\.00easyspatialswap254\.9084\.0092\.00easyobjectswap053\.0696\.00100\.00easyspatialobject053\.06100\.0096\.00easyspatialobject250\.9888\.00100\.00easygoalswap549\.0292\.00100\.00easy10swap048\.9892\.0092\.00easyobjectobject346\.67100\.00100\.00easy10swap746\.51100\.00100\.00easyobjectswap945\.45100\.0096\.00easyobjectobject545\.10100\.00100\.00easygoaltask744\.19100\.00100\.00easygoalobject541\.1896\.0096\.00mediumgoalswap838\.3388\.0088\.00mediumspatialobject535\.2992\.00100\.00mediumspatialobject633\.9696\.00100\.00mediumspatialobject333\.33100\.00100\.00mediumspatialswap930\.9180\.00100\.00mediumobjectobject229\.41100\.00100\.00mediumspatialobject828\.3388\.0096\.00mediumspatialswap828\.3388\.0096\.00mediumspatialswap628\.3092\.00100\.00mediumgoalobject227\.4564\.00100\.00medium10object925\.4564\.000\.00mediumspatialobject923\.6484\.0084\.00mediumspatialswap723\.2664\.0096\.00mediumspatialtask521\.5792\.0092\.00mediumspatialtask920\.0096\.00100\.00mediumspatialobject419\.6496\.0096\.00mediumobjectswap219\.6196\.00100\.00mediumspatialswap416\.0796\.00100\.00medium10object813\.3368\.0096\.00mediumgoalobject912\.7388\.0092\.00mediumspatialobject711\.6380\.0092\.00mediumspatialtask711\.6384\.0096\.00mediumgoalobject410\.7192\.00100\.00mediumgoalobject010\.2096\.00100\.00mediumspatialtask27\.8492\.00100\.00medium10task06\.1284\.0092\.00medium10task85\.0080\.0096\.00mediumgoalobject14\.08100\.00100\.00mediumobjectobject02\.0476\.00100\.00medium10object51\.9696\.00100\.00mediumgoaltask51\.9692\.0092\.00hard10object00\.000\.000\.00hard10object60\.0092\.00100\.00hard10object70\.000\.000\.00hard10swap30\.000\.000\.00hard10swap80\.000\.000\.00hard10swap90\.000\.000\.00hard10task10\.000\.000\.00hard10task20\.000\.000\.00hard10task30\.000\.000\.00hard10task40\.000\.000\.00hard10task50\.000\.000\.00hard10task60\.000\.000\.00hard10task70\.000\.000\.00hard10task90\.000\.000\.00hardgoalobject30\.000\.000\.00hardgoalobject60\.0036\.0048\.00hardgoalobject80\.0080\.0092\.00hardgoalswap00\.000\.000\.00hardgoalswap10\.000\.000\.00hardgoalswap20\.000\.0096\.00hardgoalswap30\.0088\.0088\.00hardgoalswap40\.000\.000\.00hardgoalswap60\.000\.000\.00hardgoalswap70\.000\.00100\.00hardgoalswap90\.000\.00100\.00hardgoaltask00\.000\.00100\.00hardgoaltask10\.000\.000\.00hardgoaltask20\.000\.000\.00hardgoaltask30\.000\.000\.00hardgoaltask40\.000\.000\.00hardgoaltask60\.000\.000\.00hardgoaltask80\.004\.000\.00hardgoaltask90\.000\.000\.00hardobjecttask00\.000\.000\.00hardobjecttask20\.000\.000\.00hardobjecttask30\.000\.000\.00hardobjecttask40\.000\.000\.00hardobjecttask50\.000\.000\.00hardobjecttask60\.000\.000\.00hardobjecttask70\.000\.000\.00hardobjecttask80\.000\.000\.00hardobjecttask90\.000\.000\.00hardspatialswap30\.000\.000\.00hardspatialtask00\.000\.0080\.00hardspatialtask10\.000\.000\.00hardspatialtask30\.0092\.000\.00hardspatialtask40\.0056\.000\.00hardspatialtask60\.0096\.00100\.00hardspatialtask80\.0080\.0092\.00Table 13:Per\-task success rate \(%\) on LIBERO\-PRO with Pi0, sorted by difficulty bucket and PPO init SR\.Diff\.SuitePert\.TaskInitPPOOurseasygoalobject193\.88100\.00100\.00easyobjectswap489\.2996\.00100\.00easygoalswap886\.67100\.00100\.00easyspatialobject286\.27100\.00100\.00easyspatialswap282\.35100\.00100\.00easyspatialobject077\.558\.0096\.00easyspatialswap077\.5592\.0088\.00easygoalobject574\.510\.0096\.00easyspatialobject366\.6796\.00100\.00easyobjectobject462\.50100\.00100\.00easyobjectswap860\.00100\.00100\.00easyspatialtask556\.86100\.0096\.00easyobjectswap956\.3692\.0092\.00easyobjectswap055\.10100\.00100\.00easyobjectswap254\.9096\.0096\.00easygoalobject853\.3380\.000\.00easygoalswap550\.98100\.0096\.00easyobjectswap547\.0696\.0096\.00easygoalobject746\.51100\.00100\.00easyspatialobject444\.6480\.0012\.00easygoaltask744\.1968\.0092\.00easyobjectswap643\.40100\.00100\.00easyobjectobject543\.1496\.0092\.00easyspatialswap441\.0788\.0064\.00easyobjectobject140\.82100\.00100\.00easyobjectswap140\.82100\.0096\.00easyobjectobject840\.00100\.00100\.00mediumobjectobject639\.6296\.00100\.00mediumspatialtask132\.6588\.0020\.00mediumgoalobject927\.2744\.0072\.00mediumspatialswap816\.6772\.0088\.00mediumspatialswap116\.33100\.0092\.00mediumspatialobject114\.29100\.00100\.00mediumobjectswap713\.9596\.0092\.00mediumspatialobject813\.3388\.0088\.00mediumobjectobject912\.7396\.0096\.00mediumobjectobject012\.24100\.00100\.00mediumspatialswap611\.32100\.0076\.00mediumgoalswap311\.11100\.0096\.00mediumspatialobject910\.9196\.0092\.00mediumobjectobject79\.30100\.00100\.00mediumgoalswap76\.980\.0096\.00mediumobjectswap36\.6792\.00100\.00mediumspatialtask86\.6792\.00100\.00mediumspatialtask25\.8888\.0084\.00mediumspatialobject65\.6668\.0096\.00mediumspatialswap95\.4588\.0084\.00mediumgoalobject34\.4416\.0072\.00mediumgoaltask53\.9252\.0080\.00mediumobjectobject23\.920\.00100\.00medium10object12\.040\.0016\.00hard10object00\.000\.000\.00hard10object20\.000\.000\.00hard10object30\.0088\.0084\.00hard10object40\.000\.0016\.00hard10object50\.000\.000\.00hard10object60\.000\.000\.00hard10object70\.000\.000\.00hard10object80\.000\.000\.00hard10object90\.000\.000\.00hard10swap00\.000\.000\.00hard10swap10\.000\.0056\.00hard10swap20\.0020\.000\.00hard10swap30\.000\.000\.00hard10swap40\.0012\.000\.00hard10swap50\.000\.000\.00hard10swap60\.000\.000\.00hard10swap70\.000\.000\.00hard10swap80\.000\.000\.00hard10swap90\.000\.000\.00hardgoalobject00\.0020\.000\.00hardgoalobject20\.000\.000\.00hardgoalobject40\.0024\.000\.00hardgoalobject60\.000\.000\.00hardgoalswap00\.000\.008\.00hardgoalswap10\.00100\.000\.00hardgoalswap20\.000\.000\.00hardgoalswap40\.000\.000\.00hardgoalswap60\.000\.000\.00hardgoalswap90\.000\.000\.00hardgoaltask00\.000\.000\.00hardgoaltask10\.00100\.000\.00hardgoaltask20\.000\.000\.00hardgoaltask30\.000\.000\.00hardgoaltask40\.000\.000\.00hardgoaltask60\.000\.000\.00hardgoaltask80\.000\.000\.00hardgoaltask90\.000\.000\.00hardobjectobject30\.0092\.0096\.00hardspatialobject50\.000\.0040\.00hardspatialobject70\.000\.000\.00hardspatialswap30\.004\.000\.00hardspatialswap50\.0052\.0020\.00hardspatialswap70\.004\.0012\.00hardspatialtask00\.000\.000\.00hardspatialtask30\.000\.0072\.00hardspatialtask40\.0076\.0020\.00hardspatialtask60\.000\.0040\.00hardspatialtask70\.004\.0092\.00hardspatialtask90\.000\.0068\.00Table 14:Per\-task success rate \(%\) on ManiSkill atτ=1\.0\\tau\{=\}1\.0, sorted by difficulty bucket and SFT init SR\.DifficultyCategoryVariantTaskSFTPPOOurshardsemanticMultiPlate023\.1030\.8033\.30hardsemanticMultiPlate1323\.1030\.8025\.00hardsemanticMultiPlate1122\.2033\.3066\.70hardexecutionPosition121\.4057\.1071\.40hardsemanticMultiPlate720\.0020\.0045\.00hardexecutionPositionChangeTo820\.0055\.0060\.00hardexecutionEEPose320\.0070\.0060\.00hardexecutionEEPose520\.0073\.3093\.30hardvisualVisionTexture051120\.0053\.30100\.00hardvisualVisionWhole051120\.0053\.30100\.00hardvisualVisionWhole051420\.0060\.0070\.00hardexecutionPosition218\.8056\.2068\.80hardexecutionPositionChangeTo418\.8068\.8062\.50hardsemanticMultiPlate918\.2045\.5036\.40hardvisualVisionWhole03618\.20100\.0084\.60hardsemanticMultiCarrot2417\.9032\.1035\.70hardexecutionPositionChangeTo1217\.6052\.9085\.70hardexecutionPosition1216\.7061\.1061\.10hardexecutionPositionChangeTo1416\.7075\.0058\.30hardsemanticMultiCarrot1815\.8052\.6057\.90hardexecutionPosition1115\.4030\.8038\.50hardvisualVisionWhole031115\.4053\.30100\.00hardsemanticMultiPlate515\.0065\.0060\.00hardsemanticPlate214\.3081\.0085\.70hardsemanticMultiPlate314\.3064\.3083\.30hardsemanticMultiPlate814\.3042\.9050\.00hardexecutionPositionChangeTo714\.3042\.9042\.90hardvisualVisionTexture05214\.3076\.2081\.00hardvisualVisionWhole05214\.3066\.7061\.90hardexecutionPositionChangeTo1013\.3040\.0050\.00hardexecutionPositionChangeTo112\.5062\.5050\.00hardvisualVisionTexture031212\.5081\.2093\.80hardsemanticMultiCarrot2111\.8026\.5044\.10hardsemanticMultiPlate611\.8029\.4040\.00hardsemanticMainCarrot219\.7077\.4067\.70hardsemanticMultiCarrot239\.7035\.5044\.40hardvisualVisionTexture0519\.5042\.9066\.70hardvisualVisionWhole0519\.5038\.1050\.00hardsemanticMultiPlate49\.1045\.5036\.40hardexecutionEEPose18\.3050\.00100\.00hardexecutionPositionChangeTo27\.7053\.80100\.00hardexecutionPositionChangeTo67\.7076\.90100\.00hardexecutionEEPose117\.1064\.3050\.00hardsemanticMultiPlate26\.2031\.2033\.30hardsemanticMultiPlate146\.2050\.0050\.00hardsemanticMainCarrot225\.9047\.1061\.80hardsemanticMultiPlate105\.905\.9017\.60hardexecutionPositionChangeTo35\.9035\.3047\.10hardexecutionPositionChangeTo95\.9047\.1050\.00hardvisualVisionWhole0315\.0061\.9066\.70hardsemanticPlate14\.8052\.4057\.10hardvisualVisionTexture0314\.8038\.1066\.70hardvisualVisionTexture0324\.80100\.00100\.00hardsemanticMultiCarrot222\.6059\.0064\.30hardsemanticMultiPlate10\.0028\.6040\.00hardsemanticMultiPlate120\.0033\.3041\.70hardsemanticMultiPlate150\.0044\.4033\.30hardexecutionPositionChangeTo110\.0075\.0033\.30hardexecutionPositionChangeTo130\.0010\.0033\.30mediumsemanticMainCarrot1850\.0068\.8081\.20mediumsemanticPlate1450\.0080\.00100\.00mediumsemanticMainCarrot1948\.6068\.6077\.10mediumsemanticMainCarrot1648\.3075\.9089\.70mediumexecutionPosition547\.8073\.9087\.00mediumvisualVisionTexture05747\.1076\.5076\.50mediumvisualVisionWhole05347\.1076\.50100\.00mediumexecutionEEPose1046\.7060\.0073\.30mediumvisualVisionImage1446\.2092\.30100\.00mediumvisualVisionImage1144\.4066\.70100\.00mediumsemanticPlate1243\.8068\.80100\.00mediumexecutionPosition943\.8025\.0062\.50mediumexecutionEEPose743\.8037\.5050\.00mediumexecutionEEPose942\.9042\.9071\.40mediumvisualVisionWhole05542\.9078\.6078\.60mediumvisualVisionWhole05942\.1057\.9073\.70mediumvisualVisionTexture051541\.2052\.9058\.80mediumvisualVisionWhole031541\.2052\.9064\.70mediumexecutionPosition840\.0066\.7073\.30mediumvisualVisionImage1240\.0085\.0095\.00mediumvisualVisionTexture051440\.0080\.00100\.00mediumvisualVisionWhole031440\.0090\.0090\.00mediumexecutionPositionChangeTo038\.1057\.1071\.40mediumsemanticMainCarrot2037\.5095\.80100\.00mediumvisualVisionWhole051237\.5068\.8081\.20mediumexecutionEEPose1436\.8073\.7084\.20mediumvisualVisionWhole03835\.3088\.2088\.20mediumexecutionPosition1435\.0065\.0075\.00mediumsemanticPlate1133\.3066\.7073\.30mediumexecutionPosition333\.3053\.3046\.70mediumexecutionEEPose633\.3077\.8077\.80mediumvisualVisionTexture031133\.3073\.30100\.00mediumvisualVisionWhole051033\.3046\.7040\.00mediumexecutionEEPose1531\.6021\.1031\.60mediumsemanticPlate630\.8069\.2084\.60mediumvisualVisionTexture03630\.80100\.00100\.00mediumvisualVisionWhole05630\.8053\.80100\.00mediumsemanticMainCarrot2330\.4034\.8056\.50mediumexecutionEEPose429\.4076\.5088\.20mediumexecutionEEPose1229\.4088\.20100\.00mediumvisualVisionWhole051529\.4029\.4047\.10mediumsemanticMultiCarrot1929\.2033\.3037\.50mediumsemanticMultiCarrot1628\.6035\.7054\.50mediumexecutionEEPose828\.6066\.7071\.40mediumvisualVisionTexture03528\.6078\.60100\.00mediumvisualVisionTexture05528\.6057\.1075\.00mediumvisualVisionWhole03227\.8090\.5081\.00mediumexecutionPosition1527\.3027\.3036\.40mediumvisualVisionImage126\.7033\.3066\.70mediumsemanticMainCarrot2426\.1065\.2073\.90mediumsemanticMultiCarrot1725\.0050\.0057\.10mediumexecutionPositionChangeTo525\.0075\.0087\.50mediumvisualVisionTexture051225\.0075\.0081\.20mediumsemanticMultiCarrot2024\.2066\.7075\.80mediumexecutionPosition623\.5047\.1088\.20mediumexecutionPosition1323\.5035\.3058\.80mediumvisualVisionTexture05323\.5082\.40100\.00mediumexecutionPositionChangeTo1523\.1034\.6030\.80mediumvisualVisionTexture05623\.1069\.20100\.00easyvisualVisionImage1392\.3061\.5076\.90easyvisualVisionTexture03088\.90100\.00100\.00easyvisualVisionWhole03088\.2088\.90100\.00easyvisualVisionImage787\.0082\.6095\.70easyvisualVisionImage886\.7086\.70100\.00easyvisualVisionImage084\.6084\.6096\.20easysemanticPlate083\.3088\.9094\.40easyvisualVisionTexture05083\.3083\.3075\.00easyvisualVisionWhole031076\.9066\.7080\.00easyvisualVisionTexture03376\.5094\.10100\.00easyvisualVisionImage1075\.0058\.3066\.70easyvisualVisionWhole03975\.0078\.9073\.70easysemanticPlate1073\.3033\.3046\.70easyexecutionPosition473\.3020\.0073\.30easyvisualVisionImage1573\.3046\.7080\.00easyvisualVisionImage571\.40100\.00100\.00easyvisualVisionImage470\.6094\.10100\.00easyvisualVisionTexture03770\.6076\.50100\.00easyvisualVisionImage370\.0090\.00100\.00easyvisualVisionTexture031470\.0090\.00100\.00easysemanticPlate968\.4057\.9068\.40easyvisualVisionTexture031066\.7040\.0080\.00easyvisualVisionWhole05066\.7072\.2075\.00easysemanticPlate364\.7094\.1094\.10easysemanticPlate764\.7070\.6088\.20easyvisualVisionTexture03864\.70100\.00100\.00easyvisualVisionTexture05864\.7094\.10100\.00easyvisualVisionWhole05864\.7070\.6083\.30easyexecutionPosition064\.3050\.0057\.10easyvisualVisionImage964\.3050\.00100\.00easyvisualVisionTexture03963\.2047\.4080\.00easyvisualVisionTexture05963\.2068\.4073\.70easyvisualVisionWhole03762\.5082\.40100\.00easyvisualVisionWhole031262\.5093\.8093\.80easysemanticPlate1360\.0050\.0080\.00easyvisualVisionTexture051360\.0040\.0080\.00easyvisualVisionWhole031360\.0070\.0060\.00easysemanticPlate858\.8094\.1094\.10easyexecutionPosition758\.3041\.7075\.00easyexecutionEEPose1358\.3041\.7066\.70easysemanticPlate557\.10100\.00100\.00easyvisualVisionImage657\.1085\.7092\.90easyvisualVisionWhole03557\.10100\.00100\.00easysemanticPlate456\.2093\.8093\.80easyvisualVisionTexture03456\.20100\.00100\.00easyvisualVisionWhole05456\.2081\.20100\.00easyexecutionEEPose053\.8084\.6092\.30easyvisualVisionTexture051053\.3026\.7053\.30easyvisualVisionWhole03453\.3093\.80100\.00easysemanticPlate1552\.9041\.2058\.80easyvisualVisionImage252\.9088\.2088\.20easyvisualVisionTexture031552\.9052\.9041\.20easyvisualVisionWhole03352\.9094\.10100\.00easyvisualVisionWhole05752\.9070\.6088\.20easysemanticMainCarrot1752\.0064\.0092\.00easyexecutionPosition1050\.0025\.0050\.00easyexecutionEEPose250\.0064\.3064\.30easyvisualVisionTexture031350\.0050\.0090\.00easyvisualVisionTexture05450\.0093\.80100\.00easyvisualVisionWhole051350\.0040\.0060\.00

## Appendix CLLM Experiment Details

This appendix provides implementation details for the language\-model experiments in Section[5\.7](https://arxiv.org/html/2607.08837#S5.SS7)\. We use the same PDE interface as in the VLA experiments: each problemgghas a canonical promptpgp\_\{g\}, PDE proposes an alternative promptp∼ρ\(⋅∣g,ℋ\)p\\sim\\rho\(\\cdot\\mid g,\\mathcal\{H\}\)for training, and evaluation is always performed underpgp\_\{g\}\.

### C\.1Experiment: AIME 2026

Setup\.We evaluate on AIME 2026, which contains3030olympiad\-style math problems, each with a unique answer given as a three\-digit integer between000000and999999\. We fine\-tune Qwen3\-4B with GRPO, using Claude Sonnet 4\.6 as the prompt samplerρ\\rho\. Training runs forT=3T=3PDE iterations\. At each iteration, we sampleK=1K=1alternative prompt per problem, generateN=8N=8rollouts per prompt, and run1010epochs of GRPO\. Both GRPO and GRPO\+PDE use learning rate3⋅10−63\\cdot 10^\{\-6\}\.

Implementation details\.For the first PDE iteration, we query the prompt samplerρ\\rhoonce per problemggto obtain an alternative promptpp\. In later iterations, we reuseppif it outperforms the canonical promptpgp\_\{g\}after the previous training iteration\. Otherwise, we queryρ\\rhofor a new prompt, conditioning on the canonical prompt, the previous alternative prompt, and the rollout historyℋ\\mathcal\{H\}\. Thus, prompt admission is determined adaptively by comparison to the canonical prompt rather than by a fixed thresholdη\\eta\.

For mixed backpropagation, this experiment uses an arithmetic\-mixture variant:

log⁡πmix​\(a∣o\)=log⁡\(πθ​\(a∣o,pg\)\+πθ​\(a∣o,p\)\),\\log\\pi\_\{\\mathrm\{mix\}\}\(a\\mid o\)=\\log\\\!\\left\(\\pi\_\{\\theta\}\(a\\mid o,p\_\{g\}\)\+\\pi\_\{\\theta\}\(a\\mid o,p\)\\right\),up to an additive constant\. This differs from the geometric\-mean form used in the VLA experiments\.

Evaluation and results\.At each checkpoint, we evaluate under the canonical promptpgp\_\{g\}only\. For each problem, we generate2020independent rollouts and report accuracy over all30×20=60030\\times 20=600generations\. Figure[11](https://arxiv.org/html/2607.08837#A3.F11)compares GRPO and GRPO\+PDE\. GRPO\+PDE improves early learning, reaching48\.9%48\.9\\%accuracy after the first iteration compared with45\.8%45\.8\\%for GRPO, and both methods reach53\.3%53\.3\\%after three iterations\.

![Refer to caption](https://arxiv.org/html/2607.08837v1/x12.png)Figure 11:AIME 2026 accuracy for GRPO and GRPO\+PDE\. Evaluation uses only the canonical problem promptpgp\_\{g\}\.
### C\.2Experiment: LiveCodeBench

Setup\.We evaluate on LiveCodeBench, which contains1,0551\{,\}055competitive\-programming problems with hidden tests\. We use600600problems for training and reserve the remaining455455for held\-out evaluation\. We fine\-tune with RLOO at learning rate5⋅10−55\\cdot 10^\{\-5\}\.

Implementation details\.For each eligible training problemgg, the policy itself serves as the prompt samplerρ\\rho\. The policy first generates1010candidate solutions under the canonical promptpgp\_\{g\}\. It is then re\-prompted to produce an alternative promptppconditioned onpgp\_\{g\}and its own raw rollouts\. The sampler observes the rollouts but not their rewards\. This refinement loop is run for33rounds, and the best alternative prompt is cached\. During training, the cached prompt is used alongside the canonical prompt in the RLOO update\. The first50%50\\%of training problems are marked as eligible for prompt refinement; held\-out evaluation uses only the canonical promptpgp\_\{g\}\.

Evaluation and results\.At each batch checkpoint, we evaluate on the first100100held\-out problems by generating one greedy rollout per problem and reporting the fraction of solutions that pass all hidden tests\. The final checkpoint is evaluated in the same way on400400held\-out problems\. Figure[12](https://arxiv.org/html/2607.08837#A3.F12)compares RLOO and PDE\+RLOO\. PDE\+RLOO learns substantially faster: it reaches50%50\\%held\-out accuracy by batch44, while RLOO reaches this level around batch77\. PDE\+RLOO also improves early accuracy, achieving42\.4%42\.4\\%vs\.29\.9%29\.9\\%at batch11,53\.1%53\.1\\%vs\.36\.0%36\.0\\%at batch33, and54\.6%54\.6\\%vs\.41\.3%41\.3\\%at batch44\. The two methods converge later in training, showing that PDE primarily improves early sample efficiency by exposing useful solution modes sooner\.

![Refer to caption](https://arxiv.org/html/2607.08837v1/x13.png)Figure 12:LiveCodeBench held\-out accuracy for RLOO and PDE\+RLOO\. Training uses alternative prompts from PDE, while evaluation uses only the canonical promptpgp\_\{g\}\.