Not All Prompts Are Equal: Exploration-Guided Prompt Scaffolding for Multimodal Reinforcement Post-Training

Hugging Face Daily Papers Papers
prompt-engineering reinforcement-learning multimodal-ai exploration-guided scaffolded-rewrites rl-post-training

Summary

This paper proposes an exploration-guided prompt scaffolding framework that dynamically adjusts training prompts for multimodal reinforcement learning, achieving up to 9.7% relative improvement in performance on benchmarks.

Training prompts in online reinforcement learning (RL) differ substantially in how informative they are for the current policy: some are already saturated while others are too difficult to yield reliable learning signals, yet both receive equal rollout budget under standard training. We propose an exploration-guided prompt scaffolding framework that adapts the training prompt distribution dynamically throughout RL post-training of multimodal large language models (MLLMs). Central to our approach is the Exploration Potential Score (EPS), a lightweight rollout-based proxy for prompt utility derived from KL-regularized policy improvement theory, computable directly from on-policy rollout statistics without additional overhead. Rather than discarding low-utility prompts, we use a teacher model to generate scaffolded rewrites that preserve the original task intent while making subsequent training more informative, reframing teacher supervision as training-data refinement rather than output imitation. Integrated with GRPO on Geo3K and MMK12, our method consistently outperforms the baseline on both in-domain and out-of-distribution benchmarks, achieving up to 9.7\% relative improvement in-domain and gains of 11.5\% on MathVision and 11.1\% on MMMU-Pro.
Original Article
View Cached Full Text

Cached at: 09/15/26, 06:41 AM

Paper page - Not All Prompts Are Equal: Exploration-Guided Prompt Scaffolding for Multimodal Reinforcement Post-Training

Source: https://huggingface.co/papers/2609.15051

Abstract

The framework dynamically adjusts training prompts via exploration potential scoring and scaffolded rewrites to improve reinforcement learning for multimodal language models.

Training prompts inonline reinforcement learning(RL) differ substantially in how informative they are for the current policy: some are already saturated while others are too difficult to yield reliable learning signals, yet both receive equal rollout budget under standard training. We propose anexploration-guided prompt scaffoldingframework that adapts the training prompt distribution dynamically throughout RL post-training ofmultimodal large language models(MLLMs). Central to our approach is theExploration Potential Score(EPS), a lightweight rollout-based proxy for prompt utility derived fromKL-regularized policy improvementtheory, computable directly from on-policy rollout statistics without additional overhead. Rather than discarding low-utility prompts, we use a teacher model to generatescaffolded rewritesthat preserve the original task intent while making subsequent training more informative, reframing teacher supervision as training-data refinement rather than output imitation. Integrated withGRPOon Geo3K and MMK12, our method consistently outperforms the baseline on both in-domain and out-of-distribution benchmarks, achieving up to 9.7\% relative improvement in-domain and gains of 11.5\% on MathVision and 11.1\% on MMMU-Pro.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2609\.15051

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.15051 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.15051 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.15051 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Prompt-Driven Exploration

arXiv cs.LG

The paper introduces Prompt-Driven Exploration (PDE), a method that uses a vision-language model to iteratively refine natural language prompts for reinforcement learning policies, enabling global exploration and successful policy learning even from zero-reward starts.

Prompt-Level Reward Specifications for Open-Ended Post-Training

arXiv cs.CL

This paper proposes a prompt-level reward specification framework that separates reward specification from computation, constructing reusable task-adaptive rubrics and executable constraint checkers offline to produce a hybrid reward for open-ended post-training without requiring human annotations or separate reward models.

From Monolithic to Modular: Segment-level Automatic Prompt Optimization

arXiv cs.AI

This paper introduces SAPO, a segment-level automatic prompt optimization method that decomposes prompts into role, context, tasks, and output format, then applies targeted improvements based on weak and strong examples. Evaluated across several benchmarks, SAPO outperforms strong APO baselines including APE, OPRO, EvoPrompt, GEPA, and StraGO on GPT-3.5-Turbo and GPT-4o-mini.