BalCapRL: A Balanced Framework for RL-Based MLLM Image Captioning

Hugging Face Daily Papers Papers
reinforcement-learning image-captioning multimodal-llms reward-modeling computer-vision

Summary

The paper introduces BalCapRL, a balanced reinforcement learning framework for multimodal large language models that jointly optimizes correctness, coverage, and linguistic quality in image captioning. It demonstrates improved performance over existing methods by addressing trade-offs between utility and fluency through reward decoupling and length-conditional masking.

Image captioning is one of the most fundamental tasks in computer vision. Owing to its open-ended nature, it has received significant attention in the era of multimodal large language models (MLLMs). In pursuit of ever more detailed and accurate captions, recent work has increasingly turned to reinforcement learning (RL). However, existing captioning-RL methods and evaluation metrics often emphasize a narrow notion of caption quality, inducing trade-offs across core dimensions of captioning. For example, utility-oriented objectives can encourage noisy, hallucinated, or overlong captions that improve downstream question answering while harming fluency, whereas arena-style objectives can favor fluent but generic descriptions with limited usefulness. To address this, we propose a more balanced RL framework that jointly optimizes utility-aware correctness, reference coverage, and linguistic quality. In order to effectively optimize the resulting continuous multi-objective reward formulation, we apply GDPO-style reward-decoupled normalization to continuous-valued captioning rewards and show that it improves performance over vanilla GRPO. Additionally, we introduce length-conditional reward masking, yielding a more suitable length penalty for captioning. Across LLaVA-1.5-7B and Qwen2.5-VL 3B and 7B base models, our method consistently improves caption quality, with peak gains of +13.6 DCScore, +9.0 CaptionQA, and +29.0 CapArena across different models.
Original Article
View Cached Full Text

Cached at: 05/12/26, 02:50 AM

Paper page - BalCapRL: A Balanced Framework for RL-Based MLLM Image Captioning

Source: https://huggingface.co/papers/2605.07394

Abstract

A balanced reinforcement learning framework for image captioning that jointly optimizes correctness, coverage, and linguistic quality while improving performance over existing methods.

Image captioningis one of the most fundamental tasks in computer vision. Owing to its open-ended nature, it has received significant attention in the era ofmultimodal large language models(MLLMs). In pursuit of ever more detailed and accurate captions, recent work has increasingly turned toreinforcement learning(RL). However, existing captioning-RL methods and evaluation metrics often emphasize a narrow notion of caption quality, inducing trade-offs across core dimensions of captioning. For example, utility-oriented objectives can encourage noisy, hallucinated, or overlong captions that improve downstream question answering while harming fluency, whereas arena-style objectives can favor fluent but generic descriptions with limited usefulness. To address this, we propose a more balanced RL framework that jointly optimizes utility-aware correctness, reference coverage, and linguistic quality. In order to effectively optimize the resultingcontinuous multi-objective reward formulation, we applyGDPO-stylereward-decoupled normalizationto continuous-valued captioning rewards and show that it improves performance over vanillaGRPO. Additionally, we introducelength-conditional reward masking, yielding a more suitable length penalty for captioning. Across LLaVA-1.5-7B and Qwen2.5-VL 3B and 7B base models, our method consistently improves caption quality, with peak gains of +13.6DCScore, +9.0CaptionQA, and +29.0CapArenaacross different models.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2605\.07394

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.07394 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.07394 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.07394 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Region-Level Policy Optimization for Fine-grained MLLM Perception

Hugging Face Daily Papers

Vision-RL² is a method for improving fine-grained perception in multimodal large language models by using region-level reinforcement learning to compress visual tokens and enhance performance across multiple benchmarks without full model fine-tuning.

Balancing Multimodal Learning through Label Space Reshaping

arXiv cs.LG

Introduces Balanced Multimodal Label Reshaping (BMLR), a method that addresses modality imbalance in multimodal learning by reshaping the label space to equalize mapping difficulty across modalities, improving performance across various architectures.

Context-Aware RL for Agentic and Multimodal LLMs

Hugging Face Daily Papers

Introduces ContextRL, a reinforcement learning approach that teaches LLMs to identify which context supports an answer, achieving gains on agentic and multimodal benchmarks.