DecoupleMix: Decoupled Ratio Search and Convex Allocation for Scalable VLM Data Recipes

Hugging Face Daily Papers Papers
vision-language-models data-curation mixture-optimization convex-optimization scalable pretraining

Summary

DecoupleMix introduces a systematic framework for optimizing pretraining data mixtures for Vision-Language Models by decoupling inter-class and intra-class ratio search, using convex optimization to improve scalability and performance over heuristic baselines.

While data curation for Vision Language Models (VLMs) is increasingly active, public practice for constructing pretraining mixtures remains largely heuristic: practitioners stack datasets that pass quality filters, set cross-domain ratios by intuition, and lack a principled, attributable criterion for admitting new data, while frontier recipes remain undisclosed. We formulate data construction as a systematic mixture-optimization problem and turn it into a reproducible engineering discipline by decoupling the mixture into two orthogonal sub-problems: inter-class ratios across capabilities and intra-class ratios within a category. For inter-class allocation, we use a single-variable iterative search; for intra-class composition, we apply a multidimensional, dataset-level assessment scoring Quality and Difficulty, and formulate selection as a constrained convex optimization with a diversity objective. The DecoupleMix framework delivers two critical capabilities: guiding what data to collect next and rendering dataset validation a controlled, attributable experiment. Experiments show our approach consistently surpasses heuristic baselines. Moreover, optimal ratios discovered on small-scale proxies transfer seamlessly to larger scales without retuning. Using 80B additional multimodal continue-pretraining tokens, our VLM is competitive with strong open-source models trained with substantially larger multimodal budgets.
Original Article
View Cached Full Text

Cached at: 07/28/26, 06:33 AM

Paper page - DecoupleMix: Decoupled Ratio Search and Convex Allocation for Scalable VLM Data Recipes

Source: https://huggingface.co/papers/2607.24516

Abstract

WhiledatacurationforVisionLanguageModels(VLMs)isincreasinglyactive,publicpracticeforconstructingpretrainingmixturesremainslargelyheuristic:practitionersstackdatasetsthatpassqualityfilters,setcross-domainratiosbyintuition,andlackaprincipled,attributablecriterionforadmittingnewdata,whilefrontierrecipesremainundisclosed.Weformulatedataconstructionasasystematicmixture-optimizationproblemandturnitintoareproducibleengineeringdisciplinebydecouplingthemixtureintotwoorthogonalsub-problems:inter-classratiosacrosscapabilitiesandintra-classratioswithinacategory.Forinter-classallocation,weuseasingle-variableiterativesearch;forintra-classcomposition,weapplyamultidimensional,dataset-levelassessmentscoringQualityandDifficulty,andformulateselectionasaconstrainedconvexoptimizationwithadiversityobjective.TheDecoupleMixframeworkdeliverstwocriticalcapabilities:guidingwhatdatatocollectnextandrenderingdatasetvalidationacontrolled,attributableexperiment.Experimentsshowourapproachconsistentlysurpassesheuristicbaselines.Moreover,optimalratiosdiscoveredonsmall-scaleproxiestransferseamlesslytolargerscaleswithoutretuning.Using80Badditionalmultimodalcontinue-pretrainingtokens,ourVLMiscompetitivewithstrongopen-sourcemodelstrainedwithsubstantiallylargermultimodalbudgets.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.24516

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.24516 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.24516 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.24516 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Data Mixing for Large Language Models Pretraining: A Survey and Outlook

arXiv cs.CL

This paper presents a comprehensive survey of data mixing methods for LLM pretraining, formalizing the problem as bilevel optimization and introducing a taxonomy that distinguishes static (rule-based and learning-based) from dynamic (adaptive and externally guided) mixing approaches. The authors analyze trade-offs, identify cross-cutting challenges, and outline future research directions including finer-grained domain partitioning and pipeline-aware designs.

Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Time

arXiv cs.CL

This paper introduces OP-Mix, a data mixing algorithm that uses low-rank adapters trained on the current model to cheaply simulate candidate data mixtures, enabling efficient and unified data mixing across pretraining, continual midtraining, and continual instruction tuning. OP-Mix consistently finds near-optimal mixtures while using a fraction of the compute of baselines, improving pretraining perplexity by 6.3% and reducing compute by 66-95% in continual learning settings.