DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models

Papers with Code Trending Papers
data-centric training framework large-language-models sample-selection domain-mixture reweighting

Summary

DataFlex is a unified framework for data-centric dynamic training of LLMs that integrates sample selection, domain mixture adjustment, and sample reweighting with compatibility for standard workflows and efficient large-scale deployment, achieving consistent improvements over static training.

Data-centric training has emerged as a promising direction for improving large language models (LLMs) by optimizing not only model parameters but also the selection, composition, and weighting of training data during optimization. However, existing approaches to data selection, data mixture optimization, and data reweighting are often developed in isolated codebases with inconsistent interfaces, hindering reproducibility, fair comparison, and practical integration. In this paper, we present DataFlex, a unified data-centric dynamic training framework built upon LLaMA-Factory. DataFlex supports three major paradigms of dynamic data optimization: sample selection, domain mixture adjustment, and sample reweighting, while remaining fully compatible with the original training workflow. It provides extensible trainer abstractions and modular components, enabling a drop-in replacement for standard LLM training, and unifies key model-dependent operations such as embedding extraction, inference, and gradient computation, with support for large-scale settings including DeepSpeed ZeRO-3. We conduct comprehensive experiments across multiple data-centric methods. Dynamic data selection consistently outperforms static full-data training on MMLU across both Mistral-7B and Llama-3.2-3B. For data mixture, DoReMi and ODM improve both MMLU accuracy and corpus-level perplexity over default proportions when pretraining Qwen2.5-1.5B on SlimPajama at 6B and 30B token scales. DataFlex also achieves consistent runtime improvements over original implementations. These results demonstrate that DataFlex provides an effective, efficient, and reproducible infrastructure for data-centric dynamic training of LLMs.
Original Article
View Cached Full Text

Cached at: 07/28/26, 06:27 AM

Paper page - DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models

Source: https://huggingface.co/papers/2603.26164 Published on Mar 27

#1 Paper of the day Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

DataFlex is a unified framework for dynamic data-centric training of large language models that supports sample selection, domain mixture adjustment, and sample reweighting while maintaining compatibility with standard training workflows and enabling efficient large-scale deployment.

Data-centric traininghas emerged as a promising direction for improvinglarge language models(LLMs) by optimizing not only model parameters but also the selection, composition, and weighting of training data during optimization. However, existing approaches to data selection, data mixture optimization, and data reweighting are often developed in isolated codebases with inconsistent interfaces, hindering reproducibility, fair comparison, and practical integration. In this paper, we present DataFlex, a unified data-centric dynamic training framework built uponLLaMA-Factory. DataFlex supports three major paradigms of dynamic data optimization:sample selection,domain mixture adjustment, andsample reweighting, while remaining fully compatible with the original training workflow. It providesextensible trainer abstractionsandmodular components, enabling a drop-in replacement for standard LLM training, and unifies key model-dependent operations such asembedding extraction, inference, andgradient computation, with support for large-scale settings includingDeepSpeed ZeRO-3. We conduct comprehensive experiments across multiple data-centric methods. Dynamic data selection consistently outperforms static full-data training onMMLUacross both Mistral-7B and Llama-3.2-3B. For data mixture, DoReMi and ODM improve bothMMLUaccuracy and corpus-levelperplexityover default proportions when pretrainingQwen2.5-1.5BonSlimPajamaat 6B and 30B token scales. DataFlex also achieves consistent runtime improvements over original implementations. These results demonstrate that DataFlex provides an effective, efficient, and reproducible infrastructure for data-centric dynamic training of LLMs.

View arXiv pageView PDFProject pageGitHub1.74kAdd to collection

Get this paper in your agent:

hf papers read 2603\.26164

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2603.26164 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2603.26164 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2603.26164 in a Space README.md to link it from this page.

Collections including this paper9

Browse 9 collections that include this paper

Similar Articles

Data Mixing for Large Language Models Pretraining: A Survey and Outlook

arXiv cs.CL

This paper presents a comprehensive survey of data mixing methods for LLM pretraining, formalizing the problem as bilevel optimization and introducing a taxonomy that distinguishes static (rule-based and learning-based) from dynamic (adaptive and externally guided) mixing approaches. The authors analyze trade-offs, identify cross-cutting challenges, and outline future research directions including finer-grained domain partitioning and pipeline-aware designs.

DataDignity: Training Data Attribution for Large Language Models

arXiv cs.AI

This paper introduces DataDignity, a framework and benchmark (FakeWiki) for pinpoint provenance, aiming to identify the specific training data sources that support an LLM's response. It proposes ScoringModel and SteerFuse methods to improve attribution accuracy over standard retrieval baselines.