Paper page - DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AI

Source: https://huggingface.co/papers/2512.16676 Published on Dec 18, 2025

#1 Paper of the day Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

DataFlow is an LLM-driven data preparation framework that enhances data quality and reproducibility for various tasks, improving LLM performance with automatically generated pipelines.

The rapidly growing demand for high-quality data inLarge Language Models (LLMs)has intensified the need for scalable, reliable, and semantically richdata preparation pipelines. However, current practices remain dominated by ad-hoc scripts and loosely specified workflows, which lack principled abstractions, hinder reproducibility, and offer limited support for model-in-the-loop data generation. To address these challenges, we presentDataFlow, a unified and extensible LLM-driven data preparation framework.DataFlowis designed withsystem-level abstractionsthat enable modular, reusable, and composable data transformations, and provides aPyTorch-style pipeline construction APIfor building debuggable and optimizabledataflows. The framework consists of nearly 200reusable operatorsand sixdomain-general pipelinesspanning text, mathematical reasoning, code,Text-to-SQL,agentic RAG, andlarge-scale knowledge extraction. To further improve usability, we introduceDataFlow-Agent, which automatically translates natural-language specifications into executable pipelines viaoperator synthesis,pipeline planning, anditerative verification. Across six representative use cases,DataFlowconsistently improves downstream LLM performance. Our math, code, and text pipelines outperform curated human datasets and specialized synthetic baselines, achieving up to +3\% execution accuracy inText-to-SQLover SynSQL, +7\% average improvements on code benchmarks, and 1--3 point gains on MATH, GSM8K, and AIME. Moreover, a unified 10K-sample dataset produced byDataFlowenables base models to surpass counterparts trained on 1M Infinity-Instruct data. These results demonstrate thatDataFlowprovides a practical and high-performance substrate for reliable, reproducible, and scalable LLM data preparation, and establishes a system-level foundation for future data-centric AI development.

View arXiv pageView PDFProject pageGitHub3.65kAdd to collection

Get this paper in your agent:

hf papers read 2512\.16676

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2512.16676 in a model README.md to link it from this page.

Datasets citing this paper9

#### OpenDCAI/dataflow-demo-Text2SQL Preview• UpdatedFeb 4 • 1.11k • 1 #### OpenDCAI/dataflow-mm-context_vqa Viewer• UpdatedJan 30 • 206k • 372 #### OpenDCAI/dataflow-instruct-10k Viewer• UpdatedDec 29, 2025 • 10k • 259 • 6 #### OpenDCAI/Infinity-Instruct-Curated-1M Viewer• UpdatedMar 16 • 1M • 113 Browse 9 datasets citing this paper### Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2512.16676 in a Space README.md to link it from this page.

Collections including this paper18

Browse 18 collections that include this paper