DataArc-SynData-Toolkit: A Unified Closed-Loop Framework for Multi-Path, Multimodal, and Multilingual Data Synthesis

# DataArc-SynData-Toolkit: A Unified Closed-Loop Framework for Multi-Path, Multimodal, and Multilingual Data Synthesis
Source: [https://arxiv.org/html/2605.08138](https://arxiv.org/html/2605.08138)
Zhichao Shi1,2,3,4,Cehao Yang1,2,511footnotemark:1,Hao Zhou1,211footnotemark:1,Xiaojun Wu1,2,5, Huajie Li1,2,Xuhui Jiang1,2,Chengjin Xu1,222footnotemark:2,Yuanzhuo Wang4Jian Guo2 1DataArc Tech Ltd\.,2IDEA Research, International Digital Economy Academy, 3School of Advanced Interdisciplinary Sciences, UCAS,4Institute of Computing Technology, CAS, 5The Hong Kong University of Science and Technology \(Guangzhou\) [https://github\.com/DataArcTech/DataArc\-SynData\-Toolkit](https://github.com/DataArcTech/DataArc-SynData-Toolkit)Both authors contributed equally to this research\.Correspondence: jiangxuhui@dataarctech\.com; xuchengjin@dataarctech\.com

###### Abstract

Synthetic data has emerged as a crucial solution to the data scarcity bottleneck in large language models \(LLMs\), particularly for specialized domains and low\-resource languages\. However, the broader adoption of existing synthetic data tools is severely hindered by convoluted workflows, fragmented data standards, and limited scalability across modalities\. To address these limitations, we develop DataArc\-SynData\-Toolkit, an open\-source framework featuring: \(1\) a configuration\-driven, end\-to\-end pipeline equipped with an intuitive visual interface and simplified CLI for exceptional usability; \(2\) a unified, quality\-controllable synthesis paradigm that standardizes multi\-source data generation to ensure high reusability; and \(3\) a highly modular architecture designed for seamless multimodal, multilingual, and multi\-task adaptation\. We apply the toolkit in multiple application scenarios\. Experimental results demonstrate that our toolkit achieves an optimal balance between generation efficiency and data quality\. By offering an end\-to\-end and visually interactive pipeline, DataArc\-SynData\-Toolkit significantly lowers the technical barrier to synthetic data generation and subsequent model training, accelerating its practical deployment in real\-world applications\.

DataArc\-SynData\-Toolkit: A Unified Closed\-Loop Framework for Multi\-Path, Multimodal, and Multilingual Data Synthesis

Zhichao Shi1,2,3,4††thanks:Both authors contributed equally to this research\., Cehao Yang1,2,511footnotemark:1, Hao Zhou1,211footnotemark:1, Xiaojun Wu1,2,5,Huajie Li1,2,Xuhui Jiang1,2††thanks:Correspondence: jiangxuhui@dataarctech\.com; xuchengjin@dataarctech\.com,Chengjin Xu1,222footnotemark:2,Yuanzhuo Wang4Jian Guo21DataArc Tech Ltd\.,2IDEA Research, International Digital Economy Academy,3School of Advanced Interdisciplinary Sciences, UCAS,4Institute of Computing Technology, CAS,5The Hong Kong University of Science and Technology \(Guangzhou\)[https://github\.com/DataArcTech/DataArc\-SynData\-Toolkit](https://github.com/DataArcTech/DataArc-SynData-Toolkit)

## 1Introduction

The rapid evolution of Large Language Models \(LLMs\) has exponentially driven the demand for high\-quality training corpora\(OpenAI,[2023](https://arxiv.org/html/2605.08138#bib.bib1); Touvronet al\.,[2023](https://arxiv.org/html/2605.08138#bib.bib2)\)\. However, acquiring real\-world data at scale remains a formidable challenge, particularly for specialized domains and low\-resource languages\(Baratiet al\.,[2025](https://arxiv.org/html/2605.08138#bib.bib3); Abdallaet al\.,[2025](https://arxiv.org/html/2605.08138#bib.bib4)\)\. Consequently, synthetic data generation has become a critical paradigm for expanding training corpora\. While several pioneering frameworks like Synthetic Data\-RL\(Guoet al\.,[2025](https://arxiv.org/html/2605.08138#bib.bib5)\)and DataFlow\(Lianget al\.,[2025](https://arxiv.org/html/2605.08138#bib.bib6)\)have significantly advanced the field, their primary design often leans towards algorithmic exploration rather than out\-of\-the\-box, end\-to\-end engineering usability\.

Table 1:Feature comparison of DataArc with other open\-source synthetic data frameworks\.Data Prep\.refers to Data Preprocessing;Struct\.andMulti\.denote Structured Data and Multimodal Data, respectively\. For Synthesis Strategies, “Local” indicates synthesis based on local corpora, “Distill” refers to knowledge distillation, and “Web” denotes synthesis via web search\. As shown, DataArc is the only framework offering an end\-to\-end pipeline covering multiple strategies, multimodal synthesis, and downstream model fine\-tuning\.Specifically, as shown in Table[1](https://arxiv.org/html/2605.08138#S1.T1), current synthetic data tools face three major challenges in practical scenarios: \(1\)Steep Learning Curves and Limited Usability\. Existing tools cater primarily to AI experts\. Users lacking specialized backgrounds struggle to navigate the fragmented processes of data generation, model training, and validation, often resorting to complex, multi\-tool scripting\. \(2\)Fragmented Workflows and Unstandardized Pipelines\. There is a distinct lack of a unified paradigm for handling multi\-source data\(Chenet al\.,[2024](https://arxiv.org/html/2605.08138#bib.bib7)\)\. This fragmentation leads to unpredictable data quality and severely limits cross\-project reusability\. \(3\)Restricted Modality and Language Scalability\. Most frameworks lack generic interfaces for multimodal and low\-resource language synthesis, resulting in prohibitively high extension and maintenance costs\.

To address these challenges, we proposeDataArc\-SynData\-Toolkit, a configuration\-driven end\-to\-end system for data synthesis, model training, and evaluation\. The toolkit unifies data collection, synthesis, quality control, post\-training, and evaluation into a single closed\-loop workflow that supports continuous iteration\. Users can construct complex synthetic datasets and optimize models through concise configuration files\. In addition, the toolkit encapsulates execution logic into simplified commands and provides an intuitive visual interface, which significantly improves usability\.

The toolkit adopts a highly modular and agent\-based system design\. It unifies the synthesis pipelines and data formats for multi\-source synthetic data, including local corpora, open\-source web datasets, and model distillation\. This design ensures high data quality and strong reusability\. Furthermore, the system abstracts unified adaptation interfaces for multimodal and multilingual data, greatly reducing the cost of future extensions\.

In summary, DataArc\-SynData\-Toolkit is designed with usability, standardization, and scalability at its core\. Its main contributions are as follows:

- •We introduce a configuration\-driven, closed\-loop system encompassing data synthesis, model training, and evaluation, accessible via a streamlined CLI and an intuitive UI to democratize synthetic data generation\.
- •We design a unified, standardized paradigm for multi\-source data synthesis equipped with robust quality\-control mechanisms, thereby guaranteeing data reliability and reusability\.
- •We deliver a highly extensible, modular framework inherently capable of multimodal and multilingual synthesis, significantly lowering the engineering overhead for future academic and industrial adaptations\.

## 2Framework

![Refer to caption](https://arxiv.org/html/2605.08138v1/figures/framework.png)Figure 1:The overview of DataArc\-SynData\-Toolkit\.User side: Users only need to set configurations in the file or the visual interface to obtain the desired synthetic data, models, and evaluation results\.System Side: The toolkit divides the end\-to\-end pipeline into three stages: \(1\) Data synthesis: synthesize data based on the configuration; \(2\) Quality control: filter and rewrite the data to ensure quality; \(3\) Model post\-training and evaluation\.We aim to develop a user\- and developer\-friendly synthetic data toolkit\. The toolkit should provide a highly extensible framework for developers, and offer users an end\-to\-end pipeline and an easy\-to\-use operational paradigm\.

Therefore, DataArc\-SynData\-Toolkit adopts a unified and modular agent\-based design\. We generalize and abstract the process from multi\-source synthetic data to model outputs\. Based on the convergence nodes in the pipeline, and considering the mergeability of different steps, we divide the entire end\-to\-end workflow into three key stages: data synthesis, data quality control, and model post\-training and evaluation, as shown in Figure[1](https://arxiv.org/html/2605.08138#S2.F1)\. In this section, we introduce the design of DataArc\-SynData\-Toolkit for implementing the key stages\.

### 2\.1Data Synthesis

To address the lack of unified workflows and data standards in existing tools, DataArc\-SynData\-Toolkit abstracts a unified data synthesis paradigm, includingtask parsing,data source preparation,constraint construction,data acquisition, andstructured processing\. \(1\)Task parsingparses the task definitions into information like task instructions and domain boundaries\. \(2\)Data source preparationselects and prepares data sources based on the configuration, including local corpora, web datasets, or models to be distilled\. \(3\)Constraint constructionbuilds constraints for data synthesis, including reference passages, field limitations, or pattern constraints\. \(4\)Data acquisitiongenerates or filters the required data from the selected sources based on the parsed information and the defined constraints\. \(5\)Structured processingnormalizes the synthesized data into a unified format\.

Based on this unified paradigm, DataArc\-SynData\-Toolkit designsBaseTaskExecutor\. As shown in Figure[2](https://arxiv.org/html/2605.08138#S2.F2),BaseTaskExecutorhas implemented basic execution logic\. Developers can flexibly extend different synthesis paths by overriding the abstract functions\. To ensure data normalization and reusability, the toolkit processes all the data into the same format in Functionstructure\_process\.

We have implemented three paths: local corpus–driven synthesis \(LocalTaskExecutor\), web dataset–based filtering \(WebTaskExecutor\), and instruction distillation–based synthesis \(DistillTaskExecutor\), which cover the common synthesis paths in current practice\.

fromsdgsystem\.tasksimportBaseTaskExecutor

executor=BaseTaskExecutor\(configurations,llm\)

parsed\_information=executor\.task\_parsing\(\)

sources=executor\.prepare\(\)

constraints=executor\.construct\_constraints\(parsed\_information\)

raw\_data=executor\.data\_acquisition\(

sources,

constraints,

parsed\_information

\)

syn\_data=executor\.structure\_process\(raw\_data\)

Figure 2:The code implementation for data synthesis in abstract classBaseTaskExecutor#### 2\.1\.1Local Task

This path synthesizes data based on local corpora\. Functiontask\_parsingextracts keywords from the task instruction and domain in configurations for retrieval\. Functionprepareloads the retriever for the local corpora and prepares the generator for synthesis\. Functionconstruct\_constraintsinvokes the retriever to retrieve reference passages from local corpora using methods such as BM25\(Robertsonet al\.,[2009](https://arxiv.org/html/2605.08138#bib.bib8)\)or dense methods\(Karpukhinet al\.,[2020](https://arxiv.org/html/2605.08138#bib.bib9); Khattab and Zaharia,[2020](https://arxiv.org/html/2605.08138#bib.bib10)\)\. Functiondata\_acquisitioncombines the task instruction, reference passages, format constraints, and optional examples to form a prompt\. The LLM generator is prompted to synthesize data\.

#### 2\.1\.2Web Task

This path selects required data from open\-source web datasets\. Functiontask\_parsingis the same as Local Task\. Functionpreparesearches candidate datasets from open platforms \(e\.g\., HuggingFace\) based on the extracted keywords\. Functionconstruct\_constraintsadopts the LLM to select the data fields that best match the task based on configurations\. Functiondata\_acquisitionextracts the contents in the selected fields, and scores the dataset by the task consistency and quality of these contents\. The final data are preferentially sampled from datasets with higher scores until the required data volume is reached\.

#### 2\.1\.3Distill Task

This path distills data from a stronger teacher model based on the task instruction\. Functiontask\_parsingparses the configurations to get parameters for the teacher model\. Functionpreparedeploys the teacher model\. Functionconstruct\_constraintsqueries the teacher model to extract high\-quality generation patterns from task instruction and optional examples\. Functiondata\_acquisitionprompts the teacher model with task instruction, pattern constraints, and format constraints to generate data\.

### 2\.2Data Quality Control

Caused by factors such as inappropriate sample difficulty\(Tonget al\.,[2024](https://arxiv.org/html/2605.08138#bib.bib11); Ferracciet al\.,[2024](https://arxiv.org/html/2605.08138#bib.bib12)\), directly training models on initial synthetic data may lead to degraded performance\. Therefore, DataArc\-SynData\-Toolkit designs a quality control stage to ensure the data quality\. This stage is divided into two key functions: evaluation and rewrite\. Through evaluation, rewriting, and re\-evaluation, high\-quality synthetic data is generated\.

To support extensibility across different algorithms, we highly abstract the evaluation and rewriting steps into base classesEvaluatorandBaseRewriter\. The evaluator takes raw synthetic data as input and outputs corresponding evaluation scores\. The rewriter takes the raw data and their scores as input and produces rewritten samples based on these scores\. Developers can override the abstract functions and design different strategies, enabling flexible extension of quality control policies\. An example of the code implementation for quality control in the toolkit is provided below\.

fromsdgsystem\.evaluationimportEvaluator

fromsdgsystem\.generation\.rewriterimportDifficultyAdjustRewriter

evaluator=Evaluator\(configs,base\_model,llm\)

rewriter=DifficultyAdjustRewriter\(configs,llm\)

init\_eval=evaluator\.evaluate\(dataset\)

rewritten\_dataset=rewriter\.rewrite\(

dataset,

init\_eval,

\)

final\_eval=evaluator\.evaluate\(

rewritten\_dataset

\)

solved,learnable,unsolved\\

=rewritten\_dataset\.categorize\_by\_score\(final\_eval\)

Figure 3:An example of code implementation for quality control strategy in the toolkit\.
### 2\.3Model Post\-Training and Evaluation

To facilitate users in examining the synthetic data’s training effectiveness, DataArc\-SynData\-Toolkit integrates model post\-training and evaluation modules, establishing a closed\-loop model iteration\.

For training, the toolkit integrates a training module powered by verl\(Shenget al\.,[2025](https://arxiv.org/html/2605.08138#bib.bib13)\), enabling users to train models directly on their synthesized data\. The toolkit supports two training methods: Supervised Fine\-tuning \(SFT\) and Group Relative Policy Optimization \(GRPO\)\(Shaoet al\.,[2024](https://arxiv.org/html/2605.08138#bib.bib14)\)\.

For evaluation, the toolkit provides a model evaluation module powered by DeepEval111[https://github\.com/confident\-ai/deepeval](https://github.com/confident-ai/deepeval)\. This module enables users to evaluate post\-trained models on user\-provided evaluation datasets and applies G\-Eval\(Liuet al\.,[2023](https://arxiv.org/html/2605.08138#bib.bib15)\)to judge model responses\. The toolkit supports three metrics: Answer Correctness, Format Compliance, and Pairwise Preference\.

### 2\.4Other Extensible Modules

To make it easy for developers to integrate various algorithms and strategies into the toolkit, we adopt a highly modular design\. In the previous subsections, we described the agent\-based abstraction and encapsulation of the three key stages of the workflow\. In this section, we introduce other extensible modules, with a focus on multimodal and multilingual adaptation modules\.

For the multimodal adapter, we developedMMProjector, which executes the task of synthesizing multimodal instructions after being provided with seed image data for cold\-start initialization\. The base model and pipeline\-driven model are switched to their multimodal versions; consequently, the final synthesized instructions include not onlyinputandoutput, but also the corresponding seed images\. This component can be optionally activated when users provide seed images or when processing local rich\-text data\.

For the multilingual adapter, we designBaseTranslator, which performs language conversion in the final step of quality control\. The toolkit currently integrates adapters for low\-resource languages such as Arabic\.

In addition, we abstract the execution logic for sequential data and designParallelExecutor\. This executor encapsulates parallelization, temporary interruption, data caching, and resumption from existing progress\. Developers can call Functionexecute\(\.\.\.\)ofParallelExecutorto perform more efficient and reliable processing of sequential data, as shown in Figure[4](https://arxiv.org/html/2605.08138#S2.F4)\.

fromsdgsystem\.parallelimportParallelExecutor

executor=ParallelExecutor\(n\_workers=10\)

defprocess\(\.\.\.\):\.\.\.

outputs=executor\.execute\(

iterable\_inputs=data,

process\_function=process,

\.\.\.

\)

Figure 4:An example ofParallelExecutor\.
### 2\.5Simplified Usage

DataArc\-SynData\-Toolkit is designed with usability as the primary goal, enabling even users without professional backgrounds to easily synthesize data\. Therefore, the toolkit provides highly simplified commands for CLI and an intuitive visual interface to launch the workflow\.

The toolkit offers simplified commands to start the pipeline\. We use uv222[https://github\.com/astral\-sh/uv](https://github.com/astral-sh/uv)as the package manager and unify the command format\. Users can complete the whole pipeline with simple commands:

uvrunsdggenerateconfigs/sdg\.yaml

uvrunsdgtrainconfigs/\[sft\|grpo\]\.yaml

uvrunsdgevalconfigs/eval\.yaml

Figure 5:Simplified commands for quick start in CLI\.In addition, we provide a visual interface to facilitate task configuration, data management, and model management for users\. The toolkit adopts a frontend–backend separated architecture, featuring a FastAPI backend and a React frontend for improved visualization and scalability\. The detailed design is described in Appendix[A](https://arxiv.org/html/2605.08138#A1)\.

The samples of data synthesized by our toolkit are detailed in Appendix[B](https://arxiv.org/html/2605.08138#A2)\. Detailed deployment instructions, command usage, and tutorial videos for the visual interface are available in our GitHub repository333[https://github\.com/DataArcTech/DataArc\-SynData\-Toolkit](https://github.com/DataArcTech/DataArc-SynData-Toolkit)\.

## 3Performance Evaluation

In this section, we systematically evaluate the performance of DataArc\-SynData\-Toolkit, including the performance across different domains, multi\-lingual, multi\-modal, and its efficiency\.

### 3\.1Benchmarks

We conduct experiments on domain\-specific, multi\-lingual, and multi\-modal benchmarks\.

- •Domain\-specific benchmarks:MedQA\(Jinet al\.,[2021](https://arxiv.org/html/2605.08138#bib.bib16)\)contains multiple\-choice questions collected from medical board exams in the US and China\. We adopt the subset MedQA\-USMLE\-4\-options444[https://huggingface\.co/datasets/GBaker/MedQA\-USMLE\-4\-options](https://huggingface.co/datasets/GBaker/MedQA-USMLE-4-options)of 1,273 questions\.Flare\-CFA555[https://huggingface\.co/datasets/TheFinAI/flare\-cfa](https://huggingface.co/datasets/TheFinAI/flare-cfa)comprises 1,032 multiple\-choice questions covering CFA exam levels I and II\.LexEval\(Liet al\.,[2024](https://arxiv.org/html/2605.08138#bib.bib17)\)evaluates LLM performance in legal applications, and 500 multiple\-choice questions involving multi\-hop reasoning are adopted\.
- •
- •Multi\-modal benchmarks:FinMME\(Luoet al\.,[2025](https://arxiv.org/html/2605.08138#bib.bib19)\)contains high\-quality financial research samples across 18 domains, and about 1,850 numerical questions are selected\.TableVQA\(Kimet al\.,[2024](https://arxiv.org/html/2605.08138#bib.bib20)\)contains table question\-answering and table structure recognition datasets of 1,500 samples\.

Table 2:The main experiment results\.
### 3\.2Implementation Details

For each benchmark, we collect seed data from HuggingFace for cold start according to the corresponding task definition\. We sample query terms from the extracted keyword set and, by default, select 5 candidate relevant datasets to ensure diversity in data distribution\. Subsequently, we sample and filter 25 data instances and clean them into seed data based on the task definition\. For multi\-modal tasks, we will collect seed images for cold\-start\. We then usegpt\-5\-minito synthesize the initial instruction data and submit it to a quality management control pipeline powered bygpt\-5\-mini\. The resulting curated datasets are constructed at specified scales for downstream training, namely 1,000 / 2,000 / 4,000 samples\. We validate our toolkit on a wide range of LLMs, including Qwen2\.5\-7B\-Instruct and Qwen3\-4B for domain and multi\-lingual tasks, and Qwen2\.5\-VL\-7B\-Instruct and Qwen3\-VL\-4B\-Instruct for multi\-modal tasks\. All experiments are conducted on an Ubuntu machine equipped with eight NVIDIA H800\-80G GPUs\.

### 3\.3Evaluation Results

![Refer to caption](https://arxiv.org/html/2605.08138v1/figures/efficiency.png)Figure 6:The efficiency of ourParallelExecutordesign in toolkit when synthesizing 500 samples\.##### Consistent Performance Gains Across Models\.

As shown in Table[2](https://arxiv.org/html/2605.08138#S3.T2), synthetic data generated by our toolkit consistently improves performance across different models and task settings\. For both Qwen2\.5\-7B and Qwen3\-4B, training on synthesized samples yields substantial accuracy gains over the corresponding base models\. For instance, Qwen2\.5\-7B improves from 42\.34 to 68\.12 on MedQA and from 19\.80 to 46\.15 on LexEval with 4,000 synthetic samples\. Similar trends are observed for Qwen3\-4B, which increases from 57\.52 to 70\.62 on MedQA and from 31\.48 to 48\.00 on LexEval\. These results demonstrate that the proposed toolkit provides robust and architecture\-agnostic gains, validating its effectiveness across heterogeneous model scales and modalities\.

##### Scaling Synthetic Data Enhances Performance Gains\.

Larger synthetic datasets lead to more pronounced performance gains\. Across nearly all benchmarks, accuracy steadily increases as the training data size expands from 1,000 to 2,000 and 4,000 samples\. For example, on Qwen2\.5\-7B, MedQA accuracy rises from 59\.02 to 64\.57 and further to 68\.12, while FinMME improves from 24\.90 to 32\.47\. The scaling behavior suggests strong potential for further gains with increased synthetic data budgets, highlighting the practicality of synthetic data as a controllable and efficient alternative to costly human annotation\.

##### Parallel Design Significantly Boosts Synthesis Efficiency\.

Experimental results in Figure[6](https://arxiv.org/html/2605.08138#S3.F6)demonstrate that adjusting the number of workers in the toolkit’sParallelExecutoryields substantial improvements in synthesis efficiency\. Without parallelization, generating merely 500 initial samples requires 909 seconds\. As the number of active workers increases, the required time decreases markedly\. Since all modules within the pipeline benefit from the parallel design, including both stages of the rewriter \(i\.e\., generation and validation\), scaling the worker count to just 10 or 25 achieves significant performance gains\.

## 4Conclusion and Future Work

We propose DataArc\-SynData\-Toolkit, a configuration\-driven, end\-to\-end open\-source toolkit for data synthesis\. The toolkit provides a highly extensible, unified, and quality\-controllable framework for multi\-path, multimodal, multilingual data synthesis\. For the next release, we plan to enable encrypted synthetic data generation to protect sensitive information for secure model training\. Future work will focus on incorporating more algorithms, modalities, and languages to offer broader options\.

## Limitations

Due to the rapid evolution of synthetic data methods, our toolkit currently integrates only a subset of data synthesis and validation approaches, such as the difficulty\-based data rewriting method shown in Figure[3](https://arxiv.org/html/2605.08138#S2.F3)\. However, our analysis of the data synthesis pipeline enables the toolkit to provide a highly abstract, extensible, and unified framework, with careful system optimizations\. New methods can be easily integrated into the toolkit, which provides users with more convenient options\.

## References

- The future of artificial intelligence in the face of data scarcity\.Computers, Materials and Continua84\(1\),pp\. 1073–1099\.Cited by:[§1](https://arxiv.org/html/2605.08138#S1.p1.1)\.
- I\. Barati, M\. Amiri, and H\. Faili \(2025\)SearchInstruct: enhancing domain adaptation via retrieval\-based instruction dataset creation\.arXiv preprint arXiv:2509\.10708\.Cited by:[§1](https://arxiv.org/html/2605.08138#S1.p1.1)\.
- D\. Chen, Y\. Huang, Z\. Ma, H\. Chen, X\. Pan, C\. Ge, D\. Gao, Y\. Xie, Z\. Liu, J\. Gao,et al\.\(2024\)Data\-juicer: a one\-stop data processing system for large language models\.InCompanion of the 2024 International Conference on Management of Data,pp\. 120–134\.Cited by:[§1](https://arxiv.org/html/2605.08138#S1.p2.1)\.
- T\. Ferracci, L\. T\. Goldmann, A\. Hinel, and F\. S\. Passino \(2024\)Targeted synthetic data generation for tabular data via hardness characterization\.arXiv preprint arXiv:2410\.00759\.Cited by:[§2\.2](https://arxiv.org/html/2605.08138#S2.SS2.p1.1)\.
- Y\. Guo, Z\. Guo, C\. Huang, Z\. Wang, Z\. Zhang, H\. Yu, H\. Zhang, and Y\. Shen \(2025\)Synthetic data rl: task definition is all you need\.arXiv preprint arXiv:2505\.17063\.Cited by:[§1](https://arxiv.org/html/2605.08138#S1.p1.1)\.
- D\. Jin, E\. Pan, N\. Oufattole, W\. Weng, H\. Fang, and P\. Szolovits \(2021\)What disease does this patient have? a large\-scale open domain question answering dataset from medical exams\.Applied Sciences11\(14\),pp\. 6421\.Cited by:[1st item](https://arxiv.org/html/2605.08138#S3.I1.i1.p1.1)\.
- V\. Karpukhin, B\. Oguz, S\. Min, P\. S\. Lewis, L\. Wu, S\. Edunov, D\. Chen, and W\. Yih \(2020\)Dense passage retrieval for open\-domain question answering\.\.InEMNLP \(1\),pp\. 6769–6781\.Cited by:[§2\.1\.1](https://arxiv.org/html/2605.08138#S2.SS1.SSS1.p1.1)\.
- O\. Khattab and M\. Zaharia \(2020\)Colbert: efficient and effective passage search via contextualized late interaction over bert\.InProceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval,pp\. 39–48\.Cited by:[§2\.1\.1](https://arxiv.org/html/2605.08138#S2.SS1.SSS1.p1.1)\.
- Y\. Kim, M\. Yim, and K\. Y\. Song \(2024\)Tablevqa\-bench: a visual question answering benchmark on multiple table domains\.arXiv preprint arXiv:2404\.19205\.Cited by:[3rd item](https://arxiv.org/html/2605.08138#S3.I1.i3.p1.1)\.
- H\. Li, Y\. Chen, Q\. Ai, Y\. Wu, R\. Zhang, and Y\. Liu \(2024\)Lexeval: a comprehensive chinese legal benchmark for evaluating large language models\.Advances in Neural Information Processing Systems37,pp\. 25061–25094\.Cited by:[1st item](https://arxiv.org/html/2605.08138#S3.I1.i1.p1.1)\.
- H\. Liang, X\. Ma, Z\. Liu, Z\. H\. Wong, Z\. Zhao, Z\. Meng, R\. He, C\. Shen, Q\. Cai, Z\. Han,et al\.\(2025\)DataFlow: an llm\-driven framework for unified data preparation and workflow automation in the era of data\-centric ai\.arXiv preprint arXiv:2512\.16676\.Cited by:[§1](https://arxiv.org/html/2605.08138#S1.p1.1)\.
- Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. Zhu \(2023\)G\-eval: nlg evaluation using gpt\-4 with better human alignment\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 2511–2522\.Cited by:[§2\.3](https://arxiv.org/html/2605.08138#S2.SS3.p3.1)\.
- J\. Luo, Z\. Kou, L\. Yang, X\. Luo, J\. Huang, Z\. Xiao, J\. Peng, C\. Liu, J\. Ji, X\. Liu,et al\.\(2025\)Finmme: benchmark dataset for financial multi\-modal reasoning evaluation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 29465–29489\.Cited by:[3rd item](https://arxiv.org/html/2605.08138#S3.I1.i3.p1.1)\.
- OpenAI \(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2605.08138#S1.p1.1)\.
- K\. Ouda \(2025\)ABBL: An Advanced Benchmark and Leaderboard for Comprehensive Evaluation of Arabic Language Models\.SILMA\.AI\.Note:Accessed: 2025\-07\-19Cited by:[2nd item](https://arxiv.org/html/2605.08138#S3.I1.i2.p1.1)\.
- S\. Robertson, H\. Zaragoza,et al\.\(2009\)The probabilistic relevance framework: bm25 and beyond\.Foundations and trends® in information retrieval3\(4\),pp\. 333–389\.Cited by:[§2\.1\.1](https://arxiv.org/html/2605.08138#S2.SS1.SSS1.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§2\.3](https://arxiv.org/html/2605.08138#S2.SS3.p2.1)\.
- G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. Wu \(2025\)Hybridflow: a flexible and efficient rlhf framework\.InProceedings of the Twentieth European Conference on Computer Systems,pp\. 1279–1297\.Cited by:[§2\.3](https://arxiv.org/html/2605.08138#S2.SS3.p2.1)\.
- Y\. Tong, X\. Zhang, R\. Wang, R\. Wu, and J\. He \(2024\)Dart\-math: difficulty\-aware rejection tuning for mathematical problem\-solving\.Advances in Neural Information Processing Systems37,pp\. 7821–7846\.Cited by:[§2\.2](https://arxiv.org/html/2605.08138#S2.SS2.p1.1)\.
- H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.\(2023\)Llama: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§1](https://arxiv.org/html/2605.08138#S1.p1.1)\.

## Appendix ADetailed Interface Design

In this section, we show the detailed interface design of DataArc\-SynData\-Toolkit\. The tutorial video is available on GitHub or on YouTube777[https://www\.youtube\.com/watch?v=zIHH3YnZKr4&t=56s](https://www.youtube.com/watch?v=zIHH3YnZKr4&t=56s)\. The interface design follows the principles of intuitiveness, ease of use, and inspectable process details\. The toolkit provides interactive interfaces that cover the entire pipeline, including synthetic task configuration \(Figure[7](https://arxiv.org/html/2605.08138#A1.F7)\), synthesis visualization \(Figure[8](https://arxiv.org/html/2605.08138#A1.F8)\), model training configuration \(Figure[9](https://arxiv.org/html/2605.08138#A1.F9)\), and training visualization \(Figure[10](https://arxiv.org/html/2605.08138#A1.F10)\)\.

![Refer to caption](https://arxiv.org/html/2605.08138v1/figures/detail_interface/i1_task_config.png)Figure 7:Interface of synthetic task configurations\.As shown in Figure[7](https://arxiv.org/html/2605.08138#A1.F7), the configuration interface provides a panel for selecting the synthesis path \(Part 1\) and a panel for flexible task parameter configuration \(Part 2\)\. In Part 1, users can choose from Local, Web, or Distill, and upload local corpora \(Local\), set the HF\_token \(Web\), or configure a teacher model \(Distill\)\. In Part 2, users can set task instructions, synthesis quantity, language, and custom prompt templates\. The toolkit provides interaction options with high flexibility\.

![Refer to caption](https://arxiv.org/html/2605.08138v1/figures/detail_interface/i2_data_synthesis.png)Figure 8:Visualization of data synthesis workflow\.As shown in Figure[8](https://arxiv.org/html/2605.08138#A1.F8), the synthesis interface visualizes the data synthesis, validation, and rewriting steps\. In Part 1, users can view completed steps and the current execution step\. In Part 2, users can click on any piece of synthesized data to inspect its details \(e\.g\., input and output\) and download them\. In Part 3, users can monitor the synthesis progress and observe time and token consumption, which allows them to stop the process when needed\.

![Refer to caption](https://arxiv.org/html/2605.08138v1/figures/detail_interface/i3_train_config.png)Figure 9:Interface of model training\.![Refer to caption](https://arxiv.org/html/2605.08138v1/figures/detail_interface/i4_train_logs.png)Figure 10:Visualization of training and evaluation\.As shown in Figure[9](https://arxiv.org/html/2605.08138#A1.F9), the configuration interface is similar to the synthesis task configuration interface\. It provides a basic configuration panel for uploading models and data \(Part 1\) and a panel of detailed hyperparameter configurations \(Part 2\)\. As shown in Figure[10](https://arxiv.org/html/2605.08138#A1.F10), users can directly observe the backend logs, which help users observe training progress, error information, and final evaluation results\. The visualization of the training process allows users to inspect process details and report issues to developers when errors occur\.

Through these visualization interfaces, DataArc\-SynData\-Toolkit provides interactive support for task configuration, data synthesis, and model training\. It supports users with different technical backgrounds and offers high flexibility\. This design significantly improves the usability of the toolkit\.

## Appendix BExamples of Synthetic Data

In this section, we show samples of using our toolkit to synthesize financial data \(Figure[11](https://arxiv.org/html/2605.08138#A2.F11)\) and multimodal data \(Figure[12](https://arxiv.org/html/2605.08138#A2.F12)\)\.

![Refer to caption](https://arxiv.org/html/2605.08138v1/x1.png)Figure 11:A sample of synthesized instruction in the finance domain\.![Refer to caption](https://arxiv.org/html/2605.08138v1/x2.png)Figure 12:A sample of synthesized instruction in the multimodal domain\.