EmbGen: Teaching with Reassembled Corpora
Summary
EmbGen is a synthetic data generation pipeline that reassembles corpora into entity-description pairs using embedding similarity to generate diverse QA pairs for fine-tuning small language models on specialized domains, showing significant improvements in factual accuracy.
View Cached Full Text
Cached at: 05/20/26, 08:25 AM
# EmbGen: Teaching with Reassembled Corpora Source: [https://arxiv.org/abs/2605.19394](https://arxiv.org/abs/2605.19394) [View PDF](https://arxiv.org/pdf/2605.19394) > Abstract:Adapting small instruction\-tuned models to specialized domains often relies on supervised fine\-tuning \(SFT\) on curated instruction\-response examples, which is expensive to collect at scale\. Synthetic training examples generated by a teacher LLM from a domain corpus can reduce this cost, but existing pipelines can produce homogenized outputs and do not consistently capture cross\-passage or cross\-document dependencies\. We introduce EmbGen, a synthetic data generation pipeline that decomposes a corpus into entity\-description pairs, reassembles them using semantic structure inferred from embedding similarity, and then generates question\-answer \(QA\) pairs via proximity, intra\-cluster, and inter\-cluster sampling with cluster\-specialized system prompts\. We evaluate EmbGen against EntiGraph, InstructLab and Knowledge\-Instruct on three datasets of varied semantic heterogeneity, under fixed token budgets \(5 and 20 million tokens\)\. We use lexical overlap metrics, an LLM\-as\-a\-judge rubric, and Binary Accuracy, a composed metric combining Factual Accuracy and Completeness for evaluation\. EmbGen improves Binary Accuracy on the most heterogeneous dataset by 12\.5% at 5M and 88\.9% at 20M tokens budget, relative to the strongest baseline, while remaining competitive across other datasets with lower heterogeneity\. ## Submission history From: Anna Leontjeva \[[view email](https://arxiv.org/show-email/cfdcde18/2605.19394)\] **\[v1\]**Tue, 19 May 2026 05:40:12 UTC \(2,573 KB\)
Similar Articles
EAR: Entity-Aware Partitioning Approach for Retrieval-Augmented Generation Development
The paper introduces EAR, an entity-aware partitioning approach for retrieval-augmented generation in multiple-choice question answering, which extracts anchors from questions and corpora to retrieve compact windows instead of fixed-size chunks.
Enhancing Biomedical Named Entity Recognition via Multiple Programming Languages Instruction Tuning and Ensemble Method
MITE reformulates biomedical NER as a structure-to-structure generation task by representing instructions and entity outputs in multiple programming-language formats (Python, C++, Java), using entity-level voting at inference to improve robustness. Experiments on six BioNER datasets show it outperforms BERT- and LLM-based baselines with strong cross-dataset generalization.
EAGER: Enhancing Generative Event Extraction via Reinforcement Learning with Verifiable Rewards
EAGER is a reinforcement learning framework that improves generative event extraction through fine-grained verifiable rewards and schema-contrastive advantage estimation, outperforming prompting, fine-tuning, and prior RL methods on seven benchmark datasets.
CogEvol: Towards Efficient and Reliable Learning Environment Generation
CogEvol is a family of models that efficiently generate structured learning artifacts like slides and interactive HTML pages in a single pass using supervised fine-tuning and reinforcement learning with vision-language rewards, reducing cost and improving reliability.
SEA-Embedding: Open and Reproducible Text Embeddings for Southeast Asia
SEA-Embedding presents a fully open and reproducible text embedding pipeline for Southeast Asian languages, trained solely on public data, achieving state-of-the-art results on the SEA-BED benchmark.