pre-training-data

UltraX proposes a function-calling refinement framework for large-scale pre-training data that introduces insertion alongside deletion and modification, enabling fine-grained instance-level editing. It builds a reliable program-supervision generation pipeline and demonstrates improved data efficiency and model performance when pretraining 1B models from scratch.