Target-Oriented Pretraining Data Selection via Neuron-Activated Graph
Summary
Introduces Neuron-Activated Graph Ranking, a training-free method that uses sparse high-impact neuron sets to select pretraining data for target tasks, boosting average benchmark performance by 4.9%.
View Cached Full Text
Cached at: 04/22/26, 06:17 AM
Paper page - Target-Oriented Pretraining Data Selection via Neuron-Activated Graph
Source: https://huggingface.co/papers/2604.15706
Abstract
A novel target-oriented language model pretraining framework uses neuron activation graphs to select informative data without additional training, demonstrating superior performance across multiple benchmarks.
Everyday tasks come with a target, and pretraining models around this target is what turns them into experts. In this paper, we study target-oriented language model (LM) pretraining by introducingNeuron-Activated Graph Ranking(NAG-based Ranking), a training-free and interpretable framework for target pretrainingdata selection. Rather than using black-box representations, our approach directly characterizes each target input by asparse set of high-impact neuronsin anyoff-the-shelf LLMs. Concretely, we quantifyneuron impactand select the most influential neurons across layers into a compactNeuron-Activated Graph(NAG), and rank candidate data byNAG similaritytotarget examples. We conduct experiments across six benchmarks, where our NAG-based Ranking improves target-oriented pretraining by 4.9% on average over random sampling, and also outperforms state-of-the-art baselines by 5.3% accuracy on HellaSwag. It also remains effective under a more applicablemulti-target setting, where our best setup surpasses two baselines by 1.1% and 4.1%, respectively. Furthermore, we provide a comprehensive analysis on why and how our NAG works, e.g., deactivating NAG-selected neurons (only 0.12% of all) causes a 23.5% performance collapse, and restricting NAG to the final layer incurs a 4.1% average drop, indicating that NAG captures a sparse “functional backbone” for learning target features. We release the code at https://github.com/asillycat/NAG.
View arXiv pageView PDFProject pageGitHub4Add to collection
Get this paper in your agent:
hf papers read 2604\.15706
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2604.15706 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2604.15706 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2604.15706 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Target-Oriented Pretraining Data Selection via Neuron-Activated Graph
This paper introduces Neuron-Activated Graph (NAG) Ranking, a training-free framework for selecting pretraining data aligned with target tasks by identifying and ranking candidate data based on similarity in neuron activation patterns. The approach achieves 4.9% average improvement over random sampling and demonstrates that sparse neuron patterns capture functional capabilities for target learning.
GRASP: Geometry-aware Residual Alignment for Scalable Pretraining Data Attribution
GRASP introduces a geometry-aware, interaction-based method for scalable pretraining data attribution that models subset dynamics, outperforming existing additive approaches by over double the task-level rank correlation while reducing computation costs.
Beyond the Golden Teacher: Enhancing Graph Learning through LLM-GNN Co-teaching
This paper proposes LLM-GNN Co-Teaching, a bidirectional framework for few-shot graph learning on text-attributed graphs. The LLM and GNN exchange confident pseudo-labels and use round-based preference optimization (RPL-PO) to mutually improve, outperforming prior methods on benchmarks.
Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning
This paper introduces Super and Supra, sparse parameter-efficient fine-tuning methods that reuse pruning-inspired saliency signals (like Wanda scores) to select trainable supports, and combine them with LoRA to achieve high accuracy on arithmetic tasks with reduced memory and compute.
A Temporally Augmented Graph Attention Network for Affordance Classification
EEG-tGAT is a temporally augmented Graph Attention Network that improves affordance classification from interaction sequences by incorporating temporal attention and dropout mechanisms. The model enhances GATv2 for sequential data where temporal dimensions are semantically non-uniform.