Target-Oriented Pretraining Data Selection via Neuron-Activated Graph

Hugging Face Daily Papers Papers
pretraining-data-selection neuron-activation-graph target-oriented-lm interpretability training-free

Summary

Introduces Neuron-Activated Graph Ranking, a training-free method that uses sparse high-impact neuron sets to select pretraining data for target tasks, boosting average benchmark performance by 4.9%.

Everyday tasks come with a target, and pretraining models around this target is what turns them into experts. In this paper, we study target-oriented language model (LM) pretraining by introducing Neuron-Activated Graph Ranking (NAG-based Ranking), a training-free and interpretable framework for target pretraining data selection. Rather than using black-box representations, our approach directly characterizes each target input by a sparse set of high-impact neurons in any off-the-shelf LLMs. Concretely, we quantify neuron impact and select the most influential neurons across layers into a compact Neuron-Activated Graph (NAG), and rank candidate data by NAG similarity to target examples. We conduct experiments across six benchmarks, where our NAG-based Ranking improves target-oriented pretraining by 4.9% on average over random sampling, and also outperforms state-of-the-art baselines by 5.3% accuracy on HellaSwag. It also remains effective under a more applicable multi-target setting, where our best setup surpasses two baselines by 1.1% and 4.1%, respectively. Furthermore, we provide a comprehensive analysis on why and how our NAG works, e.g., deactivating NAG-selected neurons (only 0.12% of all) causes a 23.5% performance collapse, and restricting NAG to the final layer incurs a 4.1% average drop, indicating that NAG captures a sparse "functional backbone" for learning target features. We release the code at https://github.com/asillycat/NAG.
Original Article
View Cached Full Text

Cached at: 04/22/26, 06:17 AM

Paper page - Target-Oriented Pretraining Data Selection via Neuron-Activated Graph

Source: https://huggingface.co/papers/2604.15706

Abstract

A novel target-oriented language model pretraining framework uses neuron activation graphs to select informative data without additional training, demonstrating superior performance across multiple benchmarks.

Everyday tasks come with a target, and pretraining models around this target is what turns them into experts. In this paper, we study target-oriented language model (LM) pretraining by introducingNeuron-Activated Graph Ranking(NAG-based Ranking), a training-free and interpretable framework for target pretrainingdata selection. Rather than using black-box representations, our approach directly characterizes each target input by asparse set of high-impact neuronsin anyoff-the-shelf LLMs. Concretely, we quantifyneuron impactand select the most influential neurons across layers into a compactNeuron-Activated Graph(NAG), and rank candidate data byNAG similaritytotarget examples. We conduct experiments across six benchmarks, where our NAG-based Ranking improves target-oriented pretraining by 4.9% on average over random sampling, and also outperforms state-of-the-art baselines by 5.3% accuracy on HellaSwag. It also remains effective under a more applicablemulti-target setting, where our best setup surpasses two baselines by 1.1% and 4.1%, respectively. Furthermore, we provide a comprehensive analysis on why and how our NAG works, e.g., deactivating NAG-selected neurons (only 0.12% of all) causes a 23.5% performance collapse, and restricting NAG to the final layer incurs a 4.1% average drop, indicating that NAG captures a sparse “functional backbone” for learning target features. We release the code at https://github.com/asillycat/NAG.

View arXiv pageView PDFProject pageGitHub4Add to collection

Get this paper in your agent:

hf papers read 2604\.15706

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2604.15706 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2604.15706 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2604.15706 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Target-Oriented Pretraining Data Selection via Neuron-Activated Graph

arXiv cs.CL

This paper introduces Neuron-Activated Graph (NAG) Ranking, a training-free framework for selecting pretraining data aligned with target tasks by identifying and ranking candidate data based on similarity in neuron activation patterns. The approach achieves 4.9% average improvement over random sampling and demonstrates that sparse neuron patterns capture functional capabilities for target learning.

Graph Machine: Towards Better Pretraining via Edges

Hugging Face Daily Papers

The paper introduces Graph Machine, a method to replace dense attention layers in transformers with sparse layers using dynamic pointers, improving efficiency and maintaining or enhancing performance during pretraining.

Signed Graph Pre-Training and Prompt Learning

arXiv cs.LG

The paper introduces TopoSIGN, a topology-guided graph pre-training and prompt learning framework for signed graphs, which combines structural encoding and persistent homology to improve transfer learning in tasks like link prediction and node classification.