Target-Oriented Pretraining Data Selection via Neuron-Activated Graph

# Target-Oriented Pretraining Data Selection via Neuron-Activated Graph Source: https://arxiv.org/html/2604.15706 Haoqin TuWeidong ZhouYiyang ZhouXiaohuan ZhouBingni ZhangWeiguo FengTaifeng WangCihang XieFengze Liu ###### Abstract Everyday tasks come with a target, and pretraining models around this target is what turns them into experts\. In this paper, we study target\-oriented language model \(LM\) pretraining by introducing*Neuron\-ActivatedGraph Ranking*\(NAG\-based Ranking\), a training\-free and interpretable framework for target pretraining data selection\. Rather than using black\-box representations, our approach directly characterizes each target input by a sparse set of high\-impact neurons in any off\-the\-shelf LLMs\. Concretely, we quantify neuron impact and select the most influential neurons across layers into a compact*Neuron\-ActivatedGraph*\(NAG\), and rank candidate data by NAG similarity to target examples\. We conduct experiments across six benchmarks, where our NAG\-based Ranking improves target\-oriented pretraining by 4\.9% on average over random sampling, and also outperforms state\-of\-the\-art baselines by 5\.3% accuracy on HellaSwag\. It also remains effective under a more applicable multi\-target setting, where our best setup surpasses two baselines by 1\.1% and 4\.1%, respectively\. Furthermore, we provide a comprehensive analysis onwhyandhowour NAG works,*e\.g*\., deactivating NAG\-selected neurons \(only 0\.12% of all\) causes a 23\.5% performance collapse, and restricting NAG to the final layer incurs a 4\.1% average drop, indicating that NAG captures a sparse “functional backbone” for learning target features\. We release the code athttps://github.com/asillycat/NAG\. Large Language Models; Pretraining Data Selection Refer to captionFigure 1:General quality\-based data selection is often misaligned with specific downstream capabilities \(left\), while prior target\-oriented methods rely on shallow similarity to target examples \(middle left\)\. Our NAG instead aligns pretraining data with target tasks by selecting inputs that activate similar neurons in the LLM, capturing the underlying capability required for the target \(middle right\), even across different domains \(*e\.g*\., economics*vs*\.math\)\.Refer to captionFigure 2:Overview of Neuron\-Activated Graph \(NAG\) target\-oriented data selection\. Given a small set of target examplesDtargetD\_\{target\}, we first characterize each input by its neuron\-level NAG features\. For a given input, we quantify the impact of individual neurons and select top\-KKof them per layer to construct a compact NAG\. NAGs from the target examples are aggregated into a target neuron\-activation profile\. Each candidate samplec∈Dpoolc\\in D\_\{pool\}is then mapped to its own NAG and ranked by its similarity to the target profile,*i\.e*\.Sim\(c,Dtarget\)\\text\{Sim\}\(c,D\_\{target\}\)\. Finally, the top\-rfr\_\{f\}ranked samples are selected for LLM pretraining\.## 1Introduction Large language models \(LLMs\) have become increasingly prevalent in everyday tasks, and people usually use these models with specific targets in mind\. Selecting high\-quality pretraining data is one of the most effective ways to improve model performance within a target domain, yielding great gains\(Penedoet al\.,2024 (https://arxiv.org/html/2604.15706#bib.bib6); Mizrahiet al\.,2025 (https://arxiv.org/html/2604.15706#bib.bib7); Gunasekaret al\.,2023 (https://arxiv.org/html/2604.15706#bib.bib32); Sorscheret al\.,2023 (https://arxiv.org/html/2604.15706#bib.bib33)\)\. Despite its importance, what makes “high\-quality” data remains surprisingly under\-defined \(Fig\.1 (https://arxiv.org/html/2604.15706#S0.F1)\)\. We argue that, in real\-world settings, high\-quality data should align with targeted scenarios that enable LLMs to acquire desired capabilities efficiently — education, medicine, or specific research domains — while ruling out other irrelevant factors that do not contribute to such capabilities\(Mizrahiet al\.,2025 (https://arxiv.org/html/2604.15706#bib.bib7)\)\. However, existing pipelines for data selection make this alignment ambiguous\. Many rely on heuristic rules\(Wenzeket al\.,2019 (https://arxiv.org/html/2604.15706#bib.bib29); Raeet al\.,2022 (https://arxiv.org/html/2604.15706#bib.bib28); Leeet al\.,2022 (https://arxiv.org/html/2604.15706#bib.bib30); Abbaset al\.,2023 (https://arxiv.org/html/2604.15706#bib.bib31)\)or implicit assumptions about “quality”\(Sachdevaet al\.,2024 (https://arxiv.org/html/2604.15706#bib.bib8); Wettiget al\.,2024 (https://arxiv.org/html/2604.15706#bib.bib27); Penedoet al\.,2024 (https://arxiv.org/html/2604.15706#bib.bib6)\), leaving a noticeable gap between how data are chosen and the specific capabilities the model ultimately needs to develop\. Prior efforts to align LLM pre\-training data with explicit targets show that target\-oriented pre\-training can yield substantial compute multipliers and consistent gains across different scales\(Mizrahiet al\.,2025 (https://arxiv.org/html/2604.15706#bib.bib7)\)\. However, most of these approaches achieve task alignment by distilling models’ internal signals, such as embedding heuristics or performance\-correlated losses, into auxiliary classifiers for scalability\(Mizrahiet al\.,2025 (https://arxiv.org/html/2604.15706#bib.bib7); Thrushet al\.,2025 (https://arxiv.org/html/2604.15706#bib.bib22); SHUMet al\.,2025 (https://arxiv.org/html/2604.15706#bib.bib21); Miyoshiet al\.,2025 (https://arxiv.org/html/2604.15706#bib.bib23)\)\. This black\-box distillation introduces an interpretability bottleneck: the learned signals are opaque and hard to diagnose or refine, thus constraining how effectively the signals can be leveraged for further performance gains\. To solve this, we introduce*Neuron\-ActivatedGraph Ranking \(NAG\-based Ranking\)*, a neuron\-centric framework for target\-oriented pretraining data selection\. As illustrated inFig\.2 (https://arxiv.org/html/2604.15706#S0.F2), our key idea is to characterize each text input by identifying which neurons in the LLM matter for processing it, rather than through black\-box representations\. Specifically, we first quantify neuron impact during inference in an off\-the\-shelf LLM \(Sec\.2\.1 (https://arxiv.org/html/2604.15706#S2.SS1)\) and then organize the most influential neurons across layers into a compact*Neuron\-ActivatedGraph \(NAG\)*\(Sec\.2\.2 (https://arxiv.org/html/2604.15706#S2.SS2)\)\. For the data selection, samples are ranked by the similarity of their NAGs to the target examples, prioritizing training inputs that trigger similar neuron patterns with the target data\. Notably, our NAG\-based Ranking requires no additional training and relies solely on interpretable signals from any off\-the\-shelf LLM\. Empirical evaluations demonstrate our NAG\-based Ranking consistently enhances target\-oriented task performance across different settings and benchmarks\. Compared to random sampling, NAG\-based Ranking yields a substantial 4\.9% average improvement in target\-oriented pretraining, outperforming both strong classifiers focusing on general data quality\(Penedoet al\.,2024 (https://arxiv.org/html/2604.15706#bib.bib6)\)and state\-of\-the\-art methods on target\-oriented data selection like BETR\(Mizrahiet al\.,2025 (https://arxiv.org/html/2604.15706#bib.bib7)\)\. Moreover, our algorithm presents more general use in the multi\-target scenario where baselines often degrade, with the best setup outperforming two baselines by 1\.1% and 4\.1%, respectively\. We also prove the broadened application of NAG by adding its signals to another quality\-based data selection method, which further boosts scores\. Finally, the performance gains of NAG remain consistent across various backbone models from 4\.7% to 5\.0%, indicating the robust and model\-agnostic nature of NAG\. Obtaining better performance is one aspect; explaininghowandwhyNAG works makes our algorithm truly interpretable\. We design dedicated experiments to probe from these two perspectives: \(i\)Why NAG works:we find that NAG works by identifying a sparse “functional backbone” of the LLM — deactivating just 0\.12% of model neurons triggers a sharp 23\.5% performance collapse \(Sec\.4\.1\.1 (https://arxiv.org/html/2604.15706#S4.SS1.SSS1)\)\. These high\-impact neurons can well represent different target information via clustering \(Sec\.4\.1\.2 (https://arxiv.org/html/2604.15706#S4.SS1.SSS2)\) — NAG captures the true drivers of the desired targets\. Finally, we show a high correlation between NAG\-based rankings and the target learning utility \(Sec\.4\.1\.3 (https://arxiv.org/html/2604.15706#S4.SS1.SSS3)\)\. \(ii\)How NAG operates:we spot the path on how NAG works, specifically, it maps a “computational trajectory” by aggregating signals across all LLM layers\. Our analysis shows that restricting NAG to the final layer results in a 4\.1% average performance drop \(Sec\.4\.2\.2 (https://arxiv.org/html/2604.15706#S4.SS2.SSS2)\)\. Furthermore, by twitching model components where NAG signals come from, we further identify that extracting signals from the internal projections \(Sec\.4\.2\.1 (https://arxiv.org/html/2604.15706#S4.SS2.SSS1)\) at a very sparse neuron ratio \(Sec\.4\.2\.3 (https://arxiv.org/html/2604.15706#S4.SS2.SSS3)\) is crucial for capturing effective task\-specific signals\. ## 2Method Prior works\(Panigrahiet al\.,2023 (https://arxiv.org/html/2604.15706#bib.bib9); Zhaoet al\.,2024 (https://arxiv.org/html/2604.15706#bib.bib1),2025 (https://arxiv.org/html/2604.15706#bib.bib10)\)suggest that model behavior is governed by a highly sparse subset of parameters, with different tasks relying on largely disjoint regions\. As a result, inputs attributing to the same capability are expected to induce similar internal parameter usage patterns\. Based on this insight, we characterize each input by the subset of neurons that exert strong influence on the model’s computation, and measure input relevance via the similarity of such neuron\-level structures\. We thus propose a neuron\-centric framework for target\-oriented data selection, which \(1\) quantifies neuron impact for each input \(Sec\.2\.1 (https://arxiv.org/html/2604.15706#S2.SS1)\), \(2\) organizes the most influential neurons across layers into a compact*Neuron\-Activated Graph \(NAG\)*\(Sec\.2\.2 (https://arxiv.org/html/2604.15706#S2.SS2)\), and \(3\) ranks candidate samples by their NAG similarity to a small set of target examples \(Sec\.2\.3 (https://arxiv.org/html/2604.15706#S2.SS3)\)\. ### 2\.1Neuron Impact Following PLND\(Zhaoet al\.,2024 (https://arxiv.org/html/2604.15706#bib.bib1)\), we define a*neuron*as a column of a projection weight matrix in a Transformer\-based language model, and focus on projection layers in both Attention \(Q, K, V\) and FFN \(UP, DOWN\) modules\(Vaswaniet al\.,2017 (https://arxiv.org/html/2604.15706#bib.bib2)\)\. Concretely, for a projection matrixW∈Rdin×doutW\\in\\mathbb\{R\}^\{d\_\{\\text\{in\}\}\\times d\_\{\\text\{out\}\}\}, each column ofWWis treated as an individual neuron, yieldingdoutd\_\{\\text\{out\}\}neurons for that layer\. For example, the FFN UP projectionWup∈Rdmodel×dinternalW\_\{\\text\{up\}\}\\in\\mathbb\{R\}^\{d\_\{\\text\{model\}\}\\times d\_\{\\text\{internal\}\}\}containsdinternald\_\{\\text\{internal\}\}neurons\. To quantify the contribution of a neuronNkN\_\{k\}in projection layerl\\ell, we measure the change induced by deactivating it\. Since evaluating its effect on the final output is expensive, we adopt a local approximation and define neuron impact based on the corresponding projection layer output \(we validate this local proxy against loss change inApp\.C (https://arxiv.org/html/2604.15706#A3)\)\. Given inputhin∈Rdinh\_\{\\text\{in\}\}\\in\\mathbb\{R\}^\{d\_\{\\text\{in\}\}\}, the layer output ishout=hin⊤Wh\_\{\\text\{out\}\}=h\_\{\\text\{in\}\}^\{\\top\}W\. DeactivatingNkN\_\{k\}amounts to zeroing thekk\-th column ofWW, denoted asW\\NkW\\backslash N\_\{k\}\. We define the neuron impact as Imp\(Nk∣hin\)=‖hin⊤W−hin⊤\(W\\Nk\)‖2=\|hin⊤W:,k\|,\\text\{Imp\}\(N\_\{k\}\\mid h\_\{\\text\{in\}\}\)=\\left\\\|h\_\{\\text\{in\}\}^\{\\top\}W\-h\_\{\\text\{in\}\}^\{\\top\}\(W\\backslash N\_\{k\}\)\\right\\\|\_\{2\}=\\left\|h\_\{\\text\{in\}\}^\{\\top\}W\_\{:,k\}\\right\|,whereW:,kW\_\{:,k\}denotes thekk\-th column ofWW\. This formulation shows that the neuron impact reduces to the magnitude of its column\-wise contribution to the layer output\. In what follows, we refer to neurons with relatively large impact values as*activated*neurons for the given input\. ### 2\.2Neuron\-Activated Graph \(NAG\) Building on the neuron impact defined inSec\.2\.1 (https://arxiv.org/html/2604.15706#S2.SS1), we construct a structured, layer\-wise representation of neuron impact patterns, termed the*Neuron\-Activated Graph \(NAG\)*\. We treat neurons with high impact scores as*activated*neurons for a given input, and use them as the basic units of the NAG\. For each layer, we rank neurons by their impact scores and select a fixed numberKKof high\-impact neurons\. Consider a model withLLlayers, where layerl\\ellcontainsdld\_\{\\ell\}neurons\. Given an inputcc, letIl,k\(c\)I\_\{\\ell,k\}\(c\)denote the impact of neuronkkin layerl\\ell\. For each layerl\\ell, we select the indices of the top\-KKneurons ranked by impact: Nl\(K\)\(c\)=TopK⁡\(\{Il,k\(c\)\}k=1dl\)⊆\{1,...,dl\},N\_\{\\ell\}^\{\(K\)\}\(c\)\\;=\\;\\operatorname\{TopK\}\\bigl\(\\\{I\_\{\\ell,k\}\(c\)\\\}\_\{k=1\}^\{d\_\{\\ell\}\}\\bigr\)\\;\\subseteq\\;\\\{1,\\dots,d\_\{\\ell\}\\\},whereTopK⁡\(⋅\)\\operatorname\{TopK\}\(\\cdot\)returns the indices of theKKlargest elements\. The*Neuron\-Activated Graph \(NAG\)*of inputccis then defined as the collection of layer\-wise neuron index sets: NAG\(c\)=\(N1\(K\)\(c\),N2\(K\)\(c\),...,NL\(K\)\(c\)\),\\mathrm\{NAG\}\(c\)\\;=\\;\\bigl\(N\_\{1\}^\{\(K\)\}\(c\),\\,N\_\{2\}^\{\(K\)\}\(c\),\\,\\dots,\\,N\_\{L\}^\{\(K\)\}\(c\)\\bigr\),or equivalently as the set of layer–neuron index pairs NAG\(c\)=\{\(l,k\)\|l∈\{1,...,L\},k∈Nl\(K\)\(c\)\}\.\\mathrm\{NAG\}\(c\)\\;=\\;\\bigl\\\{\\,\(\\ell,k\)\\;\\big\|\\;\\ell\\in\\\{1,\\dots,L\\\},\\;k\\in N\_\{\\ell\}^\{\(K\)\}\(c\)\\,\\bigr\\\}\. Table 1:Results of NAG\-based data selection under Single\-Target and Multi\-Target settings \(Sec\.3\.3 (https://arxiv.org/html/2604.15706#S3.SS3)\)\. NAG is instantiated with different backbone models \(*e\.g*\.,NAGQwen3\-1\.7B\\text\{NAG\}\_\{\\text\{Qwen3\-1\.7B\}\}\)\. For each benchmark,boldrepresents the best, andunderlinedrepresents the second; the best within the Single/Multi\-Target settings is additionally highlighted withshade\. Improvements are relative to Random and reported as subscripts \(redfor gains andbluefor drops\)\.MethodARC\-CHellaSwagTriviaQAMMLUXStoryClozeXWinogradAvg\.Random28\.5%51\.6%15\.6%30\.2%67\.1%76\.5%44\.9%FineWeb\-Edu34\.3%\+5\.8%55\.3%\+3\.7%20\.1%\+4\.5%32\.8%\+2\.6%65\.9%\-1\.2%76\.2%\-0\.3%47\.4%\+2\.5%Single\-TargetBETR32\.3%\+3\.8%57\.5%\+5\.9%20\.2%\+4\.6%31\.1%\+0\.9%\\cellcolorbaselinecolor71\.0%\+3\.9%\\cellcolorbaselinecolor80\.7%\+4\.2%48\.8%\+3\.9%NAGQwen3\-1\.7B\\text\{NAG\}\_\{\\text\{Qwen3\-1\.7B\}\}34\.0%\+5\.5%\\cellcolorbaselinecolor60\.6%\+9\.0%22\.3%\+6\.7%\\cellcolorbaselinecolor32\.2%\+2\.0%70\.0%\+2\.9%80\.1%\+3\.6%49\.8%\+4\.9%NAGLlama\-3\.2\-3B\\text\{NAG\}\_\{\\text\{Llama\-3\.2\-3B\}\}\\cellcolorbaselinecolor35\.0%\+6\.5%58\.6%\+7\.0%21\.3%\+5\.7%31\.5%\+1\.3%70\.8%\+3\.7%80\.6%\+4\.1%49\.6%\+4\.7%NAGSmolLM3\-3B\\text\{NAG\}\_\{\\text\{SmolLM3\-3B\}\}\\cellcolorbaselinecolor35\.0%\+6\.5%59\.8%\+8\.2%\\cellcolorbaselinecolor22\.6%\+7\.0%31\.2%\+1\.0%70\.5%\+3\.4%80\.6%\+4\.1%\\cellcolorbaselinecolor49\.9%\+5\.0%Multi\-TargetBETR30\.3%\+1\.8%49\.3%\-2\.3%11\.6%\-4\.0%29\.9%\-0\.3%69\.5%\+2\.4%76\.1%\-0\.4%44\.4%\-0\.5%NAGQwen3\-1\.7B\\text\{NAG\}\_\{\\text\{Qwen3\-1\.7B\}\}\\cellcolorbaselinecolor33\.4%\+4\.9%\\cellcolorbaselinecolor57\.8%