Procedural Pretraining for Molecular Property Prediction

# Procedural Pretraining for Molecular Property Prediction
Source: [https://arxiv.org/html/2609.17831](https://arxiv.org/html/2609.17831)
Zachary ShinnickAffiliation:University of OxfordAffiliation:Adelaide UniversityPhilip TorrAffiliation:University of OxfordAffiliation:Slater LabsBruno AndreisAffiliation:University of OxfordAffiliation:Slater Labs

###### Abstract

Molecular property prediction is often limited by the small size of labeled downstream datasets, motivating pretraining on large corpora of unlabeled molecules\. In this work, we ask whether useful inductive biases can instead be learned from abstract, procedurally generated data before a model sees any molecular data\. We introduce a three\-stage training pipeline consisting of procedural pretraining, molecular pretraining on SMILES, and downstream fine\-tuning, and evaluate several procedural tasks spanning sequence structure, cellular automata, and graph reasoning\. We find that procedural pretraining can improve molecular property prediction even after subsequent molecular pretraining: on Lipophilicity,Reversereduces test error by 4\.8%\. For context, the magnitude of this improvement is roughly 90% of the performance difference between our 250K\-molecule baseline and the publicly released MoLFormer checkpoint pretrained on approximately 100M molecules\. Our analysis shows that the benefit is strongest under downstream data scarcity, depends on the structure of the procedural data rather than only surface\-level statistics, and does not increase monotonically with additional procedural training\. Instead, transfer typically peaks at an intermediate procedural budget and deteriorates as the model approaches convergence on the procedural task\. We further find that, for several tasks, much of the transferable information is localized in the attention layers, while feed\-forward layers can contribute to over\-specialization\. These results show that procedural data can provide transferable structure for molecular learning and offer a complementary route to improving performance when labeled molecular data are limited\.

## 1Introduction

Figure 1:Overview of the proposed three\-stage training pipeline\. A transformer is first pretrained on procedurally generated masked\-token tasks, then pretrained on molecular SMILES, and finally fine\-tuned for molecular property prediction\. Procedural pretraining can reduce downstream error relative to molecular pretraining and fine\-tuning alone\.Modern molecular machine learning models are often trained in a regime where unlabeled molecules are abundant\([Irwin and Shoichet, 2004](https://arxiv.org/html/2609.17831#bib.bib7);[Kim et al\., 2019](https://arxiv.org/html/2609.17831#bib.bib8)\)but labeled downstream data are scarce\. Molecular property prediction datasets frequently contain only hundreds or thousands of labeled examples\([Wu et al\., 2018](https://arxiv.org/html/2609.17831#bib.bib5)\), making it difficult for high\-capacity models to learn useful representations from downstream supervision alone\. A common response has been to adopt the pretraining paradigm developed for language models: models are first trained on large collections of unlabeled molecular representations, such as SMILES strings, and subsequently fine\-tuned for a target property\([Chithrananda et al\., 2020](https://arxiv.org/html/2609.17831#bib.bib9);[Ross et al\., 2022](https://arxiv.org/html/2609.17831#bib.bib4)\)\. While such molecular pretraining can substantially improve performance, acquiring and processing increasingly large molecular corpora can be expensive, and it remains unclear whether useful inductive biases must themselves be learned from molecular data\.

Recent work suggests an alternative source of pretraining signal: procedurally generated data\. Simple algorithmic processes, such as logical operations\([Wu et al\., 2022](https://arxiv.org/html/2609.17831#bib.bib15);[Shinnick et al\., 2025](https://arxiv.org/html/2609.17831#bib.bib16)\), formal languages\([Papadimitriou and Jurafsky, 2023](https://arxiv.org/html/2609.17831#bib.bib17);[Hu et al\., 2025](https://arxiv.org/html/2609.17831#bib.bib6)\), and cellular automata\([Zhang et al\., 2025](https://arxiv.org/html/2609.17831#bib.bib18);[Lee et al\., 2026](https://arxiv.org/html/2609.17831#bib.bib3)\), can generate effectively unlimited data with controlled structural properties\. Procedural pretraining on such data has been shown to induce representations that transfer to natural language and vision tasks\([Jiang et al\., 2026](https://arxiv.org/html/2609.17831#bib.bib1);[Shinnick et al\., 2026](https://arxiv.org/html/2609.17831#bib.bib2)\)\. This raises a surprising possibility for molecular learning:can a model learn useful computational structure before seeing a single molecule, and can that structure subsequently improve molecular property prediction?

We investigate this question by introducing a procedural pretraining stage before conventional molecular pretraining and downstream fine\-tuning\. Using MoLFormer\([Ross et al\., 2022](https://arxiv.org/html/2609.17831#bib.bib4)\)as our base architecture, we pretrain on several procedural tasks spanning sequence manipulation\([Jiang et al\., 2026](https://arxiv.org/html/2609.17831#bib.bib1)\), hierarchical structure\([Hu et al\., 2025](https://arxiv.org/html/2609.17831#bib.bib6)\), dynamical systems\([Lee et al\., 2026](https://arxiv.org/html/2609.17831#bib.bib3)\), and graph reasoning, then transfer the learned transformer weights to molecular prediction\. Our experiments consider both a realistic three\-stage pipeline, procedural pretraining, molecular pretraining, and downstream fine\-tuning, and a simplified setting that removes molecular pretraining to isolate the mechanisms responsible for transfer\.

We find that procedural pretraining can improve molecular property prediction even after subsequent pretraining on real molecular data\. On Lipophilicity\([Wu et al\., 2018](https://arxiv.org/html/2609.17831#bib.bib5)\), pretraining on a simple sequence\-reversal task reduces test error by 4\.8% relative to a molecular\-pretraining\-only baseline, recovering roughly 90% of the performance difference between our 250K\-molecule baseline and the publicly released MoLFormer checkpoint pretrained on approximately 100M molecules\. We then use controlled experiments to characterize when this transfer is beneficial, what properties of the procedural data drive it, and where the transferable information is represented in the model\.

Our main contributions are:

- •We show thatprocedural pretraining can improve molecular property prediction even after subsequent molecular pretraining, demonstrating that useful non\-molecular structure can survive a realistic domain\-specific pretraining stage\.
- •We find thatstructural alignmentbetween procedural and molecular data provides an initial head start during molecular pretraining butdoes not reliably predict downstream benefit\.
- •We show that the benefit islargest under downstream data scarcity, suggesting that procedural pretraining acts as an inductive bias that is most valuable when labeled molecular data are limited\.
- •We find thatmore procedural pretraining is not always better\. Across four of five tasks, downstream performance peaks at an intermediate procedural budget and deteriorates as the model approaches the optimum of the procedural task\.
- •We demonstrate that transfer depends on thestructure of the procedural data rather than surface\-level statistics: shuffling token order while preserving vocabulary, sequence length, and token frequencies removes the benefit\.
- •Welocalize much of the transferable information to the attention layersfor several procedural tasks, while finding that feed\-forward layers can contribute to the over\-specialization observed at larger procedural budgets\.

## 2Related Work

##### Pretraining on procedural data\.

Recent work has shown that useful representations can be learned from procedurally generated data rather than natural corpora\. In language, pretraining on artificial languages, formal languages, neural cellular automata, and simple algorithmic tasks can improve downstream performance on natural language tasks\([Papadimitriou and Jurafsky, 2020](https://arxiv.org/html/2609.17831#bib.bib13);[Hu et al\., 2025](https://arxiv.org/html/2609.17831#bib.bib6);[Lee et al\., 2026](https://arxiv.org/html/2609.17831#bib.bib3);[Jiang et al\., 2026](https://arxiv.org/html/2609.17831#bib.bib1)\)\. Similar results have been observed in vision using synthetic images such as fractals or structured noise\([Kataoka et al\., 2021](https://arxiv.org/html/2609.17831#bib.bib11);[Baradad Jurjo et al\., 2021](https://arxiv.org/html/2609.17831#bib.bib10)\)\. While many of these procedures reproduce structural properties of the target domain, recent work demonstrates transfer even across domains: sequence\-based formal languages can improve vision transformers\([Shinnick et al\., 2026](https://arxiv.org/html/2609.17831#bib.bib2)\), and music pretraining can benefit language models\([Nomura, 2026](https://arxiv.org/html/2609.17831#bib.bib12)\)\. This suggests that procedural pretraining can provide useful inductive biases without closely resembling downstream data\.

Prior work has also investigated which properties make procedural pretraining transferable\. Structural properties such as diversity, long\-range dependencies, and hierarchical organization can improve transfer\([Baradad Jurjo et al\., 2021](https://arxiv.org/html/2609.17831#bib.bib10);[Chiang and Lee, 2021](https://arxiv.org/html/2609.17831#bib.bib14);[Hu et al\., 2025](https://arxiv.org/html/2609.17831#bib.bib6)\)\. However, strong domain alignment is not always necessary: even simple sequence operations retain part of the benefit of natural\-language pretraining\([Wu et al\., 2022](https://arxiv.org/html/2609.17831#bib.bib15)\), and transferable information may reside in different transformer components depending on the downstream task\([Jiang et al\., 2026](https://arxiv.org/html/2609.17831#bib.bib1)\)\. Together, these findings point to two possible sources of transfer: alignment between the procedural and target distributions, and more general changes to the model’s inductive bias\.

We extend this line of work to molecular modeling, where labeled data are comparatively scarce and procedural pretraining has received little attention\. We study whether abstract procedural tasks can complement molecular pretraining, how alignment with SMILES structure affects transfer, and which model components and properties of procedural\-task learning account for downstream gains\.

## 3Methodology

##### Training pipeline\.

We study a three\-stage training pipeline consisting of \(a\) procedural pretraining, \(b\) molecular pretraining, and \(c\) downstream fine\-tuning\. The procedural stage exposes the model only to algorithmically generated data, before it encounters any molecular examples\. We then optionally continue pretraining on unlabeled SMILES strings before fine\-tuning on a molecular property prediction task\. In[Section4\.2](https://arxiv.org/html/2609.17831#S4.SS2), we omit the molecular pretraining stage for some experiments and fine\-tune directly after procedural pretraining in order to isolate the effect of the procedural data\.

##### Procedural pretraining tasks\.

We consider five procedural tasks designed to expose the model to different forms of structure\. Three operate directly on synthetic sequences:Reverse, in which the model must recover the reversed form of a token sequence;Dyck, which consists of properly nested bracket sequences; andDyck\-Shuffle, which permits crossing bracket dependencies\.Dyckmirrors the structure of branching parentheses in SMILES, whileDyck\-Shufflemirrors ring\-closure digits\. We additionally consider neural cellular automaton \(NCA\) trajectories, which introduce spatiotemporal structure, and a shortest\-path \(SSP\) task in which the model predicts node distances from the SMILES representation of a random graph\. All tasks are formulated as masked\-token prediction problems so that the same model architecture and training objective can be used throughout procedural and molecular pretraining\. Examples of the procedural inputs are shown in[Figure2](https://arxiv.org/html/2609.17831#S3.F2)\.

01234234source\(a\)graph and source node\*CCCCCCC1\(\)101234234concatenate\*C1CC\(C\)C1CC\.01234234mask distances\*C1CC\(C\)C1CC\.?12??2?4\(b\)input sequence
Figure 2:Procedural pretraining tasks\. Top: example sequences of the sequence\-level tasks before and after masking\. Bottom: how an input sequence for theSSPtask is created\. a\) A molecule\-like random graph is generated and a source node chosen at random\. b\) The SMILES representation of the graph \(using\*for the source andCfor all other nodes\) is concatenated with shortest\-path distances from the source to all other nodes\.
##### Model\.

We use MoLFormer\([Ross et al\., 2022](https://arxiv.org/html/2609.17831#bib.bib4)\), an encoder\-only transformer with approximately4444M parameters designed for learning from SMILES representations\. MoLFormer uses rotary positional embeddings \(RoPE\), linear attention based on generalized random features, and a regular\-expression\-based SMILES tokenizer with a vocabulary of 2,362 tokens\. We use the same transformer architecture across all training stages\.

##### Molecular pretraining\.

MoLFormer is conventionally pretrained using masked token modeling \(MTM\) on large collections of unlabeled SMILES strings drawn from PubChem\([Kim et al\., 2019](https://arxiv.org/html/2609.17831#bib.bib8)\)and ZINC\([Irwin and Shoichet, 2004](https://arxiv.org/html/2609.17831#bib.bib7)\)\. The publicly released MoLFormer checkpoint was trained on approximately100100M molecules, which is beyond our available compute budget\. To evaluate procedural pretraining within a realistic molecular pretraining pipeline, we therefore construct a smaller\-scale approximation of the setup in[Ross et al\. \(2022\)](https://arxiv.org/html/2609.17831#bib.bib4)\. We pretrain for 12 epochs on a 250k\-molecule subset of PubChem using the same MTM objective and a masking probability of0\.150\.15\. When comparing procedural and non\-procedural initializations, all models receive the same molecular pretraining data and compute budget\.

##### Downstream evaluation\.

We evaluate on three molecular property prediction benchmarks with continuous regression targets: Lipophilicity from MoleculeNet\([Wu et al\., 2018](https://arxiv.org/html/2609.17831#bib.bib5)\), containing approximately44k molecules; FreeSolv, containing 642 molecules; and QM9, for which we predict the HOMO–LUMO gap and use either the full dataset or controlled subsets of the training set\. Models are fine\-tuned for 500 epochs\. We report test mean absolute error \(MAE\) on standardized targets from the checkpoint achieving the best validation performance, following[Ross et al\. \(2022\)](https://arxiv.org/html/2609.17831#bib.bib4)\. Unless stated otherwise, comparisons use identical downstream splits and optimization settings\.

##### Weight transfer\.

We partition the model parameters into token embeddings, attention projections, feed\-forward layers, LayerNorm parameters, and the output head\. The procedural and molecular domains use different token semantics, so we do not transfer learned procedural token embeddings\. Instead, the embedding matrix is held fixed at its random initialization during procedural pretraining, ensuring that any transferable information must be encoded in the transformer backbone rather than in task\-specific token representations\. We also discard the procedural output head, whose dimensionality and semantics need not match those of subsequent stages\. Our default transfer setting therefore retains the learned transformer backbone while reinitializing the input and output interfaces as required\.

To localize where transferable information is stored, we additionally consider selective transfer\. We transfer either only the attention\-layer weights or only the feed\-forward\-layer weights, while reinitializing all remaining parameters, including LayerNorms\. This allows us to measure how much of the downstream benefit can be attributed to each component of the transformer\.

## 4Experiments

### 4\.1Procedural Pretraining Improves Molecular Property Prediction

We first evaluate procedural pretraining in the full three\-stage pipeline: procedural pretraining, molecular pretraining, and downstream fine\-tuning\. We compare models initialized from each procedural task against a baseline trained only on molecular data, while keeping the molecular pretraining and downstream fine\-tuning stages fixed\. We evaluate on Lipophilicity, FreeSolv, and a 1k\-molecule subset of QM9 for HOMO–LUMO gap prediction\.

Table 1:Molecular property prediction after procedural and molecular pretraining: All models are pretrained on our 250k molecule PubChem subset before downstream fine\-tuning\. Standardized test MAE is reported \(lower is better\), withΔ\\Deltarelative to the molecular\-pretraining\-only baseline\. Results averagen=3n\{=\}3seeds \(n=6n\{=\}6for FreeSolv\); best procedural results are bold\.[Table1](https://arxiv.org/html/2609.17831#S4.T1)reports standardized test MAE and the relative change with respect to the molecular\-pretraining\-only baseline\. On Lipophilicity, two procedural tasks yield clear improvements:Reversereduces MAE by4\.80%4\.80\\%andDyck\-Shuffleby3\.53%3\.53\\%, with both improvements separated from the baseline across seeds\. The remaining three tasks also improve the mean performance, although their gains lie within seed variation\.Reverseadditionally improves performance on the 1k\-molecule QM9 subset by1\.91%1\.91\\%\.

The magnitude of the Lipophilicity improvement is notable relative to scaling molecular pretraining itself\. Fine\-tuning the publicly released MoLFormer checkpoint, pretrained on approximately100100M molecules, improves over our 250k\-molecule molecular\-pretraining baseline by5\.35%5\.35\\%\. By comparison, only 1\.25k procedural pretraining steps onReverserecover a4\.80%4\.80\\%improvement without using any additional molecular data\.

We do not observe a clear benefit on FreeSolv\. Four procedural tasks perform worse than the molecular\-pretraining\-only baseline, although the released MoLFormer checkpoint, trained on approximately400×400\\timesmore molecular data, is also worse by12\.01%12\.01\\%\. Given that FreeSolv contains only 642 molecules, the dataset appears too small to reliably resolve differences of this magnitude\.

Overall, these results show that information learned from procedural data can survive subsequent molecular pretraining and improve downstream molecular property prediction\.

### 4\.2When and Why Does Procedural Pretraining Help?

Having established that procedural pretraining can improve the full three\-stage molecular learning pipeline, we next investigate when and why this transfer occurs\. Unless otherwise noted, experiments in this section use a two\-stage diagnostic setting: procedural pretraining followed directly by downstream fine\-tuning, with no intervening molecular pretraining\. In this setting, procedurally pretrained models are compared against an identically fine\-tuned, randomly initialized baseline\. Removing the molecular stage isolates the effects of procedural pretraining from representations subsequently learned from SMILES data\. Two subsections also examine the full pipeline:[Section4\.2\.1](https://arxiv.org/html/2609.17831#S4.SS2.SSS1)retains molecular pretraining throughout, while[Section4\.2\.4](https://arxiv.org/html/2609.17831#S4.SS2.SSS4)reports shuffled\-token controls in both settings\.

#### 4\.2\.1Does Structural Alignment With Molecules Matter?

\(a\)Held\-out PubChem250k loss\.\(b\)QM9 gap MAE across training\-set sizes\.
Figure 3:Molecular pretraining dynamics and data\-scarcity evaluation: a\) Held\-out molecular pretraining loss for each procedural task at its best configuration; also includes a token\-shuffled control model discussed in[Section4\.2\.4](https://arxiv.org/html/2609.17831#S4.SS2.SSS4)\. b\) QM9 gap MAE across training\-set sizes afterDyck\(k=64k=64\) pretraining\. Values are relative to the randomly initialized baseline; error bars are±1\\pm 1standard deviation overn=3n\{=\}3seeds\. The full\-dataset point in b\) is a single seed\. Panel \(a\) also includes the token\-shuffledDyck\-Shufflecontrol discussed in[Section4\.2\.4](https://arxiv.org/html/2609.17831#S4.SS2.SSS4)Reverseprovides the strongest downstream improvement in[Section4\.1](https://arxiv.org/html/2609.17831#S4.SS1), despite having no obvious structural correspondence with SMILES\. By contrast,Dyck,Dyck\-Shuffle, andSSPwere chosen to reflect structural properties that are more closely aligned with molecular representations\. We therefore ask whether such alignment provides an advantage during molecular pretraining\.

We compare held\-out masked\-token modeling loss throughout 12 epochs of molecular pretraining on PubChem250k\. As shown in[Figure3\(a\)](https://arxiv.org/html/2609.17831#S4.F3.sf1), the SMILES\-aligned tasks,Dyck,Dyck\-Shuffle, andSSP, provide the largest advantage at the beginning of molecular pretraining\. However, this advantage decays rapidly, and after several epochs their performance becomes similar to that ofReverse\.SSPeventually becomes detrimental despite operating on real SMILES strings\.

These results suggest that structural alignment provides an initial head start during molecular pretraining, but that much of this advantage can be recovered from SMILES data alone\. More broadly, structural alignment is not a reliable predictor of which tasks will transfer best:Reverseprovides the strongest downstream improvement after the full pipeline despite having little obvious alignment with SMILES\. Thus, within our task suite, procedural tasks do not need to be structurally aligned with the molecular target domain to transfer effectively\.

#### 4\.2\.2The Benefit Is Largest Under Data Scarcity

If procedural pretraining supplies useful structure that would otherwise have to be inferred from downstream molecular data, its value should increase as the amount of labeled data decreases\. To test this hypothesis, we subsample the QM9 training set to 1k, 16k, 64k, and the full 108,446 molecules\. At each size, we compare a model pretrained onDyckagainst a randomly initialized baseline and evaluate both models on the same held\-out test set of 13,389 molecules\.

[Figure3\(b\)](https://arxiv.org/html/2609.17831#S4.F3.sf2)shows a clear relationship between downstream dataset size and transfer benefit: the advantage of procedural pretraining grows monotonically as the amount of training data decreases\. Procedural pretraining therefore acts most strongly as an inductive bias in low\-data regimes, where the downstream dataset provides less opportunity for the model to learn the relevant structure directly\.

#### 4\.2\.3More Procedural Pretraining Is Not Always Better

Figure 4:Downstream transfer across procedural pretraining budgets: Lipophilicity performance relative to random initialization across procedural budgets and task\-complexity settings\. Error bars show±1\\pm 1standard deviation overn=3n\{=\}3seeds; Across four of five tasks, downstream benefit vanishes at higher budgets\. In particular, a few seeds at the highest budget collapsed to the loss of a constant predictor, these seeds are excluded from the mean calculation\.A practical appeal of procedural data is that it can be generated in effectively unlimited quantities\. We therefore ask whether increasing the procedural pretraining budget continually improves downstream transfer\.

We vary both task complexity and procedural training duration\. ForDyckandDyck\-Shuffle, complexity is controlled by the number of distinct bracket types; forReverse, by sequence length; and forNCA, by the gzip complexity of the generated trajectories\. For each combination of task complexity and training budget, we procedurally pretrain a model and compare its downstream Lipophilicity performance against the randomly initialized baseline\.

As shown in[Figure4](https://arxiv.org/html/2609.17831#S4.F4), four of the five procedural tasks exhibit the same qualitative pattern: downstream benefit initially increases with additional procedural training, reaches a maximum at an intermediate budget, and then deteriorates\. At the largest budgets, performance is often worse than the randomly initialized baseline\.NCAis the only exception and is also the only task that the model does not learn to solve during procedural pretraining\.

This behavior suggests that transfer degrades as the model becomes increasingly specialized to the procedural objective\. To examine this more directly, we compare downstream benefit with the procedural loss reached at each training budget\. For deterministic tasks such asDyck,Reverse, andSSP, the Bayes\-optimal loss is zero, and the models approach this value at large training budgets\.Dyck\-Shuffleis not perfectly predictable because closing\-bracket identity contains aleatoric uncertainty: at a masked position, multiple currently open bracket types can yield a valid sequence\. Its Bayes\-optimal loss is therefore non\-zero and can be computed from the generating process\. Fork=128k=128, this value is approximately1\.2671\.267, which the model approaches at the largest budgets\. Procedural losses and Bayes\-optima can be found in Appendix[C](https://arxiv.org/html/2609.17831#A3)\.

The relevant factor is therefore not low procedural loss in absolute terms, but proximity to the best achievable solution for the procedural task\. Across tasks, downstream transfer is strongest before the model approaches procedural convergence, suggesting that useful general structure is acquired earlier than task\-specific specialization\.

#### 4\.2\.4Transfer Requires Procedural Structure

One alternative explanation for the observed gains is that procedural pretraining need not teach meaningful structure at all\. Any additional pretraining could move the model away from its random initialization, calibrate parameter scales, or expose it to relevant sequence lengths and token\-frequency statistics\. To distinguish these effects from learning structured dependencies, we construct shuffled\-token controls\. We randomly permute token order within each procedural sequence, thereby destroying order\-dependent structure while preserving the vocabulary, sequence lengths, and token frequencies of the original corpus\.

We evaluate these controls in two complementary settings\. First, in the two\-stage diagnostic setting used throughout most of[Section4\.2](https://arxiv.org/html/2609.17831#S4.SS2), we construct a separate shuffled\-token corpus for each of the five procedural tasks\. Each model is pretrained on the shuffled corpus and then fine\-tuned directly on Lipophilicity, without molecular pretraining\. As shown in Appendix[D](https://arxiv.org/html/2609.17831#A4), every shuffled\-token model loses the downstream benefit of its structured counterpart and performs no better than the randomly initialized fine\-tuning baseline\.

Second, we test whether this conclusion survives the full three\-stage pipeline usingDyck\-Shuffleas a representative control\. The token\-shuffled model undergoes the same molecular pretraining and downstream fine\-tuning as the structured model\. In[Figure3\(a\)](https://arxiv.org/html/2609.17831#S4.F3.sf1), its held\-out MTM loss remains worse than that of molecular pretraining from random initialization throughout all 12 epochs\. After Lipophilicity fine\-tuning, it obtains an MAE of0\.4181±0\.00620\.4181\\pm 0\.0062, a roughly2\.46%2\.46\\%degradation relative to the molecular\-pretraining\-only baseline, whereas structuredDyck\-Shuffleimproves performance by3\.53%3\.53\\%\.

These results rule out generic exposure to synthetic sequences or their token\-level statistics as a sufficient explanation for transfer\. The downstream benefit depends on the structured relationships present in the procedural data\.

### 4\.3Where Does the Transfer Reside?

The preceding experiments show that procedural structure can transfer to molecular prediction, but they do not identify where that information is represented within the transformer\. We therefore ask which model components carry the transferable information learned during procedural pretraining\.

We compare three transfer settings on Lipophilicity: transferring the full transformer backbone, transferring only the attention\-layer weights, and transferring only the feed\-forward\-layer weights\. All other parameters are reinitialized as described in[Section3](https://arxiv.org/html/2609.17831#S3)\.

Figure 5:Lipophilicity benefit for full\-backbone, attention\-only, and feed\-forward\-only transfer\. Attention\-only transfer retains most of the benefit, while feed\-forward\-only transfer generally does not\.[Figure5](https://arxiv.org/html/2609.17831#S4.F5)shows that attention\-only transfer matches or exceeds full\-backbone transfer forDyck,Dyck\-Shuffle, andReverse\. In contrast, transferring only the feed\-forward layers removes most of the downstream benefit and can even hurt performance\.SSPbehaves differently: neither component alone recovers the full\-transfer benefit, suggesting that useful information is distributed across both attention and feed\-forward layers\.NCAprovides no significant benefit under any transfer configuration\.

Figure 6:Full\-backbone transfer compared with attention\-only transfer at increasing procedural token budgets\. We report Lipophilicity benefit over the randomly initialized baseline\. Error bars are±1\\pm 1standard deviation overn=3n=3seeds\. Attention\-only transfer decays more slowly than full transfer;DyckandDyck\-Shuffleretain a benefit at the largest budgets, whileReverseand SSP still converge towards the baseline\.The deterioration observed at large procedural budgets in[Section4\.2\.3](https://arxiv.org/html/2609.17831#S4.SS2.SSS3)raises a related question: does this over\-specialization also reside in particular model components? To test this, we repeat the procedural\-budget sweep while transferring only the attention\-layer weights to the downstream task\.

[Figure6](https://arxiv.org/html/2609.17831#S4.F6)shows that attention\-only transfer is generally more robust to increasing procedural budgets than full\-backbone transfer\. ForDyckandDyck\-Shuffle, the downstream benefit no longer decays at the largest tested budgets when only attention weights are transferred\.ReverseandSSPalso decay more slowly under attention\-only transfer, although both eventually converge back towards the baseline\.

Together, these results indicate that much of the useful transferable structure learned fromDyck,Dyck\-Shuffle, andReverseis carried by the attention layers\. They also suggest that part of the degradation caused by excessive procedural pretraining is associated with the feed\-forward layers: discarding these weights preserves useful transfer over a wider range of procedural budgets\. The separation is not complete, however, sinceReverseandSSPstill lose their advantage at sufficiently large budgets even under attention\-only transfer\.

## 5Conclusion and Open Questions

We showed that procedural pretraining on abstract, non\-molecular data can improve molecular property prediction, even after subsequent pretraining on real molecular data\. The gains are largest in low\-data regimes, and in our strongest result, a simple sequence\-reversal task recovers roughly 90% of the performance difference between our 250K\-molecule baseline and the publicly released MoLFormer checkpoint pretrained on approximately 100M molecules\. The benefit depends on the structure of the procedural data rather than on simple token\-level statistics: shuffling token order removes the transfer advantage\.

Our results also show that more procedural pretraining is not always better\. Across most tasks, downstream performance peaks at an intermediate training budget and deteriorates as the model approaches procedural convergence, suggesting a transition from learning broadly useful structure to task\-specific specialization\. For several tasks, much of the transferable information is retained in the attention layers, while the feed\-forward layers appear to contribute more strongly to over\-specialization\.

Several questions remain open\. What properties make a procedural task transferable, and can they be identified without downstream evaluation? Can the optimal stopping point for procedural pretraining be predicted from procedural training dynamics alone, rather than from downstream validation? Our results were also inconclusive on FreeSolv and for the NCA task, which showed no clear downstream benefit at any procedural budget\. Determining whether these outcomes reflect dataset size, task predictability, or a more fundamental mismatch remains an important direction for future work\. More broadly, do these effects persist across larger models, alternative molecular representations, and other scientific domains? Our findings suggest that improving molecular learning may require not only more molecular data, but also better ways of shaping the model before it sees a single molecule\.

## References

- Baradad Jurjoet al\.\(2021\)M\. Baradad Jurjo, J\. Wulff, T\. Wang, P\. Isola, and A\. TorralbaLearning to See by Looking at Noise\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 2556–2569\.External Links:[Link](https://proceedings.neurips.cc/paper/2021/hash/14f2ebeab937ca128186e7ba876faef9-Abstract.html)Cited by:[§2](https://arxiv.org/html/2609.17831#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2609.17831#S2.SS0.SSS0.Px1.p2.1)\.
- Chiang and Lee \(2021\)C\. Chiang and H\. LeeOn the Transferability of Pre\-trained Language Models: A Study from Artificial Datasets\.\(en\)\.External Links:[Link](https://arxiv.org/abs/2109.03537v2)Cited by:[§2](https://arxiv.org/html/2609.17831#S2.SS0.SSS0.Px1.p2.1)\.
- Chithranandaet al\.\(2020\)S\. Chithrananda, G\. Grand, and B\. RamsundarChemBERTa: Large\-Scale Self\-Supervised Pretraining for Molecular Property Prediction\.arXiv\.Note:arXiv:2010\.09885 \[cs\.LG\]Comment: Submitted to NeurIPS 2020 ML for Molecules WorkshopExternal Links:[Link](http://arxiv.org/abs/2010.09885),[Document](https://dx.doi.org/10.48550/arXiv.2010.09885)Cited by:[§1](https://arxiv.org/html/2609.17831#S1.p1.1)\.
- Huet al\.\(2025\)M\. Y\. Hu, J\. Petty, C\. Shi, W\. Merrill, and T\. LinzenBetween Circuits and Chomsky: Pre\-pretraining on Formal Languages Imparts Linguistic Biases\.arXiv\.Note:arXiv:2502\.19249 \[cs\.CL\]Comment: ACL 2025 Camera ReadyExternal Links:[Link](http://arxiv.org/abs/2502.19249),[Document](https://dx.doi.org/10.48550/arXiv.2502.19249)Cited by:[§1](https://arxiv.org/html/2609.17831#S1.p2.1),[§1](https://arxiv.org/html/2609.17831#S1.p3.1),[§2](https://arxiv.org/html/2609.17831#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2609.17831#S2.SS0.SSS0.Px1.p2.1)\.
- Irwin and Shoichet \(2004\)J\. J\. Irwin and B\. K\. ShoichetZINC \- A Free Database of Commercially Available Compounds for Virtual Screening\.Journal of Chemical Information and Modeling45\(1\),pp\. 177–182\.External Links:ISSN 1549\-9596,[Link](https://doi.org/10.1021/ci049714+),[Document](https://dx.doi.org/10.1021/ci049714+)Cited by:[§1](https://arxiv.org/html/2609.17831#S1.p1.1),[§3](https://arxiv.org/html/2609.17831#S3.SS0.SSS0.Px4.p1.1)\.
- Jianget al\.\(2026\)L\. Jiang, Z\. Shinnick, A\. van den Hengel, H\. Saratchandran, and D\. TeneyProcedural pretraining: warming up language models with abstract data\.InProceedings of the International Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2609.17831#S1.p2.1),[§1](https://arxiv.org/html/2609.17831#S1.p3.1),[§2](https://arxiv.org/html/2609.17831#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2609.17831#S2.SS0.SSS0.Px1.p2.1)\.
- Kataokaet al\.\(2021\)H\. Kataoka, K\. Okayasu, A\. Matsumoto, E\. Yamagata, R\. Yamada, N\. Inoue, A\. Nakamura, and Y\. SatohPre\-training without Natural Images\.arXiv\.Note:arXiv:2101\.08515 \[cs\.CV\]Comment: ACCV 2020 Best Paper Honorable Mention Award, Codes are publicly available: https://github\.com/hirokatsukataoka16/FractalDB\-Pretrained\-ResNet\-PyTorchExternal Links:[Link](http://arxiv.org/abs/2101.08515),[Document](https://dx.doi.org/10.48550/arXiv.2101.08515)Cited by:[§2](https://arxiv.org/html/2609.17831#S2.SS0.SSS0.Px1.p1.1)\.
- Kimet al\.\(2019\)S\. Kim, J\. Chen, T\. Cheng, A\. Gindulyte, J\. He, S\. He, Q\. Li, B\. A\. Shoemaker, P\. A\. Thiessen, B\. Yu, L\. Zaslavsky, J\. Zhang, and E\. E\. BoltonPubChem 2019 update: improved access to chemical data\.Nucleic Acids Research47\(D1\),pp\. D1102–D1109\.External Links:ISSN 0305\-1048,[Link](https://doi.org/10.1093/nar/gky1033),[Document](https://dx.doi.org/10.1093/nar/gky1033)Cited by:[§1](https://arxiv.org/html/2609.17831#S1.p1.1),[§3](https://arxiv.org/html/2609.17831#S3.SS0.SSS0.Px4.p1.1)\.
- Leeet al\.\(2026\)D\. Lee, S\. Han, A\. Kumar, and P\. AgrawalTraining Language Models via Neural Cellular Automata\.arXiv\.Note:arXiv:2603\.10055 \[cs\.LG\]Comment: Website: https://hanseungwook\.github\.io/blog/nca\-pre\-pre\-training/External Links:[Link](http://arxiv.org/abs/2603.10055),[Document](https://dx.doi.org/10.48550/arXiv.2603.10055)Cited by:[§A\.1](https://arxiv.org/html/2609.17831#A1.SS1.p1.1),[§1](https://arxiv.org/html/2609.17831#S1.p2.1),[§1](https://arxiv.org/html/2609.17831#S1.p3.1),[§2](https://arxiv.org/html/2609.17831#S2.SS0.SSS0.Px1.p1.1)\.
- Nomura \(2026\)Y\. NomuraListen and Chant Before You Read: The Ladder of Beauty in LM Pre\-Training\.Note:arXiv:2604\.21265 \[cs\.CL\]Comment: 17 pages, 3 figuresExternal Links:[Link](http://arxiv.org/abs/2604.21265),[Document](https://dx.doi.org/10.5281/zenodo.19702183)Cited by:[§2](https://arxiv.org/html/2609.17831#S2.SS0.SSS0.Px1.p1.1)\.
- Papadimitriou and Jurafsky \(2020\)I\. Papadimitriou and D\. JurafskyLearning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 6829–6839\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.554/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.554)Cited by:[§2](https://arxiv.org/html/2609.17831#S2.SS0.SSS0.Px1.p1.1)\.
- Papadimitriou and Jurafsky \(2023\)I\. Papadimitriou and D\. JurafskyInjecting structural hints: using language models to study inductive biases in language learning\.InFindings of the Association for Computational Linguistics: EMNLP 2023,Cited by:[§1](https://arxiv.org/html/2609.17831#S1.p2.1)\.
- Rosset al\.\(2022\)J\. Ross, B\. Belgodere, V\. Chenthamarakshan, I\. Padhi, Y\. Mroueh, and P\. DasLarge\-Scale Chemical Language Representations Capture Molecular Structure and Properties\.arXiv\.Note:arXiv:2106\.09553 \[cs\.LG\]Comment: NMI 2022External Links:[Link](http://arxiv.org/abs/2106.09553),[Document](https://dx.doi.org/10.48550/arXiv.2106.09553)Cited by:[§A\.1](https://arxiv.org/html/2609.17831#A1.SS1.p1.1),[§A\.1](https://arxiv.org/html/2609.17831#A1.SS1.p2.1),[§1](https://arxiv.org/html/2609.17831#S1.p1.1),[§1](https://arxiv.org/html/2609.17831#S1.p3.1),[§3](https://arxiv.org/html/2609.17831#S3.SS0.SSS0.Px3.p1.1),[§3](https://arxiv.org/html/2609.17831#S3.SS0.SSS0.Px4.p1.1),[§3](https://arxiv.org/html/2609.17831#S3.SS0.SSS0.Px5.p1.1)\.
- Shinnicket al\.\(2025\)Z\. Shinnick, L\. Jiang, H\. Saratchandran, A\. v\. d\. Hengel, and D\. TeneyTransformers pretrained on procedural data contain modular structures for algorithmic reasoning\.arXiv preprint arXiv:2505\.22308\.Cited by:[§1](https://arxiv.org/html/2609.17831#S1.p2.1)\.
- Shinnicket al\.\(2026\)Z\. Shinnick, L\. Jiang, H\. Saratchandran, D\. Teney, and A\. van den HengelCan you learn to see without images? procedural warm\-up for vision transformers\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§1](https://arxiv.org/html/2609.17831#S1.p2.1),[§2](https://arxiv.org/html/2609.17831#S2.SS0.SSS0.Px1.p1.1)\.
- Wuet al\.\(2022\)Y\. Wu, F\. Li, and P\. LiangInsights into Pre\-training via Simpler Synthetic Tasks\.\(en\)\.External Links:[Link](https://openreview.net/forum?id=XiLasGufCM)Cited by:[§1](https://arxiv.org/html/2609.17831#S1.p2.1),[§2](https://arxiv.org/html/2609.17831#S2.SS0.SSS0.Px1.p2.1)\.
- Wuet al\.\(2018\)Z\. Wu, B\. Ramsundar, E\. N\. Feinberg, J\. Gomes, C\. Geniesse, A\. S\. Pappu, K\. Leswing, and V\. PandeMoleculeNet: A Benchmark for Molecular Machine Learning\.arXiv\.Note:arXiv:1703\.00564 \[cs\.LG\]External Links:[Link](http://arxiv.org/abs/1703.00564),[Document](https://dx.doi.org/10.48550/arXiv.1703.00564)Cited by:[§1](https://arxiv.org/html/2609.17831#S1.p1.1),[§1](https://arxiv.org/html/2609.17831#S1.p4.1),[§3](https://arxiv.org/html/2609.17831#S3.SS0.SSS0.Px5.p1.1)\.
- Zhanget al\.\(2025\)S\. Zhang, A\. Patel, S\. A\. Rizvi, N\. Liu, S\. He, A\. Karbasi, E\. Zappala, and D\. van DijkIntelligence at the edge of chaos\.External Links:2410\.02536,[Link](https://arxiv.org/abs/2410.02536)Cited by:[§1](https://arxiv.org/html/2609.17831#S1.p2.1)\.

## Appendix AReproducibility and Training Details

### A\.1Training details

All three stages use the same MoLFormer encoder\([Ross et al\., 2022](https://arxiv.org/html/2609.17831#bib.bib4)\): 12 layers, hidden size 768, 12 heads of dimension 64, feed\-forward width 768, GELU activations, linear attention with 32 generalized random features, rotary position embeddings, a maximum length of 202 tokens and the released 2,362\-token SMILES vocabulary \(46\.8M parameters with the MTM head, 48\.0M with the regression head\)\. All runs use one GPU \(NVIDIA RTX 6000 24GB or A40 48GB\), PyTorch 1\.7\.1 with PyTorch Lightning 1\.1\.5 and apex 22\.03; procedural corpora are generated with numpy, RDKit and networkx \(SSP\) and the JAX generator of[Lee et al\. \(2026\)](https://arxiv.org/html/2609.17831#bib.bib3)\(NCA\)\.

[Table2](https://arxiv.org/html/2609.17831#A1.T2)summarizes the training configuration for each stage\. All stages useapexFusedLAMB withβ1=0\.9\\beta\_\{1\}=0\.9,β2=0\.99\\beta\_\{2\}=0\.99,ϵ=10−6\\epsilon=10^\{\-6\}, no weight decay, and global gradient\-norm clipping at 1\.0\. The molecular\-pretraining and fine\-tuning configurations follow the released MoLFormer implementation\([Ross et al\., 2022](https://arxiv.org/html/2609.17831#bib.bib4)\), with three exceptions: for molecular pretraining we reduce the batch size from 1,200 to 256 and add a 500\-step linear warm\-up, and we fine\-tune on QM9 for 400 rather than 500 epochs\.

Token embeddings remain fixed at their random initialization during procedural pretraining and are trainable during molecular pretraining and downstream fine\-tuning\. Fine\-tuning targets are standardized using statistics computed from the training split, and the checkpoint with the lowest validation MAE is used for test evaluation\. Each procedural and each molecular pretraining is a single run with seed 12345\. Fine\-tuning uses seeds 12345, 42 and 43 \(FreeSolv additionally uses 44, 45 and 46\)\. The seed also draws the QM9 training subset\.

Table 2:Training configurations for the three stages of the experimental pipeline\.
### A\.2Procedural\-task generation

[Table3](https://arxiv.org/html/2609.17831#A1.T3)summarizes the construction and masking policy of each procedural task\. The values evaluated for the task complexity parameters are reported in[Figure7](https://arxiv.org/html/2609.17831#A2.F7)\. ForReverseandSSP, sequences are processed using the MoLFormer tokenizer and masked as in molecular pretraining: of the positions selected for prediction, 80% are replaced by\[MASK\], 10% by a random token, and 10% remain unchanged\.Dyck,Dyck\-Shuffle, andNCAare represented directly as integer token IDs, and every selected position is replaced by\[MASK\]\.

For each task configuration and training budget ofBBsteps, we generateB×256B\\times 256sequences once, using random seed 42, and train for a single pass over them; the checkpoint at stepBB, the end of the cosine schedule, is the one transferred, and no validation set is held out\.

Table 3:Procedural\-task generation and masking\. For each task, we report the generated sequence, the complexity parameter varied in our experiments, and the tokens eligible for masking\. Each eligible token is selected independently with probabilitypmaskp\_\{\\mathrm\{mask\}\}\.

## Appendix BChoosing Task Complexity and Procedural Budget

[Figure7](https://arxiv.org/html/2609.17831#A2.F7)shows the validation grid used to select task complexity and procedural\-pretraining budget for each task\.

![Refer to caption](https://arxiv.org/html/2609.17831v1/task001_task_grids_heatmap.png)Figure 7:Grid search over task complexity and procedural\-pretraining budget\. Cells report mean validation\-set Lipophilicity benefit over the randomly initialized baseline acrossn=3n=3seeds\.
## Appendix CProcedural Pretraining Losses and Bayes\-Floor Estimates

In[Table4](https://arxiv.org/html/2609.17831#A3.T4), we report the procedural loss reached at every budget and the downstream benefit of the respective model on Lipophilicity over the random initialization baseline\. We also mention the Bayes\-optimal achievable loss for each task setting where we are able to determine it\.

Table 4:Procedural loss and downstream benefit for each tested task\-complexity\-duration combination\.L⋆L^\{\\star\}denotes the Bayes\-optimal loss, which is unknown in closed form for NCA\. Budgetsb1b\_\{1\}–b6b\_\{6\}are\{500,1500,3000,6000,12000,24000\}\\\{500,1500,3000,6000,12000,24000\\\}optimizer steps for the sequence tasks and\{250,750,1500,3000,4500,6000\}\\\{250,750,1500,3000,4500,6000\\\}for NCA\.
## Appendix DShuffling Token Order Removes Downstream Benefit Across All Procedural Tasks

In[Section4\.2\.4](https://arxiv.org/html/2609.17831#S4.SS2.SSS4), we found that shuffling the token order in the procedural sequences eliminates all downstream benefit\. This section reports the shuffled\-token ablation in the two\-stage diagnostic setting for all five procedural tasks: procedural pretraining is directly followed by Lipophilicity fine\-tuning\.[Figure8](https://arxiv.org/html/2609.17831#A4.F8)shows that every procedural task loses its downstream benefit when the token order within the sequences is shuffled\.

Figure 8:Token shuffling removes the downstream benefit of procedural pretraining across all five tasks in the two\-stage diagnostic setting\. Models are pretrained on either the structured corpus or a version in which tokens are shuffled within each sequence, and are then fine\-tuned directly on Lipophilicity without molecular pretraining\. Shuffling preserves each sequence’s length and token counts while disrupting order\-dependent structure, indicating that marginal token statistics alone are insufficient to explain the observed benefit\.