冻结于时间：物理基础模型如何在仿真到现实的适应过程中遗忘

# Frozen in Time: How a Physics Foundation Model Forgets Under Sim-to-Real Adaptation
Source: [https://arxiv.org/html/2610.40857](https://arxiv.org/html/2610.40857)
\\workshoptitle

Sim2Science: ML with Imperfect Scientific Models

Miles CranmerAffiliation:University of CambridgeAffiliation:Polymathic AIAlessandro FaveroAffiliation:University of CambridgeAffiliation:Polymathic AIPayel MukhopadhyayAffiliation:University of CambridgeAffiliation:Polymathic AI

###### Abstract

Scientific foundation models \(SciFMs\) have achieved remarkable performance on a wide range of PDEs, including the ability to efficiently adapt to unseen data distributions with additional finetuning\. As with all neural architectures, finetuning such models comes with the risk ofcatastrophic forgetting, whereby adapting to new tasks degrades prior knowledge of the model\. We study catastrophic forgetting in Walrus\-1\.3B, a SciFM for continuum dynamics, over the course of a two\-stage sim\-to\-real adaptation: first finetuning on simulated flow around a cylinder and then adapting to experimental data of the same system\. We find that finetuning on the simulations rapidly degrades rollout performance on the original pretraining datasets, with one\-step errors increasing up to 35×\\timesafter just 20 epochs\. Subsequent adaptation to real data further erodes performance on the simulated task\. We characterize the potential drivers of forgetting, such as dataset dependence and task representation similarity\. We then show that replaying as little as 1% simulated data during real\-data finetuning recovers most of the lost simulation performance while retaining effective adaptation to the experimental dynamics\. Our results identify catastrophic forgetting as a practical obstacle to sim\-to\-real adaptation of SciFMs and show that lightweight replay strategies significantly improve retention\.

††footnotetext:∗Core technical contributors\.†Equal advising\. Correspondence:ham51@cam\.ac\.uk## 1Introduction

In recent years, large\-scale scientific foundation models \(SciFMs\), such as Walrus\[[1](https://arxiv.org/html/2610.40857#bib.bib1)\], DPOT\-L\[[2](https://arxiv.org/html/2610.40857#bib.bib2)\], Poseidon\[[3](https://arxiv.org/html/2610.40857#bib.bib3)\], and many others\[[4](https://arxiv.org/html/2610.40857#bib.bib4),[5](https://arxiv.org/html/2610.40857#bib.bib5),[6](https://arxiv.org/html/2610.40857#bib.bib6),[7](https://arxiv.org/html/2610.40857#bib.bib7),[8](https://arxiv.org/html/2610.40857#bib.bib8),[9](https://arxiv.org/html/2610.40857#bib.bib9),[10](https://arxiv.org/html/2610.40857#bib.bib10),[11](https://arxiv.org/html/2610.40857#bib.bib11),[12](https://arxiv.org/html/2610.40857#bib.bib12),[13](https://arxiv.org/html/2610.40857#bib.bib13),[14](https://arxiv.org/html/2610.40857#bib.bib14),[15](https://arxiv.org/html/2610.40857#bib.bib15)\], have adapted transformer\-based architectures similar to those used in large language and vision models to the problem of physics simulation, pretraining with vast amounts of numerically simulated PDE data\[[16](https://arxiv.org/html/2610.40857#bib.bib16),[17](https://arxiv.org/html/2610.40857#bib.bib17),[18](https://arxiv.org/html/2610.40857#bib.bib18),[19](https://arxiv.org/html/2610.40857#bib.bib19)\]\. This purely data\-driven approach to surrogate modeling has seen considerable success across a wide variety of physics domains, including fluid dynamics, weather, and astrophysics\[[20](https://arxiv.org/html/2610.40857#bib.bib20),[21](https://arxiv.org/html/2610.40857#bib.bib21),[22](https://arxiv.org/html/2610.40857#bib.bib22)\]\.

To adapt foundation models to downstream tasks or unseen data distributions, additional finetuning is generally required from the pretrained baseline\. However, the gains from finetuning are not without tradeoffs\. In particular, supervised fine\-tuning runs the risk ofcatastrophic forgetting,\[[23](https://arxiv.org/html/2610.40857#bib.bib23),[24](https://arxiv.org/html/2610.40857#bib.bib24),[25](https://arxiv.org/html/2610.40857#bib.bib25)\]well\-characterized in the vision and language model literature, whereby weight updates made to the model during finetuning result in severe degradation of the pretrained model’s prior capabilities\. This phenomenon has spurred the growth of a subfield dubbed Continual Learning \(CL\), aimed at developing methods to continually adapt models to new tasks while preserving prior knowledge and mitigating the problem of catastrophic forgetting\[[26](https://arxiv.org/html/2610.40857#bib.bib26)\]\.

To deploy SciFMs in real\-world scientific or engineering applications, models must ultimately make predictions on experimental data\. However, a large gap exists between the simulated PDE data used for pretraining and the real, noisy dynamics observed in a fluids laboratory, known as thesim\-to\-real gap\[[27](https://arxiv.org/html/2610.40857#bib.bib27),[28](https://arxiv.org/html/2610.40857#bib.bib28)\]\. Recent work demonstrated zero\-shot transfer from such models to laboratory data, and additionally showed that a lightweight second finetuning stage on few samples can further improve the results\[[29](https://arxiv.org/html/2610.40857#bib.bib29)\]\. While finetuning provides a general strategy to adapt a simulation\-trained model to experimental fluid flows, it is currently unclear how such adaptation affects the prior knowledge acquired during pretraining, or whether learning from real data further degrades recently\-gained simulation knowledge\.

In this work, we study these questions in Walrus\-1\.3B using a two\-stage adaptation setting: we first finetune the pretrained model on simulated flow around a cylinder, and then adapt it to pair experimental particle\-image\-velocimetry \(PIV\) data of the same system\. We ask how and what does a large PDE model forget upon exposure to unseen physics or experimental data? What is the severity of forgetting and its onset? Is forgotten physics recoverable or retainable through CL methods such as replay? We find that forgetting begins rapidly during simulated\-cylinder finetuning, with prediction error on some pretraining datasets increasing by up to35×35\\timesafter just 20 epochs\. Later adaptation to real PIV data further degrades performance on the simulated task\. Remarkably, we show that replaying even a small fraction of simulated data substantially restores the lost performance\.

![Refer to caption](https://arxiv.org/html/2610.40857v1/fig1_schematic.png)Figure 1:Setup\.Walrus\-1\.3B, pretrained on The Well, is fine\-tuned on simulated cylinder flow \(Phase I\), then adapted to real PIV of the same flow \(Phase II\)\. Phase I induces catastrophic forgetting, so Phase II replays simulation at a fixed fraction\.
## 2Sim\-to\-Real Adaptation

We study the adaptation of Walrus\[[1](https://arxiv.org/html/2610.40857#bib.bib1)\], a physics foundation model for continuum dynamics, to paired fluid dynamics datasets of flow around a fixed cylinder in two phases\.111See Appendix[C](https://arxiv.org/html/2610.40857#A3)for detailed experimental setup and motivation of sim\-pretraining phase\.Phase I consists of finetuning the pretrained model on simulated flows, and Phase II consists of adaptation of the Phase I model to real cylinder flows obtained from imaging\. We then perform an analysis of forgetting through inference and evaluation on prior data \(both from the Well and from cylinder simulation\) and attempt to mitigate forgetting in Phase II through a replay\-based strategy\.222Code:[https://github\.com/haarisamian/walrus\-forgetting\-paper](https://github.com/haarisamian/walrus-forgetting-paper)

#### Model\.

Walrus is a 1\.3B parameter large physics model trained on 19 different physical scenarios from the Well collection of PDE datasets\[[18](https://arxiv.org/html/2610.40857#bib.bib18)\], including acoustic scattering, shear flow, and a wide range of other continuum dynamics datasets\. Walrus is an autoregressive transformer model and was pretrained on a next\-frame prediction task\. In particular, it was trained to output a delta \(difference\) between the last input frame and the next frame\.

#### Normalization\.

An important design choice in Walrus is the normalization strategy, applied to a given channelccin the input fields before the forward pass to the model, i\.e\.,x^c=xc/σc\\hat\{x\}\_\{c\}=x\_\{c\}/\\sigma\_\{c\}whereσc=𝔼⁡\[xc2\]\\sigma\_\{c\}=\\sqrt\{\\mathbb\{E\}\[x\_\{c\}^\{2\}\]\}, Analogously,σcΔ=𝔼⁡\[Δc2\]\\sigma\_\{c\}^\{\\Delta\}=\\sqrt\{\\mathbb\{E\}\[\\Delta\_\{c\}^\{2\}\]\}is applied to the delta target outputted by the model\. This RMS normalization ensures that the model “sees” inputs from datasets with widely varying amplitudes at a roughly similar scale, stabilizing prediction and training\.

However, these RMS statistics can be computed in different ways\. Undersamplewise normalization\(SW\), the model receives an input context which is normalized by its own RMS statistics for each step during training and at inference time\. Inglobal normalization\(GN\), the model sees data normalized by RMS statistics across the whole dataset, which preserves amplitude variation between trajectories\. We analyze the impact of normalization choice in our experiments, as it strongly affects both adaptation and the dynamics of forgetting\.

![Refer to caption](https://arxiv.org/html/2610.40857v1/fig_sw_vs_gn_rollout.png)Figure 2:Predictedvxv\_\{x\}at rollout stepst=20t\{=\}20andt=60t\{=\}60after 40 epochs of Phase II PIV fine\-tuning\.GNpredictions track the downstream vortex shedding in the ground truth;SWpredictions collapse to a near\-static field, witht=60t\{=\}60nearly indistinguishable fromt=20t\{=\}20\. Both models share training data, epoch count, and rollout trajectory\.
#### Cylinder Flow Dataset\.

We finetune the model to the datasets described here, which are previously unseen as vortex shedding or cylinder data is not present in the pretraining data\. Crucially, the pretraining data contains no real or experimentally obtained data, so adapting to PIV \(Particle Image Velocimetry\) presents an even greater challenge for the model\. We obtain data from the RealPDEBench paper\[[27](https://arxiv.org/html/2610.40857#bib.bib27)\], which provided paired trajectory data for five fluid dynamics scenarios\.

The cylinder dataset explores a canonical setup in fluid dynamics, the flow around a rigid cylindrical object, through a set of 92 paired real and simulated trajectories of 3990 steps indexed by Reynolds numbers ranging from 1800–12000\. The characteristic phenomenon is von Kármán vortex shedding, whereby incoming flow leads to an alternating periodic wake of vortices downstream of the cylinder\[[30](https://arxiv.org/html/2610.40857#bib.bib30)\]\. The shedding frequency is characteristic of the experimental set\-up and scales roughly with the Reynolds number\[[31](https://arxiv.org/html/2610.40857#bib.bib31)\]\. At low Re, the flow is nearly laminar and vortices are regular and smooth, while at increasing Re, the flow becomes more turbulent, and the shedding becomes chaotic while still staying periodic\. While simulated data is obtained via numerical simulation in 2D with Lilypad solvers\[[32](https://arxiv.org/html/2610.40857#bib.bib32)\], the real data is obtained through PIV, in which tracer particles are imaged at regular intervals to infer displacement and calculate velocity flows over time\.

#### Phase I: Simulated Cylinder Finetuning\.

In this phase, we begin from the pretrained checkpoint of Walrus and finetune on cylinder simulation using a range of training protocols, varying both the normalization strategy \(samplewise vs\. global\) and learning rate, and evaluate performance on the new data using one\-step and rollout validation, as well as qualitative video rollouts\. In addition to this main sweep, we ran a separate, dedicated finetuning trial \(global normalization,lr=5×10−5\\text\{lr\}=5\\times 10^\{\-5\}, 20 epochs, checkpoints every 5 epochs\) to study forgetting dynamics at finer granularity in the earliest epochs of finetuning, evaluating each of the early checkpoints against the original Well pretraining datasets\.

#### Phase II: Real Cylinder Finetuning\.

In Phase II, we continue adapting the Phase I finetuned checkpoint to real PIV cylinder data\. Global norm finetuning was the only strategy that successfully learned the real PIV rollout dynamics; samplewise training failed to learn the PIV vortex shedding at all, so we adopt global norm as the backbone for all subsequent Phase II experiments\.

#### Samplewise normalization can cause frozen rollouts\.

Figure[2](https://arxiv.org/html/2610.40857#S2.F2)illustrates how SW fine\-tuning fails in the real data adaptation phase\. After 40 epochs of PIV training, the SW model’s rollout predictions collapse to a near\-static field\. Between rollout stepst=30t\{=\}30andt=60t\{=\}60the SW predictedvxv\_\{x\}field barely changes, with no visible vortex shedding, while the ground truth continues to evolve dynamically in line with ground truth flows\. We hypothesize this is caused by the samplewise statistics decaying over the course of rollout on out\-of\-distribution real data\. The persistent zero\-delta prediction learned under samplewise finetuning is a shortcut that prevents Walrus from predicting the core vortex dynamics of the real cylinder setup, even when exposed to simulated cylinder flows\.

#### Normalization strategy determines Phase II success\.

A central finding is that replacing Walrus’s native samplewise normalization with global normalization \(GN\) before real\-data finetuning is necessary for effective sim\-to\-real transfer\. With GN, Phase II training on real cylinder PIV converges steadily over 100 epochs, reaching a 60\-step rollout VRMSE \(variance\-scaled RMSE\) of 0\.318 on the validation set \(Figure[4](https://arxiv.org/html/2610.40857#S3.F4), left\), with the model continuing to improve through the final epoch\. The samplewise \(SW\) variant fails qualitatively before reaching any comparable performance\.

## 3Catastrophic Forgetting & Replay

Figure 3:Forgetting on the Well benchmark across both phases of finetuning\. The selected pretraining datasets from the Well degrade mostly monotonically\. Active Matter and TRL\-D degrade most severely out of these four, while MHD suffers less\.#### Evaluating & mitigating forgetting\.

To characterize forgetting at fine temporal resolution, we ran a 20\-epoch Phase 1 finetuning run \(global norm, lr=5×10−55\\times 10^\{\-5\}\) checkpointed every 5 epochs, and evaluate each checkpoint against ten held\-out Well pretraining datasets: MHD, turbulent radiative layer, turbulence gravity cooling, active matter, acoustic scattering, planetary shallow water, Helmholtz staircase, and shear flow\. Our primary metric of forgetting in this case is step\-1 VRMSE\. Additionally, we measure interphase forgetting on the cylinder simulation due to cylinder PIV finetuning\. To mitigate the forgetting of Phase I dynamics in Phase II, we explore replaying\[[33](https://arxiv.org/html/2610.40857#bib.bib33)\]a fraction of simulated cylinder data alongside real PIV data, sweeping the replay ratio coarsely in increments of 10% and then finely between 1\-10%\.

Table 1:Catastrophic forgetting during Phase I cylinder simulation fine\-tuning, under both error normalizations\. NRMSE and VRMSE differ only for fields with a non\-zero mean\. All ratios \(×\\times\) and absolute increases \(Δ\\Delta\) are at epoch 20, relative to the pretrained baseline\.*CKA*is the linear centered kernel alignment between the dataset’s and the cylinder simulation’s activation covariances at the input to the final block of the pretrained model\.*Pers\.*is persistence, the NRMSE of copying the previous frame\. Both predictors are computed before any fine\-tuning\.NRMSEVRMSEPredictorsDatasetPre×\\timesΔ\\DeltaPre×\\timesΔ\\DeltaCKAPers\.Fluid dynamicsActive Matter0\.01063\.880\.03060\.01454\.140\.04550\.0270\.312TRL\-2D0\.03802\.520\.05780\.11482\.330\.15250\.1460\.250Turbcool0\.03921\.820\.03230\.08421\.900\.07570\.4630\.096Shear Flow0\.00772\.210\.00930\.00792\.290\.01020\.0340\.078MagnetohydrodynamicsMHD\-640\.12341\.070\.00850\.14931\.070\.01010\.0150\.356Wave / acousticsHelmholtz0\.001134\.890\.03630\.001337\.660\.04660\.0500\.136Acoustic \(disc\.\)0\.006630\.630\.19430\.007230\.800\.21410\.8840\.326Acoustic \(incl\.\)0\.016611\.690\.17740\.017811\.800\.19230\.8800\.314Acoustic \(maze\)0\.039410\.280\.36540\.039810\.300\.37040\.7760\.865GeophysicsPlanet SWE0\.000728\.480\.01930\.000728\.440\.01940\.2220\.026

#### Catastrophic forgetting is rapid and dataset\-dependent\.

Even before real\-data fine\-tuning begins, Phase I cylinder simulation fine\-tuning inflicts severe forgetting across the Well benchmark \(Figure[3](https://arxiv.org/html/2610.40857#S3.F3), Table[1](https://arxiv.org/html/2610.40857#S3.T1)\)\. The onset of forgetting is remarkably fast\. Within the first five epochs of Phase I fine\-tuning, acoustic scattering datasets exhibit mean NRMSE \(normalized RMSE\) ratios of×\\times4\.6–14\.6, and Active Matter degrades by×\\times2\.3 \(Figure[3](https://arxiv.org/html/2610.40857#S3.F3)\)\. By epoch 20 the wave and geophysics datasets have collapsed catastrophically \(Helmholtz×\\times34\.9, Planet SWE×\\times28\.5\), while MHD barely moves \(×\\times1\.07\)\. Absolute degradation ranks the datasets differently: Acoustic \(maze\) suffers the largest true loss \(Δ\\DeltaVRMSE0\.370\.37\), while Helmholtz and Planet SWE post large ratios only because their pretrained errors were near zero\. Either way the spread is wide, a37×37\\timesrange inΔ\\DeltaVRMSE\.

#### Predicting forgetting before finetuning\.

Degradation varies by more than an order of magnitude across the datasets, so we ask whether it can be anticipated before any finetuning\. We consider two observables computed on the pretrained model\. We define*persistence*as the one\-step NRMSE of simply copying the previous frame, which measures how much a field changes between frames and therefore the temporal evolution the model should capture beyond a trivial copy\-forward predictor\. We then quantify*representational similarity*between each dataset and the cylinder simulation using the linear centered kernel alignment \(CKA\)\[[34](https://arxiv.org/html/2610.40857#bib.bib34)\]of their activation covariances, taken at the final block of the model\. Together persistence and CKA account for82%82\\%of the variance in absolute degradation across the ten datasets\. The two appear to capture complementary factors\.333These two predictors are effectively independent \(Variance Inflation Factor1\.031\.03\), and each remains significant after controlling for the other \(p<0\.05p<0\.05\)\.Higher persistence means more temporal structure is at stake, and so more to lose; higher CKA means a dataset relies on representations closer to those used for the cylinder task, which is consistent with greater interference during fine\-tuning\. Accordingly, MHD\-64 \(CKA=0\.015\\mathrm\{CKA\}=0\.015\) is largely preserved despite its relative high persistence error \(0\.3560\.356\), whereas the acoustic datasets have substantially higher similarity to the finetuning task \(CKA∈\[0\.776,0\.884\]\\mathrm\{CKA\}\\in\[0\.776,0\.884\]\), and degrade much more strongly\. Within the acoustic datasets, maze has the largest persistence error \(0\.8650\.865\) and also suffers the greatest absolute degradation of any dataset, consistent with persistence contributing independently of representational similarity\. We refer the reader to Appendix[C](https://arxiv.org/html/2610.40857#A3)for further details\.

Figure 4:\(Left\) Rollout VRMSE during Phase II GN training on real cylinder PIV reaches 0\.318 at epoch 100, while cylinder\-simulation skill is lost within the first 20 epochs and never recovers\. \(Right\) Cylinder\-simulation rollout VRMSE versus the replay fraction mixed into Phase II fine\-tuning\. Even a 1% mixture recovers most of the performance lost under no\-replay training \(red dashed\)\.
#### Sim replay provides an efficient remedy\.

Figure[4](https://arxiv.org/html/2610.40857#S3.F4)\(right\) shows that mixing even a small fraction of Phase I simulation data into Phase II PIV training largely restores cylinder simulation performance and mitigates catastrophic forgetting on the first task\. Without replay, Phase II GN→\\rightarrowGN training raises rollout NRMSE from the Phase I baseline of 0\.045 to 0\.377, an8×8\\timesdegradation\. At just1% simulated replay, NRMSE falls to below 0\.13, recovering\>65%\>65\\%of the lost rollout performance\. Notably, above 60% replay the model even surpasses the pure sim finetuning baseline, suggesting that at high fractions the model can continue to improve its performance on cylinder simulations without sacrificing performance on PIV data\.

## 4Conclusion

In this work, we explored the challenges of adapting large physics models to experimental and out\-of\-distribution data through fine\-tuning and its associated impact on prior task performance\. We observe catastrophic forgetting in a scientific foundation model within a few epochs of finetuning, with the severity of forgetting differing by finetuning phase \(sim vs\. real\), persistence, and representational task similarity\. Replay mitigates this forgetting on cylinder flow, preserving simulated performance while adapting to experimental data\. This controlled characterization opens the door to continual learning methods for large physics models in out\-of\-distribution settings\.

## Acknowledgments and Disclosure of Funding

Polymathic AI is funded by the Simons Foundation and Schmidt Sciences\. This work was supported in part by the AI2050 program at Schmidt Sciences \(Grant G\-25\-70028\); M\.C\. is supported by an AI2050 Early Career Fellowship and the Isaac Newton Trust\. The authors thank the Infosys–Cambridge AI Centre and the Swirles HPC system at the University of Cambridge\.

## References

- \[1\]Michael McCabe, Payel Mukhopadhyay, Tanya Marwah, Bruno Régaldo\-Saint Blancard, François Rozet, Cristiana Diaconu, Lucas Thibaut Meyer, Kaze W\. K\. Wong, Mohammad\-Hadi Sotoudeh, Alberto Bietti, Irina Espejo Morales, Rio Alexa Fear, Siavash Golkar, Tom Hehir, Keiya Hirashima, Geraud Krawezik, Francois Lanusse, Rudy Morel, Ruben Ohana, Liam Holden Parker, Mariel Pettee, Jeff Shen, Kyunghyun Cho, Miles Cranmer, and Shirley Ho\.Walrus: A cross\-domain foundation model for continuum dynamics\.In*Forty\-third International Conference on Machine Learning*, 2026\.
- \[2\]Zhongkai Hao, Chang Su, Songming Liu, Julius Berner, Chengyang Ying, Hang Su, Anima Anandkumar, Jian Song, and Jun Zhu\.Dpot: Auto\-regressive denoising operator transformer for large\-scale pde pre\-training\.*arXiv preprint arXiv:2403\.03542*, 2024\.
- \[3\]Maximilian Herde, Bogdan Raonić, Tobias Rohner, Roger Käppeli, Roberto Molinaro, Emmanuel de Bézenac, and Siddhartha Mishra\.Poseidon: Efficient foundation models for pdes\.In A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang, editors,*Advances in Neural Information Processing Systems*, volume 37, pages 72525–72624\. Curran Associates, Inc\., 2024\.doi:10\.52202/079017\-2311\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2024/file/84e1b1ec17bb11c57234e96433022a9a\-Paper\-Conference\.pdf](https://proceedings.neurips.cc/paper_files/paper/2024/file/84e1b1ec17bb11c57234e96433022a9a-Paper-Conference.pdf)\.
- \[4\]Yadi Cao, Yuxuan Liu, Liu Yang, Rose Yu, Hayden Schaeffer, and Stanley Osher\.VICON: Vision in\-context operator networks for multi\-physics fluid dynamics prediction\.*Transactions on Machine Learning Research*, 2026\.ISSN 2835\-8856\.
- \[5\]Benjamin Holzschuh, Qiang Liu, Georg Kohl, and Nils Thuerey\.Pde\-transformer: Efficient and versatile transformers for physics simulations\.*arXiv preprint arXiv:2505\.24717*, 2025\.
- \[6\]Yuxuan Liu, Jingmin Sun, Xinjie He, Griffin Pinney, Zecheng Zhang, and Hayden Schaeffer\.Prose\-fd: A multimodal pde foundation model for learning multiple operators for forecasting fluid dynamics, 2024\.URL[https://arxiv\.org/abs/2409\.09811](https://arxiv.org/abs/2409.09811)\.
- \[7\]Qiang Liu, Felix Koehler, Benjamin Holzschuh, and Nils Thuerey\.Tadpole: Autoencoders as foundation models for 3d pdes with online learning, 2026\.URL[https://arxiv\.org/abs/2605\.15284](https://arxiv.org/abs/2605.15284)\.
- \[8\]Rudy Morel, Jiequn Han, and Edouard Oyallon\.Disco: learning to discover an evolution operator for multi\-physics\-agnostic prediction, 2025\.URL[https://arxiv\.org/abs/2504\.19496](https://arxiv.org/abs/2504.19496)\.
- \[9\]Shashank Subramanian, Peter Harrington, Kurt Keutzer, Wahid Bhimji, Dmitriy Morozov, Michael Mahoney, and Amir Gholami\.Towards foundation models for scientific machine learning: Characterizing scaling and transfer behavior\.In A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine, editors,*Advances in Neural Information Processing Systems*, volume 36, pages 71242–71262\. Curran Associates, Inc\., 2023\.doi:10\.52202/075280\-3119\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2023/file/e15790966a4a9d85d688635c88ee6d8a\-Paper\-Conference\.pdf](https://proceedings.neurips.cc/paper_files/paper/2023/file/e15790966a4a9d85d688635c88ee6d8a-Paper-Conference.pdf)\.
- \[10\]Ashiqur Rahman, Robert Joseph George, Mogab Elleithy, Daniel Leibovici, Zongyi Li, Boris Bonev, Colin White, Julius Berner, Raymond A\. Yeh, Jean Kossaifi, Kamyar Azizzadenesheli, and Anima Anandkumar\.Pretraining codomain attention neural operators for solving multiphysics pdes\.In A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang, editors,*Advances in Neural Information Processing Systems*, volume 37, pages 104035–104064\. Curran Associates, Inc\., 2024\.doi:10\.52202/079017\-3306\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2024/file/bc75fa9843a7905bbed9d83895a88f7f\-Paper\-Conference\.pdf](https://proceedings.neurips.cc/paper_files/paper/2024/file/bc75fa9843a7905bbed9d83895a88f7f-Paper-Conference.pdf)\.
- \[11\]Michael McCabe, Bruno Régaldo\-Saint Blancard, Liam Parker, Ruben Ohana, Miles Cranmer, Alberto Bietti, Michael Eickenberg, Siavash Golkar, Geraud Krawezik, Francois Lanusse, Mariel Pettee, Tiberiu Tesileanu, Kyunghyun Cho, and Shirley Ho\.Multiple physics pretraining for spatiotemporal surrogate models\.In A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang, editors,*Advances in Neural Information Processing Systems*, volume 37, pages 119301–119335\. Curran Associates, Inc\., 2024\.doi:10\.52202/079017\-3791\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2024/file/d7cb9db5ade2db7814fbd01ee59f4c7b\-Paper\-Conference\.pdf](https://proceedings.neurips.cc/paper_files/paper/2024/file/d7cb9db5ade2db7814fbd01ee59f4c7b-Paper-Conference.pdf)\.
- \[12\]Tung Nguyen, Arsh Koneru, Shufan Li, and Aditya Grover\.Physix: A foundation model for physics simulations\.*arXiv preprint arXiv:2506\.17774*, 2025\.
- \[13\]Jingmin Sun, Yuxuan Liu, Zecheng Zhang, and Hayden Schaeffer\.Towards a foundation model for partial differential equations: Multioperator learning and extrapolation\.*Phys\. Rev\. E*, 111:035304, Mar 2025\.doi:10\.1103/PhysRevE\.111\.035304\.URL[https://link\.aps\.org/doi/10\.1103/PhysRevE\.111\.035304](https://link.aps.org/doi/10.1103/PhysRevE.111.035304)\.
- \[14\]Mahindra Singh Rautela, Alexander Most, Siddharth Mansingh, Bradley C\. Love, Alexander Scheinker, Diane Oyen, Nathan Debardeleben, Earl Lawrence, and Ayan Biswas\.Morph: Pde foundation models with arbitrary data modality, 2026a\.URL[https://arxiv\.org/abs/2509\.21670](https://arxiv.org/abs/2509.21670)\.
- \[15\]Louis Serrano, Jiequn Han, Edouard Oyallon, Shirley Ho, and Rudy Morel\.Test\-time generalization for physics through neural operator splitting, 2026\.URL[https://arxiv\.org/abs/2602\.00884](https://arxiv.org/abs/2602.00884)\.
- \[16\]Kalin Kanov, Randal Burns, Cristian Lalescu, and Gregory Eyink\.The johns hopkins turbulence databases: An open simulation laboratory for turbulence research\.*Computing in Science & Engineering*, 17\(5\):10–17, 2015\.doi:10\.1109/MCSE\.2015\.103\.
- \[17\]Sungduk Yu, Walter Hannah, Liran Peng, Jerry Lin, Mohamed Aziz Bhouri, Ritwik Gupta, Björn Lütjens, Justus Christopher Will, Gunnar Behrens, Julius Busecke, Nora Loose, Charles I Stern, Tom Beucler, Bryce Harrop, Benjamin R Hillman, Andrea Jenney, Savannah Ferretti, Nana Liu, Anima Anandkumar, Noah D Brenowitz, Veronika Eyring, Nicholas Geneva, Pierre Gentine, Stephan Mandt, Jaideep Pathak, Akshay Subramaniam, Carl Vondrick, Rose Yu, Laure Zanna, Tian Zheng, Ryan Abernathey, Fiaz Ahmed, David C Bader, Pierre Baldi, Elizabeth Barnes, Christopher Bretherton, Peter Caldwell, Wayne Chuang, Yilun Han, YU HUANG, Fernando Iglesias\-Suarez, Sanket Jantre, Karthik Kashinath, Marat Khairoutdinov, Thorsten Kurth, Nicholas Lutsko, Po\-Lun Ma, Griffin Mooers, J\. David Neelin, David Randall, Sara Shamekh, Mark A Taylor, Nathan Urban, Janni Yuval, Guang Zhang, and Michael Pritchard\.Climsim: A large multi\-scale dataset for hybrid physics\-ML climate emulation\.In*Thirty\-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track*, 2023\.URL[https://openreview\.net/forum?id=W5If9P1xqO](https://openreview.net/forum?id=W5If9P1xqO)\.
- \[18\]Ruben Ohana, Michael McCabe, Lucas Thibaut Meyer, Rudy Morel, Fruzsina Julia Agocs, Miguel Beneitez, Marsha Berger, Blakesley Burkhart, Stuart B\. Dalziel, Drummond Buschman Fielding, Daniel Fortunato, Jared A\. Goldberg, Keiya Hirashima, Yan\-Fei Jiang, Rich Kerswell, Suryanarayana Maddu, Jonah M\. Miller, Payel Mukhopadhyay, Stefan S\. Nixon, Jeff Shen, Romain Watteaux, Bruno Régaldo\-Saint Blancard, François Rozet, Liam Holden Parker, Miles Cranmer, and Shirley Ho\.The well: a large\-scale collection of diverse physics simulations for machine learning\.In*The Thirty\-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track*, 2024\.URL[https://openreview\.net/forum?id=00Sx577BT3](https://openreview.net/forum?id=00Sx577BT3)\.
- \[19\]Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Daniel MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert\.Pdebench: An extensive benchmark for scientific machine learning\.In S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh, editors,*Advances in Neural Information Processing Systems*, volume 35, pages 1596–1611\. Curran Associates, Inc\., 2022\.doi:10\.52202/068431\-0117\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2022/file/0a9747136d411fb83f0cf81820d44afb\-Paper\-Datasets\_and\_Benchmarks\.pdf](https://proceedings.neurips.cc/paper_files/paper/2022/file/0a9747136d411fb83f0cf81820d44afb-Paper-Datasets_and_Benchmarks.pdf)\.
- \[20\]Liam Holden Parker, Francois Lanusse, Jeff Shen, Ollie Liu, Tom Hehir, Leopoldo Sarra, Lucas Thibaut Meyer, Micah Bowles, Sebastian Wagner\-Carena, Helen Qu, Siavash Golkar, Alberto Bietti, Hatim Bourfoune, Pierre Cornette, Keiya Hirashima, Geraud Krawezik, Ruben Ohana, Nicholas Lourie, Michael McCabe, Rudy Morel, Payel Mukhopadhyay, Mariel Pettee, Kyunghyun Cho, Miles Cranmer, and Shirley Ho\.AION\-1: Omnimodal foundation model for astronomical sciences\.In*The Thirty\-ninth Annual Conference on Neural Information Processing Systems*, 2025\.URL[https://openreview\.net/forum?id=6gJ2ZykQ5W](https://openreview.net/forum?id=6gJ2ZykQ5W)\.
- \[21\]Kimberly Stachenfeld, Drummond Fielding, Dmitrii Kochkov, Miles Cranmer, Tobias Pfaff, Jonathan Godwin, Can Cui, Shirley Ho, Peter Battaglia, and Alvaro Sanchez\-Gonzalez\.Learned coarse models for efficient turbulence simulation\.In*International Conference on Learning Representations*, 2022\.
- \[22\]Remi Lam, Alvaro Sanchez\-Gonzalez, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton\-Rosen, Weihua Hu, Alexander Merose, Stephan Hoyer, George Holland, Oriol Vinyals, Jacklynn Stott, Alexander Pritzel, Shakir Mohamed, and Peter Battaglia\.Learning skillful medium\-range global weather forecasting\.*Science*, 382\(6677\):1416–1421, 2023\.doi:10\.1126/science\.adi2336\.URL[https://www\.science\.org/doi/abs/10\.1126/science\.adi2336](https://www.science.org/doi/abs/10.1126/science.adi2336)\.
- \[23\]Zhizhong Li and Derek Hoiem\.Learning without forgetting\.*IEEE Transactions on Pattern Analysis and Machine Intelligence*, 40\(12\):2935–2947, 2018\.doi:10\.1109/TPAMI\.2017\.2773081\.
- \[24\]James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A\. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska\-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell\.Overcoming catastrophic forgetting in neural networks\.*Proceedings of the National Academy of Sciences*, 114\(13\):3521–3526, 2017\.doi:10\.1073/pnas\.1611835114\.URL[https://www\.pnas\.org/doi/abs/10\.1073/pnas\.1611835114](https://www.pnas.org/doi/abs/10.1073/pnas.1611835114)\.
- \[25\]Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang\.An empirical study of catastrophic forgetting in large language models during continual fine\-tuning\.*IEEE Transactions on Audio, Speech and Language Processing*, 33:3776–3786, 2025\.doi:10\.1109/TASLPRO\.2025\.3606231\.
- \[26\]Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars\.A Continual Learning Survey: Defying Forgetting in Classification Tasks \.*IEEE Transactions on Pattern Analysis & Machine Intelligence*, 44\(07\):3366–3385, July 2022\.ISSN 1939\-3539\.doi:10\.1109/TPAMI\.2021\.3057446\.URL[https://doi\.ieeecomputersociety\.org/10\.1109/TPAMI\.2021\.3057446](https://doi.ieeecomputersociety.org/10.1109/TPAMI.2021.3057446)\.
- \[27\]Peiyan Hu, Haodong Feng, Hongyuan Liu, Tongtong Yan, Wenhao Deng, Tianrun Gao, Rong Zheng, Haoren Zheng, Chenglei Yu, Chuanrui Wang, Kaiwen Li, Zhi\-Ming Ma, Dezhi Zhou, Xingcai Lu, Dixia Fan, and Tailin Wu\.RealPDEBench: A benchmark for complex physical systems with real\-world data\.In*The Fourteenth International Conference on Learning Representations*, 2026\.URL[https://openreview\.net/forum?id=y3oHMcoItR](https://openreview.net/forum?id=y3oHMcoItR)\.
- \[28\]Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund\.Sim\-to\-real transfer in deep reinforcement learning for robotics: a survey\.In*2020 IEEE Symposium Series on Computational Intelligence \(SSCI\)*, pages 737–744, 2020\.doi:10\.1109/SSCI47803\.2020\.9308468\.
- \[29\]Payel Mukhopadhyay, Stefan S Nixon, Romain Watteaux, Michael McCabe, Alberto Bietti, Kyunghyun Cho, Cristiana Diaconu, Irina Espejo Morales, David Fouhey, Siavash Golkar, et al\.Emergent transfer of a physics foundation model from simulation to laboratory turbulence\.*arXiv preprint arXiv:2606\.01470*, 2026\.
- \[30\]C H K Williamson\.Vortex dynamics in the cylinder wake\.*Annual Review of Fluid Mechanics*, 28\(Volume 28, 1996\):477–539, 1996\.ISSN 1545\-4479\.doi:https://doi\.org/10\.1146/annurev\.fl\.28\.010196\.002401\.URL[https://www\.annualreviews\.org/content/journals/10\.1146/annurev\.fl\.28\.010196\.002401](https://www.annualreviews.org/content/journals/10.1146/annurev.fl.28.010196.002401)\.
- \[31\]Anatol Roshko\.On the development of turbulent wakes from vortex streets\.NACA Report 1191, National Advisory Committee for Aeronautics, Washington, D\.C\., 1954\.
- \[32\]Gabriel D\. Weymouth\.Lily pad: Towards real\-time interactive computational fluid dynamics\.*arXiv preprint arXiv:1510\.06886*, 2015\.URL[https://arxiv\.org/abs/1510\.06886](https://arxiv.org/abs/1510.06886)\.
- \[33\]David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne\.Experience replay for continual learning\.In H\. Wallach, H\. Larochelle, A\. Beygelzimer, F\. d'Alché\-Buc, E\. Fox, and R\. Garnett, editors,*Advances in Neural Information Processing Systems*, volume 32\. Curran Associates, Inc\., 2019\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2019/file/fa7cdfad1a5aaf8370ebeda47a1ff1c3\-Paper\.pdf](https://proceedings.neurips.cc/paper_files/paper/2019/file/fa7cdfad1a5aaf8370ebeda47a1ff1c3-Paper.pdf)\.
- \[34\]Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton\.Similarity of neural network representations revisited\.In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors,*Proceedings of the 36th International Conference on Machine Learning*, volume 97 of*Proceedings of Machine Learning Research*, pages 3519–3529\. PMLR, 09–15 Jun 2019\.URL[https://proceedings\.mlr\.press/v97/kornblith19a\.html](https://proceedings.mlr.press/v97/kornblith19a.html)\.
- \[35\]Changjian Zhou, Junfeng Fang, Negin Yousefpour, Peng Wu, Bin Yan, and Guillermo A Narsilio\.Physguard: Fisher\-guided gradient projection for sim\-to\-real neural pde surrogates, 2026\.URL[https://arxiv\.org/abs/2606\.16602](https://arxiv.org/abs/2606.16602)\.
- \[36\]Yizheng Wang, Mohammad Sadegh Eshaghi, Xiaoying Zhuang, Timon Rabczuk, and Yinghua Liu\.Replay\-based continual learning for physics\-informed neural operators, 2026\.URL[https://arxiv\.org/abs/2605\.04832](https://arxiv.org/abs/2605.04832)\.
- \[37\]Tapas Tripura and Souvik Chakraborty\.A foundational neural operator that continuously learns without forgetting, 2023\.URL[https://arxiv\.org/abs/2310\.18885](https://arxiv.org/abs/2310.18885)\.
- \[38\]Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen\.LoRA: Low\-rank adaptation of large language models\.In*International Conference on Learning Representations*, 2022\.URL[https://openreview\.net/forum?id=nZeVKeeFYf9](https://openreview.net/forum?id=nZeVKeeFYf9)\.
- \[39\]Mahindra Singh Rautela, Siddharth Mansingh, Aleksandra Pachalieva, Alexander Most, Kyle S\. Hickmann, Daniel O’Malley, Alexander Scheinker, Bradley C\. Love, Diane Oyen, Nathan A\. DeBardeleben, Earl Lawrence, and Ayan Biswas\.Out\-of\-distribution transfer of PDE foundation models to material dynamics under extreme loading\.In*AI&PDE: ICLR 2026 Workshop on AI and Partial Differential Equations*, 2026b\.URL[https://openreview\.net/forum?id=yRC7DqRYsg](https://openreview.net/forum?id=yRC7DqRYsg)\.

## Appendix ALimitations

We characterize forgetting in a single scientific foundation model \(Walrus, 1\.3B\) adapted to a single physical scenario \(cylinder flow\), with one seed per configuration, so degradation values carry no error bars and we cannot say how far the specific findings transfer\. Our predictive model is fitted on ten datasets with two predictors, and the block at which similarity is measured was selected from a sweep over those same datasets; validating it properly would require a fine\-tuning target other than cylinder flow, and the same two factors already explain considerably less variance in Phase II than in Phase I\. Forgetting is quantified through pointwise error metrics, which are insensitive to failure modes that matter physically \(e\.g\., a model may retain low pointwise error while losing small\-scale spectral fidelity, violating conservation laws, or mispredicting turbulence statistics\), so physics\-relevant diagnostics would likely reveal degradation ours does not register\. Finally, our replay experiments interleave cylinder simulation rather than Well data, addressing retention of the Phase I task rather than of the full pretraining corpus, which is left to future work\.

## Appendix BFurther Related Work

In one related study on Walrus,[Mukhopadhyay et al\. \[29\]](https://arxiv.org/html/2610.40857#bib.bib29)fine\-tune the model on a handful of direct numerical simulations of the Rayleigh–Taylor instability and apply it zero\-shot to sliding\-barrier laboratory measurements, finding that the model leaves the DNS\-like regime and enters the experimentally observed mixing\-growth band without having seen an experimental sample\. Their work asks whether simulation\-trained capability transfers to the laboratory, and we ask additionally what adapting to the laboratory costs in terms of forgetting and characterize the effects on pretraining capability\.

Closest in experimental setting,[Zhou et al\. \[35\]](https://arxiv.org/html/2610.40857#bib.bib35)study sim\-to\-real adaptation on RealPDEBench, including real cylinder flow, and propose Fisher\-guided gradient projection to protect physics\-critical parameters\. Their operators are trained from scratch on the matched numerical data \(FNO50\.450\.4M, CNO8\.08\.0M, DeepONet3\.53\.5M, Transolver4\.34\.3M\), so no broad pretraining corpus exists whose loss could be measured\. Degradation is instead assessed as low\-frequency spectral preservation on the target system itself\.

Continual learning in scientific machine learning has otherwise been studied on small operators over sequences of simulated tasks\.[Wang et al\. \[36\]](https://arxiv.org/html/2610.40857#bib.bib36)combine replay, distillation and LoRA on a0\.370\.37M\-parameter Transolver across a ten\-task Darcy sequence, and[Tripura and Chakraborty \[37\]](https://arxiv.org/html/2610.40857#bib.bib37)freeze a pretrained ensemble of operator blocks and learn only a small task\-specific gate per PDE, storing one gate per task, an adapter\-style scheme in the manner of LoRA\[[38](https://arxiv.org/html/2610.40857#bib.bib38)\]\. Shared weights are never updated after pretraining, so forgetting is avoided by construction rather than mitigated; this is unavailable when the goal is a single model that has genuinely adapted to the target domain, as here\. Work on large pretrained PDE foundation models has focused on forward transfer:[Rautela et al\. \[39\]](https://arxiv.org/html/2610.40857#bib.bib39)fine\-tune Poseidon and MORPH on OOD material dynamics simulations without evaluating retention\.

To our knowledge, no prior work fine\-tunes a billion\-parameter PDE foundation model on real experimental measurements and then measures what is lost across its pretraining corpus, which is the question this paper addresses\.

## Appendix CExperimental Setup

Table 2:Phase I simulation pretraining improves real\-PIV rollout at every stage of Phase II training\.Paired ablation: both arms start from Walrus\-1\.3B and are trained on real PIV with identical settings \(lr=5×10−5\\mathrm\{lr\}=5\{\\times\}10^\{\-5\}, 200 steps/epoch\); they differ only in whether the cylinder\-simulation Phase I stage was run first\. Epoch 0 is the zero\-shot rollout of each starting checkpoint\. Metrics are 64\-step rollout means on the held\-out PIV split\. Bold marks the better arm\.VRMSE↓\\downarrowNRMSE↓\\downarrowPearsonrr↑\\uparrowPIV epochsnoPhase I\+ Phase IgainnoPhase I\+ Phase IgainnoPhase I\+ Phase I03\.9020\.7655\.10×\\times1\.8170\.4474\.06×\\times0\.5480\.653200\.4840\.3711\.31×\\times0\.3120\.2381\.31×\\times0\.8620\.898400\.4710\.3581\.31×\\times0\.3170\.2291\.38×\\times0\.8560\.903600\.5170\.3381\.53×\\times0\.3460\.2111\.64×\\times0\.8340\.916800\.4410\.3441\.28×\\times0\.2800\.2101\.34×\\times0\.8690\.9141000\.3500\.3181\.10×\\times0\.2190\.1981\.11×\\times0\.9130\.924Table 3:The benefit of Phase I is concentrated at long rollout horizons\.Rollout VRMSE \(↓\\downarrow\) on real PIV, decomposed into the time windows logged during validation, for the same paired arms as Table[2](https://arxiv.org/html/2610.40857#A3.T2)\. Once Phase II training has run, the two arms are indistinguishable over the first few steps; the gap that remains is almost entirely in theT=8:22T\{=\}8\{:\}22andT=22:64T\{=\}22\{:\}64windows, indicating that Phase I buys long\-horizon stability rather than one\-step accuracy\. Epoch 0 is the zero\-shot rollout of each starting checkpoint\.PIV epochsArmT=0:1T\{=\}0\{:\}1T=1:2T\{=\}1\{:\}2T=2:8T\{=\}2\{:\}8T=8:22T\{=\}8\{:\}22T=22:64T\{=\}22\{:\}640no Phase I0\.0760\.1430\.5762\.7164\.910\+ Phase I0\.0540\.0890\.2070\.4780\.96420no Phase I0\.0470\.0690\.1340\.2780\.614\+ Phase I0\.0470\.0680\.1330\.2540\.45340no Phase I0\.0470\.0680\.1340\.2800\.595\+ Phase I0\.0470\.0680\.1310\.2510\.43660no Phase I0\.0460\.0680\.1340\.2860\.661\+ Phase I0\.0470\.0680\.1310\.2500\.40780no Phase I0\.0460\.0680\.1300\.2680\.553\+ Phase I0\.0460\.0680\.1330\.2520\.415100no Phase I0\.0440\.0650\.1260\.2420\.428\+ Phase I0\.0440\.0650\.1250\.2320\.383### C\.1Cylinder Data

We prepared the data for our experiments by converting the datasets to HDF5 in the Well format, and by downsampling the simulated data \(originally 128 x 256\) to match the resolution of our real PIV data \(64 x 128\), so that the sim\-to\-real gap could be studied independently of resolution differences\. We also limited our training and evaluation to the two shared fields between the datasets \(x\- and y\-velocity\)\. The 92 trajectories were divided into train/valid/test splits at an approximate ratio of 80/10/10, with each split containing a range of Reynolds numbers across the dataset\.

### C\.2Phase I: Simulation Finetuning

We used AdamW \(weight decay10−410^\{\-4\}\) with three learning rates \(2×10−52\\times 10^\{\-5\},1×10−41\\times 10^\{\-4\}, and5×10−55\\times 10^\{\-5\}\), crossed with both normalization strategies, for six finetuning runs total, each initialized from the same pretrained checkpoint with an inverse\-square\-root learning rate schedule \(20\-epoch linear warmup, 10\-epoch cooldown\)\. Runs were trained for up to 51 epochs, with checkpoints saved every 20 epochs and one\-step/rollout validation every 5 epochs and every epoch consisting of 200 batched optimizer steps\. Based on Phase I results, the2×10−52\\times 10^\{\-5\}condition was not promising and was dropped from Phase II\.

### C\.3Phase II: PIV Data Finetuning

We keep the same architecture, optimizer, and per\-epoch validation cadence as Phase I \(AdamW, batch size 1, delta prediction, one\-step and rollout validation every 5 epochs, 60\-step rollouts\), but training for up to 100 epochs at lr=5×10−55\\times 10^\{\-5\}with a shortened warmup/cooldown \(3/5 epochs, vs\. 20/10 in Phase I\) appropriate to continuing from an already\-adapted checkpoint rather than initializing from scratch\. We compare three initialization/normalization conditions: continuing from the Phase I global\-norm checkpoint under global norm, continuing from the Phase I samplewise checkpoint but switching to global norm, and continuing from the Phase I samplewise checkpoint and remaining samplewise throughout\.

#### Persistence baseline\.

Persistence is the error of the trivial predictory⁡\(t\+1\)=y⁡\(t\)y\(t\+1\)=y\(t\): each frame is predicted by copying its predecessor\. We score it with NRMSE computed per trajectory, timestep and channel over the spatial axes and then averaged, using the same channel set the model is scored on\. Frames are consecutive—the configuration setsmin\_dt\_stride==max\_dt\_stride=1=1for every dataset, so consecutive stored frames are what the model steps between\. We average over up to 32 validation trajectories per dataset \(as few as nine where the validation split is smaller\) and up to 128 frames per trajectory, apply the same field transforms used in training \(base\-10 logarithm of density and temperature for turbulence gravity cooling\), and reduce the two three\-dimensional datasets to their centralzzslice\. The baseline uses no model and no training, and is computed directly from the raw data\. As a sanity check, the pretrained model outperforms it on every dataset, by between2\.5×2\.5\\times\(turbulence gravity cooling\) and126×126\\times\(Helmholtz\)\.

#### Representational similarity\.

We record activations by instrumenting the forward pass of the evaluation above, so that data loading, normalization and checkpoint handling are identical\. For each dataset we accumulate the first and second moments of the 1408\-dimensional hidden state at the input of transformer blocks\{0,5,10,15,20,25,30,39\}\\\{0,5,10,15,20,25,30,39\\\}, over 64 batches with up to 4096 randomly subsampled space\-time tokens each \(262,144262\{,\}144tokens per dataset\), and form the centered covariance\. Recording the first moment matters, since at block 0 the mean direction alone carries roughly63%63\\%of the total energy, and an uncentered second moment is dominated by it\. Similarity between two datasets is the normalized Frobenius inner product of their covariances,⟨CA,CB⟩F/\(∥CA∥F​∥CB∥F\)\\langle C\_\{A\},C\_\{B\}\\rangle\_\{F\}/\(\\lVert C\_\{A\}\\rVert\_\{F\}\\lVert C\_\{B\}\\rVert\_\{F\}\), which is confined to\[0,1\]\[0,1\]because covariances are positive semi\-definite\. All covariances are computed with pretrained weights, before any fine\-tuning, and the reference is the cylinder simulation\. We report block 39; by that depth every dataset occupies the same 1408\-dimensional space irrespective of its field count \(two to eleven\), resolution, or spatial dimensionality, which is what makes the comparison well defined\.

The encoders themselves do differ by dimensionality\. Since the cylinder data is two\-dimensional, finetuning leaves the 3D encoder and decoder of MHD\-64 and turbulence gravity cooling exactly unchanged, which may potentially contribute to MHD\-64’s weak forgetting\. The effect appears limited, however, since turbulence gravity cooling degrades substantially under the same untouched encoder, and the encoders hold only6\.8%6\.8\\%of parameters against86\.4%86\.4\\%in the shared processor, suggesting that forgetting propagates mainly through shared weights, as it must also for Helmholtz, which has no velocity channel yet shows the largest relative degradation\.

#### Statistical analysis\.

Degradation and both predictors span more than an order of magnitude, and the proposed mechanism is multiplicative, so we fit ordinary least squares oflog10⁡Δ​VRMSE\\log\_\{10\}\\Delta\\mathrm\{VRMSE\}onlog10\\log\_\{10\}of the two predictors\. Withn=10n=10datasets and two predictors we report adjustedR2R^\{2\}throughout, together with the variance inflation factor and the coefficienttt\-tests, which establish that each predictor contributes after controlling for the other\. The fitted model isΔVRMSE∝CKA0\.60Pers\.0\.61\\Delta\\mathrm\{VRMSE\}\\propto\\mathrm\{CKA\}^\{0\.60\}\\,\\mathrm\{Pers\.\}^\{0\.61\}\(R2=86\.3%R^\{2\}=86\.3\\%, adjusted82\.4%82\.4\\%;F2,7=22\.1F\_\{2,7\}=22\.1,p<0\.001p<0\.001\), with no dataset mispredicted by more than2\.3×2\.3\\timesacross a37×37\\timesrange\.

#### Compute\.

All experiments ran on an internal Slurm cluster using NVIDIA L40S and A100 GPUs\. Fine\-tuning used four GPUs per job under a 12\-hour wall\-clock limit, with longer runs chained across successive jobs; evaluation and analysis jobs were single\-GPU\. The six\-condition Phase I sweep consumed approximately 290 GPU\-hours and the three Phase II conditions approximately 600\. The replay sweep dominates the budget at roughly 3,300 GPU\-hours across 150 training jobs, since each replay fraction requires a full Phase II run\. Evaluation is comparatively cheap: held\-out forgetting evaluations averaged 31 minutes each \(224 evaluations, 97 GPU\-hours\), replay evaluations a further 88 GPU\-hours, and activation capture 22 minutes each \(55 captures, 16 GPU\-hours\)\. The persistence baseline is CPU\-only and completes in under 10 minutes for all ten datasets\. In total the experiments reported here consumed approximately 4,400 GPU\-hours; the broader project, including preliminary runs, ablations and experiments held for future work, consumed several hundred GPU\-hours more\.

## Appendix DAdditional Forgetting Detail

Table 4:Well benchmark mean per\-field error at each Phase I forgetting\-dynamics checkpoint, under both normalizations\. Metric: mean of per\-field\{N,V\}RMSE\_T=all\_meanfrom the one\-step validation pass\. NRMSE divides byRMS⁡\(y\)\\mathrm\{RMS\}\(y\), VRMSE byσ⁡\(y\)\\sigma\(y\); the two coincide for zero\-mean fields and diverge in proportion to a channel’s uniform background, so VRMSE is the comparable metric across physically different channels andVRMSE=1\\mathrm\{VRMSE\}=1means no better than predicting the spatial mean\. Bold = lowest \(best\) per row\.*Pretrained*: Walrus\-1\.3B before any cylinder finetuning\.*Stepkk*: afterkkepochs of Phase I cylinder finetuning \(200 gradient steps/epoch\)\.NRMSEVRMSEDatasetpretrainedstep 5step 10step 15step 20pretrainedstep 5step 10step 15step 20acoustic\_disc\.0\.00660\.09560\.16430\.20160\.20080\.00720\.10600\.18100\.22290\.2213acoustic\_incl\.0\.01660\.09740\.16790\.19610\.19400\.01780\.10580\.18150\.21210\.2101acoustic\_maze0\.03940\.17930\.30190\.38950\.40470\.03980\.18180\.30610\.39500\.4102active\_matter0\.01060\.02460\.03280\.03910\.04130\.01450\.03880\.04930\.05710\.0599helmholtz0\.00110\.00520\.01650\.03040\.03740\.00130\.00740\.02150\.03830\.0478MHD\_640\.12340\.12580\.12850\.13070\.13180\.14930\.15220\.15540\.15800\.1594planetswe0\.00070\.01250\.01780\.01990\.02000\.00070\.01260\.01790\.02000\.0201shear\_flow0\.00770\.01220\.01490\.01680\.01700\.00790\.01290\.01580\.01790\.0181TRL\_2D0\.03800\.06050\.07770\.09480\.09580\.11480\.16910\.21790\.26430\.2674turbcool0\.03920\.04640\.06160\.07020\.07150\.08420\.10290\.13830\.15720\.1599![Refer to caption](https://arxiv.org/html/2610.40857v1/fig_appendix_well_acoustic_scattering_discontinuous_pretrained.png)

![Refer to caption](https://arxiv.org/html/2610.40857v1/fig_appendix_well_acoustic_scattering_discontinuous.png)

Figure 5:Acoustic Scattering \(Discontinuous\)\.GT, prediction, and signed error \(shared colorscale per field\) at rollout stepst=1,20,40,60t=1,20,40,60\.*Top*: pretrained Walrus\-1\.3B \(before Phase I finetuning\)\.*Bottom*: after Phase I cylinder finetuning \(step 20\)\.![Refer to caption](https://arxiv.org/html/2610.40857v1/fig_appendix_well_acoustic_scattering_inclusions_pretrained.png)

![Refer to caption](https://arxiv.org/html/2610.40857v1/fig_appendix_well_acoustic_scattering_inclusions.png)

Figure 6:Acoustic Scattering \(Inclusions\)\.*Top*: pretrained\.*Bottom*: Phase I step 20\.![Refer to caption](https://arxiv.org/html/2610.40857v1/fig_appendix_well_acoustic_scattering_maze_pretrained.png)

![Refer to caption](https://arxiv.org/html/2610.40857v1/fig_appendix_well_acoustic_scattering_maze.png)

Figure 7:Acoustic Scattering \(Maze\)\.*Top*: pretrained\.*Bottom*: Phase I step 20\.![Refer to caption](https://arxiv.org/html/2610.40857v1/fig_appendix_well_active_matter_pretrained.png)

![Refer to caption](https://arxiv.org/html/2610.40857v1/fig_appendix_well_active_matter.png)

Figure 8:Active Matter\.11 fields; time groups stacked vertically\.*Top*: pretrained\.*Bottom*: Phase I step 20\.![Refer to caption](https://arxiv.org/html/2610.40857v1/fig_appendix_well_helmholtz_staircase_pretrained.png)

![Refer to caption](https://arxiv.org/html/2610.40857v1/fig_appendix_well_helmholtz_staircase.png)

Figure 9:Helmholtz Staircase\.*Top*: pretrained\.*Bottom*: Phase I step 20\.![Refer to caption](https://arxiv.org/html/2610.40857v1/fig_appendix_well_MHD_64_pretrained.png)

![Refer to caption](https://arxiv.org/html/2610.40857v1/fig_appendix_well_MHD_64.png)

Figure 10:MHD\-64\.7 fields from a centralzz\-slice of the 3D volume; time groups stacked vertically\.*Top*: pretrained\.*Bottom*: Phase I step 20\.![Refer to caption](https://arxiv.org/html/2610.40857v1/fig_appendix_well_planetswe_pretrained.png)

![Refer to caption](https://arxiv.org/html/2610.40857v1/fig_appendix_well_planetswe.png)

Figure 11:PlanetSWE\.*Top*: pretrained\.*Bottom*: Phase I step 20\.![Refer to caption](https://arxiv.org/html/2610.40857v1/fig_appendix_well_shear_flow_pretrained.png)

![Refer to caption](https://arxiv.org/html/2610.40857v1/fig_appendix_well_shear_flow.png)

Figure 12:Shear Flow\.*Top*: pretrained\.*Bottom*: Phase I step 20\.![Refer to caption](https://arxiv.org/html/2610.40857v1/fig_appendix_well_turbulent_radiative_layer_2D_pretrained.png)

![Refer to caption](https://arxiv.org/html/2610.40857v1/fig_appendix_well_turbulent_radiative_layer_2D.png)

Figure 13:Turbulent Radiative Layer 2D\.*Top*: pretrained\.*Bottom*: Phase I step 20\.![Refer to caption](https://arxiv.org/html/2610.40857v1/fig_appendix_well_turbulence_gravity_cooling_pretrained.png)

![Refer to caption](https://arxiv.org/html/2610.40857v1/fig_appendix_well_turbulence_gravity_cooling.png)

Figure 14:Turbulence Gravity Cooling\.Density and temperature fields log10\-transformed before prediction\.*Top*: pretrained\.*Bottom*: Phase I step 20\.