When Physics Meets AI: Physical Elements in Deep Learning (Part 2)

Introduction

The first part, “When Physics Meets AI: Physical Elements in Deep Learning (Part 1),” laid out the parts of physics already embedded in AI. Energy, entropy, free energy, Hopfield networks, symmetry, renormalization group — these are established facts, set in stone. Throughout the mathematical skeleton of deep learning, tools from physics have been borrowed and used very effectively.

But beyond the established facts, AI has a set of deeper problems. Why is the Scaling Law a power law? Why do certain abilities suddenly appear at a certain scale? Why does the double descent curve contradict classical statistics? What exactly is grokking — the phenomenon of “suddenly learning to generalize after overfitting”? All AI researchers see these phenomena, but no one can explain them theoretically. They are known within the industry as AI’s open questions.

A physicist looking at these problems has a peculiar feeling: déjà vu. Over the past hundred-plus years, physics has encountered structurally similar puzzles when studying nature. Power laws correspond to critical scaling laws, sudden emergence corresponds to phase transitions, “sudden improvement after overfitting” corresponds to crystallization in supercooled liquids, and double descent, which defies classical statistical expectations, corresponds to phase transition phenomena in high-dimensional statistics. Every AI open question has a structural sibling in the physics toolbox.

How much of this “déjà vu” is real? Can the tools from the physics toolbox truly be applied to derive AI phenomena, or is it just a superficial resemblance? This is the first question this essay seeks to answer.

The second, more important, question is: How do physicists place their bets in AI? This has been going on for two decades, following a set of standard procedures tacitly agreed upon by the community. It has produced a few winning lottery tickets (diffusion models being the most famous) and many losing ones (spin glasses → loss landscapes, renormalization group → deep learning, tensor networks → classifiers). Placing successful and unsuccessful cases side-by-side reveals the true hit rate of the physics methodology in AI, and whether there’s a better way forward.

These two questions together point to a larger issue. Physics has been stagnant in its own domain for half a century, its tools being co-opted by AI without gaining new research questions in return. Could AI conversely give physics a second spring? The answer lies in physical lineages that have not yet been seriously tried: integrable systems, Lindblad open systems, KPZ surface growth, Hubbard-Heisenberg. The next winning lottery ticket might be hidden in one of them.

1. Physical Analogies in AI: Which Are True Isomorphisms, Which Just Look Similar

The previous part discussed two layers of physics in AI. The bottom layer is direct embedding of physical mathematics into AI: Boltzmann distributions, variational free energy, Langevin dynamics — these tools are borrowed as training objectives and sampling mechanisms. The middle layer uses physical intuition to shape network architectures: Hopfield’s Ising energy function, CNN’s translational symmetry, AlphaFold’s E(3) equivariance, the “layer” structure of deep networks corresponding to the layer-by-layer coarse-graining of renormalization group.

These two layers share a common feature: specific physical tools are used in AI. Either the formulas are directly borrowed, or physical intuition guides the architecture design. The reader can see a clear causal chain: physics is actively contributing.

The third layer is different. It describes another kind of relationship: “There’s a phenomenon in AI whose behavioral patterns are mathematically isomorphic to a known phenomenon in physics.” The objects of study on both sides are completely different, but the same mathematical description applies.

If this isomorphism holds, its significance is enormous. Physics already has a complete theory for dealing with that phenomenon. This theory can be directly applied to predict AI behavior and provide quantitative predictions. There’s no need to reinvent the wheel in AI.

In 1900, Max Planck faced the blackbody radiation problem. The experimental curve of blackbody radiation completely disagreed with classical theory, remaining unexplained for years, until Planck found the isomorphism with quantized harmonic oscillators. The entire quantum mechanics exploded from this moment. The isomorphism itself was the discovery; once found, the remaining work had existing mathematics.

Among AI’s open questions, is there one that is “waiting for its own Planck moment” blackbody radiation? That is the question this essay seriously pursues.

Let’s state the conclusion upfront. Among AI’s several core open questions, the strength of evidence for the third physical layer varies greatly.

Strong evidence (two): The formal correspondence between Scaling Law and critical scaling laws is almost seamless. Double descent has been rigorously derived using random matrix theory on linear models. These may genuinely be mathematical isomorphisms.

Moderate evidence (two): The loss landscape has a rigorous equivalence proof to spherical spin glasses (but requires the random data assumption). Grokking aligns well with the physical picture of supercooled liquid crystallization, but the mathematics hasn’t been established yet.

Weak but promising (third tier): Emergence looks formally like phase transitions, but work by Schaeffer et al. has made this correspondence controversial. The geometric structure of neural network representation spaces has a formal correspondence with topological matter, but it’s still early. In-Context Learning has almost no physical counterpart.

This tiered map forms the skeleton of the first half of this essay, expanded in the following five chapters.

2. The First Dark Cloud: Scaling Law and Critical Scaling Laws

The direction with the strongest evidence and most solid mathematical work in the third physical layer is the Scaling Law.

2.1 A Power Law Without Explanation

In January 2020, Jared Kaplan et al. at OpenAI uploaded “Scaling Laws for Neural Language Models” (arXiv:2001.08361). This paper systematically measured one thing: the curves of language model performance (test loss) versus parameter count N, data size D, and compute C.

The results were surprisingly clean. All three curves were power laws: Loss ∝ N^(-α), Loss ∝ D^(-β). On a log-log plot, they were straight lines. The exponent α_N was about 0.076, α_D about 0.095.

[Figure from Scaling Laws for Neural Language Models]

Kaplan measured parameter ranges from about 1,000 to 1.5 billion (6 orders of magnitude), data sizes from 22M to 23B tokens (also 6 orders of magnitude), and it held throughout. He also reported that architectural details like network width and depth had little effect on the power-law exponents. The relationship between model performance and scale is governed by a law relatively insensitive to architecture.

But no one could theoretically derive this power law. No one knows why the exponents are those specific numbers. A core empirical law guiding the training of GPT-3 and GPT-4 remains a complete theoretical black box.

In 2022, Jordan Hoffmann et al. at DeepMind re-examined Kaplan’s exponents in the Chinchilla paper, finding that Kaplan systematically underestimated the importance of data size. The corrected core conclusion: for each doubling of model size, the number of tokens should be doubled proportionally (about 20 tokens/parameter, significantly different from Kaplan’s ratio). This correction didn’t change the judgment that “the power law itself exists,” just the specific exponents. The Scaling Law as an empirical phenomenon became even more robust, but its precise form is still being refined.

[Figure from Jordan Hoffmann et al. (2022)]

2.2 Physicists Recognized This Shape

Physicists have seen this shape many times before. In physics, there’s a standard structure called critical scaling.

Near a phase transition critical point, almost all observable physical quantities obey power laws. For example, in a ferromagnet near the Curie temperature T_c, magnetization M ∝ (T_c - T)^β, and heat capacity C ∝ |T - T_c|^(-α). A key fact: these critical exponents are the same for all materials with the same symmetry. Different materials, different microscopic interactions, but identical critical exponents. Physics calls this a universality class, meaning that behavior near the critical point is determined by the deep symmetry of the system, independent of microscopic details.

AI’s Scaling Law has exactly this form. Both are power laws. The exponents are insensitive to implementation details, spanning multiple orders of magnitude. The statement in Kaplan’s paper, “changing architecture doesn’t affect the exponent,” is, to a physicist, a hallmark of universality class.

There’s another layer: behind critical scaling lies the renormalization group. The behavior of a system near a critical point is determined by the fixed point of the renormalization group. If AI’s Scaling Law truly is a critical scaling law, then the renormalization group discussed in Section 5.x of Part 1 appears once again. The same tool from the physics toolbox is used at a different level in AI.

2.3 People Have Started Working Seriously on This

The paper “The Quantization Model of Neural Scaling” by Eric Michaud, Liu, Girit, and Tegmark (arXiv:2303.13506, 2023; accepted at NeurIPS 2024) is one of the deeper attempts in this direction.

Their idea is to decompose the Scaling Law into the accumulation of “quanta” (discrete ability blocks). Each quantum is a specific skill learned from the data by the model. When quanta are learned in descending order of usage frequency, the power-law nature of the frequency distribution itself can explain the power-law decay of loss with scale. This framework simultaneously predicts the Scaling Law and emergence, because the moment a new quantum is learned corresponds to a sudden appearance of an ability.

This kind of work goes further. It aims to theoretically derive what the exponents α and β should be, then compare with experiments, rather than merely concluding that “the Scaling Law looks like critical scaling.” If this path succeeds, AI’s Scaling Law would transition from an “empirically observed regularity” to a “theoretically derived prediction from first principles.” The significance would be enormous.

2.4 Honestly: The Mechanism Level Hasn’t Been Cracked Yet

The physical origin of critical scaling is the renormalization group fixed point. To rigorously apply this theory to AI, one needs to answer several questions: What is the “fixed point” in AI? In which dimension is the “critical point”? What is the “order parameter” in the AI context? These questions currently have no clear answers.

The formal correspondence is extremely strong: universal exponents, spanning orders of magnitude, insensitivity to details — both AI and physical critical phenomena share these features. But the mechanism-level correspondence is still one step away. AI may not actually be near a critical point; it might be some deep statistical universality that coincidentally produces a similar form.

Honest assessment: This is the strongest third-layer evidence among AI’s open questions. It might be a true isomorphism, but it hasn’t been rigorously proven. This is the direction most worth long-term tracking in this essay.

3. The Second Dark Cloud: Emergence and Phase Transitions

The second open question is emergence. This is the most dramatic correspondence in the third physical layer, but also the most challenged.

3.1 A Puzzling Phenomenon

Language models exhibit a peculiar behavior: a certain ability is completely absent at small scales, but suddenly appears beyond a certain scale, as if a switch is flipped.

In June 2022, Jason Wei et al. at Google uploaded “Emergent Abilities of Large Language Models” (arXiv:2206.07682), published in TMLR the same year. The paper systematically documented a set of “emergent abilities”: arithmetic reasoning, multi-step reasoning, language analogies, TruthfulQA, etc.

[Emergent abilities of models from Wei et al. (2022)]

How bizarre are these abilities? Take three-digit addition: below 100 billion parameters, all models of various sizes have accuracy close to zero. Beyond a certain scale threshold, accuracy suddenly jumps to over 70%. The intermediate process is almost entirely invisible.

This “suddenness” is problematic in engineering. It means that the capability curves of small models cannot extrapolate to large models. Before GPT-3, no one could use GPT-2’s curves to predict what new abilities GPT-3 would have.

3.2 Physicist’s Reaction: Isn’t This a Phase Transition?

Water suddenly turns to steam at 100 degrees Celsius; a ferromagnet loses its magnetization beyond the Curie point. This “suddenness” has a strict definition in physics: it’s a discontinuity in the thermodynamic limit. Some macroscopic quantity (density, magnetization, order parameter) undergoes a discontinuous jump at a certain control parameter.

Lev Landau’s 1937 theory of phase transitions provided a mathematical skeleton for this. Define an order parameter; it jumps from zero to non-zero at the phase transition point, corresponding to the breaking of a symmetry. First-order transitions have a discontinuous jump in the order parameter itself; second-order transitions have a discontinuous derivative.

[Phase transitions of matter from wiki]

AI emergence looks formally almost identical. Treat accuracy as the order parameter and parameter scale as the control parameter. At a certain scale, accuracy jumps from near zero to a high value, exactly like a phase transition in Landau theory. That’s almost always the first reaction of physicists upon seeing Wei’s paper.

3.3 A Serious Challenge

But in April 2023, Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo at Stanford uploaded “Are Emergent Abilities of Large Language Models a Mirage?” (arXiv:2304.15004). The paper won the NeurIPS 2023 Outstanding Paper Award.

They posed a question: Is AI’s “emergence” an illusion caused by the measurement method?

Their finding: if you use a more continuous metric (e.g., token-level log-likelihood instead of binary 0-1 accuracy for entire problems), the emergence phenomenon disappears, and the curves become continuously smooth.

In other words, the discrete accuracy metric itself creates the illusion of “sudden appearance,” while the underlying continuous quantity actually grows smoothly. When a model goes from “getting 5% of steps correct” to “getting 95% of steps correct,” the binary correctness of the whole problem jumps from 0 to 1, but the underlying step accuracy grows continuously.

This discovery shook the foundation of the “AI emergence = real phase transition” analogy. If measurement artifacts explain most emergence, then the third physical layer here is just a measurement artifact, not a true isomorphism.

3.4 The Controversy Isn’t Over

Schaeffer’s challenge has been counter-challenged. Not all emergence can be dissolved by continuous metrics. Certain algorithmic capabilities (e.g., multi-digit multiplication, specific reasoning tasks) do appear to have true threshold behavior even under finer metrics. Subsequent work by Anthropic and Google DeepMind has reported that some tasks exhibit genuine phase transition features.

Michaud’s Quantization Model also offers a new perspective here. If model capabilities arise from the accumulation of discrete quanta, with each new quantum learned corresponding to a jump in ability, then it is physically possible to resemble a first-order phase transition. But the frequency distribution of quanta itself determines that the macroscopic curve is smooth, which aligns with Schaeffer’s “measurement artifact” claim. Both views can coexist within this framework.

Honest assessment: This is the most dramatic analogy in the third physical layer, but the evidence is contested. The current state is “it could be a real phase transition, or just a measurement artifact, or both,” with no final conclusion. This is the direction most in need of continued experimental observation in this essay.

4. The Third Dark Cloud: Double Descent and Random Matrices

Double descent is the deepest penetration of physical tools into AI. For other open questions, physical tools mostly remain at the “looks formally similar” level. This one has gone the next step. Physical mathematics has been directly applied to precisely derive this AI curve. The two sides can be equated.

4.1 A Curve That Flips Statistics Textbooks

Chapter one of every machine learning textbook first introduces a U-shaped curve called the bias-variance tradeoff. The story goes like this: when the model is too simple, it can’t learn (underfitting). As the model becomes more complex, error decreases. At a certain sweet spot, the model starts to memorize training data, failing on new data, so error increases again (overfitting). The overall shape is a U, with the optimum at the bottom. This law dominated statistics for decades, a near-universal common sense.

In late 2018, Mikhail Belkin and collaborators at Ohio State did something surprising. They didn’t stop at the bottom of the U; they kept making the model larger to see what happened. At a specific point (where the number of model parameters equals the number of training data points), the error, instead of continuing to rise, began to decrease again, eventually lower than the bottom of the U. This “first decrease, then increase, then decrease” curve was named double descent (paper published in PNAS 2019, arXiv:1812.11118).

[Double Descent schematic]

This directly contradicted decades of statistics textbooks. But its significance for today’s large model era is immense. GPT-4 has trillions of parameters; by classical statistics it should be hopelessly overfitted, yet it gets better with size. Double descent is a key facet of the puzzle “why large models work.” It tells everyone that the mechanism by which large models work is entirely different from what textbooks say.

4.2 Physical Tools Directly Derived It

Classical statistics couldn’t explain double descent, but physicists had a ready-made set of tools that directly derived it.

That tool is called random matrix theory. It originated in nuclear physics. In the 1950s, physicists wanted to calculate the energy level distribution of hundreds of nucleons in a heavy atomic nucleus, but it was impossible to compute individually due to complexity. The physicist Eugene Wigner changed perspective: since the precise solution was incalculable, he used a random matrix as a proxy for nucleon interactions, asking “what is the approximate distribution of eigenvalues of a random matrix?” This seemingly crude simplification gave predictions that matched experiments. Physicists polished this tool for nearly 70 years, compiling a complete set of mathematics describing the spectral properties of random matrices.

In 2019, Trevor Hastie (Stanford) together with Andrea Montanari et al. applied this nuclear physics tool to double descent (arXiv:1903.08560, formally published in Annals of Statistics 2022). The approach: reformulate the neural network training problem as a high-dimensional matrix problem, then use the physicists’ polished tools to compute the behavior of this matrix. The result was a set of quantitative predictions, including where the double descent peak occurs, how high the peak is, and how fast it decays after the peak. All could be theoretically derived and matched experiments exactly.

What’s the significance here? The physical correspondences discussed in previous chapters still remained at the analogical level: “there’s a phenomenon in AI, a similar phenomenon in physics, the forms match.” This one is different: physical mathematics directly gave an exact mathematical description of the AI phenomenon. The two sides can be equated.

Physicists see an intuitive picture here: at the critical point where model parameters equal data points, the AI system switches from one “phase” to another; the entire behavioral mechanism changes. This is the same class of phenomenon as water turning to steam at 100°C (called a phase transition). The peak of double descent is the phase transition point.

4.3 Still Pushing Towards True Deep Networks

Currently, rigorous random matrix analysis has been mainly completed for simpler objects like linear models and kernel methods. True deep neural networks (tens of layers, nonlinear activations, billions of parameters) still lack a complete rigorous derivation, but the direction is set, working step by step towards that goal.

Representative works: Song Mei and Andrea Montanari (2019, arXiv:1908.05355) pushed the analysis to a simplified version of two-layer neural networks. Jeffrey Pennington, starting from ICML 2017, used free probability (an advanced branch of random matrix theory) to study key properties of deep networks. Each step pushes physical tools closer to the true complexity of neural networks.

[Figure from Song Mei & Andrea Montanari (2019)]

A natural question: can we compute the double descent curve for real-scale networks like GPT-4? The mathematical complexity jumps several orders from linear models to true deep networks. Whether and when it’s possible are open questions.

Honest assessment: This is the deepest penetration of physical tools into AI. Linear models have been rigorously completed; deep networks are the next step. Among all open questions in this essay, this is the closest to the ideal state of “physics directly applied.”

5. Grokking and Metastability

Grokking is the most mysterious of the open questions. Physicists have a very fitting image (crystallization of supercooled liquids), but strict mathematics hasn’t been worked out yet.

5.1 A Phenomenon Like “Sudden Enlightenment”

Imagine a student memorizing all homework answers in the first week, getting perfect scores on homework, but failing immediately on a new problem. Keep making them do problems for tens of times longer, and one day they suddenly “get it,” switching from “memorizing answers” to “understanding underlying principles,” and can solve new problems. This phenomenon of “sudden understanding after long mechanical memorization” is also observed in machine learning, called grokking (the term from Heinlein’s sci-fi, meaning “to understand thoroughly”).

In early 2022, Alethea Power and collaborators at OpenAI discovered this phenomenon in a class of small algorithmic tasks (e.g., modular arithmetic addition) (arXiv:2201.02177). Their Transformer reduced training error to near zero within a few hundred steps — clearly overfitting. At that point, test accuracy was near random. According to classical learning theory, that’s a dead end: memorization is complete, further training would only worsen overfitting. But they didn’t stop; they continued training for tens or even hundreds of times longer. Then, at some moment, test accuracy suddenly climbed rapidly, eventually reaching near-perfect generalization.

This event overturned a piece of common sense. In all of machine learning, there’s a rule called “early stopping”: stop when training error decreases and test error starts rising. Grokking shows that this rule is wrong in some cases. Continuing training might lead to “enlightenment.” The mechanism behind this is still an open question today.

5.2 Physical Picture: Crystallization of Supercooled Liquids

The closest physical picture is metastability and crystallization. Put bottled water in a fridge and slowly cool it below zero degrees Celsius; the water remains liquid, not freezing. This is a metastable state: its energy is higher than that of ice, but it is blocked by an energy barrier, unable to jump directly. Given enough time or a small perturbation (like a tap), it will suddenly crystallize at some moment, crossing the barrier into the stable state.

The neural network in the overfitting state can be seen as a metastable state. It has found a “memorization solution” that perfectly fits the training set, but hasn’t truly learned the underlying law. Continuing training is like waiting for it to “crystallize” into an “algorithmic solution” with lower energy. This algorithmic solution is blocked by some barrier, requiring time or random perturbations (noise in gradient descent) to cross.

This picture was later confirmed by experiments on neural network interpretability. In early 2023, Neel Nanda et al. (arXiv:2301.05217, ICLR 2023) “opened up” the Transformer after grokking to see what it had learned. They found that the network had indeed implemented a specific algorithm internally: using discrete Fourier transforms plus trigonometric identities for modular addition. Grokking is a genuine mode-switching process, from “rote memorization” to “derived algorithm,” not a coincidence shaken out of noise. “Memory” is the metastable state, “algorithm” is the stable state, fitting the supercooled liquid crystallization picture very well.

5.3 Physical Picture Clear, Mathematics Not Yet Done

The physical picture is clear, but strict mathematics hasn’t been established. To rigorously apply supercooled liquid phase transition theory to grokking, one needs to map the loss landscape of the neural network to a physical system with a clear barrier structure. No one has done this yet. Nanda’s paper is an experimental observation at the mechanism level, not a derivation at the level of physical mathematics.

Honest assessment: This is the open question with the tightest physical correspondence, most worth a serious effort by physicists. The physical picture exists, the mathematical tools for metastability and crystallization are mature in statistical physics; the engineering gap is a concrete bridge to neural networks. Whoever does this first might create the next “physics → AI” success story like diffusion models.

6. Other Open Questions Worth Noting

The previous four chapters covered four open questions with relatively concentrated evidence. The remaining few have varying degrees of physical correspondence, not enough to support a full chapter each. This chapter covers them together, providing a complete map.

6.1 Loss Landscape and Spin Glasses

Neural network training is finding a solution in a high-dimensional space. This space has countless “local optima” — points where any small change worsens performance. Common sense suggests gradient descent should fall into a poor local optimum and get stuck. Yet in practice, gradient descent consistently finds decent solutions. Why?

Physicists encountered a very similar situation in a class of disordered magnetic materials called spin glasses. In these materials, millions of tiny magnetic spins are coupled, and the energy landscape also has countless local optima. Giorgio Parisi (one of the 2021 Nobel Prize in Physics winners) studied spin glasses decades ago and made a counterintuitive prediction: the energies of the vast majority of local optima are close to the global optimum. In other words, falling into any one of them isn’t bad.

If the neural network loss landscape is truly equivalent to a spin glass, this prediction would explain why gradient descent works. In 2015, Anna Choromanska et al. (collaborating with Yann LeCun) published a paper at AISTATS (arXiv:1412.0233) that rigorously proved under the assumption of random data, the loss landscape of deep networks is mathematically equivalent to a spherical spin glass model.

However, this equivalence has a core assumption: data is random. Real data has structure, very different from random data. Subsequent experimental observations found that the real loss landscape is much friendlier than a spherical spin glass, with large flat minima regions and low-loss connecting paths, diverging significantly from the “disordered spin glass” rugged image. Choromanska’s framework was half right, its prediction half correct, but engineering impact limited.

6.2 Representation Space Geometry and Topological Matter

Inside a trained neural network, words and concepts are represented as high-dimensional vectors. These vectors have peculiar geometric relationships. The most famous example: king - man + woman ≈ queen: vector arithmetic captures semantic relationships.

In physics, there is a similar phenomenon called topological protection. In condensed matter, certain properties (like current direction) of materials are robust to small parameter perturbations; they don’t disappear due to detail changes, protected by the global structure of “topology.” The geometric structures in neural network representation spaces that are “stable across models” (different models trained independently still exhibit king - man + woman ≈ queen) have a formal correspondence with topological protection.

Researchers are already using topological data analysis (a mathematical tool that identifies “holes,” “loops,” and other global structures in high-dimensional data) to study neural network representation spaces. But this is a growing direction, far from being able to give quantitative predictions.

Honest assessment: Physical correspondence exists, but still early.

6.3 In-Context Learning: Weakest Physical Correspondence

You give GPT-4 five addition examples (“1+1=2, 3+5=8, 7+2=9, 4+4=8, 6+3=9”), then ask “8+5=?”. It answers correctly. Crucially, it hasn’t updated any parameters; purely by reading these five examples during the forward pass, it has “learned” what you’re asking. This phenomenon is called in-context learning (ICL).

ICL can do more than addition. Give it a few English-Chinese translation examples, and it follows. Give it a few sentiment classification examples, and it follows. Give it demonstrations of “output in JSON format,” and it complies. Any task that can be exemplified in the prompt can be learned on the fly. When the GPT-3 paper (Brown et al., 2020) introduced this as a core demo, it became the foundation for all of today’s “few-shot abilities” and “instruction following” of large models.

Mechanistically, this is strange. In traditional machine learning, “learning” has a clear definition: parameters are updated step by step via gradient descent, causing a decrease in the loss function. No parameter update means no learning. ICL completely bypasses this: a few examples in the input sequence, and the model “understands” the task. This forces everyone to re-ask: “What is learning anyway?”

Existing explanatory paths are not on the physics line. Garg et al. (arXiv:2208.01066, 2022) demonstrated that Transformers can internally run a gradient descent algorithm during inference, meaning “learning” still happens, just moved from parameter space during training to intermediate activation space during inference. Catherine Olsson et al. at Anthropic (2022) discovered a circuit called induction heads in Transformers that specifically spots “patterns that appeared earlier and reproduces them,” serving as the mechanistic hardware for ICL.

These explanations all belong to the computer science level (explaining what computation the Transformer performs). A physical isomorphism (“this matches a known physical phenomenon in mathematical structure”) has not been found yet. The physical tools used in previous chapters (statistical mechanics, phase transitions, random matrices, metastability) don’t apply directly.

If someone finds a rigorous physical correspondence, it’s an open potential direction. But currently, ICL may require a new mathematical framework completely unrelated to physics, possibly from computability theory, dynamical systems, or other fields. This is the open question most out of reach of physical tools in this essay.

6.4 Chapter Summary

Putting the six or seven open questions together, ranked by the strength of physical correspondence:

Strong: Scaling Law (critical scaling), Double Descent (strict derivation via random matrix theory)

Moderate: Loss landscape (spin glass, but requires random data assumption), Grokking (clear metastability picture, mathematics not rigorous)

Weak but promising: Emergence (phase transition picture, challenged by Schaeffer), Representation geometry (topological matter correspondence, still early)

Almost blank: In-Context Learning

There’s a notable pattern in this map: the directions with the strongest evidence overlap with the directions with the most mature mathematical tools. Scaling Law corresponds to critical scaling and renormalization group, core achievements of condensed matter physics polished over decades. Double descent corresponds to random matrix theory, a mature tool of statistical physics. Grokking’s metastability theory is an off-the-shelf tool of non-equilibrium statistical mechanics.

This makes sense: the isomorphisms physicists see in AI are limited by the tools they have. “Those with tools can see correspondences” is both the power and the blind spot of physics methodology. Phenomena like In-Context Learning, without existing physical tools, may require a completely new mathematical framework unrelated to physics.

7. Building AI Models with Physical Tools: A “Lottery Ticket” Activity

The previous six chapters discussed the third layer of physics-AI integration: using physics to analogize AI problems. Starting from this chapter, we move to the fourth layer, the methodological level: can physical tools be directly applied to AI modeling, mapping physical models to AI models? What’s the hit rate of such methods, and who is doing it?

The physics toolbox contains hundreds of tools; each can be used to attempt building an AI model, like buying a lottery ticket. The vast majority of tickets don’t win, a few do. Chapter 7 describes the activity of buying tickets itself, Chapter 8 presents several losing tickets, and Chapter 9 fully details the winning ticket (diffusion models).

7.1 The Physics Toolbox: Raw Materials for Building AI Models

Generations of physicists have accumulated a vast mathematical toolbox. Each tool can, in principle, be used to attempt building an AI model or a training method.

Partial list:

• Statistical mechanics (Boltzmann distribution, partition function) → Boltzmann machines, energy-based models (EBM)

• Ising / Hopfield model → Hopfield networks, associative memory, Modern Hopfield

• Variational free energy → VAE, ELBO training objective

• Langevin / Fokker-Planck equation → Diffusion models, Flow Matching

• Symmetry and group theory → CNN, E(3) equivariant networks (AlphaFold core)

• Renormalization group → Hierarchical interpretation of deep networks

• Tensor networks (MPS, PEPS) → Used as classifiers and compression tools

• Spin glass → Loss landscape analysis

• Plus path integrals, chaos dynamics, critical phenomena, Hamiltonian mechanics, topological matter, and a dozen more

The toolbox has at least a hundred such tools. Each behind it has decades to centuries of accumulation by physicists. Before AI emerged, these tools already had complete mathematics and intuition in physics. Moving them to AI and seeing what happens is a series of trials.

7.2 Buying a Lottery Ticket: How to Try Building an AI Model with a Physical Tool

“Buying a lottery ticket” operationally involves common steps. No one prescribed these steps; they emerge from the group’s repeated trial and error over the past twenty years.

Step 1: Identify a mathematical shape in an AI phenomenon. Power laws, phase transitions, disordered minima, symmetry breaking — physicists have seen these shapes many times and recognize them instantly. This is the entry point for identifying “which physical tool might fit.”

Step 2: Choose the corresponding physical tool from the toolbox and build a simplified AI model. Push network width to infinity (mean-field limit), assume data is Gaussian, replace nonlinearity with simple forms. Derive a specific training objective or architecture from this toy version.

Step 3: Run this model on MNIST or CIFAR-10 to see if it works. Physicists’ standard for validation is “curve shape is right, qualitative conclusions hold,” not pursuing SOTA.

Step 4: Write a paper, title it with a physics keyword. Words like thermodynamics, renormalization, phase transition, spin glass, free energy feel familiar to physicists and make ML reviewers think “the theory is interesting.”

Step 5: Submit to NeurIPS, ICML, or physics top journals. Wait.

Each completed round is like buying a lottery ticket. Next paper, pick another physical tool, buy another ticket.

7.3 Hit Rate: Several Hundred to One or Worse

The hit rate of this lottery-ticket activity is very low.

Over the past twenty years, serious attempts to build AI models with physical tools have produced hundreds to thousands of papers. Only a few have become everyday engineering tools:

• Winners: Diffusion models (Sohl-Dickstein 2015 → DDPM → mainstream image/video generators), Hopfield networks (revived in 2020 by Modern Hopfield, proven mathematically equivalent to Transformer attention), equivariant networks (core architecture of AlphaFold)

• Losers: Spin glass loss landscape (Choromanska 2015), renormalization group RBM (Mehta-Schwab 2014), tensor network classifiers (Stoudenmire-Schwab 2016), free energy principle universal AI (Friston, two decades)

This is a typical long-tail distribution. A few winning works changed the entire industry; the vast majority ended up in “theoretically beautiful, engineeringly unused” after publication. Each paper is a lottery ticket; the probability of winning is at most a few percent.

Chapter 8 presents four losing tickets in detail; Chapter 9 fully describes the winning ticket (diffusion models). When reading these two chapters, remember: each case represents a specific attempt to “use a physical tool to build an AI model,” and all attempts used the above five-step process.

7.4 Who Buys Lottery Tickets

The players of this lottery ticket activity come in two groups.

The first group is the academic school, concentrated in a few core labs, with a named research sub-community called statistical physics of machine learning, totaling a few dozen core PIs and a few hundred PhDs/postdocs:

• EPFL: Lenka Zdeborová, Florent Krzakala, Matthieu Wyart

• Stanford: Surya Ganguli, Andrea Montanari

• MIT: Max Tegmark’s group (including Eric Michaud, Ziming Liu), and the IAIFI institute

• Boston University: Pankaj Mehta

• Harvard: Matthew Schwartz

The academic school’s output mainly stays at the toy model level and theoretical papers, with low hit rate but fast output.

The second group is physics-trained researchers aiming to build usable models, spanning academia and industry, without a unified community name, but contributing most of the truly winning works:

• John Hopfield: PhD in condensed matter physics, early years at AT&T Bell Labs industrial research, moved to Caltech in the 1980s, later back to Princeton. The key 1982 paper was from his Caltech period, mapping the Ising model into neural networks to create Hopfield networks. 2024 Nobel Prize in Physics.

• Geoffrey Hinton: Undergraduate in theoretical physics, PhD in cognitive science, long at University of Toronto, ten years at Google Brain. Advanced Hopfield to Boltzmann machines and RBMs, initiating the deep learning renaissance in the 2000s. The other half of the 2024 Nobel Prize in Physics.

• Jascha Sohl-Dickstein: PhD in biophysics from UC Berkeley, former NASA Mars rover researcher, later Google Brain,