几年来我一直在思考这个问题。统计力学和AI之间的关系。统计力学用分子动力学的统计方式，复现了热力学优美的基本定理，尤其是复现了一些热力学里极为漂亮的宏观量之间的关系，比如熵、自由能，当然还有温度、压强等。

问题是，AI是否有这些热力学的宏观量？如果AI无法用宏观量表示，那么本质上无法用统计力学的方法进行研究。几年前我做了一些思考，在机器学习系统中，重新提出温度的概念，把训练过程看成热机的状态变化过程。

https://arxiv.org/abs/2404.13218

---

On the Temperature of Machine Learning Systems

Source: https://arxiv.org/html/2404.13218

Abstract

We develop a thermodynamic theory for machine learning (ML) systems. Similar to physical thermodynamic systems which are characterized by energy and entropy, ML systems possess these characteristics as well. This comparison inspire us to integrate the concept of temperature into ML systems grounded in the fundamental principles of thermodynamics, and establish a basic thermodynamic framework for machine learning systems with non-Boltzmann distributions. We introduce the concept of states within a ML system, identify two typical types of state, and interpret model training and refresh as a process of state phase transition. We consider that the initial potential energy of a ML system is described by the model’s loss functions, and the energy adheres to the principle of minimum potential energy. For a variety of energy forms and parameter initialization methods, we derive the temperature of systems during the phase transition both analytically and asymptotically, highlighting temperature as a vital indicator of system data distribution and ML training complexity. Moreover, we perceive deep neural networks as complex heat engines with both global temperature and local temperatures in each layer. The concept of work efficiency is introduced within neural networks, which mainly depends on the neural activation functions. We then classify neural networks based on their work efficiency, and describe neural networks as two types of heat engines.

Keywords: machine learning system, thermodynamics, temperature, entropy, energy, phase transition, heat engine, work efficiency

1Introduction

From the perspective of information theory, data carries entropy. The concept of entropy originated from thermodynamics and statistical mechanics, where it describes the disorder or randomness in a physical system. Claude Shannon later extended the idea of entropy to information theory to measure the uncertainty of random variables[1], while Norbert Wiener also discussed entropy in the context of cybernetics, especially differential entropy[2]. The employment of entropy in machine learning is an adaptation from information theory. For example, cross-entropy and information gain are used for splitting nodes in decision trees and random forests. In unsupervised learning, entropy can be used to evaluate the quality of clusters. Overall, the usage of entropy in data systems and machine learning is fundamentally rooted in the principles of thermodynamics and information theory, demonstrating a diverse and interdisciplinary application of the concept.

On the other hand, a physical system has energy, as well as entropy. If the concept of entropy can be introduced into a data system, does data also have energy? In the field of machine learning, there is a category of models known as energy-based models (EBMs)[3,4]. The origins of these EBMs can be traced back to the Ising model in statistical physics[5,6]and the Amari-Hopfield network[7,8]. The Boltzmann Machines (BMs) were proposed as stochastic recurrent neural networks[9], inspired by the Ising model as well as spin-glass model in physics[10]. To simplify the training process and improve computational efficiency, the Restricted Boltzmann Machines (RBMs) were later developed[11,12]. The RBMs introduced a restriction that the neurons must form a bipartite graph, which significantly improved the training efficiency. Since the advent of RBMs, a variety of methods and applications have been proposed under the umbrella of EBMs, contributing to the evolution and expansion of this field[13,14,15,16,17,18,19]. The fundamental concept of an EBM is to define an energy function that satisfiesEμ⁢(x)=−log⁡pμ⁢(x)subscript𝐸𝜇𝑥subscript𝑝𝜇𝑥E_{\mu}(x)=-\log p_{\mu}(x)italic_E start_POSTSUBSCRIPT italic_μ end_POSTSUBSCRIPT ( italic_x ) = - roman_log italic_p start_POSTSUBSCRIPT italic_μ end_POSTSUBSCRIPT ( italic_x ), orpμ⁢(x)=exp⁡[−Eθ⁢(x)/Zθ]subscript𝑝𝜇𝑥subscript𝐸𝜃𝑥subscript𝑍𝜃p_{\mu}(x)=\exp[-E_{\theta}(x)/Z_{\theta}]italic_p start_POSTSUBSCRIPT italic_μ end_POSTSUBSCRIPT ( italic_x ) = roman_exp [ - italic_E start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_x ) / italic_Z start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ], whereEμ⁢(x)subscript𝐸𝜇𝑥E_{\mu}(x)italic_E start_POSTSUBSCRIPT italic_μ end_POSTSUBSCRIPT ( italic_x )is the energy function with parameter setμ𝜇\muitalic_μ, andZθsubscript𝑍𝜃Z_{\theta}italic_Z start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPTis the partition function as the normalizing constant. This relationship between probability and energy aligns with the principles of statistical physics, and we can optimize either the loss function or the energy function to train the models.

Of course, the so-called “energy of data” is not physical energy in the real world. Instead, it is an analogy drawn between data systems and the real physical world, serving as an indicator to describe the properties of data and machine learning systems associated with the data. However, just like the introduction of entropy in information theory, we can consider the energy in data systems as a kind of generalized energy. Building on this concept, it leads us to consider:if a machine learning (ML) system itself has energy, and incorporates the concept of entropy, the machine learning (ML) system can essentially be analogized to a thermodynamic system.This raises an important question:could we define temperature-like quantities to characterize the properties of a ML system?The concept of temperature already exists in machine learning as a scaling parameter used to control the randomness of predictions made by models. For example, in BMs and RBMs, the temperature parameter appears in the Boltzmann distribution, with higher temperatures leading to more uniform distributions over states[20,21,22]. Similarly, the temperature parameter in softmax classifiers for multi-class classification is applied to the logits before the softmax function, with higher temperatures giving more similar probabilities to all classes[23,24,25]. Temperature can be used to control the creativity of a generative model, where higher temperatures will make more novel and unexpected predictions more likely[26,27]. However, the existing concept of temperature in machine learning is merely a single model parameter, which cannot be derived from first principles, nor can it reflect the overall thermodynamic properties of the ML system.Overall, although the concepts of energy, entropy, and temperature exist in the field of machine learning, no one has yet unified the three concepts together, nor viewed ML systems as thermodynamic systems from first principles.

This paper systematically proposes a theory of thermodynamics and statistical mechanics in machine learning, with a particular focus on discussing the concept of temperature for ML systems. We can gain inspiration from the thermodynamic potentials in the real physical world. The thermodynamic potentials are fundamental concepts that describes the energy characteristics of a thermodynamic system. They are scalar quantities that provide information about the system state, and are used to understand how the system will respond to changes in temperature, pressure, and volume. There are several thermodynamic potentials, including internal energy (U𝑈Uitalic_U), Helmholtz free energy (F𝐹Fitalic_F), enthalpy (H𝐻Hitalic_H) and Gibbs free energy (G𝐺Gitalic_G). Correspondingly, we have a set of four equations known as the fundamental thermodynamic relations to describe these thermodynamic potentials, which are essential in understanding the behavior of thermodynamic systems[28]

d⁢U𝑑𝑈\displaystyle dUitalic_d italic_U=\displaystyle==T⁢d⁢S−P⁢d⁢V+∑iμi⁢d⁢Ni𝑇𝑑𝑆𝑃𝑑𝑉subscript𝑖subscript𝜇𝑖𝑑subscript𝑁𝑖\displaystyle TdS-PdV+\sum_{i}\mu_{i}dN_{i}italic_T italic_d italic_S - italic_P italic_d italic_V + ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_d italic_N start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT(1.1)d⁢F𝑑𝐹\displaystyle dFitalic_d italic_F=\displaystyle==−S⁢d⁢T−P⁢d⁢V+∑iμi⁢d⁢Ni𝑆𝑑𝑇𝑃𝑑𝑉subscript𝑖subscript𝜇𝑖𝑑subscript𝑁𝑖\displaystyle-SdT-PdV+\sum_{i}\mu_{i}dN_{i}- italic_S italic_d italic_T - italic_P italic_d italic_V + ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_d italic_N start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT(1.2)d⁢H𝑑𝐻\displaystyle dHitalic_d italic_H=\displaystyle==T⁢d⁢S+V⁢d⁢P+∑iμi⁢d⁢Ni𝑇𝑑𝑆𝑉𝑑𝑃subscript𝑖subscript𝜇𝑖𝑑subscript𝑁𝑖\displaystyle TdS+VdP+\sum_{i}\mu_{i}dN_{i}italic_T italic_d italic_S + italic_V italic_d italic_P + ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_d italic_N start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT(1.3)d⁢G𝑑𝐺\displaystyle dGitalic_d italic_G=\displaystyle==−S⁢d⁢T+V⁢d⁢P+∑iμi⁢d⁢Ni,𝑆𝑑𝑇𝑉𝑑𝑃subscript𝑖subscript𝜇𝑖𝑑subscript𝑁𝑖\displaystyle-SdT+VdP+\sum_{i}\mu_{i}dN_{i},- italic_S italic_d italic_T + italic_V italic_d italic_P + ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_d italic_N start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ,(1.4)whereS𝑆Sitalic_S,T𝑇Titalic_T,P𝑃Pitalic_P,V𝑉Vitalic_Vare entropy, temperature, pressure and volume of the system respectively. For fixed number of particles, volume or pressure, we have the following equations of state for temperature:

T=(∂U∂S)V,{Ni}=(∂H∂S)P,{Ni}.𝑇subscript𝑈𝑆𝑉subscript𝑁𝑖subscript𝐻𝑆𝑃subscript𝑁𝑖T=\left(\frac{\partial U}{\partial S}\right)_{V,\{N_{i}\}}=\left(\frac{% \partial H}{\partial S}\right)_{P,\{N_{i}\}}.italic_T = ( divide start_ARG ∂ italic_U end_ARG start_ARG ∂ italic_S end_ARG ) start_POSTSUBSCRIPT italic_V , { italic_N start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT } end_POSTSUBSCRIPT = ( divide start_ARG ∂ italic_H end_ARG start_ARG ∂ italic_S end_ARG ) start_POSTSUBSCRIPT italic_P , { italic_N start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT } end_POSTSUBSCRIPT .(1.5) Figure 1:A machine learning (ML) system includes the initial model design, setting of initial parameters, importing data for training, importing new data for prediction tasks, and the process of keeping the model refreshed with new data. From a physics perspective, such a system with a series of steps is analogous to a heat engine. We can examine the various temperatures of the system during these processes, as well as the changes in energy and entropy.On the other hand, a ML system has energy and entropy, but lacks some other physical quantities such as volume and pressure. We can use a method similar to equation (1.5) to calculate the ML system temperature. Consider a system transitioning from one state to another, where the change in energy isΔ⁢EΔ𝐸\Delta Eroman_Δ italic_E, and the change in entropy isΔ⁢SΔ𝑆\Delta Sroman_Δ italic_S, the for an equilibrium system, we can useT=Δ⁢E/Δ⁢S𝑇Δ𝐸Δ𝑆T=\Delta E/\Delta Sitalic_T = roman_Δ italic_E / roman_Δ italic_Sto calculate the temperature of the system. Note that for a machine learning system, we should not simply view it as a static data processor that goes from raw data input to prediction output. Instead, we should see it as a dynamic process, which includes a series of processes including the initial design of the model architecture, parameter initialization, optimization of the loss function and parameter tuning, and keep refreshing the model dynamically due to data shifts. Therefore, the temperature of a ML system must also be used to describe the entire process above. Figure1shows such a system with multiple processes, and we can explore the various system temperature in different steps.

This paper is organized as follows. In Section2, we develop the general theory to build a thermodynamic framework for ML systems. Three pivotal elements define an ML system: the data flow, model structure with its parameters, and system energy. In particular, we introduce two states of an ML system, corresponding to the stages of parameter initialization and data shifting respectively (Section2.1). The ML training process can be viewed as an isothermal phase transition process, and the two states can be unified into a global picture (AppendixC.1). In Section2.2, we assign new physical meaning to the model loss function, viewing it as the internal potential energy of a ML system, which follows the principle of minimum potential energy. We emphasize that the meaning of energy in our theory differs from energy in energy-based models in literature. We then review system entropy in Section2.3and discuss the relation between discrete and differential entropy under dimension collapse scenarios (AppendixB). The comparison of the ML system from an ML perspective versus a thermodynamic perspective is presented in Section2.5.

Next, we derive the temperature of various ML systems based on different system energies and parameter initialization methods. In Section3, we develop the temperature theory in ML systems with a linear regression model and mean square error (MSE) as the internal energy, while parameters are initialized by normal distribution (Section3.1), uniform distribution (Section3.2), and mixed distribution (Section3.3). We also delve into alternative energy forms in ML systems, specifically the forms of Mean Square Error (MSE) and Mean Absolute Error (MAE) with regularization. The temperatures for these systems are derived in Sections3.4and3.5respectively. Note that we prioritize deriving analytical expressions for system temperature. However, in cases where an analytical solution is elusive, we resort to asymptotic methods. We highlight the physical meaning of our temperature theory in Section3.6, demonstrating temperature in equilibrium systems, and energy transfer and temperatures changes in non-equilibrium systems. Furthermore, we derive temperature in ML systems with logistic regression and cross-entropy energy in Section4.

The thermodynamic theory of artificial neural networks (NNs) is developed in Section5. For simplicity, we only consider MSE energy and asymptotic solutions for NN systems in this paper. We discuss global and local temperature in NN systems, while the global temperature is for the whole system while local temperature is for each individual NN layer. In Section5.3, we argue that an NN system can be viewed as a complex heat engine, where the engine work efficiency is defined as the ratio between the energy output from the last layer to the total energy released from the system. The system work efficiency is highly dependent on activation functions in NNs. Based on this, we define two types of heat engines categorized by their work efficiency.

2General Theory

Figure 2:Three fundamental elements of a basic machine learning (ML) system.Let us first clarify the definition of ML system in this paper. Figure2shows the three core elements of a basic ML system – model, data, and energy. The model includes its architecture and parameters𝝁𝝁\boldsymbol{\mu}bold_italic_μ. The data is used to train the model, optimize its parameters, and compare with the model output. The energy, as the intrinsic form of the system, will be studied from a thermodynamic perspective in this paper.

2.1State of a Machine Learning system

In thermodynamics, a thermodynamic state refers to the macroscopic state of a system that is characterized and specified by a set of measurable physical properties known asstate variables. This definition of a thermodynamic state can be extended to ML systems.

Let𝒳∈ℝK𝒳superscriptℝ𝐾\mathcal{X}\in\mathbb{R}^{K}caligraphic_X ∈ blackboard_R start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPTdenote aK−limit-from𝐾K-italic_K -dimensional real valued random input vector, and𝒴∈ℝ𝒴ℝ\mathcal{Y}\in\mathbb{R}caligraphic_Y ∈ blackboard_Ra real valued random output variable, and the data domain of a ML system is𝒟=𝒳×𝒴𝒟𝒳𝒴\mathcal{D}=\mathcal{X}\times\mathcal{Y}caligraphic_D = caligraphic_X × caligraphic_Y. A model functionf𝑓fitalic_fgives𝒴^=f⁢(𝒳)^𝒴𝑓𝒳\hat{\mathcal{Y}}=f(\mathcal{X})over^ start_ARG caligraphic_Y end_ARG = italic_f ( caligraphic_X ), where𝒴^^𝒴\hat{\mathcal{Y}}over^ start_ARG caligraphic_Y end_ARGis the prediction which can be compared with the output variable𝒴𝒴\mathcal{Y}caligraphic_Y. From this we can define a class of functions

𝔉={f|𝒴^=f𝝁⁢(𝒳),𝝁∈ℝn},𝔉conditional-set𝑓formulae-sequence^𝒴subscript𝑓𝝁𝒳𝝁superscriptℝ𝑛\mathfrak{F}=\{f|\hat{\mathcal{Y}}=f_{\boldsymbol{\mu}}(\mathcal{X}),{% \boldsymbol{\mu}}\in\mathbb{R}^{n}\},fraktur_F = { italic_f | over^ start_ARG caligraphic_Y end_ARG = italic_f start_POSTSUBSCRIPT bold_italic_μ end_POSTSUBSCRIPT ( caligraphic_X ) , bold_italic_μ ∈ blackboard_R start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT } ,(2.1)where𝝁𝝁{\boldsymbol{\mu}}bold_italic_μis the set of parameters, which form the parameter spaceℝnsuperscriptℝ𝑛\mathbb{R}^{n}blackboard_R start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT. A state in the ML system can be considered as a subset of𝒳×Y×𝝁𝒳𝑌𝝁\mathcal{X}\times Y\times{\boldsymbol{\mu}}caligraphic_X × italic_Y × bold_italic_μ.

In particular, we have two types of states as follows.

Type I State and Phase Transition.

For a given dataset from the domain𝒟=𝒳×Y𝒟𝒳𝑌\mathcal{D}=\mathcal{X}\times Ycaligraphic_D = caligraphic_X × italic_Y, all possible𝝁𝝁{\boldsymbol{\mu}}bold_italic_μin the parameter space forms a state. State I from Figure3shows an example of this kind of state. For a given dataset, we have a series of parameters𝝁={𝝁i}i=1,2,3,..,n{\boldsymbol{\mu}}=\{\boldsymbol{\mu}_{i}\}_{i=1,2,3,..,n}bold_italic_μ = { bold_italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT } start_POSTSUBSCRIPT italic_i = 1 , 2 , 3 , . . , italic_n end_POSTSUBSCRIPT, where each𝝁isubscript𝝁𝑖\boldsymbol{\mu}_{i}bold_italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPTrepresents a point in the parameter space, corresponding to a functionfisubscript𝑓𝑖f_{i}italic_f start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPTin𝔉𝔉\mathfrak{F}fraktur_Ffrom Equation (2.1). Also, the parameter set𝝁isubscript𝝁𝑖\boldsymbol{\mu}_{i}bold_italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPTand functionfisubscript𝑓𝑖f_{i}italic_f start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPTcorrespond to an energy levelEisubscript𝐸𝑖E_{i}italic_E start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT, which is a function of𝝁isubscript𝝁𝑖\boldsymbol{\mu}_{i}bold_italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPTand𝒟𝒟\mathcal{D}caligraphic_D. All of the{Ei,𝝁i}subscript𝐸𝑖subscript𝝁𝑖\{E_{i},\boldsymbol{\mu}_{i}\}{ italic_E start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , bold_italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT }form a state, and each{Ei,𝝁i}subscript𝐸𝑖subscript𝝁𝑖\{E_{i},\boldsymbol{\mu}_{i}\}{ italic_E start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , bold_italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT }can be considered as a “particle“ in the system.

The total energy of the state can be calculated by

E0=∑i=1nEi⁢pi,subscript𝐸0superscriptsubscript𝑖1𝑛subscript𝐸𝑖subscript𝑝𝑖E_{0}=\sum_{i=1}^{n}E_{i}p_{i},italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT = ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT italic_E start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_p start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ,(2.2)wherepisubscript𝑝𝑖p_{i}italic_p start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPTis the probability ofμisubscript𝜇𝑖\mu_{i}italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPTin the parameter space. On the other hand, the (Sharon) entropy of the state is

S0=−∑i=1pi⁢log⁡pi,subscript𝑆0subscript𝑖1subscript𝑝𝑖subscript𝑝𝑖S_{0}=-\sum_{i=1}p_{i}\log p_{i},italic_S start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT = - ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT italic_p start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT roman_log italic_p start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ,(2.3)which is the entropy in the parameter space. Since the dataset in𝒟𝒟\mathcal{D}caligraphic_Dis given and fixed,S0subscript𝑆0S_{0}italic_S start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPTgives the overall entropy of the system.

Figure 3:Type I state of a ML system (State I), which is the state for a given dataset with a set of parameters𝝁𝝁{\boldsymbol{\mu}}bold_italic_μ. This state represents a ML system that has not yet been trained, and each{Ei,𝝁i}subscript𝐸𝑖subscript𝝁𝑖\{E_{i},\boldsymbol{\mu}_{i}\}{ italic_E start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , bold_italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT }can be considered as a particle. After training, all the particles converge to{E^,𝝁^}^𝐸^𝝁\{\hat{E},\hat{\boldsymbol{\mu}}\}{ over^ start_ARG italic_E end_ARG , over^ start_ARG bold_italic_μ end_ARG }, which is State II. The transition from State I to State II is the process of the ML system going from initial state to trained state. Meanwhile, it can also be viewed as a phase transition process from State I to State II. We can use an isothermal phase transition process to calculate the temperature of the system.Now let us consider the training process of the ML system. For any given initial parameter𝝁isubscript𝝁𝑖{\boldsymbol{\mu}_{i}}bold_italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT, the system converges to an optimized state with energyE^^𝐸\hat{E}over^ start_ARG italic_E end_ARGand parameter𝝁^^𝝁\hat{\boldsymbol{\mu}}over^ start_ARG bold_italic_μ end_ARGafter training. Ideally, regardless of the initial parameters, the system will eventually converge to the same{E^,𝝁^}^𝐸^𝝁\{\hat{E},\hat{\boldsymbol{\mu}}\}{ over^ start_ARG italic_E end_ARG , over^ start_ARG bold_italic_μ end_ARG }, and the corresponding functionf^^𝑓\hat{f}over^ start_ARG italic_f end_ARGis the optimal function among all functions in the function class𝔉𝔉\mathfrak{F}fraktur_Fto fit the relation between𝒳𝒳\mathcal{X}caligraphic_Xand𝒴𝒴\mathcal{Y}caligraphic_Y.

State I in Figure3corresponds to the parameter initialization process of the ML system. The ML training process is essentially transitioning from the parameter initialization state to the parameter convergence state. In Figure3, this is thephase transitionprocess from State I to State II. During the phase transition, the system energy changes fromE0subscript𝐸0E_{0}italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT(equation2.2) in State I toE^^𝐸\hat{E}over^ start_ARG italic_E end_ARGin State II, and the system entropy changes fromS0subscript𝑆0S_{0}italic_S start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT(equation2.3) to zero. Clearly, the phase transition is also the process of entropy reduction. Assuming the phase transition is an isothermal process, we can calculate the temperature of the system as

T𝑇\displaystyle Titalic_T=\displaystyle==Estage⁢II−Estage⁢ISstage⁢II−Sstage⁢Isubscript𝐸stageIIsubscript𝐸stageIsubscript𝑆stageIIsubscript𝑆stageI\displaystyle\frac{E_{\rm stage\;II}-E_{\rm stage\;I}}{S_{\rm stage\;II}-S_{% \rm stage\;I}}divide start_ARG italic_E start_POSTSUBSCRIPT roman_stage roman_II end_POSTSUBSCRIPT - italic_E start_POSTSUBSCRIPT roman_stage roman_I end_POSTSUBSCRIPT end_ARG start_ARG italic_S start_POSTSUBSCRIPT roman_stage roman_II end_POSTSUBSCRIPT - italic_S start_POSTSUBSCRIPT roman_stage roman_I end_POSTSUBSCRIPT end_ARG(2.4)=\displaystyle==E^−∑i=1nEi⁢pi∑i=1pi⁢log⁡pi.^𝐸superscriptsubscript𝑖1𝑛subscript𝐸𝑖subscript𝑝𝑖subscript𝑖1subscript𝑝𝑖subscript𝑝𝑖\displaystyle\frac{\hat{E}-\sum_{i=1}^{n}E_{i}p_{i}}{\sum_{i=1}p_{i}\log p_{i}}.divide start_ARG over^ start_ARG italic_E end_ARG - ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT italic_E start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_p start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT italic_p start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT roman_log italic_p start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG .Note that Equation (2.4) and Figure3are for the discrete case. For the continuous case, we have

E0=∫E⁢(𝝁)⁢p⁢(𝝁)⁢𝑑𝝁,subscript𝐸0𝐸𝝁𝑝𝝁differential-d𝝁\displaystyle E_{0}=\int E({\boldsymbol{\mu}})p({\boldsymbol{\mu}})d{% \boldsymbol{\mu}},italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT = ∫ italic_E ( bold_italic_μ ) italic_p ( bold_italic_μ ) italic_d bold_italic_μ ,(2.5)S0=−∫p⁢(𝝁)⁢log⁡[p⁢(𝝁)]⁢𝑑𝝁,subscript𝑆0𝑝𝝁𝑝𝝁differential-d𝝁\displaystyle S_{0}=-\int p({\boldsymbol{\mu}})\log[p({\boldsymbol{\mu}})]d{% \boldsymbol{\mu}},italic_S start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT = - ∫ italic_p ( bold_italic_μ ) roman_log [ italic_p ( bold_italic_μ ) ] italic_d bold_italic_μ ,(2.6)T=∫E𝝁⁢𝑑𝝁−E^−∫p⁢(𝝁)⁢log⁡[p⁢(𝝁)]⁢𝑑𝝁,𝑇subscript𝐸𝝁differential-d𝝁^𝐸𝑝𝝁𝑝𝝁differential-d𝝁\displaystyle T=\frac{\int E_{\boldsymbol{\mu}}d{\boldsymbol{\mu}}-\hat{E}}{-% \int p({\boldsymbol{\mu}})\log[p({\boldsymbol{\mu}})]d{\boldsymbol{\mu}}},italic_T = divide start_ARG ∫ italic_E start_POSTSUBSCRIPT bold_italic_μ end_POSTSUBSCRIPT italic_d bold_italic_μ - over^ start_ARG italic_E end_ARG end_ARG start_ARG - ∫ italic_p ( bold_italic_μ ) roman_log [ italic_p ( bold_italic_μ ) ] italic_d bold_italic_μ end_ARG ,(2.7)wherep⁢(𝝁)𝑝𝝁p({\boldsymbol{\mu}})italic_p ( bold_italic_μ )is the probability density function of𝝁𝝁{\boldsymbol{\mu}}bold_italic_μ. However, it should be aware of that Equation (2.6) gives differential entropy, which is fundamentally different from the discrete Shannon entropy. We will have more detailed discussion in Section2.3.

Type II State.

Different from the Type I state, the Type II state of a ML system is a state given by shifiting of the dataset after specifying a parameter set𝝁^^𝝁\hat{\boldsymbol{\mu}}over^ start_ARG bold_italic_μ end_ARG. Assume the ML system’s model is trained on a dataset𝒟1subscript𝒟1\mathcal{D}_{1}caligraphic_D start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT, where𝒟1subscript𝒟1\mathcal{D}_{1}caligraphic_D start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPTis a finite sequence of pairs in the data domain𝒳×𝒴𝒳𝒴\mathcal{X}\times\mathcal{Y}caligraphic_X × caligraphic_Ythat𝒟1=𝒳1×𝒴1subscript𝒟1subscript𝒳1subscript𝒴1\mathcal{D}_{1}=\mathcal{X}_{1}\times\mathcal{Y}_{1}caligraphic_D start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT = caligraphic_X start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT × caligraphic_Y start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPTwith𝒳1⊂𝒳subscript𝒳1𝒳\mathcal{X}_{1}\subset\mathcal{X}caligraphic_X start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ⊂ caligraphic_X,𝒴1⊂𝒴subscript𝒴1𝒴\mathcal{Y}_{1}\subset\mathcal{Y}caligraphic_Y start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ⊂ caligraphic_Y, and𝒟1⊂𝒟subscript𝒟1𝒟\mathcal{D}_{1}\subset\mathcal{D}caligraphic_D start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ⊂ caligraphic_D. We useE⁢(𝒟1)𝐸subscript𝒟1E(\mathcal{D}_{1})italic_E ( caligraphic_D start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT )to denote the energy of the system under this state. Thus, the entropy of the state can be written as

S⁢(𝒟1)=−∑(xi,yi)∈𝒟1P⁢(xi,yi)⁢log⁡[P⁢(xi,yi)],𝑆subscript𝒟1subscriptsubscript𝑥𝑖subscript𝑦𝑖subscript𝒟1𝑃subscript𝑥𝑖subscript𝑦𝑖𝑃subscript𝑥𝑖subscript𝑦𝑖S(\mathcal{D}_{1})=-\sum_{(x_{i},y_{i})\in\mathcal{D}_{1}}P(x_{i},y_{i})\log[P% (x_{i},y_{i})],italic_S ( caligraphic_D start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ) = - ∑ start_POSTSUBSCRIPT ( italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) ∈ caligraphic_D start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPT italic_P ( italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) roman_log [ italic_P ( italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) ] ,(2.8)whereP⁢(xi,yi)𝑃subscript𝑥𝑖subscript𝑦𝑖P(x_{i},y_{i})italic_P ( italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT )is the joint probability in the domain, and each(xi,yi)subscript𝑥𝑖subscript𝑦𝑖(x_{i},y_{i})( italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT )can be considered as a particle in Type II state. Note that𝒟1=𝒳1×𝒴1subscript𝒟1subscript𝒳1subscript𝒴1\mathcal{D}_{1}=\mathcal{X}_{1}\times\mathcal{Y}_{1}caligraphic_D start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT = caligraphic_X start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT × caligraphic_Y start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPTis just a sample from the entire domain𝒟=𝒳×𝒴𝒟𝒳𝒴\mathcal{D}=\mathcal{X}\times\mathcal{Y}caligraphic_D = caligraphic_X × caligraphic_Y. Suppose the data used by the ML system is constantly shifting:𝒟1=𝒳1×𝒴1subscript𝒟1subscript𝒳1subscript𝒴1\mathcal{D}_{1}=\mathcal{X}_{1}\times\mathcal{Y}_{1}caligraphic_D start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT = caligraphic_X start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT × caligraphic_Y start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT(State 1)→𝒟2=𝒳2×𝒴2→absentsubscript𝒟2subscript𝒳2subscript𝒴2\rightarrow\mathcal{D}_{2}=\mathcal{X}_{2}\times\mathcal{Y}_{2}→ caligraphic_D start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT = caligraphic_X start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT × caligraphic_Y start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT(State 2)→𝒟3=𝒳3×𝒴3→absentsubscript𝒟3subscript𝒳3subscript𝒴3\rightarrow\mathcal{D}_{3}=\mathcal{X}_{3}\times\mathcal{Y}_{3}→ caligraphic_D start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT = caligraphic_X start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT × caligraphic_Y start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT(State 3) ……., then the ML system can be viewed as a thermodynamic system that continuously absorbs or releases energy, as shown in Figure4. When the dataset of the system changes from𝒟jsubscript𝒟𝑗\mathcal{D}_{j}caligraphic_D start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPTto𝒟j+1subscript𝒟𝑗1\mathcal{D}_{j+1}caligraphic_D start_POSTSUBSCRIPT italic_j + 1 end_POSTSUBSCRIPT, and the energy changes fromE⁢(𝒟j)𝐸subscript𝒟𝑗E(\mathcal{D}_{j})italic_E ( caligraphic_D start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT )toE⁢(𝒟j+1)𝐸subscript𝒟𝑗1E(\mathcal{D}_{j+1})italic_E ( caligraphic_D start_POSTSUBSCRIPT italic_j + 1 end_POSTSUBSCRIPT ), the temperature (Tj,j+1subscript𝑇𝑗𝑗1T_{j,j+1}italic_T start_POSTSUBSCRIPT italic_j , italic_j + 1 end_POSTSUBSCRIPT) of the system during this process is

Tj,j+1=E⁢(𝒟j+1)−E⁢(𝒟j)∑(xi,yi)∈𝒟jP⁢(xi,yi)⁢log⁡[P⁢(xi,yi)]−∑(xi,yi)∈𝒟j+1P⁢(xi,yi)⁢log⁡[P⁢(xi,yi)],subscript𝑇𝑗𝑗1𝐸subscript𝒟𝑗1𝐸subscript𝒟𝑗subscriptsubscript𝑥𝑖subscript𝑦𝑖subscript𝒟𝑗𝑃subscript𝑥𝑖subscript𝑦𝑖𝑃subscript𝑥𝑖subscript𝑦𝑖subscriptsubscript𝑥𝑖subscript𝑦𝑖subscript𝒟𝑗1𝑃subscript𝑥𝑖subscript𝑦𝑖𝑃subscript𝑥𝑖subscript𝑦𝑖T_{j,j+1}=\frac{E(\mathcal{D}_{j+1})-E(\mathcal{D}_{j})}{\sum_{(x_{i},y_{i})% \in\mathcal{D}_{j}}P(x_{i},y_{i})\log[P(x_{i},y_{i})]-\sum_{(x_{i},y_{i})\in% \mathcal{D}_{j+1}}P(x_{i},y_{i})\log[P(x_{i},y_{i})]},italic_T start_POSTSUBSCRIPT italic_j , italic_j + 1 end_POSTSUBSCRIPT = divide start_ARG italic_E ( caligraphic_D start_POSTSUBSCRIPT italic_j + 1 end_POSTSUBSCRIPT ) - italic_E ( caligraphic_D start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT ) end_ARG start_ARG ∑ start_POSTSUBSCRIPT ( italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) ∈ caligraphic_D start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT end_POSTSUBSCRIPT italic_P ( italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) roman_log [ italic_P ( italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) ] - ∑ start_POSTSUBSCRIPT ( italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) ∈ caligraphic_D start_POSTSUBSCRIPT italic_j + 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPT italic_P ( italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) roman_log [ italic_P ( italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) ] end_ARG ,(2.9)where in Equation (2.9), the denominator shows the change in the system entropy, and the numerator gives the change in the system energy.

Figure 4:The evolution of the Type II state of a ML system (discrete case). The training dataset of the system is constantly evolving from𝒟1=𝒳1×𝒴1subscript𝒟1subscript𝒳1subscript𝒴1\mathcal{D}_{1}=\mathcal{X}_{1}\times\mathcal{Y}_{1}caligraphic_D start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT = caligraphic_X start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT × caligraphic_Y start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPTto𝒟2=𝒳2×𝒴2subscript𝒟2subscript𝒳2subscript𝒴2\mathcal{D}_{2}=\mathcal{X}_{2}\times\mathcal{Y}_{2}caligraphic_D start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT = caligraphic_X start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT × caligraphic_Y start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPTto𝒟3subscript𝒟3\mathcal{D}_{3}caligraphic_D start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT…, forming a sequence𝒟ksubscript𝒟𝑘{\mathcal{D}_{k}}caligraphic_D start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT, and the fixed parameter for the model𝝁ksubscript𝝁𝑘{\boldsymbol{\mu}_{k}}bold_italic_μ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPTand𝒟ksubscript𝒟𝑘\mathcal{D}_{k}caligraphic_D start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPTgive a Type II state with energyE^⁢(𝒟k)^𝐸subscript𝒟𝑘\hat{E}(\mathcal{D}_{k})over^ start_ARG italic_E end_ARG ( caligraphic_D start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT ), and the entropy corresponding to the training dataset can be calculated using Equation (2.8). As the Type II state evolves, we can calculate the temperature of the state and its evolution through the changes in energy and entropy.Figure4shows the discrete changes of the Type II state. If the training dataset shifting goes through a continuous changing process, i.e., the streamed training dataset can be written as𝒟⁢(λ)=𝒳⁢(λ)×𝒴⁢(λ)𝒟𝜆𝒳𝜆𝒴𝜆\mathcal{D}(\lambda)=\mathcal{X}(\lambda)\times\mathcal{Y}(\lambda)caligraphic_D ( italic_λ ) = caligraphic_X ( italic_λ ) × caligraphic_Y ( italic_λ ), whereλ𝜆\lambdaitalic_λrepresents a continuous parameter that governs the streamed shifting. In this way, the differential entropy of the system is

S⁢(λ)=−∫P⁢[𝒳⁢(λ),𝒴⁢(λ)]⁢log⁡[P⁢(𝒳⁢(λ),𝒴⁢(λ))]⁢𝑑𝒳⁢𝑑𝒴,𝑆𝜆𝑃𝒳𝜆𝒴𝜆𝑃𝒳𝜆𝒴𝜆differential-d𝒳differential-d𝒴S(\lambda)=-\int P[\mathcal{X}(\lambda),\mathcal{Y}(\lambda)]\log[P(\mathcal{X% }(\lambda),\mathcal{Y}(\lambda))]d\mathcal{X}d\mathcal{Y},italic_S ( italic_λ ) = - ∫ italic_P [ caligraphic_X ( italic_λ ) , caligraphic_Y ( italic_λ ) ] roman_log [ italic_P ( caligraphic_X ( italic_λ ) , caligraphic_Y ( italic_λ ) ) ] italic_d caligraphic_X italic_d caligraphic_Y ,(2.10)and we can obtain the temperature

T⁢(λ)=d⁢E⁢(𝒳,𝒴)/d⁢λ−∫{1+log⁡[P⁢(𝒳,𝒴)]}⁢(d⁢Pd⁢𝒳⁢d⁢𝒳d⁢λ+d⁢Pd⁢𝒴⁢d⁢𝒴d⁢λ)⁢𝑑𝒳⁢𝑑𝒴.𝑇𝜆𝑑𝐸𝒳𝒴𝑑𝜆1𝑃𝒳𝒴𝑑𝑃𝑑𝒳𝑑𝒳𝑑𝜆𝑑𝑃𝑑𝒴𝑑𝒴𝑑𝜆differential-d𝒳differential-d𝒴T(\lambda)=\frac{dE(\mathcal{X},\mathcal{Y})/d\lambda}{-\int\{1+\log[P(% \mathcal{X},\mathcal{Y})]\}\left(\frac{dP}{d\mathcal{X}}\frac{d\mathcal{X}}{d% \lambda}+\frac{dP}{d\mathcal{Y}}\frac{d\mathcal{Y}}{d\lambda}\right)d\mathcal{% X}d\mathcal{Y}}.italic_T ( italic_λ ) = divide start_ARG italic_d italic_E ( caligraphic_X , caligraphic_Y ) / italic_d italic_λ end_ARG start_ARG - ∫ { 1 + roman_log [ italic_P ( caligraphic_X , caligraphic_Y ) ] } ( divide start_ARG italic_d italic_P end_ARG start_ARG italic_d caligraphic_X end_ARG divide start_ARG italic_d caligraphic_X end_ARG start_ARG italic_d italic_λ end_ARG + divide start_ARG italic_d italic_P end_ARG start_ARG italic_d caligraphic_Y end_ARG divide start_ARG italic_d caligraphic_Y end_ARG start_ARG italic_d italic_λ end_ARG ) italic_d caligraphic_X italic_d caligraphic_Y end_ARG .(2.11) Different from the phase transition of Type I state, Type II state can be seen as gradually change or continuously change. The biggest question with Type II state is how to calculate the joint probabilityP⁢(𝒳,𝒴)𝑃𝒳𝒴P(\mathcal{X},\mathcal{Y})italic_P ( caligraphic_X , caligraphic_Y ). In most realistic cases, the probability distribution of the data domain and of the training dataset sample are unknown. In order to effectively calculate the temperature change for Type II state, we must use a model to approximate and inferP⁢(𝒳,𝒴)𝑃𝒳𝒴P(\mathcal{X},\mathcal{Y})italic_P ( caligraphic_X , caligraphic_Y ). A commonly used method is to employ Bayes’ theorem such that

P⁢(𝒳,𝒴)=P⁢(𝒴|𝒳)⁢P⁢(𝒳)𝑃𝒳𝒴𝑃conditional𝒴𝒳𝑃𝒳P(\mathcal{X,Y})=P(\mathcal{Y|X})P(\mathcal{X})italic_P ( caligraphic_X , caligraphic_Y ) = italic_P ( caligraphic_Y | caligraphic_X ) italic_P ( caligraphic_X )(2.12)withP⁢(𝒴|𝒳)𝑃conditional𝒴𝒳P(\mathcal{Y|X})italic_P ( caligraphic_Y | caligraphic_X )obtained from modeling, and additional assumptions onP⁢(𝒳)𝑃𝒳P(\mathcal{X})italic_P ( caligraphic_X ). AppendixAprovides details on how to calculateP⁢(𝒳,𝒴)𝑃𝒳𝒴P(\mathcal{X},\mathcal{Y})italic_P ( caligraphic_X , caligraphic_Y ), and how to unify Type I and Type II states together within a single framework.

The temperature in the Type II state is highly dependent on the changes in the system data, while the temperature in the Type I state depends more on the method of system parameter initialization and the form of energy expression, and can be derived from first principles.This paper gives a comprehensive discussion on global temperature for ML systems in this section, with subsequent discussions focusing mainly on the Type I state.

2.2System Energy

Figure 5:Energy of the system in𝒴𝒴\mathcal{Y}caligraphic_Yspace. The left figure shows that the system has “long-range energy“, that is, there is energy𝒱⁢(yi,yj)𝒱subscript𝑦𝑖subscript𝑦𝑗\mathscr{V}(y_{i},y_{j})script_V ( italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT )between any internal particlesyisubscript𝑦𝑖y_{i}italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPTandyjsubscript𝑦𝑗y_{j}italic_y start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPTwithin the system. Also, there is energy𝒱⁢(y^k,yi)𝒱subscript^𝑦𝑘subscript𝑦𝑖\mathscr{V}(\hat{y}_{k},y_{i})script_V ( over^ start_ARG italic_y end_ARG start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT )between any “position“yk^^subscript𝑦𝑘\hat{y_{k}}over^ start_ARG italic_y start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT end_ARGand any internal particleyisubscript𝑦𝑖y_{i}italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT. See Equation (2.14) to describe the energy of such as system. The right figure shows a simplified energy model where the system has “short-range energy“. In this model, any position within the systemy^ksubscript^𝑦𝑘\hat{y}_{k}over^ start_ARG italic_y end_ARG start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPTinteracts only with its nearest internal particleyksubscript𝑦𝑘y_{k}italic_y start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT. The mutual energy between internal particles is ignored due to its lack of variation. See Equation (2.15) to describe such a system.In classical physics, the energy of a system can be divided into the potential and kinetic energy, the potential energy includes both the external background contributions, like gravitational potential energy, and the energy arising from particle interactions within the system. For example, for a molecular system, the potential energy of interaction between molecules can be described by the Lennard-Jones potential, which is written as[29,30]

EL−J=∑i=1N∑j>iN4⁢ϵ⁢[(σri⁢j)12−(σri⁢j)6],subscript𝐸𝐿𝐽superscriptsubscript𝑖1𝑁superscriptsubscript𝑗𝑖𝑁4italic-ϵdelimited-[]superscript𝜎subscript𝑟𝑖𝑗12superscript𝜎subscript𝑟𝑖𝑗6E_{L-J}=\sum_{i=1}^{N}\sum_{j>i}^{N}4\epsilon\left[\left(\frac{\sigma}{r_{ij}}% \right)^{12}-\left(\frac{\sigma}{r_{ij}}\right)^{6}\right],italic_E start_POSTSUBSCRIPT italic_L - italic_J end_POSTSUBSCRIPT = ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT ∑ start_POSTSUBSCRIPT italic_j > italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT 4 italic_ϵ [ ( divide start_ARG italic_σ end_ARG start_ARG italic_r start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT end_ARG ) start_POSTSUPERSCRIPT 12 end_POSTSUPERSCRIPT - ( divide start_ARG italic_σ end_ARG start_ARG italic_r start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT end_ARG ) start_POSTSUPERSCRIPT 6 end_POSTSUPERSCRIPT ] ,(2.13)whereri⁢jsubscript𝑟𝑖𝑗r_{ij}italic_r start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPTis the distance between particles, andϵitalic-ϵ\epsilonitalic_ϵis the depth of the potential well. For a ML system, we can also construct a potential energy similar to the interaction between molecules. For systems with labels, the potential energy can be directly established in the𝒴𝒴\mathcal{Y}caligraphic_Yspace. Figure5shows demonstrations of such a potential. Lety^k=f𝝁⁢(xk)subscript^𝑦𝑘subscript𝑓𝝁subscript𝑥𝑘\hat{y}_{k}=f_{\boldsymbol{\mu}}(x_{k})over^ start_ARG italic_y end_ARG start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT = italic_f start_POSTSUBSCRIPT bold_italic_μ end_POSTSUBSCRIPT ( italic_x start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT ), the left panel of Figure5shows a general scenario for the potential energy of the system

Ep=∑i=1N∑k=1N𝒱⁢(y^k,yi)+∑i=1N∑j>iN𝒱⁢(yj,yi),subscript𝐸𝑝superscriptsubscript𝑖1𝑁superscriptsubscript𝑘1𝑁𝒱subscript^𝑦𝑘subscript𝑦𝑖superscriptsubscript𝑖1𝑁superscriptsubscript𝑗𝑖𝑁𝒱subscript𝑦𝑗subscript𝑦𝑖E_{p}=\sum_{i=1}^{N}\sum_{k=1}^{N}\mathscr{V}(\hat{y}_{k},y_{i})+\sum_{i=1}^{N% }\sum_{j>i}^{N}\mathscr{V}(y_{j},y_{i}),italic_E start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT = ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT ∑ start_POSTSUBSCRIPT italic_k = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT script_V ( over^ start_ARG italic_y end_ARG start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) + ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT ∑ start_POSTSUBSCRIPT italic_j > italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_N end_POSTSUPERSCRIPT script_V ( italic_y start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) ,(2.14)where the first term𝒱⁢(yk,yi)𝒱subscript𝑦𝑘subscript𝑦𝑖\mathscr{V}(y_{k},y_{i})script_V ( italic_y start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT )is the interaction energy between all positionsy^ksubscript^𝑦𝑘\hat{y}_{k}over^ start_ARG italic_y end_ARG start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPTandyisubscript𝑦𝑖y_{i}italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT. As for the second term, if the system dataset remains unchanged, then this term is also constant, and thus we can ignore it.

Equation (2.14) accounts for the interactions among all particles in the system. However, if these interactions are short-ranged, a simplified approach is to only consider interactions between neighboring particles. This is often used in certain lattice gas models where particles on a lattice interact exclusively with their nearest neighbors. For example, in the Ising model, we consider a lattice where spins interact solely with their immediate neighbors. The right panel of Figure5shows a similar concept within the context of a ML system, wherey^isubscript^𝑦𝑖\hat{y}_{i}over^ start_ARG italic_y end_ARG start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPTcan be viewed as the displacement ofyisubscript𝑦𝑖y_{i}italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT, andyisubscript𝑦𝑖y_{i}italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPTinteracts only withy^isubscript^𝑦𝑖\hat{y}_{i}over^ start_ARG italic_y end_ARG start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT. The interaction energy can be formulated as

Ep=∑i𝒱⁢(y^i,yi)=∑i‖f𝝁⁢(yi)−yi‖.subscript𝐸𝑝subscript𝑖𝒱subscript^𝑦𝑖subscript𝑦𝑖subscript𝑖normsubscript𝑓𝝁subscript𝑦𝑖subscript𝑦𝑖E_{p}=\sum_{i}\mathscr{V}(\hat{y}_{i},y_{i})=\sum_{i}||f_{\boldsymbol{\mu}}(y_% {i})-y_{i}||.italic_E start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT = ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT script_V ( over^ start_ARG italic_y end_ARG start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) = ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT | | italic_f start_POSTSUBSCRIPT bold_italic_μ end_POSTSUBSCRIPT ( italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT | | .(2.15) Equation (2.15) can be considered as the simplified version of Equation (2.14), and is also a commonly used version of energy in machine learning. The most common forms of energy expressions are

• •The mean squared error (MSE): EMSE=1n⁢∑i=1n(y^i−yi)2,subscript𝐸MSE1𝑛superscriptsubscript𝑖1𝑛superscriptsubscript^𝑦𝑖subscript𝑦𝑖2E_{\rm MSE}=\frac{1}{n}\sum_{i=1}^{n}(\hat{y}_{i}-y_{i})^{2},italic_E start_POSTSUBSCRIPT roman_MSE end_POSTSUBSCRIPT = divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ( over^ start_ARG italic_y end_ARG start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ,(2.16)
• •The mean absolute error (MAE): EMAE=1n⁢∑i=1n|y^i−yi|,subscript𝐸MAE1𝑛superscriptsubscript𝑖1𝑛subscript^𝑦𝑖subscript𝑦𝑖E_{\rm MAE}=\frac{1}{n}\sum_{i=1}^{n}|\hat{y}_{i}-y_{i}|,italic_E start_POSTSUBSCRIPT roman_MAE end_POSTSUBSCRIPT = divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT | over^ start_ARG italic_y end_ARG start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT | ,(2.17)
• •The mean bias error (MBA): EMBA=1n⁢∑i=1n(y^i−yi),subscript𝐸MBA1𝑛superscriptsubscript𝑖1𝑛subscript^𝑦𝑖subscript𝑦𝑖E_{\rm MBA}=\frac{1}{n}\sum_{i=1}^{n}(\hat{y}_{i}-y_{i}),italic_E start_POSTSUBSCRIPT roman_MBA end_POSTSUBSCRIPT = divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ( over^ start_ARG italic_y end_ARG start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) ,(2.18)
• •The Cross Entropy (CE): ECE=−1n⁢∑i=1n[yi⁢log⁡(yi^)+(1−yi)⁢log⁡(1−y^i)].subscript𝐸CE1𝑛superscriptsubscript𝑖1𝑛delimited-[]subscript𝑦𝑖^subscript𝑦𝑖1subscript𝑦𝑖1subscript^𝑦𝑖E_{\rm CE}=-\frac{1}{n}\sum_{i=1}^{n}[y_{i}\log(\hat{y_{i}})+(1-y_{i})\log(1-% \hat{y}_{i})].italic_E start_POSTSUBSCRIPT roman_CE end_POSTSUBSCRIPT = - divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT [ italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT roman_log ( over^ start_ARG italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG ) + ( 1 - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) roman_log ( 1 - over^ start_ARG italic_y end_ARG start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) ] .(2.19)

Different from EBMs, we tend to interpret the loss function as the potential energy of the system. In the context of EBMs, inference and learning refer to two distinct processes, where the goal of learning is to adjust the model parameters, while the goal of inference is to find configurations of the variables that optimize the energy function. For the state changes and phase transition described in Section2.1, we actually view learning as the process of system energy change, so we also view the loss function corresponding to learning as the potential energy of the system. For example, loss functions with L1 or L2 regularization are

L1:Ep=||f𝝁(yi)−yi||+λ∑j|μj|\displaystyle{\rm L1:}\quad E_{p}=||f_{\boldsymbol{\mu}}(y_{i})-y_{i}||+% \lambda\sum_{j}|\mu_{j}|L1 : italic_E start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT = | | italic_f start_POSTSUBSCRIPT bold_italic_μ end_POSTSUBSCRIPT ( italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT | | + italic_λ ∑ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT | italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT |(2.20)L2:Ep=||f𝝁(yi)−yi||+λ∑j|μj2|.\displaystyle{\rm L2:}\quad E_{p}=||f_{\boldsymbol{\mu}}(y_{i})-y_{i}||+% \lambda\sum_{j}|\mu_{j}^{2}|.L2 : italic_E start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT = | | italic_f start_POSTSUBSCRIPT bold_italic_μ end_POSTSUBSCRIPT ( italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT | | + italic_λ ∑ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT | italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT | .(2.21)They can also be treated as two forms of potential energy. From a physical perspective, compared to Equation (2.15), Equations (2.20) and (2.21) have extra terms added a “background potential“ that is unrelated to the internal potential of the system.

Another example is the energy for binary classification problem. In addition to the cross entropy energy in the aforementioned Equation (2.19), the energy for binary classification can also be expressed as (see Lecun et al.[4])E=∑i|2⁢yi−1|⁢f𝝁(i)⁢(x)𝐸subscript𝑖2subscript𝑦𝑖1superscriptsubscript𝑓𝝁𝑖𝑥E=\sum_{i}|2y_{i}-1|f_{\boldsymbol{\mu}}^{(i)}(x)italic_E = ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT | 2 italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT - 1 | italic_f start_POSTSUBSCRIPT bold_italic_μ end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_i ) end_POSTSUPERSCRIPT ( italic_x ), wheref𝝁⁢(x)subscript𝑓𝝁𝑥f_{\boldsymbol{\mu}}(x)italic_f start_POSTSUBSCRIPT bold_italic_μ end_POSTSUBSCRIPT ( italic_x )is typically a linear combination of𝝁𝝁{\boldsymbol{\mu}}bold_italic_μandx𝑥xitalic_x, i.e.,𝝁⁢x+𝒃𝝁𝑥𝒃{\boldsymbol{\mu}}x+{\boldsymbol{b}}bold_italic_μ italic_x + bold_italic_b. Clearly, we prefer to use the cross entropy expression as the energy model for binary classification.

From the perspective of the loss function, the setting of model parameters is determined by minimizing a chosen loss function, i.e.,

𝝁^=argmin𝝁⁢1n⁢∑i=0nℒ⁢(y^⁢(xi,𝝁),yi).^𝝁subscriptargmin𝝁1𝑛superscriptsubscript𝑖0𝑛ℒ^𝑦subscript𝑥𝑖𝝁subscript𝑦𝑖\hat{\boldsymbol{\mu}}={\rm argmin}_{\boldsymbol{\mu}}\frac{1}{n}\sum_{i=0}^{n% }\mathcal{L}(\hat{y}(x_{i},{\boldsymbol{\mu}}),y_{i}).over^ start_ARG bold_italic_μ end_ARG = roman_argmin start_POSTSUBSCRIPT bold_italic_μ end_POSTSUBSCRIPT divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT caligraphic_L ( over^ start_ARG italic_y end_ARG ( italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , bold_italic_μ ) , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) .(2.22)This gives us a hint that training a model involves adjusting its parameters to minimize this loss function, much like finding the position where potential energy is minimized.If we consider the loss functionℒℒ\mathcal{L}caligraphic_Las the internal potential energy of the system, we can view the process of optimizing the loss function through the lens of the principle of minimum potential energy, which states that a stable physical system seeks to establish an equilibrium state that minimizes its potential energyEpsubscript𝐸𝑝E_{p}italic_E start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPTsuch that when an infinitesimal variation from such position involves no change in energyδ⁢Ep=0𝛿subscript𝐸𝑝0\delta E_{p}=0italic_δ italic_E start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT = 0.

That is to say, if we consider the loss function as the system internal energy, then the process of optimizing the loss function, from a physics perspective, is equivalent to the process of the system minimizing its internal energy. This also shows the reasonableness of viewing the ML system as a physical system that conforms to physical laws.

It is important to clarify that the system energy we propose is fundamentally different from the energy in EBMs. Within the EBM framework, both the loss function and the energy function coexist. The energy function is minimized by the inference process, while the loss functional is minimized by the learning process. Differently, in our Type I state phase transition, as energy is released, the system parameters gradually change, while the data remains constant (see Figures1and2). This process corresponds to the traditional learning process. In our theoretical framework, we do not distinguish between the loss function and the energy function.

2.3System Entropy

Discrete Entropy and Differential Entropy.

In this section, let us take a deep dive into the entropy changes of the Type I state system. Given a distribution in either parameter or data space, how can we determine the entropy of all particles? One direct approach is to segment the space into grids, and assess the probability of particles within each grid. Since the distribution for parameter initialization in parameter space and data spatial distribution derived from models are generally continuous, we can calculate the differential entropy of the system. However, given that the particle count in a ML system is always finite, it appears we should evaluate the particle entropy using discrete entropy. In this section, let us investigate the connection between differential entropy and discrete entropy, and confirm how to calculate entropy change in a ML system.

Assume there areN𝑁Nitalic_Nparticles in the parameter space{𝝁}𝝁\{\boldsymbol{\mu}\}{ bold_italic_μ }, and the particles are independent of each other. Therefore, the entropy of the particles is the linear superposition of the entropy contributed by each particle. To calculate the contribution of a single particle, assume the parameter space is divided into a series of grids with the volume of thei−limit-from𝑖i-italic_i -th grid beingΔisubscriptΔ𝑖\Delta_{i}roman_Δ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT, and the probability of the particle in that grid ispisubscript𝑝𝑖p_{i}italic_p start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT, so the discrete entropy of the particle is

Sdiscrete=−∑ipi⁢log⁡pi.subscript𝑆discretesubscript𝑖subscript𝑝𝑖subscript𝑝𝑖S_{\rm discrete}=-\sum_{i}p_{i}\log p_{i}.italic_S start_POSTSUBSCRIPT roman_discrete end_POSTSUBSCRIPT = - ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_p start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT roman_log italic_p start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT .(2.23)Assume the probability distribution of the particle isf⁢(𝝁)𝑓𝝁f({\boldsymbol{\mu}})italic_f ( bold_italic_μ ), we havepi=f⁢(𝝁)⁢Δisubscript𝑝𝑖𝑓𝝁subscriptΔ𝑖p_{i}=f({\boldsymbol{\mu}})\Delta_{i}italic_p start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT = italic_f ( bold_italic_μ ) roman_Δ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT, and the above equation becomesSdiscrete=−∑if⁢(𝝁i)⁢Δi⁢log⁡[f⁢(𝝁i)⁢Δi]subscript𝑆discretesubscript𝑖𝑓subscript𝝁𝑖subscriptΔ𝑖𝑓subscript𝝁𝑖subscriptΔ𝑖S_{\rm discrete}=-\sum_{i}f({\boldsymbol{\mu}_{i}})\Delta_{i}\log[f({% \boldsymbol{\mu}_{i}})\Delta_{i}]italic_S start_POSTSUBSCRIPT roman_discrete end_POSTSUBSCRIPT = - ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_f ( bold_italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) roman_Δ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT roman_log [ italic_f ( bold_italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) roman_Δ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ], and for allΔi→δ→0→subscriptΔ𝑖𝛿→0\Delta_{i}\rightarrow\delta\rightarrow 0roman_Δ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT → italic_δ → 0, We have obtained the connection between discrete and differential entropy as

Sdiscretesubscript𝑆discrete\displaystyle S_{\rm discrete}italic_S start_POSTSUBSCRIPT roman_discrete end_POSTSUBSCRIPT=\displaystyle==−∫f⁢(𝝁)⁢log⁡[f⁢(𝝁)]⁢𝑑𝝁−log⁡δ𝑓𝝁𝑓𝝁differential-d𝝁𝛿\displaystyle-\int f({\boldsymbol{\mu}})\log[f({\boldsymbol{\mu}})]d{% \boldsymbol{\mu}}-\log\delta- ∫ italic_f ( bold_italic_μ ) roman_log [ italic_f ( bold_italic_μ ) ] italic_d bold_italic_μ - roman_log italic_δ(2.24)=\displaystyle==Sdiff−log⁡δ,subscript𝑆diff𝛿\displaystyle S_{\rm diff}-\log\delta,italic_S start_POSTSUBSCRIPT roman_diff end_POSTSUBSCRIPT - roman_log italic_δ ,(2.25)whereSdiffsubscript𝑆diffS_{\rm diff}italic_S start_POSTSUBSCRIPT roman_diff end_POSTSUBSCRIPTis differential entropy for the distributionf⁢(𝝁)𝑓𝝁f({\boldsymbol{\mu}})italic_f ( bold_italic_μ ). Thus, the entropy forN𝑁Nitalic_Nparticles is

S0=N⁢Sdiff−N⁢log⁡δ.subscript𝑆0𝑁subscript𝑆diff𝑁𝛿S_{0}=NS_{\rm diff}-N\log\delta.italic_S start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT = italic_N italic_S start_POSTSUBSCRIPT roman_diff end_POSTSUBSCRIPT - italic_N roman_log italic_δ .(2.26)On the other hand, the energy possessed byN𝑁Nitalic_Nparticles (before phase transition) is

Einitialsubscript𝐸initial\displaystyle E_{\rm initial}italic_E start_POSTSUBSCRIPT roman_initial end_POSTSUBSCRIPT=\displaystyle==∑iE⁢(𝝁i)⁢ni⁢Δi=∑iE⁢(𝝁)⁢(N⁢f𝝁⁢Δ)subscript𝑖𝐸subscript𝝁𝑖subscript𝑛𝑖subscriptΔ𝑖subscript𝑖𝐸𝝁𝑁subscript𝑓𝝁Δ\displaystyle\sum_{i}E({\boldsymbol{\mu}}_{i})n_{i}\Delta_{i}=\sum_{i}E({% \boldsymbol{\mu}})(Nf_{\boldsymbol{\mu}}\Delta)∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_E ( bold_italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) italic_n start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT roman_Δ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT = ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_E ( bold_italic_μ ) ( italic_N italic_f start_POSTSUBSCRIPT bold_italic_μ end_POSTSUBSCRIPT roman_Δ )(2.27)→→\displaystyle\rightarrow→N⁢∫E⁢(𝝁)⁢f⁢(𝝁)⁢𝑑𝝁.𝑁𝐸𝝁𝑓𝝁differential-d𝝁\displaystyle N\int E({\boldsymbol{\mu}})f({\boldsymbol{\mu}})d{\boldsymbol{% \mu}}.italic_N ∫ italic_E ( bold_italic_μ ) italic_f ( bold_italic_μ ) italic_d bold_italic_μ .After the phase transition, the total energy is assumed to beN⁢Ef𝑁subscript𝐸𝑓NE_{f}italic_N italic_E start_POSTSUBSCRIPT italic_f end_POSTSUBSCRIPTand the final discrete entropy drops to zero, so we have the temperature

T=N⁢∫E⁢(𝝁)⁢f⁢(𝝁)⁢𝑑𝝁−N⁢EfN⁢Sdiff−N⁢log⁡δ=∫E⁢(𝝁)⁢f⁢(𝝁)⁢𝑑𝝁−EfSdiff−log⁡δ,𝑇𝑁𝐸𝝁𝑓𝝁differential-d𝝁𝑁subscript𝐸𝑓𝑁subscript𝑆diff𝑁𝛿𝐸𝝁𝑓𝝁differential-d𝝁subscript𝐸𝑓subscript𝑆diff𝛿T=\frac{N\int E({\boldsymbol{\mu}})f({\boldsymbol{\mu}})d{\boldsymbol{\mu}}-NE% _{f}}{NS_{\rm diff}-N\log\delta}=\frac{\int E({\boldsymbol{\mu}})f({% \boldsymbol{\mu}})d{\boldsymbol{\mu}}-E_{f}}{S_{\rm diff}-\log\delta},italic_T = divide start_ARG italic_N ∫ italic_E ( bold_italic_μ ) italic_f ( bold_italic_μ ) italic_d bold_italic_μ - italic_N italic_E start_POSTSUBSCRIPT italic_f end_POSTSUBSCRIPT end_ARG start_ARG italic_N italic_S start_POSTSUBSCRIPT roman_diff end_POSTSUBSCRIPT - italic_N roman_log italic_δ end_ARG = divide start_ARG ∫ italic_E ( bold_italic_μ ) italic_f ( bold_italic_μ ) italic_d bold_italic_μ - italic_E start_POSTSUBSCRIPT italic_f end_POSTSUBSCRIPT end_ARG start_ARG italic_S start_POSTSUBSCRIPT roman_diff end_POSTSUBSCRIPT - roman_log italic_δ end_ARG ,(2.28)The termlog⁡δ→−∞→𝛿\log\delta\rightarrow-\inftyroman_log italic_δ → - ∞forδ→0→𝛿0\delta\rightarrow 0italic_δ → 0is trivial. In AppendixB, we discuss from a higher perspective thatlog⁡δ𝛿\log\deltaroman_log italic_δrepresents the dimensional collapse of the system, andSdiffsubscript𝑆diffS_{\rm diff}italic_S start_POSTSUBSCRIPT roman_diff end_POSTSUBSCRIPTshows the real entropy change for Type I state phase transition. Therefore, we can use the different entropy as the indicator of entropy change, and write

T=Ef−∫E⁢(𝝁)⁢f⁢(𝝁)⁢𝑑𝝁∫f⁢(𝝁)⁢log⁡[f⁢(𝝁)]⁢𝑑𝝁,𝑇subscript𝐸𝑓𝐸𝝁𝑓𝝁differential-d𝝁𝑓𝝁𝑓𝝁differential-d𝝁T=\frac{E_{f}-\int E({\boldsymbol{\mu}})f({\boldsymbol{\mu}})d{\boldsymbol{\mu% }}}{\int f({\boldsymbol{\mu}})\log[f({\boldsymbol{\mu}})]d{\boldsymbol{\mu}}},italic_T = divide start_ARG italic_E start_POSTSUBSCRIPT italic_f end_POSTSUBSCRIPT - ∫ italic_E ( bold_italic_μ ) italic_f ( bold_italic_μ ) italic_d bold_italic_μ end_ARG start_ARG ∫ italic_f ( bold_italic_μ ) roman_log [ italic_f ( bold_italic_μ ) ] italic_d bold_italic_μ end_ARG ,(2.29)as shown in Equation (2.7) from Section2.1.

Parameter Entropy and Data Entropy.

Let us distinguish between the concepts of parameter entropy and data entropy. The parameter entropy, denoted asSparametersubscript𝑆parameterS_{\rm parameter}italic_S start_POSTSUBSCRIPT roman_parameter end_POSTSUBSCRIPT, arises from the randomness of parameter initialization, while the data entropySdatasubscript𝑆dataS_{\rm data}italic_S start_POSTSUBSCRIPT roman_data end_POSTSUBSCRIPT, originates from the randomness of the system data. For a complete ML system, the system entropySsyssubscript𝑆sysS_{\rm sys}italic_S start_POSTSUBSCRIPT roman_sys end_POSTSUBSCRIPThas

Ssys=Sparameter+Sdata.subscript𝑆syssubscript𝑆parametersubscript𝑆dataS_{\rm sys}=S_{\rm parameter}+S_{\rm data}.italic_S start_POSTSUBSCRIPT roman_sys end_POSTSUBSCRIPT = italic_S start_POSTSUBSCRIPT roman_parameter end_POSTSUBSCRIPT + italic_S start_POSTSUBSCRIPT roman_data end_POSTSUBSCRIPT .(2.30) For the Type I state, given that the data remains constant with a fixedSdatasubscript𝑆dataS_{\rm data}italic_S start_POSTSUBSCRIPT roman_data end_POSTSUBSCRIPT, our primary focus is on the entropy contributed by parameter initialization, as described by Equation (2.25). Conversely, for the Type II state, since the parameters are already optimized with a fixedSparametersubscript𝑆parameterS_{\rm parameter}italic_S start_POSTSUBSCRIPT roman_parameter end_POSTSUBSCRIPT, butSdatasubscript𝑆dataS_{\rm data}italic_S start_POSTSUBSCRIPT roman_data end_POSTSUBSCRIPTvaries due to data shifts, our attention is solely on data entropy, which can be obtained by Equations (2.8) and (2.10).

2.4Non-Boltzmann Distribution

In much of the literature about energy-based models (EBMs), energy and probability distribution are linked together. In classical statistical, there is a relation between probabilityP⁢(x)𝑃𝑥P(x)italic_P ( italic_x )and energyE𝝁subscript𝐸𝝁E_{\boldsymbol{\mu}}italic_E start_POSTSUBSCRIPT bold_italic_μ end_POSTSUBSCRIPT, whereP⁢(x)𝑃𝑥P(x)italic_P ( italic_x )is proportional toexp⁡[−β⁢E𝝁⁢(x)]𝛽subscript𝐸𝝁𝑥\exp[-\beta E_{\boldsymbol{\mu}}(x)]roman_exp [ - italic_β italic_E start_POSTSUBSCRIPT bold_italic_μ end_POSTSUBSCRIPT ( italic_x ) ]withβ=1/T𝛽1𝑇\beta=1/Titalic_β = 1 / italic_T, andT𝑇Titalic_Tbeing the temperature of the system. After normalization, we can get

P⁢(x)=exp⁡[−E𝝁⁢(x)/T]∫x∈𝒳exp⁡[−E𝝁⁢(x)/T]⁢𝑑x𝑃𝑥subscript𝐸𝝁𝑥𝑇subscript𝑥𝒳subscript𝐸𝝁𝑥𝑇differential-d𝑥P(x)=\frac{\exp[-E_{\boldsymbol{\mu}}(x)/T]}{\int_{x\in\mathcal{X}}\exp[-E_{% \boldsymbol{\mu}}(x)/T]dx}italic_P ( italic_x ) = divide start_ARG roman_exp [ - italic_E start_POSTSUBSCRIPT bold_italic_μ end_POSTSUBSCRIPT ( italic_x ) / italic_T ] end_ARG start_ARG ∫ start_POSTSUBSCRIPT italic_x ∈ caligraphic_X end_POSTSUBSCRIPT roman_exp [ - italic_E start_POSTSUBSCRIPT bold_italic_μ end_POSTSUBSCRIPT ( italic_x ) / italic_T ] italic_d italic_x end_ARG(2.31)or the conditional probability distribution

P⁢(y|x)=exp⁡[−E𝝁⁢(x,y)/T]∫y∈𝒴exp⁡[−E𝝁⁢(x,y′)/T]⁢𝑑y′.𝑃conditional𝑦𝑥subscript𝐸𝝁𝑥𝑦𝑇subscript𝑦𝒴subscript𝐸𝝁𝑥superscript𝑦′𝑇differential-dsuperscript𝑦′P(y|x)=\frac{\exp[-E_{\boldsymbol{\mu}}(x,y)/T]}{\int_{y\in\mathcal{Y}}\exp[-E% _{\boldsymbol{\mu}}(x,y^{\prime})/T]dy^{\prime}}.italic_P ( italic_y | italic_x ) = divide start_ARG roman_exp [ - italic_E start_POSTSUBSCRIPT bold_italic_μ end_POSTSUBSCRIPT ( italic_x , italic_y ) / italic_T ] end_ARG start_ARG ∫ start_POSTSUBSCRIPT italic_y ∈ caligraphic_Y end_POSTSUBSCRIPT roman_exp [ - italic_E start_POSTSUBSCRIPT bold_italic_μ end_POSTSUBSCRIPT ( italic_x , italic_y start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ) / italic_T ] italic_d italic_y start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT end_ARG .(2.32) Equations (2.31) and (2.32) have also been used in EBMs withT𝑇Titalic_Tas a scalar parameter for the models. However, we must point out that equations (2.31) and (2.32) do not always hold true, even within the realm of statistical mechanics. In AppendixC, we discusses the cases ofnon-Boltzmann distributions, which can also be referred to asgeneralized Boltzmann distributions. The probability distribution also depends on the non-thermodynamic properties of the system, such as gravity and the choice of reference frame. For most thermodynamic systems, the generalized Boltzmann distribution is written as

P⁢(x)=f⁢(𝒢i)⁢exp⁡[−E𝝁⁢(x)/T]∫x∈𝒳f⁢(𝒢i)⁢exp⁡[−E𝝁⁢(x)/T]⁢𝑑x,𝑃𝑥𝑓subscript𝒢𝑖subscript𝐸𝝁𝑥𝑇subscript𝑥𝒳𝑓subscript𝒢𝑖subscript𝐸𝝁𝑥𝑇differential-d𝑥P(x)=\frac{f(\mathcal{G}_{i})\exp[-E_{\boldsymbol{\mu}}(x)/T]}{\int_{x\in% \mathcal{X}}f(\mathcal{G}_{i})\exp[-E_{\boldsymbol{\mu}}(x)/T]dx},italic_P ( italic_x ) = divide start_ARG italic_f ( caligraphic_G start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) roman_exp [ - italic_E start_POSTSUBSCRIPT bold_italic_μ end_POSTSUBSCRIPT ( italic_x ) / italic_T ] end_ARG start_ARG ∫ start_POSTSUBSCRIPT italic_x ∈ caligraphic_X end_POSTSUBSCRIPT italic_f ( caligraphic_G start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) roman_exp [ - italic_E start_POSTSUBSCRIPT bold_italic_μ end_POSTSUBSCRIPT ( italic_x ) / italic_T ] italic_d italic_x end_ARG ,(2.33)wheref⁢(𝒢)𝑓𝒢f(\mathcal{G})italic_f ( caligraphic_G )is a gravity-like factor to shape the probability.

Hence, initiating from the classical Boltzmann distribution and trying to forge a link between energy through probability equations (2.31) and (2.32) may not be the most effective approach. We would prefer to calculate the temperature of the system from first principles, specifically by assessing the changes in the system’s energy and entropy. In AppendixD, we briefly review research from the literature on the Lorentz transformation of temperature. In general, the fundamental thermodynamic equations are covariant in any reference frame, and the relationT=∂U/∂S𝑇𝑈𝑆T=\partial U/\partial Sitalic_T = ∂ italic_U / ∂ italic_S(equation1.5) always holds. The principle is more fundamental than the Boltzmann distribution.

2.5ML vs Physics Perspective, and the Following Content

Based on the discussions above in Section2, we establish an analogy between thermodynamic systems and ML systems. Table1provides the terminology descriptions of various concepts in ML from a physics perspective. Throughout this paper, we will frequently use various physics terms. In this paper, we regard the terms from both the ML and physics perspectives as interchangeable and mutually equivalent.

ML PerspectivePhysics PerspectiveSection(s)ML systemThermodynamic systemSection1ML system temperaturePhysical temperatureSection1Information entropyPhysical entropySection1, Section2.3Differential vs. discrete entropyDimension collapseSection2.3, AppendixBModel parameter initializationType I state particlesSection2.1Model trainingType I state phase transitionSection2.1System data shiftType II state evolutionSection2.1Loss functionInternal potential energySection2.2Loss functions optimizationMinimum potential energySection2.2Parameter entropyType I state entropySection2.3Data entropyType II state entropySection2.3Model retrainingThermodynamic equilibriumSection3.6Entire process of ML (re)training cyclesUnified Scenario of Type I & II statesAppendixANeural networkComplex heat engineSection5.3Tanh and Sigmoid activationLow work efficiency (Type I heat engine)Section5.3ReLU activationHigh work efficiency (Type II heat engine)Section5.3Table 1:A comparison of various terms from the ML perspective and the physics perspective. The last three pairs of terms are from the following Sections. The ML terms and physics terms are used interchangeable in this paper.Next, let us calculate the temperature for a series of common machine learning systems. Based on the systems, they can be categorized as follows:

• •Parameter Initialization:The most common ways to initialize (model) parameters of the system are through normal distribution and uniform distribution.
• •Energy Forms:Common energy forms, as mentioned in Section2.2, are borrowed from ML loss functions, which include Linear Regression (MSE, MAE, etc.), Logistic Regression (Cross Entropy), and original energy forms adjusted byL1subscript𝐿1L_{1}italic_L start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPTandL2subscript𝐿2L_{2}italic_L start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPTregularization.

Table 2:The temperatures of various ML systems. The energy forms of the systems include linear regression with MSE and MAE, logistic regression with Cross Entropy, and neural networks with MSE. The initial parameter distributions for the systems are either normal or uniform. We employed either analytical method or asymptotic approximations to investigate the ML systems and their temperatures.Henceforce, the remainder of this paper will discuss the temperature of various ML systems based on the two aforementioned types of parameter initialization and various energy forms. In particular, we focus on the system temperature during the phase transition process of system State I. Table2gives the various systems we will study. Note that if the study “Method” in Table2is “analytic”, it means that the we can derive the system temperature analytically, while “asymptotic” means that the temperature does not have an analytical solution, and we consider the extreme cases, such as the standard deviation→∞→absent\rightarrow\infty→ ∞for the normal distribution, or the range of the uniform distribution expands to infinity in order to obtain an approximate solution for the temperature. The approximated asymptotic solutions still have clear meanings for the ML systems. In this paper we only consider analytical and asymptotic solutions, and do not expand into numerical solutions and simulations.

3Linear Regression with MSE

3.1Parameter Initialization: Normal Distribution

Assuming a ML system has an initial Type I state with set of parameters{𝝁}𝝁\{\boldsymbol{\mu}\}{ bold_italic_μ }withK𝐾Kitalic_Kdimensions, and the parameters are initialized by normal distribution. The differential entropy of the multivariate Gaussian is

S=12⁢ln⁡|Σ|+K2⁢[1+ln⁡(2⁢π)],𝑆12Σ𝐾2delimited-[]12𝜋S=\frac{1}{2}\ln|\Sigma|+\frac{K}{2}[1+\ln(2\pi)],italic_S = divide start_ARG 1 end_ARG start_ARG 2 end_ARG roman_ln | roman_Σ | + divide start_ARG italic_K end_ARG start_ARG 2 end_ARG [ 1 + roman_ln ( 2 italic_π ) ] ,(3.1)In our case we assume each dimension has independent distribution, i.e.,

Σ=[σ12σ22σ32…σK2],Σdelimited-[]superscriptsubscript𝜎12missing-subexpressionmissing-subexpressionmissing-subexpressionmissing-subexpressionmissing-subexpressionsuperscriptsubscript𝜎22missing-subexpressionmissing-subexpressionmissing-subexpressionmissing-subexpressionmissing-subexpressionsuperscriptsubscript𝜎32missing-subexpressionmissing-subexpressionmissing-subexpressionmissing-subexpressionmissing-subexpression…missing-subexpressionmissing-subexpressionmissing-subexpressionmissing-subexpressionmissing-subexpressionsuperscriptsubscript𝜎𝐾2\Sigma=\left[\begin{array}[]{ccccc}\sigma_{1}^{2}&&&&\\ &\sigma_{2}^{2}&&&\\ &&\sigma_{3}^{2}&&\\ &&&...&\\ &&&&\sigma_{K}^{2}\end{array}\right],roman_Σ = [ start_ARRAY start_ROW start_CELL italic_σ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_CELL start_CELL end_CELL start_CELL end_CELL start_CELL end_CELL start_CELL end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL italic_σ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_CELL start_CELL end_CELL start_CELL end_CELL start_CELL end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL end_CELL start_CELL italic_σ start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_CELL start_CELL end_CELL start_CELL end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL end_CELL start_CELL end_CELL start_CELL … end_CELL start_CELL end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL end_CELL start_CELL end_CELL start_CELL end_CELL start_CELL italic_σ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_CELL end_ROW end_ARRAY ] ,From Equation (3.1) the initial entropy of the system becomes

S0=ln⁡(σ1⁢σ2⁢…⁢σK)+K2⁢[1+ln⁡(2⁢π)].subscript𝑆0subscript𝜎1subscript𝜎2…subscript𝜎𝐾𝐾2delimited-[]12𝜋S_{0}=\ln(\sigma_{1}\sigma_{2}...\sigma_{K})+\frac{K}{2}[1+\ln(2\pi)].italic_S start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT = roman_ln ( italic_σ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT italic_σ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT … italic_σ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT ) + divide start_ARG italic_K end_ARG start_ARG 2 end_ARG [ 1 + roman_ln ( 2 italic_π ) ] .(3.2)

3.1.12D Linear Regression

Let us start from 2-dimensional linear regression with the loss function MSE1n⁢∑(μ1⁢xi+μ2−yi)21𝑛superscriptsubscript𝜇1subscript𝑥𝑖subscript𝜇2subscript𝑦𝑖2\frac{1}{n}\sum(\mu_{1}x_{i}+\mu_{2}-y_{i})^{2}divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ ( italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT + italic_μ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT, wheren𝑛nitalic_nis the number of data points{xi,yi}subscript𝑥𝑖subscript𝑦𝑖\{x_{i},y_{i}\}{ italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT }(see Equation [2.16]). As we initialize the parameter set with normal distribution, the average energy of the state is

⟨E0⟩delimited-⟨⟩subscript𝐸0\displaystyle\langle E_{0}\rangle⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩=\displaystyle==∫1n⁢∑in(μ1⁢xi+μ2−yi)2⁢12⁢π⁢σ1⁢e−μ122⁢σ12⁢12⁢π⁢σ2⁢e−μ222⁢σ22⁢d⁢μ1⁢d⁢μ21𝑛superscriptsubscript𝑖𝑛superscriptsubscript𝜇1subscript𝑥𝑖subscript𝜇2subscript𝑦𝑖212𝜋subscript𝜎1superscriptesuperscriptsubscript𝜇122superscriptsubscript𝜎1212𝜋subscript𝜎2superscriptesuperscriptsubscript𝜇222superscriptsubscript𝜎22𝑑subscript𝜇1𝑑subscript𝜇2\displaystyle\int\frac{1}{n}\sum_{i}^{n}(\mu_{1}x_{i}+\mu_{2}-y_{i})^{2}\frac{% 1}{\sqrt{2\pi}\sigma_{1}}\textrm{e}^{-\frac{\mu_{1}^{2}}{2\sigma_{1}^{2}}}% \frac{1}{\sqrt{2\pi}\sigma_{2}}\textrm{e}^{-\frac{\mu_{2}^{2}}{2\sigma_{2}^{2}% }}d\mu_{1}d\mu_{2}∫ divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ( italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT + italic_μ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT divide start_ARG 1 end_ARG start_ARG square-root start_ARG 2 italic_π end_ARG italic_σ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_ARG e start_POSTSUPERSCRIPT - divide start_ARG italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_σ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG end_POSTSUPERSCRIPT divide start_ARG 1 end_ARG start_ARG square-root start_ARG 2 italic_π end_ARG italic_σ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_ARG e start_POSTSUPERSCRIPT - divide start_ARG italic_μ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_σ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG end_POSTSUPERSCRIPT italic_d italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT italic_d italic_μ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT(3.3)=\displaystyle==∫σ222⁢π⁢n⁢𝑑μ1⁢e−μ122⁢∑∫[μ2+(μ1⁢σ1σ2⁢xi−yiσ2)]2⁢e−μ222⁢𝑑μ2superscriptsubscript𝜎222𝜋𝑛differential-dsubscript𝜇1superscriptesuperscriptsubscript𝜇122superscriptdelimited-[]subscript𝜇2subscript𝜇1subscript𝜎1subscript𝜎2subscript𝑥𝑖subscript𝑦𝑖subscript𝜎22superscriptesuperscriptsubscript𝜇222differential-dsubscript𝜇2\displaystyle\int\frac{\sigma_{2}^{2}}{2\pi n}d\mu_{1}\textrm{e}^{-\frac{\mu_{% 1}^{2}}{2}}\sum\int\left[\mu_{2}+\left(\mu_{1}\frac{\sigma_{1}}{\sigma_{2}}x_{% i}-\frac{y_{i}}{\sigma_{2}}\right)\right]^{2}\textrm{e}^{-\frac{\mu_{2}^{2}}{2% }}d\mu_{2}∫ divide start_ARG italic_σ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_π italic_n end_ARG italic_d italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT e start_POSTSUPERSCRIPT - divide start_ARG italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 end_ARG end_POSTSUPERSCRIPT ∑ ∫ [ italic_μ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT + ( italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT divide start_ARG italic_σ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_ARG start_ARG italic_σ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_ARG italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT - divide start_ARG italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG start_ARG italic_σ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_ARG ) ] start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT e start_POSTSUPERSCRIPT - divide start_ARG italic_μ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 end_ARG end_POSTSUPERSCRIPT italic_d italic_μ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT=\displaystyle==∑∫σ222⁢π⁢n⁢e−μ122⁢[(μ1⁢σ1σ2⁢xi−yiσ2)2+1]⁢𝑑μ1superscriptsubscript𝜎222𝜋𝑛superscriptesuperscriptsubscript𝜇122delimited-[]superscriptsubscript𝜇1subscript𝜎1subscript𝜎2subscript𝑥𝑖subscript𝑦𝑖subscript𝜎221differential-dsubscript𝜇1\displaystyle\sum\int\frac{\sigma_{2}^{2}}{\sqrt{2\pi}n}\textrm{e}^{-\frac{\mu% _{1}^{2}}{2}}\left[\left(\mu_{1}\frac{\sigma_{1}}{\sigma_{2}}x_{i}-\frac{y_{i}% }{\sigma_{2}}\right)^{2}+1\right]d\mu_{1}∑ ∫ divide start_ARG italic_σ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG square-root start_ARG 2 italic_π end_ARG italic_n end_ARG e start_POSTSUPERSCRIPT - divide start_ARG italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 end_ARG end_POSTSUPERSCRIPT [ ( italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT divide start_ARG italic_σ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_ARG start_ARG italic_σ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_ARG italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT - divide start_ARG italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG start_ARG italic_σ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_ARG ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + 1 ] italic_d italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT=\displaystyle==∑σ12⁢xi2n⁢[1+yi2xi2⁢σ12+σ22σ12⁢xi2]superscriptsubscript𝜎12superscriptsubscript𝑥𝑖2𝑛delimited-[]1superscriptsubscript𝑦𝑖2superscriptsubscript𝑥𝑖2superscriptsubscript𝜎12superscriptsubscript𝜎22superscriptsubscript𝜎12superscriptsubscript𝑥𝑖2\displaystyle\sum\frac{\sigma_{1}^{2}x_{i}^{2}}{n}\left[1+\frac{y_{i}^{2}}{x_{% i}^{2}\sigma_{1}^{2}}+\frac{\sigma_{2}^{2}}{\sigma_{1}^{2}x_{i}^{2}}\right]∑ divide start_ARG italic_σ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG italic_n end_ARG [ 1 + divide start_ARG italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_σ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + divide start_ARG italic_σ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG italic_σ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ]=\displaystyle==∑σ12⁢xi2+yi2+σ22nsuperscriptsubscript𝜎12superscriptsubscript𝑥𝑖2superscriptsubscript𝑦𝑖2superscriptsubscript𝜎22𝑛\displaystyle\sum\frac{\sigma_{1}^{2}x_{i}^{2}+y_{i}^{2}+\sigma_{2}^{2}}{n}∑ divide start_ARG italic_σ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + italic_σ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG italic_n end_ARG=\displaystyle==σ12⁢X2¯+Y2¯+σ22,superscriptsubscript𝜎12¯superscript𝑋2¯superscript𝑌2superscriptsubscript𝜎22\displaystyle\sigma_{1}^{2}\overline{X^{2}}+\overline{Y^{2}}+\sigma_{2}^{2},italic_σ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + over¯ start_ARG italic_Y start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + italic_σ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ,whereX2¯=∑xi2/n¯superscript𝑋2superscriptsubscript𝑥𝑖2𝑛\overline{X^{2}}=\sum x_{i}^{2}/nover¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG = ∑ italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT / italic_nandY2¯=∑yi2/n¯superscript𝑌2superscriptsubscript𝑦𝑖2𝑛\overline{Y^{2}}=\sum y_{i}^{2}/nover¯ start_ARG italic_Y start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG = ∑ italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT / italic_nare denoted as the mean of the squared values of the data in the system.

After Type I state phase transition, the internal energy of the system goes to the minimum point that

Efsubscript𝐸𝑓\displaystyle E_{f}italic_E start_POSTSUBSCRIPT italic_f end_POSTSUBSCRIPT=\displaystyle==min⁢[1n⁢∑(μ1⁢xi+μ2−yi)2]mindelimited-[]1𝑛superscriptsubscript𝜇1subscript𝑥𝑖subscript𝜇2subscript𝑦𝑖2\displaystyle\textrm{min}\left[\frac{1}{n}\sum(\mu_{1}x_{i}+\mu_{2}-y_{i})^{2}\right]min [ divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ ( italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT + italic_μ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ](3.4)=\displaystyle==∑(yi−y¯)2n−∑[(xi−x¯)⁢(yi−y¯)]2n⁢(xi−x¯)2superscriptsubscript𝑦𝑖¯𝑦2𝑛superscriptdelimited-[]subscript𝑥𝑖¯𝑥subscript𝑦𝑖¯𝑦2𝑛superscriptsubscript𝑥𝑖¯𝑥2\displaystyle\frac{\sum(y_{i}-\bar{y})^{2}}{n}-\frac{\sum[(x_{i}-\bar{x})(y_{i% }-\bar{y})]^{2}}{n(x_{i}-\bar{x})^{2}}divide start_ARG ∑ ( italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT - over¯ start_ARG italic_y end_ARG ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG italic_n end_ARG - divide start_ARG ∑ [ ( italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT - over¯ start_ARG italic_x end_ARG ) ( italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT - over¯ start_ARG italic_y end_ARG ) ] start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG italic_n ( italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT - over¯ start_ARG italic_x end_ARG ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG=\displaystyle==Var⁢(Y)−Cov⁢(X,Y)2Var⁢(X)Var𝑌Covsuperscript𝑋𝑌2Var𝑋\displaystyle\textrm{Var}(Y)-\frac{\textrm{Cov}(X,Y)^{2}}{\textrm{Var}(X)}Var ( italic_Y ) - divide start_ARG Cov ( italic_X , italic_Y ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG Var ( italic_X ) end_ARG=\displaystyle==(1−ρ2)⁢Var⁢(Y).1superscript𝜌2Var𝑌\displaystyle(1-\rho^{2})\textrm{Var}(Y).( 1 - italic_ρ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ) Var ( italic_Y ) .Thus, the temperature of the system is

T𝑇\displaystyle Titalic_T=\displaystyle==⟨E0⟩−EfS0delimited-⟨⟩subscript𝐸0subscript𝐸𝑓subscript𝑆0\displaystyle\frac{\langle E_{0}\rangle-E_{f}}{S_{0}}divide start_ARG ⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩ - italic_E start_POSTSUBSCRIPT italic_f end_POSTSUBSCRIPT end_ARG start_ARG italic_S start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_ARG(3.5)=\displaystyle==σ12⁢X2¯+Y2¯+σ22−(1−ρ2)⁢Var⁢(Y)1+ln⁡(2⁢π⁢σ1⁢σ2)superscriptsubscript𝜎12¯superscript𝑋2¯superscript𝑌2superscriptsubscript𝜎221superscript𝜌2Var𝑌12𝜋subscript𝜎1subscript𝜎2\displaystyle\frac{\sigma_{1}^{2}\overline{X^{2}}+\overline{Y^{2}}+\sigma_{2}^% {2}-(1-\rho^{2})\textrm{Var}(Y)}{1+\ln(2\pi\sigma_{1}\sigma_{2})}divide start_ARG italic_σ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + over¯ start_ARG italic_Y start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + italic_σ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT - ( 1 - italic_ρ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ) Var ( italic_Y ) end_ARG start_ARG 1 + roman_ln ( 2 italic_π italic_σ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT italic_σ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ) end_ARGEquation (3.5) gives the analytical solution of the system temperature for a linear regression MSE energy form with initial normal distribution. The temperature is a function of data distributionX2¯¯superscript𝑋2\overline{X^{2}}over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG,Y2¯¯superscript𝑌2\overline{Y^{2}}over¯ start_ARG italic_Y start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG, as well as initial parameter distributionσ1subscript𝜎1\sigma_{1}italic_σ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPTandσ2subscript𝜎2\sigma_{2}italic_σ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT.

For the asymptotic caseσ1,σ2→σ→∞→subscript𝜎1subscript𝜎2𝜎→\sigma_{1},\sigma_{2}\rightarrow\sigma\rightarrow\inftyitalic_σ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_σ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT → italic_σ → ∞, where the parameter initialization distribution has the maximum randomness, we obtain the asymptotic property of temperature that

T∼σ2+σ2⁢X2¯+Y2¯2⁢ln⁡σ∼(σ22⁢ln⁡σ)⁢(1+X2¯).similar-to𝑇superscript𝜎2superscript𝜎2¯superscript𝑋2¯superscript𝑌22𝜎similar-tosuperscript𝜎22𝜎1¯superscript𝑋2T\sim\frac{\sigma^{2}+\sigma^{2}\overline{X^{2}}+\overline{Y^{2}}}{2\ln\sigma}% \sim\left(\frac{\sigma^{2}}{2\ln\sigma}\right)(1+\overline{X^{2}}).italic_T ∼ divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + over¯ start_ARG italic_Y start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG end_ARG start_ARG 2 roman_ln italic_σ end_ARG ∼ ( divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 roman_ln italic_σ end_ARG ) ( 1 + over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ) .(3.6)

3.1.2High-dimensional Linear Regression

The 2D cases in Section3.1.1can be extended to higher dimensions. The energy form in high dimensions is written as

MSE=1n⁢∑i(∑j=1K−1μj⁢xi⁢j+μK−yi)2,MSE1𝑛subscript𝑖superscriptsuperscriptsubscript𝑗1𝐾1subscript𝜇𝑗subscript𝑥𝑖𝑗subscript𝜇𝐾subscript𝑦𝑖2{\rm MSE}=\frac{1}{n}\sum_{i}\left(\sum_{j=1}^{K-1}\mu_{j}x_{ij}+\mu_{K}-y_{i}% \right)^{2},roman_MSE = divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT + italic_μ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ,(3.7)whereK𝐾Kitalic_Kis the dimension of the parameter space, andxi⁢jsubscript𝑥𝑖𝑗x_{ij}italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPTis thej𝑗jitalic_j-th component of thei𝑖iitalic_i-th data point𝒙isubscript𝒙𝑖\boldsymbol{x}_{i}bold_italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT, Based on this, we can obtain the initial energy of the system

⟨E0⟩delimited-⟨⟩subscript𝐸0\displaystyle\langle E_{0}\rangle⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩=\displaystyle==∫1n⁢∑i(∑j=1K−1μj⁢xi⁢j+μK−yi)2⁢1(2⁢π)K/2⁢∏j=1Kσj⁢e−∑j=1Kμj22⁢σj2⁢∏j=1Kd⁢μj1𝑛subscript𝑖superscriptsuperscriptsubscript𝑗1𝐾1subscript𝜇𝑗subscript𝑥𝑖𝑗subscript𝜇𝐾subscript𝑦𝑖21superscript2𝜋𝐾2superscriptsubscriptproduct𝑗1𝐾subscript𝜎𝑗superscriptesuperscriptsubscript𝑗1𝐾superscriptsubscript𝜇𝑗22superscriptsubscript𝜎𝑗2superscriptsubscriptproduct𝑗1𝐾𝑑subscript𝜇𝑗\displaystyle\int\frac{1}{n}\sum_{i}\left(\sum_{j=1}^{K-1}\mu_{j}x_{ij}+\mu_{K% }-y_{i}\right)^{2}\frac{1}{(2\pi)^{K/2}\prod_{j=1}^{K}\sigma_{j}}\textrm{e}^{-% \sum_{j=1}^{K}\frac{\mu_{j}^{2}}{2\sigma_{j}^{2}}}\prod_{j=1}^{K}d\mu_{j}∫ divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT + italic_μ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT divide start_ARG 1 end_ARG start_ARG ( 2 italic_π ) start_POSTSUPERSCRIPT italic_K / 2 end_POSTSUPERSCRIPT ∏ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT end_ARG e start_POSTSUPERSCRIPT - ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT divide start_ARG italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG end_POSTSUPERSCRIPT ∏ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT italic_d italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT(3.8)=\displaystyle==∑i1(2⁢π)K/2⁢n⁢∫e−∑j=1Kμj22⁢σj2⁢∏j=1K−1d⁢μj⁢(∑j=1k−1μj⁢σj⁢xi⁢j+μk⁢σK−yi)2subscript𝑖1superscript2𝜋𝐾2𝑛superscriptesuperscriptsubscript𝑗1𝐾superscriptsubscript𝜇𝑗22superscriptsubscript𝜎𝑗2superscriptsubscriptproduct𝑗1𝐾1𝑑subscript𝜇𝑗superscriptsuperscriptsubscript𝑗1𝑘1subscript𝜇𝑗subscript𝜎𝑗subscript𝑥𝑖𝑗subscript𝜇𝑘subscript𝜎𝐾subscript𝑦𝑖2\displaystyle\sum_{i}\frac{1}{(2\pi)^{K/2}n}\int\textrm{e}^{-\sum_{j=1}^{K}% \frac{\mu_{j}^{2}}{2\sigma_{j}^{2}}}\prod_{j=1}^{K-1}d\mu_{j}\left(\sum_{j=1}^% {k-1}\mu_{j}\sigma_{j}x_{ij}+\mu_{k}\sigma_{K}-y_{i}\right)^{2}∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT divide start_ARG 1 end_ARG start_ARG ( 2 italic_π ) start_POSTSUPERSCRIPT italic_K / 2 end_POSTSUPERSCRIPT italic_n end_ARG ∫ e start_POSTSUPERSCRIPT - ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT divide start_ARG italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG end_POSTSUPERSCRIPT ∏ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT italic_d italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT ( ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_k - 1 end_POSTSUPERSCRIPT italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT + italic_μ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT italic_σ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT=\displaystyle==∑1(2⁢π)K−12⁢n⁢∫e−∑j=1K−1μj22⁢σj2⁢∏j=1K−1d⁢μj⁢[σK2+(∑j=1K−1μj⁢σj⁢xi⁢j−yi)2]1superscript2𝜋𝐾12𝑛superscriptesuperscriptsubscript𝑗1𝐾1superscriptsubscript𝜇𝑗22superscriptsubscript𝜎𝑗2superscriptsubscriptproduct𝑗1𝐾1𝑑subscript𝜇𝑗delimited-[]superscriptsubscript𝜎𝐾2superscriptsuperscriptsubscript𝑗1𝐾1subscript𝜇𝑗subscript𝜎𝑗subscript𝑥𝑖𝑗subscript𝑦𝑖2\displaystyle\sum\frac{1}{(2\pi)^{\frac{K-1}{2}}n}\int\textrm{e}^{-\sum_{j=1}^% {K-1}\frac{\mu_{j}^{2}}{2\sigma_{j}^{2}}}\prod_{j=1}^{K-1}d\mu_{j}\left[\sigma% _{K}^{2}+\left(\sum_{j=1}^{K-1}\mu_{j}\sigma_{j}x_{ij}-y_{i}\right)^{2}\right]∑ divide start_ARG 1 end_ARG start_ARG ( 2 italic_π ) start_POSTSUPERSCRIPT divide start_ARG italic_K - 1 end_ARG start_ARG 2 end_ARG end_POSTSUPERSCRIPT italic_n end_ARG ∫ e start_POSTSUPERSCRIPT - ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT divide start_ARG italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG end_POSTSUPERSCRIPT ∏ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT italic_d italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT [ italic_σ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + ( ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ]=\displaystyle==∑1(2⁢π)K−22⁢n⁢∫e−∑j=1K−2μj22⁢σj2⁢∏j=1K−2d⁢μj⁢[σK2+σK−12⁢xi,K−12+(∑j=1K−2μj⁢σj⁢xi⁢j−yi)2]1superscript2𝜋𝐾22𝑛superscriptesuperscriptsubscript𝑗1𝐾2superscriptsubscript𝜇𝑗22superscriptsubscript𝜎𝑗2superscriptsubscriptproduct𝑗1𝐾2𝑑subscript𝜇𝑗delimited-[]superscriptsubscript𝜎𝐾2superscriptsubscript𝜎𝐾12superscriptsubscript𝑥𝑖𝐾12superscriptsuperscriptsubscript𝑗1𝐾2subscript𝜇𝑗subscript𝜎𝑗subscript𝑥𝑖𝑗subscript𝑦𝑖2\displaystyle\sum\frac{1}{(2\pi)^{\frac{K-2}{2}}n}\int\textrm{e}^{-\sum_{j=1}^% {K-2}\frac{\mu_{j}^{2}}{2\sigma_{j}^{2}}}\prod_{j=1}^{K-2}d\mu_{j}\left[\sigma% _{K}^{2}+\sigma_{K-1}^{2}x_{i,K-1}^{2}+\left(\sum_{j=1}^{K-2}\mu_{j}\sigma_{j}% x_{ij}-y_{i}\right)^{2}\right]∑ divide start_ARG 1 end_ARG start_ARG ( 2 italic_π ) start_POSTSUPERSCRIPT divide start_ARG italic_K - 2 end_ARG start_ARG 2 end_ARG end_POSTSUPERSCRIPT italic_n end_ARG ∫ e start_POSTSUPERSCRIPT - ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 2 end_POSTSUPERSCRIPT divide start_ARG italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG end_POSTSUPERSCRIPT ∏ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 2 end_POSTSUPERSCRIPT italic_d italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT [ italic_σ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + italic_σ start_POSTSUBSCRIPT italic_K - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_x start_POSTSUBSCRIPT italic_i , italic_K - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + ( ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 2 end_POSTSUPERSCRIPT italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ]=\displaystyle==…⁢………\displaystyle......… …=\displaystyle==∑i1n⁢[σK2+σK−12⁢xi,K−12+σK−22⁢xi,K−22+…+σ12⁢xi,12+yi2]subscript𝑖1𝑛delimited-[]superscriptsubscript𝜎𝐾2superscriptsubscript𝜎𝐾12superscriptsubscript𝑥𝑖𝐾12superscriptsubscript𝜎𝐾22superscriptsubscript𝑥𝑖𝐾22…superscriptsubscript𝜎12superscriptsubscript𝑥𝑖12superscriptsubscript𝑦𝑖2\displaystyle\sum_{i}\frac{1}{n}\left[\sigma_{K}^{2}+\sigma_{K-1}^{2}x_{i,K-1}% ^{2}+\sigma_{K-2}^{2}x_{i,K-2}^{2}+...+\sigma_{1}^{2}x_{i,1}^{2}+y_{i}^{2}\right]∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT divide start_ARG 1 end_ARG start_ARG italic_n end_ARG [ italic_σ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + italic_σ start_POSTSUBSCRIPT italic_K - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_x start_POSTSUBSCRIPT italic_i , italic_K - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + italic_σ start_POSTSUBSCRIPT italic_K - 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_x start_POSTSUBSCRIPT italic_i , italic_K - 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + … + italic_σ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_x start_POSTSUBSCRIPT italic_i , 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ]=\displaystyle==σK2+σK−12⁢XK−12¯+σK−22⁢XK−22¯+…⁢…+σ22⁢X22¯+σ12⁢X12¯+Y2¯.superscriptsubscript𝜎𝐾2superscriptsubscript𝜎𝐾12¯superscriptsubscript𝑋𝐾12superscriptsubscript𝜎𝐾22¯superscriptsubscript𝑋𝐾22……superscriptsubscript𝜎22¯superscriptsubscript𝑋22superscriptsubscript𝜎12¯superscriptsubscript𝑋12¯superscript𝑌2\displaystyle\sigma_{K}^{2}+\sigma_{K-1}^{2}\overline{X_{K-1}^{2}}+\sigma_{K-2% }^{2}\overline{X_{K-2}^{2}}+......+\sigma_{2}^{2}\overline{X_{2}^{2}}+\sigma_{% 1}^{2}\overline{X_{1}^{2}}+\overline{Y^{2}}.italic_σ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + italic_σ start_POSTSUBSCRIPT italic_K - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_K - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + italic_σ start_POSTSUBSCRIPT italic_K - 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_K - 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + … … + italic_σ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + italic_σ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + over¯ start_ARG italic_Y start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG . On the other hand, after phase transition the system reaches to its lowest energy point that

Ef=min MSE=1n⁢tr⁢{Cx−Cx⁢y⁢Cy−1⁢Cy⁢x},subscript𝐸𝑓min MSE1𝑛trsubscript𝐶𝑥subscript𝐶𝑥𝑦superscriptsubscript𝐶𝑦1subscript𝐶𝑦𝑥E_{f}=\textrm{min\;MSE}=\frac{1}{n}{\rm tr}\{C_{x}-C_{xy}C_{y}^{-1}C_{yx}\},italic_E start_POSTSUBSCRIPT italic_f end_POSTSUBSCRIPT = min MSE = divide start_ARG 1 end_ARG start_ARG italic_n end_ARG roman_tr { italic_C start_POSTSUBSCRIPT italic_x end_POSTSUBSCRIPT - italic_C start_POSTSUBSCRIPT italic_x italic_y end_POSTSUBSCRIPT italic_C start_POSTSUBSCRIPT italic_y end_POSTSUBSCRIPT start_POSTSUPERSCRIPT - 1 end_POSTSUPERSCRIPT italic_C start_POSTSUBSCRIPT italic_y italic_x end_POSTSUBSCRIPT } ,(3.9)whereCxsubscript𝐶𝑥C_{x}italic_C start_POSTSUBSCRIPT italic_x end_POSTSUBSCRIPTandCysubscript𝐶𝑦C_{y}italic_C start_POSTSUBSCRIPT italic_y end_POSTSUBSCRIPTare auto-covariance matrix ofx𝑥xitalic_xandy𝑦yitalic_y, respectively, andCx⁢ysubscript𝐶𝑥𝑦C_{xy}italic_C start_POSTSUBSCRIPT italic_x italic_y end_POSTSUBSCRIPTis cross-covariance matrix betweenx𝑥xitalic_xandy𝑦yitalic_y.

Therefore, the temperature of the system is

T=∑j=1Kσj2⁢Xj2¯+Y2¯−tr⁢{Cx−Cx⁢y⁢Cy−1⁢Cy⁢x}¯∑j=1Kln⁡σj+K2⁢[1+ln⁡(2⁢π)],𝑇superscriptsubscript𝑗1𝐾superscriptsubscript𝜎𝑗2¯superscriptsubscript𝑋𝑗2¯superscript𝑌2¯trsubscript𝐶𝑥subscript𝐶𝑥𝑦superscriptsubscript𝐶𝑦1subscript𝐶𝑦𝑥superscriptsubscript𝑗1𝐾subscript𝜎𝑗𝐾2delimited-[]12𝜋\displaystyle T=\frac{\sum_{j=1}^{K}\sigma_{j}^{2}\overline{X_{j}^{2}}+% \overline{Y^{2}}-\overline{{\rm tr}\{C_{x}-C_{xy}C_{y}^{-1}C_{yx}\}}}{\sum_{j=% 1}^{K}\ln\sigma_{j}+\frac{K}{2}[1+\ln(2\pi)]},italic_T = divide start_ARG ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + over¯ start_ARG italic_Y start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG - over¯ start_ARG roman_tr { italic_C start_POSTSUBSCRIPT italic_x end_POSTSUBSCRIPT - italic_C start_POSTSUBSCRIPT italic_x italic_y end_POSTSUBSCRIPT italic_C start_POSTSUBSCRIPT italic_y end_POSTSUBSCRIPT start_POSTSUPERSCRIPT - 1 end_POSTSUPERSCRIPT italic_C start_POSTSUBSCRIPT italic_y italic_x end_POSTSUBSCRIPT } end_ARG end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT roman_ln italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT + divide start_ARG italic_K end_ARG start_ARG 2 end_ARG [ 1 + roman_ln ( 2 italic_π ) ] end_ARG ,(3.10)where we denotexi⁢K=1subscript𝑥𝑖𝐾1x_{iK}=1italic_x start_POSTSUBSCRIPT italic_i italic_K end_POSTSUBSCRIPT = 1andXK2¯=1¯superscriptsubscript𝑋𝐾21\overline{X_{K}^{2}}=1over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG = 1.

For the asymptotic case{σj}→σ→∞→subscript𝜎𝑗𝜎→\{\sigma_{j}\}\rightarrow\sigma\rightarrow\infty{ italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT } → italic_σ → ∞, we obtain the temperature

T𝑇\displaystyle Titalic_T∼similar-to\displaystyle\sim∼σ2⁢∑j=1KXj2¯K⁢ln⁡σ∼(∑j=1KXj2¯K)⁢(σ2ln⁡σ)similar-tosuperscript𝜎2superscriptsubscript𝑗1𝐾¯superscriptsubscript𝑋𝑗2𝐾𝜎superscriptsubscript𝑗1𝐾¯superscriptsubscript𝑋𝑗2𝐾superscript𝜎2𝜎\displaystyle\frac{\sigma^{2}\sum_{j=1}^{K}\overline{X_{j}^{2}}}{K\ln\sigma}% \sim\left(\frac{\sum_{j=1}^{K}\overline{X_{j}^{2}}}{K}\right)\left(\frac{% \sigma^{2}}{\ln\sigma}\right)divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG end_ARG start_ARG italic_K roman_ln italic_σ end_ARG ∼ ( divide start_ARG ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG end_ARG start_ARG italic_K end_ARG ) ( divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ end_ARG )(3.11)∼similar-to\displaystyle\sim∼X2¯⁢(σ2ln⁡σ).¯superscript𝑋2superscript𝜎2𝜎\displaystyle\overline{X^{2}}\left(\frac{\sigma^{2}}{\ln\sigma}\right).over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ( divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ end_ARG ) .This result combined with Section3.1.1shows that the temperature of a ML system is determined by its data distribution, as well as parameter initialized structure.

3.2Parameter Initialization: Uniform Distribution

If the parameters of the system initial distribution is a uniform distribution, where theK𝐾Kitalic_K-dimensional parameters{𝝁}𝝁\{\boldsymbol{\mu}\}{ bold_italic_μ }distribution satisfies

f⁢(𝝁)=∏k=1K1lk𝑓𝝁superscriptsubscriptproduct𝑘1𝐾1subscript𝑙𝑘f({\boldsymbol{\mu}})=\prod_{k=1}^{K}\frac{1}{l_{k}}italic_f ( bold_italic_μ ) = ∏ start_POSTSUBSCRIPT italic_k = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT divide start_ARG 1 end_ARG start_ARG italic_l start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT end_ARG(3.12)for{−lk/2≤μk<lk/2}k=1,2,3,…,Ksubscriptsubscript𝑙𝑘2subscript𝜇𝑘subscript𝑙𝑘2𝑘123…𝐾\{-l_{k}/2\leq\mu_{k}<l_{k}/2\}_{k=1,2,3,...,K}{ - italic_l start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT / 2 ≤ italic_μ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT < italic_l start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT / 2 } start_POSTSUBSCRIPT italic_k = 1 , 2 , 3 , … , italic_K end_POSTSUBSCRIPT. Thus, the differential entropy of the system initial state is

S=ln⁡(∏k=1Klk).𝑆superscriptsubscriptproduct𝑘1𝐾subscript𝑙𝑘S=\ln\left(\prod_{k=1}^{K}l_{k}\right).italic_S = roman_ln ( ∏ start_POSTSUBSCRIPT italic_k = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT ) .(3.13)

3.2.12D Uniform Distribution

In this special case, the average energy of the state can be calculated by

⟨E0⟩delimited-⟨⟩subscript𝐸0\displaystyle\langle E_{0}\rangle⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩=\displaystyle==∫1n⁢∑i=1n(μ1⁢xi+μ2−yi)2⁢1l⁢1⁢d⁢μ1⁢1l⁢2⁢d⁢μ21𝑛superscriptsubscript𝑖1𝑛superscriptsubscript𝜇1subscript𝑥𝑖subscript𝜇2subscript𝑦𝑖21𝑙1𝑑subscript𝜇11𝑙2𝑑subscript𝜇2\displaystyle\int\frac{1}{n}\sum_{i=1}^{n}(\mu_{1}x_{i}+\mu_{2}-y_{i})^{2}% \frac{1}{l1}d\mu_{1}\frac{1}{l2}d\mu_{2}∫ divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ( italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT + italic_μ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT divide start_ARG 1 end_ARG start_ARG italic_l 1 end_ARG italic_d italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT divide start_ARG 1 end_ARG start_ARG italic_l 2 end_ARG italic_d italic_μ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT(3.14)=\displaystyle==∫−l⁢1/2l⁢1/2∑d⁢μ1n⁢l1⁢l2⁢∫−l⁢2/2l⁢2/2[μ2+(μ1⁢xi−yi)]2⁢𝑑μ2superscriptsubscript𝑙12𝑙12𝑑subscript𝜇1𝑛subscript𝑙1subscript𝑙2superscriptsubscript𝑙22𝑙22superscriptdelimited-[]subscript𝜇2subscript𝜇1subscript𝑥𝑖subscript𝑦𝑖2differential-dsubscript𝜇2\displaystyle\int_{-l1/2}^{l1/2}\sum\frac{d\mu_{1}}{nl_{1}l_{2}}\int_{-l2/2}^{% l2/2}[\mu_{2}+(\mu_{1}x_{i}-y_{i})]^{2}d\mu_{2}∫ start_POSTSUBSCRIPT - italic_l 1 / 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_l 1 / 2 end_POSTSUPERSCRIPT ∑ divide start_ARG italic_d italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_ARG start_ARG italic_n italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_ARG ∫ start_POSTSUBSCRIPT - italic_l 2 / 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_l 2 / 2 end_POSTSUPERSCRIPT [ italic_μ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT + ( italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) ] start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_d italic_μ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT=\displaystyle==∑∫−l1/2l1/2xi2n⁢l1⁢[l2212⁢xi2+(μ1−yixi)2]⁢𝑑μ1superscriptsubscriptsubscript𝑙12subscript𝑙12superscriptsubscript𝑥𝑖2𝑛subscript𝑙1delimited-[]superscriptsubscript𝑙2212superscriptsubscript𝑥𝑖2superscriptsubscript𝜇1subscript𝑦𝑖subscript𝑥𝑖2differential-dsubscript𝜇1\displaystyle\sum\int_{-l_{1}/2}^{l_{1}/2}\frac{x_{i}^{2}}{nl_{1}}\left[\frac{% l_{2}^{2}}{12x_{i}^{2}}+\left(\mu_{1}-\frac{y_{i}}{x_{i}}\right)^{2}\right]d% \mu_{1}∑ ∫ start_POSTSUBSCRIPT - italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT / 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT / 2 end_POSTSUPERSCRIPT divide start_ARG italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG italic_n italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_ARG [ divide start_ARG italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + ( italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT - divide start_ARG italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG start_ARG italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ] italic_d italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT=\displaystyle==∑1n⁢[l12⁢xi212+yi2+l2212]1𝑛delimited-[]superscriptsubscript𝑙12superscriptsubscript𝑥𝑖212superscriptsubscript𝑦𝑖2superscriptsubscript𝑙2212\displaystyle\sum\frac{1}{n}\left[\frac{l_{1}^{2}x_{i}^{2}}{12}+y_{i}^{2}+% \frac{l_{2}^{2}}{12}\right]∑ divide start_ARG 1 end_ARG start_ARG italic_n end_ARG [ divide start_ARG italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG + italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + divide start_ARG italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG ]=\displaystyle==l1212⁢X2¯+Y2¯+l2212.superscriptsubscript𝑙1212¯superscript𝑋2¯superscript𝑌2superscriptsubscript𝑙2212\displaystyle\frac{l_{1}^{2}}{12}\overline{X^{2}}+\overline{Y^{2}}+\frac{l_{2}% ^{2}}{12}.divide start_ARG italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + over¯ start_ARG italic_Y start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + divide start_ARG italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG .While the final stateEfsubscript𝐸𝑓E_{f}italic_E start_POSTSUBSCRIPT italic_f end_POSTSUBSCRIPTfollows Equation (3.4). Therefore, the temperature of the system is

T=l12⁢X2¯+l22+12⁢Y2¯−12⁢(1−ρ2)⁢Var⁢(Y)12⁢ln⁡(l1⁢l2).𝑇superscriptsubscript𝑙12¯superscript𝑋2superscriptsubscript𝑙2212¯superscript𝑌2121superscript𝜌2Var𝑌12subscript𝑙1subscript𝑙2\displaystyle T=\frac{l_{1}^{2}\overline{X^{2}}+l_{2}^{2}+12\overline{Y^{2}}-1% 2(1-\rho^{2})\textrm{Var}(Y)}{12\ln(l_{1}l_{2})}.italic_T = divide start_ARG italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + 12 over¯ start_ARG italic_Y start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG - 12 ( 1 - italic_ρ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ) Var ( italic_Y ) end_ARG start_ARG 12 roman_ln ( italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ) end_ARG .(3.15)We can show that asl1=l2=l→∞subscript𝑙1subscript𝑙2𝑙→l_{1}=l_{2}=l\rightarrow\inftyitalic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT = italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT = italic_l → ∞, the asymptotic temperature of the system is

T∼l2⁢(1+X2¯)+12⁢Y2¯12⁢ln⁡l2∼(l224⁢ln⁡l)⁢(1+X2¯).similar-to𝑇superscript𝑙21¯superscript𝑋212¯superscript𝑌212superscript𝑙2similar-tosuperscript𝑙224𝑙1¯superscript𝑋2T\sim\frac{l^{2}(1+\overline{X^{2}})+12\overline{Y^{2}}}{12\ln l^{2}}\sim\left% (\frac{l^{2}}{24\ln l}\right)(1+\overline{X^{2}}).italic_T ∼ divide start_ARG italic_l start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ( 1 + over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ) + 12 over¯ start_ARG italic_Y start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG end_ARG start_ARG 12 roman_ln italic_l start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ∼ ( divide start_ARG italic_l start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 24 roman_ln italic_l end_ARG ) ( 1 + over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ) .(3.16) We can compare the temperature in Equation (3.16) for a system with parameters initialized by uniform distributions (hereafter “Uniform System” with Type I State temperatureTuniformsubscript𝑇uniformT_{\rm uniform}italic_T start_POSTSUBSCRIPT roman_uniform end_POSTSUBSCRIPT), to the temperature in Equation (3.6) for a system with parameters initialized by normal distributions (hereafter “Normal System” with Type I State temperatureTnormalsubscript𝑇normalT_{\rm normal}italic_T start_POSTSUBSCRIPT roman_normal end_POSTSUBSCRIPT). We cut out alKsuperscript𝑙𝐾l^{K}italic_l start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPTsized cube in the parameter space of the Normal System, wherel𝑙litalic_lcorresponds to the distribution region in the parameter space of System Uniform. Ifl=4⁢σ𝑙4𝜎l=4\sigmaitalic_l = 4 italic_σ, the cube encompasses approximately0.9545Ksuperscript0.9545𝐾0.9545^{K}0.9545 start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPTprobability of all particles in the Normal System. Therefore, we requirel≥4⁢σ𝑙4𝜎l\geq 4\sigmaitalic_l ≥ 4 italic_σto ensure that the Uniform System has at least a∼0.9545similar-toabsent0.9545\sim 0.9545∼ 0.9545similarity to the Normal System. Consequently, the two temperatures under this constraint have

Tuniformsubscript𝑇uniform\displaystyle T_{\rm uniform}italic_T start_POSTSUBSCRIPT roman_uniform end_POSTSUBSCRIPT∼similar-to\displaystyle\sim∼(l224⁢ln⁡l)⁢(1+X2¯)≳(16⁢σ224⁢ln⁡σ)⁢(1+X2¯)greater-than-or-equivalent-tosuperscript𝑙224𝑙1¯superscript𝑋216superscript𝜎224𝜎1¯superscript𝑋2\displaystyle\left(\frac{l^{2}}{24\ln l}\right)(1+\overline{X^{2}})\gtrsim% \left(\frac{16\sigma^{2}}{24\ln\sigma}\right)(1+\overline{X^{2}})( divide start_ARG italic_l start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 24 roman_ln italic_l end_ARG ) ( 1 + over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ) ≳ ( divide start_ARG 16 italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 24 roman_ln italic_σ end_ARG ) ( 1 + over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG )(3.17)>\displaystyle>>(σ22⁢ln⁡σ)⁢(1+X2¯)∼Tnormal,similar-tosuperscript𝜎22𝜎1¯superscript𝑋2subscript𝑇normal\displaystyle\left(\frac{\sigma^{2}}{2\ln\sigma}\right)(1+\overline{X^{2}})% \sim T_{\rm normal},( divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 roman_ln italic_σ end_ARG ) ( 1 + over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ) ∼ italic_T start_POSTSUBSCRIPT roman_normal end_POSTSUBSCRIPT ,which means for the Normal System and the Uniform System which are similar, the Uniform System has a higher temperatureTuniform>Tnormalsubscript𝑇uniformsubscript𝑇normalT_{\rm uniform}>T_{\rm normal}italic_T start_POSTSUBSCRIPT roman_uniform end_POSTSUBSCRIPT > italic_T start_POSTSUBSCRIPT roman_normal end_POSTSUBSCRIPT.

3.2.2High-dimensional Distribution

Generally, assuming the initial parameters of the ML system follow a high-dimensional uniform distribution as shown in Equation (3.12), we can calculate the initial energy of this system as

⟨E0⟩delimited-⟨⟩subscript𝐸0\displaystyle\langle E_{0}\rangle⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩=\displaystyle==∫1n⁢∑i(∑j=1K−1μj⁢xi⁢j+μK−yi)2⁢∏j=1K(d⁢μjlj)1𝑛subscript𝑖superscriptsuperscriptsubscript𝑗1𝐾1subscript𝜇𝑗subscript𝑥𝑖𝑗subscript𝜇𝐾subscript𝑦𝑖2superscriptsubscriptproduct𝑗1𝐾𝑑subscript𝜇𝑗subscript𝑙𝑗\displaystyle\int\frac{1}{n}\sum_{i}\left(\sum_{j=1}^{K-1}\mu_{j}x_{ij}+\mu_{K% }-y_{i}\right)^{2}\prod_{j=1}^{K}\left(\frac{d\mu_{j}}{l_{j}}\right)∫ divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT + italic_μ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ∏ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT ( divide start_ARG italic_d italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT end_ARG start_ARG italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT end_ARG )(3.18)=\displaystyle==∑i1n⁢∫∏j=1K−1(d⁢μjlj)⁢[lK212+(∑j=1K−1μj⁢xi⁢j−yi)2]subscript𝑖1𝑛superscriptsubscriptproduct𝑗1𝐾1𝑑subscript𝜇𝑗subscript𝑙𝑗delimited-[]superscriptsubscript𝑙𝐾212superscriptsuperscriptsubscript𝑗1𝐾1subscript𝜇𝑗subscript𝑥𝑖𝑗subscript𝑦𝑖2\displaystyle\sum_{i}\frac{1}{n}\int\prod_{j=1}^{K-1}\left(\frac{d\mu_{j}}{l_{% j}}\right)\left[\frac{l_{K}^{2}}{12}+\left(\sum_{j=1}^{K-1}\mu_{j}x_{ij}-y_{i}% \right)^{2}\right]∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∫ ∏ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT ( divide start_ARG italic_d italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT end_ARG start_ARG italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT end_ARG ) [ divide start_ARG italic_l start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG + ( ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ]=\displaystyle==lK212+lK−1212⁢XK−12¯+∑i1n⁢∫∏j=1K−3(d⁢μjlj)⁢[xi,K−22⁢lK−2212+(∑j=1K−3μj⁢xi⁢j−yi)2]superscriptsubscript𝑙𝐾212superscriptsubscript𝑙𝐾1212¯superscriptsubscript𝑋𝐾12subscript𝑖1𝑛superscriptsubscriptproduct𝑗1𝐾3𝑑subscript𝜇𝑗subscript𝑙𝑗delimited-[]superscriptsubscript𝑥𝑖𝐾22superscriptsubscript𝑙𝐾2212superscriptsuperscriptsubscript𝑗1𝐾3subscript𝜇𝑗subscript𝑥𝑖𝑗subscript𝑦𝑖2\displaystyle\frac{l_{K}^{2}}{12}+\frac{l_{K-1}^{2}}{12}\overline{X_{K-1}^{2}}% +\sum_{i}\frac{1}{n}\int\prod_{j=1}^{K-3}\left(\frac{d\mu_{j}}{l_{j}}\right)% \left[x_{i,K-2}^{2}\frac{l_{K-2}^{2}}{12}+\left(\sum_{j=1}^{K-3}\mu_{j}x_{ij}-% y_{i}\right)^{2}\right]divide start_ARG italic_l start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG + divide start_ARG italic_l start_POSTSUBSCRIPT italic_K - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_K - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∫ ∏ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 3 end_POSTSUPERSCRIPT ( divide start_ARG italic_d italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT end_ARG start_ARG italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT end_ARG ) [ italic_x start_POSTSUBSCRIPT italic_i , italic_K - 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT divide start_ARG italic_l start_POSTSUBSCRIPT italic_K - 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG + ( ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 3 end_POSTSUPERSCRIPT italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ]=\displaystyle==…….=lK212+lK−1212XK−12¯+…+l2212X22¯+∑i1n∫−l1/2l1/2(μ1xi⁢1−yi)2dμ1\displaystyle.......=\frac{l_{K}^{2}}{12}+\frac{l_{K-1}^{2}}{12}\overline{X_{K% -1}^{2}}+...+\frac{l_{2}^{2}}{12}\overline{X_{2}^{2}}+\sum_{i}\frac{1}{n}\int_% {-l_{1}/2}^{l_{1}/2}(\mu_{1}x_{i1}-y_{i})^{2}d\mu_{1}… … . = divide start_ARG italic_l start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG + divide start_ARG italic_l start_POSTSUBSCRIPT italic_K - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_K - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + … + divide start_ARG italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG over¯ start_ARG italic_X start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∫ start_POSTSUBSCRIPT - italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT / 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT / 2 end_POSTSUPERSCRIPT ( italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i 1 end_POSTSUBSCRIPT - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_d italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT=\displaystyle==lK212+lK−1212⁢XK−12¯+…+l2212⁢X22¯+l1212⁢X12¯+Y2¯superscriptsubscript𝑙𝐾212superscriptsubscript𝑙𝐾1212¯superscriptsubscript𝑋𝐾12…superscriptsubscript𝑙2212¯superscriptsubscript𝑋22superscriptsubscript𝑙1212¯superscriptsubscript𝑋12¯superscript𝑌2\displaystyle\frac{l_{K}^{2}}{12}+\frac{l_{K-1}^{2}}{12}\overline{X_{K-1}^{2}}% +...+\frac{l_{2}^{2}}{12}\overline{X_{2}^{2}}+\frac{l_{1}^{2}}{12}\overline{X_% {1}^{2}}+\overline{Y^{2}}divide start_ARG italic_l start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG + divide start_ARG italic_l start_POSTSUBSCRIPT italic_K - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_K - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + … + divide start_ARG italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG over¯ start_ARG italic_X start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + divide start_ARG italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG over¯ start_ARG italic_X start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + over¯ start_ARG italic_Y start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG=\displaystyle==112⁢∑j=1Klj2⁢Xj2¯+Y2¯112superscriptsubscript𝑗1𝐾superscriptsubscript𝑙𝑗2¯superscriptsubscript𝑋𝑗2¯superscript𝑌2\displaystyle\frac{1}{12}\sum_{j=1}^{K}l_{j}^{2}\overline{X_{j}^{2}}+\overline% {Y^{2}}divide start_ARG 1 end_ARG start_ARG 12 end_ARG ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + over¯ start_ARG italic_Y start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARGwhere we assume thek𝑘kitalic_k-th component of𝒙𝒙\boldsymbol{x}bold_italic_xthatxi⁢K=1subscript𝑥𝑖𝐾1x_{iK}=1italic_x start_POSTSUBSCRIPT italic_i italic_K end_POSTSUBSCRIPT = 1always holds, soXK2¯=1¯superscriptsubscript𝑋𝐾21\overline{X_{K}^{2}}=1over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG = 1. Thus, the temperature of the system is

T𝑇\displaystyle Titalic_T=\displaystyle==∑j=1Klj2⁢Xj2¯/12+Y¯2−tr⁢{Cx−Cx⁢y⁢Cy−1⁢Cy⁢x}∑j=1Kln⁡lj.superscriptsubscript𝑗1𝐾superscriptsubscript𝑙𝑗2¯superscriptsubscript𝑋𝑗212superscript¯𝑌2trsubscript𝐶𝑥subscript𝐶𝑥𝑦superscriptsubscript𝐶𝑦1subscript𝐶𝑦𝑥superscriptsubscript𝑗1𝐾subscript𝑙𝑗\displaystyle\frac{\sum_{j=1}^{K}l_{j}^{2}\overline{X_{j}^{2}}/12+\overline{Y}% ^{2}-{\rm tr}\{C_{x}-C_{xy}C_{y}^{-1}C_{yx}\}}{\sum_{j=1}^{K}\ln l_{j}}.divide start_ARG ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG / 12 + over¯ start_ARG italic_Y end_ARG start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT - roman_tr { italic_C start_POSTSUBSCRIPT italic_x end_POSTSUBSCRIPT - italic_C start_POSTSUBSCRIPT italic_x italic_y end_POSTSUBSCRIPT italic_C start_POSTSUBSCRIPT italic_y end_POSTSUBSCRIPT start_POSTSUPERSCRIPT - 1 end_POSTSUPERSCRIPT italic_C start_POSTSUBSCRIPT italic_y italic_x end_POSTSUBSCRIPT } end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT roman_ln italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT end_ARG .(3.19)For the asymptotic case{lj}→l→∞→subscript𝑙𝑗𝑙→\{l_{j}\}\rightarrow l\rightarrow\infty{ italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT } → italic_l → ∞, we have

T∼l2⁢∑1KXj2¯12⁢K⁢ln⁡l∼(l212⁢ln⁡l)⁢X2¯.similar-to𝑇superscript𝑙2superscriptsubscript1𝐾¯superscriptsubscript𝑋𝑗212𝐾𝑙similar-tosuperscript𝑙212𝑙¯superscript𝑋2\displaystyle T\sim\frac{l^{2}\sum_{1}^{K}\overline{X_{j}^{2}}}{12K\ln l}\sim% \left(\frac{l^{2}}{12\ln l}\right)\overline{X^{2}}.italic_T ∼ divide start_ARG italic_l start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ∑ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG end_ARG start_ARG 12 italic_K roman_ln italic_l end_ARG ∼ ( divide start_ARG italic_l start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 roman_ln italic_l end_ARG ) over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG .(3.20) For two similar Normal and Uniform Systems with finite regionl≥4⁢σ𝑙4𝜎l\geq 4\sigmaitalic_l ≥ 4 italic_σ, we obtainTuniform>Tnormalsubscript𝑇uniformsubscript𝑇normalT_{\rm uniform}>T_{\rm normal}italic_T start_POSTSUBSCRIPT roman_uniform end_POSTSUBSCRIPT > italic_T start_POSTSUBSCRIPT roman_normal end_POSTSUBSCRIPT.

3.3Mixed Distribution

In general, the initial parameters of a system typically follow either a uniform distribution or a normal distribution. If we consider the most common system, where the distribution of the initial parameters is a mixture of uniform and normal distribution, suppose the parameter space is of dimensionK+Q𝐾𝑄K+Qitalic_K + italic_Qwith the firstK𝐾Kitalic_Kdimensions being normal distribution, denoted as{μj(n)}j=1,2,…,Ksubscriptsuperscriptsubscript𝜇𝑗𝑛𝑗12…𝐾\{\mu_{j}^{(n)}\}_{j=1,2,...,K}{ italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_n ) end_POSTSUPERSCRIPT } start_POSTSUBSCRIPT italic_j = 1 , 2 , … , italic_K end_POSTSUBSCRIPTwithfn⁢(μj)subscript𝑓𝑛subscript𝜇𝑗f_{n}(\mu_{j})italic_f start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT )being normal distribution with standard deviationσjsubscript𝜎𝑗\sigma_{j}italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT, and dimensions fromK+1𝐾1K+1italic_K + 1st toK+Q−𝐾limit-from𝑄K+Q-italic_K + italic_Q -th being uniform distributions, denoted as{μj(u)}j=K+1,…,K+Qsubscriptsuperscriptsubscript𝜇𝑗𝑢𝑗𝐾1…𝐾𝑄\{\mu_{j}^{(u)}\}_{j=K+1,...,K+Q}{ italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_u ) end_POSTSUPERSCRIPT } start_POSTSUBSCRIPT italic_j = italic_K + 1 , … , italic_K + italic_Q end_POSTSUBSCRIPTwithfu⁢(μj)=1/ljsubscript𝑓𝑢subscript𝜇𝑗1subscript𝑙𝑗f_{u}(\mu_{j})=1/l_{j}italic_f start_POSTSUBSCRIPT italic_u end_POSTSUBSCRIPT ( italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT ) = 1 / italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPTfor−lj/2≤μj≤lj/2subscript𝑙𝑗2subscript𝜇𝑗subscript𝑙𝑗2-l_{j}/2\leq\mu_{j}\leq l_{j}/2- italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT / 2 ≤ italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT ≤ italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT / 2, then the energy of the system can be written as

⟨E0⟩delimited-⟨⟩subscript𝐸0\displaystyle\langle E_{0}\rangle⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩=\displaystyle==1n⁢∑i∫(∑j=1Kμj(n)⁢xi⁢j+∑k=K+1K+Q−1μj(u)⁢xi⁢j+μK+Q−yi)2⁢∏j=1K[fn⁢(μj(n))⁢d⁢μj]⁢∏j=K+1K+Q[fu⁢(μj(u))⁢d⁢μj]1𝑛subscript𝑖superscriptsuperscriptsubscript𝑗1𝐾superscriptsubscript𝜇𝑗𝑛subscript𝑥𝑖𝑗superscriptsubscript𝑘𝐾1𝐾𝑄1superscriptsubscript𝜇𝑗𝑢subscript𝑥𝑖𝑗subscript𝜇𝐾𝑄subscript𝑦𝑖2superscriptsubscriptproduct𝑗1𝐾delimited-[]subscript𝑓𝑛superscriptsubscript𝜇𝑗𝑛𝑑subscript𝜇𝑗superscriptsubscriptproduct𝑗𝐾1𝐾𝑄delimited-[]subscript𝑓𝑢superscriptsubscript𝜇𝑗𝑢𝑑subscript𝜇𝑗\displaystyle\frac{1}{n}\sum_{i}\int\left(\sum_{j=1}^{K}\mu_{j}^{(n)}x_{ij}+% \sum_{k=K+1}^{K+Q-1}\mu_{j}^{(u)}x_{ij}+\mu_{K+Q}-y_{i}\right)^{2}\prod_{j=1}^% {K}[f_{n}(\mu_{j}^{(n)})d\mu_{j}]\prod_{j=K+1}^{K+Q}[f_{u}(\mu_{j}^{(u)})d\mu_% {j}]divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ∫ ( ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_n ) end_POSTSUPERSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT + ∑ start_POSTSUBSCRIPT italic_k = italic_K + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K + italic_Q - 1 end_POSTSUPERSCRIPT italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_u ) end_POSTSUPERSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT + italic_μ start_POSTSUBSCRIPT italic_K + italic_Q end_POSTSUBSCRIPT - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ∏ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT [ italic_f start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_n ) end_POSTSUPERSCRIPT ) italic_d italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT ] ∏ start_POSTSUBSCRIPT italic_j = italic_K + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K + italic_Q end_POSTSUPERSCRIPT [ italic_f start_POSTSUBSCRIPT italic_u end_POSTSUBSCRIPT ( italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_u ) end_POSTSUPERSCRIPT ) italic_d italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT ](3.21)=\displaystyle==lK+Q212+lK+Q−1212⁢XK+Q−12¯+…+lK+1212⁢XK+12¯+1n⁢∑i∫(∑j=1Kμj(n)⁢xi⁢j−yi)2⁢∏j=1K[fn⁢(μj(n))⁢d⁢μj]superscriptsubscript𝑙𝐾𝑄212superscriptsubscript𝑙𝐾𝑄1212¯superscriptsubscript𝑋𝐾𝑄12…superscriptsubscript𝑙𝐾1212¯superscriptsubscript𝑋𝐾121𝑛subscript𝑖superscriptsuperscriptsubscript𝑗1𝐾superscriptsubscript𝜇𝑗𝑛subscript𝑥𝑖𝑗subscript𝑦𝑖2superscriptsubscriptproduct𝑗1𝐾delimited-[]subscript𝑓𝑛superscriptsubscript𝜇𝑗𝑛𝑑subscript𝜇𝑗\displaystyle\frac{l_{K+Q}^{2}}{12}+\frac{l_{K+Q-1}^{2}}{12}\overline{X_{K+Q-1% }^{2}}+...+\frac{l_{K+1}^{2}}{12}\overline{X_{K+1}^{2}}+\frac{1}{n}\sum_{i}% \int\left(\sum_{j=1}^{K}\mu_{j}^{(n)}x_{ij}-y_{i}\right)^{2}\prod_{j=1}^{K}[f_% {n}(\mu_{j}^{(n)})d\mu_{j}]divide start_ARG italic_l start_POSTSUBSCRIPT italic_K + italic_Q end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG + divide start_ARG italic_l start_POSTSUBSCRIPT italic_K + italic_Q - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_K + italic_Q - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + … + divide start_ARG italic_l start_POSTSUBSCRIPT italic_K + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_K + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ∫ ( ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_n ) end_POSTSUPERSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ∏ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT [ italic_f start_POSTSUBSCRIPT italic_n end_POSTSUBSCRIPT ( italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ( italic_n ) end_POSTSUPERSCRIPT ) italic_d italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT ]=\displaystyle==lK+Q212+lK+Q−1212⁢XK+Q−12¯+…+lK+1212⁢XK+12¯+σK2⁢XK2¯+σK−12⁢XK−12¯+…+σ12⁢X12¯+Y2¯superscriptsubscript𝑙𝐾𝑄212superscriptsubscript𝑙𝐾𝑄1212¯superscriptsubscript𝑋𝐾𝑄12…superscriptsubscript𝑙𝐾1212¯superscriptsubscript𝑋𝐾12superscriptsubscript𝜎𝐾2¯superscriptsubscript𝑋𝐾2superscriptsubscript𝜎𝐾12¯superscriptsubscript𝑋𝐾12…superscriptsubscript𝜎12¯superscriptsubscript𝑋12¯superscript𝑌2\displaystyle\frac{l_{K+Q}^{2}}{12}+\frac{l_{K+Q-1}^{2}}{12}\overline{X_{K+Q-1% }^{2}}+...+\frac{l_{K+1}^{2}}{12}\overline{X_{K+1}^{2}}+\sigma_{K}^{2}% \overline{X_{K}^{2}}+\sigma_{K-1}^{2}\overline{X_{K-1}^{2}}+...+\sigma_{1}^{2}% \overline{X_{1}^{2}}+\overline{Y^{2}}divide start_ARG italic_l start_POSTSUBSCRIPT italic_K + italic_Q end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG + divide start_ARG italic_l start_POSTSUBSCRIPT italic_K + italic_Q - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_K + italic_Q - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + … + divide start_ARG italic_l start_POSTSUBSCRIPT italic_K + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_K + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + italic_σ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + italic_σ start_POSTSUBSCRIPT italic_K - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_K - 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + … + italic_σ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + over¯ start_ARG italic_Y start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG=\displaystyle==∑j=1Kσj2⁢Xj2¯+∑j=K+1K+Qlj212⁢Xj2¯+Y2¯.superscriptsubscript𝑗1𝐾superscriptsubscript𝜎𝑗2¯superscriptsubscript𝑋𝑗2superscriptsubscript𝑗𝐾1𝐾𝑄superscriptsubscript𝑙𝑗212¯superscriptsubscript𝑋𝑗2¯superscript𝑌2\displaystyle\sum_{j=1}^{K}\sigma_{j}^{2}\overline{X_{j}^{2}}+\sum_{j=K+1}^{K+% Q}\frac{l_{j}^{2}}{12}\overline{X_{j}^{2}}+\overline{Y^{2}}.∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + ∑ start_POSTSUBSCRIPT italic_j = italic_K + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K + italic_Q end_POSTSUPERSCRIPT divide start_ARG italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + over¯ start_ARG italic_Y start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG .Let we denoteσj=lj/12subscript𝜎𝑗subscript𝑙𝑗12\sigma_{j}=l_{j}/\sqrt{12}italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT = italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT / square-root start_ARG 12 end_ARG, we have

⟨E0⟩delimited-⟨⟩subscript𝐸0\displaystyle\langle E_{0}\rangle⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩=\displaystyle==∑j=1Kσj2⁢Xj2¯+∑j=K+1K+Qσj2⁢Xj2¯+Y2¯=∑j=1K+Qσj2⁢Xj2¯+Y2¯,superscriptsubscript𝑗1𝐾superscriptsubscript𝜎𝑗2¯superscriptsubscript𝑋𝑗2superscriptsubscript𝑗𝐾1𝐾𝑄superscriptsubscript𝜎𝑗2¯superscriptsubscript𝑋𝑗2¯superscript𝑌2superscriptsubscript𝑗1𝐾𝑄superscriptsubscript𝜎𝑗2¯superscriptsubscript𝑋𝑗2¯superscript𝑌2\displaystyle\sum_{j=1}^{K}\sigma_{j}^{2}\overline{X_{j}^{2}}+\sum_{j=K+1}^{K+% Q}\sigma_{j}^{2}\overline{X_{j}^{2}}+\overline{Y^{2}}=\sum_{j=1}^{K+Q}\sigma_{% j}^{2}\overline{X_{j}^{2}}+\overline{Y^{2}},∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + ∑ start_POSTSUBSCRIPT italic_j = italic_K + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K + italic_Q end_POSTSUPERSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + over¯ start_ARG italic_Y start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG = ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K + italic_Q end_POSTSUPERSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + over¯ start_ARG italic_Y start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ,(3.22)which shows in the same the unified form as Equation (3.8).

On the other hand, the initial entropy of the system is

S𝑆\displaystyle Sitalic_S=\displaystyle==ln(σ1σ2..σK)+K2ln[1+ln(2π)]+ln(lK+1lK+2…lK+Q)\displaystyle\ln(\sigma_{1}\sigma_{2}..\sigma_{K})+\frac{K}{2}\ln[1+\ln(2\pi)]% +\ln(l_{K+1}l_{K+2}...l_{K+Q})roman_ln ( italic_σ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT italic_σ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT . . italic_σ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT ) + divide start_ARG italic_K end_ARG start_ARG 2 end_ARG roman_ln [ 1 + roman_ln ( 2 italic_π ) ] + roman_ln ( italic_l start_POSTSUBSCRIPT italic_K + 1 end_POSTSUBSCRIPT italic_l start_POSTSUBSCRIPT italic_K + 2 end_POSTSUBSCRIPT … italic_l start_POSTSUBSCRIPT italic_K + italic_Q end_POSTSUBSCRIPT )(3.23)=\displaystyle==ln(σ1σ2..σK)+K2ln[1+ln(2π)]+ln(σK+1σK+2…σK+Q)+Q2ln12\displaystyle\ln(\sigma_{1}\sigma_{2}..\sigma_{K})+\frac{K}{2}\ln[1+\ln(2\pi)]% +\ln(\sigma_{K+1}\sigma_{K+2}...\sigma_{K+Q})+\frac{Q}{2}\ln 12roman_ln ( italic_σ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT italic_σ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT . . italic_σ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT ) + divide start_ARG italic_K end_ARG start_ARG 2 end_ARG roman_ln [ 1 + roman_ln ( 2 italic_π ) ] + roman_ln ( italic_σ start_POSTSUBSCRIPT italic_K + 1 end_POSTSUBSCRIPT italic_σ start_POSTSUBSCRIPT italic_K + 2 end_POSTSUBSCRIPT … italic_σ start_POSTSUBSCRIPT italic_K + italic_Q end_POSTSUBSCRIPT ) + divide start_ARG italic_Q end_ARG start_ARG 2 end_ARG roman_ln 12=\displaystyle==ln⁡(∏jK+Qσj)+K2⁢ln⁡[1+ln⁡(2⁢π)]+Q2⁢ln⁡12.superscriptsubscriptproduct𝑗𝐾𝑄subscript𝜎𝑗𝐾212𝜋𝑄212\displaystyle\ln\left(\prod_{j}^{K+Q}\sigma_{j}\right)+\frac{K}{2}\ln[1+\ln(2% \pi)]+\frac{Q}{2}\ln 12.roman_ln ( ∏ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K + italic_Q end_POSTSUPERSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT ) + divide start_ARG italic_K end_ARG start_ARG 2 end_ARG roman_ln [ 1 + roman_ln ( 2 italic_π ) ] + divide start_ARG italic_Q end_ARG start_ARG 2 end_ARG roman_ln 12 .Thus, the temperature of the system is

T=∑j=1K+Qσj2Xj2¯+Y2¯+const1.ln(∏jK+Qσj)+const2.T=\frac{\sum_{j=1}^{K+Q}\sigma_{j}^{2}\overline{X_{j}^{2}}+\overline{Y^{2}}+{% \rm const1.}}{\ln\left(\prod_{j}^{K+Q}\sigma_{j}\right)+{\rm const2}.}italic_T = divide start_ARG ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K + italic_Q end_POSTSUPERSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + over¯ start_ARG italic_Y start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + const1 . end_ARG start_ARG roman_ln ( ∏ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K + italic_Q end_POSTSUPERSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT ) + const2 . end_ARG(3.24) (1) For allσj→σ→∞→subscript𝜎𝑗𝜎→\sigma_{j}\rightarrow\sigma\rightarrow\inftyitalic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT → italic_σ → ∞,

T𝑇\displaystyle Titalic_T≈\displaystyle\approx≈σ2⁢∑1K+QXj2¯(K+Q)⁢ln⁡σ≈X2¯⁢(σ2ln⁡σ),superscript𝜎2superscriptsubscript1𝐾𝑄¯superscriptsubscript𝑋𝑗2𝐾𝑄𝜎¯superscript𝑋2superscript𝜎2𝜎\displaystyle\frac{\sigma^{2}\sum_{1}^{K+Q}\overline{X_{j}^{2}}}{(K+Q)\ln% \sigma}\approx\overline{X^{2}}\left(\frac{\sigma^{2}}{\ln\sigma}\right),divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ∑ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K + italic_Q end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG end_ARG start_ARG ( italic_K + italic_Q ) roman_ln italic_σ end_ARG ≈ over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ( divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ end_ARG ) ,(3.25)which is the same as normal distribution.

(2) As discussed in Section3.1.2, if we useσj=σsubscript𝜎𝑗𝜎\sigma_{j}=\sigmaitalic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT = italic_σonly forj=1,2,3,…,K𝑗123…𝐾j=1,2,3,...,Kitalic_j = 1 , 2 , 3 , … , italic_Kwhilelj≥4⁢σsubscript𝑙𝑗4𝜎l_{j}\geq 4\sigmaitalic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT ≥ 4 italic_σ, i.e.,σj≥2⁢σ/3subscript𝜎𝑗2𝜎3\sigma_{j}\geq 2\sigma/\sqrt{3}italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT ≥ 2 italic_σ / square-root start_ARG 3 end_ARGforj=K+1,K+2,…,K+Q𝑗𝐾1𝐾2…𝐾𝑄j=K+1,K+2,...,K+Qitalic_j = italic_K + 1 , italic_K + 2 , … , italic_K + italic_Q, the temperature

T𝑇\displaystyle Titalic_T≳greater-than-or-equivalent-to\displaystyle\gtrsim≳σ2⁢∑1KXj2¯+43⁢σ2⁢∑K+1K+QXj2¯(K+Q)⁢ln⁡σ>X2¯⁢(σ2ln⁡σ),superscript𝜎2superscriptsubscript1𝐾¯superscriptsubscript𝑋𝑗243superscript𝜎2superscriptsubscript𝐾1𝐾𝑄¯superscriptsubscript𝑋𝑗2𝐾𝑄𝜎¯superscript𝑋2superscript𝜎2𝜎\displaystyle\frac{\sigma^{2}\sum_{1}^{K}\overline{X_{j}^{2}}+\frac{4}{3}% \sigma^{2}\sum_{K+1}^{K+Q}\overline{X_{j}^{2}}}{(K+Q)\ln\sigma}>\overline{X^{2% }}\left(\frac{\sigma^{2}}{\ln\sigma}\right),divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ∑ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + divide start_ARG 4 end_ARG start_ARG 3 end_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ∑ start_POSTSUBSCRIPT italic_K + 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K + italic_Q end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG end_ARG start_ARG ( italic_K + italic_Q ) roman_ln italic_σ end_ARG > over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ( divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ end_ARG ) ,(3.26)which gives a higher temperature.

3.4MSE with Regularization

In this section, we explore the system energy with regularization, especiallyL1subscript𝐿1L_{1}italic_L start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPTandL2subscript𝐿2L_{2}italic_L start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPTregularization. For the MSE-based energy, the energy withL1subscript𝐿1L_{1}italic_L start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPTregularization isEL1=EMSE+λ⁢∑j|μj|subscript𝐸subscript𝐿1subscript𝐸MSE𝜆subscript𝑗subscript𝜇𝑗E_{L_{1}}=E_{\rm MSE}+\lambda\sum_{j}|\mu_{j}|italic_E start_POSTSUBSCRIPT italic_L start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPT = italic_E start_POSTSUBSCRIPT roman_MSE end_POSTSUBSCRIPT + italic_λ ∑ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT | italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT |, while the energy withL2subscript𝐿2L_{2}italic_L start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPTregularization isEL2=EMSE+λ⁢∑j|μj2|subscript𝐸subscript𝐿2subscript𝐸MSE𝜆subscript𝑗superscriptsubscript𝜇𝑗2E_{L_{2}}=E_{\rm MSE}+\lambda\sum_{j}|\mu_{j}^{2}|italic_E start_POSTSUBSCRIPT italic_L start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_POSTSUBSCRIPT = italic_E start_POSTSUBSCRIPT roman_MSE end_POSTSUBSCRIPT + italic_λ ∑ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT | italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT |.

3.4.1Normal Distribution

If the initial distribution of the ML system parameters follow normal distributions, then the initial energy of the system withL1subscript𝐿1L_{1}italic_L start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPTregularization is calculated as

⟨EL1⟩delimited-⟨⟩subscript𝐸subscript𝐿1\displaystyle\langle E_{L_{1}}\rangle⟨ italic_E start_POSTSUBSCRIPT italic_L start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPT ⟩=\displaystyle==⟨E0⟩+λ⁢∑j=1K(∫|μj|2⁢π⁢σj⁢e−μj22⁢σj2⁢𝑑μj)delimited-⟨⟩subscript𝐸0𝜆superscriptsubscript𝑗1𝐾subscript𝜇𝑗2𝜋subscript𝜎𝑗superscriptesuperscriptsubscript𝜇𝑗22superscriptsubscript𝜎𝑗2differential-dsubscript𝜇𝑗\displaystyle\langle E_{0}\rangle+\lambda\sum_{j=1}^{K}\left(\int\frac{|\mu_{j% }|}{\sqrt{2\pi}\sigma_{j}}\textrm{e}^{-\frac{\mu_{j}^{2}}{2\sigma_{j}^{2}}}d% \mu_{j}\right)⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩ + italic_λ ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT ( ∫ divide start_ARG | italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT | end_ARG start_ARG square-root start_ARG 2 italic_π end_ARG italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT end_ARG e start_POSTSUPERSCRIPT - divide start_ARG italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG end_POSTSUPERSCRIPT italic_d italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT )(3.27)=\displaystyle==⟨E0⟩+λ⁢2π⁢∑j=1Kσj=∑j=1Kσj⁢(σj⁢Xj2¯+λ′)+Y2¯,delimited-⟨⟩subscript𝐸0𝜆2𝜋superscriptsubscript𝑗1𝐾subscript𝜎𝑗superscriptsubscript𝑗1𝐾subscript𝜎𝑗subscript𝜎𝑗¯superscriptsubscript𝑋𝑗2superscript𝜆′¯superscript𝑌2\displaystyle\langle E_{0}\rangle+\lambda\sqrt{\frac{2}{\pi}}\sum_{j=1}^{K}% \sigma_{j}=\sum_{j=1}^{K}\sigma_{j}(\sigma_{j}\overline{X_{j}^{2}}+\lambda^{% \prime})+\overline{Y^{2}},⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩ + italic_λ square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_π end_ARG end_ARG ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT = ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT ( italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + italic_λ start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ) + over¯ start_ARG italic_Y start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ,where we takeλ′=λ⁢2/πsuperscript𝜆′𝜆2𝜋\lambda^{\prime}=\lambda\sqrt{2/\pi}italic_λ start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT = italic_λ square-root start_ARG 2 / italic_π end_ARG.

The system withL2subscript𝐿2L_{2}italic_L start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPTregularization is

⟨EL1⟩delimited-⟨⟩subscript𝐸subscript𝐿1\displaystyle\langle E_{L_{1}}\rangle⟨ italic_E start_POSTSUBSCRIPT italic_L start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPT ⟩=\displaystyle==⟨E0⟩+λ⁢∑j=1K(∫|μj2|2⁢π⁢σj⁢e−μj22⁢σj2⁢𝑑μj)delimited-⟨⟩subscript𝐸0𝜆superscriptsubscript𝑗1𝐾superscriptsubscript𝜇𝑗22𝜋subscript𝜎𝑗superscriptesuperscriptsubscript𝜇𝑗22superscriptsubscript𝜎𝑗2differential-dsubscript𝜇𝑗\displaystyle\langle E_{0}\rangle+\lambda\sum_{j=1}^{K}\left(\int\frac{|\mu_{j% }^{2}|}{\sqrt{2\pi}\sigma_{j}}\textrm{e}^{-\frac{\mu_{j}^{2}}{2\sigma_{j}^{2}}% }d\mu_{j}\right)⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩ + italic_λ ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT ( ∫ divide start_ARG | italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT | end_ARG start_ARG square-root start_ARG 2 italic_π end_ARG italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT end_ARG e start_POSTSUPERSCRIPT - divide start_ARG italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG end_POSTSUPERSCRIPT italic_d italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT )(3.28)=\displaystyle==⟨E0⟩+λ⁢∑j=1Kσj2=∑j=1Kσj2⁢(Xj2¯+λ)+Y2¯.delimited-⟨⟩subscript𝐸0𝜆superscriptsubscript𝑗1𝐾superscriptsubscript𝜎𝑗2superscriptsubscript𝑗1𝐾subscriptsuperscript𝜎2𝑗¯superscriptsubscript𝑋𝑗2𝜆¯superscript𝑌2\displaystyle\langle E_{0}\rangle+\lambda\sum_{j=1}^{K}\sigma_{j}^{2}=\sum_{j=% 1}^{K}\sigma^{2}_{j}\left(\overline{X_{j}^{2}}+\lambda\right)+\overline{Y^{2}}.⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩ + italic_λ ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT = ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT ( over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + italic_λ ) + over¯ start_ARG italic_Y start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG . For the asymptotic case allσj→σ→∞→subscript𝜎𝑗𝜎→\sigma_{j}\rightarrow\sigma\rightarrow\inftyitalic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT → italic_σ → ∞, we obtain two temperature forms

TL1∼X2¯⁢(σ2ln⁡σ)∼Tnormal,similar-tosubscript𝑇subscript𝐿1¯superscript𝑋2superscript𝜎2𝜎similar-tosubscript𝑇normalT_{L_{1}}\sim\overline{X^{2}}\left(\frac{\sigma^{2}}{\ln\sigma}\right)\sim T_{% \rm normal},italic_T start_POSTSUBSCRIPT italic_L start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPT ∼ over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ( divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ end_ARG ) ∼ italic_T start_POSTSUBSCRIPT roman_normal end_POSTSUBSCRIPT ,(3.29)TL2∼(X2¯+λ)⁢(σ2ln⁡σ)∼(1+λX2¯)⁢Tnormal.similar-tosubscript𝑇subscript𝐿2¯superscript𝑋2𝜆superscript𝜎2𝜎similar-to1𝜆¯superscript𝑋2subscript𝑇normalT_{L_{2}}\sim(\overline{X^{2}}+\lambda)\left(\frac{\sigma^{2}}{\ln\sigma}% \right)\sim\left(1+\frac{\lambda}{\overline{X^{2}}}\right)T_{\rm normal}.italic_T start_POSTSUBSCRIPT italic_L start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_POSTSUBSCRIPT ∼ ( over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + italic_λ ) ( divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ end_ARG ) ∼ ( 1 + divide start_ARG italic_λ end_ARG start_ARG over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG end_ARG ) italic_T start_POSTSUBSCRIPT roman_normal end_POSTSUBSCRIPT .(3.30)This implies thatL1subscript𝐿1L_{1}italic_L start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPTregularization, in the asymptotic sense, does not change the system temperature, whileL2subscript𝐿2L_{2}italic_L start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPTregularization maintains the temperature form (∝σ2ln⁡σproportional-toabsentsuperscript𝜎2𝜎\propto\frac{\sigma^{2}}{\ln\sigma}∝ divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ end_ARG), but adds(λ/X2¯)⁢Tnormal𝜆¯superscript𝑋2subscript𝑇normal(\lambda/\overline{X^{2}})T_{\rm normal}( italic_λ / over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ) italic_T start_POSTSUBSCRIPT roman_normal end_POSTSUBSCRIPTto the existing system temperature.

3.4.2Uniform Distribution

Similar result can be obtained for initially uniform distributed parameters. The initial energy withL1subscript𝐿1L_{1}italic_L start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPTandL2subscript𝐿2L_{2}italic_L start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPTregularization are

⟨EL1⟩=∑j=1Klj12⁢(lj⁢Xj2¯+3⁢λ)+Y2¯,delimited-⟨⟩subscript𝐸subscript𝐿1superscriptsubscript𝑗1𝐾subscript𝑙𝑗12subscript𝑙𝑗¯superscriptsubscript𝑋𝑗23𝜆¯superscript𝑌2\langle E_{L_{1}}\rangle=\sum_{j=1}^{K}\frac{l_{j}}{12}(l_{j}\overline{X_{j}^{% 2}}+3\lambda)+\overline{Y^{2}},⟨ italic_E start_POSTSUBSCRIPT italic_L start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPT ⟩ = ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT divide start_ARG italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT end_ARG start_ARG 12 end_ARG ( italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + 3 italic_λ ) + over¯ start_ARG italic_Y start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ,(3.31)⟨EL2⟩=∑j=1Klj212⁢(Xj2¯+λ)+Y2¯,delimited-⟨⟩subscript𝐸subscript𝐿2superscriptsubscript𝑗1𝐾superscriptsubscript𝑙𝑗212¯superscriptsubscript𝑋𝑗2𝜆¯superscript𝑌2\langle E_{L_{2}}\rangle=\sum_{j=1}^{K}\frac{l_{j}^{2}}{12}(\overline{X_{j}^{2% }}+\lambda)+\overline{Y^{2}},⟨ italic_E start_POSTSUBSCRIPT italic_L start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_POSTSUBSCRIPT ⟩ = ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT divide start_ARG italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG ( over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + italic_λ ) + over¯ start_ARG italic_Y start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ,(3.32)respectively. So for the asymptotic case alllj=l→∞subscript𝑙𝑗𝑙→l_{j}=l\rightarrow\inftyitalic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT = italic_l → ∞, the two temperatures are

TL1∼X2¯⁢(l212⁢ln⁡l)∼Tuniform,similar-tosubscript𝑇subscript𝐿1¯superscript𝑋2superscript𝑙212𝑙similar-tosubscript𝑇uniformT_{L_{1}}\sim\overline{X^{2}}\left(\frac{l^{2}}{12\ln l}\right)\sim T_{\rm uniform},italic_T start_POSTSUBSCRIPT italic_L start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPT ∼ over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ( divide start_ARG italic_l start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 roman_ln italic_l end_ARG ) ∼ italic_T start_POSTSUBSCRIPT roman_uniform end_POSTSUBSCRIPT ,(3.33)TL2∼(X2¯+λ)⁢(l212⁢ln⁡l)∼(1+λX2¯)⁢Tuniform.similar-tosubscript𝑇subscript𝐿2¯superscript𝑋2𝜆superscript𝑙212𝑙similar-to1𝜆¯superscript𝑋2subscript𝑇uniformT_{L_{2}}\sim(\overline{X^{2}}+\lambda)\left(\frac{l^{2}}{12\ln l}\right)\sim% \left(1+\frac{\lambda}{\overline{X^{2}}}\right)T_{\rm uniform}.italic_T start_POSTSUBSCRIPT italic_L start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_POSTSUBSCRIPT ∼ ( over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + italic_λ ) ( divide start_ARG italic_l start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 roman_ln italic_l end_ARG ) ∼ ( 1 + divide start_ARG italic_λ end_ARG start_ARG over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG end_ARG ) italic_T start_POSTSUBSCRIPT roman_uniform end_POSTSUBSCRIPT .(3.34)Thus we have the similar result thatL1subscript𝐿1L_{1}italic_L start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPTregularization does not change the temperature, whileL2subscript𝐿2L_{2}italic_L start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPTregularization also does not change the energy form∝l212⁢ln⁡lproportional-toabsentsuperscript𝑙212𝑙\propto\frac{l^{2}}{12\ln l}∝ divide start_ARG italic_l start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 roman_ln italic_l end_ARG.

3.5Linear Regression with MAE

Also, we consider systems with energy follow the MAE form thatEp=1n⁢|∑jK−1μj⁢xi⁢j+μK−yi|subscript𝐸𝑝1𝑛superscriptsubscript𝑗𝐾1subscript𝜇𝑗subscript𝑥𝑖𝑗subscript𝜇𝐾subscript𝑦𝑖E_{p}=\frac{1}{n}|\sum_{j}^{K-1}\mu_{j}x_{ij}+\mu_{K}-y_{i}|italic_E start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT = divide start_ARG 1 end_ARG start_ARG italic_n end_ARG | ∑ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT + italic_μ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT |. For initialized normal distributed parameters, the initial energy of the state is

⟨E0⟩delimited-⟨⟩subscript𝐸0\displaystyle\langle E_{0}\rangle⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩=\displaystyle==∫1n⁢∑i=1n|∑jK−1μj⁢xi⁢j+μK−yi|⁢∏j=1K(12⁢π⁢σj⁢e−μj22⁢σj2⁢d⁢μj)1𝑛superscriptsubscript𝑖1𝑛superscriptsubscript𝑗𝐾1subscript𝜇𝑗subscript𝑥𝑖𝑗subscript𝜇𝐾subscript𝑦𝑖superscriptsubscriptproduct𝑗1𝐾12𝜋subscript𝜎𝑗superscriptesuperscriptsubscript𝜇𝑗22superscriptsubscript𝜎𝑗2𝑑subscript𝜇𝑗\displaystyle\int\frac{1}{n}\sum_{i=1}^{n}\left|\sum_{j}^{K-1}\mu_{j}x_{ij}+% \mu_{K}-y_{i}\right|\prod_{j=1}^{K}\left(\frac{1}{\sqrt{2\pi}\sigma_{j}}% \textrm{e}^{-\frac{\mu_{j}^{2}}{2\sigma_{j}^{2}}}d\mu_{j}\right)∫ divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT | ∑ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT + italic_μ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT | ∏ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT ( divide start_ARG 1 end_ARG start_ARG square-root start_ARG 2 italic_π end_ARG italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT end_ARG e start_POSTSUPERSCRIPT - divide start_ARG italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG end_POSTSUPERSCRIPT italic_d italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT )(3.35)Different from MSE, the energy in the form of MAE does not have an analytical solution. We look into the asymptotic properties of the energy. Assumingσ1≫{σ2,σ3,…,σK}much-greater-thansubscript𝜎1subscript𝜎2subscript𝜎3…subscript𝜎𝐾\sigma_{1}\gg\{\sigma_{2},\sigma_{3},...,\sigma_{K}\}italic_σ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ≫ { italic_σ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , italic_σ start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT , … , italic_σ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT }, Equation (3.35) becomes

⟨E0⟩∼∫12⁢π⁢n⁢∑i=1n|μ1⁢σ1⁢xi⁢1|⁢e−μ122⁢d⁢μ1∼2π⁢σ1⁢|x1|¯,similar-todelimited-⟨⟩subscript𝐸012𝜋𝑛superscriptsubscript𝑖1𝑛subscript𝜇1subscript𝜎1subscript𝑥𝑖1superscriptesuperscriptsubscript𝜇122𝑑subscript𝜇1similar-to2𝜋subscript𝜎1¯subscript𝑥1\displaystyle\langle E_{0}\rangle\sim\int\frac{1}{\sqrt{2\pi}n}\sum_{i=1}^{n}|% \mu_{1}\sigma_{1}x_{i1}|\textrm{e}^{-\frac{\mu_{1}^{2}}{2}}d\mu_{1}\sim\sqrt{% \frac{2}{\pi}}\sigma_{1}\overline{|x_{1}|},⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩ ∼ ∫ divide start_ARG 1 end_ARG start_ARG square-root start_ARG 2 italic_π end_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT | italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT italic_σ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i 1 end_POSTSUBSCRIPT | e start_POSTSUPERSCRIPT - divide start_ARG italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 end_ARG end_POSTSUPERSCRIPT italic_d italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ∼ square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_π end_ARG end_ARG italic_σ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT over¯ start_ARG | italic_x start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT | end_ARG ,(3.36)where|x1|¯¯subscript𝑥1\overline{|x_{1}|}over¯ start_ARG | italic_x start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT | end_ARGis the average of absolute values of the first component of𝒙𝒙\boldsymbol{x}bold_italic_x. Therefore, for{σj}j=1,2,…,K→∞→subscriptsubscript𝜎𝑗𝑗12…𝐾\{\sigma_{j}\}_{j=1,2,...,K}\rightarrow\infty{ italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT } start_POSTSUBSCRIPT italic_j = 1 , 2 , … , italic_K end_POSTSUBSCRIPT → ∞, the asymptotic energy is approximated as

⟨E0⟩∼2π⁢∑j=1Kσj⁢|xj|¯.similar-todelimited-⟨⟩subscript𝐸02𝜋superscriptsubscript𝑗1𝐾subscript𝜎𝑗¯subscript𝑥𝑗\langle E_{0}\rangle\sim\sqrt{\frac{2}{\pi}}\sum_{j=1}^{K}\sigma_{j}\overline{% |x_{j}|}.⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩ ∼ square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_π end_ARG end_ARG ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT over¯ start_ARG | italic_x start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT | end_ARG .(3.37)The temperature of the system is

T∼2π⁢∑j=1Kσj⁢|xj|¯∑j=1Kln⁡σj+K2⁢[1+ln⁡(2⁢π)].similar-to𝑇2𝜋superscriptsubscript𝑗1𝐾subscript𝜎𝑗¯subscript𝑥𝑗superscriptsubscript𝑗1𝐾subscript𝜎𝑗𝐾2delimited-[]12𝜋T\sim\frac{\sqrt{\frac{2}{\pi}}\sum_{j=1}^{K}\sigma_{j}\overline{|x_{j}|}}{% \sum_{j=1}^{K}\ln\sigma_{j}+\frac{K}{2}[1+\ln(2\pi)]}.italic_T ∼ divide start_ARG square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_π end_ARG end_ARG ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT over¯ start_ARG | italic_x start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT | end_ARG end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT roman_ln italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT + divide start_ARG italic_K end_ARG start_ARG 2 end_ARG [ 1 + roman_ln ( 2 italic_π ) ] end_ARG .(3.38)For{σj}j=1,2,…,K→σ→∞→subscriptsubscript𝜎𝑗𝑗12…𝐾𝜎→\{\sigma_{j}\}_{j=1,2,...,K}\rightarrow\sigma\rightarrow\infty{ italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT } start_POSTSUBSCRIPT italic_j = 1 , 2 , … , italic_K end_POSTSUBSCRIPT → italic_σ → ∞, we have

T∼2π⁢(σln⁡σ)⁢∑j=1K|xj|¯K∼2π⁢(σln⁡σ)⁢|X|¯.similar-to𝑇2𝜋𝜎𝜎superscriptsubscript𝑗1𝐾¯subscript𝑥𝑗𝐾similar-to2𝜋𝜎𝜎¯𝑋T\sim\sqrt{\frac{2}{\pi}}\left(\frac{\sigma}{\ln\sigma}\right)\frac{\sum_{j=1}% ^{K}\overline{|x_{j}|}}{K}\sim\sqrt{\frac{2}{\pi}}\left(\frac{\sigma}{\ln% \sigma}\right)\overline{|X|}.italic_T ∼ square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_π end_ARG end_ARG ( divide start_ARG italic_σ end_ARG start_ARG roman_ln italic_σ end_ARG ) divide start_ARG ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT over¯ start_ARG | italic_x start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT | end_ARG end_ARG start_ARG italic_K end_ARG ∼ square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_π end_ARG end_ARG ( divide start_ARG italic_σ end_ARG start_ARG roman_ln italic_σ end_ARG ) over¯ start_ARG | italic_X | end_ARG .(3.39) According to Section3.1.2, the temperature of a system under the same conditions but with MSE energy isTMSE∼X2¯⁢(σ2ln⁡σ)≫TMAE∼2π⁢(σln⁡σ)⁢|X|¯similar-tosubscript𝑇MSE¯superscript𝑋2superscript𝜎2𝜎much-greater-thansubscript𝑇MAEsimilar-to2𝜋𝜎𝜎¯𝑋T_{\rm MSE}\sim\overline{X^{2}}\left(\frac{\sigma^{2}}{\ln\sigma}\right)\gg T_% {\rm MAE}\sim\sqrt{\frac{2}{\pi}}\left(\frac{\sigma}{\ln\sigma}\right)% \overline{|X|}italic_T start_POSTSUBSCRIPT roman_MSE end_POSTSUBSCRIPT ∼ over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ( divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ end_ARG ) ≫ italic_T start_POSTSUBSCRIPT roman_MAE end_POSTSUBSCRIPT ∼ square-root start_ARG divide start_ARG 2 end_ARG start_ARG italic_π end_ARG end_ARG ( divide start_ARG italic_σ end_ARG start_ARG roman_ln italic_σ end_ARG ) over¯ start_ARG | italic_X | end_ARG. SinceTMSE∝σ2proportional-tosubscript𝑇MSEsuperscript𝜎2T_{\rm MSE}\propto\sigma^{2}italic_T start_POSTSUBSCRIPT roman_MSE end_POSTSUBSCRIPT ∝ italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPTwhileTMAE∝σproportional-tosubscript𝑇MAE𝜎T_{\rm MAE}\propto\sigmaitalic_T start_POSTSUBSCRIPT roman_MAE end_POSTSUBSCRIPT ∝ italic_σ, system with MSE energy has much higher temperature than that with MAE energy.

For the case where the initial distribution of the parameters is uniform, we can calculate the analytical solution for⟨E0⟩delimited-⟨⟩subscript𝐸0\langle E_{0}\rangle⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩. For example, for the two-dimensional case:

⟨E0⟩=∫1n⁢∑i|μ1⁢xi+μ2−yi|⁢d⁢μ1l1⁢d⁢μ2l2,delimited-⟨⟩subscript𝐸01𝑛subscript𝑖subscript𝜇1subscript𝑥𝑖subscript𝜇2subscript𝑦𝑖𝑑subscript𝜇1subscript𝑙1𝑑subscript𝜇2subscript𝑙2\displaystyle\langle E_{0}\rangle=\int\frac{1}{n}\sum_{i}|\mu_{1}x_{i}+\mu_{2}% -y_{i}|\frac{d\mu_{1}}{l_{1}}\frac{d\mu_{2}}{l_{2}},⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩ = ∫ divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT | italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT + italic_μ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT | divide start_ARG italic_d italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_ARG start_ARG italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_ARG divide start_ARG italic_d italic_μ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_ARG start_ARG italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_ARG ,(3.40)we derive

6⁢n⁢l1⁢l2⁢xi⁢⟨E0⟩6𝑛subscript𝑙1subscript𝑙2subscript𝑥𝑖delimited-⟨⟩subscript𝐸0\displaystyle 6nl_{1}l_{2}x_{i}\langle E_{0}\rangle6 italic_n italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩=\displaystyle==∑(l12⁢xi+l22−yi)2⁢|l12⁢xi+l22−yi|superscriptsubscript𝑙12subscript𝑥𝑖subscript𝑙22subscript𝑦𝑖2subscript𝑙12subscript𝑥𝑖subscript𝑙22subscript𝑦𝑖\displaystyle\sum\left(\frac{l_{1}}{2}x_{i}+\frac{l_{2}}{2}-y_{i}\right)^{2}% \left|\frac{l_{1}}{2}x_{i}+\frac{l_{2}}{2}-y_{i}\right|∑ ( divide start_ARG italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_ARG start_ARG 2 end_ARG italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT + divide start_ARG italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_ARG start_ARG 2 end_ARG - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT | divide start_ARG italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_ARG start_ARG 2 end_ARG italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT + divide start_ARG italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_ARG start_ARG 2 end_ARG - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT |(3.41)−\displaystyle--∑(l12⁢xi−l22+yi)2⁢|l12⁢xi−l22+yi|superscriptsubscript𝑙12subscript𝑥𝑖subscript𝑙22subscript𝑦𝑖2subscript𝑙12subscript𝑥𝑖subscript𝑙22subscript𝑦𝑖\displaystyle\sum\left(\frac{l_{1}}{2}x_{i}-\frac{l_{2}}{2}+y_{i}\right)^{2}% \left|\frac{l_{1}}{2}x_{i}-\frac{l_{2}}{2}+y_{i}\right|∑ ( divide start_ARG italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_ARG start_ARG 2 end_ARG italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT - divide start_ARG italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_ARG start_ARG 2 end_ARG + italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT | divide start_ARG italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_ARG start_ARG 2 end_ARG italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT - divide start_ARG italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_ARG start_ARG 2 end_ARG + italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT |−\displaystyle--∑(l12⁢xi−l22−yi)2⁢|l12⁢xi−l22−yi|superscriptsubscript𝑙12subscript𝑥𝑖subscript𝑙22subscript𝑦𝑖2subscript𝑙12subscript𝑥𝑖subscript𝑙22subscript𝑦𝑖\displaystyle\sum\left(\frac{l_{1}}{2}x_{i}-\frac{l_{2}}{2}-y_{i}\right)^{2}% \left|\frac{l_{1}}{2}x_{i}-\frac{l_{2}}{2}-y_{i}\right|∑ ( divide start_ARG italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_ARG start_ARG 2 end_ARG italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT - divide start_ARG italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_ARG start_ARG 2 end_ARG - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT | divide start_ARG italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_ARG start_ARG 2 end_ARG italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT - divide start_ARG italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_ARG start_ARG 2 end_ARG - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT |+\displaystyle++∑(l12⁢xi+l22+yi)2⁢|l12⁢xi+l22+yi|.superscriptsubscript𝑙12subscript𝑥𝑖subscript𝑙22subscript𝑦𝑖2subscript𝑙12subscript𝑥𝑖subscript𝑙22subscript𝑦𝑖\displaystyle\sum\left(\frac{l_{1}}{2}x_{i}+\frac{l_{2}}{2}+y_{i}\right)^{2}% \left|\frac{l_{1}}{2}x_{i}+\frac{l_{2}}{2}+y_{i}\right|.∑ ( divide start_ARG italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_ARG start_ARG 2 end_ARG italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT + divide start_ARG italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_ARG start_ARG 2 end_ARG + italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT | divide start_ARG italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_ARG start_ARG 2 end_ARG italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT + divide start_ARG italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_ARG start_ARG 2 end_ARG + italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT | .Forl1≫l2much-greater-thansubscript𝑙1subscript𝑙2l_{1}\gg l_{2}italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ≫ italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT, from Equation3.41we can obtain⟨E0⟩∼14⁢l1⁢|xi|¯similar-todelimited-⟨⟩subscript𝐸014subscript𝑙1¯subscript𝑥𝑖\langle E_{0}\rangle\sim\frac{1}{4}l_{1}\overline{|x_{i}|}⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩ ∼ divide start_ARG 1 end_ARG start_ARG 4 end_ARG italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT over¯ start_ARG | italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT | end_ARG. Similarly, forl2≫l1much-greater-thansubscript𝑙2subscript𝑙1l_{2}\gg l_{1}italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ≫ italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT, we obtain⟨E0⟩∼14⁢l2similar-todelimited-⟨⟩subscript𝐸014subscript𝑙2\langle E_{0}\rangle\sim\frac{1}{4}l_{2}⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩ ∼ divide start_ARG 1 end_ARG start_ARG 4 end_ARG italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT.

For high dimensions,

⟨E0⟩delimited-⟨⟩subscript𝐸0\displaystyle\langle E_{0}\rangle⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩=\displaystyle==∫1n⁢∑i|∑j=2K−1μj⁢xi⁢j+μK−yi|⁢∏j=1K(d⁢μjlj).1𝑛subscript𝑖superscriptsubscript𝑗2𝐾1subscript𝜇𝑗subscript𝑥𝑖𝑗subscript𝜇𝐾subscript𝑦𝑖superscriptsubscriptproduct𝑗1𝐾𝑑subscript𝜇𝑗subscript𝑙𝑗\displaystyle\int\frac{1}{n}\sum_{i}\left|\sum_{j=2}^{K-1}\mu_{j}x_{ij}+\mu_{K% }-y_{i}\right|\prod_{j=1}^{K}\left(\frac{d\mu_{j}}{l_{j}}\right).∫ divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT | ∑ start_POSTSUBSCRIPT italic_j = 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT + italic_μ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT | ∏ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT ( divide start_ARG italic_d italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT end_ARG start_ARG italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT end_ARG ) .(3.42)We can get the asymptotic solution of the energy

⟨E0⟩∼14⁢[l1⁢|xi⁢1|+l2⁢|xi⁢2|+l3⁢|xi⁢3|+…+lk1⁢|xi,K−1|+lK],similar-todelimited-⟨⟩subscript𝐸014delimited-[]subscript𝑙1subscript𝑥𝑖1subscript𝑙2subscript𝑥𝑖2subscript𝑙3subscript𝑥𝑖3…subscript𝑙subscript𝑘1subscript𝑥𝑖𝐾1subscript𝑙𝐾\langle E_{0}\rangle\sim\frac{1}{4}[l_{1}|x_{i1}|+l_{2}|x_{i2}|+l_{3}|x_{i3}|+% ...+l_{k_{1}}|x_{i,K-1}|+l_{K}],⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩ ∼ divide start_ARG 1 end_ARG start_ARG 4 end_ARG [ italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT | italic_x start_POSTSUBSCRIPT italic_i 1 end_POSTSUBSCRIPT | + italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT | italic_x start_POSTSUBSCRIPT italic_i 2 end_POSTSUBSCRIPT | + italic_l start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT | italic_x start_POSTSUBSCRIPT italic_i 3 end_POSTSUBSCRIPT | + … + italic_l start_POSTSUBSCRIPT italic_k start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPT | italic_x start_POSTSUBSCRIPT italic_i , italic_K - 1 end_POSTSUBSCRIPT | + italic_l start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT ] ,(3.43)and the temperature is

T∼∑j=1Klj⁢|xi⁢j|4⁢∑j=1Kln⁡lj.similar-to𝑇superscriptsubscript𝑗1𝐾subscript𝑙𝑗subscript𝑥𝑖𝑗4superscriptsubscript𝑗1𝐾subscript𝑙𝑗T\sim\frac{\sum_{j=1}^{K}l_{j}|x_{ij}|}{4\sum_{j=1}^{K}\ln l_{j}}.italic_T ∼ divide start_ARG ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT | italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT | end_ARG start_ARG 4 ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT roman_ln italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT end_ARG .(3.44)For{li}→l→∞→subscript𝑙𝑖𝑙→\{l_{i}\}\rightarrow l\rightarrow\infty{ italic_l start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT } → italic_l → ∞, we have

T∼14⁢(lln⁡l)⁢|X¯|.similar-to𝑇14𝑙𝑙¯𝑋T\sim\frac{1}{4}\left(\frac{l}{\ln l}\right)|\overline{X}|.italic_T ∼ divide start_ARG 1 end_ARG start_ARG 4 end_ARG ( divide start_ARG italic_l end_ARG start_ARG roman_ln italic_l end_ARG ) | over¯ start_ARG italic_X end_ARG | .(3.45)Compare systems under same conditions but with different energy forms,

TMAE∝lln⁡l≪TMSE∝l2ln⁡l.proportional-tosubscript𝑇MAE𝑙𝑙much-less-thansubscript𝑇MSEproportional-tosuperscript𝑙2𝑙T_{\rm MAE}\propto\frac{l}{\ln l}\ll T_{\rm MSE}\propto\frac{l^{2}}{\ln l}.italic_T start_POSTSUBSCRIPT roman_MAE end_POSTSUBSCRIPT ∝ divide start_ARG italic_l end_ARG start_ARG roman_ln italic_l end_ARG ≪ italic_T start_POSTSUBSCRIPT roman_MSE end_POSTSUBSCRIPT ∝ divide start_ARG italic_l start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_l end_ARG .(3.46)Again, we find that MSE gives much higher temperature than MAE.

3.6Physical Explanation of Temperature

Let us consider two ML systemsA𝐴Aitalic_AandB𝐵Bitalic_B. For simplification, we assume the systems have a linear regression model structure and MSE energy form, and the initial normal distribution of the parameters is determined by a sufficiently largeσ𝜎\sigmaitalic_σ111For the system with initial uniform distribution of parameters, we can use a sufficiently largel𝑙litalic_lto derive very similar results in this section.. Thus, the temperature of the system can be calculated by Equation (3.11), which means the system temperature is determined by data distributionX𝑋Xitalic_Xand the initialσ𝜎\sigmaitalic_σfrom the parameter space.

Figure 6:ML systems: mixing (left panel) and training (right panel). For two systemsA𝐴Aitalic_AandB𝐵Bitalic_Bthat have not yet reached equilibrium, suppose they have the same parameter distribution (σ𝜎\sigmaitalic_σ), but different temperatures and energies(TA,EA)subscript𝑇𝐴subscript𝐸𝐴(T_{A},E_{A})( italic_T start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT , italic_E start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT )and(TB,EB)subscript𝑇𝐵subscript𝐸𝐵(T_{B},E_{B})( italic_T start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT , italic_E start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT )respectively. IfTA>TBsubscript𝑇𝐴subscript𝑇𝐵T_{A}>T_{B}italic_T start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT > italic_T start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT, then after mixing, the system’s temperatureTA∪Bsubscript𝑇𝐴𝐵T_{A\cup B}italic_T start_POSTSUBSCRIPT italic_A ∪ italic_B end_POSTSUBSCRIPTis betweenTAsubscript𝑇𝐴T_{A}italic_T start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPTandTBsubscript𝑇𝐵T_{B}italic_T start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT, with the direction of energy flow fromA𝐴Aitalic_AtoB𝐵Bitalic_B, meaning the energy of systemA𝐴Aitalic_Adecreases while the energy of systemB𝐵Bitalic_Bincreases. This process by which systemsA𝐴Aitalic_AandB𝐵Bitalic_Breach equilibrium can be analogized to a physical process in thermodynamic systems, with very strong similarities. On the other hand, if the data of a system changes fromA𝐴Aitalic_AtoC𝐶Citalic_Cand needs to be retrained, we can view the training process as the mixing process of systemsA𝐴Aitalic_AandB𝐵Bitalic_B, whereB=C−A𝐵𝐶𝐴B=C-Aitalic_B = italic_C - italic_A.##### Equilibrium Systems

From the above assumption we can obtain the temperatures of the two systems:

TA∼σ2ln⁡σ⁢XA2¯,TB∼σ2ln⁡σ⁢XB2¯,formulae-sequencesimilar-tosubscript𝑇𝐴superscript𝜎2𝜎¯superscriptsubscript𝑋𝐴2similar-tosubscript𝑇𝐵superscript𝜎2𝜎¯superscriptsubscript𝑋𝐵2T_{A}\sim\frac{\sigma^{2}}{\ln\sigma}\overline{X_{A}^{2}},\quad T_{B}\sim\frac% {\sigma^{2}}{\ln\sigma}\overline{X_{B}^{2}},italic_T start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT ∼ divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ end_ARG over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG , italic_T start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT ∼ divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ end_ARG over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ,(3.47)IfXAsubscript𝑋𝐴X_{A}italic_X start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPTandXBsubscript𝑋𝐵X_{B}italic_X start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPThave the same data distribution, i.e.,XA2¯=XB2¯¯superscriptsubscript𝑋𝐴2¯superscriptsubscript𝑋𝐵2\overline{X_{A}^{2}}=\overline{X_{B}^{2}}over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG = over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG, we can deriveTA=TBsubscript𝑇𝐴subscript𝑇𝐵T_{A}=T_{B}italic_T start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT = italic_T start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT.

Next we mix systemsA𝐴Aitalic_AandB𝐵Bitalic_Bto form a new systemA∪B𝐴𝐵A\cup Bitalic_A ∪ italic_B, i.e., the data of the two systems are combined. SinceXA2¯=XB2¯=XA∪B2¯¯superscriptsubscript𝑋𝐴2¯superscriptsubscript𝑋𝐵2¯superscriptsubscript𝑋𝐴𝐵2\overline{X_{A}^{2}}=\overline{X_{B}^{2}}=\overline{X_{A\cup B}^{2}}over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG = over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG = over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_A ∪ italic_B end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG, we get the temperature of the mixed system that

TA∪B∼σ2ln⁡σ⁢XA∪B2=TA=TB.similar-tosubscript𝑇𝐴𝐵superscript𝜎2𝜎superscriptsubscript𝑋𝐴𝐵2subscript𝑇𝐴subscript𝑇𝐵T_{A\cup B}\sim\frac{\sigma^{2}}{\ln\sigma}X_{A\cup B}^{2}=T_{A}=T_{B}.italic_T start_POSTSUBSCRIPT italic_A ∪ italic_B end_POSTSUBSCRIPT ∼ divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ end_ARG italic_X start_POSTSUBSCRIPT italic_A ∪ italic_B end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT = italic_T start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT = italic_T start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT .(3.48)From a physical perspective, systemsA𝐴Aitalic_AandB𝐵Bitalic_Bare in equilibrium with the same temperature. Therefore, systems with the same temperature still have the same temperature after mixture.

Non-Equilibrium Systems

Now let us image that the data distributions of systemsA𝐴Aitalic_AandB𝐵Bitalic_Bare significantly different, such as

x>1for∀x∈XAformulae-sequence𝑥1forfor-all𝑥subscript𝑋𝐴\displaystyle x>1\quad\textrm{for}\quad\forall x\in X_{A}italic_x > 1 for ∀ italic_x ∈ italic_X start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT(3.49)x<1for∀x∈XB.formulae-sequence𝑥1forfor-all𝑥subscript𝑋𝐵\displaystyle x<1\quad\textrm{for}\quad\forall x\in X_{B}.italic_x < 1 for ∀ italic_x ∈ italic_X start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT .(3.50)Thus, the temperatures of the two systems are

TA∼σ2ln⁡σ⁢XA2¯>TB∼σ2ln⁡σ⁢XB2¯.similar-tosubscript𝑇𝐴superscript𝜎2𝜎¯superscriptsubscript𝑋𝐴2subscript𝑇𝐵similar-tosuperscript𝜎2𝜎¯superscriptsubscript𝑋𝐵2T_{A}\sim\frac{\sigma^{2}}{\ln\sigma}\overline{X_{A}^{2}}>T_{B}\sim\frac{% \sigma^{2}}{\ln\sigma}\overline{X_{B}^{2}}.italic_T start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT ∼ divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ end_ARG over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG > italic_T start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT ∼ divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ end_ARG over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG .(3.51)For the mixed systemA∪B𝐴𝐵A\cup Bitalic_A ∪ italic_B, sinceXB2¯<XA∪B2¯<XA2¯¯superscriptsubscript𝑋𝐵2¯superscriptsubscript𝑋𝐴𝐵2¯superscriptsubscript𝑋𝐴2\overline{X_{B}^{2}}<\overline{X_{A\cup B}^{2}}<\overline{X_{A}^{2}}over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG < over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_A ∪ italic_B end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG < over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG, we can get

TB<TA∪B∼σ2ln⁡σ⁢XA∪B2¯<TA.subscript𝑇𝐵subscript𝑇𝐴𝐵similar-tosuperscript𝜎2𝜎¯superscriptsubscript𝑋𝐴𝐵2subscript𝑇𝐴T_{B}<T_{A\cup B}\sim\frac{\sigma^{2}}{\ln\sigma}\overline{X_{A\cup B}^{2}}<T_% {A}.italic_T start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT < italic_T start_POSTSUBSCRIPT italic_A ∪ italic_B end_POSTSUBSCRIPT ∼ divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ end_ARG over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_A ∪ italic_B end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG < italic_T start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT .(3.52)The temperature of the mixed systemA∪B𝐴𝐵A\cup Bitalic_A ∪ italic_Bis between the two pre-mixed systems.

Note that the energy of systemsA𝐴Aitalic_AandB𝐵Bitalic_Bare

EA∼KA⁢XA2¯,EB∼KB⁢XB2¯,formulae-sequencesimilar-tosubscript𝐸𝐴subscript𝐾𝐴¯superscriptsubscript𝑋𝐴2similar-tosubscript𝐸𝐵subscript𝐾𝐵¯superscriptsubscript𝑋𝐵2\displaystyle E_{A}\sim K_{A}\overline{X_{A}^{2}},\quad E_{B}\sim K_{B}% \overline{X_{B}^{2}},italic_E start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT ∼ italic_K start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG , italic_E start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT ∼ italic_K start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ,(3.53)whereKAsubscript𝐾𝐴K_{A}italic_K start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPTandKBsubscript𝐾𝐵K_{B}italic_K start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPTare the dimensions of parameter space inA𝐴Aitalic_AandB𝐵Bitalic_B, respectively. The mixed system redistributed energy as

EA′∼KA⁢XA∪B2¯,EB′∼KB⁢XA∪B2¯.formulae-sequencesimilar-tosuperscriptsubscript𝐸𝐴′subscript𝐾𝐴¯superscriptsubscript𝑋𝐴𝐵2similar-tosuperscriptsubscript𝐸𝐵′subscript𝐾𝐵¯superscriptsubscript𝑋𝐴𝐵2\displaystyle E_{A}^{\prime}\sim K_{A}\overline{X_{A\cup B}^{2}},\quad E_{B}^{% \prime}\sim K_{B}\overline{X_{A\cup B}^{2}}.italic_E start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ∼ italic_K start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_A ∪ italic_B end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG , italic_E start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT ∼ italic_K start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_A ∪ italic_B end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG .(3.54)ForTA≥TBsubscript𝑇𝐴subscript𝑇𝐵T_{A}\geq T_{B}italic_T start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT ≥ italic_T start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT, the energy changes inA𝐴Aitalic_AandB𝐵Bitalic_Bare

Δ⁢EAΔsubscript𝐸𝐴\displaystyle\Delta E_{A}roman_Δ italic_E start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT=\displaystyle==KA⁢σ2⁢(XA∪B2¯−XA2¯)<0,subscript𝐾𝐴superscript𝜎2¯superscriptsubscript𝑋𝐴𝐵2¯superscriptsubscript𝑋𝐴20\displaystyle K_{A}\sigma^{2}(\overline{X_{A\cup B}^{2}}-\overline{X_{A}^{2}})% <0,italic_K start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ( over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_A ∪ italic_B end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG - over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ) < 0 ,(3.55)Δ⁢EBΔsubscript𝐸𝐵\displaystyle\Delta E_{B}roman_Δ italic_E start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT=\displaystyle==KB⁢σ2⁢(XA∪B2¯−XB2¯)>0.subscript𝐾𝐵superscript𝜎2¯superscriptsubscript𝑋𝐴𝐵2¯superscriptsubscript𝑋𝐵20\displaystyle K_{B}\sigma^{2}(\overline{X_{A\cup B}^{2}}-\overline{X_{B}^{2}})% >0.italic_K start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ( over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_A ∪ italic_B end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG - over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ) > 0 .(3.56)The energy is transferred from systemA𝐴Aitalic_Ato systemB𝐵Bitalic_B. In other words, the energy flows from a high-temperature, high-energy system to a low-temperature, low-energy system once they reach equilibrium. This phenomenon aligns with the behavior of physical systems in a thermodynamic context.

Physical Scenario of Model Retraining

Now, let us consider a ML system with dataXAsubscript𝑋𝐴X_{A}italic_X start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT. The system obtains new dataXBsubscript𝑋𝐵X_{B}italic_X start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPTwith a different distribution thatXA2¯≠XB2¯¯superscriptsubscript𝑋𝐴2¯superscriptsubscript𝑋𝐵2\overline{X_{A}^{2}}\neq\overline{X_{B}^{2}}over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ≠ over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG. We mixA𝐴Aitalic_AandB𝐵Bitalic_Btogether to form a new systemA∪B𝐴𝐵A\cup Bitalic_A ∪ italic_Band retrain the model. Since two non-equilibrium systems are mixed, the temperatureTA∪Bsubscript𝑇𝐴𝐵T_{A\cup B}italic_T start_POSTSUBSCRIPT italic_A ∪ italic_B end_POSTSUBSCRIPTof the new system has the following characteristics: IfTA<TBsubscript𝑇𝐴subscript𝑇𝐵T_{A}<T_{B}italic_T start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT < italic_T start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT, we haveTA<TA∪B<TBsubscript𝑇𝐴subscript𝑇𝐴𝐵subscript𝑇𝐵T_{A}<T_{A\cup B}<T_{B}italic_T start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT < italic_T start_POSTSUBSCRIPT italic_A ∪ italic_B end_POSTSUBSCRIPT < italic_T start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT, else ifTA>TBsubscript𝑇𝐴subscript𝑇𝐵T_{A}>T_{B}italic_T start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT > italic_T start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT, we haveTA>TA∪B>TBsubscript𝑇𝐴subscript𝑇𝐴𝐵subscript𝑇𝐵T_{A}>T_{A\cup B}>T_{B}italic_T start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT > italic_T start_POSTSUBSCRIPT italic_A ∪ italic_B end_POSTSUBSCRIPT > italic_T start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT.

That is to say, if the new data added to the ML system has a higher temperature, then the temperature of the system after retraining will also increase, and vice versa. The scenarios discussed above temperature changes in equilibrium, non-equilibrium, and model retraining systems are very similar to traditional thermodynamic systems. This is the thermodynamic landscape of the temperature in ML systems.

4Logistic Regression with Cross Entropy

4.1Parameter Initialization: Normal Distribution

We move forward to discuss ML systems with cross entropy energy, which is written as

Epsubscript𝐸𝑝\displaystyle E_{p}italic_E start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT=\displaystyle==−∑i1n⁢[yi⁢ln⁡yi^+(1−yi)⁢ln⁡(1−y^i)]subscript𝑖1𝑛delimited-[]subscript𝑦𝑖^subscript𝑦𝑖1subscript𝑦𝑖1subscript^𝑦𝑖\displaystyle-\sum_{i}\frac{1}{n}[y_{i}\ln\hat{y_{i}}+(1-y_{i})\ln(1-\hat{y}_{% i})]- ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT divide start_ARG 1 end_ARG start_ARG italic_n end_ARG [ italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT roman_ln over^ start_ARG italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG + ( 1 - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) roman_ln ( 1 - over^ start_ARG italic_y end_ARG start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) ](4.1)=\displaystyle==∑i1n⁢[ln⁡(1+e−zi)+(1−yi)⁢zi],subscript𝑖1𝑛delimited-[]1superscriptesubscript𝑧𝑖1subscript𝑦𝑖subscript𝑧𝑖\displaystyle\sum_{i}\frac{1}{n}[\ln(1+\textrm{e}^{-z_{i}})+(1-y_{i})z_{i}],∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT divide start_ARG 1 end_ARG start_ARG italic_n end_ARG [ roman_ln ( 1 + e start_POSTSUPERSCRIPT - italic_z start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ) + ( 1 - italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) italic_z start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ] ,givenzi=∑j=1K−1μj⁢xi⁢j+μKsubscript𝑧𝑖superscriptsubscript𝑗1𝐾1subscript𝜇𝑗subscript𝑥𝑖𝑗subscript𝜇𝐾z_{i}=\sum_{j=1}^{K-1}\mu_{j}x_{ij}+\mu_{K}italic_z start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT = ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT + italic_μ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT.

Note that

∫zi⁢f⁢(𝝁)⁢𝑑𝝁=0subscript𝑧𝑖𝑓𝝁differential-d𝝁0\int z_{i}f({\boldsymbol{\mu}})d{\boldsymbol{\mu}}=0∫ italic_z start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_f ( bold_italic_μ ) italic_d bold_italic_μ = 0(4.2)if the distribution functionf⁢(𝝁)𝑓𝝁f({\boldsymbol{\mu}})italic_f ( bold_italic_μ )is an even function. Thus, the average energy of the initial state with parameter normal distribution is

⟨E0⟩delimited-⟨⟩subscript𝐸0\displaystyle\langle E_{0}\rangle⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩=\displaystyle==∫Ep⁢f⁢(𝝁)⁢𝑑𝝁=∫∑i1n⁢ln⁡(1+e−zi)⁢∏j(12⁢π⁢σj⁢e−μj22⁢σj2⁢d⁢μj)subscript𝐸𝑝𝑓𝝁differential-d𝝁subscript𝑖1𝑛1superscriptesubscript𝑧𝑖subscriptproduct𝑗12𝜋subscript𝜎𝑗superscriptesuperscriptsubscript𝜇𝑗22superscriptsubscript𝜎𝑗2𝑑subscript𝜇𝑗\displaystyle\int E_{p}f({\boldsymbol{\mu}})d{\boldsymbol{\mu}}=\int\sum_{i}% \frac{1}{n}\ln(1+\textrm{e}^{-z_{i}})\prod_{j}\left(\frac{1}{\sqrt{2\pi}\sigma% _{j}}\textrm{e}^{-\frac{\mu_{j}^{2}}{2\sigma_{j}^{2}}}d\mu_{j}\right)∫ italic_E start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT italic_f ( bold_italic_μ ) italic_d bold_italic_μ = ∫ ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT divide start_ARG 1 end_ARG start_ARG italic_n end_ARG roman_ln ( 1 + e start_POSTSUPERSCRIPT - italic_z start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ) ∏ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT ( divide start_ARG 1 end_ARG start_ARG square-root start_ARG 2 italic_π end_ARG italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT end_ARG e start_POSTSUPERSCRIPT - divide start_ARG italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG end_POSTSUPERSCRIPT italic_d italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT )(4.3)=\displaystyle==∫∑i1n⁢ln⁡[1+exp⁡(−∑j=1K−1μj⁢σj⁢xi⁢j−μK⁢σK)]⁢∏j(12⁢π⁢e−μj22⁢d⁢μj).subscript𝑖1𝑛1superscriptsubscript𝑗1𝐾1subscript𝜇𝑗subscript𝜎𝑗subscript𝑥𝑖𝑗subscript𝜇𝐾subscript𝜎𝐾subscriptproduct𝑗12𝜋superscriptesuperscriptsubscript𝜇𝑗22𝑑subscript𝜇𝑗\displaystyle\int\sum_{i}\frac{1}{n}\ln\left[1+\exp\left(-\sum_{j=1}^{K-1}\mu_% {j}\sigma_{j}x_{ij}-\mu_{K}\sigma_{K}\right)\right]\prod_{j}\left(\frac{1}{% \sqrt{2\pi}}\textrm{e}^{-\frac{\mu_{j}^{2}}{2}}d\mu_{j}\right).∫ ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT divide start_ARG 1 end_ARG start_ARG italic_n end_ARG roman_ln [ 1 + roman_exp ( - ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT - italic_μ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT italic_σ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT ) ] ∏ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT ( divide start_ARG 1 end_ARG start_ARG square-root start_ARG 2 italic_π end_ARG end_ARG e start_POSTSUPERSCRIPT - divide start_ARG italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 end_ARG end_POSTSUPERSCRIPT italic_d italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT ) .Since⟨E0⟩delimited-⟨⟩subscript𝐸0\langle E_{0}\rangle⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩does not have an analytic solution, we examine the asymptotic solution of⟨E0⟩delimited-⟨⟩subscript𝐸0\langle E_{0}\rangle⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩for anyσjsubscript𝜎𝑗\sigma_{j}italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPTin{σ1,σ2,…⁢σK}→∞→subscript𝜎1subscript𝜎2…subscript𝜎𝐾\{\sigma_{1},\sigma_{2},...\sigma_{K}\}\rightarrow\infty{ italic_σ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_σ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , … italic_σ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT } → ∞. We discuss this by considering two cases: Case I and Case II as follows.

Case I

:zi′=∑j=1K−1μj⁢σj⁢xi⁢j+μK⁢σK→∞superscriptsubscript𝑧𝑖′superscriptsubscript𝑗1𝐾1subscript𝜇𝑗subscript𝜎𝑗subscript𝑥𝑖𝑗subscript𝜇𝐾subscript𝜎𝐾→z_{i}^{\prime}=\sum_{j=1}^{K-1}\mu_{j}\sigma_{j}x_{ij}+\mu_{K}\sigma_{K}\rightarrow\inftyitalic_z start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT = ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT + italic_μ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT italic_σ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT → ∞. Thus, we haveln⁡(1+e−zi′)→e−zi′→1superscriptesuperscriptsubscript𝑧𝑖′superscriptesuperscriptsubscript𝑧𝑖′\ln(1+\textrm{e}^{-z_{i}^{\prime}})\rightarrow\textrm{e}^{-z_{i}^{\prime}}roman_ln ( 1 + e start_POSTSUPERSCRIPT - italic_z start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT end_POSTSUPERSCRIPT ) → e start_POSTSUPERSCRIPT - italic_z start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT end_POSTSUPERSCRIPT. From Equation (4.3) we can derive

⟨E0⟩isubscriptdelimited-⟨⟩subscript𝐸0𝑖\displaystyle\langle E_{0}\rangle_{i}⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT∼similar-to\displaystyle\sim∼∫1n⁢e−zi′⁢∏j(12⁢π⁢e−μj22⁢d⁢μj)1𝑛superscriptesuperscriptsubscript𝑧𝑖′subscriptproduct𝑗12𝜋superscriptesuperscriptsubscript𝜇𝑗22𝑑subscript𝜇𝑗\displaystyle\int\frac{1}{n}\textrm{e}^{-z_{i}^{\prime}}\prod_{j}\left(\frac{1% }{\sqrt{2\pi}}\textrm{e}^{-\frac{\mu_{j}^{2}}{2}}d\mu_{j}\right)∫ divide start_ARG 1 end_ARG start_ARG italic_n end_ARG e start_POSTSUPERSCRIPT - italic_z start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT end_POSTSUPERSCRIPT ∏ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT ( divide start_ARG 1 end_ARG start_ARG square-root start_ARG 2 italic_π end_ARG end_ARG e start_POSTSUPERSCRIPT - divide start_ARG italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 end_ARG end_POSTSUPERSCRIPT italic_d italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT )(4.4)∼similar-to\displaystyle\sim∼∫12⁢π⁢∏j=1K−1exp⁡[−μj⁢σj⁢xi⁢j−μj22]⁢d⁢μj⁢exp⁡[−μK⁢σK−μK22]⁢d⁢μK12𝜋superscriptsubscriptproduct𝑗1𝐾1subscript𝜇𝑗subscript𝜎𝑗subscript𝑥𝑖𝑗superscriptsubscript𝜇𝑗22𝑑subscript𝜇𝑗subscript𝜇𝐾subscript𝜎𝐾superscriptsubscript𝜇𝐾22𝑑subscript𝜇𝐾\displaystyle\int\frac{1}{\sqrt{2\pi}}\prod_{j=1}^{K-1}\exp\left[-\mu_{j}% \sigma_{j}x_{ij}-\frac{\mu_{j}^{2}}{2}\right]d\mu_{j}\exp\left[-\mu_{K}\sigma_% {K}-\frac{\mu_{K}^{2}}{2}\right]d\mu_{K}∫ divide start_ARG 1 end_ARG start_ARG square-root start_ARG 2 italic_π end_ARG end_ARG ∏ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT roman_exp [ - italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT - divide start_ARG italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 end_ARG ] italic_d italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT roman_exp [ - italic_μ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT italic_σ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT - divide start_ARG italic_μ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 end_ARG ] italic_d italic_μ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT∼similar-to\displaystyle\sim∼1n⁢exp⁡[12⁢∑j=1K−1σj2⁢xi⁢j2+12⁢σK2].1𝑛12superscriptsubscript𝑗1𝐾1superscriptsubscript𝜎𝑗2superscriptsubscript𝑥𝑖𝑗212superscriptsubscript𝜎𝐾2\displaystyle\frac{1}{n}\exp\left[\frac{1}{2}\sum_{j=1}^{K-1}\sigma_{j}^{2}x_{% ij}^{2}+\frac{1}{2}\sigma_{K}^{2}\right].divide start_ARG 1 end_ARG start_ARG italic_n end_ARG roman_exp [ divide start_ARG 1 end_ARG start_ARG 2 end_ARG ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + divide start_ARG 1 end_ARG start_ARG 2 end_ARG italic_σ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ] .

Case II

:zi′=∑j=1K−1μj⁢σj⁢xi⁢j+μK⁢σK→−∞superscriptsubscript𝑧𝑖′superscriptsubscript𝑗1𝐾1subscript𝜇𝑗subscript𝜎𝑗subscript𝑥𝑖𝑗subscript𝜇𝐾subscript𝜎𝐾→z_{i}^{\prime}=\sum_{j=1}^{K-1}\mu_{j}\sigma_{j}x_{ij}+\mu_{K}\sigma_{K}% \rightarrow-\inftyitalic_z start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT = ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT + italic_μ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT italic_σ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT → - ∞. In this case, we haveln⁡(1+e−zi′)=ln⁡[e−zi′⁢(1+ezi′)]=−zi′+ln⁡(1+ezi′)→−zi′+ezi′1superscriptesuperscriptsubscript𝑧𝑖′superscriptesuperscriptsubscript𝑧𝑖′1superscriptesuperscriptsubscript𝑧𝑖′superscriptsubscript𝑧𝑖′1superscriptesuperscriptsubscript𝑧𝑖′→superscriptsubscript𝑧𝑖′superscriptesuperscriptsubscript𝑧𝑖′\ln(1+\textrm{e}^{-z_{i}^{\prime}})=\ln[\textrm{e}^{-z_{i}^{\prime}}(1+\textrm% {e}^{z_{i}^{\prime}})]=-z_{i}^{\prime}+\ln(1+\textrm{e}^{z_{i}^{\prime}})% \rightarrow-z_{i}^{\prime}+\textrm{e}^{z_{i}^{\prime}}roman_ln ( 1 + e start_POSTSUPERSCRIPT - italic_z start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT end_POSTSUPERSCRIPT ) = roman_ln [ e start_POSTSUPERSCRIPT - italic_z start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT end_POSTSUPERSCRIPT ( 1 + e start_POSTSUPERSCRIPT italic_z start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT end_POSTSUPERSCRIPT ) ] = - italic_z start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT + roman_ln ( 1 + e start_POSTSUPERSCRIPT italic_z start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT end_POSTSUPERSCRIPT ) → - italic_z start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT + e start_POSTSUPERSCRIPT italic_z start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT end_POSTSUPERSCRIPT. Note that∫zi′⁢f⁢(𝝁)⁢𝑑𝝁=0superscriptsubscript𝑧𝑖′𝑓𝝁differential-d𝝁0\int z_{i}^{\prime}f({\boldsymbol{\mu}})d{\boldsymbol{\mu}}=0∫ italic_z start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT italic_f ( bold_italic_μ ) italic_d bold_italic_μ = 0also holds, so Equation (4.3) becomes

⟨E0⟩delimited-⟨⟩subscript𝐸0\displaystyle\langle E_{0}\rangle⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩∼similar-to\displaystyle\sim∼∫1n⁢exp⁡[∑j=1K−1μj⁢σj⁢xi⁢j+μK⁢σK]⁢∏j(12⁢π⁢e−μj22⁢d⁢μj)1𝑛superscriptsubscript𝑗1𝐾1subscript𝜇𝑗subscript𝜎𝑗subscript𝑥𝑖𝑗subscript𝜇𝐾subscript𝜎𝐾subscriptproduct𝑗12𝜋superscriptesuperscriptsubscript𝜇𝑗22𝑑subscript𝜇𝑗\displaystyle\int\frac{1}{n}\exp\left[\sum_{j=1}^{K-1}\mu_{j}\sigma_{j}x_{ij}+% \mu_{K}\sigma_{K}\right]\prod_{j}\left(\frac{1}{\sqrt{2\pi}}\textrm{e}^{-\frac% {\mu_{j}^{2}}{2}}d\mu_{j}\right)∫ divide start_ARG 1 end_ARG start_ARG italic_n end_ARG roman_exp [ ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT + italic_μ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT italic_σ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT ] ∏ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT ( divide start_ARG 1 end_ARG start_ARG square-root start_ARG 2 italic_π end_ARG end_ARG e start_POSTSUPERSCRIPT - divide start_ARG italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 end_ARG end_POSTSUPERSCRIPT italic_d italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT )(4.5)∼similar-to\displaystyle\sim∼1n⁢exp⁡[12⁢∑j=1K−1σj2⁢xi⁢j2+12⁢σK2],1𝑛12superscriptsubscript𝑗1𝐾1superscriptsubscript𝜎𝑗2superscriptsubscript𝑥𝑖𝑗212superscriptsubscript𝜎𝐾2\displaystyle\frac{1}{n}\exp\left[\frac{1}{2}\sum_{j=1}^{K-1}\sigma_{j}^{2}x_{% ij}^{2}+\frac{1}{2}\sigma_{K}^{2}\right],divide start_ARG 1 end_ARG start_ARG italic_n end_ARG roman_exp [ divide start_ARG 1 end_ARG start_ARG 2 end_ARG ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + divide start_ARG 1 end_ARG start_ARG 2 end_ARG italic_σ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ] ,which is exactly the same as Equation (4.4). So we can write

⟨E0⟩delimited-⟨⟩subscript𝐸0\displaystyle\langle E_{0}\rangle⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩∼similar-to\displaystyle\sim∼exp⁡[σK22]⁢exp⁡[12⁢∑j=1K−1σj2⁢xi⁢j2]¯superscriptsubscript𝜎𝐾22¯12superscriptsubscript𝑗1𝐾1superscriptsubscript𝜎𝑗2superscriptsubscript𝑥𝑖𝑗2\displaystyle\exp\left[\frac{\sigma_{K}^{2}}{2}\right]\overline{\exp\left[% \frac{1}{2}\sum_{j=1}^{K-1}\sigma_{j}^{2}x_{ij}^{2}\right]}roman_exp [ divide start_ARG italic_σ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 end_ARG ] over¯ start_ARG roman_exp [ divide start_ARG 1 end_ARG start_ARG 2 end_ARG ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ] end_ARG(4.6)>\displaystyle>>eσK2/2⁢exp⁡[12⁢σj2⁢(xjmin)2]superscriptesuperscriptsubscript𝜎𝐾2212superscriptsubscript𝜎𝑗2superscriptsuperscriptsubscript𝑥𝑗min2\displaystyle\textrm{e}^{\sigma_{K}^{2}/2}\exp\left[\frac{1}{2}\sigma_{j}^{2}(% x_{j}^{\rm min})^{2}\right]e start_POSTSUPERSCRIPT italic_σ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT / 2 end_POSTSUPERSCRIPT roman_exp [ divide start_ARG 1 end_ARG start_ARG 2 end_ARG italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ( italic_x start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_min end_POSTSUPERSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ]>\displaystyle>>exp⁡[12⁢σj2⁢(xjmin)2],12superscriptsubscript𝜎𝑗2superscriptsuperscriptsubscript𝑥𝑗min2\displaystyle\exp\left[\frac{1}{2}\sigma_{j}^{2}(x_{j}^{\rm min})^{2}\right],roman_exp [ divide start_ARG 1 end_ARG start_ARG 2 end_ARG italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ( italic_x start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_min end_POSTSUPERSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ] ,(4.7)wherexjmin=min⁢{x1⁢j,x2⁢j,…⁢xn⁢j}superscriptsubscript𝑥𝑗minminsubscript𝑥1𝑗subscript𝑥2𝑗…subscript𝑥𝑛𝑗x_{j}^{\rm min}={\rm min}\{x_{1j},x_{2j},...x_{nj}\}italic_x start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_min end_POSTSUPERSCRIPT = roman_min { italic_x start_POSTSUBSCRIPT 1 italic_j end_POSTSUBSCRIPT , italic_x start_POSTSUBSCRIPT 2 italic_j end_POSTSUBSCRIPT , … italic_x start_POSTSUBSCRIPT italic_n italic_j end_POSTSUBSCRIPT }. Therefore, the temperature of the system is

T∼⟨E0⟩−Ef∑jln⁡σj+K2⁢[1+ln⁡(2⁢π)]>eσj2⁢(xjmin)2/2−Ef∑jln⁡σj+K2⁢[1+ln⁡(2⁢π)].similar-to𝑇delimited-⟨⟩subscript𝐸0subscript𝐸𝑓subscript𝑗subscript𝜎𝑗𝐾2delimited-[]12𝜋superscriptesuperscriptsubscript𝜎𝑗2superscriptsuperscriptsubscript𝑥𝑗min22subscript𝐸𝑓subscript𝑗subscript𝜎𝑗𝐾2delimited-[]12𝜋\displaystyle T\sim\frac{\langle E_{0}\rangle-E_{f}}{\sum_{j}\ln\sigma_{j}+% \frac{K}{2}[1+\ln(2\pi)]}>\frac{\textrm{e}^{\sigma_{j}^{2}(x_{j}^{\rm min})^{2% }/2}-E_{f}}{\sum_{j}\ln\sigma_{j}+\frac{K}{2}[1+\ln(2\pi)]}.italic_T ∼ divide start_ARG ⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩ - italic_E start_POSTSUBSCRIPT italic_f end_POSTSUBSCRIPT end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT roman_ln italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT + divide start_ARG italic_K end_ARG start_ARG 2 end_ARG [ 1 + roman_ln ( 2 italic_π ) ] end_ARG > divide start_ARG e start_POSTSUPERSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ( italic_x start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_min end_POSTSUPERSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT / 2 end_POSTSUPERSCRIPT - italic_E start_POSTSUBSCRIPT italic_f end_POSTSUBSCRIPT end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT roman_ln italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT + divide start_ARG italic_K end_ARG start_ARG 2 end_ARG [ 1 + roman_ln ( 2 italic_π ) ] end_ARG .(4.8)If we have only one parameterσj∈{σ1,σ2,…⁢σK}→∞subscript𝜎𝑗subscript𝜎1subscript𝜎2…subscript𝜎𝐾→\sigma_{j}\in\{\sigma_{1},\sigma_{2},...\sigma_{K}\}\rightarrow\inftyitalic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT ∈ { italic_σ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_σ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , … italic_σ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT } → ∞, the system temperature is

T≳eσj2⁢(xjmin)2/2−Efln⁡σj∼eσj2⁢(xjmin)2/2ln⁡σj.greater-than-or-equivalent-to𝑇superscriptesuperscriptsubscript𝜎𝑗2superscriptsuperscriptsubscript𝑥𝑗min22subscript𝐸𝑓subscript𝜎𝑗similar-tosuperscriptesuperscriptsubscript𝜎𝑗2superscriptsuperscriptsubscript𝑥𝑗min22subscript𝜎𝑗T\gtrsim\frac{\textrm{e}^{\sigma_{j}^{2}(x_{j}^{\rm min})^{2}/2}-E_{f}}{\ln% \sigma_{j}}\sim\frac{\textrm{e}^{\sigma_{j}^{2}(x_{j}^{\rm min})^{2}/2}}{\ln% \sigma_{j}}.italic_T ≳ divide start_ARG e start_POSTSUPERSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ( italic_x start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_min end_POSTSUPERSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT / 2 end_POSTSUPERSCRIPT - italic_E start_POSTSUBSCRIPT italic_f end_POSTSUBSCRIPT end_ARG start_ARG roman_ln italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT end_ARG ∼ divide start_ARG e start_POSTSUPERSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ( italic_x start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_min end_POSTSUPERSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT / 2 end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT end_ARG .(4.9)For all parameters{σ1,σ2,…,σK}→σ→∞→subscript𝜎1subscript𝜎2…subscript𝜎𝐾𝜎→\{\sigma_{1},\sigma_{2},...,\sigma_{K}\}\rightarrow\sigma\rightarrow\infty{ italic_σ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_σ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , … , italic_σ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT } → italic_σ → ∞, we obtain the temperature

T𝑇\displaystyle Titalic_T∼similar-to\displaystyle\sim∼exp⁡[σ22]⁢exp⁡[12⁢∑j=1K−1σ2⁢xi⁢j2]¯−EfK⁢ln⁡σsuperscript𝜎22¯12superscriptsubscript𝑗1𝐾1superscript𝜎2superscriptsubscript𝑥𝑖𝑗2subscript𝐸𝑓𝐾𝜎\displaystyle\frac{\exp\left[\frac{\sigma^{2}}{2}\right]\overline{\exp\left[% \frac{1}{2}\sum_{j=1}^{K-1}\sigma^{2}x_{ij}^{2}\right]}-E_{f}}{K\ln\sigma}divide start_ARG roman_exp [ divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 end_ARG ] over¯ start_ARG roman_exp [ divide start_ARG 1 end_ARG start_ARG 2 end_ARG ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ] end_ARG - italic_E start_POSTSUBSCRIPT italic_f end_POSTSUBSCRIPT end_ARG start_ARG italic_K roman_ln italic_σ end_ARG(4.10)>\displaystyle>>eσ2/2K⁢ln⁡σ∝eσ2/2ln⁡σ.proportional-tosuperscriptesuperscript𝜎22𝐾𝜎superscriptesuperscript𝜎22𝜎\displaystyle\frac{\textrm{e}^{\sigma^{2}/2}}{K\ln\sigma}\propto\frac{\textrm{% e}^{\sigma^{2}/2}}{\ln\sigma}.divide start_ARG e start_POSTSUPERSCRIPT italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT / 2 end_POSTSUPERSCRIPT end_ARG start_ARG italic_K roman_ln italic_σ end_ARG ∝ divide start_ARG e start_POSTSUPERSCRIPT italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT / 2 end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ end_ARG .Note that for MSE and MAE energy the asymptotic temperature is∝σ2/(ln⁡σ)proportional-toabsentsuperscript𝜎2𝜎\propto\sigma^{2}/(\ln\sigma)∝ italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT / ( roman_ln italic_σ )andσ/(ln⁡σ)𝜎𝜎\sigma/(\ln\sigma)italic_σ / ( roman_ln italic_σ )respectively, so the temperature from system with cross entropy energyTC.E.≫TMSE,TMAEmuch-greater-thansubscript𝑇formulae-sequenceCEsubscript𝑇MSEsubscript𝑇MAET_{\rm C.E.}\gg T_{\rm MSE},T_{\rm MAE}italic_T start_POSTSUBSCRIPT roman_C . roman_E . end_POSTSUBSCRIPT ≫ italic_T start_POSTSUBSCRIPT roman_MSE end_POSTSUBSCRIPT , italic_T start_POSTSUBSCRIPT roman_MAE end_POSTSUBSCRIPT.

4.2Parameter Initialization: Uniform Distribution

For parameters initialized by uniform distribution, the initial average energy is written as

⟨E0⟩=∫∑i1n⁢[1+exp⁡(−∑j=1K−1μj⁢lj⁢xi⁢j−μK⁢lK)]⁢∏j=1Kd⁢μj.delimited-⟨⟩subscript𝐸0subscript𝑖1𝑛delimited-[]1superscriptsubscript𝑗1𝐾1subscript𝜇𝑗subscript𝑙𝑗subscript𝑥𝑖𝑗subscript𝜇𝐾subscript𝑙𝐾superscriptsubscriptproduct𝑗1𝐾𝑑subscript𝜇𝑗\displaystyle\langle E_{0}\rangle=\int\sum_{i}\frac{1}{n}\left[1+\exp\left(-% \sum_{j=1}^{K-1}\mu_{j}l_{j}x_{ij}-\mu_{K}l_{K}\right)\right]\prod_{j=1}^{K}d% \mu_{j}.⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩ = ∫ ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT divide start_ARG 1 end_ARG start_ARG italic_n end_ARG [ 1 + roman_exp ( - ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT - italic_μ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT italic_l start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT ) ] ∏ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT italic_d italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT .(4.11)Given the asymptotic case∑j=1K−1μj⁢lj⁢xi⁢j−μK⁢lK→∞→superscriptsubscript𝑗1𝐾1subscript𝜇𝑗subscript𝑙𝑗subscript𝑥𝑖𝑗subscript𝜇𝐾subscript𝑙𝐾\sum_{j=1}^{K-1}\mu_{j}l_{j}x_{ij}-\mu_{K}l_{K}\rightarrow\infty∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT - italic_μ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT italic_l start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT → ∞(Case I), Equation (4.11) becomes

⟨E0⟩delimited-⟨⟩subscript𝐸0\displaystyle\langle E_{0}\rangle⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩∼similar-to\displaystyle\sim∼∑i1n⁢∫−1/21/2exp⁡(−∑j=1K−1μj⁢lj⁢xi⁢j)⁢𝑑μj⁢∫−1/21/2exp⁡(−μK⁢lK)⁢𝑑μKsubscript𝑖1𝑛superscriptsubscript1212superscriptsubscript𝑗1𝐾1subscript𝜇𝑗subscript𝑙𝑗subscript𝑥𝑖𝑗differential-dsubscript𝜇𝑗superscriptsubscript1212subscript𝜇𝐾subscript𝑙𝐾differential-dsubscript𝜇𝐾\displaystyle\sum_{i}\frac{1}{n}\int_{-1/2}^{1/2}\exp\left(-\sum_{j=1}^{K-1}% \mu_{j}l_{j}x_{ij}\right)d\mu_{j}\int_{-1/2}^{1/2}\exp\left(-\mu_{K}l_{K}% \right)d\mu_{K}∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∫ start_POSTSUBSCRIPT - 1 / 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 / 2 end_POSTSUPERSCRIPT roman_exp ( - ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT ) italic_d italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT ∫ start_POSTSUBSCRIPT - 1 / 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 1 / 2 end_POSTSUPERSCRIPT roman_exp ( - italic_μ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT italic_l start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT ) italic_d italic_μ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT(4.12)∼similar-to\displaystyle\sim∼∑i1n⁢∏j=1K−1(elj⁢|xi⁢j|/2−e−lj⁢|xi⁢j|/2lj⁢|xi⁢j|)⁢(elK/2−e−lK/2lK)subscript𝑖1𝑛superscriptsubscriptproduct𝑗1𝐾1superscriptesubscript𝑙𝑗subscript𝑥𝑖𝑗2superscriptesubscript𝑙𝑗subscript𝑥𝑖𝑗2subscript𝑙𝑗subscript𝑥𝑖𝑗superscriptesubscript𝑙𝐾2superscriptesubscript𝑙𝐾2subscript𝑙𝐾\displaystyle\sum_{i}\frac{1}{n}\prod_{j=1}^{K-1}\left(\frac{\textrm{e}^{l_{j}% |x_{ij}|/2}-\textrm{e}^{-l_{j}|x_{ij}|/2}}{l_{j}|x_{ij}|}\right)\left(\frac{% \textrm{e}^{l_{K}/2}-\textrm{e}^{-l_{K}/2}}{l_{K}}\right)∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∏ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT ( divide start_ARG e start_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT | italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT | / 2 end_POSTSUPERSCRIPT - e start_POSTSUPERSCRIPT - italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT | italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT | / 2 end_POSTSUPERSCRIPT end_ARG start_ARG italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT | italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT | end_ARG ) ( divide start_ARG e start_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT / 2 end_POSTSUPERSCRIPT - e start_POSTSUPERSCRIPT - italic_l start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT / 2 end_POSTSUPERSCRIPT end_ARG start_ARG italic_l start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT end_ARG )∼similar-to\displaystyle\sim∼1n⁢∑i∏j=1K−1(elj⁢|xi⁢j|/2lj⁢|xi⁢j|)⁢(elK/2lK).1𝑛subscript𝑖superscriptsubscriptproduct𝑗1𝐾1superscriptesubscript𝑙𝑗subscript𝑥𝑖𝑗2subscript𝑙𝑗subscript𝑥𝑖𝑗superscriptesubscript𝑙𝐾2subscript𝑙𝐾\displaystyle\frac{1}{n}\sum_{i}\prod_{j=1}^{K-1}\left(\frac{\textrm{e}^{l_{j}% |x_{ij}|/2}}{l_{j}|x_{ij}|}\right)\left(\frac{\textrm{e}^{l_{K}/2}}{l_{K}}% \right).divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ∏ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT ( divide start_ARG e start_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT | italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT | / 2 end_POSTSUPERSCRIPT end_ARG start_ARG italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT | italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT | end_ARG ) ( divide start_ARG e start_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT / 2 end_POSTSUPERSCRIPT end_ARG start_ARG italic_l start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT end_ARG ) .Note that we assume{l1,l2,…⁢lK}→∞→subscript𝑙1subscript𝑙2…subscript𝑙𝐾\{l_{1},l_{2},...l_{K}\}\rightarrow\infty{ italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , … italic_l start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT } → ∞for the last step derivation in Equation4.12. On the other hand, for another asymptotic case∑j=1K−1μj⁢lj⁢xi⁢j−μK⁢lK→−∞→superscriptsubscript𝑗1𝐾1subscript𝜇𝑗subscript𝑙𝑗subscript𝑥𝑖𝑗subscript𝜇𝐾subscript𝑙𝐾\sum_{j=1}^{K-1}\mu_{j}l_{j}x_{ij}-\mu_{K}l_{K}\rightarrow-\infty∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT italic_μ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_l start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT - italic_μ start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT italic_l start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT → - ∞(Case II), we can check that Equation (4.12) still holds. Also note that the functionex/2/xsuperscript𝑒𝑥2𝑥e^{x/2}/xitalic_e start_POSTSUPERSCRIPT italic_x / 2 end_POSTSUPERSCRIPT / italic_xhas a minimum valuee/2𝑒2e/2italic_e / 2. For{l1,l2,…⁢lK}→l→∞→subscript𝑙1subscript𝑙2…subscript𝑙𝐾𝑙→\{l_{1},l_{2},...l_{K}\}\rightarrow l\rightarrow\infty{ italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , … italic_l start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT } → italic_l → ∞, the energy shows

⟨E0⟩>1n⁢∑i(e2)K−1⁢(el/2l)=(e2)K−1⁢(el/2l).delimited-⟨⟩subscript𝐸01𝑛subscript𝑖superscript𝑒2𝐾1superscript𝑒𝑙2𝑙superscript𝑒2𝐾1superscript𝑒𝑙2𝑙\displaystyle\langle E_{0}\rangle>\frac{1}{n}\sum_{i}\left(\frac{e}{2}\right)^% {K-1}\left(\frac{e^{l/2}}{l}\right)=\left(\frac{e}{2}\right)^{K-1}\left(\frac{% e^{l/2}}{l}\right).⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩ > divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( divide start_ARG italic_e end_ARG start_ARG 2 end_ARG ) start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT ( divide start_ARG italic_e start_POSTSUPERSCRIPT italic_l / 2 end_POSTSUPERSCRIPT end_ARG start_ARG italic_l end_ARG ) = ( divide start_ARG italic_e end_ARG start_ARG 2 end_ARG ) start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT ( divide start_ARG italic_e start_POSTSUPERSCRIPT italic_l / 2 end_POSTSUPERSCRIPT end_ARG start_ARG italic_l end_ARG ) .(4.13)Thus, the temperature of the system is

T=⟨E0⟩−EfK⁢ln⁡l≳(e2)K−1⁢el/2K⁢l⁢ln⁡l>0.614⁢el/2l⁢ln⁡l∝el/2l⁢ln⁡l𝑇delimited-⟨⟩subscript𝐸0subscript𝐸𝑓𝐾𝑙greater-than-or-equivalent-tosuperscript𝑒2𝐾1superscript𝑒𝑙2𝐾𝑙𝑙0.614superscript𝑒𝑙2𝑙𝑙proportional-tosuperscript𝑒𝑙2𝑙𝑙\displaystyle T=\frac{\langle E_{0}\rangle-E_{f}}{K\ln l}\gtrsim\left(\frac{e}% {2}\right)^{K-1}\frac{e^{l/2}}{Kl\ln l}>0.614\frac{e^{l/2}}{l\ln l}\propto% \frac{e^{l/2}}{l\ln l}italic_T = divide start_ARG ⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩ - italic_E start_POSTSUBSCRIPT italic_f end_POSTSUBSCRIPT end_ARG start_ARG italic_K roman_ln italic_l end_ARG ≳ ( divide start_ARG italic_e end_ARG start_ARG 2 end_ARG ) start_POSTSUPERSCRIPT italic_K - 1 end_POSTSUPERSCRIPT divide start_ARG italic_e start_POSTSUPERSCRIPT italic_l / 2 end_POSTSUPERSCRIPT end_ARG start_ARG italic_K italic_l roman_ln italic_l end_ARG > 0.614 divide start_ARG italic_e start_POSTSUPERSCRIPT italic_l / 2 end_POSTSUPERSCRIPT end_ARG start_ARG italic_l roman_ln italic_l end_ARG ∝ divide start_ARG italic_e start_POSTSUPERSCRIPT italic_l / 2 end_POSTSUPERSCRIPT end_ARG start_ARG italic_l roman_ln italic_l end_ARG(4.14)Compared with temperature in system with MSE or MAE energy thatTMSE∝l2/ln⁡lproportional-tosubscript𝑇MSEsuperscript𝑙2𝑙T_{\rm MSE}\propto l^{2}/\ln litalic_T start_POSTSUBSCRIPT roman_MSE end_POSTSUBSCRIPT ∝ italic_l start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT / roman_ln italic_landTMAE∝l/ln⁡lproportional-tosubscript𝑇MAE𝑙𝑙T_{\rm MAE}\propto l/\ln litalic_T start_POSTSUBSCRIPT roman_MAE end_POSTSUBSCRIPT ∝ italic_l / roman_ln italic_l, we get the conclusion thatTC.E.≫TMSE,TMAEmuch-greater-thansubscript𝑇formulae-sequenceCEsubscript𝑇MSEsubscript𝑇MAET_{\rm C.E.}\gg T_{\rm MSE},T_{\rm MAE}italic_T start_POSTSUBSCRIPT roman_C . roman_E . end_POSTSUBSCRIPT ≫ italic_T start_POSTSUBSCRIPT roman_MSE end_POSTSUBSCRIPT , italic_T start_POSTSUBSCRIPT roman_MAE end_POSTSUBSCRIPT.

5Temperature in Neural Network

In this section, we look into a standard artificial neural network system. We assume the neural network hasL𝐿Litalic_Llayers. For thep𝑝pitalic_p-th layer, the input data is𝒂p−1superscript𝒂𝑝1{\boldsymbol{a}}^{p-1}bold_italic_a start_POSTSUPERSCRIPT italic_p - 1 end_POSTSUPERSCRIPT, the output data is𝒂psuperscript𝒂𝑝{\boldsymbol{a}}^{p}bold_italic_a start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT, and the size of the layer islpsubscript𝑙𝑝l_{p}italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT. We have

𝐳p=𝐖p⁢𝐚p−1+𝐛psuperscript𝐳𝑝superscript𝐖𝑝superscript𝐚𝑝1superscript𝐛𝑝\displaystyle{\bf z}^{p}={\bf W}^{p}{\bf a}^{p-1}+{\bf b}^{p}bold_z start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT = bold_W start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT bold_a start_POSTSUPERSCRIPT italic_p - 1 end_POSTSUPERSCRIPT + bold_b start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT(5.1)𝐚p=ϕ⁢(𝐳p),superscript𝐚𝑝italic-ϕsuperscript𝐳𝑝\displaystyle{\bf a}^{p}=\phi({\bf z}^{p}),bold_a start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT = italic_ϕ ( bold_z start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT ) ,(5.2)whereϕitalic-ϕ\phiitalic_ϕis the activation function. Some typical activation function including Sigmoid functionσ⁢(z)=1/(1+e−z)𝜎𝑧11superscripte𝑧\sigma(z)=1/(1+\textrm{e}^{-z})italic_σ ( italic_z ) = 1 / ( 1 + e start_POSTSUPERSCRIPT - italic_z end_POSTSUPERSCRIPT ), Tanh functiontanh⁢(z)=(ez−e−z)/(ez+e−z)tanh𝑧superscripte𝑧superscripte𝑧superscripte𝑧superscripte𝑧{\rm tanh}(z)=(\textrm{e}^{z}-\textrm{e}^{-z})/(\textrm{e}^{z}+\textrm{e}^{-z})roman_tanh ( italic_z ) = ( e start_POSTSUPERSCRIPT italic_z end_POSTSUPERSCRIPT - e start_POSTSUPERSCRIPT - italic_z end_POSTSUPERSCRIPT ) / ( e start_POSTSUPERSCRIPT italic_z end_POSTSUPERSCRIPT + e start_POSTSUPERSCRIPT - italic_z end_POSTSUPERSCRIPT )and ReLU function ReLU(z)=max⁢{0,z}𝑧max0𝑧(z)={\rm max}\{0,z\}( italic_z ) = roman_max { 0 , italic_z }.

5.1Initial Normal Distribution

5.1.1Default Case: Tanh Activation Function

We assume the system follows MSE to set up its energy

E=∑(𝐖L⁢𝐚L−1+𝐛L−y)2,𝐸superscriptsuperscript𝐖𝐿superscript𝐚𝐿1superscript𝐛𝐿𝑦2E=\sum({\bf W}^{L}{\bf a}^{L-1}+{\bf b}^{L}-y)^{2},italic_E = ∑ ( bold_W start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT bold_a start_POSTSUPERSCRIPT italic_L - 1 end_POSTSUPERSCRIPT + bold_b start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT - italic_y ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ,(5.3)with𝐖L={Wi⁢jL}superscript𝐖𝐿subscriptsuperscript𝑊𝐿𝑖𝑗{\bf W}^{L}=\{W^{L}_{ij}\}bold_W start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT = { italic_W start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT }andi∈[1,lL]𝑖1subscript𝑙𝐿i\in[1,l_{L}]italic_i ∈ [ 1 , italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT ],j∈[1,lL−1]𝑗1subscript𝑙𝐿1j\in[1,l_{L-1}]italic_j ∈ [ 1 , italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT ]being the weight matrix connecting the last two layers.

Let us consider the asymptotic solution of the energy equation above when all parameters have initially normal distributions. For any elementWi⁢jpsuperscriptsubscript𝑊𝑖𝑗𝑝W_{ij}^{p}italic_W start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPTin any weight matrix in the neural network, as the asymptotic standard deviationσi⁢jp→∞→superscriptsubscript𝜎𝑖𝑗𝑝\sigma_{ij}^{p}\rightarrow\inftyitalic_σ start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT → ∞,𝒛p→±∞→superscript𝒛𝑝plus-or-minus\boldsymbol{z}^{p}\rightarrow\pm\inftybold_italic_z start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT → ± ∞. We first discuss the case where the activation function istanh⁢(z)tanh𝑧{\rm tanh}(z)roman_tanh ( italic_z )with|tanh⁢(z)|→1→tanh𝑧1|{\rm tanh}(z)|\rightarrow 1| roman_tanh ( italic_z ) | → 1so that|𝒂p|=|tanh⁢(z)|→1superscript𝒂𝑝tanh𝑧→1|{\boldsymbol{a}}^{p}|=|{\rm tanh}(z)|\rightarrow 1| bold_italic_a start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT | = | roman_tanh ( italic_z ) | → 1. The energy of the system is written as

⟨E0⟩delimited-⟨⟩subscript𝐸0\displaystyle\langle E_{0}\rangle⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩=\displaystyle==1n⁢∫∑i=1n∑k=1lL(∑j=1lL−1Wk⁢jL⁢ai⁢jL−1+bkL−yi⁢k)2⁢f⁢(𝑾L)⁢d⁢𝑾L⁢d⁢bL1𝑛superscriptsubscript𝑖1𝑛superscriptsubscript𝑘1subscript𝑙𝐿superscriptsuperscriptsubscript𝑗1subscript𝑙𝐿1superscriptsubscript𝑊𝑘𝑗𝐿superscriptsubscript𝑎𝑖𝑗𝐿1superscriptsubscript𝑏𝑘𝐿subscript𝑦𝑖𝑘2𝑓superscript𝑾𝐿𝑑superscript𝑾𝐿𝑑superscript𝑏𝐿\displaystyle\frac{1}{n}\int\sum_{i=1}^{n}\sum_{k=1}^{l_{L}}\left(\sum_{j=1}^{% l_{L-1}}W_{kj}^{L}a_{ij}^{L-1}+b_{k}^{L}-y_{ik}\right)^{2}f({\boldsymbol{W}}^{% L})d{\boldsymbol{W}^{L}}db^{L}divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∫ ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ∑ start_POSTSUBSCRIPT italic_k = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ( ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT italic_W start_POSTSUBSCRIPT italic_k italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT italic_a start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L - 1 end_POSTSUPERSCRIPT + italic_b start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT - italic_y start_POSTSUBSCRIPT italic_i italic_k end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_f ( bold_italic_W start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT ) italic_d bold_italic_W start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT italic_d italic_b start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT(5.4)=\displaystyle==1n⁢∑i∑k[∑j(σk⁢jL)2⁢(ai⁢jL−1)2+(σkL)2+(yi⁢k)2]1𝑛subscript𝑖subscript𝑘delimited-[]subscript𝑗superscriptsuperscriptsubscript𝜎𝑘𝑗𝐿2superscriptsuperscriptsubscript𝑎𝑖𝑗𝐿12superscriptsuperscriptsubscript𝜎𝑘𝐿2superscriptsubscript𝑦𝑖𝑘2\displaystyle\frac{1}{n}\sum_{i}\sum_{k}\left[\sum_{j}(\sigma_{kj}^{L})^{2}(a_% {ij}^{L-1})^{2}+(\sigma_{k}^{L})^{2}+(y_{ik})^{2}\right]divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ∑ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT [ ∑ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT ( italic_σ start_POSTSUBSCRIPT italic_k italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ( italic_a start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L - 1 end_POSTSUPERSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + ( italic_σ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + ( italic_y start_POSTSUBSCRIPT italic_i italic_k end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ]∼similar-to\displaystyle\sim∼1n⁢∑i∑k[∑j(σk⁢jL)2+(σkL)2+(yi⁢k)2].1𝑛subscript𝑖subscript𝑘delimited-[]subscript𝑗superscriptsuperscriptsubscript𝜎𝑘𝑗𝐿2superscriptsuperscriptsubscript𝜎𝑘𝐿2superscriptsubscript𝑦𝑖𝑘2\displaystyle\frac{1}{n}\sum_{i}\sum_{k}\left[\sum_{j}(\sigma_{kj}^{L})^{2}+(% \sigma_{k}^{L})^{2}+(y_{ik})^{2}\right].divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ∑ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT [ ∑ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT ( italic_σ start_POSTSUBSCRIPT italic_k italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + ( italic_σ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + ( italic_y start_POSTSUBSCRIPT italic_i italic_k end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ] .Givenσk⁢jL,σkL→σ→∞→superscriptsubscript𝜎𝑘𝑗𝐿superscriptsubscript𝜎𝑘𝐿𝜎→\sigma_{kj}^{L},\sigma_{k}^{L}\rightarrow\sigma\rightarrow\inftyitalic_σ start_POSTSUBSCRIPT italic_k italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT , italic_σ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT → italic_σ → ∞, Equation (5.4) can be simplified to

⟨E0⟩delimited-⟨⟩subscript𝐸0\displaystyle\langle E_{0}\rangle⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩∼similar-to\displaystyle\sim∼1n⁢∑i,k[σ2⁢(lL−1+1)+yi⁢k2]∼1n⁢∑i[σ2⁢(lL−1+1)⁢lL+yi2]similar-to1𝑛subscript𝑖𝑘delimited-[]superscript𝜎2subscript𝑙𝐿11superscriptsubscript𝑦𝑖𝑘21𝑛subscript𝑖delimited-[]superscript𝜎2subscript𝑙𝐿11subscript𝑙𝐿superscriptsubscript𝑦𝑖2\displaystyle\frac{1}{n}\sum_{i,k}[\sigma^{2}(l_{L-1}+1)+y_{ik}^{2}]\sim\frac{% 1}{n}\sum_{i}[\sigma^{2}(l_{L-1}+1)l_{L}+y_{i}^{2}]divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i , italic_k end_POSTSUBSCRIPT [ italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ( italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT + 1 ) + italic_y start_POSTSUBSCRIPT italic_i italic_k end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ] ∼ divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT [ italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ( italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT + 1 ) italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT + italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ](5.5)∼similar-to\displaystyle\sim∼σ2⁢(lL−1+1)⁢lL+Y2¯∼σ2⁢(lL−1+1)⁢lLsimilar-tosuperscript𝜎2subscript𝑙𝐿11subscript𝑙𝐿¯superscript𝑌2superscript𝜎2subscript𝑙𝐿11subscript𝑙𝐿\displaystyle\sigma^{2}(l_{L-1}+1)l_{L}+\overline{Y^{2}}\sim\sigma^{2}(l_{L-1}% +1)l_{L}italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ( italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT + 1 ) italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT + over¯ start_ARG italic_Y start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ∼ italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ( italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT + 1 ) italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPTNote that the total number of parameters in the neural network is𝒟=lL⁢(lL−1+1)+lL−1⁢(lL−2+1)+…+l1⁢(l0+1)=∑p=1lLlp⁢(lp−1+1)𝒟subscript𝑙𝐿subscript𝑙𝐿11subscript𝑙𝐿1subscript𝑙𝐿21…subscript𝑙1subscript𝑙01superscriptsubscript𝑝1subscript𝑙𝐿subscript𝑙𝑝subscript𝑙𝑝11\mathcal{D}=l_{L}(l_{L-1}+1)+l_{L-1}(l_{L-2}+1)+...+l_{1}(l_{0}+1)=\sum_{p=1}^% {l_{L}}l_{p}(l_{p-1}+1)caligraphic_D = italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT + 1 ) + italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_L - 2 end_POSTSUBSCRIPT + 1 ) + … + italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT + 1 ) = ∑ start_POSTSUBSCRIPT italic_p = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT end_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPT + 1 ). The temperature of the neural network system is

T𝑇\displaystyle Titalic_T=\displaystyle==⟨E0⟩−Ef∑1𝒟ln⁡σ+𝒟2⁢[1+ln⁡(2⁢π)]delimited-⟨⟩subscript𝐸0subscript𝐸𝑓superscriptsubscript1𝒟𝜎𝒟2delimited-[]12𝜋\displaystyle\frac{\langle E_{0}\rangle-E_{f}}{\sum_{1}^{\mathcal{D}}\ln\sigma% +\frac{\mathcal{D}}{2}[1+\ln(2\pi)]}divide start_ARG ⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩ - italic_E start_POSTSUBSCRIPT italic_f end_POSTSUBSCRIPT end_ARG start_ARG ∑ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT caligraphic_D end_POSTSUPERSCRIPT roman_ln italic_σ + divide start_ARG caligraphic_D end_ARG start_ARG 2 end_ARG [ 1 + roman_ln ( 2 italic_π ) ] end_ARG(5.6)∼similar-to\displaystyle\sim∼lL⁢(lL−1+1)∑p=1lLlp⁢(lp−1+1)⁢(σ2ln⁡σ).subscript𝑙𝐿subscript𝑙𝐿11superscriptsubscript𝑝1subscript𝑙𝐿subscript𝑙𝑝subscript𝑙𝑝11superscript𝜎2𝜎\displaystyle\frac{l_{L}(l_{L-1}+1)}{\sum_{p=1}^{l_{L}}l_{p}(l_{p-1}+1)}\left(% \frac{\sigma^{2}}{\ln\sigma}\right).divide start_ARG italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT + 1 ) end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_p = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT end_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPT + 1 ) end_ARG ( divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ end_ARG ) .

5.1.2Sigmoid Function

If the activation function is the sigmoid function, i.e.,a=σ⁢(z)→{1,0}𝑎𝜎𝑧→10a=\sigma(z)\rightarrow\{1,0\}italic_a = italic_σ ( italic_z ) → { 1 , 0 }forz→∞→𝑧z\rightarrow\inftyitalic_z → ∞, Equation (5.4) becomes

⟨E0⟩delimited-⟨⟩subscript𝐸0\displaystyle\langle E_{0}\rangle⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩∼similar-to\displaystyle\sim∼1n⁢∑i∑j[σ2⁢(ξ⁢lL−1+1)+yi⁢k2]1𝑛subscript𝑖subscript𝑗delimited-[]superscript𝜎2𝜉subscript𝑙𝐿11superscriptsubscript𝑦𝑖𝑘2\displaystyle\frac{1}{n}\sum_{i}\sum_{j}[\sigma^{2}(\xi l_{L-1}+1)+y_{ik}^{2}]divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ∑ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT [ italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ( italic_ξ italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT + 1 ) + italic_y start_POSTSUBSCRIPT italic_i italic_k end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ](5.7)∼similar-to\displaystyle\sim∼σ2⁢lL⁢(ξ⁢lL−1+1),superscript𝜎2subscript𝑙𝐿𝜉subscript𝑙𝐿11\displaystyle\sigma^{2}l_{L}(\xi l_{L-1}+1),italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT ( italic_ξ italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT + 1 ) ,whereξ𝜉\xiitalic_ξis the fraction ofajL→1→subscriptsuperscript𝑎𝐿𝑗1a^{L}_{j}\rightarrow 1italic_a start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT → 1. Thus, the temperature of the system is

T∼lL⁢(ξ⁢lL−1+1)∑p=1lLlp⁢(lp−1+1)⁢(σ2ln⁡σ).similar-to𝑇subscript𝑙𝐿𝜉subscript𝑙𝐿11superscriptsubscript𝑝1subscript𝑙𝐿subscript𝑙𝑝subscript𝑙𝑝11superscript𝜎2𝜎\displaystyle T\sim\frac{l_{L}(\xi l_{L-1}+1)}{\sum_{p=1}^{l_{L}}l_{p}(l_{p-1}% +1)}\left(\frac{\sigma^{2}}{\ln\sigma}\right).italic_T ∼ divide start_ARG italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT ( italic_ξ italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT + 1 ) end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_p = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT end_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPT + 1 ) end_ARG ( divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ end_ARG ) .(5.8)Compared with Equation (5.6), the neural network with Sigmoid activation function has lower temperature than the that with tanh activation function, with an approximationTsigmoid∼ξ⁢lL−1+1lL−1+1⁢Ttanh∼ξ⁢Ttanhsimilar-tosubscript𝑇sigmoid𝜉subscript𝑙𝐿11subscript𝑙𝐿11subscript𝑇tanhsimilar-to𝜉subscript𝑇tanhT_{\rm sigmoid}\sim\frac{\xi l_{L-1}+1}{l_{L-1}+1}T_{\rm tanh}\sim\xi T_{\rm tanh}italic_T start_POSTSUBSCRIPT roman_sigmoid end_POSTSUBSCRIPT ∼ divide start_ARG italic_ξ italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT + 1 end_ARG start_ARG italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT + 1 end_ARG italic_T start_POSTSUBSCRIPT roman_tanh end_POSTSUBSCRIPT ∼ italic_ξ italic_T start_POSTSUBSCRIPT roman_tanh end_POSTSUBSCRIPT.

5.1.3ReLU Function

If the activation function is the ReLu function, i.e.,ϕ⁢(z)=zitalic-ϕ𝑧𝑧\phi(z)=zitalic_ϕ ( italic_z ) = italic_zforz>0𝑧0z>0italic_z > 0andϕ⁢(z)=0italic-ϕ𝑧0\phi(z)=0italic_ϕ ( italic_z ) = 0forz⩽0𝑧0z\leqslant 0italic_z ⩽ 0, for thep𝑝pitalic_p-th layer and thej𝑗jitalic_j-th component of𝒛psuperscript𝒛𝑝{\boldsymbol{z}}^{p}bold_italic_z start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT,zjp=∑k=1lp−1Wj⁢kp⁢akp−1+bjpsuperscriptsubscript𝑧𝑗𝑝superscriptsubscript𝑘1subscript𝑙𝑝1superscriptsubscript𝑊𝑗𝑘𝑝superscriptsubscript𝑎𝑘𝑝1superscriptsubscript𝑏𝑗𝑝z_{j}^{p}=\sum_{k=1}^{l_{p-1}}W_{jk}^{p}a_{k}^{p-1}+b_{j}^{p}italic_z start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT = ∑ start_POSTSUBSCRIPT italic_k = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT italic_W start_POSTSUBSCRIPT italic_j italic_k end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT italic_a start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p - 1 end_POSTSUPERSCRIPT + italic_b start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT. For anyzjp>0superscriptsubscript𝑧𝑗𝑝0z_{j}^{p}>0italic_z start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT > 0, we can use parameters set−Wj⁢kpsuperscriptsubscript𝑊𝑗𝑘𝑝-W_{jk}^{p}- italic_W start_POSTSUBSCRIPT italic_j italic_k end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPTand−bjpsuperscriptsubscript𝑏𝑗𝑝-b_{j}^{p}- italic_b start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPTto findz^jp<0superscriptsubscript^𝑧𝑗𝑝0\hat{z}_{j}^{p}<0over^ start_ARG italic_z end_ARG start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT < 0while|z^jp|=|zjp|superscriptsubscript^𝑧𝑗𝑝superscriptsubscript𝑧𝑗𝑝|\hat{z}_{j}^{p}|=|z_{j}^{p}|| over^ start_ARG italic_z end_ARG start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT | = | italic_z start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT |. From symmetry we can derive∫zjp>0f⁢(𝑾p,𝒃p)⁢𝑑𝑾p⁢𝑑𝒃p=∫zjp⩽0f⁢(𝑾p,𝒃p)⁢𝑑𝑾p⁢𝑑𝒃psubscriptsuperscriptsubscript𝑧𝑗𝑝0𝑓superscript𝑾𝑝superscript𝒃𝑝differential-dsuperscript𝑾𝑝differential-dsuperscript𝒃𝑝subscriptsuperscriptsubscript𝑧𝑗𝑝0𝑓superscript𝑾𝑝superscript𝒃𝑝differential-dsuperscript𝑾𝑝differential-dsuperscript𝒃𝑝\int_{z_{j}^{p}>0}f({\boldsymbol{W}^{p}},{\boldsymbol{b}^{p}})d{\boldsymbol{W}% ^{p}}d{\boldsymbol{b}^{p}}=\int_{z_{j}^{p}\leqslant 0}f({\boldsymbol{W}^{p}},{% \boldsymbol{b}^{p}})d{\boldsymbol{W}^{p}}d{\boldsymbol{b}^{p}}∫ start_POSTSUBSCRIPT italic_z start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT > 0 end_POSTSUBSCRIPT italic_f ( bold_italic_W start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT , bold_italic_b start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT ) italic_d bold_italic_W start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT italic_d bold_italic_b start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT = ∫ start_POSTSUBSCRIPT italic_z start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT ⩽ 0 end_POSTSUBSCRIPT italic_f ( bold_italic_W start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT , bold_italic_b start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT ) italic_d bold_italic_W start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT italic_d bold_italic_b start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT, therefore, we have

∫(ajp)2⁢f⁢(𝑾p,𝒃p)⁢𝑑𝑾p⁢𝑑𝒃p=12⁢∫(zjp)2⁢f⁢(𝑾p,𝒃p)⁢𝑑𝑾p⁢𝑑𝒃psuperscriptsuperscriptsubscript𝑎𝑗𝑝2𝑓superscript𝑾𝑝superscript𝒃𝑝differential-dsuperscript𝑾𝑝differential-dsuperscript𝒃𝑝12superscriptsuperscriptsubscript𝑧𝑗𝑝2𝑓superscript𝑾𝑝superscript𝒃𝑝differential-dsuperscript𝑾𝑝differential-dsuperscript𝒃𝑝\displaystyle\int(a_{j}^{p})^{2}f({\boldsymbol{W}^{p}},{\boldsymbol{b}^{p}})d{% \boldsymbol{W}^{p}}d{\boldsymbol{b}^{p}}=\frac{1}{2}\int(z_{j}^{p})^{2}f({% \boldsymbol{W}^{p}},{\boldsymbol{b}^{p}})d{\boldsymbol{W}^{p}}d{\boldsymbol{b}% ^{p}}∫ ( italic_a start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_f ( bold_italic_W start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT , bold_italic_b start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT ) italic_d bold_italic_W start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT italic_d bold_italic_b start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT = divide start_ARG 1 end_ARG start_ARG 2 end_ARG ∫ ( italic_z start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_f ( bold_italic_W start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT , bold_italic_b start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT ) italic_d bold_italic_W start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT italic_d bold_italic_b start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT=12⁢∫(∑k=1lpWj⁢kp⁢akp−1+bjp)2⁢f⁢(𝑾p,𝒃p)⁢𝑑𝑾p⁢𝑑𝒃pabsent12superscriptsuperscriptsubscript𝑘1subscript𝑙𝑝superscriptsubscript𝑊𝑗𝑘𝑝superscriptsubscript𝑎𝑘𝑝1superscriptsubscript𝑏𝑗𝑝2𝑓superscript𝑾𝑝superscript𝒃𝑝differential-dsuperscript𝑾𝑝differential-dsuperscript𝒃𝑝\displaystyle=\frac{1}{2}\int\left(\sum_{k=1}^{l_{p}}W_{jk}^{p}a_{k}^{p-1}+b_{% j}^{p}\right)^{2}f({\boldsymbol{W}^{p}},{\boldsymbol{b}^{p}})d{\boldsymbol{W}^% {p}}d{\boldsymbol{b}^{p}}= divide start_ARG 1 end_ARG start_ARG 2 end_ARG ∫ ( ∑ start_POSTSUBSCRIPT italic_k = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT end_POSTSUPERSCRIPT italic_W start_POSTSUBSCRIPT italic_j italic_k end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT italic_a start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p - 1 end_POSTSUPERSCRIPT + italic_b start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_f ( bold_italic_W start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT , bold_italic_b start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT ) italic_d bold_italic_W start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT italic_d bold_italic_b start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT≈12⁢[∑k=1lp−1(σj⁢kp)2⁢(ai⁢kp−1)2+(σjp)2]absent12delimited-[]superscriptsubscript𝑘1subscript𝑙𝑝1superscriptsuperscriptsubscript𝜎𝑗𝑘𝑝2superscriptsuperscriptsubscript𝑎𝑖𝑘𝑝12superscriptsuperscriptsubscript𝜎𝑗𝑝2\displaystyle\approx\frac{1}{2}\left[\sum_{k=1}^{l_{p-1}}(\sigma_{jk}^{p})^{2}% (a_{ik}^{p-1})^{2}+(\sigma_{j}^{p})^{2}\right]≈ divide start_ARG 1 end_ARG start_ARG 2 end_ARG [ ∑ start_POSTSUBSCRIPT italic_k = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ( italic_σ start_POSTSUBSCRIPT italic_j italic_k end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ( italic_a start_POSTSUBSCRIPT italic_i italic_k end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p - 1 end_POSTSUPERSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + ( italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ]≈12⁢[∑1lp−1σ2⁢(ai⁢kp−1)2+σ2].absent12delimited-[]superscriptsubscript1subscript𝑙𝑝1superscript𝜎2superscriptsuperscriptsubscript𝑎𝑖𝑘𝑝12superscript𝜎2\displaystyle\approx\frac{1}{2}\left[\sum_{1}^{l_{p-1}}\sigma^{2}(a_{ik}^{p-1}% )^{2}+\sigma^{2}\right].≈ divide start_ARG 1 end_ARG start_ARG 2 end_ARG [ ∑ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ( italic_a start_POSTSUBSCRIPT italic_i italic_k end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p - 1 end_POSTSUPERSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ] .(5.9) Equation (5.9) can be used to calculate the initial energy

⟨E0⟩delimited-⟨⟩subscript𝐸0\displaystyle\langle E_{0}\rangle⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩∼similar-to\displaystyle\sim∼1n⁢∫∑i∑k[∑jσ2⁢(ai⁢j)2+σ2]⁢f⁢(𝑾,𝒃)⁢d⁢𝑾⁢d⁢𝒃1𝑛subscript𝑖subscript𝑘delimited-[]subscript𝑗superscript𝜎2superscriptsubscript𝑎𝑖𝑗2superscript𝜎2𝑓𝑾𝒃𝑑𝑾𝑑𝒃\displaystyle\frac{1}{n}\int\sum_{i}\sum_{k}\left[\sum_{j}\sigma^{2}(a_{ij})^{% 2}+\sigma^{2}\right]f({\boldsymbol{W},\boldsymbol{b}})d{\boldsymbol{W}}d{% \boldsymbol{b}}divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∫ ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ∑ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT [ ∑ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ( italic_a start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ] italic_f ( bold_italic_W , bold_italic_b ) italic_d bold_italic_W italic_d bold_italic_b(5.10)∼similar-to\displaystyle\sim∼12⁢n⁢∑i∫lL⁢lL−1⁢(σ2)2⁢[∑j=1lL−2(ai⁢jL−1)2+1]⁢f⁢(𝑾,𝒃)⁢𝑑𝑾⁢𝑑𝒃12𝑛subscript𝑖subscript𝑙𝐿subscript𝑙𝐿1superscriptsuperscript𝜎22delimited-[]superscriptsubscript𝑗1subscript𝑙𝐿2superscriptsuperscriptsubscript𝑎𝑖𝑗𝐿121𝑓𝑾𝒃differential-d𝑾differential-d𝒃\displaystyle\frac{1}{2n}\sum_{i}\int l_{L}l_{L-1}(\sigma^{2})^{2}\left[\sum_{% j=1}^{l_{L-2}}(a_{ij}^{L-1})^{2}+1\right]f({\boldsymbol{W},\boldsymbol{b}})d{% \boldsymbol{W}}d{\boldsymbol{b}}divide start_ARG 1 end_ARG start_ARG 2 italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ∫ italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT ( italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT [ ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_L - 2 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ( italic_a start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L - 1 end_POSTSUPERSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + 1 ] italic_f ( bold_italic_W , bold_italic_b ) italic_d bold_italic_W italic_d bold_italic_b∼similar-to\displaystyle\sim∼j=122⁢n⁢∑i∫lL⁢lL−1⁢lL−2⁢(σ2)3⁢[∑1lL−3(ai⁢jL−2)2+1]⁢f⁢(𝑾,𝒃)⁢𝑑𝑾⁢𝑑𝒃𝑗1superscript22𝑛subscript𝑖subscript𝑙𝐿subscript𝑙𝐿1subscript𝑙𝐿2superscriptsuperscript𝜎23delimited-[]superscriptsubscript1subscript𝑙𝐿3superscriptsuperscriptsubscript𝑎𝑖𝑗𝐿221𝑓𝑾𝒃differential-d𝑾differential-d𝒃\displaystyle\frac{j=1}{2^{2}n}\sum_{i}\int l_{L}l_{L-1}l_{L-2}(\sigma^{2})^{3% }\left[\sum_{1}^{l_{L-3}}(a_{ij}^{L-2})^{2}+1\right]f({\boldsymbol{W},% \boldsymbol{b}})d{\boldsymbol{W}}d{\boldsymbol{b}}divide start_ARG italic_j = 1 end_ARG start_ARG 2 start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ∫ italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT italic_l start_POSTSUBSCRIPT italic_L - 2 end_POSTSUBSCRIPT ( italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ) start_POSTSUPERSCRIPT 3 end_POSTSUPERSCRIPT [ ∑ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_L - 3 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ( italic_a start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L - 2 end_POSTSUPERSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + 1 ] italic_f ( bold_italic_W , bold_italic_b ) italic_d bold_italic_W italic_d bold_italic_b∼similar-to\displaystyle\sim∼…∼12L−1⁢n⁢∑i(∏p=1Llp)⁢(σ2)L⁢[∑j=1l0(xi⁢j)2+1]similar-to…1superscript2𝐿1𝑛subscript𝑖superscriptsubscriptproduct𝑝1𝐿subscript𝑙𝑝superscriptsuperscript𝜎2𝐿delimited-[]superscriptsubscript𝑗1subscript𝑙0superscriptsubscript𝑥𝑖𝑗21\displaystyle...\sim\frac{1}{2^{L-1}n}\sum_{i}\left(\prod_{p=1}^{L}l_{p}\right% )(\sigma^{2})^{L}\left[\sum_{j=1}^{l_{0}}(x_{ij})^{2}+1\right]… ∼ divide start_ARG 1 end_ARG start_ARG 2 start_POSTSUPERSCRIPT italic_L - 1 end_POSTSUPERSCRIPT italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( ∏ start_POSTSUBSCRIPT italic_p = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ) ( italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ) start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT [ ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ( italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + 1 ]∼similar-to\displaystyle\sim∼12L−1⁢(∏p=1Llp)⁢σ2⁢L⁢X2¯,1superscript2𝐿1superscriptsubscriptproduct𝑝1𝐿subscript𝑙𝑝superscript𝜎2𝐿¯superscript𝑋2\displaystyle\frac{1}{2^{L-1}}\left(\prod_{p=1}^{L}l_{p}\right)\sigma^{2L}% \overline{X^{2}},divide start_ARG 1 end_ARG start_ARG 2 start_POSTSUPERSCRIPT italic_L - 1 end_POSTSUPERSCRIPT end_ARG ( ∏ start_POSTSUBSCRIPT italic_p = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ) italic_σ start_POSTSUPERSCRIPT 2 italic_L end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ,where we denoteXi2=∑j=1l0xi⁢j2+1superscriptsubscript𝑋𝑖2superscriptsubscript𝑗1subscript𝑙0superscriptsubscript𝑥𝑖𝑗21X_{i}^{2}=\sum_{j=1}^{l_{0}}x_{ij}^{2}+1italic_X start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT = ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT italic_x start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + 1, andX2¯=1n⁢Xi2¯superscript𝑋21𝑛superscriptsubscript𝑋𝑖2\overline{X^{2}}=\frac{1}{n}X_{i}^{2}over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG = divide start_ARG 1 end_ARG start_ARG italic_n end_ARG italic_X start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT. Thus, the temperature of the system is calculated as

T𝑇\displaystyle Titalic_T∼similar-to\displaystyle\sim∼21−L⁢(∏p=1Llp)⁢σ2⁢L⁢X2¯∑p=1Llp⁢(lp−1+1)⁢ln⁡σsuperscript21𝐿superscriptsubscriptproduct𝑝1𝐿subscript𝑙𝑝superscript𝜎2𝐿¯superscript𝑋2superscriptsubscript𝑝1𝐿subscript𝑙𝑝subscript𝑙𝑝11𝜎\displaystyle\frac{2^{1-L}\left(\prod_{p=1}^{L}l_{p}\right)\sigma^{2L}% \overline{X^{2}}}{\sum_{p=1}^{L}l_{p}(l_{p-1}+1)\ln\sigma}divide start_ARG 2 start_POSTSUPERSCRIPT 1 - italic_L end_POSTSUPERSCRIPT ( ∏ start_POSTSUBSCRIPT italic_p = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ) italic_σ start_POSTSUPERSCRIPT 2 italic_L end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_p = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPT + 1 ) roman_ln italic_σ end_ARG(5.11)∝proportional-to\displaystyle\propto∝(σ2⁢Lln⁡σ)⁢X2¯,superscript𝜎2𝐿𝜎¯superscript𝑋2\displaystyle\left(\frac{\sigma^{2L}}{\ln\sigma}\right)\overline{X^{2}},( divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 italic_L end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ end_ARG ) over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ,which is much higher than the temperature from the system using Sigmoid or Tanh activation function that∝σ2/ln⁡σproportional-toabsentsuperscript𝜎2𝜎\propto\sigma^{2}/\ln\sigma∝ italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT / roman_ln italic_σ.

5.2Temperature in Individual Layers

From Section5.1.1to Section5.1.3, we calculated the temperature of the entire neural network system corresponding to different activation functions. In fact, for a neural network, we can also calculate the temperature of each individual layer. To do this, we can write the MSE energy of thep𝑝pitalic_p-th layer as

(𝑾p⁢𝒂p−1+𝒃p−𝒛p^)2,superscriptsuperscript𝑾𝑝superscript𝒂𝑝1superscript𝒃𝑝^superscript𝒛𝑝2({\boldsymbol{W}^{p}}{\boldsymbol{a}^{p-1}}+{\boldsymbol{b}}^{p}-\hat{% \boldsymbol{z}^{p}})^{2},( bold_italic_W start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT bold_italic_a start_POSTSUPERSCRIPT italic_p - 1 end_POSTSUPERSCRIPT + bold_italic_b start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT - over^ start_ARG bold_italic_z start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT end_ARG ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ,(5.12)where𝒛^psuperscript^𝒛𝑝\hat{\boldsymbol{z}}^{p}over^ start_ARG bold_italic_z end_ARG start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPTrepresents the “ideal“𝒛psuperscript𝒛𝑝\boldsymbol{z}^{p}bold_italic_z start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPTvalue for that layer. This value can either be back-propagated from the labelyisubscript𝑦𝑖y_{i}italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPTto thep𝑝pitalic_p-th layer, or be the optimized value after the neural network training converged. In any case, under asymptotic condition, we have|𝑾p⁢𝒂p−1|superscript𝑾𝑝superscript𝒂𝑝1|{\boldsymbol{W}^{p}}{\boldsymbol{a}^{p-1}}|| bold_italic_W start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT bold_italic_a start_POSTSUPERSCRIPT italic_p - 1 end_POSTSUPERSCRIPT |,|𝒃p|≫𝒛^pmuch-greater-thansuperscript𝒃𝑝superscript^𝒛𝑝|{\boldsymbol{b}}^{p}|\gg\hat{\boldsymbol{z}}^{p}| bold_italic_b start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT | ≫ over^ start_ARG bold_italic_z end_ARG start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPTand(𝑾p⁢𝒂p−1+𝒃p−𝒛p^)2≈(𝑾p⁢𝒂p−1+𝒃p)2superscriptsuperscript𝑾𝑝superscript𝒂𝑝1superscript𝒃𝑝^superscript𝒛𝑝2superscriptsuperscript𝑾𝑝superscript𝒂𝑝1superscript𝒃𝑝2({\boldsymbol{W}^{p}}{\boldsymbol{a}^{p-1}}+{\boldsymbol{b}}^{p}-\hat{% \boldsymbol{z}^{p}})^{2}\approx({\boldsymbol{W}^{p}}{\boldsymbol{a}^{p-1}}+{% \boldsymbol{b}}^{p})^{2}( bold_italic_W start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT bold_italic_a start_POSTSUPERSCRIPT italic_p - 1 end_POSTSUPERSCRIPT + bold_italic_b start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT - over^ start_ARG bold_italic_z start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT end_ARG ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ≈ ( bold_italic_W start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT bold_italic_a start_POSTSUPERSCRIPT italic_p - 1 end_POSTSUPERSCRIPT + bold_italic_b start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT. The averaged energy in thep𝑝pitalic_p-th layer is

⟨Ep⟩delimited-⟨⟩subscript𝐸𝑝\displaystyle\langle E_{p}\rangle⟨ italic_E start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ⟩∼similar-to\displaystyle\sim∼1n⁢∫∑i=1n∑k=1lp(∑j=1lp−1Wk⁢jp⁢ai⁢jp−1+bkp−yi⁢k)2⁢f⁢(𝑾L)⁢f⁢(𝒃L)⁢d⁢𝑾L⁢d⁢𝒃L1𝑛superscriptsubscript𝑖1𝑛superscriptsubscript𝑘1subscript𝑙𝑝superscriptsuperscriptsubscript𝑗1subscript𝑙𝑝1superscriptsubscript𝑊𝑘𝑗𝑝superscriptsubscript𝑎𝑖𝑗𝑝1superscriptsubscript𝑏𝑘𝑝subscript𝑦𝑖𝑘2𝑓superscript𝑾𝐿𝑓superscript𝒃𝐿𝑑superscript𝑾𝐿𝑑superscript𝒃𝐿\displaystyle\frac{1}{n}\int\sum_{i=1}^{n}\sum_{k=1}^{l_{p}}\left(\sum_{j=1}^{% l_{p-1}}W_{kj}^{p}a_{ij}^{p-1}+b_{k}^{p}-y_{ik}\right)^{2}f({\boldsymbol{W}}^{% L})f({\boldsymbol{b}^{L}})d{\boldsymbol{W}^{L}}d{\boldsymbol{b}^{L}}divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∫ ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ∑ start_POSTSUBSCRIPT italic_k = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ( ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT italic_W start_POSTSUBSCRIPT italic_k italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT italic_a start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p - 1 end_POSTSUPERSCRIPT + italic_b start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT - italic_y start_POSTSUBSCRIPT italic_i italic_k end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_f ( bold_italic_W start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT ) italic_f ( bold_italic_b start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT ) italic_d bold_italic_W start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT italic_d bold_italic_b start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT(5.13)∼similar-to\displaystyle\sim∼1n⁢∑i=1n∑1lp[∑j=1lp−1σ2⁢(ai⁢jp−1)2+σ2]1𝑛superscriptsubscript𝑖1𝑛superscriptsubscript1subscript𝑙𝑝delimited-[]superscriptsubscript𝑗1subscript𝑙𝑝1superscript𝜎2superscriptsuperscriptsubscript𝑎𝑖𝑗𝑝12superscript𝜎2\displaystyle\frac{1}{n}\sum_{i=1}^{n}\sum_{1}^{l_{p}}\left[\sum_{j=1}^{l_{p-1% }}\sigma^{2}(a_{ij}^{p-1})^{2}+\sigma^{2}\right]divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ∑ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT end_POSTSUPERSCRIPT [ ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ( italic_a start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p - 1 end_POSTSUPERSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ]

Tanh Activation Function.

We have

⟨Ep⟩∼1n⁢∑i=1nlp⁢(lp−1+1)⁢σ2∼lp⁢(lp−1+1)⁢σ2,similar-todelimited-⟨⟩subscript𝐸𝑝1𝑛superscriptsubscript𝑖1𝑛subscript𝑙𝑝subscript𝑙𝑝11superscript𝜎2similar-tosubscript𝑙𝑝subscript𝑙𝑝11superscript𝜎2\langle E_{p}\rangle\sim\frac{1}{n}\sum_{i=1}^{n}l_{p}(l_{p-1}+1)\sigma^{2}% \sim l_{p}(l_{p-1}+1)\sigma^{2},⟨ italic_E start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ⟩ ∼ divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPT + 1 ) italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ∼ italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPT + 1 ) italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ,(5.14)and the temperature of thep𝑝pitalic_p-th layer is

Tp∼⟨Ep⟩lp⁢(lp−1+1)⁢ln⁡σ∼(σ2ln⁡σ)similar-tosubscript𝑇𝑝delimited-⟨⟩subscript𝐸𝑝subscript𝑙𝑝subscript𝑙𝑝11𝜎similar-tosuperscript𝜎2𝜎\displaystyle T_{p}\sim\frac{\langle E_{p}\rangle}{l_{p}(l_{p-1}+1)\ln\sigma}% \sim\left(\frac{\sigma^{2}}{\ln\sigma}\right)italic_T start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ∼ divide start_ARG ⟨ italic_E start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ⟩ end_ARG start_ARG italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPT + 1 ) roman_ln italic_σ end_ARG ∼ ( divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ end_ARG )(5.15)forp≥2𝑝2p\geq 2italic_p ≥ 2, while since⟨E1⟩∼l1⁢X2¯⁢σ2similar-todelimited-⟨⟩subscript𝐸1subscript𝑙1¯superscript𝑋2superscript𝜎2\langle E_{1}\rangle\sim l_{1}\overline{X^{2}}\sigma^{2}⟨ italic_E start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ⟩ ∼ italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPTand

T1∼l1⁢X2¯l0+1⁢(σ2ln⁡σ),similar-tosubscript𝑇1subscript𝑙1¯superscript𝑋2subscript𝑙01superscript𝜎2𝜎T_{1}\sim\frac{l_{1}\overline{X^{2}}}{l_{0}+1}\left(\frac{\sigma^{2}}{\ln% \sigma}\right),italic_T start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ∼ divide start_ARG italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG end_ARG start_ARG italic_l start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT + 1 end_ARG ( divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ end_ARG ) ,(5.16)so the asymptotic temperatures of layers are

T1∝T2∼T3∼…∼TL,proportional-tosubscript𝑇1subscript𝑇2similar-tosubscript𝑇3similar-to…similar-tosubscript𝑇𝐿T_{1}\propto T_{2}\sim T_{3}\sim...\sim T_{L},italic_T start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ∝ italic_T start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ∼ italic_T start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT ∼ … ∼ italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT ,(5.17)which are independent from layers.

Sigmoid Activation Function.

In this case, the energy of thep𝑝pitalic_p-th layer can be written as

⟨Ep⟩∼lp⁢(ξp−1⁢lp−1+1)⁢σ2,similar-todelimited-⟨⟩subscript𝐸𝑝subscript𝑙𝑝subscript𝜉𝑝1subscript𝑙𝑝11superscript𝜎2\langle E_{p}\rangle\sim l_{p}(\xi_{p-1}l_{p-1}+1)\sigma^{2},⟨ italic_E start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ⟩ ∼ italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ( italic_ξ start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPT italic_l start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPT + 1 ) italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ,(5.18)whereξp−1subscript𝜉𝑝1\xi_{p-1}italic_ξ start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPTmeasures the fraction ofzp−1superscript𝑧𝑝1z^{p-1}italic_z start_POSTSUPERSCRIPT italic_p - 1 end_POSTSUPERSCRIPTtends towards one in the asymptotic case. And the temperature of the layer is

Tp∼(ξp−1⁢lp−1+1lp1+1)⁢(σ2ln⁡σ).similar-tosubscript𝑇𝑝subscript𝜉𝑝1subscript𝑙𝑝11subscript𝑙subscript𝑝11superscript𝜎2𝜎T_{p}\sim\left(\frac{\xi_{p-1}l_{p-1}+1}{l_{p_{1}}+1}\right)\left(\frac{\sigma% ^{2}}{\ln\sigma}\right).italic_T start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ∼ ( divide start_ARG italic_ξ start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPT italic_l start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPT + 1 end_ARG start_ARG italic_l start_POSTSUBSCRIPT italic_p start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPT + 1 end_ARG ) ( divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ end_ARG ) .(5.19)Althoughξpsubscript𝜉𝑝\xi_{p}italic_ξ start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPTmakes some fluctuation, sinceTp∝(σ2ln⁡σ)proportional-tosubscript𝑇𝑝superscript𝜎2𝜎T_{p}\propto\left(\frac{\sigma^{2}}{\ln\sigma}\right)italic_T start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ∝ ( divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ end_ARG )still holds, we see the temperature relation Equation (5.19) also holds for the case of Sigmoid activation function.

ReLU Activation Function.

Similar to Equation (5.10), we can derive the energy of thep𝑝pitalic_p-th layer as

⟨Ep⟩∼2⁢[∏k=1p(lk2)]⁢σ2⁢p⁢X2¯,similar-todelimited-⟨⟩subscript𝐸𝑝2delimited-[]superscriptsubscriptproduct𝑘1𝑝subscript𝑙𝑘2superscript𝜎2𝑝¯superscript𝑋2\langle E_{p}\rangle\sim 2\left[\prod_{k=1}^{p}\left(\frac{l_{k}}{2}\right)% \right]\sigma^{2p}\overline{X^{2}},⟨ italic_E start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ⟩ ∼ 2 [ ∏ start_POSTSUBSCRIPT italic_k = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT ( divide start_ARG italic_l start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT end_ARG start_ARG 2 end_ARG ) ] italic_σ start_POSTSUPERSCRIPT 2 italic_p end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ,(5.20)while the temperature in the layer is

Tp∼[∏k=1p−1(lk2)]⁢σ2⁢p⁢X2¯(lp−1+1)⁢ln⁡σ∝(σ2⁢pln⁡σ)⁢X2¯.similar-tosubscript𝑇𝑝delimited-[]superscriptsubscriptproduct𝑘1𝑝1subscript𝑙𝑘2superscript𝜎2𝑝¯superscript𝑋2subscript𝑙𝑝11𝜎proportional-tosuperscript𝜎2𝑝𝜎¯superscript𝑋2T_{p}\sim\left[\prod_{k=1}^{p-1}\left(\frac{l_{k}}{2}\right)\right]\frac{% \sigma^{2p}\overline{X^{2}}}{(l_{p-1}+1)\ln\sigma}\propto\left(\frac{\sigma^{2% p}}{\ln\sigma}\right)\overline{X^{2}}.italic_T start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ∼ [ ∏ start_POSTSUBSCRIPT italic_k = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_p - 1 end_POSTSUPERSCRIPT ( divide start_ARG italic_l start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT end_ARG start_ARG 2 end_ARG ) ] divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 italic_p end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG end_ARG start_ARG ( italic_l start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPT + 1 ) roman_ln italic_σ end_ARG ∝ ( divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 italic_p end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ end_ARG ) over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG .(5.21)As a result, the temperatures in each layer have

T1≪T2≪T3≪..≪TL.T_{1}\ll T_{2}\ll T_{3}\ll..\ll T_{L}.italic_T start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ≪ italic_T start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ≪ italic_T start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT ≪ . . ≪ italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT .(5.22)

5.3Heat Engine Analogy and Work Efficiency

Figure 7:The structure of a typical artificial neural network. The upper figure shows the neural network’s structure from a connectionism perspective, which includes input data and output result. In between, there areL𝐿Litalic_Llayers, with each layer’s size beingl1,l2,l3,…,lLsubscript𝑙1subscript𝑙2subscript𝑙3…subscript𝑙𝐿l_{1},l_{2},l_{3},...,l_{L}italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT , italic_l start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT , … , italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT. The connection between two adjacent layers, i.e., thep−1𝑝1p-1italic_p - 1and thep𝑝pitalic_p-th layers, can be expressed using the weight matrix𝑾p={Wi⁢jp}superscript𝑾𝑝subscriptsuperscript𝑊𝑝𝑖𝑗{\boldsymbol{W}^{p}}=\{W^{p}_{ij}\}bold_italic_W start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT = { italic_W start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT }. On the other hand, the lower figure views the neural network from a physical perspective, considering it as a complex heat engine system, where each layer represents a part of this system. Thep𝑝pitalic_p-th part of the system has a temperatureTpsubscript𝑇𝑝T_{p}italic_T start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT, and the released energy isΔ⁢EpΔsubscript𝐸𝑝\Delta E_{p}roman_Δ italic_E start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT. Only the energy released by the final part (theL𝐿Litalic_L-th layer) is useful externally and can be considered as the actual energy output of the system. From this we can calculate the work efficiency of this heat engine and compare the overall system temperatureTsycsubscript𝑇sycT_{\rm syc}italic_T start_POSTSUBSCRIPT roman_syc end_POSTSUBSCRIPTwith the temperature of each part{Tp}subscript𝑇𝑝\{T_{p}\}{ italic_T start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT }.The above results for neural networks can be interpreted from the perspective of thermodynamics and heat engine efficiency. Figure7shows a typical artificial neural network structure, we can view the system from a connectionist perspective (upper figure), or we can revisit this system from an energy perspective (lower figure). The process of data moving from input to output through the system is also a process of the series of layers releasing energy: thep𝑝pitalic_p-th layer has a local temperatureTpsubscript𝑇𝑝T_{p}italic_T start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPTand releases energyΔ⁢EpΔsubscript𝐸𝑝\Delta E_{p}roman_Δ italic_E start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT, while the eventually “useful“ energy released by the system (i.e. work done externally) isΔ⁢ELΔsubscript𝐸𝐿\Delta E_{L}roman_Δ italic_E start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT, and the system temperature isTsyssubscript𝑇sysT_{\rm sys}italic_T start_POSTSUBSCRIPT roman_sys end_POSTSUBSCRIPT.

We can define the work efficiencyη𝜂\etaitalic_ηof the system as the ratio between work done externally and total energy released, i.e.

η=Δ⁢ELΔ⁢Etot=Δ⁢EL∑p=1LΔ⁢Ep.𝜂Δsubscript𝐸𝐿Δsubscript𝐸totΔsubscript𝐸𝐿superscriptsubscript𝑝1𝐿Δsubscript𝐸𝑝\eta=\frac{\Delta E_{L}}{\Delta E_{\rm tot}}=\frac{\Delta E_{L}}{\sum_{p=1}^{L% }\Delta E_{p}}.italic_η = divide start_ARG roman_Δ italic_E start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT end_ARG start_ARG roman_Δ italic_E start_POSTSUBSCRIPT roman_tot end_POSTSUBSCRIPT end_ARG = divide start_ARG roman_Δ italic_E start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_p = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT roman_Δ italic_E start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT end_ARG .(5.23)The temperature of the system is

Tsyssubscript𝑇sys\displaystyle T_{\rm sys}italic_T start_POSTSUBSCRIPT roman_sys end_POSTSUBSCRIPT=\displaystyle==Δ⁢ELStot=Δ⁢ELΔ⁢Etot⁢Δ⁢EtotStotΔsubscript𝐸𝐿subscript𝑆totΔsubscript𝐸𝐿Δsubscript𝐸totΔsubscript𝐸totsubscript𝑆tot\displaystyle\frac{\Delta E_{L}}{S_{\rm tot}}=\frac{\Delta E_{L}}{\Delta E_{% \rm tot}}\frac{\Delta E_{\rm tot}}{S_{\rm tot}}divide start_ARG roman_Δ italic_E start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT end_ARG start_ARG italic_S start_POSTSUBSCRIPT roman_tot end_POSTSUBSCRIPT end_ARG = divide start_ARG roman_Δ italic_E start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT end_ARG start_ARG roman_Δ italic_E start_POSTSUBSCRIPT roman_tot end_POSTSUBSCRIPT end_ARG divide start_ARG roman_Δ italic_E start_POSTSUBSCRIPT roman_tot end_POSTSUBSCRIPT end_ARG start_ARG italic_S start_POSTSUBSCRIPT roman_tot end_POSTSUBSCRIPT end_ARG(5.24)=\displaystyle==η⁢(∑p=1LTp⁢Sp∑p=1LSp)𝜂superscriptsubscript𝑝1𝐿subscript𝑇𝑝subscript𝑆𝑝superscriptsubscript𝑝1𝐿subscript𝑆𝑝\displaystyle\eta\left(\frac{\sum_{p=1}^{L}T_{p}S_{p}}{\sum_{p=1}^{L}S_{p}}\right)italic_η ( divide start_ARG ∑ start_POSTSUBSCRIPT italic_p = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT italic_T start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT italic_S start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_p = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT italic_S start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT end_ARG )If we assume that the entropy of each individual layer has the same orderS1∼S2∼…∼SLsimilar-tosubscript𝑆1subscript𝑆2similar-to…similar-tosubscript𝑆𝐿S_{1}\sim S_{2}\sim...\sim S_{L}italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ∼ italic_S start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ∼ … ∼ italic_S start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT, thusη=TL/(∑pTp)𝜂subscript𝑇𝐿subscript𝑝subscript𝑇𝑝\eta=T_{L}/(\sum_{p}T_{p})italic_η = italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT / ( ∑ start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT italic_T start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ), and the relation between temperatures in individual layers and overall system temperature is

Tsys∼TLL∼(ηL)⁢∑p=1LTp∼η⁢T¯,similar-tosubscript𝑇syssubscript𝑇𝐿𝐿similar-to𝜂𝐿superscriptsubscript𝑝1𝐿subscript𝑇𝑝similar-to𝜂¯𝑇T_{\rm sys}\sim\frac{T_{L}}{L}\sim\left(\frac{\eta}{L}\right)\sum_{p=1}^{L}T_{% p}\sim\eta\overline{T},italic_T start_POSTSUBSCRIPT roman_sys end_POSTSUBSCRIPT ∼ divide start_ARG italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT end_ARG start_ARG italic_L end_ARG ∼ ( divide start_ARG italic_η end_ARG start_ARG italic_L end_ARG ) ∑ start_POSTSUBSCRIPT italic_p = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT italic_T start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ∼ italic_η over¯ start_ARG italic_T end_ARG ,(5.25)which shows that the overall system temperature is proportional to the average temperature of individual layers, with the work efficiencyη𝜂\etaitalic_ηas the coefficient.

5.3.1First Type of Heat Engine

For system using Tanh activation functions, we have

{Δ⁢E1=l1⁢X2¯⁢σ2,S1=l1⁢(l0+1)⁢ln⁡σΔ⁢E2=l2⁢(l1+1)⁢σ2,S2=l2⁢(l1+1)⁢ln⁡σΔ⁢E3=l3⁢(l2+1)⁢σ2,S3=l3⁢(l2+1)⁢ln⁡σ…⁢…Δ⁢EL=lL⁢(lL−1+1)⁢σ2,SL=lL⁢(lL−1+1)⁢ln⁡σ\left\{\begin{aligned} &\Delta E_{1}=l_{1}\overline{X^{2}}\sigma^{2},&S_{1}=l_% {1}(l_{0}+1)\ln\sigma\\ &\Delta E_{2}=l_{2}(l_{1}+1)\sigma^{2},&S_{2}=l_{2}(l_{1}+1)\ln\sigma\\ &\Delta E_{3}=l_{3}(l_{2}+1)\sigma^{2},&S_{3}=l_{3}(l_{2}+1)\ln\sigma\\ &......\\ &\Delta E_{L}=l_{L}(l_{L-1}+1)\sigma^{2},&S_{L}=l_{L}(l_{L-1}+1)\ln\sigma\end{% aligned}\right.{ start_ROW start_CELL end_CELL start_CELL roman_Δ italic_E start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT = italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT , end_CELL start_CELL italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT = italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT + 1 ) roman_ln italic_σ end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL roman_Δ italic_E start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT = italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT + 1 ) italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT , end_CELL start_CELL italic_S start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT = italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT + 1 ) roman_ln italic_σ end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL roman_Δ italic_E start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT = italic_l start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT + 1 ) italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT , end_CELL start_CELL italic_S start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT = italic_l start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT + 1 ) roman_ln italic_σ end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL … … end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL roman_Δ italic_E start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT = italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT + 1 ) italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT , end_CELL start_CELL italic_S start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT = italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT + 1 ) roman_ln italic_σ end_CELL end_ROW(5.26) The work efficiency of the system is

η=lL⁢(lL−1+1)l1⁢X2¯+∑p=1L−1lp+1⁢(lp+1).𝜂subscript𝑙𝐿subscript𝑙𝐿11subscript𝑙1¯superscript𝑋2superscriptsubscript𝑝1𝐿1subscript𝑙𝑝1subscript𝑙𝑝1\eta=\frac{l_{L}(l_{L-1}+1)}{l_{1}\overline{X^{2}}+\sum_{p=1}^{L-1}l_{p+1}(l_{% p}+1)}.italic_η = divide start_ARG italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT + 1 ) end_ARG start_ARG italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + ∑ start_POSTSUBSCRIPT italic_p = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L - 1 end_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_p + 1 end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT + 1 ) end_ARG .(5.27)From Equation (5.17) thatT2∼T3∼..∼TL∼TT_{2}\sim T_{3}\sim..\sim T_{L}\sim Titalic_T start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ∼ italic_T start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT ∼ . . ∼ italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT ∼ italic_Tand Equation (5.24), it is straightforward to obtain the temperature

Tsyssubscript𝑇sys\displaystyle T_{\rm sys}italic_T start_POSTSUBSCRIPT roman_sys end_POSTSUBSCRIPT=\displaystyle==lL⁢(lL−1+1)∑p=0L−1lp+1⁢(lp+1)⁢Tsubscript𝑙𝐿subscript𝑙𝐿11superscriptsubscript𝑝0𝐿1subscript𝑙𝑝1subscript𝑙𝑝1𝑇\displaystyle\frac{l_{L}(l_{L-1}+1)}{\sum_{p=0}^{L-1}l_{p+1}(l_{p}+1)}Tdivide start_ARG italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT + 1 ) end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_p = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L - 1 end_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_p + 1 end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT + 1 ) end_ARG italic_T(5.28)=\displaystyle==η⁢T⁢[1+S1Stot⁢(T1T−1)].𝜂𝑇delimited-[]1subscript𝑆1subscript𝑆totsubscript𝑇1𝑇1\displaystyle\eta T\left[1+\frac{S_{1}}{S_{\rm tot}}\left(\frac{T_{1}}{T}-1% \right)\right].italic_η italic_T [ 1 + divide start_ARG italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_ARG start_ARG italic_S start_POSTSUBSCRIPT roman_tot end_POSTSUBSCRIPT end_ARG ( divide start_ARG italic_T start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_ARG start_ARG italic_T end_ARG - 1 ) ] .whereStot=∑p=1LSpsubscript𝑆totsuperscriptsubscript𝑝1𝐿subscript𝑆𝑝S_{\rm tot}=\sum_{p=1}^{L}S_{p}italic_S start_POSTSUBSCRIPT roman_tot end_POSTSUBSCRIPT = ∑ start_POSTSUBSCRIPT italic_p = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT italic_S start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT. Note that Equation (5.28) also holds for system using Sigmoid activation function. For sufficient large total entropy or number of layers, we can writeTsys∼η⁢T∼η⁢TLsimilar-tosubscript𝑇sys𝜂𝑇similar-to𝜂subscript𝑇𝐿T_{\rm sys}\sim\eta T\sim\eta T_{L}italic_T start_POSTSUBSCRIPT roman_sys end_POSTSUBSCRIPT ∼ italic_η italic_T ∼ italic_η italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT.

From a thermodynamic perspective, for systems where each layer has a comparable temperature (this cap is caused by the constraints of the activation functions), and the work efficiency of the systemη≪1much-less-than𝜂1\eta\ll 1italic_η ≪ 1, and the system temperatureTsys∼η⁢TLsimilar-tosubscript𝑇sys𝜂subscript𝑇𝐿T_{\rm sys}\sim\eta T_{L}italic_T start_POSTSUBSCRIPT roman_sys end_POSTSUBSCRIPT ∼ italic_η italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT, we refer to such neural network systems as theFirst Type of Heat Engine.

5.3.2Second Type of Heat Engine

On the other hand, for system using ReLU activation function, we have

{Δ⁢E1=l1⁢σ2⁢X2¯,S1=l1⁢(l0+1)⁢ln⁡σΔ⁢E2=12⁢l2⁢l1⁢σ4⁢X2¯,S2=l2⁢(l1+1)⁢ln⁡σΔ⁢E3=122⁢l3⁢l2⁢l1⁢σ6⁢X2¯,S3=l3⁢(l2+1)⁢ln⁡σ…⁢…Δ⁢EL=12L−1⁢(∏p=1Llp)⁢σ2⁢L⁢X2¯,SL=lL⁢(lL−1+1)⁢ln⁡σ\left\{\begin{aligned} &\Delta E_{1}=l_{1}\sigma^{2}\overline{X^{2}},&S_{1}=l_% {1}(l_{0}+1)\ln\sigma\\ &\Delta E_{2}=\frac{1}{2}l_{2}l_{1}\sigma^{4}\overline{X^{2}},&S_{2}=l_{2}(l_{% 1}+1)\ln\sigma\\ &\Delta E_{3}=\frac{1}{2^{2}}l_{3}l_{2}l_{1}\sigma^{6}\overline{X^{2}},&S_{3}=% l_{3}(l_{2}+1)\ln\sigma\\ &......\\ &\Delta E_{L}=\frac{1}{2^{L-1}}\left(\prod_{p=1}^{L}l_{p}\right)\sigma^{2L}% \overline{X^{2}},&S_{L}=l_{L}(l_{L-1}+1)\ln\sigma\end{aligned}\right.{ start_ROW start_CELL end_CELL start_CELL roman_Δ italic_E start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT = italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG , end_CELL start_CELL italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT = italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT + 1 ) roman_ln italic_σ end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL roman_Δ italic_E start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT = divide start_ARG 1 end_ARG start_ARG 2 end_ARG italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT italic_σ start_POSTSUPERSCRIPT 4 end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG , end_CELL start_CELL italic_S start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT = italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT + 1 ) roman_ln italic_σ end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL roman_Δ italic_E start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT = divide start_ARG 1 end_ARG start_ARG 2 start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG italic_l start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT italic_σ start_POSTSUPERSCRIPT 6 end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG , end_CELL start_CELL italic_S start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT = italic_l start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT + 1 ) roman_ln italic_σ end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL … … end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL roman_Δ italic_E start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT = divide start_ARG 1 end_ARG start_ARG 2 start_POSTSUPERSCRIPT italic_L - 1 end_POSTSUPERSCRIPT end_ARG ( ∏ start_POSTSUBSCRIPT italic_p = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ) italic_σ start_POSTSUPERSCRIPT 2 italic_L end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG , end_CELL start_CELL italic_S start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT = italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT + 1 ) roman_ln italic_σ end_CELL end_ROW(5.29)Note thatΔ⁢E1≪Δ⁢E2≪…⁢Δ⁢ELmuch-less-thanΔsubscript𝐸1Δsubscript𝐸2much-less-than…Δsubscript𝐸𝐿\Delta E_{1}\ll\Delta E_{2}\ll...\Delta E_{L}roman_Δ italic_E start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ≪ roman_Δ italic_E start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ≪ … roman_Δ italic_E start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPTandT1≪T2≪…⁢TLmuch-less-thansubscript𝑇1subscript𝑇2much-less-than…subscript𝑇𝐿T_{1}\ll T_{2}\ll...T_{L}italic_T start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ≪ italic_T start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ≪ … italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT. We have the work efficiency

η=12L−1⁢(∏p=1Llp)⁢σ2⁢L∑k=1L12k−1⁢(∏p=1klp)⁢σ2⁢k≈1.𝜂1superscript2𝐿1superscriptsubscriptproduct𝑝1𝐿subscript𝑙𝑝superscript𝜎2𝐿superscriptsubscript𝑘1𝐿1superscript2𝑘1superscriptsubscriptproduct𝑝1𝑘subscript𝑙𝑝superscript𝜎2𝑘1\eta=\frac{\frac{1}{2^{L-1}}\left(\prod_{p=1}^{L}l_{p}\right)\sigma^{2L}}{\sum% _{k=1}^{L}\frac{1}{2^{k-1}}\left(\prod_{p=1}^{k}l_{p}\right)\sigma^{2k}}% \approx 1.italic_η = divide start_ARG divide start_ARG 1 end_ARG start_ARG 2 start_POSTSUPERSCRIPT italic_L - 1 end_POSTSUPERSCRIPT end_ARG ( ∏ start_POSTSUBSCRIPT italic_p = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ) italic_σ start_POSTSUPERSCRIPT 2 italic_L end_POSTSUPERSCRIPT end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_k = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT divide start_ARG 1 end_ARG start_ARG 2 start_POSTSUPERSCRIPT italic_k - 1 end_POSTSUPERSCRIPT end_ARG ( ∏ start_POSTSUBSCRIPT italic_p = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ) italic_σ start_POSTSUPERSCRIPT 2 italic_k end_POSTSUPERSCRIPT end_ARG ≈ 1 .(5.30)And

Tsys=TL⁢SL∑i=1LSi∼η⁢TL⁢(SLStot).subscript𝑇syssubscript𝑇𝐿subscript𝑆𝐿superscriptsubscript𝑖1𝐿subscript𝑆𝑖similar-to𝜂subscript𝑇𝐿subscript𝑆𝐿subscript𝑆tot\displaystyle T_{\rm sys}=\frac{T_{L}S_{L}}{\sum_{i=1}^{L}S_{i}}\sim\eta T_{L}% \left(\frac{S_{L}}{S_{\rm tot}}\right).italic_T start_POSTSUBSCRIPT roman_sys end_POSTSUBSCRIPT = divide start_ARG italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT italic_S start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT italic_S start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG ∼ italic_η italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT ( divide start_ARG italic_S start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT end_ARG start_ARG italic_S start_POSTSUBSCRIPT roman_tot end_POSTSUBSCRIPT end_ARG ) .(5.31)For sufficient complex system with large number of layers,Tsys≪η⁢TLmuch-less-thansubscript𝑇sys𝜂subscript𝑇𝐿T_{\rm sys}\ll\eta T_{L}italic_T start_POSTSUBSCRIPT roman_sys end_POSTSUBSCRIPT ≪ italic_η italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT, which is different from the result in Section5.3.1thatTsys∼η⁢TLsimilar-tosubscript𝑇sys𝜂subscript𝑇𝐿T_{\rm sys}\sim\eta T_{L}italic_T start_POSTSUBSCRIPT roman_sys end_POSTSUBSCRIPT ∼ italic_η italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT. Activation functions can be used to control the temperatures in the system.

From a thermodynamic perspective, in principle there is no upper limit on the local temperature of the system. As the number of layers increases, the local temperatures gradually increase, the work efficiencyη𝜂\etaitalic_ηof the system approaches 1η≈1𝜂1\eta\approx 1italic_η ≈ 1, and the system temperatureTsys∼TL⁢SL/Stotsimilar-tosubscript𝑇syssubscript𝑇𝐿subscript𝑆𝐿subscript𝑆totT_{\rm sys}\sim T_{L}S_{L}/S_{\rm tot}italic_T start_POSTSUBSCRIPT roman_sys end_POSTSUBSCRIPT ∼ italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT italic_S start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT / italic_S start_POSTSUBSCRIPT roman_tot end_POSTSUBSCRIPT. We refer to neural network systems with these properties as the as theSecond Type of Heat Engine.

5.4Other Parameter Initializations

In the discussions from Section5.1to Section5.3, we assumed all the parameters initially follow normal distributions with the same standard deviationσ𝜎\sigmaitalic_σ. However, in practice, there are different methods to initialize the parameters.

One common method is to use a different standard deviation for the initial normal distribution of parameters in each layer, for example, setσp∝1/lp−1proportional-tosubscript𝜎𝑝1subscript𝑙𝑝1\sigma_{p}\propto\sqrt{1/l_{p-1}}italic_σ start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ∝ square-root start_ARG 1 / italic_l start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPT end_ARG, or

σp=σ0lp−1.subscript𝜎𝑝subscript𝜎0subscript𝑙𝑝1\sigma_{p}=\frac{\sigma_{0}}{\sqrt{l_{p-1}}}.italic_σ start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT = divide start_ARG italic_σ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_ARG start_ARG square-root start_ARG italic_l start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPT end_ARG end_ARG .(5.32)So for the default case (Section5.1.1), the energy can be written as

⟨E0⟩∼1n⁢∑ilL⁢(lL−1+1)lL−1⁢σ02∼σ02⁢lL⁢(1+1lL−1).similar-todelimited-⟨⟩subscript𝐸01𝑛subscript𝑖subscript𝑙𝐿subscript𝑙𝐿11subscript𝑙𝐿1superscriptsubscript𝜎02similar-tosuperscriptsubscript𝜎02subscript𝑙𝐿11subscript𝑙𝐿1\displaystyle\langle E_{0}\rangle\sim\frac{1}{n}\sum_{i}\frac{l_{L}(l_{L-1}+1)% }{l_{L-1}}\sigma_{0}^{2}\sim\sigma_{0}^{2}l_{L}\left(1+\frac{1}{l_{L-1}}\right).⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩ ∼ divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT divide start_ARG italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT + 1 ) end_ARG start_ARG italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT end_ARG italic_σ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ∼ italic_σ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT ( 1 + divide start_ARG 1 end_ARG start_ARG italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT end_ARG ) .(5.33)Note that for any layer the entropySp=lp⁢(lp−1+1)⁢ln⁡σp=lp⁢(lp−1+1)⁢ln⁡σ0+const.→lp⁢(lp−1+1)⁢ln⁡σ0subscript𝑆𝑝subscript𝑙𝑝subscript𝑙𝑝11subscript𝜎𝑝subscript𝑙𝑝subscript𝑙𝑝11subscript𝜎0const.→subscript𝑙𝑝subscript𝑙𝑝11subscript𝜎0S_{p}=l_{p}(l_{p-1}+1)\ln\sigma_{p}=l_{p}(l_{p-1}+1)\ln\sigma_{0}+\textrm{% const.}\rightarrow l_{p}(l_{p-1}+1)\ln\sigma_{0}italic_S start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT = italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPT + 1 ) roman_ln italic_σ start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT = italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPT + 1 ) roman_ln italic_σ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT + const. → italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPT + 1 ) roman_ln italic_σ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPTfor asymptotic case. Thus, the system temperature is

T∼lL⁢σ02∑p=1Llp⁢(lp+1)⁢ln⁡σ0⁢(1+1lL−1)∼lL⁢σ02∑p=1Llp⁢(lp+1)⁢ln⁡σ0.similar-to𝑇subscript𝑙𝐿superscriptsubscript𝜎02superscriptsubscript𝑝1𝐿subscript𝑙𝑝subscript𝑙𝑝1subscript𝜎011subscript𝑙𝐿1similar-tosubscript𝑙𝐿superscriptsubscript𝜎02superscriptsubscript𝑝1𝐿subscript𝑙𝑝subscript𝑙𝑝1subscript𝜎0\displaystyle T\sim\frac{l_{L}\sigma_{0}^{2}}{\sum_{p=1}^{L}l_{p}(l_{p}+1)\ln% \sigma_{0}}\left(1+\frac{1}{l_{L-1}}\right)\sim\frac{l_{L}\sigma_{0}^{2}}{\sum% _{p=1}^{L}l_{p}(l_{p}+1)\ln\sigma_{0}}.italic_T ∼ divide start_ARG italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT italic_σ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_p = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT + 1 ) roman_ln italic_σ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_ARG ( 1 + divide start_ARG 1 end_ARG start_ARG italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT end_ARG ) ∼ divide start_ARG italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT italic_σ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_p = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT + 1 ) roman_ln italic_σ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_ARG .(5.34)Compared to Equation (5.6), the relationT∝σ2ln⁡σproportional-to𝑇superscript𝜎2𝜎T\propto\frac{\sigma^{2}}{\ln\sigma}italic_T ∝ divide start_ARG italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ end_ARGstill holds, but the temperature is1/(lL−1+1)1subscript𝑙𝐿111/(l_{L-1}+1)1 / ( italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT + 1 )of the original version. For each layer, the local temperatures are

Tp∼σ02lp−1⁢ln⁡σ0similar-tosubscript𝑇𝑝superscriptsubscript𝜎02subscript𝑙𝑝1subscript𝜎0\displaystyle T_{p}\sim\frac{\sigma_{0}^{2}}{l_{p-1}\ln\sigma_{0}}italic_T start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ∼ divide start_ARG italic_σ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG italic_l start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPT roman_ln italic_σ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_ARG(5.35)forp≥2𝑝2p\geq 2italic_p ≥ 2. We haveTL∼lL−2lL−1⁢TL−1∼lL−3lL−1⁢TL−2∼…∼l1lL−1⁢T2similar-tosubscript𝑇𝐿subscript𝑙𝐿2subscript𝑙𝐿1subscript𝑇𝐿1similar-tosubscript𝑙𝐿3subscript𝑙𝐿1subscript𝑇𝐿2similar-to…similar-tosubscript𝑙1subscript𝑙𝐿1subscript𝑇2T_{L}\sim\frac{l_{L-2}}{l_{L-1}}T_{L-1}\sim\frac{l_{L-3}}{l_{L-1}}T_{L-2}\sim.% ..\sim\frac{l_{1}}{l_{L-1}}T_{2}italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT ∼ divide start_ARG italic_l start_POSTSUBSCRIPT italic_L - 2 end_POSTSUBSCRIPT end_ARG start_ARG italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT end_ARG italic_T start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT ∼ divide start_ARG italic_l start_POSTSUBSCRIPT italic_L - 3 end_POSTSUBSCRIPT end_ARG start_ARG italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT end_ARG italic_T start_POSTSUBSCRIPT italic_L - 2 end_POSTSUBSCRIPT ∼ … ∼ divide start_ARG italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_ARG start_ARG italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT end_ARG italic_T start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT. The work efficiency of the system

η∼lLl1l0⁢X2¯+∑p=2Llp,similar-to𝜂subscript𝑙𝐿subscript𝑙1subscript𝑙0¯superscript𝑋2superscriptsubscript𝑝2𝐿subscript𝑙𝑝\eta\sim\frac{l_{L}}{\frac{l_{1}}{l_{0}}\overline{X^{2}}+\sum_{p=2}^{L}l_{p}},italic_η ∼ divide start_ARG italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT end_ARG start_ARG divide start_ARG italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_ARG start_ARG italic_l start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_ARG over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + ∑ start_POSTSUBSCRIPT italic_p = 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT end_ARG ,(5.36)and the system temperature is

Tsys=η⁢TL⁢S1+∑p=2LlL−1⁢Splp−1Stot−η⁢Δ⁢T⁢S1Stot,subscript𝑇sys𝜂subscript𝑇𝐿subscript𝑆1superscriptsubscript𝑝2𝐿subscript𝑙𝐿1subscript𝑆𝑝subscript𝑙𝑝1subscript𝑆tot𝜂Δ𝑇subscript𝑆1subscript𝑆tot\displaystyle T_{\rm sys}=\eta T_{L}\frac{S_{1}+\sum_{p=2}^{L}\frac{l_{L-1}S_{% p}}{l_{p-1}}}{S_{\rm tot}}-\eta\Delta T\frac{S_{1}}{S_{\rm tot}},italic_T start_POSTSUBSCRIPT roman_sys end_POSTSUBSCRIPT = italic_η italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT divide start_ARG italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT + ∑ start_POSTSUBSCRIPT italic_p = 2 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT divide start_ARG italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT italic_S start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT end_ARG start_ARG italic_l start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPT end_ARG end_ARG start_ARG italic_S start_POSTSUBSCRIPT roman_tot end_POSTSUBSCRIPT end_ARG - italic_η roman_Δ italic_T divide start_ARG italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_ARG start_ARG italic_S start_POSTSUBSCRIPT roman_tot end_POSTSUBSCRIPT end_ARG ,(5.37)whereΔ⁢T=TL−T1Δ𝑇subscript𝑇𝐿subscript𝑇1\Delta T=T_{L}-T_{1}roman_Δ italic_T = italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT - italic_T start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT. Ifl1∼l2∼l3∼…∼lL−1similar-tosubscript𝑙1subscript𝑙2similar-tosubscript𝑙3similar-to…similar-tosubscript𝑙𝐿1l_{1}\sim l_{2}\sim l_{3}\sim...\sim l_{L-1}italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ∼ italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ∼ italic_l start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT ∼ … ∼ italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT,

Tsys∼η⁢(TL−S1Stot⁢Δ⁢T)∼η⁢TL,similar-tosubscript𝑇sys𝜂subscript𝑇𝐿subscript𝑆1subscript𝑆totΔ𝑇similar-to𝜂subscript𝑇𝐿T_{\rm sys}\sim\eta\left(T_{L}-\frac{S_{1}}{S_{\rm tot}}\Delta T\right)\sim% \eta T_{L},italic_T start_POSTSUBSCRIPT roman_sys end_POSTSUBSCRIPT ∼ italic_η ( italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT - divide start_ARG italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_ARG start_ARG italic_S start_POSTSUBSCRIPT roman_tot end_POSTSUBSCRIPT end_ARG roman_Δ italic_T ) ∼ italic_η italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT ,(5.38)which is similar to Equation (5.28).

For the system with ReLU activation function (see Section5.1.3), usingσp=σ0/lp−1subscript𝜎𝑝subscript𝜎0subscript𝑙𝑝1\sigma_{p}=\sigma_{0}/\sqrt{l_{p-1}}italic_σ start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT = italic_σ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT / square-root start_ARG italic_l start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPT end_ARG, we obtain

⟨E0⟩∼12L−1⁢lLl0⁢σ02⁢L⁢X2¯,similar-todelimited-⟨⟩subscript𝐸01superscript2𝐿1subscript𝑙𝐿subscript𝑙0superscriptsubscript𝜎02𝐿¯superscript𝑋2\displaystyle\langle E_{0}\rangle\sim\frac{1}{2^{L-1}}\frac{l_{L}}{l_{0}}% \sigma_{0}^{2L}\overline{X^{2}},⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩ ∼ divide start_ARG 1 end_ARG start_ARG 2 start_POSTSUPERSCRIPT italic_L - 1 end_POSTSUPERSCRIPT end_ARG divide start_ARG italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT end_ARG start_ARG italic_l start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_ARG italic_σ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 italic_L end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ,(5.39)We still have

T∝(σ02⁢Lln⁡σ0)⁢X2¯,proportional-to𝑇superscriptsubscript𝜎02𝐿subscript𝜎0¯superscript𝑋2T\propto\left(\frac{\sigma_{0}^{2L}}{\ln\sigma_{0}}\right)\overline{X^{2}},italic_T ∝ ( divide start_ARG italic_σ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 italic_L end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_ARG ) over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG ,(5.40)but with a lower temperature compared with Equation (5.21). The local temperature in each layer is

Tp∼X2¯2p−1⁢l0⁢(lp−1+1)⁢σ02⁢pln⁡σ0,similar-tosubscript𝑇𝑝¯superscript𝑋2superscript2𝑝1subscript𝑙0subscript𝑙𝑝11superscriptsubscript𝜎02𝑝subscript𝜎0T_{p}\sim\frac{\overline{X^{2}}}{2^{p-1}l_{0}(l_{p-1}+1)}\frac{\sigma_{0}^{2p}% }{\ln\sigma_{0}},italic_T start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ∼ divide start_ARG over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG end_ARG start_ARG 2 start_POSTSUPERSCRIPT italic_p - 1 end_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_p - 1 end_POSTSUBSCRIPT + 1 ) end_ARG divide start_ARG italic_σ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 italic_p end_POSTSUPERSCRIPT end_ARG start_ARG roman_ln italic_σ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT end_ARG ,(5.41)andT1=σ02⁢X2¯/[l0⁢(l0+1)⁢ln⁡σ]subscript𝑇1superscriptsubscript𝜎02¯superscript𝑋2delimited-[]subscript𝑙0subscript𝑙01𝜎T_{1}=\sigma_{0}^{2}\overline{X^{2}}/[l_{0}(l_{0}+1)\ln\sigma]italic_T start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT = italic_σ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG / [ italic_l start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT + 1 ) roman_ln italic_σ ]. Such, we haveT1≪T2≪T2≪…≪TLmuch-less-thansubscript𝑇1subscript𝑇2much-less-thansubscript𝑇2much-less-than…much-less-thansubscript𝑇𝐿T_{1}\ll T_{2}\ll T_{2}\ll...\ll T_{L}italic_T start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ≪ italic_T start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ≪ italic_T start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ≪ … ≪ italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT, the work efficiencyη≈1𝜂1\eta\approx 1italic_η ≈ 1, and

Tsys∼η⁢TL⁢(SLStot),similar-tosubscript𝑇sys𝜂subscript𝑇𝐿subscript𝑆𝐿subscript𝑆tot\displaystyle T_{\rm sys}\sim\eta T_{L}\left(\frac{S_{L}}{S_{\rm tot}}\right),italic_T start_POSTSUBSCRIPT roman_sys end_POSTSUBSCRIPT ∼ italic_η italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT ( divide start_ARG italic_S start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT end_ARG start_ARG italic_S start_POSTSUBSCRIPT roman_tot end_POSTSUBSCRIPT end_ARG ) ,(5.42)The key results for Type II heat engine still holds.

As a conclusion, if the initial normal distribution of the parameters is different, the relation between the system temperature and the local temperatures in the first type of heat engine varies, and the system’s efficiency also differs slightly. For the second type of heat engine, the expressions for system efficiency and temperature remain the same.

If the initial setting of the system parameters is uniform distribution, the aforementioned conclusions still hold. We discuss the case where the parameters are initially set to a uniform distribution in the AppendixE.

6Conclusions

We develop a thermodynamic framework for machine learning (ML) systems. Similar to physical thermodynamic systems, which are characterized by energy and entropy, ML systems also possess these characteristics. This comparison inspired us to integrate the concept of temperature into ML systems, grounded in the fundamental principles of thermodynamics, and to establish a basic thermodynamic framework for machine learning systems with non-Boltzmann distributions.

We first introduce the concept of states into ML systems. We propose three fundamental elements of a basic ML system: the model (with parameter set𝝁𝝁\boldsymbol{\mu}bold_italic_μ), data (𝒟=𝒳×Y𝒟𝒳𝑌\mathcal{D}=\mathcal{X}\times Ycaligraphic_D = caligraphic_X × italic_Y), and energy (E𝐸Eitalic_E) (see Figure2). ML systems have two types of states. For a given dataset, all possible𝝁𝝁{\boldsymbol{\mu}}bold_italic_μin the parameter space forms a state, known as theType I State. This state represents an ML system that has not yet been trained, where each{Ei,𝝁i}subscript𝐸𝑖subscript𝝁𝑖\{E_{i},\boldsymbol{\mu}_{i}\}{ italic_E start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , bold_italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT }can be considered as a particle. After model training, all the particles converge to{E^,𝝁^}^𝐸^𝝁\{\hat{E},\hat{\boldsymbol{\mu}}\}{ over^ start_ARG italic_E end_ARG , over^ start_ARG bold_italic_μ end_ARG }, and the transition from all{Ei,𝝁i}subscript𝐸𝑖subscript𝝁𝑖\{E_{i},\boldsymbol{\mu}_{i}\}{ italic_E start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , bold_italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT }to converged{E^,𝝁^}^𝐸^𝝁\{\hat{E},\hat{\boldsymbol{\mu}}\}{ over^ start_ARG italic_E end_ARG , over^ start_ARG bold_italic_μ end_ARG }can be considered an isothermal phase transition, allowing us to calculate the corresponding temperature of the system. On the other hand, unlike the Type I state, theType II stateof an ML system is defined by the dataset after specifying a set of parameters𝝁^^𝝁\hat{\boldsymbol{\mu}}over^ start_ARG bold_italic_μ end_ARG. The transformation of the Type II state corresponds to shifts in ML data and the continuous refresh of the model.

After providing a comprehensive discussion and unified scenario on global temperature of ML systems, we primarily discuss the temperature of Type I state in phase transitions. We consider that the initial potential energy of an ML system is described by the model loss functions, and the energy adheres to the principle of minimum potential energy. Regarding the probability distributionf⁢(𝝁)𝑓𝝁f({\boldsymbol{\mu}})italic_f ( bold_italic_μ )within the parameter space, we argue that the temperature of the Type I state is (Equation2.29)

T=Ef−∫E⁢(𝝁)⁢f⁢(𝝁)⁢𝑑𝝁∫f⁢(𝝁)⁢log⁡[f⁢(𝝁)]⁢𝑑𝝁.𝑇subscript𝐸𝑓𝐸𝝁𝑓𝝁differential-d𝝁𝑓𝝁𝑓𝝁differential-d𝝁T=\frac{E_{f}-\int E({\boldsymbol{\mu}})f({\boldsymbol{\mu}})d{\boldsymbol{\mu% }}}{\int f({\boldsymbol{\mu}})\log[f({\boldsymbol{\mu}})]d{\boldsymbol{\mu}}}.italic_T = divide start_ARG italic_E start_POSTSUBSCRIPT italic_f end_POSTSUBSCRIPT - ∫ italic_E ( bold_italic_μ ) italic_f ( bold_italic_μ ) italic_d bold_italic_μ end_ARG start_ARG ∫ italic_f ( bold_italic_μ ) roman_log [ italic_f ( bold_italic_μ ) ] italic_d bold_italic_μ end_ARG .It is important to clarify that the system energy we propose is fundamentally different from the energy in EBMs. Within the EBM framework, both the loss function and the energy function coexist. The energy function is minimized by the inference process, while the loss functional is minimized by the learning process. In our theoretical framework, we do not distinguish between the loss function and the energy function.

Next, we derive the temperatures of a various ML toy systems, which have models with initial parameters following either normal or uniform distributions, while their energy forms include linear regression with Mean Squared Error (MSE) or Mean Absolute Error (MAE) and regularization, as well as logistic regression with cross entropy. Certain system temperatures under certain inertial parameter distributions and energy forms have analytical solutions. For example, for an initial normal distribution of parameters and linear regression, the system temperature is (Equation [3.10])

T=∑j=1Kσj2⁢Xj2¯+Y2¯−tr⁢{Cx−Cx⁢y⁢Cy−1⁢Cy⁢x}¯∑j=1Kln⁡σj+K2⁢[1+ln⁡(2⁢π)],𝑇superscriptsubscript𝑗1𝐾superscriptsubscript𝜎𝑗2¯superscriptsubscript𝑋𝑗2¯superscript𝑌2¯trsubscript𝐶𝑥subscript𝐶𝑥𝑦superscriptsubscript𝐶𝑦1subscript𝐶𝑦𝑥superscriptsubscript𝑗1𝐾subscript𝜎𝑗𝐾2delimited-[]12𝜋\displaystyle T=\frac{\sum_{j=1}^{K}\sigma_{j}^{2}\overline{X_{j}^{2}}+% \overline{Y^{2}}-\overline{{\rm tr}\{C_{x}-C_{xy}C_{y}^{-1}C_{yx}\}}}{\sum_{j=% 1}^{K}\ln\sigma_{j}+\frac{K}{2}[1+\ln(2\pi)]},italic_T = divide start_ARG ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG + over¯ start_ARG italic_Y start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG - over¯ start_ARG roman_tr { italic_C start_POSTSUBSCRIPT italic_x end_POSTSUBSCRIPT - italic_C start_POSTSUBSCRIPT italic_x italic_y end_POSTSUBSCRIPT italic_C start_POSTSUBSCRIPT italic_y end_POSTSUBSCRIPT start_POSTSUPERSCRIPT - 1 end_POSTSUPERSCRIPT italic_C start_POSTSUBSCRIPT italic_y italic_x end_POSTSUBSCRIPT } end_ARG end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_K end_POSTSUPERSCRIPT roman_ln italic_σ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT + divide start_ARG italic_K end_ARG start_ARG 2 end_ARG [ 1 + roman_ln ( 2 italic_π ) ] end_ARG ,whereK𝐾Kitalic_Kis the dimension of the parameter space, andXisubscript𝑋𝑖X_{i}italic_X start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPTis thei𝑖iitalic_i-th component of the data point𝑿𝑿\boldsymbol{X}bold_italic_X. In more general cases, we consider the asymptotic solutions for the system temperatures, in particular for normal distributions{σi}→σ→∞→subscript𝜎𝑖𝜎→\{\sigma_{i}\}\rightarrow\sigma\rightarrow\infty{ italic_σ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT } → italic_σ → ∞, or uniform distributions length{li}→l→∞→subscript𝑙𝑖𝑙→\{l_{i}\}\rightarrow l\rightarrow\infty{ italic_l start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT } → italic_l → ∞, Table3gives the asymptotic solutions.

Table 3:The asymptotic temperatures of various ML systems. The energy forms of the systems include linear regression (LineR) with MSE and MAE and regularization (reg), logistic regression (logR) with Cross Entropy (CE), and neural network (NN) with Tanh, Sigmoid and ReLU activation functions. The initial parameter (𝝁𝝁{\boldsymbol{\mu}}bold_italic_μ) distributions for the systems are either normal or uniform.We also give a physical explanation of the ML temperature. If two systems,A𝐴Aitalic_AandB𝐵Bitalic_B, have the same data distributionXA2¯=XB2¯¯superscriptsubscript𝑋𝐴2¯superscriptsubscript𝑋𝐵2\overline{X_{A}^{2}}=\overline{X_{B}^{2}}over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_A end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG = over¯ start_ARG italic_X start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG, they also share the same temperature. From a physical perspective, the two systems are in equilibrium. The process of mixing non-equilibrium systems to achieve equilibrium essentially involves the transfer of energy from the higher-temperature system to the lower-temperature system. The scenarios discussed above regarding temperature changes in equilibrium, non-equilibrium, and model retraining systems bear a strong resemblance to traditional thermodynamic systems. This forms the thermodynamic landscape of temperature in ML systems.

Next we discuss ML systems with neural network models. For a neural network withL𝐿Litalic_Llayers, we propose that each layers of a neural network has its own local temperatureTisubscript𝑇𝑖T_{i}italic_T start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT(i=1,2,3,…,L𝑖123…𝐿i=1,2,3,...,Litalic_i = 1 , 2 , 3 , … , italic_L), also the system has its global temperatureTsyssubscript𝑇sysT_{\rm sys}italic_T start_POSTSUBSCRIPT roman_sys end_POSTSUBSCRIPT. We found that the relationship between the local temperatures of a neural network depends on the model activation function. For example, with Tanh and Sigmoid activation functions, we haveT1∝T2∼T3∼…∼TLproportional-tosubscript𝑇1subscript𝑇2similar-tosubscript𝑇3similar-to…similar-tosubscript𝑇𝐿T_{1}\propto T_{2}\sim T_{3}\sim...\sim T_{L}italic_T start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ∝ italic_T start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ∼ italic_T start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT ∼ … ∼ italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT(Equation [5.17]), while for ReLU activation function, we haveT1≪T2≪T3≪..≪TLT_{1}\ll T_{2}\ll T_{3}\ll..\ll T_{L}italic_T start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ≪ italic_T start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ≪ italic_T start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT ≪ . . ≪ italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT(Equation [5.22]). These results for neural networks can be interpreted from the perspective of thermodynamics andheat engine efficiency. Viewing a neural network system as aheat engine(see Figure7), where the energy output of each layer isΔ⁢EiΔsubscript𝐸𝑖\Delta E_{i}roman_Δ italic_E start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT, we can define the efficiency of this heat engine system as

η=Δ⁢ELΔ⁢Etot=Δ⁢EL∑p=1LΔ⁢Ep.𝜂Δsubscript𝐸𝐿Δsubscript𝐸totΔsubscript𝐸𝐿superscriptsubscript𝑝1𝐿Δsubscript𝐸𝑝\eta=\frac{\Delta E_{L}}{\Delta E_{\rm tot}}=\frac{\Delta E_{L}}{\sum_{p=1}^{L% }\Delta E_{p}}.italic_η = divide start_ARG roman_Δ italic_E start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT end_ARG start_ARG roman_Δ italic_E start_POSTSUBSCRIPT roman_tot end_POSTSUBSCRIPT end_ARG = divide start_ARG roman_Δ italic_E start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_p = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT roman_Δ italic_E start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT end_ARG .(6.1)We then classify neural networks based on their work efficiency, identifying them as one of two types of heat engines. For systems using Tanh activation functions, we observe thatT1∝T2∼T3∼…∼TL∼Tproportional-tosubscript𝑇1subscript𝑇2similar-tosubscript𝑇3similar-to…similar-tosubscript𝑇𝐿similar-to𝑇T_{1}\propto T_{2}\sim T_{3}\sim...\sim T_{L}\sim Titalic_T start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ∝ italic_T start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ∼ italic_T start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT ∼ … ∼ italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT ∼ italic_T,η≪1much-less-than𝜂1\eta\ll 1italic_η ≪ 1and the system tempeatureTsys∼η⁢TLsimilar-tosubscript𝑇sys𝜂subscript𝑇𝐿T_{\rm sys}\sim\eta T_{L}italic_T start_POSTSUBSCRIPT roman_sys end_POSTSUBSCRIPT ∼ italic_η italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT. We define such low-efficient neural network systems as theFirst Type of Heat Engine. Conversely, for systems employing the ReLU activation function, we find thatη≈1𝜂1\eta\approx 1italic_η ≈ 1, and the system temperatureTsys≪η⁢TLmuch-less-thansubscript𝑇sys𝜂subscript𝑇𝐿T_{\rm sys}\ll\eta T_{L}italic_T start_POSTSUBSCRIPT roman_sys end_POSTSUBSCRIPT ≪ italic_η italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT. We refer to neural network systems with these properties as the as theSecond Type of Heat Engine.

Appendix AAppendix I: The Unified Scenario for Type I and Type II States

Figure 8:The entire process of training and retraining a ML system from a thermodynamic perspective, which shows a unified scenario to combine Type I and Type II states from Section2.1for discrete case (upper figure (a)) and continuous case (lower figure (b)). The black dots in upper figures represent Type I state phase transition. This figure develops the idea in Figure1. For detailed discussion of the figure see AppendixA.For Type II State from Section2.1, we need to calculate the system entropy through the joint distributionP⁢(𝒳,𝒴)𝑃𝒳𝒴P(\mathcal{X},\mathcal{Y})italic_P ( caligraphic_X , caligraphic_Y )in equation (2.8), which can be normalized by

P⁢(xi,yi)=P⁢(xi,yi)∑(xi,yi)∈𝒟P⁢(xi,yi),𝑃subscript𝑥𝑖subscript𝑦𝑖𝑃subscript𝑥𝑖subscript𝑦𝑖subscriptsubscript𝑥𝑖subscript𝑦𝑖𝒟𝑃subscript𝑥𝑖subscript𝑦𝑖P(x_{i},y_{i})=\frac{P(x_{i},y_{i})}{\sum_{{(x_{i},y_{i})}\in\mathcal{D}}P(x_{% i},y_{i})},italic_P ( italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) = divide start_ARG italic_P ( italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) end_ARG start_ARG ∑ start_POSTSUBSCRIPT ( italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) ∈ caligraphic_D end_POSTSUBSCRIPT italic_P ( italic_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_y start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) end_ARG ,(A.1)However, the probability distribution of the input and output data domain for most ML systems is unknown. We can only estimate the distribution via some models, i.e., using generative models to estimateP⁢(𝒳,𝒴)𝑃𝒳𝒴P(\mathcal{X},\mathcal{Y})italic_P ( caligraphic_X , caligraphic_Y ), or discriminative models to estimate the conditional probabilityP⁢(𝒴|𝒳)𝑃conditional𝒴𝒳P(\mathcal{Y}|\mathcal{X})italic_P ( caligraphic_Y | caligraphic_X ), and we need to know or assume the probability distribution ofP⁢(𝒳)𝑃𝒳P(\mathcal{X})italic_P ( caligraphic_X ), so we can estimateP⁢(𝒳,𝒴)𝑃𝒳𝒴P(\mathcal{X},\mathcal{Y})italic_P ( caligraphic_X , caligraphic_Y )byP⁢(𝒴|𝒳)⁢P⁢(𝒳)𝑃conditional𝒴𝒳𝑃𝒳P(\mathcal{Y}|\mathcal{X})P(\mathcal{X})italic_P ( caligraphic_Y | caligraphic_X ) italic_P ( caligraphic_X ).

Figure8shows the entire scenario of combining Type I and Type II states, where the upper figure (a) illustrates the discrete case. Suppose the initial system (training) dataset is𝒟1:𝒳1×𝒴1:subscript𝒟1subscript𝒳1subscript𝒴1\mathcal{D}_{1}:\mathcal{X}_{1}\times\mathcal{Y}_{1}caligraphic_D start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT : caligraphic_X start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT × caligraphic_Y start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT, and the model parameter space is{𝝁}𝝁\{\boldsymbol{\mu}\}{ bold_italic_μ }. The fixed dataset𝒟1subscript𝒟1\mathcal{D}_{1}caligraphic_D start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPTand the parameter space{𝝁}𝝁\{\boldsymbol{\mu}\}{ bold_italic_μ }with a certain distribution gives a Type I state of the system. The ML training process is equivalent to the phase transition process into a new state with energyE^⁢(𝒟1)^𝐸subscript𝒟1\hat{E}(\mathcal{D}_{1})over^ start_ARG italic_E end_ARG ( caligraphic_D start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT )and with an optimized (fixed) parameter set𝝁1^^subscript𝝁1\hat{\boldsymbol{\mu}_{1}}over^ start_ARG bold_italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_ARG, with energyΔ⁢(Ep)1Δsubscriptsubscript𝐸𝑝1\Delta(E_{p})_{1}roman_Δ ( italic_E start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ) start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPTreleased from phase transition. We can use Equation (2.4) to calculate the corresponding temperatureT1subscript𝑇1T_{1}italic_T start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPTthrough the Type I phase transition. Next, through the modelf^^𝑓\hat{f}over^ start_ARG italic_f end_ARGcorresponding to𝝁1^^subscript𝝁1\hat{\boldsymbol{\mu}_{1}}over^ start_ARG bold_italic_μ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_ARG, we can compute the conditional probabilityP⁢(𝒴1|𝒳1)𝑃conditionalsubscript𝒴1subscript𝒳1P(\mathcal{Y}_{1}|\mathcal{X}_{1})italic_P ( caligraphic_Y start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT | caligraphic_X start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT )on dataset𝒟1subscript𝒟1\mathcal{D}_{1}caligraphic_D start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT, and obtain an approximation ofP⁢(𝒳1,𝒴1)=P⁢(𝒴1|𝒳1)⁢P⁢(𝒳1)𝑃subscript𝒳1subscript𝒴1𝑃conditionalsubscript𝒴1subscript𝒳1𝑃subscript𝒳1P(\mathcal{X}_{1},\mathcal{Y}_{1})=P(\mathcal{Y}_{1}|\mathcal{X}_{1})P(% \mathcal{X}_{1})italic_P ( caligraphic_X start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , caligraphic_Y start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ) = italic_P ( caligraphic_Y start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT | caligraphic_X start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ) italic_P ( caligraphic_X start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT )by assuming the distribution ofP⁢(𝒳1)𝑃subscript𝒳1P(\mathcal{X}_{1})italic_P ( caligraphic_X start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ). Thus, we can then use Equation (2.8) to calculate the corresponding entropyS⁢(𝒟1)𝑆subscript𝒟1S(\mathcal{D}_{1})italic_S ( caligraphic_D start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ), which is the data entropy of the Type II state.

As the system data varies from𝒟1subscript𝒟1\mathcal{D}_{1}caligraphic_D start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPTto𝒟2=𝒳2×𝒴2subscript𝒟2subscript𝒳2subscript𝒴2\mathcal{D}_{2}=\mathcal{X}_{2}\times\mathcal{Y}_{2}caligraphic_D start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT = caligraphic_X start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT × caligraphic_Y start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT, the Type I state changes to temperatureT2subscript𝑇2T_{2}italic_T start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT, and undergoes a phase transition into a state with energyE^⁢(𝒟2)^𝐸subscript𝒟2\hat{E}(\mathcal{D}_{2})over^ start_ARG italic_E end_ARG ( caligraphic_D start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT )with an updated optimized parameter set𝝁2^^subscript𝝁2\hat{\boldsymbol{\mu}_{2}}over^ start_ARG bold_italic_μ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_ARG, and the data entropy of the system changes toS⁢(𝒟2)𝑆subscript𝒟2S(\mathcal{D}_{2})italic_S ( caligraphic_D start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ). Using Equation (2.9), we can calculate the corresponding temperature of Type II state. As shown in Figure8, this calculation can be continued as the training dataset sequence{𝒟k}subscript𝒟𝑘\{\mathcal{D}_{k}\}{ caligraphic_D start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT }evolves.

The lower panel of Figure8shows the continuous case of system evolution, where the system data(𝒳⁢(λ),𝒴⁢(λ))𝒳𝜆𝒴𝜆(\mathcal{X}(\lambda),\mathcal{Y}(\lambda))( caligraphic_X ( italic_λ ) , caligraphic_Y ( italic_λ ) )changes continuously. We can still start from the model parameter initialization to form a Type I state𝒟k=𝒳⁢(λk)×𝒴⁢(λk)subscript𝒟𝑘𝒳subscript𝜆𝑘𝒴subscript𝜆𝑘\mathcal{D}_{k}=\mathcal{X}(\lambda_{k})\times\mathcal{Y}(\lambda_{k})caligraphic_D start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT = caligraphic_X ( italic_λ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT ) × caligraphic_Y ( italic_λ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT )at a certainλksubscript𝜆𝑘\lambda_{k}italic_λ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPTpoint, and this state undergoes a phase transition through ML training, with its temperature and energy beingTksubscript𝑇𝑘T_{k}italic_T start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPTandE^⁢(𝒟k)^𝐸subscript𝒟𝑘\hat{E}(\mathcal{D}_{k})over^ start_ARG italic_E end_ARG ( caligraphic_D start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT )respectively. We can then calculate the data entropy of the system after the phase transition from the perspective of Type II state. The temperature of Type II state can be calculated as

T⁢(𝒟k)=d⁢E^⁢(𝒟k)d⁢S⁢(𝒟k).𝑇subscript𝒟𝑘𝑑^𝐸subscript𝒟𝑘𝑑𝑆subscript𝒟𝑘T(\mathcal{D}_{k})=\frac{d\hat{E}(\mathcal{D}_{k})}{dS(\mathcal{D}_{k})}.italic_T ( caligraphic_D start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT ) = divide start_ARG italic_d over^ start_ARG italic_E end_ARG ( caligraphic_D start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT ) end_ARG start_ARG italic_d italic_S ( caligraphic_D start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT ) end_ARG .(A.2) In short, Figure8combines Type I and Type II states, describing the entire process of training and retraining a ML system from a thermodynamic perspective.

Appendix BAppendix II: Entropy Change and Dimension Collapse

Let us begin with a simple example. We consider a two-dimensional (2D) uniform distribution in a rectangle region that

f⁢(x,y)=1a⁢b𝑓𝑥𝑦1𝑎𝑏f(x,y)=\frac{1}{ab}italic_f ( italic_x , italic_y ) = divide start_ARG 1 end_ARG start_ARG italic_a italic_b end_ARG(B.1)forx0≤x≤x0+asubscript𝑥0𝑥subscript𝑥0𝑎x_{0}\leq x\leq x_{0}+aitalic_x start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ≤ italic_x ≤ italic_x start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT + italic_aandy0≤y≤y0+bsubscript𝑦0𝑦subscript𝑦0𝑏y_{0}\leq y\leq y_{0}+bitalic_y start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ≤ italic_y ≤ italic_y start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT + italic_b, otherwisef⁢(x,y)=0𝑓𝑥𝑦0f(x,y)=0italic_f ( italic_x , italic_y ) = 0. The differential entropy of the uniform distribution is given by

S2⁢(x,y)=log⁡(a⁢b).subscript𝑆2𝑥𝑦𝑎𝑏S_{2}(x,y)=\log(ab).italic_S start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ( italic_x , italic_y ) = roman_log ( italic_a italic_b ) .(B.2)Next, we squeeze they𝑦yitalic_y-direction of the 2D region such thatb𝑏bitalic_bapproach a small numberδ𝛿\deltaitalic_δ. Then the differential entropy Equation (B.2) becomes

Sδ⁢(x,y)=log⁡a+log⁡δ.subscript𝑆𝛿𝑥𝑦𝑎𝛿S_{\delta}(x,y)=\log a+\log\delta.italic_S start_POSTSUBSCRIPT italic_δ end_POSTSUBSCRIPT ( italic_x , italic_y ) = roman_log italic_a + roman_log italic_δ .(B.3)Asδ→0→𝛿0\delta\rightarrow 0italic_δ → 0, the 2D probability distribution actually becomes a 1D (line) distribution. From the 1D perspective, the entropy of the new distribution is

S1⁢(x)=log⁡a.subscript𝑆1𝑥𝑎S_{1}(x)=\log a.italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ( italic_x ) = roman_log italic_a .(B.4)Clearly, the termlog⁡δ𝛿\log\deltaroman_log italic_δin Equation (B.3) represents the entropy change resulting from dimensional collapse from two dimensions to one dimension, whileS2⁢(x,y)−S1⁢(x)=log⁡bsubscript𝑆2𝑥𝑦subscript𝑆1𝑥𝑏S_{2}(x,y)-S_{1}(x)=\log bitalic_S start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ( italic_x , italic_y ) - italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ( italic_x ) = roman_log italic_bis the actually entropy from 1D.

Using the same logic, let us re-examine the connection between differential entropy and discrete entropy in Section2.3. For a probability distribution functionf⁢(𝝁)𝑓𝝁f({\boldsymbol{\mu}})italic_f ( bold_italic_μ ), we assume the parameter space of𝝁𝝁{\boldsymbol{\mu}}bold_italic_μis divided into a series of small grids{Δi}subscriptΔ𝑖\{\Delta_{i}\}{ roman_Δ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT }, and the probability distribution collapses into a Dirac delta functionδD⁢(𝝁−𝝁i)subscript𝛿𝐷𝝁subscript𝝁𝑖\delta_{D}({\boldsymbol{\mu}-\boldsymbol{\mu}_{i}})italic_δ start_POSTSUBSCRIPT italic_D end_POSTSUBSCRIPT ( bold_italic_μ - bold_italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT )within each gridΔisubscriptΔ𝑖\Delta_{i}roman_Δ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPTandf⁢(𝝁i)=pi𝑓subscript𝝁𝑖subscript𝑝𝑖f({\boldsymbol{\mu}_{i}})=p_{i}italic_f ( bold_italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) = italic_p start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT, we see the entropy change as

Sdiffsubscript𝑆diff\displaystyle S_{\rm diff}italic_S start_POSTSUBSCRIPT roman_diff end_POSTSUBSCRIPT=\displaystyle==−∫f⁢(𝝁)⁢log⁡[f⁢(𝝁)]⁢𝑑𝝁𝑓𝝁𝑓𝝁differential-d𝝁\displaystyle-\int f({\boldsymbol{\mu}})\log[f({\boldsymbol{\mu}})]d{% \boldsymbol{\mu}}- ∫ italic_f ( bold_italic_μ ) roman_log [ italic_f ( bold_italic_μ ) ] italic_d bold_italic_μ(B.5)→→\displaystyle\rightarrow→−∑i∫Δif⁢(𝝁)⁢δD⁢(𝝁−𝝁i)⁢log⁡[f⁢(𝝁)⁢δD⁢(𝝁−𝝁i)]⁢𝑑𝝁subscript𝑖subscriptsubscriptΔ𝑖𝑓𝝁subscript𝛿𝐷𝝁subscript𝝁𝑖𝑓𝝁subscript𝛿𝐷𝝁subscript𝝁𝑖differential-d𝝁\displaystyle-\sum_{i}\int_{\Delta_{i}}f({\boldsymbol{\mu}})\delta_{D}({% \boldsymbol{\mu}-\boldsymbol{\mu}_{i}})\log[f({\boldsymbol{\mu}})\delta_{D}({% \boldsymbol{\mu}-\boldsymbol{\mu}_{i}})]d{\boldsymbol{\mu}}- ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ∫ start_POSTSUBSCRIPT roman_Δ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUBSCRIPT italic_f ( bold_italic_μ ) italic_δ start_POSTSUBSCRIPT italic_D end_POSTSUBSCRIPT ( bold_italic_μ - bold_italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) roman_log [ italic_f ( bold_italic_μ ) italic_δ start_POSTSUBSCRIPT italic_D end_POSTSUBSCRIPT ( bold_italic_μ - bold_italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) ] italic_d bold_italic_μ=\displaystyle==−∑if⁢(𝝁i)⁢log⁡[f⁢(𝝁i)⁢δ⁢(0)]=−∑ipi⁢log⁡[pi⁢δD𝒩⁢(0)]subscript𝑖𝑓subscript𝝁𝑖𝑓subscript𝝁𝑖𝛿0subscript𝑖subscript𝑝𝑖subscript𝑝𝑖superscriptsubscript𝛿𝐷𝒩0\displaystyle-\sum_{i}f({\boldsymbol{\mu}_{i}})\log[f({\boldsymbol{\mu}_{i}})% \delta(0)]=-\sum_{i}p_{i}\log[p_{i}\delta_{D}^{\mathcal{N}}(0)]- ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_f ( bold_italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) roman_log [ italic_f ( bold_italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) italic_δ ( 0 ) ] = - ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_p start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT roman_log [ italic_p start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_δ start_POSTSUBSCRIPT italic_D end_POSTSUBSCRIPT start_POSTSUPERSCRIPT caligraphic_N end_POSTSUPERSCRIPT ( 0 ) ]=\displaystyle==Sdiscrete+𝒩⁢log⁡δ,subscript𝑆discrete𝒩𝛿\displaystyle S_{\rm discrete}+\mathcal{N}\log\delta,italic_S start_POSTSUBSCRIPT roman_discrete end_POSTSUBSCRIPT + caligraphic_N roman_log italic_δ ,where we denoteδ⁢(𝝁i−𝝁i)=[δD⁢(0)]𝒩=(1/δ)𝒩𝛿subscript𝝁𝑖subscript𝝁𝑖superscriptdelimited-[]subscript𝛿𝐷0𝒩superscript1𝛿𝒩\delta({\boldsymbol{\mu}_{i}}-{\boldsymbol{\mu}_{i}})=[\delta_{D}(0)]^{% \mathcal{N}}=(1/\delta)^{\mathcal{N}}italic_δ ( bold_italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT - bold_italic_μ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) = [ italic_δ start_POSTSUBSCRIPT italic_D end_POSTSUBSCRIPT ( 0 ) ] start_POSTSUPERSCRIPT caligraphic_N end_POSTSUPERSCRIPT = ( 1 / italic_δ ) start_POSTSUPERSCRIPT caligraphic_N end_POSTSUPERSCRIPTwith𝒟𝒟\mathcal{D}caligraphic_Dbeing the dimension of𝝁𝝁\boldsymbol{\mu}bold_italic_μandδ→0→𝛿0\delta\rightarrow 0italic_δ → 0as shown in Equation (B.3). The change in entropy, in addition to the change from differential entropySdiffsubscript𝑆diffS_{\rm diff}italic_S start_POSTSUBSCRIPT roman_diff end_POSTSUBSCRIPTto discrete entropySdicretesubscript𝑆dicreteS_{\rm dicrete}italic_S start_POSTSUBSCRIPT roman_dicrete end_POSTSUBSCRIPT, also includes the entropy change due to dimensionality reduction𝒩⁢log⁡δ𝒩𝛿\mathcal{N}\log\deltacaligraphic_N roman_log italic_δ.

In general, for entropy change from dimension𝒟1subscript𝒟1\mathcal{D}_{1}caligraphic_D start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPTto𝒟2subscript𝒟2\mathcal{D}_{2}caligraphic_D start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT(where𝒟1>𝒟2subscript𝒟1subscript𝒟2\mathcal{D}_{1}>\mathcal{D}_{2}caligraphic_D start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT > caligraphic_D start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT), the overall entropy change is written as

Δ⁢S=S𝒟2−S𝒟1+(𝒟1−𝒟2)⁢log⁡δ,Δ𝑆subscript𝑆subscript𝒟2subscript𝑆subscript𝒟1subscript𝒟1subscript𝒟2𝛿\Delta S=S_{\mathcal{D}_{2}}-S_{\mathcal{D}_{1}}+(\mathcal{D}_{1}-\mathcal{D}_% {2})\log\delta,roman_Δ italic_S = italic_S start_POSTSUBSCRIPT caligraphic_D start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_POSTSUBSCRIPT - italic_S start_POSTSUBSCRIPT caligraphic_D start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPT + ( caligraphic_D start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT - caligraphic_D start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ) roman_log italic_δ ,(B.6)whereS𝒟1subscript𝑆subscript𝒟1S_{\mathcal{D}_{1}}italic_S start_POSTSUBSCRIPT caligraphic_D start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPTandS𝒟2subscript𝑆subscript𝒟2S_{\mathcal{D}_{2}}italic_S start_POSTSUBSCRIPT caligraphic_D start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_POSTSUBSCRIPTare entropy in the corresponding dimensional spaces.Δ⁢S1=S𝒟2−S𝒟1Δsubscript𝑆1subscript𝑆subscript𝒟2subscript𝑆subscript𝒟1\Delta S_{1}=S_{\mathcal{D}_{2}}-S_{\mathcal{D}_{1}}roman_Δ italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT = italic_S start_POSTSUBSCRIPT caligraphic_D start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_POSTSUBSCRIPT - italic_S start_POSTSUBSCRIPT caligraphic_D start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPTshows the “real“ finite entropy change, and(𝒟1−𝒟2)⁢log⁡δsubscript𝒟1subscript𝒟2𝛿(\mathcal{D}_{1}-\mathcal{D}_{2})\log\delta( caligraphic_D start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT - caligraphic_D start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ) roman_log italic_δis caused by dimension collapse. For a probability distribution in the space𝝁𝝁\boldsymbol{\mu}bold_italic_μcollapses to a single pointμ0subscript𝜇0\mu_{0}italic_μ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT, we have𝒟1=𝒩subscript𝒟1𝒩{\mathcal{D}_{1}}=\mathcal{N}caligraphic_D start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT = caligraphic_N,𝒟2=0subscript𝒟20\mathcal{D}_{2}=0caligraphic_D start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT = 0,S𝒟1=Sdiffsubscript𝑆subscript𝒟1subscript𝑆diffS_{\mathcal{D}_{1}}=S_{\rm diff}italic_S start_POSTSUBSCRIPT caligraphic_D start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT end_POSTSUBSCRIPT = italic_S start_POSTSUBSCRIPT roman_diff end_POSTSUBSCRIPT,S𝒟2=Sdiscretesubscript𝑆subscript𝒟2subscript𝑆discreteS_{\mathcal{D}_{2}}=S_{\rm discrete}italic_S start_POSTSUBSCRIPT caligraphic_D start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_POSTSUBSCRIPT = italic_S start_POSTSUBSCRIPT roman_discrete end_POSTSUBSCRIPT, and in the zero-dimension space with a single point, there is zero entropy thatSdiscrete=0subscript𝑆discrete0S_{\rm discrete}=0italic_S start_POSTSUBSCRIPT roman_discrete end_POSTSUBSCRIPT = 0. So from the general Equation (B.6) we can use

Δ⁢S1=0−Sdiff=−SdiffΔsubscript𝑆10subscript𝑆diffsubscript𝑆diff\Delta S_{1}=0-S_{\rm diff}=-S_{\rm diff}roman_Δ italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT = 0 - italic_S start_POSTSUBSCRIPT roman_diff end_POSTSUBSCRIPT = - italic_S start_POSTSUBSCRIPT roman_diff end_POSTSUBSCRIPT(B.7)to represent the change in entropy, while𝒩⁢log⁡δ𝒩𝛿\mathcal{N}\log\deltacaligraphic_N roman_log italic_δcorresponds to the entropy change due to dimensionality reduction.

For a ML system with Type I state phase transition, given that the energy changeΔ⁢EΔ𝐸\Delta Eroman_Δ italic_Eis a finite quantity, we also use a finite quantitySdiffsubscript𝑆diffS_{\rm diff}italic_S start_POSTSUBSCRIPT roman_diff end_POSTSUBSCRIPTto express the change in entropy. Thus, the temperature of Type I state phase transitionT=(Ef−E0)/(−Sdiff)=(E0−Ef)/Sdiff𝑇subscript𝐸𝑓subscript𝐸0subscript𝑆diffsubscript𝐸0subscript𝐸𝑓subscript𝑆diffT=(E_{f}-E_{0})/(-S_{\rm diff})=(E_{0}-E_{f})/S_{\rm diff}italic_T = ( italic_E start_POSTSUBSCRIPT italic_f end_POSTSUBSCRIPT - italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ) / ( - italic_S start_POSTSUBSCRIPT roman_diff end_POSTSUBSCRIPT ) = ( italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT - italic_E start_POSTSUBSCRIPT italic_f end_POSTSUBSCRIPT ) / italic_S start_POSTSUBSCRIPT roman_diff end_POSTSUBSCRIPT. This is exactly Equation (2.7) we showed in Section2.1, also Equation (2.29) in Section2.3.

Above we discussed the change in system entropy from the perspective of dimensional collapse. Next, we will look into the issue of infinite entropy from the perspective of a physical thermodynamic system. Suppose a system is made up by two subsystems. The energy change is

Δ⁢E=Δ⁢Q1+Δ⁢Q2=T1⁢Δ⁢S1+T2⁢Δ⁢S2.Δ𝐸Δsubscript𝑄1Δsubscript𝑄2subscript𝑇1Δsubscript𝑆1subscript𝑇2Δsubscript𝑆2\Delta E=\Delta Q_{1}+\Delta Q_{2}=T_{1}\Delta S_{1}+T_{2}\Delta S_{2}.roman_Δ italic_E = roman_Δ italic_Q start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT + roman_Δ italic_Q start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT = italic_T start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT roman_Δ italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT + italic_T start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT roman_Δ italic_S start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT .(B.8)The total entropy change in the system isSdiff−log⁡δsubscript𝑆diff𝛿S_{\rm diff}-\log\deltaitalic_S start_POSTSUBSCRIPT roman_diff end_POSTSUBSCRIPT - roman_log italic_δ(see Equqation [2.25]). We allocateΔ⁢S1=−log⁡δΔsubscript𝑆1𝛿\Delta S_{1}=-\log\deltaroman_Δ italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT = - roman_log italic_δ,Δ⁢S2=SdiffΔsubscript𝑆2subscript𝑆diff\Delta S_{2}=S_{\rm diff}roman_Δ italic_S start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT = italic_S start_POSTSUBSCRIPT roman_diff end_POSTSUBSCRIPT, andΔ⁢Q1=δΔsubscript𝑄1𝛿\Delta Q_{1}=\deltaroman_Δ italic_Q start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT = italic_δ,Δ⁢Q2=Δ⁢E−δΔsubscript𝑄2Δ𝐸𝛿\Delta Q_{2}=\Delta E-\deltaroman_Δ italic_Q start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT = roman_Δ italic_E - italic_δ. So the temperature of the subsystem I is

T1subscript𝑇1\displaystyle T_{1}italic_T start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT=\displaystyle==δ−log⁡δ=δ∑k=1∞(1−δ)kk≈δ∑k=1∞1k𝛿𝛿𝛿superscriptsubscript𝑘1superscript1𝛿𝑘𝑘𝛿superscriptsubscript𝑘11𝑘\displaystyle\frac{\delta}{-\log\delta}=\frac{\delta}{\sum_{k=1}^{\infty}\frac% {(1-\delta)^{k}}{k}}\approx\frac{\delta}{\sum_{k=1}^{\infty}\frac{1}{k}}divide start_ARG italic_δ end_ARG start_ARG - roman_log italic_δ end_ARG = divide start_ARG italic_δ end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_k = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ∞ end_POSTSUPERSCRIPT divide start_ARG ( 1 - italic_δ ) start_POSTSUPERSCRIPT italic_k end_POSTSUPERSCRIPT end_ARG start_ARG italic_k end_ARG end_ARG ≈ divide start_ARG italic_δ end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_k = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ∞ end_POSTSUPERSCRIPT divide start_ARG 1 end_ARG start_ARG italic_k end_ARG end_ARG(B.9)∼similar-to\displaystyle\sim∼δlimk→∞⁢ln⁡k+0.57721→0,→𝛿subscriptlim→𝑘𝑘0.577210\displaystyle\frac{\delta}{{\rm lim}_{{k\rightarrow\infty}}\ln k+0.57721}% \rightarrow 0,divide start_ARG italic_δ end_ARG start_ARG roman_lim start_POSTSUBSCRIPT italic_k → ∞ end_POSTSUBSCRIPT roman_ln italic_k + 0.57721 end_ARG → 0 ,On the other hand, the temperature of the subsystem II is

T2=(Δ⁢E−δ)/Sdiff≈Δ⁢E/Sdiff,subscript𝑇2Δ𝐸𝛿subscript𝑆diffΔ𝐸subscript𝑆diffT_{2}=(\Delta E-\delta)/S_{\rm diff}\approx\Delta E/S_{\rm diff},italic_T start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT = ( roman_Δ italic_E - italic_δ ) / italic_S start_POSTSUBSCRIPT roman_diff end_POSTSUBSCRIPT ≈ roman_Δ italic_E / italic_S start_POSTSUBSCRIPT roman_diff end_POSTSUBSCRIPT ,(B.10)which is the temperature we calculate in this paper.

Note that ifSdiff<0subscript𝑆diff0S_{\rm diff}<0italic_S start_POSTSUBSCRIPT roman_diff end_POSTSUBSCRIPT < 0,T2subscript𝑇2T_{2}italic_T start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPTcan be negative. Negative thermodynamic temperature has been widely discussed in statistical and condense physics so this is not an unfamiliar concept[31,32].

Appendix CAppendix III: Non-Boltzmann and Generalized Boltzmann Distributions

We can derive the Boltzmann distribution of a gaseous system from first principles[28]. If the gaseous system is in a constant gravitational potential with accelerationg𝑔gitalic_g, for a gas particle with velocityv𝑣vitalic_vand at a heighthℎhitalic_h, we have

εv,h=12⁢m⁢v2+m⁢g⁢h.subscript𝜀𝑣ℎ12𝑚superscript𝑣2𝑚𝑔ℎ\varepsilon_{v,h}=\frac{1}{2}mv^{2}+mgh.italic_ε start_POSTSUBSCRIPT italic_v , italic_h end_POSTSUBSCRIPT = divide start_ARG 1 end_ARG start_ARG 2 end_ARG italic_m italic_v start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + italic_m italic_g italic_h .(C.1)The configuration is subject to the constraints

∬nv,h⁢𝑑v⁢𝑑h=Ndouble-integralsubscript𝑛𝑣ℎdifferential-d𝑣differential-dℎ𝑁\displaystyle\iint n_{v,h}dvdh=N∬ italic_n start_POSTSUBSCRIPT italic_v , italic_h end_POSTSUBSCRIPT italic_d italic_v italic_d italic_h = italic_N(C.2)∬nv,h⁢εv,h⁢𝑑v⁢𝑑h=E,double-integralsubscript𝑛𝑣ℎsubscript𝜀𝑣ℎdifferential-d𝑣differential-dℎ𝐸\displaystyle\iint n_{v,h}\varepsilon_{v,h}dvdh=E,∬ italic_n start_POSTSUBSCRIPT italic_v , italic_h end_POSTSUBSCRIPT italic_ε start_POSTSUBSCRIPT italic_v , italic_h end_POSTSUBSCRIPT italic_d italic_v italic_d italic_h = italic_E ,(C.3)whereN𝑁Nitalic_NandE𝐸Eitalic_Eare total number of particles and energy respectively. The number of ways of making a given configuration is

Ω⁢({nvi,hj})=N!∏i,jnvi,hj!.Ωsubscript𝑛subscript𝑣𝑖subscriptℎ𝑗𝑁subscriptproduct𝑖𝑗subscript𝑛subscript𝑣𝑖subscriptℎ𝑗\Omega(\{n_{v_{i},h_{j}}\})=\frac{N!}{\prod_{i,j}n_{v_{i},h_{j}}!}.roman_Ω ( { italic_n start_POSTSUBSCRIPT italic_v start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT end_POSTSUBSCRIPT } ) = divide start_ARG italic_N ! end_ARG start_ARG ∏ start_POSTSUBSCRIPT italic_i , italic_j end_POSTSUBSCRIPT italic_n start_POSTSUBSCRIPT italic_v start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , italic_h start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT end_POSTSUBSCRIPT ! end_ARG .(C.4)We want to maximizeΩΩ\Omegaroman_Ω, therefore we have

∂∂nv,h⁢(ln⁡Ω−β⁢E−α⁢N)=0subscript𝑛𝑣ℎΩ𝛽𝐸𝛼𝑁0\frac{\partial}{\partial n_{v,h}}(\ln\Omega-\beta E-\alpha N)=0divide start_ARG ∂ end_ARG start_ARG ∂ italic_n start_POSTSUBSCRIPT italic_v , italic_h end_POSTSUBSCRIPT end_ARG ( roman_ln roman_Ω - italic_β italic_E - italic_α italic_N ) = 0(C.5)nv,hN=e−α−β⁢εv,h,subscript𝑛𝑣ℎ𝑁superscripte𝛼𝛽subscript𝜀𝑣ℎ\frac{n_{v,h}}{N}=\textrm{e}^{-\alpha-\beta\varepsilon_{v,h}},divide start_ARG italic_n start_POSTSUBSCRIPT italic_v , italic_h end_POSTSUBSCRIPT end_ARG start_ARG italic_N end_ARG = e start_POSTSUPERSCRIPT - italic_α - italic_β italic_ε start_POSTSUBSCRIPT italic_v , italic_h end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ,(C.6)Using equation (C.1), equation (C.6) can be written as

nv,h∝exp⁡(−α−m⁢v2+2⁢m⁢g⁢h2⁢kB⁢T),proportional-tosubscript𝑛𝑣ℎ𝛼𝑚superscript𝑣22𝑚𝑔ℎ2subscript𝑘𝐵𝑇n_{v,h}\propto\exp\left(-\alpha-\frac{mv^{2}+2mgh}{2k_{B}T}\right),italic_n start_POSTSUBSCRIPT italic_v , italic_h end_POSTSUBSCRIPT ∝ roman_exp ( - italic_α - divide start_ARG italic_m italic_v start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + 2 italic_m italic_g italic_h end_ARG start_ARG 2 italic_k start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT italic_T end_ARG ) ,(C.7)where we can proveβ=1/kB⁢T𝛽1subscript𝑘𝐵𝑇\beta=1/k_{B}Titalic_β = 1 / italic_k start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT italic_Tusing thermodynamic relations.

The Boltzmann distribution has already been extended to the cases of special and general relativity[33,34,35,36]. However, how to establish the theories of thermodynamics and statistical mechanics within the framework of relativity remains an open question[37,38,39]. In this appendix, we continue to discuss within the framework of classical mechanics. We have given the above calculations in the rest frame stationary relative to the ground. Next, let us consider an observer moving relative to the ground with velocity𝑽𝑽{\boldsymbol{V}}bold_italic_V. For this observer, a gas particle with velocity𝒗~~𝒗\tilde{\boldsymbol{v}}over~ start_ARG bold_italic_v end_ARGwould appear to have a velocity𝒗~−𝑽~𝒗𝑽\tilde{\boldsymbol{v}}-{\boldsymbol{V}}over~ start_ARG bold_italic_v end_ARG - bold_italic_Vin the observer’s reference frame. We can establish the velocity transformation between the two frames of reference:

(rest⁢frame)⁢𝒗~⟷𝒗=𝒗~−𝑽⁢(observer′⁢s⁢frame),⟷restframe~𝒗𝒗~𝒗𝑽superscriptobserver′sframe({\rm rest\,frame})\;\tilde{\boldsymbol{v}}\;{\longleftrightarrow}\;{% \boldsymbol{v}}=\tilde{\boldsymbol{v}}-{\boldsymbol{V}}\;({\rm observer^{% \prime}s\,frame}),( roman_rest roman_frame ) over~ start_ARG bold_italic_v end_ARG ⟷ bold_italic_v = over~ start_ARG bold_italic_v end_ARG - bold_italic_V ( roman_observer start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT roman_s roman_frame ) ,(C.8)Then, from the observer’s frame of reference,

nv,hsubscript𝑛𝑣ℎ\displaystyle n_{v,h}italic_n start_POSTSUBSCRIPT italic_v , italic_h end_POSTSUBSCRIPT∝proportional-to\displaystyle\propto∝nv~,h∝exp⁡(−α−m⁢v~2+2⁢m⁢g⁢h2⁢kB⁢T)proportional-tosubscript𝑛~𝑣ℎ𝛼𝑚superscript~𝑣22𝑚𝑔ℎ2subscript𝑘𝐵𝑇\displaystyle n_{\tilde{v},h}\propto\exp\left(-\alpha-\frac{m\tilde{v}^{2}+2% mgh}{2k_{B}T}\right)italic_n start_POSTSUBSCRIPT over~ start_ARG italic_v end_ARG , italic_h end_POSTSUBSCRIPT ∝ roman_exp ( - italic_α - divide start_ARG italic_m over~ start_ARG italic_v end_ARG start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + 2 italic_m italic_g italic_h end_ARG start_ARG 2 italic_k start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT italic_T end_ARG )(C.9)∝proportional-to\displaystyle\propto∝exp⁡[−m⁢(𝒗+𝑽)2+2⁢m⁢g⁢h2⁢kB⁢T]𝑚superscript𝒗𝑽22𝑚𝑔ℎ2subscript𝑘𝐵𝑇\displaystyle\exp\left[-\frac{m({\boldsymbol{v}}+{\boldsymbol{V}})^{2}+2mgh}{2% k_{B}T}\right]roman_exp [ - divide start_ARG italic_m ( bold_italic_v + bold_italic_V ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + 2 italic_m italic_g italic_h end_ARG start_ARG 2 italic_k start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT italic_T end_ARG ]∝proportional-to\displaystyle\propto∝exp⁡(m⁢V22⁢kB⁢T)⁢exp⁡(−m⁢v2+2⁢m⁢g⁢h2⁢kB⁢T)⁢⟨exp⁡(−mkB⁢T⁢𝒗⋅𝑽)⟩𝑚superscript𝑉22subscript𝑘𝐵𝑇𝑚superscript𝑣22𝑚𝑔ℎ2subscript𝑘𝐵𝑇delimited-⟨⟩⋅𝑚subscript𝑘𝐵𝑇𝒗𝑽\displaystyle\exp\left(\frac{mV^{2}}{2k_{B}T}\right)\exp\left(-\frac{mv^{2}+2% mgh}{2k_{B}T}\right)\left\langle\exp(-\frac{m}{k_{B}T}{\boldsymbol{v}}\cdot{% \boldsymbol{V}})\right\rangleroman_exp ( divide start_ARG italic_m italic_V start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_k start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT italic_T end_ARG ) roman_exp ( - divide start_ARG italic_m italic_v start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + 2 italic_m italic_g italic_h end_ARG start_ARG 2 italic_k start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT italic_T end_ARG ) ⟨ roman_exp ( - divide start_ARG italic_m end_ARG start_ARG italic_k start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT italic_T end_ARG bold_italic_v ⋅ bold_italic_V ) ⟩∝proportional-to\displaystyle\propto∝exp⁡(m⁢V22⁢kB⁢T)⁢exp⁡(−εv,h2⁢kB⁢T).𝑚superscript𝑉22subscript𝑘𝐵𝑇subscript𝜀𝑣ℎ2subscript𝑘𝐵𝑇\displaystyle\exp\left(\frac{mV^{2}}{2k_{B}T}\right)\exp\left(-\frac{% \varepsilon_{v,h}}{2k_{B}T}\right).roman_exp ( divide start_ARG italic_m italic_V start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 2 italic_k start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT italic_T end_ARG ) roman_exp ( - divide start_ARG italic_ε start_POSTSUBSCRIPT italic_v , italic_h end_POSTSUBSCRIPT end_ARG start_ARG 2 italic_k start_POSTSUBSCRIPT italic_B end_POSTSUBSCRIPT italic_T end_ARG ) .We can also start from the first principles. From the observer’s frame, the energy in the gaseous system gives

∬nv,h⁢εv,h⁢𝑑v⁢𝑑hdouble-integralsubscript𝑛𝑣ℎsubscript𝜀𝑣ℎdifferential-d𝑣differential-dℎ\displaystyle\iint n_{v,h}\varepsilon_{v,h}dvdh∬ italic_n start_POSTSUBSCRIPT italic_v , italic_h end_POSTSUBSCRIPT italic_ε start_POSTSUBSCRIPT italic_v , italic_h end_POSTSUBSCRIPT italic_d italic_v italic_d italic_h=\displaystyle==∬[12⁢m⁢(𝒗~−𝑽)2+m⁢g⁢h]⁢nv,h⁢𝑑v⁢𝑑hdouble-integraldelimited-[]12𝑚superscript~𝒗𝑽2𝑚𝑔ℎsubscript𝑛𝑣ℎdifferential-d𝑣differential-dℎ\displaystyle\iint\left[\frac{1}{2}m(\tilde{\boldsymbol{v}}-{\boldsymbol{V}})^% {2}+mgh\right]n_{v,h}dvdh∬ [ divide start_ARG 1 end_ARG start_ARG 2 end_ARG italic_m ( over~ start_ARG bold_italic_v end_ARG - bold_italic_V ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + italic_m italic_g italic_h ] italic_n start_POSTSUBSCRIPT italic_v , italic_h end_POSTSUBSCRIPT italic_d italic_v italic_d italic_h(C.10)=\displaystyle==∬(12⁢m⁢v~2+m⁢g⁢h)⁢nv~,h⁢𝑑v~⁢𝑑h−∬m⁢𝐯⋅𝐕⁢nv~,h⁢𝑑v~⁢𝑑h⁢𝑑ωdouble-integral12𝑚superscript~𝑣2𝑚𝑔ℎsubscript𝑛~𝑣ℎdifferential-d~𝑣differential-dℎdouble-integral⋅𝑚𝐯𝐕subscript𝑛~𝑣ℎdifferential-d~𝑣differential-dℎdifferential-d𝜔\displaystyle\iint~\left(\frac{1}{2}m\tilde{v}^{2}+mgh\right)n_{\tilde{v},h}d% \tilde{v}dh-\iint m{\bf v}\cdot{\bf V}n_{\tilde{v},h}d\tilde{v}dhd\omega∬ ( divide start_ARG 1 end_ARG start_ARG 2 end_ARG italic_m over~ start_ARG italic_v end_ARG start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + italic_m italic_g italic_h ) italic_n start_POSTSUBSCRIPT over~ start_ARG italic_v end_ARG , italic_h end_POSTSUBSCRIPT italic_d over~ start_ARG italic_v end_ARG italic_d italic_h - ∬ italic_m bold_v ⋅ bold_V italic_n start_POSTSUBSCRIPT over~ start_ARG italic_v end_ARG , italic_h end_POSTSUBSCRIPT italic_d over~ start_ARG italic_v end_ARG italic_d italic_h italic_d italic_ω+\displaystyle++12⁢m⁢V2⁢∬nv,h⁢𝑑v⁢𝑑h12𝑚superscript𝑉2double-integralsubscript𝑛𝑣ℎdifferential-d𝑣differential-dℎ\displaystyle\frac{1}{2}mV^{2}\iint n_{v,h}dvdhdivide start_ARG 1 end_ARG start_ARG 2 end_ARG italic_m italic_V start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ∬ italic_n start_POSTSUBSCRIPT italic_v , italic_h end_POSTSUBSCRIPT italic_d italic_v italic_d italic_h=\displaystyle==E+12⁢m⁢V2⁢∬nv,h⁢𝑑v⁢𝑑h,𝐸12𝑚superscript𝑉2double-integralsubscript𝑛𝑣ℎdifferential-d𝑣differential-dℎ\displaystyle E+\frac{1}{2}mV^{2}\iint n_{v,h}dvdh,italic_E + divide start_ARG 1 end_ARG start_ARG 2 end_ARG italic_m italic_V start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ∬ italic_n start_POSTSUBSCRIPT italic_v , italic_h end_POSTSUBSCRIPT italic_d italic_v italic_d italic_h ,Therefore we have two constraints in the observer’s frame that

∬nv,h⁢𝑑v⁢𝑑h=Ndouble-integralsubscript𝑛𝑣ℎdifferential-d𝑣differential-dℎ𝑁\displaystyle\iint n_{v,h}dvdh=N∬ italic_n start_POSTSUBSCRIPT italic_v , italic_h end_POSTSUBSCRIPT italic_d italic_v italic_d italic_h = italic_N(C.11)∬nv,h⁢εv,h⁢𝑑v⁢𝑑h−12⁢m⁢V2⁢∬nv,h⁢𝑑v⁢𝑑h=E,double-integralsubscript𝑛𝑣ℎsubscript𝜀𝑣ℎdifferential-d𝑣differential-dℎ12𝑚superscript𝑉2double-integralsubscript𝑛𝑣ℎdifferential-d𝑣differential-dℎ𝐸\displaystyle\iint n_{v,h}\varepsilon_{v,h}dvdh-\frac{1}{2}mV^{2}\iint n_{v,h}% dvdh=E,∬ italic_n start_POSTSUBSCRIPT italic_v , italic_h end_POSTSUBSCRIPT italic_ε start_POSTSUBSCRIPT italic_v , italic_h end_POSTSUBSCRIPT italic_d italic_v italic_d italic_h - divide start_ARG 1 end_ARG start_ARG 2 end_ARG italic_m italic_V start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ∬ italic_n start_POSTSUBSCRIPT italic_v , italic_h end_POSTSUBSCRIPT italic_d italic_v italic_d italic_h = italic_E ,(C.12)Thus we can obtain

ln⁡(nv,hN)−α−β⁢[εv,h−12⁢m⁢V2]=0,subscript𝑛𝑣ℎ𝑁𝛼𝛽delimited-[]subscript𝜀𝑣ℎ12𝑚superscript𝑉20\ln\left(\frac{n_{v,h}}{N}\right)-\alpha-\beta\left[\varepsilon_{v,h}-\frac{1}% {2}mV^{2}\right]=0,roman_ln ( divide start_ARG italic_n start_POSTSUBSCRIPT italic_v , italic_h end_POSTSUBSCRIPT end_ARG start_ARG italic_N end_ARG ) - italic_α - italic_β [ italic_ε start_POSTSUBSCRIPT italic_v , italic_h end_POSTSUBSCRIPT - divide start_ARG 1 end_ARG start_ARG 2 end_ARG italic_m italic_V start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ] = 0 ,(C.13)or

nv,h∝exp⁡(12⁢β⁢m⁢V2)⁢e−β⁢εv,h,proportional-tosubscript𝑛𝑣ℎ12𝛽𝑚superscript𝑉2superscripte𝛽subscript𝜀𝑣ℎn_{v,h}\propto\exp\left(\frac{1}{2}\beta mV^{2}\right){\rm e}^{-\beta% \varepsilon_{v,h}},italic_n start_POSTSUBSCRIPT italic_v , italic_h end_POSTSUBSCRIPT ∝ roman_exp ( divide start_ARG 1 end_ARG start_ARG 2 end_ARG italic_β italic_m italic_V start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ) roman_e start_POSTSUPERSCRIPT - italic_β italic_ε start_POSTSUBSCRIPT italic_v , italic_h end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ,(C.14)which is the same as equation (C.9). Clearly,nv,hsubscript𝑛𝑣ℎn_{v,h}italic_n start_POSTSUBSCRIPT italic_v , italic_h end_POSTSUBSCRIPTdiffers from the traditional Boltzmann distribution by a factor off⁢(V)=12⁢m⁢V2𝑓𝑉12𝑚superscript𝑉2f(V)=\frac{1}{2}mV^{2}italic_f ( italic_V ) = divide start_ARG 1 end_ARG start_ARG 2 end_ARG italic_m italic_V start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT. IfV𝑉Vitalic_Vvaries, thennv,hsubscript𝑛𝑣ℎn_{v,h}italic_n start_POSTSUBSCRIPT italic_v , italic_h end_POSTSUBSCRIPTalso redistributes withV𝑉Vitalic_V. this can be referred to as a non-Boltzmann distribution or a generalized Boltzmann distribution.

Some research divides the energy of a gaseous system into internal energy and bulk energy. However, from the perspective of relativity, there is no special reference frame among various reference frames. Therefore, setting a specific center-of-mass reference from is also unreasonable, especially in many cases, we cannot accurately determine the velocity of the center of mass, nor split energy of each gas particles to internal and bulk energy.

In the above discussion, we assumed that the gravitational potential energy is a constant. We can consider more general cases. We split the energy to internal and others for thei𝑖iitalic_i-th state of a system:

εi=εi^+𝒢i,subscript𝜀𝑖^subscript𝜀𝑖subscript𝒢𝑖\varepsilon_{i}=\hat{\varepsilon_{i}}+\mathcal{G}_{i},italic_ε start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT = over^ start_ARG italic_ε start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG + caligraphic_G start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ,(C.15)where𝒢isubscript𝒢𝑖\mathcal{G}_{i}caligraphic_G start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPTis contributed by gravitational potential and other input/output energies. And we assume the internal energy is conserved, so we have

∫ni⁢εi⁢𝑑Ωsubscript𝑛𝑖subscript𝜀𝑖differential-dΩ\displaystyle\int n_{i}\varepsilon_{i}d\Omega∫ italic_n start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_ε start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_d roman_Ω=\displaystyle==∫ni⁢εi^⁢𝑑Ω+∫ni⁢𝒢i⁢𝑑Ωsubscript𝑛𝑖^subscript𝜀𝑖differential-dΩsubscript𝑛𝑖subscript𝒢𝑖differential-dΩ\displaystyle\int n_{i}\hat{\varepsilon_{i}}d\Omega+\int n_{i}\mathcal{G}_{i}d\Omega∫ italic_n start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT over^ start_ARG italic_ε start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG italic_d roman_Ω + ∫ italic_n start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT caligraphic_G start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_d roman_Ω(C.16)=\displaystyle==E+∫ni⁢𝒢i⁢𝑑Ω,𝐸subscript𝑛𝑖subscript𝒢𝑖differential-dΩ\displaystyle E+\int n_{i}\mathcal{G}_{i}d\Omega,italic_E + ∫ italic_n start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT caligraphic_G start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_d roman_Ω ,so the constraints are given by

N=∫ni⁢𝑑Ω𝑁subscript𝑛𝑖differential-dΩ\displaystyle N=\int n_{i}d\Omegaitalic_N = ∫ italic_n start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_d roman_Ω(C.17)E=∫ni⁢εi⁢𝑑Ω−∫ni⁢𝒢i⁢𝑑Ω.𝐸subscript𝑛𝑖subscript𝜀𝑖differential-dΩsubscript𝑛𝑖subscript𝒢𝑖differential-dΩ\displaystyle E=\int n_{i}\varepsilon_{i}d\Omega-\int n_{i}\mathcal{G}_{i}d\Omega.italic_E = ∫ italic_n start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_ε start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_d roman_Ω - ∫ italic_n start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT caligraphic_G start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_d roman_Ω .(C.18)The distribution of the system is

ln⁡(niN)−α−β⁢εi+β⁢𝒢i=0,subscript𝑛𝑖𝑁𝛼𝛽subscript𝜀𝑖𝛽subscript𝒢𝑖0\ln\left(\frac{n_{i}}{N}\right)-\alpha-\beta\varepsilon_{i}+\beta\mathcal{G}_{% i}=0,roman_ln ( divide start_ARG italic_n start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG start_ARG italic_N end_ARG ) - italic_α - italic_β italic_ε start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT + italic_β caligraphic_G start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT = 0 ,(C.19)i.e.,

nisubscript𝑛𝑖\displaystyle n_{i}italic_n start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT∝proportional-to\displaystyle\propto∝exp⁡(−α−β⁢εi+β⁢𝒢i)𝛼𝛽subscript𝜀𝑖𝛽subscript𝒢𝑖\displaystyle\exp\left(-\alpha-\beta\varepsilon_{i}+\beta\mathcal{G}_{i}\right)roman_exp ( - italic_α - italic_β italic_ε start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT + italic_β caligraphic_G start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT )(C.20)∝proportional-to\displaystyle\propto∝fi⁢(𝒢)⁢e−α−β⁢εi.subscript𝑓𝑖𝒢superscripte𝛼𝛽subscript𝜀𝑖\displaystyle f_{i}(\mathcal{G}){\rm e}^{-\alpha-\beta\varepsilon_{i}}.italic_f start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( caligraphic_G ) roman_e start_POSTSUPERSCRIPT - italic_α - italic_β italic_ε start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUPERSCRIPT .Here, the factorfi⁢(𝒢)=eβ⁢Gisubscript𝑓𝑖𝒢superscripte𝛽subscript𝐺𝑖f_{i}(\mathcal{G})={\rm e}^{\beta G_{i}}italic_f start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ( caligraphic_G ) = roman_e start_POSTSUPERSCRIPT italic_β italic_G start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_POSTSUPERSCRIPTcan be called themetric of the partition function. Therefore, Boltzmann distribution is just s special cases for a system without varying gravitational background. And Equation (C.20) can be called the generalized Boltzmann distribution.

Appendix DAppendix IV: Temperature and Thermodynamics Laws with Lorentz Transformation

In the framework of relativity, how does temperature transform between difference reference frames? This remains an open question and has been debated for a long time. Assume a body is measured with a temperatureT𝑇Titalic_Tin the rest frame, while its temperature isT′superscript𝑇′T^{\prime}italic_T start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPTin another frame where the body has a velocityv𝑣vitalic_v. How do we establish the Lorentz-like transformation fromT𝑇Titalic_TtoT′superscript𝑇′T^{\prime}italic_T start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT? There have been three prevailing viewpoints:

• •The observed moving body has the temperatureT′=T/γsuperscript𝑇′𝑇𝛾T^{\prime}=T/\gammaitalic_T start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT = italic_T / italic_γ[40,41], whereγ=(1−v2/c2)−1/2𝛾superscript1superscript𝑣2superscript𝑐212\gamma=(1-v^{2}/c^{2})^{-1/2}italic_γ = ( 1 - italic_v start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT / italic_c start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ) start_POSTSUPERSCRIPT - 1 / 2 end_POSTSUPERSCRIPTis the Lorentz factor.
• •The moving body has the temperatureT′=T⁢γsuperscript𝑇′𝑇𝛾T^{\prime}=T\gammaitalic_T start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT = italic_T italic_γ[42,43].
• •The body temperature is an invariant thatT′=Tsuperscript𝑇′𝑇T^{\prime}=Titalic_T start_POSTSUPERSCRIPT ′ end_POSTSUPERSCRIPT = italic_T[44].

Discussing whether the temperatureT𝑇Titalic_Tof a body is a Lorentz invariant is beyond the scope of this paper. However, many studies have considered the thermodynamic relation

T⁢d⁢S=d⁢U+P⁢d⁢V𝑇𝑑𝑆𝑑𝑈𝑃𝑑𝑉TdS=dU+PdVitalic_T italic_d italic_S = italic_d italic_U + italic_P italic_d italic_V(D.1)to be Lorentz covariant, which means the fundamental thermodynamic relation holds in all reference frames. Based on this, we can still use

T=(∂U∂S)V𝑇subscript𝑈𝑆𝑉T=\left(\frac{\partial U}{\partial S}\right)_{V}italic_T = ( divide start_ARG ∂ italic_U end_ARG start_ARG ∂ italic_S end_ARG ) start_POSTSUBSCRIPT italic_V end_POSTSUBSCRIPT(D.2)to calculate temperature in any reference frames.

Appendix EAppendix V: Neural Network with Initially Uniform Distributed Parameters

All discussions in Section5are based on the condition that the initial parameters follow normal distributions. In this appendix, we discuss the scenario where the parameters of the neural network follow uniform distributions. Assuming all parameters are randomly distributed between[−l/2,l/2]𝑙2𝑙2[-l/2,l/2][ - italic_l / 2 , italic_l / 2 ], the asymptotic energy of the system withl→∞→𝑙l\rightarrow\inftyitalic_l → ∞and Tanh activation function is

⟨E0⟩delimited-⟨⟩subscript𝐸0\displaystyle\langle E_{0}\rangle⟨ italic_E start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟩=\displaystyle==1n⁢∫∑i=1n∑k=1lL(∑j=1lL−1Wk⁢jL⁢ai⁢jL−1+bkL−yi⁢k)2⁢f⁢(𝑾L)⁢d⁢𝑾L⁢d⁢bL1𝑛superscriptsubscript𝑖1𝑛superscriptsubscript𝑘1subscript𝑙𝐿superscriptsuperscriptsubscript𝑗1subscript𝑙𝐿1superscriptsubscript𝑊𝑘𝑗𝐿superscriptsubscript𝑎𝑖𝑗𝐿1superscriptsubscript𝑏𝑘𝐿subscript𝑦𝑖𝑘2𝑓superscript𝑾𝐿𝑑superscript𝑾𝐿𝑑superscript𝑏𝐿\displaystyle\frac{1}{n}\int\sum_{i=1}^{n}\sum_{k=1}^{l_{L}}\left(\sum_{j=1}^{% l_{L-1}}W_{kj}^{L}a_{ij}^{L-1}+b_{k}^{L}-y_{ik}\right)^{2}f({\boldsymbol{W}}^{% L})d{\boldsymbol{W}^{L}}db^{L}divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∫ ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ∑ start_POSTSUBSCRIPT italic_k = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT end_POSTSUPERSCRIPT ( ∑ start_POSTSUBSCRIPT italic_j = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT end_POSTSUPERSCRIPT italic_W start_POSTSUBSCRIPT italic_k italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT italic_a start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L - 1 end_POSTSUPERSCRIPT + italic_b start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT - italic_y start_POSTSUBSCRIPT italic_i italic_k end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_f ( bold_italic_W start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT ) italic_d bold_italic_W start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT italic_d italic_b start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT(E.1)=\displaystyle==1n⁢∑i∑k[∑jl212⁢(ai⁢jL−1)2+l212+(yi⁢k)2]1𝑛subscript𝑖subscript𝑘delimited-[]subscript𝑗superscript𝑙212superscriptsuperscriptsubscript𝑎𝑖𝑗𝐿12superscript𝑙212superscriptsubscript𝑦𝑖𝑘2\displaystyle\frac{1}{n}\sum_{i}\sum_{k}\left[\sum_{j}\frac{l^{2}}{12}(a_{ij}^% {L-1})^{2}+\frac{l^{2}}{12}+(y_{ik})^{2}\right]divide start_ARG 1 end_ARG start_ARG italic_n end_ARG ∑ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ∑ start_POSTSUBSCRIPT italic_k end_POSTSUBSCRIPT [ ∑ start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT divide start_ARG italic_l start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG ( italic_a start_POSTSUBSCRIPT italic_i italic_j end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L - 1 end_POSTSUPERSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT + divide start_ARG italic_l start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG + ( italic_y start_POSTSUBSCRIPT italic_i italic_k end_POSTSUBSCRIPT ) start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ]∼similar-to\displaystyle\sim∼l212⁢lL⁢(lL−1+1).superscript𝑙212subscript𝑙𝐿subscript𝑙𝐿11\displaystyle\frac{l^{2}}{12}l_{L}(l_{L-1}+1).divide start_ARG italic_l start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT + 1 ) .From this, the temperature of the system is

Tsys∼lL−1+1∑p=1Llp⁢(lp+1)⁢(l212⁢ln⁡l)similar-tosubscript𝑇syssubscript𝑙𝐿11superscriptsubscript𝑝1𝐿subscript𝑙𝑝subscript𝑙𝑝1superscript𝑙212𝑙T_{\rm sys}\sim\frac{l_{L-1}+1}{\sum_{p=1}^{L}l_{p}(l_{p}+1)}\left(\frac{l^{2}% }{12\ln l}\right)italic_T start_POSTSUBSCRIPT roman_sys end_POSTSUBSCRIPT ∼ divide start_ARG italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT + 1 end_ARG start_ARG ∑ start_POSTSUBSCRIPT italic_p = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_L end_POSTSUPERSCRIPT italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT + 1 ) end_ARG ( divide start_ARG italic_l start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 roman_ln italic_l end_ARG )(E.2)For the energy and temperature of each layer, we can calculate as follows

{Δ⁢E1=l1⁢l212⁢X2¯,S1=l1⁢(l0+1)⁢ln⁡lΔ⁢E2=l2⁢(l1+1)⁢l212,S2=l2⁢(l1+1)⁢ln⁡lΔ⁢E3=l3⁢(l2+1)⁢l212,S3=l3⁢(l2+1)⁢ln⁡l…⁢…Δ⁢EL=lL⁢(lL−1+1)⁢l212,SL=lL⁢(lL−1+1)⁢ln⁡l\left\{\begin{aligned} &\Delta E_{1}=l_{1}\frac{l^{2}}{12}\overline{X^{2}},&S_% {1}=l_{1}(l_{0}+1)\ln l\\ &\Delta E_{2}=l_{2}(l_{1}+1)\frac{l^{2}}{12},&S_{2}=l_{2}(l_{1}+1)\ln l\\ &\Delta E_{3}=l_{3}(l_{2}+1)\frac{l^{2}}{12},&S_{3}=l_{3}(l_{2}+1)\ln l\\ &......\\ &\Delta E_{L}=l_{L}(l_{L-1}+1)\frac{l^{2}}{12},&S_{L}=l_{L}(l_{L-1}+1)\ln l% \end{aligned}\right.{ start_ROW start_CELL end_CELL start_CELL roman_Δ italic_E start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT = italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT divide start_ARG italic_l start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG over¯ start_ARG italic_X start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG , end_CELL start_CELL italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT = italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT + 1 ) roman_ln italic_l end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL roman_Δ italic_E start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT = italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT + 1 ) divide start_ARG italic_l start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG , end_CELL start_CELL italic_S start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT = italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT + 1 ) roman_ln italic_l end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL roman_Δ italic_E start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT = italic_l start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT + 1 ) divide start_ARG italic_l start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG , end_CELL start_CELL italic_S start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT = italic_l start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT + 1 ) roman_ln italic_l end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL … … end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL roman_Δ italic_E start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT = italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT + 1 ) divide start_ARG italic_l start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG , end_CELL start_CELL italic_S start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT = italic_l start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT ( italic_l start_POSTSUBSCRIPT italic_L - 1 end_POSTSUBSCRIPT + 1 ) roman_ln italic_l end_CELL end_ROW(E.3)And we obtainT2∼T3∼T4∼…∼TL∼l2/ln⁡lsimilar-tosubscript𝑇2subscript𝑇3similar-tosubscript𝑇4similar-to…similar-tosubscript𝑇𝐿similar-tosuperscript𝑙2𝑙T_{2}\sim T_{3}\sim T_{4}\sim...\sim T_{L}\sim l^{2}/\ln litalic_T start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ∼ italic_T start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT ∼ italic_T start_POSTSUBSCRIPT 4 end_POSTSUBSCRIPT ∼ … ∼ italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT ∼ italic_l start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT / roman_ln italic_l, which belongs to the first type of heat engine. Clearly, by simply substituting

l212→σ2,→superscript𝑙212superscript𝜎2\frac{l^{2}}{12}\rightarrow\sigma^{2},divide start_ARG italic_l start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT end_ARG start_ARG 12 end_ARG → italic_σ start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT ,(E.4)the energy and temperatures of the system in mathematical form is identical to that in Section5. For the case where the activation function is ReLU, the system remains the second type of heat engine, corresponding to the work efficiencyη≈1𝜂1\eta\approx 1italic_η ≈ 1andTsys≈TL⁢(SL/Stot)subscript𝑇syssubscript𝑇𝐿subscript𝑆𝐿subscript𝑆totT_{\rm sys}\approx T_{L}(S_{L}/S_{\rm tot})italic_T start_POSTSUBSCRIPT roman_sys end_POSTSUBSCRIPT ≈ italic_T start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT ( italic_S start_POSTSUBSCRIPT italic_L end_POSTSUBSCRIPT / italic_S start_POSTSUBSCRIPT roman_tot end_POSTSUBSCRIPT ). Other details will not be elaborated further.

References

• [1]C. E. Shannon.A mathematical theory of communication.The Bell System Technical Journal, 27(3):379–423, 1948.
• [2]N. Wiener.Cybernetics: Or control and communication in the animal and the machine.1948.Chapter III.
• [3]Y. W. Teh, M. Welling, S. Osindero, and G. E. Hinton.Energy-based models for sparse overcomplete representations.JMLR, 4:1235–1260, 2003.
• [4]Y. Lecun, S. Chopra, R. Hadsell, M. A. Ranzato, and F. J. Huang.A tutorial on energy-based learning.MIT Press, 2006.
• [5]E. Ising.Beitrag zur theorie des ferromagnetismus.Zeitschrift für Physik, 31:253–258, 1925.
• [6]B. A.. Cipra.An introduction to the ising model.The American Mathematical Monthly, 94(10):937–959, 1987.
• [7]S.-I. Amari.Learning patterns and pattern sequences by self-organizing nets of threshold elements.IEEE Transactions. C, 21(11):1197–1206, 1972.
• [8]J. J. Hopfield.Neural networks and physical systems with emergent collective computational abilities.Proceedings of the National Academy of Sciences, 79(8):2554–2558, 1982.
• [9]D. H. Ackley, G. E. Hinton, and T. J. Sejnowski.A learning algorithm for boltzmann machines.Cognitive Science, 9(1):147–169, 1985.
• [10]D. Sherrington and S. Kirkpatrick.Solvable model of a spin-glass.Physical Review Letters, 35:1792–1796, 1975.
• [11]P. Smolensky.Information processing in dynamical systems: Foundations of harmony theory.Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Volume 1: Foundations, 35:194–281, 1986.
• [12]H. Larochelle and Y. Bengio.Classification using discriminative restricted boltzmann machines.ICML, pages 536–543, 2008.
• [13]C. Finn, P. Christiano, P. Abbeel, and S. Levine.A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models.NIPS 2016 Workshop on Adversarial Training, 2016.arXiv preprint, arXiv:1611.03852.
• [14]J. Zhao, M. Mathieu, and Y. LeCun.Energy-based generative adversarial network.International Conference on Learning Representations, 2017.arXiv preprint, arXiv:1609.03126.
• [15]I. Mordatch.Concept learning with energy-based models.2018.arXiv preprint, arXiv:1811.02486.
• [16]R. G. Melko, G. Carleo, J. Carrasquilla, and J. I. Cirac.Restricted boltzmann machines in quantum physics.Nature Physics, 15:887–892, 2019.
• [17]Y. Du and I. Mordatch.Implicit generation and generalization in energy-based models.2019.arXiv preprint, arXiv:1903.08689.
• [18]M. Arbel, L. Zhou, and A. Gretton.Generalized energy based models.2020.arXiv preprint, arXiv:2003.05033.
• [19]Y. Song and D. P. Kingma.How to train your energy-based models.2021.arXiv preprint, arXiv:2101.03288.
• [20]G. Li, L. Deng, Y. Xu, C. Wen, W. Wang, J. Pei, and L. Shi.Temperature based restricted boltzmann machines.Nature, Scientific Reports, 6(19133), 2016.
• [21]L. Aparecido, P. Junior, and J. P. Papa.Temperature-based deep boltzmann machines.Neural Processing Letters, 48:95–107, 2018.
• [22]R. Veiga and R. Vicente.Restricted boltzmann machine flows and the critical temperature of ising models.2022.arXiv preprint, arXiv:2006.10176.
• [23]J. C.. Platt.Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods.Advances in large margin classifiers, 10(3):61–74, 1999.
• [24]G. Hinton, O. Vinyals, and J. Dean.Distilling the knowledge in a neural network.NIPS 2014 Deep Learning Workshop, 2015.arXiv preprint, arXiv:1503.02531.
• [25]C. Guo, G. Pleiss, Y. Sun, and K. Q.. Weinberger.On calibration of modern neural networks.ICML, 70:1321–1330, 2017.
• [26]C. J. Maddison, A. Mnih, and Y. W. Teh.The concrete distribution: A continuous relaxation of discrete random variables.2016.arXiv preprint, arXiv:1611.00712.
• [27]J. Ficler and Y. Goldberg.Controlling linguistic style aspects in neural language generation.2017.arXiv preprint, arXiv:1707.02633.
• [28]L. D. Landau and E. M. Lifshitz.Statistical Physics.Course of Theoretical Physics. Butterworth-Heinemann, 3rd ed. edition, 1980.Chapter I.
• [29]J. E. Lennard-Jones.On the determination of molecular fields. -i. from the variation of the viscosity of a gas with temperature.Proceedings of the Royal Society of London. Series A, Containing Papers of a Mathematical and Physical Character, 106(738):441–462, 1924.
• [30]J. E. Lennard-Jones.On the determination of molecular fields. -ii. from the equation of state of a gas.Proceedings of the Royal Society of London. Series A, Containing Papers of a Mathematical and Physical Character, 106(738):463–477, 1924.
• [31]L. Onsager.Statistical hydrodynamics.Il Nuovo Cimento, 6(2 Supplement):279–287, 1949.
• [32]N. F. Ramsey.Thermodynamics and statistical mechanics at negative absolute temperatures.Physical Review, 103(1):20–28, 1956.
• [33]F. Jüttner.Das maxwellsche gesetz der geschwindigkeitsverteilung in der relativtheorie.Annalen der Physik, 339(5):856–882, 1911.
• [34]A. G. Walker.The boltzmann equations in general relativity.Proceedings of the Edinburgh Mathematical Society, 4(4):238–253, 1936.
• [35]G. M.. Kremer.The boltzmann equation in special and general relativity.28th International Symposium on Rarefied Gas Dynamics, AIP Conference Proceedings, 1501(1):160–167, 2012.
• [36]L. Zaninetti.New probability distributions in astrophysics: Iv. the relativistic maxwell-boltzmann distribution.International Journal of Astronomy and Astrophysics, 10(04):302–318, 2020.
• [37]C. Rovelli.General relativistic statistical mechanics.Physical Review D, 87(8):084055, 2013.
• [38]C. S. Lopez-Monsalvo and N. Andersson.Thermal dynamics in general relativity.Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 467(2127):738–759, 2011.
• [39]W. M. Mendes and B. P. e Silva.On the laws of thermodynamics in classical non-inertial frames.2021.arXiv preprint, arXiv:2104.13128.
• [40]M. Planck.Zur dynamik bewegter systeme.Sitzungsberichte der Königlich-Preussischen Akademie der Wissenschaften, pages 542–570, 1907.
• [41]A. Einstein.Über das relativitätsprinzip und die aus demselben gezogenen folgerungen.Jahrb. Radioakt. Elektron., 4:411–462, 1907.
• [42]H. Ott.Lorentz-transformation der wärme und der temperatur.Zeitschrift für Physik, 175(1):70–104, 1963.
• [43]H. Arzeliès.Transformation relativiste de la température et de quelques autres grandeurs thermodynamiques.Nuovo Cimento, 35:792–804, 1965.
• [44]P. T. Landsberg.Does a moving body appear cool?Nature, 212:571–572, 1966.

Phoenix Yin (@Phoenixyin13): 最近思考到，统计物理、信息科学和AI之间存在非常深刻、本质的统一关系。

先说统计物理。

统计物理研究大量粒子如何通过统计规律产生宏观热力学性质。 核心概念是概率分布、系综、熵（Boltzmann熵）、自由能、最概然态。

再说信息论。