@BetaTomorrow: #DeepManifoldInterpretation Paper: On the Entropy Calibration of Language Models Author: Steven Cao, Gregory Valiant, a…

#DeepManifoldInterpretation Paper: On the Entropy Calibration of Language Models Author: Steven Cao, Gregory Valiant, and Percy Liang (@percyliang)

The paper’s entropy-calibration result can be interpreted as intrinsic-pathway instability under recursively changing boundary conditions.

In autoregressive generation, every generated token becomes part of the next context, so each step shifts the boundary condition of the iterated integral and may redirect inference into a different, more weakly constrained region of the learned manifold.

What the paper measures as rising entropy can therefore be viewed as increasing diffusion of the accessible pathway rather than simply token-by-token error accumulation.

Scaling helps little for natural language because heavy-tailed data continually expose new sparse regions of the manifold; increasing model size expands coverage, but also reveals additional poorly constrained geometry.

Truncation and lower-temperature decoding stabilize generation by narrowing the admissible pathways, at the cost of diversity.

Most interestingly, the paper’s future-entropy idea suggests a pathway-aware alternative: choose the present token not only from its local probability, but also from the expected stability of the continuation it opens.

#DeepManifoldInterpretation Paper: Small-Scale Experiments: Are We There Yet? Author: Nicholas Lourie (@NickLourie), Kyunghyun Cho(@kchonyc), Karen Ullrich(@karen_ullrich), Sanae Lotfi (@LotfiSanae)

The paper’s finding that carefully tuned small models can reproduce large-scale trends connects naturally to ** data-driven architecture **, but also exposes its limit. Learning is ** an inverse problem **: the structure and degrees of freedom required to represent the data are not known in advance.

Deep Manifold therefore argues that architecture should follow the learning space rather than be predetermined by scale alone; specifically,

Deep Manifold Part 2: Neural Network Mathematics, §4.7 Data Driven Architecture,

argues that model size should be driven primarily by the nonlinearity of the data within that learning space.

Small-scale experiments can reveal scaling tendencies within a chosen architecture family, but they cannot by themselves determine how many degrees of freedom the underlying learning problem ultimately requires.

The paper’s second major observation that small models are highly sensitive to hyperparameters while this sensitivity fades with scale, can be read through Deep Manifold as a difference in rigidity and training dynamics.

A small model has fewer effective degrees of freedom and is therefore more rigid: changes in learning rate, warmup, normalization, or other training conditions more strongly constrain its evolving manifold.

A larger model has greater representational elasticity and can tolerate or reorganize around such perturbations.

More importantly, training is a dynamical process, so hyperparameters do not act only at the moment they are applied; their effects propagate through subsequent iterations and may appear much later in the learned geometry.

As discussed in Mathematical Considerations for Training Dynamics, training conditions have temporal and delayed effects because each update changes the state from which all later learning proceeds.

From this perspective, larger models are not simply less sensitive to hyperparameters, they are dynamically more tolerant of their delayed effects.