Mathematical Considerations for Transfer Learning through Category Theory

English Edition: Mathematical Considerations for Transfer Learning through Category Theory

I have not received formal education or training in category theory. However, through Deep Manifold, we discovered that Transformers ultimately perform classification via propertyless numerical counting. This observation prompted me to explore category theory: not to claim that neural networks formally implement category theory, but as a possible language to understand how neural networks classify, abstract, and transfer learned structures across different domains.

The Deep Manifold Perspective

There is already considerable research on the connection between neural networks and category theory. Many works use category theory as a formal language to describe composition, representation, equivariance, logic, or learning systems. Graph neural networks are perhaps the closest to a categorical implementation among neural architectures, as they inherently operate on explicit relational structures: nodes, edges, neighborhoods, paths, and message passing. In this setting, category theory can naturally describe how information is preserved, transformed, and composed within graph structures.

But the Deep Manifold starts from a different point.

First, Deep Manifold begins with propertyless numerical counting. A neural network does not start from symbolic meanings, object attributes, or human-defined categories. It begins from numerical states with no inherent properties. Through model architecture, data, and training, these propertyless numerical states are shaped into learned relations, transformations, and compositional paths. Classification does not arise from stored semantic objects; it emerges from learned numerical structures.

Second, Deep Manifold is not limited to architectures with explicit relational structures like graph neural networks. In graph neural networks, the graph structure is often given: nodes and edges provide the relational skeleton. Deep Manifold poses a broader question: when the input initially consists of propertyless numerical states rather than an explicit graph structure, how do relational abstractions emerge in ordinary neural networks, including Transformers? In this perspective, relational structure is not directly provided by the data format; it is learned through model architecture, stacked manifold coverings, and data and training strategies grounded in fixed-point class theory.

This is where Deep Manifold provides a unique unifying perspective. It connects category-theoretic abstractions, model architecture, and data and training strategies grounded in fixed-point class theory. Category theory provides the language of relations, morphisms, composition, and classification. Model architecture provides the computational structures in which these relations are learned. Data and training strategies guided by fixed-point class theory provide the boundary conditions that determine which relational structures can stabilize, be reused, and transfer.

This does not mean that Transformers formally implement category theory. That claim would be too strong. More precisely, Deep Manifold suggests that neural networks exhibit a category-theoretic-like abstraction because their learned computations are relational, compositional, and transferable across stacked manifold coverings. The contribution of Deep Manifold is not to replace category theory, nor to reduce neural networks to formal category theory, but to ask: how can relational abstractions emerge from propertyless numerical computation?

Category Theory and the Inner Logic of Classification

Category theory is often regarded as one of the most abstract branches of modern mathematics. But its abstraction is not vacuous. Its power comes from a simple shift of attention: instead of asking what is inside an object, ask how objects relate to each other, how they transform, how they compose, and what structures are preserved through these relations.

This is why category theory is so deeply connected to classification. Classification is not just putting things into different boxes. Shallow classification merely lists similarities. Deeper classification explains why different things belong to the same category, how one structure can transform into another, and what remains invariant under these transformations.

In this sense, category theory is not just a tool for classification; it is the language of the inner logic of classification. It defines objects not solely by their internal entities, but through their relations, morphisms, transformations, and roles within larger structures.

This perspective is important for understanding neural networks. A neural network does not just memorize object attributes. It does not store “catness,” “redness,” “grammar,” or “reasoning” as fixed symbolic objects in its weights. Instead, it learns numerical relations. It learns how one propertyless numerical state transforms into another, how local structures compose across layers, and how the final output is classified from a long chain of learned transformations.

From the Deep Manifold perspective, neural networks therefore exhibit a category-theoretic-like abstraction. Meaning does not reside in isolated activations or stored object attributes. Meaning emerges from learned relations, transformations, and compositions among propertyless numerical states across stacked manifold coverings.

Meaning Emerges in Stacked Manifold Coverings

This also reinterprets next-token prediction. The final step of a Transformer is often described as predicting the next token, but mathematically, it is essentially a large-scale classification task. Given a context, the model generates a probability distribution over all possible next tokens, then selects, samples, or ranks tokens from an extremely large discrete vocabulary.

However, the final classification is only the visible endpoint. Before reaching this endpoint, the model has performed deep, composite transformations on the prompt. Each layer reshapes the numerical states; each attention and feedforward module modifies local relationships between tokens; each residual path carries and continuously updates the evolving state.

Therefore, it is misleading to say in an overly simplistic way that language models are “merely next-token predictors.” Yes, the training objective is next-token prediction; yes, the final step is classification over the vocabulary. But the learned structure that makes this classification possible is not simple. It is built through stacked manifold coverings, learned relations, composite computational paths, and boundary conditions provided jointly by data, model architecture, and training strategy.

Category theory helps us name this level of abstraction. It reminds us that structure does not always reside inside isolated objects. Structure can exist in relations. A single token has no meaning by itself. An activation value has no meaning by itself. A neuron has no meaning by itself. Meaning only appears when these propertyless numerical states are placed into a learned system of relations and transformed across layers.

In the language of Deep Manifold, mathematical coverings are the deeper learned structures, while physical coverings are the observable behaviors: activations, logits, classifications, generated tokens, and model outputs. The output token is not the manifold itself. It is merely a trace left by the learned manifold computation under specific boundary conditions.

Why Cross-Domain Transfer Learning Is Possible

This category-theoretic-like abstraction also helps explain why cross-domain transfer learning is theoretically possible. A neural network does not need to transfer a fixed object attribute from one domain to another. It can transfer relational and compositional structures.

For example, a model may learn certain relational structures in language that later prove useful for code tasks. It may learn correspondences between vision and text that later aid multimodal reasoning. It may learn compositional patterns in one domain that become useful in another. Transfer is not magic. It is possible because learned computations are not bound to surface attributes, but are organized in deeper numerical relations.

When model architecture, data, and training strategies provide compatible manifold coverings and boundary conditions, these learned structures can be reused, transformed, and stabilized in new domains. This is why the same base model can often transfer from text to code, from language to reasoning, from images to descriptions, or from scientific data to prediction tasks. The model does not just copy surface features; it reuses learned relational structures.

This is also where Deep Manifold complements the category-theoretic perspective. Category theory gives us a high-level language of relations, transformations, and classification. Deep Manifold provides a numerical and geometric interpretation of how this abstraction can be learned inside neural networks. A neural network does not directly implement category theory as formal mathematics. More precisely, it exhibits a category-theoretic-like abstraction because its learned computations are relational, compositional, and transferable across stacked manifold coverings.

In this perspective, classification is not a narrow endpoint. It is the final expression of a deeper process. The model starts from propertyless numerical states. Through training, these states are shaped by data into manifold coverings. Through inference, the prompt becomes a boundary condition. Through layer-by-layer computation, the model performs learned transformations across these coverings. Finally, the system returns to classification: a token, a label, an answer, or an action.

Category theory tells us that classification can be understood through relations, not through internal essences. Deep Manifold further suggests that neural networks learn in this way. They do not first have meaning and then classify. They learn relational numerical structures; when those structures become stable enough to support classification, transfer, and composition, meaning emerges.

This may be one reason why neural networks exhibit remarkable generalization. Their abstraction is not classical symbolic abstraction. It is not purely logical, not purely semantic, not purely statistical. It is learned numerical abstraction. It is propertyless computation organized into relations. And when these relations are sufficiently stable across domains, transfer learning is not only practically effective but also theoretically natural.

• What is Deep Manifold ?

• Neural Network Fixed-Point Field

• Deep Manifold in the Real World

Single Token Geometry Series

• Single Token Geometry 01: Topology

• Single Token Geometry 02: DeepSeek V4 and Manifold Tearing

• Single Token Geometry 03: Data Complexity

• Single Token Geometry 04: A Critique of Manifold Steering

• Single Token Geometry 05: Numerical Manifold Method

• Single Token Geometry 06: Stacked Piecewise Manifold

• Single Token Geometry 07: Attention

deepmanifold.ai