@simplifyinAI: DeepSeek has dropped a fundamental rewrite of the Transformer architecture. And it solves the "identity crisis" that br…

X AI KOLs Timeline Papers
deepseek transformer-architecture manifold-constrained-hyper-connections ai-research residual-connections model-stability

Summary

DeepSeek has published a paper introducing mHC (Manifold-Constrained Hyper-Connections), a fundamental rewrite of the Transformer architecture that stabilizes large models by replacing standard residual connections with mathematically constrained multi-stream pathways.

DeepSeek has dropped a fundamental rewrite of the Transformer architecture. And it solves the "identity crisis" that breaks massive AI models. For the last decade, every major AI has relied on Residual Connections. Think of them as a fast lane that lets information skip layers to keep the signal pure. Without them, deep networks literally forget what they are doing and become untrainable. But there is a problem: As we make models bigger and deeper, these simple "skip paths" aren't enough anymore. Information gets diluted. Gradients explode. The math breaks. DeepSeek (and researchers including founder Wenfeng Liang) just released a paper introducing mHC: Manifold-Constrained Hyper-Connections. It is a complete overhaul of how data moves inside an AI. Instead of a single "skip lane," they widened the highway into multiple parallel streams. They call these Hyper-Connections. But they didn't stop there. When you have multiple streams, they usually descend into chaos. The AI loses its "identity mapping"—it stops being able to pass information forward without distorting it. DeepSeek’s breakthrough was forcing these connections to live on a specific mathematical "manifold." By projecting the connections onto the Birkhoff polytope (using the Sinkhorn-Knopp algorithm), they forced the AI to stay stable. It keeps the richness of multiple pathways, but ensures the signal never gets lost or blown out. The results are staggering: Stability: It successfully trained a 27B parameter model that was previously impossible to stabilize with standard Hyper-Connections. Performance: It crushed baselines on coding, math, and reasoning benchmarks (BBH and DROP). Efficiency: Despite the complexity, they engineered a custom kernel that adds only ~6.7% training overhead. We spent the last few years trying to make models smarter by making them bigger. DeepSeek just proved that the real gains come from fixing the plumbing. The future of scaling isn't just about more layers. It's about better connections between them.
Original Article

Similar Articles

xHC: Expanded Hyper-Connections

Hugging Face Daily Papers

This paper introduces xHC (Expanded Hyper-Connections), a method that enables meaningful expansion of residual streams in Transformers beyond the previous N=4 limit, achieving consistent improvements on 18B and 28B MoE models while adding only modest training FLOPs. It also presents xHC-Flash to reduce memory traffic, making large-N residual-stream expansion practical for LLM pre-training.

DeepSeek Introduces Vision

Hacker News Top

DeepSeek announces a new vision capability, likely a vision-language model, expanding its AI offerings.

@ZhihuFrontier: Half a year ago, a Zhihu contributor predicted that the next Transformer would absorb loops, recurrent state, sparse ro…

X AI KOLs Timeline

A Zhihu contributor's half-year-old prediction that the next Transformer would absorb loops, recurrent state, sparse routing, and latent reasoning is gaining relevance as Loop Engineering advances. The article explores how future Transformer architectures may evolve into hybrid models blending linear-complexity layers for background context with attention for precise reasoning, plus finer-grained sparsity and native System 2 reasoning.