@mubeitech: The Transformer is not the endgame of AI, says NVIDIA VP of AI Research Sanja Fidler.
Summary
Sanja Fidler, VP of AI Research at NVIDIA and head of the company’s spatial-intelligence lab, says the Transformer’s Achilles heel is clear: training costs are sky-high and the hunger for data is bottomless. A new architectural breakthrough is overdue, and next-gen variants are already emerging.
View Cached Full Text
Cached at: Unknown
The Transformer is by no means the endgame for AI.
That’s the verdict from Sanja Fidler, VP of AI Research at NVIDIA.
The head of NVIDIA’s spatial-intelligence lab sees a fatal flaw in today’s architectures:
training costs are astronomical,
the hunger for data is bottomless,
and a fundamental rethink is the only way forward.
The next-generation mutants are already emerging.
Similar Articles
@snowboat84: Continuing the discussion on applying physical models in AI. Today's Transformer uses attention to allow information at different positions in a sequence to interact, but this mixing is very likely lossy and irreversible. Independent pieces of information get blurred and lost as they mix, and it's impossible to precisely reverse-engineer the input from the output. What we want is a different kind of interaction...
The author discusses the problem of information loss and irreversibility caused by the Transformer's attention mechanism, and proposes drawing inspiration from the physical model of soliton propagation to design a reversible, zero-loss interaction layer as a direction for improving existing AI model architectures.
@NFTCPS: You keep talking about AI, but can't even explain what a Transformer is? There's a repo that goes all out — builds a GPT from scratch without using any high-level libraries. It lays out exactly how Attention, Multi-Head, Feed-Forward, Embedding, Residual connections, and Layer Norm are pieced together. And it's not just the model; the entire pipeline is covered…
A GitHub open-source project that implements the complete GPT training pipeline from scratch, including data preprocessing, pretraining, SFT, and RLHF post-training, all based on native PyTorch. Ideal for developers who want to deeply understand the Transformer architecture.
@Phoenixyin13: AI has fallen into an either-or trap. On one side is the world-dominating Transformer architecture — excellent memory, but its quadratic computational explosion makes long contexts increasingly expensive, a real resource hog. On the other is the classic RNN architecture — lightning fast and cheap, but a total scatterbrain that forgets earlier content after a few more lines.
This article introduces a new method proposed by Google Research, Cornell, and USC that takes snapshots of RNN memory and caches them, enabling RNNs to efficiently handle long contexts. It combines Transformer-like strong memory with RNN-like low cost, offering a new direction for long-context AI.
@AYi_AInotes: A counter-intuitive judgment: 80% of Agent production crashes have nothing to do with model IQ — they're all from context overflow, tool misconfiguration, sub-agent runaway. The real watershed in 2026 is Harness and Loop, not the model. Bro, @wizardly_ai's engineering note...
This article points out that 80% of AI Agent production crashes are not due to model intelligence, but are caused by context overflow, tool misconfiguration, and sub-agent runaway. The author emphasizes that the watershed in 2026 lies in Harness (office systems, security) and Loop (automatic cycling mechanism), not the model itself.
@0xLogicrw: Noam Shazeer, Google AI key figure and Gemini model technical lead, leaves Google again and officially joins rival OpenAI. OpenAI announced to employees that Shazeer will focus on finding entirely new underlying architectures for large models and advancing the Transformer...
Noam Shazeer, co-author of the Transformer architecture and technical lead of Google's Gemini model, has left Google again and officially joined OpenAI. He will focus on discovering new underlying architectures for large models and driving research into the evolution of Transformers.