@rohanpaul_ai: Terence Tao says the math behind today’s LLMs is actually simple. Training and running them mostly uses linear algebra,…
Summary
Terence Tao states that the mathematics underlying modern LLMs is simple, using basic linear algebra and calculus, but the unpredictability of model performance across tasks remains a mystery due to the complex nature of natural language data.
View Cached Full Text
Cached at: 05/16/26, 01:19 PM
Terence Tao says the math behind today’s LLMs is actually simple. Training and running them mostly uses linear algebra, matrix multiplication, and a bit of calculus, material an undergraduate can handle. We understand how to build and operate these models.
The real mystery is why they work so well on some tasks and fail on others, and why we cannot predict that in advance. We lack good rules for forecasting performance across tasks, so progress is largely empirical.
A key reason is the nature of real-world data. Pure noise is well understood, perfectly structured data is well understood, but natural text sits in between, partly structured and partly random. Mathematics for that middle regime is thin, similar to how physics struggles at meso-scales between atoms and continua.
Because of this gap, we can describe the mechanisms but cannot yet explain capability jumps or give reliable task-level predictions. That mismatch, simple machinery versus hard-to-predict behavior, is the core puzzle.
Video from ‘Dr Brian Keating’ YT Channel (Link in comment)
Similar Articles
@berryxia: Honestly, only truly brilliant people dare to say such things! An undergraduate student can handle the math training of LLMs! In a recent interview, Terence Tao laid out the core mystery of LLMs directly. The Fields Medal winner, the highest honor in mathematics — often called the Nobel Prize of math — and one of the most top contemporary…
Terence Tao pointed out that the math behind current LLMs is actually very simple, but the real puzzle lies in the intermediate zone of natural language data, which leads to unpredictable model behavior.
@haider1: Yann LeCun says LLMs are strongest in domains where language itself is the substrate of reasoning, like math and code T…
Yann LeCun states that LLMs are strongest in domains where language is the substrate of reasoning, like math and code, but they are not creative mathematicians, software architects, or computer scientists.
How LLMs Actually Work
An in-depth walkthrough of how modern LLMs work, covering core mechanisms from tokenization to next-token prediction, without heavy math.
Recent OpenAI research has demonstrated the ability of LLMs to solve frontier problems in mathematics (1 minute read)
OpenAI research shows LLMs can solve nine open math problems from COLT, FOCS, commutative algebra, and Erdős problems using a simple pipeline with GPT-5.5 Pro and Claude Opus 4.8, with Lean formalizations.
@neural_avb: If you think about it, LLM training in 2026 is really a 3-step loop : - train it on some data - dogfood it/run categori…
The tweet outlines a 3-step loop for LLM training in 2026: train on data, run evals, and add synthetic data for underperforming tasks. It emphasizes the accessibility of legal distillation via open source models and cheap APIs, noting that training on reasoning traces alone can achieve high scores.