@_yusufknl: In 1948, Claude Shannon invented the math behind every LLM you use today. He tested it by making his wife guess the nex…

In 1948, Claude Shannon invented the math behind every LLM you use today. He tested it by making his wife guess the next letter in a book.

A Stanford-trained mathematician just released a 32-minute walkthrough of this exact history - and why the “next-token prediction” story of GPT-5 is actually wrong.

Bookmark & watch this weekend. The alternative is a graduate info-theory course + 3 semesters of your life.

TL;DR: Claude Shannon’s 1948 information theory laid the mathematical foundation for LLMs (e.g., cross-entropy loss). Compression and prediction are mathematically equivalent. Through the example of encoding robot instructions, the video shows how optimal compression naturally leads to the definition of entropy.

The Limits of Text Compression and the Origins of Information Theory

When you encode text into binary, you always want to use as little data as possible. So the natural question is: is there a fundamental limit to how efficiently text can be compressed? ASCII uses eight bits per character—very inefficient. A slightly smarter approach—assign shorter bitstrings to common characters—can bring the average down to about four bits. Even smarter methods that exploit patterns in long sequences of text can do much better. But where is the limit? More interesting than any single numerical answer is: how would you even begin to answer this question?

This problem goes back at least to the 1940s, when Claude Shannon’s pioneering work launched information theory. Interestingly, the mathematical methods he developed to answer such questions turn out to be remarkably useful for modern machine learning. For example, when training large language models, pre-training is often described as “next-token prediction,” especially using the cross-entropy loss. That term comes from information theory. One conclusion of information theory is that prediction and compression are mathematically equivalent—they are two sides of the same coin. This means we can rethink the pre-training objective: it is fundamentally not about next-token prediction, but about creating the most efficient text compressor. Understanding this clarifies what cross-entropy is and why we use it.

Also, the phrase “compression is intelligence” is hard to evaluate rigorously (since “intelligence” is vaguely defined), but the mathematical theory of compression has a curious connection to artificial intelligence. This video is the first part of a trilogy, aiming to lay the mathematical foundation so we can later assess what this claim really means.

Warm-up Problem: Encoding Robot Instructions

Imagine a robot roaming the surface of a moon, receiving four movement commands: up, down, left, right, each with different probabilities: 1/2 for up, 1/4 for down, 1/8 for left, 1/8 for right. Each command is independent (a simplifying assumption). Sending a stream of bits to the robot is expensive, so the natural question is: what is the most efficient way to encode these instructions into a bitstream?

Three Student Approaches

Direct student: Use two fixed bits per instruction, e.g., 00=up, 01=down, 10=left, 11=right. Decoding is simple, but the average is 2 bits per instruction, not exploiting the probability differences.

Clever student: Exploit frequency differences by using variable-length bitstrings:

• up: 0 (1 bit)
• down: 10 (2 bits)
• left: 110 (3 bits)
• right: 111 (3 bits)

Average bits = 1/2×1 + 1/4×2 + 1/8×3 + 1/8×3 = 1.75 bits, better than the fixed 2 bits. When decoding, the robot reads the bitstream until it forms a complete codeword. The key: no codeword is a prefix of another—this kind of code is called a prefix code (or prefix-free code). Violating this rule (e.g., introducing a fifth instruction with 100) would cause ambiguity.

Visualizing with a Binary Tree

Each layer shows all binary strings of a given length. Strings starting with 0 are on the left half, those starting with 1 on the right. In the clever student’s assignment, “up” consumes half the space (all strings starting with 0 are banned), “down” consumes a quarter of the remainder, and “left” and “right” each consume an eighth. These proportions exactly equal the probabilities of each command. This perfect alignment suggests the scheme may have reached the limit.

Perfect Compression and Random Noise

The cloud-headed student argues: a perfectly optimal encoding should make the output bitstream indistinguishable from random noise—that is, each bit is independent and equally likely (50% 0, 50% 1). The clever student’s encoding indeed satisfies this: each new bit is like an independent coin flip. For example, the probability the first bit is 0 equals the probability of “up” = 1/2; otherwise it’s 1, then the probability the second bit is 0 equals the probability of “down” among the remaining cases = 1/2, and so on.

Why is random noise incompressible? From the receiver’s perspective: if a compressed message is represented by n bits, it is just one of 2^n possible messages, and if the output looks like random noise, all 2^n messages must be equally likely (probability 2^{-n}). If you try to make some message use fewer bits (moving down in the binary tree), you crowd out other messages, forcing them upward to use more bits, leading to overall inefficiency. For equally likely messages, the most efficient scheme gives all messages the same number of bits—intuitively reasonable, and the binary tree provides a more specific trade-off argument.

This line of thinking leads to the key definition: a message that uses n bits in perfect compression has probability 2^{-n}, which is the embryonic idea of entropy. The core idea of Shannon’s noiseless coding theorem—that the compression limit is determined by the probability distribution of messages—naturally emerges from this.

Source: YouTube video (https://www.youtube.com/watch?app=desktop&v=l6DKRf-fAAM&t=239s&pp=ygUNM2JsdWVvbmVicm93bdIHCQmbCwGHKiGM7w%3D%3D)

fintex (@_yusufknl): As someone who’s spent 3 years fine-tuning ML models, this lecture on neural networks from a Stanford math grad is the closest thing to a no-bullshit “day one of ML” briefing I’ve ever seen released publicly for free.

Everyone thinks neural nets are magic. They aren’t. They’re