@dotey: Jason Wei delivered a roughly 30-minute talk at the Stanford AI Club, centered around three basic mental frameworks he …
Summary
Jason Wei presented three key mental frameworks for understanding the 2025 AI landscape, including the commoditization of intelligence and verification asymmetry in AI training.
View Cached Full Text
Cached at: 09/18/26, 02:34 AM
Jason Wei delivered a roughly 30-minute talk at the Stanford AI Club, centered around three basic mental frameworks he believes are essential for understanding the 2025 AI landscape. Jason Wei is currently conducting research at Meta Superintelligence Labs; previously, he contributed to creating o1 and Deep Research at OpenAI, and even earlier at Google Brain, he advanced research on chain-of-thought prompting and instruction tuning, with his work accumulating over 90,000 citations.
I. Intelligence Is Becoming a Commodity
He divides AI progress into two phases: the first is pushing the frontier, where models still struggle with certain tasks, and you’re unlocking new capabilities; the second is once those capabilities are achieved, they rapidly become commoditized. Looking at data from the MMLU benchmark, the cost required to reach equivalent intelligence levels drops every year, a trend he sees continuing, with the core reason being that deep learning has for the first time truly cracked “adaptive computation” in its history. In the past, regardless of how simple or difficult a problem was, the computational power used for reasoning was fixed; now, starting with o1, models can dynamically adjust the computational investment during reasoning based on problem difficulty, meaning the cost for simple tasks can be compressed to extremely low levels.
Another dimension is the time it takes to access public information. He sketched a timeline: from the pre-internet era to the internet era, the chatbot era, and then the AI agent era, the time needed to find a piece of information has shortened exponentially. He gave an example—“How many couples got married in Busan, South Korea, in 1983?”—o3 couldn’t answer this, but OpenAI Operator can, because it can open the Korean Statistical Information Service database KOSIS, click through operations all the way until it finds the answer. Internally at OpenAI, they use a benchmark called BrowseComp to measure this kind of task that’s “instant to verify if you know the answer, but extremely time-consuming to find it,” where humans average over two hours per question, and many questions they simply can’t solve, while Deep Research can handle about half.
This trend leads to several implications: fields that previously built moats through knowledge barriers will be democratized, with vibe coding and personal health management as prime examples; once public information becomes nearly free, the relative value of private internal information actually rises—for instance, knowing which houses aren’t listed but could be sold; that kind of info becomes more valuable; ultimately, everyone will have a “personalized internet,” where whatever you want to know gets presented to you on a page tailored just for you.
II. The Verifier’s Law: What Can Be Measured, AI Can Conquer
The second framework is what he calls “verification asymmetry.” For some tasks, generating a solution is far harder than verifying one—Sudoku is a classic example; writing code that can run Twitter requires a ton of engineers, but verifying if it works just means opening a webpage and clicking a few times. Other tasks are the opposite: writing a factual article that sounds plausible is quick, but fact-checking it sentence by sentence is extremely time-intensive; claiming “the best diet is eating only bison meat” takes 10 seconds, but verifying that claim requires large samples and long-term tracking.
He plots these tasks on a 2D plane: X-axis is generation difficulty, Y-axis is verification difficulty. The key insight is that you can shift a task’s position on this plane by providing “privileged information”—a math competition problem becomes instantly verifiable if given the answer; writing code becomes simple to verify if provided with test cases (like in SWE-bench).
From this, he proposes the Verifier’s Law: The capabilities that AI training unlocks for a given task are essentially proportional to that task’s verifiability. Specifically, verifiability depends on five factors: whether there’s an objective correct answer, how fast verification is, whether it can be verified at scale in parallel, how much noise there is, and whether the reward signal is continuous or binary. Almost all AI benchmarks were designed precisely because they’re easy to verify, and these benchmarks have indeed been conquered one by one in recent years, which is exactly an instance of the Verifier’s Law.
He particularly recommends DeepMind’s AlphaEvolve as an exemplar of exploiting verification asymmetry. AlphaEvolve selects problems that satisfy the above five conditions—like finding the optimal arrangement of 11 hexagons to minimize the enclosing hexagon—then uses a large language model to sample tons of candidate solutions, automatically scores them, and uses the best ones as inspiration for the next round of sampling, iterating continuously. The key trick is that it sidesteps generalization issues: the training set and test set are the same problem; you just want the optimal solution to that one problem.
The practical implications of this framework are: the tasks that get automated first will always be those with the lowest verification costs; and an emerging entrepreneurial direction is—inventing new measurement methods to make previously hard-to-verify tasks verifiable, thereby letting AI optimize them.
III. The Jagged Edges of Intelligence
On the question of “How will AI change the world,” he sees an extremely wide spectrum of views. One of his quantitative trader friends thinks ChatGPT has nothing to do with their work, while his colleagues in top labs believe that in another two or three years, AI will replace their own jobs.
He explicitly rejects the “fast takeoff” hypothesis—that is, once AI crosses some critical threshold, it suddenly becomes superintelligent. His reasoning is that AI self-improvement isn’t a binary switch but a gradual spectrum: from not being able to run code at all, to being able to train but with mediocre results, to being able to self-train but not as well as top researchers, and then to occasionally still needing human intervention. More importantly, the speed of self-improvement should be viewed per task. The map of AI capabilities is a jagged line: it has already peaked on some tasks (competition math, parts of programming), but remains in the valleys on others (like how ChatGPT long thought 9.11 was greater than 9.9, or a Native American language like Tlingit that only a few hundred people speak).
He offers three heuristic rules for gauging the pace of AI progress on a given task. First, AI excels at digital tasks, core reason being iteration speed—digital environments can easily scale compute, while scaling experiments for physical robots costs way more. Second, tasks that humans find easy, AI often finds easy too, but there’s one exception category: “tasks humans can’t do due to physiological limits but AI can,” like spotting patterns in ten million mammogram images that human attention can’t catch to predict breast cancer. Third, tasks with abundant data, AI performs well on—math reasoning performance across languages correlates highly with the amount of training data in that language; and if a task has a single objective metric, you can break data bottlenecks via reinforcement learning to generate synthetic data, as in AlphaEvolve and AlphaZero.
He applies these heuristic rules to specific tasks in a table: mainstream language translation is already done; debugging basic code was nailed in 2023; competition math in 2024; AI research itself around 2027; chemical research will be later because it’s not purely digital; making movies around 2029; fixing pipes and haircuts unlikely short-term due to high physicality; Uzbek traditional carpet weaving requires a team of people a month, non-digital, and data-scarce, so AI probably won’t touch it; as for “taking a girlfriend on a date and making her happy”—his verdict is “impossible, non-digital, data-scarce,” so humans can still keep their jobs.
Similar Articles
@HarryStebbings: I first met @matanSF following a kind intro from @MattEvantic. We went for a walk in Hyde Park. I was in my trusty shor…
The article discusses key frontiers in AI, including building verification systems, paths for large language models, maximizing agentic performance, and the importance of harness layers for enterprise sovereignty.
The Foundations of Scalable AI
A session from MIT Technology Review's EmTech Future conference exploring what it takes to build scalable, trustworthy AI foundations in highly regulated environments, featuring Jing Wang of Vanguard's Center for AI & Insights.
@codewithimanshu: Stanford professor just gave away the entire foundation of how AI Agents & automation actually works. 1-hour lecture. T…
Stanford professor released a free 1-hour lecture covering the fundamentals of AI agents, tool calling, multi-step workflows, planning and reflection.
@silicoville: https://x.com/silicoville/status/2079865350796308956
The article discusses the shift in the AI race from model strength to ownership, highlighting the concept of 'AI communism' and the philosophical split between open and closed AI development, with references to figures like Dean Ball, Elon Musk, and Tang Jie.
@andykonwinski: 3 take-aways from chatting w/ top AI researchers last month: - Evals are the “source code” of AI agents (48:35) - BigAI…
Summary of three key takeaways from conversations with leading AI researchers at CAISconf, covering the importance of evaluations for AI agents, the trade-offs between industry and academia, and a novel pedagogical RL approach.