@Michaelzsguo: 这是我最近看到的关于强化学习基础，以及它和现代 AI 关系的最好深度讨论之一。 Eric Jang 和 Dwarkesh 把一个看起来有点复古的练习，也就是用今天的工具重新构建 AlphaGo，变成了一堂非常清晰的大师课：为什么“搜索 +…

这是我最近看到的关于强化学习基础，以及它和现代 AI 关系的最好深度讨论之一。

Eric Jang 和 Dwarkesh 把一个看起来有点复古的练习，也就是用今天的工具重新构建 AlphaGo，变成了一堂非常清晰的大师课：为什么“搜索 + 自我对弈”仍然是我们工程化出来的、最干净的智能展示之一。

几个关键take-away：

• AlphaGo 的核心循环，是把神经网络和蒙特卡洛树搜索结合起来。神经网络负责两件事：一个 policy network，用来预测下一步棋的概率；一个 value network，用来评估当前局面的价值。MCTS 则让系统高效模拟许多可能的未来，把围棋早期接近 361 的巨大分支空间，压缩成可执行的决策。神经网络本质上是在“摊销”深度搜索的成本，让系统不需要每一次都从头穷尽模拟所有可能性。

• 最有启发的是 credit assignment，也就是如何判断到底是哪一步导致了成功。对 LLM 来说，这也是最难的问题之一。在 AlphaGo 式的自我对弈 + MCTS 里，每一步都能从搜索中得到一个更干净、更高质量的训练目标：更优的动作建议，以及对应的价值评估。相比之下，如果对 LLM 做朴素的 policy gradient RL，你面对的是十万 token 级别的长轨迹，却要猜到底哪些 token 真正推动了成功。这个信号非常嘈杂，也非常低效。人类学习和 AlphaGo，似乎都更接近这种干净信号的版本。

• 还有一个很现代的角度：Jang 用大约 1,000 到 10,000 美元的算力，借助 LLM 做编码和实验，重新构建了一个很强的版本。这个开源项目也在测试自动化研究循环：LLM 已经很擅长实现实验、调参和跑工程流程，但它们仍然不擅长选择真正有前途的问题，也不擅长跳出局部最优。这一点和我们现在讨论“什么时候会出现完全自动化的 AI 研究员”高度相关。

Eric Jang 是一位 AI 研究者和工程师，曾在 Google Brain / Robotics 等团队做过机器人学习、强化学习和生成模型相关工作，也长期写高质量技术文章，把复杂概念讲得很清楚。

这个 podcast 的黑板形式也特别适合这个主题。看他们一步步画出博弈树、value function 和 self-play loop，比看幻灯片更容易真正理解这些概念。

Dwarkesh Patel (@dwarkesh_sp): New blackboard lecture w @ericjang11

He walks through how to build AlphaGo from scratch, but with modern AI tools.

Sometimes you understand the future better by stepping backward. AlphaGo is still the cleanest worked example of the primitives of intelligence: search, learning

This is one of the best in-depth discussions I’ve seen recently on the basics of reinforcement learning and its relationship to modern AI.

Eric Jang and Dwarkesh take what seems like a somewhat retro exercise—rebuilding AlphaGo with today’s tools—and turn it into a remarkably clear masterclass: why “search + self-play” remains one of the cleanest demonstrations of engineered intelligence we’ve come up with.

A few key takeaways:

• The core loop of AlphaGo combines neural networks with Monte Carlo Tree Search. The neural networks handle two things: a policy network to predict the probabilities of the next move; and a value network to evaluate the worth of the current board position. MCTS then allows the system to efficiently simulate many possible futures, compressing Go’s enormous early-game branching space of nearly 361 possibilities into actionable decisions. Neural networks essentially “amortize” the cost of deep search, so the system doesn’t have to exhaustively simulate every possibility from scratch each time.

• The most insightful part is credit assignment—figuring out exactly which move led to success. For LLMs, this is one of the hardest problems too. In AlphaGo-style self-play + MCTS, every move gets a cleaner, higher-quality training signal from the search: better action suggestions and corresponding value estimates. By contrast, if you do naive policy gradient RL on an LLM, you’re dealing with trajectories at the scale of hundreds of thousands of tokens, yet you have to guess which ones truly drove success. That signal is incredibly noisy and inefficient. Human learning and AlphaGo both seem closer to this version with clean signals.

• There’s also a very modern angle: Jang rebuilt a strong version using about $1,000 to $10,000 in compute, leveraging LLMs for coding and experimentation. This open-source project is also testing automated research loops: LLMs are already great at implementing experiments, tuning parameters, and running engineering workflows, but they’re still not good at picking truly promising problems or escaping local optima. This ties directly into our current debates about “when will fully automated AI researchers emerge.”

Eric Jang is an AI researcher and engineer who has worked on robot learning, reinforcement learning, and generative models in teams like Google Brain / Robotics, and he regularly writes high-quality technical articles that explain complex concepts with exceptional clarity.

The podcast’s blackboard format is also particularly well-suited to this topic. Watching them step-by-step draw out game trees, value functions, and self-play loops makes it easier to truly grasp these concepts than watching slides.

Jang’s open-source autogo

Codex CLI has a new spashpage. Pretty cool. @OpenAIDevs