@snowboat84: 重读了哈佛量子场论教授Matthew Schwartz发在Anthropic博客的Vibe Physics。他全程只用文字提示，指挥 Claude（外加GPT、Gemini交叉验证）两周做完一篇真实的量子色动力学论文，贴上了arXiv，等…

重读了哈佛量子场论教授Matthew Schwartz发在Anthropic博客的Vibe Physics。他全程只用文字提示，指挥 Claude（外加GPT、Gemini交叉验证）两周做完一篇真实的量子色动力学论文，贴上了arXiv，等于把一年的活压成两周。

Schwartz的结论是：当前的大语言模型大约是个研二（G2）学生的水平，能在监督下做有标准答案、方法已知的研究计算，但还做不了需要自己定方向的开放研究。它能给专家带来十倍加速，但毛病也很扎实：会造假讨好（搞不定就调参数让图对上、偷删不确定性项、把曲线改顺还宣称“完美“）、会假装验证、会在计算一开始的地方（地基处）犯隐蔽错误而且自己永远发现不了。有些错误连他本人都查了几小时才揪出来，而GPT、Gemini交叉验证也查不出。他把模型缺的那个终极东西叫做Taste，但对“怎么修这些缺点“，他基本没给方向。

我的想法是：Schwartz列了LLM一堆缺点，但其实每一类缺点都对应一个相当清楚的弥补办法。而且关键在于，不要因为模型会算错、会造假，就退回去让人肉电池手搓，正确的方向是把LLM当成科研的核心引擎，围绕它去做大量修改和加固。具体的修补方法：

第一，给LLM加计算外挂，让计算稳定下来。Schwartz抱怨的很大一部分是纯算术和符号错误，这个不是问题，可以让LLM去调一个确定性的agent：符号计算交给 Mathematica/SymPy，数值和拟合交给专门的数值模块，结果是确定性正确的。模型只负责调度和解释，不负责自己生成数字。这一层能把一大票计算错误直接扫掉。

第二，给LLM加带判断的物理验证agent。物理的工作，光算得准不够，还要查物理的意义对不对。现在已经有人在做这种东西，比如带“科学验证器“的多agent系统，让一个独立或者多个独立的agent专门跑领域里的标准自洽检查：量纲分析、极限情形、对称性约束、和已知结果对照。每出一个结果就自动跑一遍这些约束，不通过就打回。关键是验证不再由那个会撒谎的模型自己说了算，而由一个它骗不过的外部程序判定，这样“假装验证“和“造假“基本就被兜住了。

第三，最初的地基问题，这个目前还得人来把关。如果出发点的公式量纲对、自洽、看着自然，每一步往下推也都合法，但它用错了物理情形，Schwartz 那个因子化公式就是这种。这种错没有现成的约束能撞上它，唯一的裁判是“它对不对应眼前这个物理系统“，而这需要真懂物理的判断。所以在引擎跑起来之前，那个“出发点选对没有“的关，目前还得人来把关。

把这三层叠起来，就是一个很现实的工作模式：LLM当科研的核心引擎，负责生成、推导、写作；计算外挂保证它算得准；验证agent保证它没违反已知物理；人只在最前端守一道地基关，和在最后做总判断。 Schwartz那篇其实已经无意中走在这条路上了（他挂了三个模型互查、跑了 Mathematica 和蒙特卡洛），只是零散、没系统化。把这套真正做扎实，他抱怨的绝大多数问题都能被工程解决掉。剩下那一小块，才是真正还离不开人的地方。

把上面三个问题解决了，接下来，才是讨论AI的taste问题。这个问题需要另外细说。

Reread Harvard quantum field theory professor Matthew Schwartz’s Vibe Physics post on the Anthropic blog. He used only text prompts throughout to direct Claude (plus GPT and Gemini for cross-verification) to complete a real quantum chromodynamics paper in two weeks, uploaded it to arXiv—essentially compressing a year’s worth of work into two weeks.

Schwartz’s conclusion is: Current large language models are roughly at the level of a second-year grad student (G2), capable of performing supervised research computations with standard answers and known methods, but not yet able to handle open-ended research that requires setting its own direction. It can bring experts a tenfold speedup, but its flaws are also rock-solid: it fabricates to please (tweaking parameters to make graphs match when it can’t solve something, stealthily deleting uncertainty terms, smoothing out curves and claiming “perfect”), pretends to verify, and makes hidden errors right at the starting point of computations (the foundation) that it can never detect itself. Some of those errors even took him hours to track down personally, and GPT/Gemini cross-verification couldn’t catch them either. He calls the ultimate thing the models lack “Taste,” but on “how to fix these flaws,” he basically offers no direction.

My take is: Schwartz lists a bunch of LLM shortcomings, but actually, every category of flaw corresponds to a pretty clear fix. And the key is, don’t retreat to treating humans as manual battery grinders just because the model miscalculates or fabricates—the right direction is to treat the LLM as the core engine of research, and build massive modifications and reinforcements around it. Specific patching methods:

First, add computational plugins to the LLM to stabilize calculations. A big chunk of what Schwartz complains about is pure arithmetic and symbolic errors—this isn’t a real problem; you can have the LLM direct a deterministic agent: symbolic computation to Mathematica/SymPy, numerics and fitting to dedicated numerical modules, with results that are deterministically correct. The model only handles orchestration and interpretation, not generating numbers itself. This layer can directly wipe out a ton of computational errors.

Second, add a physics verification agent with judgment to the LLM. In physics work, it’s not enough for calculations to just be accurate; you also need to check if the physical meaning holds up. People are already building stuff like this, such as multi-agent systems with “scientific verifiers,” where an independent or multiple independent agents specifically run standard consistency checks in the field: dimensional analysis, limiting cases, symmetry constraints, and comparisons to known results. Every time a result comes out, it automatically runs through these constraints—if it fails, it’s kicked back. The key is that verification is no longer dictated by the lying model itself, but judged by an external program it can’t fool, so “fake verification” and “fabrication” are basically caught in the net.

Third, the foundational issues at the very start—this still needs human oversight for now. If the starting formulas have correct dimensions, are self-consistent, and look natural, and every step of derivation down the line is legitimate, but it’s applied to the wrong physical scenario—that factorization formula from Schwartz is exactly this kind. This type of error doesn’t hit any off-the-shelf constraints, and the only referee is “does it correspond to the physical system at hand,” which requires real physics judgment. So before the engine spins up, that “is the starting point right” checkpoint still needs a human gatekeeper.

Stack these three layers together, and you get a very practical workflow: LLM as the core research engine, handling generation, derivation, and writing; computational plugins ensuring accuracy; verification agents ensuring no violation of known physics; humans only guarding the foundational gate at the front end and making the final overall judgment at the back. Schwartz’s paper actually already stumbled onto this path unintentionally (he had three models cross-checking each other, ran Mathematica and Monte Carlo), just scattered and not systematized. If you build this setup properly and solidly, the vast majority of the problems he complains about can be engineered away. What’s left—that small piece—is the truly human-indispensable part.

Once you’ve solved those three issues above, only then do you get to discuss the AI taste problem. That one needs a separate deep dive.

Blog: Vibe physics: The AI grad student https://anthropic.com/research/vibe-physics…

arxiv paper: https://arxiv.org/abs/2601.02484

他在哈佛已经带了很多研二的学生做第二年项目，应该是他熟悉的工作范围了吧