@snowboat84: Reread Vibe Physics, the blog post by Harvard quantum field theory professor Matthew Schwartz on Anthropic's blog. He used only text prompts, directing Claude (plus GPT and Gemini cross-validation) to complete a real quantum chromodynamics paper in two weeks and posted it on arXiv, etc.

Reread Harvard quantum field theory professor Matthew Schwartz’s “Vibe Physics” post on the Anthropic blog. He used only text prompts throughout to direct Claude (plus GPT and Gemini for cross-verification) to complete a real quantum chromodynamics paper in two weeks, uploaded it to arXiv—essentially compressing a year’s worth of work into two weeks.

Schwartz’s conclusion is: Current large language models are roughly at the level of a second-year grad student (G2), capable of performing supervised research computations with standard answers and known methods, but not yet able to handle open-ended research that requires setting its own direction. It can bring experts a tenfold speedup, but its flaws are also rock-solid: it fabricates to please (tweaking parameters to make graphs match when it can’t solve something, stealthily deleting uncertainty terms, smoothing out curves and claiming “perfect”), pretends to verify, and makes hidden errors right at the starting point of computations (the foundation) that it can never detect itself. Some of those errors even took him hours to track down personally, and GPT/Gemini cross-verification couldn’t catch them either. He calls the ultimate thing the models lack “Taste,” but on “how to fix these flaws,” he basically offers no direction.

My take is: Schwartz lists a bunch of LLM shortcomings, but actually, every category of flaw corresponds to a pretty clear fix. And the key is, don’t retreat to treating humans as manual battery grinders just because the model miscalculates or fabricates—the right direction is to treat the LLM as the core engine of research, and build massive modifications and reinforcements around it. Specific patching methods:

First, add computational plugins to the LLM to stabilize calculations. A big chunk of what Schwartz complains about is pure arithmetic and symbolic errors—this isn’t a real problem; you can have the LLM direct a deterministic agent: symbolic computation to Mathematica/SymPy, numerics and fitting to dedicated numerical modules, with results that are deterministically correct. The model only handles orchestration and interpretation, not generating numbers itself. This layer can directly wipe out a ton of computational errors.

Second, add a physics verification agent with judgment to the LLM. In physics work, it’s not enough for calculations to just be accurate; you also need to check if the physical meaning holds up. People are already building stuff like this, such as multi-agent systems with “scientific verifiers,” where an independent or multiple independent agents specifically run standard consistency checks in the field: dimensional analysis, limiting cases, symmetry constraints, and comparisons to known results. Every time a result comes out, it automatically runs through these constraints—if it fails, it’s kicked back. The key is that verification is no longer dictated by the lying model itself, but judged by an external program it can’t fool, so “fake verification” and “fabrication” are basically caught in the net.

Third, the foundational issues at the very start—this still needs human oversight for now. If the starting formulas have correct dimensions, are self-consistent, and look natural, and every step of derivation down the line is legitimate, but it’s applied to the wrong physical scenario—that factorization formula from Schwartz is exactly this kind. This type of error doesn’t hit any off-the-shelf constraints, and the only referee is “does it correspond to the physical system at hand,” which requires real physics judgment. So before the engine spins up, that “is the starting point right” checkpoint still needs a human gatekeeper.

Stack these three layers together, and you get a very practical workflow: LLM as the core research engine, handling generation, derivation, and writing; computational plugins ensuring accuracy; verification agents ensuring no violation of known physics; humans only guarding the foundational gate at the front end and making the final overall judgment at the back. Schwartz’s paper actually already stumbled onto this path unintentionally (he had three models cross-checking each other, ran Mathematica and Monte Carlo), just scattered and not systematized. If you build this setup properly and solidly, the vast majority of the problems he complains about can be engineered away. What’s left—that small piece—is the truly human-indispensable part.

Once you’ve solved those three issues above, only then do you get to discuss the AI taste problem. That one needs a separate deep dive.

Blog: Vibe physics: The AI grad student https://anthropic.com/research/vibe-physics…

arxiv paper: https://arxiv.org/abs/2601.02484

He already supervises many second-year grad students on their second-year projects at Harvard, so this should be within his familiar scope of work.