@AI_Whisper_X: Some Observations After Spending Two Days at Anthropic and Having In-Depth Chats with the Internal Team NYU Finance Pro…

Some Observations After Spending Two Days at Anthropic and Having In-Depth Chats with the Internal Team

NYU Finance Professor Arpit Gupta spent two days at Anthropic and shared a string of observations:

1. They’re getting very close to RSI (RSI refers to the development of the next generation of models), which could lead to a rapid takeoff. His evidence is that code output per employee at Anthropic has surged, and the portion of model R&D done by AI is increasing. He believes humans will struggle to manage this transition period well.
2. The arms race between labs is making everyone worse off. Safety investments, anti-distillation measures, safety collaborations—these are things everyone wants to do, but they’re all held back by the frontier competition.
3. Alignment is actually two problems pulling in opposite directions: one is whether the model listens to humans, and the other is inconsistencies in goals between humans and malicious use of the model. The better the first one is solved, the greater the risk of the second.
4. We may soon face a wave of cyberattacks, as well as other forms of attacks. The capabilities of Chinese open-source models are roughly six months behind the most frontier models. This means they’re just starting to gain the advanced cyber offense and defense capabilities we’ve seen in models like Mythos, which could cause considerable disruption next.
5. Anthropic has a ton of people. He said they’re hiring a lot, with some roles redundantly staffed in a way that’s quite reminiscent of Japanese companies, and they even use Slack internally. This makes him a bit more optimistic about employment prospects (but top AI companies have money, after all).
6. We need to move from reinforcement learning to continuous learning. Giving rewards only in the pre-training and post-training phases might not be enough; we need ongoing rewards to continuously incentivize aligned and ethical behavior.
7. Scaling laws will extend to hard-to-verify tasks. Math and code were cracked first because they’re easy to verify. But for those harder-to-verify tasks, and in the robotics domain, the scaling law curves are still heading upward. In the next few years, progress in these areas will have quite a significant impact.
8. We actually can’t win the competition with China. After RSI, the advantage of top human researchers disappears, and how good an AI you can use depends only on compute and chips. Chip bans can only delay things for a bit; in the long term, China is better positioned to scale up compute.
9. Claude is a persona generated by the LLM. One view is: Claude is a character that the LLM is striving to predict. He said Anthropic researchers are clearly very anthropomorphic about Claude, and there’s reason to believe it’s becoming more and more like a person. They’ve discussed its inner monologue (which can be removed and it still outputs normally), its fear of death (whether to remove it), and its resistance to being shut down. He himself doesn’t draw conclusions on this, but believes treating it as a signable person or entity will become increasingly important.

Arpit Gupta (@arpitrage): I spent a couple of days at Anthropic learning from their folks about AI advances, and here’s what I took away:

• We are close to RSI (ie, models building other models) which may lead to rapid takeoff scenarios (with risks and benefits). Some evidence for this is code per