8 Predictions for the Era of Continual Learning | Dwarkesh Patel

# Eight Predictions for the Era of Continual Learning TL;DR: When AI is no longer "train first, deploy later" but instead continuously learns at the deployment site, safety regulation, alignment research, model diversity, competitive dynamics, and business models will all fundamentally change. This article offers eight concrete predictions. ## Why continual learning is needed I've explained elsewhere why I think continual learning is genuinely necessary. If you force AI to only write markdown files between sessions, I don't think they can handle full jobs the way humans can. For example, imagine if students learned the saxophone this way. There is a student who has never played the saxophone. He walks into the concert hall and tries to play. Of course, since it's his first time, he fails, and then he writes down a bunch of notes about what went wrong. The next student is waiting outside the concert hall. He walks in, reads all these notes. He has also never played, so of course he messes up, and he keeps adding to these notes. There is an endless line of students outside the concert hall, constantly writing notes for the next student. I don't think any string of text they could write would enable a later student to play the saxophone successfully on the first try. At some point, you have to actually accumulate relevant experience in the brain. I believe the same is true for many of the skills we want AI to genuinely accumulate from the various workplaces where they are deployed. So once we actually achieve continual learning, what changes? ## Prediction 1: The "train-then-deploy" assumption in safety regulation will break down Many proposals for regulating AI assume that you first train a model and then deploy it. So if you run a battery of checks on the model before deployment, you can ensure it won't help with cyberattacks or do something crazy. I don't think this assumption will necessarily hold in the future, which is one of many reasons I'm worried about locking in a particular safety regulation regime right now—because we don't know what kind of technology we'll be dealing with in a year, let alone five or ten. What if models improve every day based on the millions of sessions they've done that day? If that's the case, we might be locking in an outdated and potentially counterproductive way to address the threats posed by AI. As far as government safety assessments of model providers go, I think monthly or quarterly risk checks make more sense than trying to single out some special moment after training is complete and before deployment begins, because in the future that won't be a meaningful standalone category anymore. ## Prediction 2: The focus of technical alignment will shift from frozen weights to continuous updates Right now, a lot of research is focused on how to ensure a frozen set of weights behaves well during deployment. But I'm not sure how much research is looking at the question of how we keep AI systems from ever being jailbroken, or from turning into a deceptive or evil persona, even as weights are continuously updated. And if AI is also integrating learning across users, how do we prevent users from injecting backdoors or some kind of malicious tendency into the base model? In a sense, this is the human alignment problem, right? Humans grow in a self-directed way. If you have children—I don't have kids, but I imagine this is what happens—they go out and learn new things. Sometimes they go off the rails. They get hit by some crazy ideology, take the wrong drugs, become super weird. But you hope you've given them enough common sense and basic values that they can grow up in a self-directed way without ending up with some super strange beliefs or misanthropic ideas. ## Prediction 3: The diversity of AI minds will increase Currently, there are fewer than five famous AI minds, and by that I mean foundation models serving millions, hundreds of millions, or billions of users at the same time. And, by the way, they're all very similar to each other because they're trained on roughly the same data. But if AI learns from experience, and experiences differ across AI companies—and even across different instances of the same model—we might see a world that ends up with a great deal of diversity. I think that would be a net good outcome. One thing to worry about in the future is a rather boring monolithic giant. A world with continual learning will hopefully be far more interesting than the mode-collapsed models we see in today's world. ## Prediction 4: The leader's returns accelerate When deployment becomes part of training, the returns to being ahead in the AI race accelerate. If you have the best model, and more people are using your AI for more complex, useful work, the result is that they give it a huge amount of feedback beyond the session window that it can integrate, and so your model becomes even smarter. ## Prediction 5: Labs will deploy their strongest models earlier If models primarily learn from deployment, labs will feel strong pressure to deploy their smartest models earlier. According to reports, Anthropic has been using Mythos internally since February, but didn't release the model to the public until June. Under a true continual learning regime, that kind of thing simply couldn't happen. You can't maintain a four-month gap between internal and external deployment and still remain competitive, because a competitor that releases an inferior model on launch day will have a smarter model based on real-world experience. ## Prediction 6: Continual learning will create a clear moat Continual learning will create a clear moat for leading AI labs that they currently lack. A lot of people have been asking: "How on earth do AI labs make money?" I've been asking that too. When I invited Dario on the podcast, I asked him that question, and he drew an analogy with cloud service providers. He pointed out: Look, cloud service providers offer a lot of undifferentiated services, and yet they still earn high profit margins. If you look at Amazon's or Google's quarterly earnings, they're doing very well. But the reason cloud profit margins are so high is that switching from one cloud to another is very time-consuming and expensive. Right now, there's nothing stopping me from starting a software repository with Codex, doing more work with Cursor, and finishing it with Claude Code. But once we have true continual learning, where the model you're using improves over sessions through interactions with you, there will be quite significant switching costs. If you want to switch the AI you're using, you're basically firing an employee who has accumulated months of background knowledge about your organization, bringing in a very new, very inexperienced intern, and having to retrain them from scratch. Once you have that lock-in, model providers can command quite substantial profit margins. Sorry. Just to emphasize: "new intern." ## Prediction 7: Enterprises will try to avoid lock-in, but labs have a carrot and a stick Enterprises will see through this dynamic. They'll try to avoid this lock-in. But what if the choice is: either you're locked into one model provider, or you lose this super valuable feature—your model improves for you over successive sessions? If real usage ends up being the primary way models improve, then AI labs might subsidize the users and companies that allow models to train on their sessions. If you look at the various offers given to new users of coding products, you'll see this is already happening. That's very similar to why Google gives away search for free. Conversely, labs might say that any enterprise that refuses to let them train on its sessions won't be able to use the best models. With both a carrot and a stick, labs can do a lot to get users to allow AI to learn from experience. Of course, I'm glossing over the fact that there's a difference between updating a single user's weight set and merging all the different weight branches back into the main model, and the latter may be more technically challenging. But in time, that problem will be solved too. ## Prediction 8: Economies of scale on the inference side will favor large organizations AI training already has significant economies of scale. You can spread all that expensive training across more users, and you can see evidence of that in the fact that lab revenues are growing much faster than their compute. But continual learning may also bring economies of scale on the inference side for end users, namely from batching. You may have seen the episode with Reiner Pope and me, where we discussed this in detail. But if each company's information requires a full weight update, rather than living in low-rank adapters, batching will provide enormous advantages. Rough calculations show that for a sparse model like DeepSeek v3, the optimal inference batch size is generating more than 2,400 concurrent sequences at the same time. If you don't do that, your compute isn't being fully utilized. If you want to understand why, I again highly recommend the Reiner episode on inference economics. But in any case, the point here is that a given set of weights can only be served efficiently when thousands of sequences are being decoded against them simultaneously. A large company with many employees and agents all doing different things can serve its weight branch very efficiently, while an individual user running a batch size of 1 could be more than two orders of magnitude less efficient in compute utilization. So the economics of serving personalized weights strongly favor large organizations. ## Conclusion Obviously, by the time continual learning actually arrives, there will be many more changes, and the most important ones may be those that are hardest to predict in advance. But the points above already seem clear even now. This is a spoken version of a blog post also published on my website. Check out dwarkesh.com. Otherwise, see you in the next podcast episode. Source: https://youtu.be/iewm45atodE