@ziv_ravid: 1/ On Training in Imagination - Dwarkesh's episode has a segment on dreaming as one of the next training paradigms. The…
Summary
A tweet thread discussing Dwarkesh Patel's podcast episode on 'dreaming' as a next training paradigm for AI models, linking to a recent paper on this topic.
View Cached Full Text
Cached at: 07/03/26, 08:34 AM
1/ On Training in Imagination -
Dwarkesh’s episode has a segment on dreaming as one of the next training paradigms. The idea is that a model learns mostly inside its own, by imagining what would happen, instead of trying out for real.
We have a recent paper on exactly this
2/ The question is simple. If a model practices inside a world it imagined rather than the real one, how much can you trust what it picks up?
Two things can be off. Its sense of how the world works, and its sense of what counts as a good outcome.
3/ We pull the two apart and measure each one on its own.
The how the world works part is the risky one. Small mistakes there pile up the longer the model imagines ahead, so a long daydream slowly drifts away from reality.
4/ That points to a fix. If the imagined world changes smoothly and predictably from one step to the next, the mistakes stay small and the whole thing stays trustworthy for longer.
This lines up nicely with how modern world models are already built.
5/ Then the money question. You have a fixed budget and two things to spend it on. Teaching the model how the world moves is cheap. Teaching it what is good, which usually needs human feedback, is expensive.
We work out the best way to divide the budget between the two.
6/ The results are quite surprising. The what is good part is learned much faster than the how the world works part, roughly nine times faster as you add more data.
So you only need to label a small slice of examples with feedback, and can spend the rest on cheap practice.
7/ We also showed that even if the feedback is noisy, as long as the mistakes are random they cancel out once you collect enough tries. So cheap and slightly sloppy feedback is usually fine.
The real danger is feedback that leans in one direction. That never washes out :(
8/ Dreaming is a nice framing. Our paper says when it holds up and when it breaks.
With @NadavTimor, @micahgoldblum , @ylecun, and David Harel.
Paper: arxiv: 2605.06732
Similar Articles
@dwarkesh_sp: What does the next training paradigm look like? 0:00:00 – The big research bet the labs are making 0:02:12 – Grindabili…
A discussion on the next training paradigm for AI, covering research bets, grindability, RLVR, and a vision for 2027.
@omarsar0: By the way, a highly recommended talk by Anthropic on Memory and the new Dream feature. Lots of cool ideas there about …
A recommendation of Anthropic's talk on Memory and the new Dream feature, discussing future memory in AI agents.
On Training in Imagination
This paper analyzes the 'training in imagination' paradigm in model-based reinforcement learning, deriving optimal sample allocation strategies and characterizing how dynamics and reward model errors affect policy returns.
Sleep for Continual Learning (24 minute read)
Google researchers propose a 'Sleep' paradigm for continual learning that consolidates short-term in-context knowledge into long-term model parameters via distillation and replay. A 'Dreaming' stage uses reinforcement learning to generate synthetic curricula for self-improvement.
The Next Paradigm (7 minute read)
The article argues that training AI on millions of verifiable tasks across diverse RL environments could lead to AGI, and that scaling may overcome current limitations like sample inefficiency. It also examines why progress on computer use has been slower due to lack of grindable environments.