@rohanpaul_ai: 新微软论文认为，Transformer在学会紧凑内部状态时泛化更好，而不仅仅是预测下一个token……

New Microsoft paper argues that transformers generalize better when they learn compact internal states, not just next tokens.

微软新论文指出，Transformer在学会紧凑内部状态时泛化能力更强，而不仅仅是预测下一个词元。

The problem is that normal transformers can look back at every earlier token, so they do not have to squeeze the past into a clean summary. token prediction alone can reward shortcuts that do not become coherent world models.

问题在于，普通Transformer可以回顾每一个之前的词元，因此它们无需将过去压缩成简洁的摘要。仅靠词元预测可能会奖励那些不会形成连贯世界模型的捷径。

That can work beautifully on familiar data and still fail when the model has to plan, detour, reason, or carry a hidden structure forward.

这在熟悉的数据上效果很好，但当模型需要规划、绕路、推理或携带隐藏结构时仍可能失败。

NextLat fixes this by adding a training task where the model must predict its next hidden state, not just the next word.

NextLat通过添加一个训练任务来解决这一问题：模型必须预测其下一个隐藏状态，而不仅仅是下一个词。

A hidden state is the model’s private summary of what it has seen, so predicting the next one pushes the model to learn how situations change over time.

隐藏状态是模型对所见内容的私有摘要，因此预测下一个隐藏状态会促使模型学习情境如何随时间变化。

The authors tested this on map-like world modeling, math reasoning, graph planning, story prediction, and regular language modeling.

作者在类似地图的世界建模、数学推理、图规划、故事预测和常规语言建模上进行了测试。

The main result is that NextLat often learned more compact and useful internal states, solved planning tasks better, and sped up generation by up to 3.3x.

主要结果是，NextLat通常能学到更紧凑且有用的内部状态，更好地解决规划任务，并将生成速度提升高达3.3倍。

Overall, it gives transformers some of the useful memory behavior of recurrent models without changing the transformer architecture or slowing normal inference.

总体而言，它在不改变Transformer架构或降低正常推理速度的情况下，为Transformer提供了类似循环模型的一些有用记忆行为。

Link – arxiv. org/abs/2511.05963

Title: “Next-Latent Prediction Transformers Learn Compact World Models”

Jayden Teoh (@jayden_teoh_): Next-token prediction is myopic. What if transformers learn to predict their own next latent state?

🌠 We present Next-Latent Prediction (NextLat): a self-supervised learning method that teaches transformers to form compact world models for reasoning