interleaved-modalities

WM-VLM 论文为预训练 VLM 增加轻量级世界模型分支，通过两阶段训练生成中间视觉状态以支持语言与视觉双通道推理，在 2D/3D 心智旋转任务上较监督微调基线最高提升 39.25 个百分点。

This paper reveals that interleaved speech-text language models implicitly transcribe speech into text in intermediate layers, then predict in text space before converting back to speech, shedding light on internal modality interaction.