@VictoriaLinML: The video of my Stanford CS25 guest lecture, From Language Models to Native Multimodal Intelligence, is now online. I d…
Summary
Victoria Lin's Stanford CS25 lecture discusses the paradigm shift from language models to native multimodal intelligence, covering tokenization approaches and comparing models like Chameleon and Transfusion.
View Cached Full Text
Cached at: 07/03/26, 08:43 PM
The video of my Stanford CS25 guest lecture, From Language Models to Native Multimodal Intelligence, is now online.
I discussed how the core ideas behind LLMs have shaped multimodal AI, from architectures to training paradigms and scaling, and where the next challenges may lie.
TL;DR: Victoria Lin systematically introduced the paradigm of native multimodal language models in her Stanford CS25 lecture—unified tokenization converts text, images, audio, and video into sequences, then autoregressive modeling is applied just like in language models. She focused on the design philosophies and trade-offs of full-modality models Chameleon (discrete tokens) and Transfusion (continuous representations + diffusion).
Lecture Background
Victoria Lin is from Thinking Machines Lab, specializing in native multimodal intelligence. She previously worked as a Research Scientist at Meta AI and Salesforce AI Research, and holds a PhD from the University of Washington. This lecture is entirely based on publicly available materials and reflects only her personal views.
Why Do We Need Multimodal Language Models?
Large Language Models (LLMs) trained on massive text data by predicting the next token exhibit emergent capabilities such as knowledge acquisition, instruction following, chain-of-thought reasoning, and even complex task planning. But relying solely on text is insufficient—the digital and physical world we encounter daily is filled with images, audio, video, and other multimodal information. Building AI systems that can interact with humans in real time requires seamless handling of visual and auditory information.
Many state-of-the-art models (e.g., Gemini, Qwen, Kimi) already claim to be “natively multimodal,” capable of tasks like image and video understanding. The first part of this lecture dives into the design philosophy behind such models.
Definition and Paradigm of Native Multimodal Language Models
The most straightforward way to understand this paradigm is by analogy with language models: state-of-the-art multimodal language models fully embrace tokenization—regardless of whether the input is an image, video, or audio, it is converted into a sequence of tokens that can be processed by a Transformer, and then global autoregressive generation modeling is performed exactly like in language models.
Tokenization Methods for Different Modalities
- Text: Split into small token chunks using Byte-Pair Encoding (BPE).
- Image: Perform “patchify,” dividing the entire image into fixed-size patches (e.g., 16×16 pixels), then obtain vector representations via an encoder, and serialize them into image tokens.
- Audio: Transform the waveform and tokenize the resulting representation.
- Video: Treat video as a sequence of images, patchify each frame, and concatenate all patches to obtain a token sequence.
These tokens do not have to be discrete; encoding them into dense vectors is still referred to as tokenization.
Two Main Types
- Multimodal Input + Text Output: Input a multimodal sequence, but compute loss only on text tokens. Examples include Gemini, Qwen, Kimi, etc.—they understand images/video and answer in text. Many core products today still produce only text output.
- Full-Modality Models (Omni): Both input and output are multimodal—they can generate not only text but also images and audio. Representative: GPT-4o.
Benefits of Transferring the LLM Paradigm
Through the lens of tokenization, LLM architectural and training principles can be transferred to the multimodal domain:
- Prompting & Instruction Following: Prompt the model with mixed modalities to solve complex tasks that require multimodal capabilities.
- Planning & Reasoning: The model can leverage multimodal information for planning and reasoning.
- Scaling Laws: The pattern of performance improvement with larger data and model size seen in LLMs also applies to multimodal models. Although exact scaling laws for multimodality are not yet clear (text modeling has power-law fits), the rough trend still shows scalability.
- Architecture Transfer: Many scaling techniques (e.g., Mixture-of-Experts and its variants) can be directly applied to multimodal language models.
Spotlight on Full-Modality Models
This lecture focuses on the second category—full-modality models—because Victoria Lin’s research group has been working on developing such models. Key research questions: (1) How can we enable models to generate non-text modalities? (2) Is there transfer between understanding and generation of non-text modalities?
Chameleon Family: Discrete Tokenization
Chameleon adopts a simple assumption: tokenize every modality into discrete tokens, and then model the multimodal world as a sequence of discrete tokens, just like a language model.
How to Discretize Images: After patchifying the image, use a learned vector codebook (VQ-VAE technique) to match each image patch embedding to the nearest discrete index in the codebook, thereby converting the image into a sequence of discrete representations.
Training & Capabilities: Train with a cross-entropy language model objective on interleaved image and text sequences. The model can generate interleaved text and images in their original order, producing mixed-modality documents. It can also handle multiple tasks such as chat, brainstorming, and comparing images.
Limitations:
- The discretization causes information loss in image understanding tasks, resulting in a significant performance gap compared to models using continuous image encoders (e.g., SigLIP).
- In generation, discrete tokens are inefficient; they require large amounts of data to sample well-shaped images.
Transfusion: Continuous Representations + Diffusion
Transfusion was proposed around the same time as Chameleon and attempts to overcome its limitations. It takes continuous representations of images as input and directly predicts these continuous representations.
Method: Combine autoregressive language modeling with diffusion-based image generation within a single Transformer. Input is still interleaved text and image sequences. Standard autoregressive modeling is performed on text; for the image portion, instead of autoregressive next-token prediction, a segment of image representations is taken and processed with diffusion (multiple diffusion steps to produce a clear image), and then the clear image is fed back as input to continue autoregressive processing.
Structural Changes: Text uses causal attention, while images use bidirectional attention for better performance.
Advantages: According to the paper, Transfusion can generate images of better quality with fewer steps than discrete token methods.
Summary
The transition from language models to native multimodal intelligence centers on incorporating multimodal information into an autoregressive framework through unified tokenization. Chameleon demonstrates the feasibility of training interleaved sequences from scratch, but discretization causes information loss; Transfusion combines continuous representations and diffusion, achieving better generation efficiency and quality. Future topics worth exploring include multimodal scaling laws and the transfer between understanding and generation of non-text modalities.
Source: https://www.youtube.com/watch?v=NDdc39KYqDU
Similar Articles
@swyx: full writeup and links here
A Latent Space podcast episode discusses the thesis that video models derive intelligence from LLMs, and that the next frontier is video agents. Guest Ethan He, who built Grok Imagine at xAI, shares insights on building frontier image and video systems.
From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models
This survey paper systematically reviews the paradigm evolution of unified vision-language perception in multimodal large language models (MLLMs), proposing a five-stage taxonomy and identifying open challenges toward general multimodal intelligence.
Vokenization: Multimodel Learning for Vision and Language
The article explains 'Vokenization,' a multimodal learning technique that bridges computer vision and natural language processing by using weak supervision to link visual data with language tokens. It contrasts this approach with text-only models like GPT-3 and BERT, highlighting how visual grounding can improve language understanding.
@seclink: https://x.com/seclink/status/2067968283492712846
This article, based on the sharing of researcher Victoria Lin, systematically reviews the mainstream technical approaches of native multimodal large models (Chameleon, Transfusion, MOT) and their pros and cons. It points out that multimodal AI is still in the early exploration stage, with open problems such as gaps in scaling laws, inconsistency between image understanding and generation encoding, and connection with the physical world.
VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding
VideoChat3 is a fully open, efficient, and generalist video-centric multimodal large language model that introduces Inflated 3D Vision Transformer (I3D-ViT) and Adaptive Frame Resolution for streaming video perception, along with scalable video data synthesis pipelines, achieving superior performance with only 4B parameters.