Let’s remove VAEs and ViTs from video models!

🚀 Introducing 𝗣𝗶𝘅𝗲𝗹𝗨𝗠𝗠: an encoder-free unified multimodal model for image and video understanding and generation, directly in pixel space.

Code and model available today!

🌐 https://nv-tlabs.github.io/PixelUMM 📄 https://arxiv.org/abs/2609.38597

Encoder-Free Unified Image and Video Understanding and Generation

Source: https://nv-tlabs.github.io/PixelUMM/

Overview

PixelUMM reads and writes raw pixels with a single decoder-only Transformer — no VAE and no vision encoder. Images become 16×16 patches and videos become 4-frame tubes; a Qwen3-8B backbone with separate understanding and generation experts shares one self-attention across text, clean pixels, and noisy pixels.

Architecture overview. The answer text shown for the understanding branch is illustrative. Videos could not be loaded. Please reload the page.

Empirical findings

Eight experiment families · F1–F8

The ablations behind PixelUMM are grouped into eight experiment families, F1–F8, with configurations numbered R01, R02, and so on. We start with patch artifacts, the most visible failure mode of a linear pixel head.

F3### Patch artifacts

With the linear pixel decoder, generated images and videos can show faint grid-aligned intensity steps in smooth, low-texture regions such as sky. As also observed in MiniT2I, these patch artifacts become more pronounced at high classifier-free guidance, around CFG 6.

Linear pixel heads (F3-R01). Left: a generated image above a crop of its smooth sky. Right: frame 0 of theUpright pianovideo above a crop of the smooth wall behind the pianist. Open the figure to view it at full resolution.Note thatthe released checkpoint, the benchmarked model and all online demos, including every video on this page, are generated with the default linear output heads, because switching from a linear to a convolutional head requires further training.

Three convolutional video heads

Each head replacesvideo\_linear\_outproj, is initialized from Gen Stage 1, and maps the Transformer’sT/4 × H/16 × W/16hidden grid back toT × H × WRGB video.

RunOutput headGFLOPs*ParamsF3-R01Linear (baseline)13312.6MF3-R02Wan-style upsample + conv2,274(17.1×)18.3M(1.5×)F3-R03PixelShuffle, temporal-first (T-S)1,279(9.6×)52.9M(4.2×)F3-R04PixelShuffle, spatial-first (S-T)747(5.6×)15.1M(1.2×)*Convolution-only forward cost for one 96×176×320 clip.

The three convolutional output heads. T-Conv and S-Conv denote 3×1×1 temporal and 1×3×3 spatial convolutions.#### What the heads trade off

• **Training loss.**Both PixelShuffle heads converge to a T2V pixel-flow loss near 0.02, while the Wan-style head stays above 0.05.
• **Boundary probe.**Across 72 evaluation prompts, measured on pre-clamp float32 outputs, the linear head shows the largest elevations at 16-pixel patch boundaries (1.078) and 4-frame tubelet boundaries (1.160). The Wan-style head sits near the normalized baseline; the PixelShuffle heads leave smaller residual elevations.
• **Generated video.**The Wan-style head gives the cleanest boundaries but blurrier outputs. PixelShuffle offers the better trade-off, with temporal-first (F3-R03) slightly ahead of spatial-first (F3-R04).

Same prompt, one row per output head. The red box marks a fixed crop, shown at frames 0, 32, 64 and 95 of the 96-frame video (lossless frames captured before video encoding).TakeawayConvolutional heads reduce patch artifacts. PixelShuffle offers a better trade-off between training loss and artifact suppression than the upsampling-and-convolution head.

Note thatthe released checkpoint, the benchmarked model and all online demos, including every video on this page, are generated with the default linear output heads, because switching from a linear to a convolutional head requires further training.

F1### Image patch size

16×16 (F1-R01) and 32×32 (F1-R02) patches differ only in the generation input projection and output head. With the same sequence-length budget, 32×32 fits four times as many images per step, yet 16×16 keeps a lower T2I training loss throughout late training.

T2I pixel-flow MSE: full trajectory (left) and 10K–15K zoom-in (right).TakeawayStronger spatial compression makes image generation harder to learn.

F2### Video patch size

Four tubelet-to-token mappings: p32/t4, p32/t2, p16/t4 and p32/t1 (spatial patch / frames per token). Less aggressive spatiotemporal compression generally gives lower T2V loss, with p32/t1 the lowest. The final model uses p16/t4 to match the compression of common video VAEs such as Wan2.2.

T2V MSE over 0–68K steps (left) and 40K–68K (right).TakeawayCompression ranges from 1,024 to 4,096 pixels per video token; stronger compression makes video generation harder to learn.

F4### Pixel-space vs. VAE-space training

A 32×32 pixel-patch run with x-prediction (F4-R01) against a frozen Wan2.2 VAE with 2×2 latent patchification and v-prediction (F4-R02), at the same effective token stride. At 10K steps the VAE-space loss is about 4.3× the pixel-space loss, but because the two losses live in different spaces, this does not show that pixel space learns faster.

Left: T2I velocity MSE in pixel and VAE space. Right: pre-clip global gradient norm.TakeawayGradient norms are nearly equal; pixel-space training shows occasional loss spikes.

F5### Model size

1.7B (F5-R01) and 8B (F5-R02) models with the same recipe and a global batch size of 256. The 8B model reaches MSE 0.060 at about 14K instead of 40K steps, and CE 0.50 at about 10K instead of 31K steps.

Joint Stage 1 MSE (left) and CE (right).TakeawayThe 8B model reaches comparable generation and text losses in roughly one third of the training steps.

F6### Compute scaling

The same 1.7B model and matched recipe on 8 GPUs (F6-R01) versus 128 GPUs (F6-R02). With more GPUs, CE reaches 1.0 at about 2K instead of 17.5K steps (≈9× fewer), while MSE reaches 0.065 at about 17K instead of 23K steps (≈1.3× fewer). These are steps to a fixed loss, not wall-clock speedups.

MSE (left) and CE (right); opaque curves are EMAs of the raw measurements.TakeawayMore GPUs speed up text (CE) convergence far more than generation (MSE) convergence.

F7### Multimodal context conditioning

Clean conditions (text, reference images, an input video) enter the understanding expert, noisy target tokens enter the generation expert, and both meet in shared self-attention. This extends SenseNova-U1’s encoder-free image conditioning to image-to-video and video editing, with no separate VAE-token stream. After 15K steps of multi-task fine-tuning on eight tasks, image understanding moves both ways (BLINK +2.37, CV-Bench +2.27, SEED-I −5.41), while all four video benchmarks improve by 0.89–3.49 points.

Interleaved visual-context conditioning.Image-to-video: the clean first frame, then frames 0, 47 and 95 of the generated clip.TakeawayClean visual conditions can be routed through the understanding expert without systematically degrading understanding.

F8### Video understanding interfaces

dense\_modesamples at 4 FPS and forms 4-frame tubelets;sparse\_modesamples at 1 FPS and projects frames independently. Both use the same 448² per-frame budget. At F8-R01 (Und Stage 3, 9K steps):

InterfaceMVBenchVideo-MMELongVideoBenchLVBenchsparse\_mode71.1557.8959.3941.38dense\_mode71.2257.6758.7141.38Δ+0.07−0.22−0.680.00 Takeaway4-FPS tubelet input performs comparably to 1-FPS frame input.

Understanding benchmarks

The released PixelUMM checkpoint, compared with unified models and vision-language models (VLMs). Its overall performance is comparable to the baselines. Since training data differ across models, these results cannot establish which architecture is superior or which model converges faster. Dark and light green mark the best and second-best result in each row.

Image understanding

Unified modelsVLMsBenchmarkPixelUMM8B MoTBAGEL7B MoTTUNA7B+5BTUNA-27B+5BQwen2.5-VL7BLLaVA-OV-1.58BLLaVA-OV-28BQwen3-VL-Inst.8BNEO-ov8BMMMU41.6755.3049.8050.7051.3055.40–69.6068.10MMStar53.99–61.20–62.5067.7064.3070.9067.30RWQA71.6372.8066.1067.7068.5068.1069.7071.5067.80SEED-I70.39–74.70–77.5077.30––76.60AI2D80.1289.2079.3079.6082.6084.2084.3085.7085.40DocVQA90.42–––94.9095.0095.2096.1091.90ChartQA82.9678.5085.8085.6084.1086.5085.9089.6086.20InfoVQA63.67–––81.7078.4074.4083.10–TextVQA78.76–––84.90–––78.50OCRBench78.0073.3074.3079.7084.2082.9078.2089.6081.60MME1809.542388.00––2347.00––––GQA61.9066.4063.9065.0060.70––––MMVP80.3385.0070.7077.3078.00––––SEED2+64.5171.9052.7061.1070.9069.20–––CV-Bench77.60–––80.0080.70–––CountBench94.3082.5073.5081.7086.4088.2089.0089.80–PixMo-Count73.03–––63.3062.2064.0062.40–V*76.9670.2052.4059.2077.0078.0085.9085.30–MMMU-Pro27.63–––36.3037.40–––BLINK53.46–––56.4048.3063.5069.1062.80MuirBench38.31–––59.60––64.4058.20 PixelUMM is evaluated with the official LMMS-Eval protocol (64,750 generations over 21 tasks).

Video understanding

Unified modelsVLMsBenchmarkPixelUMM8B MoTShow-o21.5B+0.5BTUNA1.5B+?Lance3B MoTLLaVA-OV-28BQwen3-VL8BKeye-VL-1.58BInternVL-3.58BPLM8BLLaVA-OV-1.58BMVBench70.5349.8054.4062.0066.2069.0056.9072.1077.1051.20Video-MME (w/o sub.)57.3348.0049.10–71.9071.4073.0065.9060.5061.10LongVideoBench59.6149.2049.70–66.9068.0066.0062.4059.6056.20LVBench40.41–27.40–55.5058.0042.8046.7044.5040.10 PixelUMM is evaluated insparse\_modewith up to 96 sampled frames. Video-MME is evaluated without subtitles. Published models keep their original evaluation protocols.

Limitations

PixelUMM has weaknesses. Generating directly in pixel space removes the VAE, but PixelUMM still shares the typical failure modes of latent video diffusion models.

• **Many similar entities.**When several animals or people overlap, their bodies can merge, split, or appear out of nowhere, so their number drifts over the clip.
• **Hands and fine anatomy.**Hands can have the wrong number of fingers, and small limbs can blur into each other.
• **Physics and interactions.**Motion can be physically implausible, rigid objects can bend or morph, and interactions between objects or characters may not have the expected effect.