@CongWei1230: Thanks for sharing! We open-sourced code and models. Huggingface Paper: https://huggingface.co/papers/2609.38597… Code:…
Summary
NVIDIA Labs released PixelUMM, an encoder-free unified multimodal model that operates directly in raw pixel space for both image and video understanding and generation, along with open-sourced code and models.
View Cached Full Text
Cached at: 10/05/26, 11:24 AM
Thanks for sharing! We open-sourced code and models. Huggingface Paper: https://huggingface.co/papers/2609.38597… Code: https://github.com/nv-tlabs/PixelUMM… Model: https://huggingface.co/nvidia/PixelUMM
Paper page - PixelUMM: Encoder-Free Unified Image and Video Understanding and Generation
Source: https://huggingface.co/papers/2609.38597 Authors:
,
,
,
,
,
,
,
,
,
Abstract
UnifiedMultimodalModels(UMMs)oftenrelyonseparatevisualrepresentationsforunderstandingandgeneration,increasingvisualcontextlengthandcomplicatingintegrationwithestablishedvision-languagepretrainingpipelines.Recentadvancesinpixel-spacemodelingofferanencoder-freealternative,butextendingthisparadigmfromimagestovideosisnon-trivial:videounderstandingandgenerationadoptdifferenttemporalrepresentations,leavingthedesignofaunifiedvisualinterfaceanopenquestion.WepresentPixelUMM,anencoder-freemodelforunifiedimageandvideounderstandingandgenerationdirectlyinpixelspace.PixelUMMrepresentsimagesasspatialpatchesandvideosasspatiotemporaltubelets,connectingrawpixelstoasharedmultimodalbackbonethroughsingle-layerlinearprojections.ItsMixture-of-Transformersarchitecturecombinessharedattentionwithtask-specificparametersandextendsclean-pixelpredictiontovideogeneration,jointlysupportingautoregressivetextpredictionandpixel-spaceflowmatching.ExperimentsshowthatPixelUMMachievescompetitiveperformanceacrossimageandvideounderstandingandgenerationtasks.Wefurtherconductempiricalstudiesofkeydesignchoices,includingdecoderdesignandspatial-temporalpatchsize,providinginsightsforfuturepixel-spaceunifiedmultimodalmodels.
View arXiv pageView PDFProject pageGitHub122Add to collection
Get this paper in your agent:
hf papers read 2609\.38597
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2609.38597 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.38597 in a dataset README.md to link it from this page.
Spaces citing this paper2
Collections including this paper3
Robots Digest 🤖 (@robotsdigest): NVIDIA researchers just released PixelUMM, a unified multimodal model that fundamentally changes how models process visual data. Groundbreaking highlights: Entirely removes VAEs and ViTs from the architecture Operates directly in raw pixel space for both image and video tasks
Similar Articles
@CongWei1230: Let's remove VAEs and ViTs from video models! Introducing 𝗣𝗶𝘅𝗲𝗹𝗨𝗠𝗠: an encoder-free unified multimodal model fo…
NVIDIA introduces PixelUMM, an encoder-free unified multimodal model that reads and writes raw pixels with a single decoder-only Transformer (Qwen3-8B backbone), removing VAEs and ViT encoders from image and video understanding and generation. Code and model checkpoints are released alongside detailed ablations on patch artifacts and decoder head design.
PixelUMM: Encoder-Free Unified Image and Video Understanding and Generation
PixelUMM is an encoder-free unified multimodal model that operates directly in pixel space, representing images as spatial patches and videos as spatiotemporal tubelets fed to a shared backbone via single-layer linear projections. Its Mixture-of-Transformers architecture supports both autoregressive text prediction and pixel-space flow matching for image and video understanding and generation, achieving competitive performance while providing design insights for future pixel-space UMMs.
@omarsar0: Banger compression paper from NVIDIA. (bookmark it) Bigger MoE models keep winning on quality, but serving them at inte…
NVIDIA's paper introduces Puzzle-75B-A9B, a compressed hybrid MoE model that doubles server throughput while preserving quality, enabling cost-effective deployment of large language models.
@lqiao: Congrats @huggingface and @NVIDIAAI for this merge to strengthen open model ecosystem!
The article congratulates Hugging Face and NVIDIA on their merge to strengthen the open model ecosystem, highlighting how open models enhance AI safety, cybersecurity, and innovation.
@HuggingPapers: Microsoft just released Lens on Hugging Face A 3.8B parameter text-to-image model delivering efficient training and hig…
Microsoft released Lens, a 3.8B parameter text-to-image model on Hugging Face, capable of efficient training and high-resolution generation up to 1440×1440.