Tag
NVIDIA Labs released PixelUMM, an encoder-free unified multimodal model that operates directly in raw pixel space for both image and video understanding and generation, along with open-sourced code and models.
NVIDIA introduces PixelUMM, an encoder-free unified multimodal model that reads and writes raw pixels with a single decoder-only Transformer (Qwen3-8B backbone), removing VAEs and ViT encoders from image and video understanding and generation. Code and model checkpoints are released alongside detailed ablations on patch artifacts and decoder head design.
PixelUMM is an encoder-free unified multimodal model that operates directly in pixel space, representing images as spatial patches and videos as spatiotemporal tubelets fed to a shared backbone via single-layer linear projections. Its Mixture-of-Transformers architecture supports both autoregressive text prediction and pixel-space flow matching for image and video understanding and generation, achieving competitive performance while providing design insights for future pixel-space UMMs.
This paper systematically compares scaling laws for encoder-free and encoder-based multimodal models, revealing that encoder-free architectures underperform at small scales but catch up at larger compute budgets, with the visual encoder's advantage diminishing as scale increases.
SenseNova-U1.5 is an 8B native unified multimodal model that performs visual understanding, reasoning, and generation without encoders or VAEs, achieving high fidelity and instruction following through patch reconstruction, curated data, and expert optimization.
Thinking Machines Lab releases Inkling, a 975B-parameter open-weights multimodal MoE model with 41B active parameters and a controllable thinking effort feature. It uses an encoder-free approach for multimodality and achieves state-of-the-art on several benchmarks.
Researchers trained a vision-language model without a vision encoder for only $100, inspired by Gemma 4 12B, achieving a 30% reduction in end-to-end latency on an M3 Pro MacBook.
Discusses leveraging Gemma 4 12B's encoder-free architecture for native voice input, seeking out-of-the-box solutions for low-latency streaming audio ingestion.
Google DeepMind announces Gemma 4 12B, a novel encoder-free multimodal AI model that integrates vision and audio directly into the LLM backbone, delivering advanced reasoning and agentic capabilities on laptops with 16GB of RAM, released under Apache 2.0 license.
Google's Gemma 4 12B introduces an encoder-free multimodal architecture that competes with larger models, though benchmark comparisons show it trailing Qwen 2.5 9B on most tasks. The article also covers related developments including open-weight model security risks, Uber's Claude Code spending caps, and NeurIPS's misuse of an uncalibrated AI detector.
Google's new Gemma 4 12B is a single decoder-only transformer with encoder-free multimodal input, achieving strong benchmarks while being small enough to run locally on a budget GPU. It is released under Apache 2.0 license.
Google DeepMind researcher announces the release of Gemma 4 12B, a dense encoder-free model that processes text, image, and audio inputs, continuing work on unifying models across modalities.
Google's Gemma 4 12B model enables local multimodal AI using an encoder-free architecture.
Google launches Gemma 4 12B, an encoder-free multimodal model with native audio support, optimized for local execution on laptops under Apache 2.0.