Vision as Unified Multimodal Generation

Hugging Face Daily Papers Papers
multimodal-generation computer-vision unified-model senseNova-vision instruction-response foundation-model

Summary

This paper presents SenseNova-Vision, a unified multimodal model that formulates computer vision tasks as generation problems, achieving performance comparable to specialized systems across diverse vision tasks. It introduces a large-scale instruction-response corpus and publicly releases the model and datasets.

We formulate computer vision as unified multimodal generation, where heterogeneous visual tasks are expressed in the native text and image generation spaces of a unified multimodal model, without task-specific architectures. Under this formulation, SenseNova-Vision uses natural-language instructions and optional visual prompts to specify tasks, target regions or views, and decoding conventions, and generates responses as text for symbolic outputs, images for dense spatial predictions, or mixed text-and-image outputs for compositional tasks. To support large-scale training, we convert diverse computer vision annotations into instruction-response examples compatible with these generation spaces, resulting in the SenseNova-Vision Corpus, a computer-vision instruction-response corpus spanning text, image, and mixed targets. Starting from an off-the-shelf pretrained unified multimodal model, SenseNova-Vision is trained primarily on this corpus, with auxiliary multimodal data used as a capability-preserving mixture, and requires no task-specific prediction heads or architectural modifications. The resulting model covers a broad range of vision tasks, including detection, OCR, keypoint estimation, segmentation, depth estimation, surface normal prediction, point maps, and camera pose estimation, while supporting language-defined variants that combine category, color, region, and other visual cues. Experiments show that a single unified model can match leading task-specialized systems across structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry. These results suggest unified multimodal generation as a scalable route for integrating computer vision capabilities into general-purpose foundation models. The model and corpus are publicly available.
Original Article
View Cached Full Text

Cached at: 07/08/26, 06:48 AM

Paper page - Vision as Unified Multimodal Generation

Source: https://huggingface.co/papers/2607.06560 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

A unified multimodal model formulates computer vision tasks as generation problems using natural language and visual prompts, achieving performance comparable to specialized systems across diverse vision tasks.

We formulate computer vision as unifiedmultimodal generation, where heterogeneous visual tasks are expressed in the native text and image generation spaces of aunified multimodal model, without task-specific architectures. Under this formulation, SenseNova-Vision uses natural-language instructions and optional visual prompts to specify tasks, target regions or views, and decoding conventions, and generates responses as text for symbolic outputs, images for dense spatial predictions, or mixed text-and-image outputs for compositional tasks. To support large-scale training, we convert diverse computer vision annotations intoinstruction-response examplescompatible with these generation spaces, resulting in theSenseNova-Vision Corpus, a computer-vision instruction-response corpus spanning text, image, and mixed targets. Starting from an off-the-shelfpretrained unified multimodal model, SenseNova-Vision is trained primarily on this corpus, with auxiliary multimodal data used as a capability-preserving mixture, and requires no task-specific prediction heads or architectural modifications. The resulting model covers a broad range of vision tasks, including detection, OCR, keypoint estimation,segmentation, depth estimation, surface normal prediction, point maps, and camera pose estimation, while supporting language-defined variants that combine category, color, region, and other visual cues. Experiments show that a single unified model can match leadingtask-specialized systemsacross structured visual understanding,dense geometric prediction,segmentation, andmulti-view visual geometry. These results suggest unifiedmultimodal generationas a scalable route for integrating computer vision capabilities into general-purpose foundation models. The model and corpus are publicly available.

View arXiv pageView PDFGitHub5Add to collection

Get this paper in your agent:

hf papers read 2607\.06560

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### sensenova/SenseNova-Vision-7B-MoT Depth Estimation• Updated11 minutes ago • 33 • 6

Datasets citing this paper2

#### sensenova/SenseNova-Vision-Corpus-50M Viewer• Updated8 minutes ago • 900 • 4 • 1 #### sensenova/SenseNova-Vision-Benchmark Preview• Updated11 minutes ago • 3 • 1

Spaces citing this paper1

Collections including this paper1

Similar Articles

sensenova/SenseNova-U1-8B-MoT

Hugging Face Models Trending

SenseNova U1 is a new series of native multimodal models that unify understanding and generation within a single architecture using the NEO-Unify framework, eliminating the need for separate visual encoders or VAEs.

Video Generation Models are General-Purpose Vision Learners

Hugging Face Daily Papers

This paper proposes that large-scale text-to-video generation can serve as a powerful pre-training paradigm for computer vision, introducing GenCeption which achieves state-of-the-art performance across diverse vision tasks with high data efficiency and emergent generalization to unseen domains.