From RGB Generation to Dense Field Readout: Pixel-Space Dense Prediction with Text-to-Image Models

Hugging Face Daily Papers Papers
diffusion-transformers dense-prediction text-to-image rechannel flux task-lora state-of-the-art

Summary

ReChannel adapts pretrained diffusion transformers (e.g., FLUX-Klein) for dense prediction tasks by mapping tokens directly to pixel-space patches, achieving state-of-the-art results on trimap-free matting, KITTI depth, and referring segmentation with minimal additional parameters.

Large-scale text-to-image models are attractive backbones for dense prediction because RGB generation pretraining learns rich semantic, structural, and geometric priors. Existing generative and editing approaches reuse these priors by casting dense prediction as target generation: annotations such as depth, normals, alpha mattes, masks, and heatmaps are encoded into an RGB-trained VAE latent space and decoded back as image-like targets. We argue this inherits more of the generative output interface than dense prediction requires: unlike RGB synthesis, dense prediction asks for pixel-correct, task-native fields on the same image plane, not new RGB content to be rendered. Our key observation is that a pretrained DiT already organizes RGB inputs through a patch-to-token-to-patch lattice on the image plane, so each token indexes a fixed output patch whose channels can carry task-native quantities instead of RGB appearance. We instantiate this as ReChannel: we keep the VAE encoder for the DiT's input distribution but drop the target-side decoder, adapt the frozen DiT with task LoRA, and map each token to its p x p x K_t pixel-space patch through a shared token-local linear head--about 33K parameters, no spatial mixing. Using FLUX-Klein, we evaluate on six dense prediction tasks and over a dozen benchmarks. This minimal interface sets new state-of-the-art on trimap-free matting, KITTI depth, and referring segmentation, and stays competitive on normals, saliency, and pose. In a matched 4B setting it is more accurate and 2.48x faster than an edit-plus-latent-decode counterpart--dense perception can benefit from generative pretraining without inheriting its output interface.
Original Article
View Cached Full Text

Cached at: 07/13/26, 07:49 AM

Paper page - From RGB Generation to Dense Field Readout: Pixel-Space Dense Prediction with Text-to-Image Models

Source: https://huggingface.co/papers/2607.06553

Abstract

Pretrained diffusion transformers can be adapted for dense prediction tasks by mapping tokens to task-native outputs instead of generating RGB images, achieving state-of-the-art results with minimal additional parameters.

Large-scaletext-to-image modelsare attractive backbones fordense predictionbecause RGB generation pretraining learns rich semantic, structural, and geometric priors. Existing generative and editing approaches reuse these priors by castingdense predictionas target generation: annotations such as depth,normals, alpha mattes, masks, and heatmaps are encoded into an RGB-trainedVAE latent spaceand decoded back as image-like targets. We argue this inherits more of the generative output interface thandense predictionrequires: unlike RGB synthesis,dense predictionasks for pixel-correct, task-native fields on the same image plane, not new RGB content to be rendered. Our key observation is that a pretrainedDiTalready organizes RGB inputs through a patch-to-token-to-patch lattice on the image plane, so each token indexes a fixed output patch whose channels can carry task-native quantities instead of RGB appearance. We instantiate this as ReChannel: we keep the VAE encoder for theDiT’s input distribution but drop the target-side decoder, adapt the frozenDiTwithtask LoRA, and map each token to its p x p x K_t pixel-space patch through a sharedtoken-local linear head--about 33K parameters, no spatial mixing. UsingFLUX-Klein, we evaluate on sixdense predictiontasks and over a dozen benchmarks. This minimal interface sets new state-of-the-art ontrimap-free matting,KITTI depth, andreferring segmentation, and stays competitive onnormals,saliency, andpose. In a matched 4B setting it is more accurate and 2.48x faster than an edit-plus-latent-decode counterpart--dense perception can benefit from generative pretraining without inheriting its output interface.

View arXiv pageView PDFGitHub9Add to collection

Get this paper in your agent:

hf papers read 2607\.06553

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.06553 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.06553 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.06553 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Representations Before Pixels: Semantics-Guided Hierarchical Video Prediction

Hugging Face Daily Papers

Re2Pix is a hierarchical video prediction framework that improves future video generation by first predicting semantic representations using frozen vision foundation models, then conditioning a latent diffusion model on these predictions to generate photorealistic frames. The approach addresses train-test mismatches through nested dropout and mixed supervision strategies, achieving improved temporal semantic consistency and perceptual quality on autonomous driving benchmarks.

L2P: Unlocking Latent Potential for Pixel Generation

Hugging Face Daily Papers

The L2P paper introduces a Latent-to-Pixel transfer paradigm that leverages pre-trained latent diffusion models to create efficient pixel-space models capable of 4K generation with minimal training overhead.