Cross-modal Steering Vectors Exist Across Both Image and Text Generation

Hugging Face Daily Papers Papers
steering-vectors multimodal-models cross-modal-transfer interpretability image-generation text-generation

Summary

The paper shows that steering vectors derived in one modality (text or image) transfer zero-shot to the other in unified multimodal models, revealing shared causal control mechanisms despite a modality gap, along with a geometric signature where transferable steering vectors point in similar directions across modalities.

Are concepts and mechanisms unified across modalities in current AI models, or instead implemented separately for each one? We operationalize this broad question of modality unification by studying cross-modal steering: 1) finding vectors that successfully steer generation on tokens of one modality, which then 2) transfer zero-shot to steer generation when added to tokens of another modality. Our study goes beyond simple correlational alignment (e.g. representational similarity) because steering is fundamentally causal. We study cross-modal steering in several unified multimodal models (UMMs), since they possess shared parameters for language and image generation, focusing on these two modalities. Across diverse steered concepts such as emotion and object size, we show that steering vectors transfer zero-shot in both directions (text-to-image & image-to-text). Even for concepts that are highly unique to one modality (e.g. formal vs. colloquial language), steering vectors often induce a consistent, interpretable effect in the other modality, such as formal clothing and design. This transfer has a geometric signature too: despite a clear modality gap separating text and image representations, steering vectors with strong cross-modal transferability tend to point in similar directions for the same concept. Our findings shed light on how vision and language interact inside unified multimodal models and point towards more general, modality-agnostic approaches for controlling their behavior.
Original Article
View Cached Full Text

Cached at: 10/28/26, 05:29 AM

Paper page - Cross-modal Steering Vectors Exist Across Both Image and Text Generation

Source: https://huggingface.co/papers/2610.36269

Abstract

Cross-modal steering vectors transfer between text and image generation in unified multimodal models, revealing shared causal control mechanisms despite a modality gap.

Are concepts and mechanisms unified across modalities in current AI models, or instead implemented separately for each one? We operationalize this broad question of modality unification by studyingcross-modal steering: 1) finding vectors that successfully steer generation on tokens of one modality, which then 2) transfer zero-shot to steer generation when added to tokens of another modality. Our study goes beyond simple correlational alignment (e.g. representational similarity) because steering is fundamentally causal. We studycross-modal steeringin severalunified multimodal models(UMMs), since they possess shared parameters for language and image generation, focusing on these two modalities. Across diverse steered concepts such as emotion and object size, we show thatsteering vectorstransfer zero-shot in both directions (text-to-image & image-to-text). Even for concepts that are highly unique to one modality (e.g. formal vs. colloquial language),steering vectorsoften induce a consistent, interpretable effect in the other modality, such as formal clothing and design. This transfer has a geometric signature too: despite a clearmodality gapseparating text and image representations,steering vectorswith strong cross-modal transferability tend to point in similar directions for the same concept. Our findings shed light on how vision and language interact insideunified multimodal modelsand point towards more general, modality-agnostic approaches for controlling their behavior.

View arXiv pageView PDFProject pageGitHub0Add to collection

Get this paper in your agent:

hf papers read 2610\.36269

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.36269 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.36269 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.36269 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

On the Generalization of Steering Vectors for Chain-of-Thought Faithfulness

arXiv cs.AI

This paper investigates how well activation steering for improving chain-of-thought faithfulness generalizes across cue types, datasets, and steering vector construction methods across several Gemma and Qwen models. The authors find that when steering is effective, it generalizes broadly, and the evaluation setting matters more than the training setting or vector construction method.

Cross-Lingual Steering for Figurative Language Generation

arXiv cs.CL

This paper explores cross-lingual transfer of internal representations for figurative language generation in multilingual LLMs, showing that activation directions learned in one language can effectively steer generation in other languages.

Inverted Detection and Control in Steering Vectors

arXiv cs.LG

This paper identifies an 'inverted detection-control' phenomenon where some discriminative steering vectors, despite aligning with positive concept representations, consistently promote the opposite behavior. The authors propose a method to detect such inverted steering vectors without generation, enabling sign flips that improve a detection-based steering pipeline across multiple LLMs and concepts.