Tag
This paper introduces Visualized Task Semantics (VTS), a controlled intervention to study when multimodal LLMs are asked questions in the image rather than text. It reveals a consistent accuracy drop across all tested models and tasks, and proposes prompt-region grounding to recover the clean task representation, improving VTS accuracy from 58.0 to 66.3.