Lens: Bringing the Right Semantic Perspective into Focus for Training-Free Multimodal Representation Learning

# Lens: Bringing the Right Semantic Perspective into Focus for Training-Free Multimodal Representation Learning
Source: [https://arxiv.org/html/2609.20252](https://arxiv.org/html/2609.20252)
Xinran Liu, Shouqian Shi†, Yixian Chen, Ruizhi Chen,Xin\-Wei Yao, Sheng Zhong†Corresponding author:sqlite@nju\.edu\.cn

###### Abstract

High\-quality representations are essential for a wide range of downstream tasks\. Dedicated embedding models are explicitly optimized for representation learning, yet their training data are often more limited in scale and diversity than the massive corpora used to pretrain modern large language models and multimodal large language models\. Large\-scale pretraining and instruction following enable autoregressive models to select relevant evidence, integrate multimodal information, and infer semantics under different task perspectives, creating a distinctive opportunity for training\-free representation learning\. However, our analysis reveals that existing semantic\-elicitation methods do not reliably orient the extracted states toward the semantic perspective required by the downstream task\. Consequently, the resulting representations often remain dominated by salient input content\. We characterize this problem as*semantic perspective misalignment*and proposeLens, a training\-free framework that makes representation readout task\-directed\.*Semantic Perspective Anchoring*associates the task\-required perspective with a task\-specific readout phrase, specifying the interpretive role of the positions later used for extraction\.*Contextualized Phrase Readout*places the same phrase after the complete input and aggregates its token states, combining full\-context access with the anchored perspective\. The resulting representation reflects task\-conditioned evidence integration and inference rather than a generic summary of salient content\. Without parameter updates, architectural modification, or reranking, Lens achieves an overall Precision@1 of 63\.9 across all 36 MMEB datasets, outperforming the closest same\-backbone training\-free embedding baseline by 10\.2 points\.

## Introduction

Autoregressive models are increasingly being repurposed as general\-purpose representation models\. Unlike dedicated embedding encoders, large language models \(LLMs\) and multimodal large language models \(MLLMs\) are pretrained to interpret content under diverse instructions and contexts\. Their hidden states can therefore reflect not only what is explicitly present in an input, but also how the input should be understood for a particular task\. This ability makes frozen autoregressive models a promising foundation for training\-free representation learning\.

Recent studies have begun to convert decoder hidden states into embeddings\. Training\-based approaches adapt pretrained models through contrastive objectives, embedding supervision, or architectural modificationBehnamGhader et al\. \([2024](https://arxiv.org/html/2609.20252#bib.bib2)\); Jiang et al\. \([2025](https://arxiv.org/html/2609.20252#bib.bib8)\); Lin et al\. \([2025](https://arxiv.org/html/2609.20252#bib.bib11)\); Zhang et al\. \([2025](https://arxiv.org/html/2609.20252#bib.bib19)\)\. Training\-free approaches instead elicit representations directly from frozen hidden statesJiang et al\. \([2024a](https://arxiv.org/html/2609.20252#bib.bib6)\); Lei et al\. \([2024](https://arxiv.org/html/2609.20252#bib.bib9)\); Springer et al\. \([2025](https://arxiv.org/html/2609.20252#bib.bib15)\); Jiang et al\. \([2024b](https://arxiv.org/html/2609.20252#bib.bib7)\); Zhu et al\. \([2025](https://arxiv.org/html/2609.20252#bib.bib21)\)\. In multimodal settings, E5\-V introduces semantic guidance for hidden\-state extractionJiang et al\. \([2024b](https://arxiv.org/html/2609.20252#bib.bib7)\), while FreeRet further incorporates task instructions, noise suppression, and layer selectionZhu et al\. \([2025](https://arxiv.org/html/2609.20252#bib.bib21)\)\. These methods demonstrate the potential of frozen decoders for direct representation extraction without additional optimization\.

Despite this progress, our analysis of representative failure cases produced by a prior semantic\-elicitation baseline reveals a consistent pattern across tasks: the extracted representations capture plausible input semantics, yet emphasize a perspective different from that required for downstream comparison\.

![Refer to caption](https://arxiv.org/html/2609.20252v1/x1.png)Figure 1:Representative failure cases produced by a prior semantic\-elicitation baseline across different downstream tasks\.As illustrated in Fig\.[1](https://arxiv.org/html/2609.20252#Sx1.F1), classification requires abstracting instance appearance into category identity; visual question answering requires integrating the question with visual evidence to derive the answer; and compositional retrieval requires resolving the semantic effect of a specified modification\. Yet the extracted representations remain centered on immediately salient content\. This observed perspective mismatch reflects a deeper limitation: existing readouts do not fully translate the decoder’s task\-conditioned understanding into the resulting embedding\. We define the mismatch between the semantic perspective required by the task and that expressed by the representation as*semantic perspective misalignment*\.

To address this misalignment, we proposeLens, a training\-free framework that directs representation readout toward the required semantic perspective\. Lens associates a task\-specific readout phrase with this perspective and reuses the phrase after the complete task specification and input\. Its contextualized states combine an explicit semantic role with the full preceding context, and their aggregation yields a task\-directed representation of the model’s deeper task\-relevant understanding\. Component ablations, controlled representation analysis, and qualitative cases jointly support this mechanism, showing that Lens redirects the readout toward the intended semantics and better captures category abstraction, question\-conditioned inference, and compositional understanding\. Lens requires no parameter updates, architectural modification, or reranking\.

Our contributions are summarized as follows:

- •Through cross\-task error analysis, we identify and characterize*semantic perspective misalignment*, a recurring limitation in which the semantic perspective expressed by a representation differs from that required for downstream comparison\.
- •We introduceLens, a general training\-free framework that converts the task\-conditioned understanding of frozen autoregressive models into task\-directed and directly comparable representations, establishing a new perspective for exploiting their semantic and reasoning capabilities without additional optimization\.
- •We provide comprehensive empirical evidence on all 36 MMEB datasets spanning classification, visual question answering, retrieval, and grounding\. Lens achieves an overall Precision@1 of 63\.9 and outperforms the closest same\-backbone training\-free embedding baseline by 10\.2 points\. Component ablations, controlled representation analysis, and qualitative diagnostics further support the proposed mechanism and demonstrate its effectiveness\.

## Related Work

### Multimodal Representation Learning

Multimodal representation learning maps heterogeneous inputs into an embedding space that supports efficient cross\-modal comparison\. Dual\-encoder models such as CLIPRadford et al\. \([2021](https://arxiv.org/html/2609.20252#bib.bib14)\), ALIGNJia et al\. \([2021](https://arxiv.org/html/2609.20252#bib.bib5)\), and SigLIPZhai et al\. \([2023](https://arxiv.org/html/2609.20252#bib.bib17)\)learn transferable vision–language representations from large\-scale paired data\. More recent methods repurpose multimodal large language models \(MLLMs\) as general\-purpose embedding models\. VLM2Vec introduces the Massive Multimodal Embedding Benchmark and contrastively adapts MLLMs across diverse embedding tasksJiang et al\. \([2025](https://arxiv.org/html/2609.20252#bib.bib8)\)\. MM\-Embed improves universal multimodal retrieval through modality\-aware hard\-negative miningLin et al\. \([2025](https://arxiv.org/html/2609.20252#bib.bib11)\), while GME explores large\-scale training for general multimodal embeddingsZhang et al\. \([2025](https://arxiv.org/html/2609.20252#bib.bib19)\)\. These approaches obtain strong representations through additional optimization\. Lens instead extracts task\-directed representations directly from a frozen multimodal decoder\.

### Training\-Free Representations from Autoregressive Models

Autoregressive models have also been adapted for representation learning without parameter updates\. In the text domain, PromptEOL uses natural\-language prompts to elicit sentence representationsJiang et al\. \([2024a](https://arxiv.org/html/2609.20252#bib.bib6)\), while MetaEOL combines multiple elicited views to capture complementary semanticsLei et al\. \([2024](https://arxiv.org/html/2609.20252#bib.bib9)\)\. Echo embeddings repeat the input so that later token states receive fuller context under causal attentionSpringer et al\. \([2025](https://arxiv.org/html/2609.20252#bib.bib15)\)\. These methods demonstrate that prompt design and readout position substantially affect the representational quality of frozen decoders\.

Training\-free multimodal representation learning extends this idea to MLLMs\. E5\-V derives embeddings from semantically elicited hidden statesJiang et al\. \([2024b](https://arxiv.org/html/2609.20252#bib.bib7)\)\. FreeRet further improves task alignment, semantic grounding, noise suppression, and hidden\-state selection, and employs the frozen model for retrieval and rerankingZhu et al\. \([2025](https://arxiv.org/html/2609.20252#bib.bib21)\)\. These methods primarily improve how meaningful semantic states are elicited and selected\. Lens addresses a complementary limitation: whether the selected states express the semantic perspective required for downstream comparison\.

![Refer to caption](https://arxiv.org/html/2609.20252v1/x2.png)Figure 2:Overview of Lens\. It anchors the task\-required perspective to a readout phrase and aggregates the phrase’s full\-context states as the representation\.

## Method

### Overview

Given a downstream taskτ\\tau, Lens independently encodes a queryqqand a candidateccwith a frozen multimodal autoregressive model, producing representations whose similarity reflects task\-defined compatibility\.

User\-side prompt

\{Task Instruction\}

\{Role Mapping\}

Answer format:

\{Readout Phrase\} <\{Task\-Relevant Semantics\}\>

\{Semantic Guidance\}

\{Input Content\}

Assistant\-side prefill

\{Readout Phrase\}

Braced expressions are instantiated separately for each task and for the query and candidate\. The italicized headers distinguish the user and assistant sides and are not included in the model input\. Complete task\-specific query and candidate prompts are provided in the supplementary material\.

As illustrated in Fig\.[2](https://arxiv.org/html/2609.20252#Sx2.F2), Lens consists of two coupled components\.*Semantic Perspective Anchoring*establishes the semantics with respect to which the query and candidate should be compared\.*Contextualized Phrase Readout*extracts the input\-specific realization of those semantics from the model’s internal states\. The former determines what the representation should express, while the latter determines where that representation is read\.

The task\-required semantic perspective is the task\-specific semantic content on which query–candidate compatibility is judged\. The two sides may derive or express this content differently, but their representations must organize it in a directly comparable form\. The task instruction defines the compatibility relation, while role mapping specifies how each side contributes the required semantics\. The answer format associates these semantics with a shared readout phrase, and semantic guidance specifies how the current input should be interpreted under the resulting perspective\.

For news classification, the query\-side prompt derives the topic conveyed by an article, whereas the candidate\-side prompt represents the meaning of a topic category\. Both sides share the answer format “The news topic is: <news topic\>” and the readout phrase “The news topic is:”, while their role mappings and semantic guidance are instantiated separately\.

The two occurrences of the readout phrase serve complementary roles\. Its occurrence in the answer format establishes its association with the task\-required semantic perspective\. Its assistant\-side occurrence follows the complete task and input context and supplies the hidden states used to construct the representation\. Lens thus couples semantic specification and contextual readout through the same linguistic phrase\.

### Semantic Perspective Anchoring

Representation similarity is meaningful only when it reflects the semantics on which task compatibility depends\. A representation may capture salient and plausible properties of an input yet remain unsuitable for comparison if those properties are not the semantics under which the query and candidate should be matched\. Semantic Perspective Anchoring addresses this mismatch by establishing the task\-required perspective that representation similarity should reflect\.

For a given task, this perspective must apply to both input sides while respecting their distinct informational roles\. A query may require inference from multimodal context, whereas a candidate may directly express the relevant semantics\. Role mapping specifies this distinction while organizing both sides around the same perspective\.

Lens associates the task\-required perspective with a shared readout phrase through the answer format:

\{Readout Phrase\} <\{Task\-Relevant Semantics\}\>\.\\text\{\\\{Readout Phrase\\\} \\textless\\\{Task\-Relevant Semantics\\\}\\textgreater\}\.The complete declaration appears before the input and places the phrase and the task\-relevant semantics in an explicit linguistic relation\. The first occurrence of the phrase is not used as the representation\. Instead, the complete answer format becomes part of the preceding context available to the later assistant\-side occurrence\.

The anchored perspective must correspond to the semantics that govern task compatibility rather than to an unrestricted description of either input\. It must remain consistent across the query and candidate, even when their roles differ, and the same readout phrase must be used in the answer format and assistant\-side prefill\. These conditions place both representations under a common task\-defined interpretation\.

Semantic guidance specifies how the task\-required perspective should be derived from each input side\. It directs the model to organize the relevant evidence, perform the required abstraction or inference, and preserve the semantic distinctions needed for task compatibility\. Unlike the answer format, which defines the shared perspective, semantic guidance may differ between the query and candidate according to their informational roles\.

By establishing an explicit association between the readout phrase and the task\-required perspective, Semantic Perspective Anchoring targets semantic perspective misalignment at its source\. It also yields a testable prediction: modifying or removing this association should systematically redirect the resulting representation, as examined in Table[3](https://arxiv.org/html/2609.20252#Sx4.T3)\.

### Contextualized Phrase Readout

Semantic Perspective Anchoring establishes the interpretation associated with the readout phrase\. Contextualized Phrase Readout places the same phrase after the complete prompt and extracts its token states\. Because these tokens occur after the task instruction, role mapping, answer format, semantic guidance, and input, their hidden states are conditioned jointly on the anchored perspective and the current input context\.

The resulting states encode the current input at the readout phrase whose semantic role has been established by the preceding context\. Lens extracts the states of the phrase itself without generating an answer or reading any subsequently generated content\.

Suppose that the readout phrase containsmmtokens at positions\{p1,…,pm\}\\\{p\_\{1\},\\ldots,p\_\{m\}\\\}\. For each sides∈\{q,c\}s\\in\\\{q,c\\\}, Lens extracts their final\-layer states after the self\-attention residual addition and before the post\-attention layer normalization and MLP\. The representation is computed as

𝐞~τs=1m​∑j=1m𝐡L,pjs,𝐞τs=𝐞~τs‖𝐞~τs‖2,\\widetilde\{\\mathbf\{e\}\}\_\{\\tau\}^\{s\}=\\frac\{1\}\{m\}\\sum\_\{j=1\}^\{m\}\\mathbf\{h\}\_\{L,p\_\{j\}\}^\{s\},\\qquad\\mathbf\{e\}\_\{\\tau\}^\{s\}=\\frac\{\\widetilde\{\\mathbf\{e\}\}\_\{\\tau\}^\{s\}\}\{\\left\\\|\\widetilde\{\\mathbf\{e\}\}\_\{\\tau\}^\{s\}\\right\\\|\_\{2\}\},\(1\)where𝐡L,pjs\\mathbf\{h\}\_\{L,p\_\{j\}\}^\{s\}denotes the hidden state of thejj\-th token in the assistant\-side readout phrase\.

Lens aggregates all phrase\-token states because the semantic role is assigned to the readout phrase as a whole rather than to any individual token\. Averaging its contextualized token states avoids privileging an arbitrary lexical position and retains the phrase\-level realization of the anchored perspective\.

For taskτ\\tau, query–candidate compatibility is measured by cosine similarity:

simτ⁡\(q,c\)=\(𝐞τq\)⊤​𝐞τc\.\\operatorname\{sim\}\_\{\\tau\}\(q,c\)=\\left\(\\mathbf\{e\}\_\{\\tau\}^\{q\}\\right\)^\{\\top\}\\mathbf\{e\}\_\{\\tau\}^\{c\}\.\(2\)
Query and candidate inputs are encoded independently, allowing candidate representations to be precomputed and reused\. Lens requires one forward pass per input, without parameter updates, architectural modification, answer generation, or reranking\.

## Experiments

### Experimental Setup

#### Benchmark and Evaluation Protocol\.

We evaluate Lens on the Massive Multimodal Embedding Benchmark \(MMEB\)Jiang et al\. \([2025](https://arxiv.org/html/2609.20252#bib.bib8)\), which contains 36 datasets spanning 10 classification, 10 visual question answering \(VQA\), 12 retrieval, and 4 grounding datasets\. We select MMEB because its unified candidate\-ranking protocol covers substantially different forms of multimodal compatibility, allowing us to evaluate whether Lens generalizes across task semantics without task\-specific training\. Following the standard MMEB protocol, all tasks are formulated as candidate\-ranking problems, and we report Precision@1 \(%\)\. Task\-family scores are averaged over their constituent datasets, while the overall average is computed across all 36 datasets\.

#### Baselines\.

We compare Lens with three groups of multimodal representation methods\.*Conventional embedding models*include CLIPRadford et al\. \([2021](https://arxiv.org/html/2609.20252#bib.bib14)\), BLIP\-2Li et al\. \([2023](https://arxiv.org/html/2609.20252#bib.bib10)\), SigLIPZhai et al\. \([2023](https://arxiv.org/html/2609.20252#bib.bib17)\), OpenCLIPCherti et al\. \([2023](https://arxiv.org/html/2609.20252#bib.bib3)\), and MagicLensZhang et al\. \([2024](https://arxiv.org/html/2609.20252#bib.bib18)\)\.*Training\-based MLLM embedding methods*include VLM2VecJiang et al\. \([2025](https://arxiv.org/html/2609.20252#bib.bib8)\), UniMEGu et al\. \([2025](https://arxiv.org/html/2609.20252#bib.bib4)\), MMRetZhou et al\. \([2025](https://arxiv.org/html/2609.20252#bib.bib20)\), MM\-EmbedLin et al\. \([2025](https://arxiv.org/html/2609.20252#bib.bib11)\), LamRA\-RetLiu et al\. \([2025](https://arxiv.org/html/2609.20252#bib.bib12)\), and GMEZhang et al\. \([2025](https://arxiv.org/html/2609.20252#bib.bib19)\)\.*Training\-free MLLM embedding methods*include E5\-VJiang et al\. \([2024b](https://arxiv.org/html/2609.20252#bib.bib7)\)and FreeRetZhu et al\. \([2025](https://arxiv.org/html/2609.20252#bib.bib21)\)\. We report the embedding\-only variant of FreeRet, denoted FreeRet\-embed, without reranking\. Its Qwen2\.5\-VL\-7B result provides the closest same\-backbone comparison\.

Results for the conventional embedding models and VLM2Vec are taken from VLM2VecJiang et al\. \([2025](https://arxiv.org/html/2609.20252#bib.bib8)\)\. The remaining baseline results are taken from FreeRetZhu et al\. \([2025](https://arxiv.org/html/2609.20252#bib.bib21)\)\.

#### Implementation Details\.

We use Qwen2\.5\-VL\-7B\-InstructBai et al\. \([2025](https://arxiv.org/html/2609.20252#bib.bib1)\)as the primary frozen backbone and retain the original FreeRet task instructions across all 36 MMEB datasets for controlled comparison\. We further evaluate Qwen2\.5\-VL\-3B\-InstructBai et al\. \([2025](https://arxiv.org/html/2609.20252#bib.bib1)\)and Qwen3\.5\-4B/9BQwen Team \([2026](https://arxiv.org/html/2609.20252#bib.bib13)\)without backbone\-specific adaptation\. Complete task\-specific query and candidate prompts are provided in the supplementary material\.

For each input, we extract the final\-layer states of the assistant\-side readout phrase after the self\-attention residual addition and before the post\-attention layer normalization and MLP\. Phrase\-token states are mean\-pooled andℓ2\\ell\_\{2\}\-normalized\. Queries and candidates are encoded independently in a single forward pass, allowing candidate representations to be precomputed and reused\.

Lens requires no parameter updates, additional training data, architectural modification, answer generation, or reranking\.

### Main Results

Table 1:Main results on MMEB in Precision@1 \(%\)\. The classification, VQA, retrieval, and grounding columns report averages over 10, 10, 12, and 4 datasets, respectively\. Baseline results, including the reported averages, follow the corresponding sources: conventional embedding models and VLM2Vec are taken from VLM2VecJiang et al\. \([2025](https://arxiv.org/html/2609.20252#bib.bib8)\), while the remaining baselines are taken from FreeRetZhu et al\. \([2025](https://arxiv.org/html/2609.20252#bib.bib21)\)\. Boldface denotes the best result among training\-free methods, while underlining denotes the best result among all methods shown\. A result satisfying both criteria is both boldfaced and underlined\.†E5\-V is reproduced by FreeRet with Qwen2\.5\-VL\-7B\.Table[1](https://arxiv.org/html/2609.20252#Sx4.T1)reports the average performance across the four MMEB task families\. Lens with Qwen2\.5\-VL\-7B achieves the best training\-free result in every task family and reaches an overall Precision@1 of 63\.9\. Compared with FreeRet\-embed using the same backbone, Lens improves the overall average by 10\.2 points, with gains of 10\.6, 10\.6, 10\.4, and 7\.0 points on classification, VQA, retrieval, and grounding, respectively\.

Lens obtains particularly strong results on classification and VQA, where successful matching depends on identifying the specific label or answer semantics required by the query, directly benefiting from the task\-oriented representations constructed by Lens\. The substantial gain on retrieval further shows that aligning heterogeneous inputs under a shared semantic perspective improves global semantic matching\. The gain on grounding is comparatively smaller because region candidates already provide strong localized visual cues, leaving less room for semantic reorganization; nevertheless, Lens still achieves the best training\-free result by directing these cues toward the relation specified by the query\.

#### Different Backbones and Model Sizes\.

With Qwen2\.5\-VL\-3B, Lens achieves an overall Precision@1 of 53\.9, comparable to the 53\.7 obtained by FreeRet\-embed with the larger Qwen2\.5\-VL\-7B backbone\. Lens further obtains overall scores of 53\.2 and 55\.2 with Qwen3\.5\-4B and Qwen3\.5\-9B, respectively\. Their classification, VQA, retrieval, and grounding scores are 57\.1/54\.8/45\.8/61\.8 and 60\.0/55\.9/47\.4/65\.1, respectively\. These results show that the same Lens formulation can be applied across different multimodal model families and scales without backbone\-specific adaptation\.

Qwen2\.5\-VL\-7B nevertheless achieves the strongest overall performance, indicating that the effectiveness of training\-free readout depends not only on model size, but also on how reliably the backbone exposes task\-conditioned multimodal semantics at the selected internal position\.

### Ablation Studies

Table 2:Ablation results on MMEB in Precision@1 \(%\)\. The first variant removes the answer\-format declaration that associates the task\-required perspective with the readout phrase\. The second retains the complete prompt and assistant\-side phrase but replaces phrase\-level aggregation with the last phrase\-token state\. All other settings remain unchanged\. The best result in each column is boldfaced\.Table[2](https://arxiv.org/html/2609.20252#Sx4.T2)evaluates the two defining operations of Lens across all 36 MMEB datasets\. Removing answer\-format anchoring reduces the overall Precision@1 by 5\.1 points, while replacing phrase\-level aggregation with the last phrase\-token state causes an 8\.3\-point decrease, demonstrating that both operations are important to the broad effectiveness of Lens\. The two ablated variants obtain higher grounding scores, where the candidates are already localized image regions and successful matching depends more directly on preserving fine\-grained local visual correspondence\. In this setting, weaker task\-level semantic reorganization may retain more of the backbone’s native local visual evidence\. Nevertheless, the complete design provides substantially stronger classification, VQA, and retrieval results and achieves the highest overall performance\.

#### Semantic Perspective Anchoring\.

Removing the answer\-format anchor lowers the overall Precision@1 from 63\.9 to 58\.8\. This variant retains the task instruction, role mapping, semantic guidance, and assistant\-side readout phrase, but no longer explicitly associates the phrase with the semantic perspective required by the task\. This causes decreases of 8\.7, 5\.0, and 4\.6 points on classification, VQA, and retrieval, respectively\. These results show that merely providing the relevant task information is insufficient: explicitly anchoring the readout phrase to the task\-required perspective is important for organizing heterogeneous queries and candidates under a consistent basis of comparison\.

#### Contextualized Phrase Readout\.

Using only the last phrase\-token state reduces the overall Precision@1 from 63\.9 to 55\.6 while retaining the answer\-format anchor, complete prompt, and assistant\-side phrase\. It produces decreases of 8\.9, 9\.2, and 11\.0 points on classification, VQA, and retrieval, respectively\. This substantial degradation indicates that the task\-relevant evidence organized at the readout phrase is distributed across its contextualized token states rather than being fully represented by a single lexical position\. Aggregating the complete phrase therefore preserves a more comprehensive task\-directed representation for subsequent similarity comparison\.

Table 3:Average cosine similarities on the N24News test split\. The first block compares each query representation with its correct topic representation; the second compares the unanchored representation with its content\-oriented and task\-oriented counterparts\.![Refer to caption](https://arxiv.org/html/2609.20252v1/x3.png)Figure 3:Representative cases corrected by Lens, illustrating question\-conditioned inference, compositional\-constraint preservation, and knowledge\-supported inference\.

### Representation\-Level Effect of Semantic Perspective Anchoring

We further examine whether the answer format can control the semantic perspective expressed at the readout phrase\. We conduct the analysis on the full N24News test splitWang et al\. \([2022](https://arxiv.org/html/2609.20252#bib.bib16)\)using the frozen Qwen2\.5\-VL\-7B backbone\. Candidate topic representations are encoded once with the task\-oriented target prompt and remain fixed across all query conditions\. We construct three query conditions while keeping the input, task instruction, semantic guidance, assistant\-side readout phrase, and extraction procedure unchanged\. The*task\-oriented*condition uses “The news topic is: <news topic\>” as the answer format, the*content\-oriented*condition replaces it with “The input content is: <input content\>”, and the*unanchored*condition removes the answer format\. Thus, the experiment tests whether changing only the semantic perspective specified by the answer format redirects the representation extracted from the same readout phrase\.

As shown in Table[3](https://arxiv.org/html/2609.20252#Sx4.T3), the task\-oriented representation has the highest similarity to the correct topic representation, whereas the content\-oriented and unanchored variants are less aligned with the task target\. Moreover, the unanchored representation is closer to the content\-oriented representation than to the task\-oriented one\. These results show that changing the answer format redirects the semantic perspective expressed at the fixed readout phrase: task\-oriented anchoring moves the representation toward the news\-topic semantics required for comparison, whereas changing or removing the anchor shifts it away from this task\-oriented state\. This directly supports the central mechanism of Semantic Perspective Anchoring\.

### Qualitative Analysis of Corrected Cases

Figure[3](https://arxiv.org/html/2609.20252#Sx4.F3)presents representative cases that are incorrectly predicted by FreeRet\-embed but correctly resolved by Lens under the same Qwen2\.5\-VL\-7B backbone\. Since the input, candidate set, and frozen model remain unchanged, the differences reflect how the two methods organize the backbone’s internal semantics for comparison\. Across the examples, FreeRet\-embed tends to emphasize globally salient content, whereas Lens focuses on the evidence required by the query\.

In the donut example, the task requires a historical relation rather than object identity, and Lens shifts the representation toward the queried year*1847*\. The*left*and*closed silver laptop*cases similarly require preserving spatial relations and compositional constraints beyond broad object similarity\. The ScienceQA cases further suggest that Lens can organize not only explicit visual evidence, but also task\-relevant knowledge and reasoning encoded in the frozen backbone when the answer is not directly stated in the input\.

These examples illustrate semantic perspective misalignment: the same input can support multiple plausible interpretations, but only one provides the appropriate basis for candidate comparison\. Semantic Perspective Anchoring specifies this task\-required interpretation, while Contextualized Phrase Readout preserves the corresponding evidence for direct comparison\. Lens therefore goes beyond reorganizing surface input content, providing a training\-free way to expose and reuse the broader knowledge and reasoning capabilities already acquired by multimodal autoregressive models\.

## Conclusion

We introducedLens, a training\-free framework for deriving multimodal representations from frozen autoregressive models\. Through cross\-task error analysis, we identified*semantic perspective misalignment*: existing readouts can capture plausible input semantics while failing to express the perspective required for downstream comparison\. Lens addresses this problem by explicitly associating the task\-required perspective with a readout phrase and extracting its contextualized states after the complete input\. Without parameter updates, architectural modification, or reranking, Lens achieves an overall Precision@1 of 63\.9 across all 36 MMEB datasets, outperforming the closest same\-backbone training\-free embedding baseline by 10\.2 points\. Component ablations demonstrate the contributions of Semantic Perspective Anchoring and Contextualized Phrase Readout, while controlled representation analysis shows that anchoring redirects the readout toward the intended semantic perspective\. Qualitative cases further show improvements in question\-conditioned inference, compositional understanding, and knowledge\-supported reasoning\. These findings support a broader view of training\-free representation learning: embeddings need not be limited to compressed descriptions of observed content, but can instead reflect the task\-conditioned semantics that a frozen multimodal decoder organizes, integrates, and infers\.

## References

- Bai et al\. \(2025\)Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin\.Qwen2\.5\-VL technical report\.*arXiv preprint arXiv:2502\.13923*, 2025\.doi:10\.48550/arXiv\.2502\.13923\.URLhttps://arxiv\.org/abs/2502\.13923\.
- BehnamGhader et al\. \(2024\)Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy\.LLM2vec: Large language models are secretly powerful text encoders\.In*First Conference on Language Modeling*, 2024\.URLhttps://openreview\.net/forum?id=IW1PR7vEBf\.
- Cherti et al\. \(2023\)Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev\.Reproducible scaling laws for contrastive language\-image learning\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 2818–2829, June 2023\.
- Gu et al\. \(2025\)Tiancheng Gu, Kaicheng Yang, Ziyong Feng, Xingjun Wang, Yanzhao Zhang, Dingkun Long, Yingda Chen, Weidong Cai, and Jiankang Deng\.Breaking the modality barrier: Universal embedding learning with multimodal llms\.In*Proceedings of the 33rd ACM International Conference on Multimedia*, MM ’25, page 2860–2869, New York, NY, USA, 2025\. Association for Computing Machinery\.ISBN 9798400720352\.doi:10\.1145/3746027\.3754845\.URLhttps://doi\.org/10\.1145/3746027\.3754845\.
- Jia et al\. \(2021\)Chao Jia, Yinfei Yang, Ye Xia, Yi\-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun\-Hsuan Sung, Zhen Li, and Tom Duerig\.Scaling up visual and vision\-language representation learning with noisy text supervision\.In Marina Meila and Tong Zhang, editors,*Proceedings of the 38th International Conference on Machine Learning*, volume 139 of*Proceedings of Machine Learning Research*, pages 4904–4916\. PMLR, 18–24 Jul 2021\.URLhttps://proceedings\.mlr\.press/v139/jia21b\.html\.
- Jiang et al\. \(2024a\)Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, and Fuzhen Zhuang\.Scaling sentence embeddings with large language models\.In Yaser Al\-Onaizan, Mohit Bansal, and Yun\-Nung Chen, editors,*Findings of the Association for Computational Linguistics: EMNLP 2024*, pages 3182–3196, Miami, Florida, USA, November 2024a\. Association for Computational Linguistics\.doi:10\.18653/v1/2024\.findings\-emnlp\.181\.URLhttps://aclanthology\.org/2024\.findings\-emnlp\.181/\.
- Jiang et al\. \(2024b\)Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang\.E5\-V: Universal embeddings with multimodal large language models\.*arXiv preprint arXiv:2407\.12580*, 2024b\.
- Jiang et al\. \(2025\)Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen\.VLM2vec: Training vision\-language models for massive multimodal embedding tasks\.In*The Thirteenth International Conference on Learning Representations*, 2025\.URLhttps://openreview\.net/forum?id=TE0KOzWYAF\.
- Lei et al\. \(2024\)Yibin Lei, Di Wu, Tianyi Zhou, Tao Shen, Yu Cao, Chongyang Tao, and Andrew Yates\.Meta\-task prompting elicits embeddings from large language models\.In Lun\-Wei Ku, Andre Martins, and Vivek Srikumar, editors,*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 10141–10157, Bangkok, Thailand, August 2024\. Association for Computational Linguistics\.doi:10\.18653/v1/2024\.acl\-long\.546\.URLhttps://aclanthology\.org/2024\.acl\-long\.546/\.
- Li et al\. \(2023\)Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi\.BLIP\-2: Bootstrapping language\-image pre\-training with frozen image encoders and large language models\.In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors,*Proceedings of the 40th International Conference on Machine Learning*, volume 202 of*Proceedings of Machine Learning Research*, pages 19730–19742\. PMLR, 23–29 Jul 2023\.URLhttps://proceedings\.mlr\.press/v202/li23q\.html\.
- Lin et al\. \(2025\)Sheng\-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping\.MM\-EMBED: UNIVERSAL MULTIMODAL RETRIEVAL WITH MULTIMODAL LLMS\.In*The Thirteenth International Conference on Learning Representations*, 2025\.URLhttps://openreview\.net/forum?id=i45NQb2iKO\.
- Liu et al\. \(2025\)Yikun Liu, Yajie Zhang, Jiayin Cai, Xiaolong Jiang, Yao Hu, Jiangchao Yao, Yanfeng Wang, and Weidi Xie\.Lamra: Large multimodal model as your advanced retrieval assistant\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 4015–4025, June 2025\.
- Qwen Team \(2026\)Qwen Team\.Qwen3\.5: Towards native multimodal agents, February 2026\.URLhttps://qwen\.ai/blog?id=qwen3\.5\.
- Radford et al\. \(2021\)Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever\.Learning transferable visual models from natural language supervision\.In Marina Meila and Tong Zhang, editors,*Proceedings of the 38th International Conference on Machine Learning*, volume 139 of*Proceedings of Machine Learning Research*, pages 8748–8763\. PMLR, 18–24 Jul 2021\.URLhttps://proceedings\.mlr\.press/v139/radford21a\.html\.
- Springer et al\. \(2025\)Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, and Aditi Raghunathan\.Repetition improves language model embeddings\.In*The Thirteenth International Conference on Learning Representations*, 2025\.URLhttps://openreview\.net/forum?id=Ahlrf2HGJR\.
- Wang et al\. \(2022\)Zhen Wang, Xu Shan, Xiangxie Zhang, and Jie Yang\.N24News: A new dataset for multimodal news classification\.In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, and Stelios Piperidis, editors,*Proceedings of the Thirteenth Language Resources and Evaluation Conference*, pages 6768–6775, Marseille, France, June 2022\. European Language Resources Association\.URLhttps://aclanthology\.org/2022\.lrec\-1\.729/\.
- Zhai et al\. \(2023\)Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer\.Sigmoid loss for language image pre\-training\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\)*, pages 11975–11986, October 2023\.
- Zhang et al\. \(2024\)Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lee, Siyuan Qiao, Wenhu Chen, Yu Su, and Ming\-Wei Chang\.MagicLens: Self\-supervised image retrieval with open\-ended instructions\.In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors,*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pages 59403–59420\. PMLR, 21–27 Jul 2024\.URLhttps://proceedings\.mlr\.press/v235/zhang24an\.html\.
- Zhang et al\. \(2025\)Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang\.Bridging modalities: Improving universal multimodal retrieval by multimodal large language models\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 9274–9285, June 2025\.
- Zhou et al\. \(2025\)Junjie Zhou, Yongping Xiong, Zheng Liu, Ze Liu, Shitao Xiao, Yueze Wang, Bo Zhao, Chen Jason Zhang, and Defu Lian\.MegaPairs: Massive data synthesis for universal multimodal retrieval\.In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 19076–19095, Vienna, Austria, July 2025\. Association for Computational Linguistics\.ISBN 979\-8\-89176\-251\-0\.doi:10\.18653/v1/2025\.acl\-long\.935\.URLhttps://aclanthology\.org/2025\.acl\-long\.935/\.
- Zhu et al\. \(2025\)Yuhan Zhu, Xiangyu Zeng, Chenting Wang, Xinhao Li, Chunxu Liu, Yicheng Xu, Ziang Yan, Yi Wang, and Limin Wang\.FreeRet: MLLMs as training\-free retrievers\.*arXiv preprint arXiv:2509\.24621*, 2025\.