Follow the Flow: On Information Flow Across Textual Tokens in Text-to-Image Models

# Follow the Flow: On Information Flow Across Textual Tokens in Text-to-Image Models Source: https://arxiv.org/html/2504.01137 Guy Kaplan1, Michael Toker211footnotemark:1, Yuval Reif1, Yonatan Belinkov2,3, Roy Schwartz1 1Hebrew University of Jerusalem \{guy\.kaplan3,yuval\.reif,roy\.schwartz1\}@mail\.huji\.ac\.il 2Technion – Israel Institute of Technology tok@campus\.technion\.ac\.il,belinkov@technion\.ac\.il 3Kempner Institute, Harvard University ###### Abstract Text\-to\-image generation models suffer from alignment problems, where generated images fail to accurately capture the objects and relations in the text prompt\. Prior work has focused on improving alignment by refining the diffusion process, ignoring the role of the text encoder, which guides the diffusion\. In this work, we investigate how semantic information is distributed across token representations in text\-to\-image prompts, analyzing it at two levels: \(1\)*in\-item representation*—whether individual tokens represent their lexical item \(i\.e\., a word or expression conveying a single concept\), and \(2\)*cross\-item interaction*—whether information flows between tokens of different lexical items\. We use patching techniques to uncover encoding patterns, and find that information is usually concentrated in only one or two of the item’s tokens; for example, in the item “San Francisco’s Golden Gate Bridge”, the token “Gate” sufficiently captures the entire expression while the other tokens could effectively be discarded\. Lexical items also tend to remain isolated; for instance, in the prompt “a green dog”, the token “dog” encodes no visual information about “green”\. However, in some cases, items do influence each other’s representation, often leading to misinterpretations—e\.g\., in the prompt “a pool by a table”, the token “pool” represents a “pool table” after contextualization\. Our findings highlight the critical role of token\-level encoding in image generation, and demonstrate that simple interventions at the encoding stage can substantially improve alignment and generation quality\.111Project repository:https://github.com/tokeron/lens Follow the Flow: On Information Flow Across Textual Tokens in Text\-to\-Image Models Guy Kaplan1††thanks:Equal contribution\., Michael Toker211footnotemark:1, Yuval Reif1, Yonatan Belinkov2,3, Roy Schwartz11Hebrew University of Jerusalem\{guy\.kaplan3,yuval\.reif,roy\.schwartz1\}@mail\.huji\.ac\.il2Technion – Israel Institute of Technologytok@campus\.technion\.ac\.il,belinkov@technion\.ac\.il3Kempner Institute, Harvard University Refer to captionFigure 1:Our main findings\.Left:When generating an image based on a single input token, we find that information within a lexical item is unevenly distributed across its tokens’ contextualized representations\. In this example, one token carries the meaning of the entire item \(e\.g\.,licrepresents a pelican, whilepeandando not\)\.Right:Contextual items may distort a token’s encoding, leading to misaligned interpretations in the generated image\. Top: generations from the full prompts “apoolby atable” \(misaligned\) and “apoolby achair” \(aligned\)\. Bottom: generations from the contextualized tokenpoolalone\. Withtable,poolencodes a pool table; withchair, it retains the intended swimming pool meaning\.## 1Introduction Text\-to\-image \(T2I\) models typically consist of two main components: a text encoder and a diffusion model\(Hoet al\.,2020 (https://arxiv.org/html/2504.01137#bib.bib49); Song and Ermon,2019 (https://arxiv.org/html/2504.01137#bib.bib50)\)\. The former processes the user’s prompt, transforming it into a representation that guides the latter in generating the image\. Though widely used, T2I models often exhibit prompt\-image misalignment, where generated images fail to capture key concepts from the user’s prompt\(Cheferet al\.,2023a (https://arxiv.org/html/2504.01137#bib.bib28); Rassinet al\.,2022 (https://arxiv.org/html/2504.01137#bib.bib27); Huanget al\.,2023a (https://arxiv.org/html/2504.01137#bib.bib5)\)\. Prior work has attempted to address these issues by modifying the diffusion stage, and particularly the cross\-attention mechanism\(Rassinet al\.,2023 (https://arxiv.org/html/2504.01137#bib.bib29); Cheferet al\.,2023a (https://arxiv.org/html/2504.01137#bib.bib28); Daharyet al\.,2024 (https://arxiv.org/html/2504.01137#bib.bib51)\), under the implicit assumption that each textual token reliably encodes the item it is intended to convey\. This raises two fundamental questions regardingtextual encoding: \(1\) is the meaning of a lexical item evenly distributed across its tokens, or concentrated in just one or two? and \(2\) does each token exclusively encode its lexical item, or can it also absorb information from surrounding items? In this work, we examine these questions by studying how information is distributed across tokens after the textual encoding stage\. We focus on*lexical items*—words or phrases that convey a single meaning, such as “pelican” or “golden gate bridge”\. We trace how item information is distributed both*within*the tokens of a single item \(*in\-item*\), and*across*tokens of different items \(*cross\-item*\), using the same causal framework for both analyses\.222SeeFig\.1 (https://arxiv.org/html/2504.01137#S0.F1)for examples of the different cases\.To do so, we use a causal intervention framework\(Tokeret al\.,2025 (https://arxiv.org/html/2504.01137#bib.bib3)\)that assesses the information encoded in each contextual token representation at the encoder’s output \(2 (https://arxiv.org/html/2504.01137#S2)\)\.333Throughout this work, we study token representations at the output of the text encoder\. For brevity, we sometimes omit the word “representation”\.We then evaluate this framework on prompts drawn from widely used T2I benchmarks \(3 (https://arxiv.org/html/2504.01137#S3)\)\. For*in\-item*representation \(4 (https://arxiv.org/html/2504.01137#S4)\), we find that a lexical item’s meaning is typically concentrated in one or tworepresentative tokens\(tokens that alone suffice to convey the full item, e\.g\., “lic” for “pelican” inFig\. ̃1 (https://arxiv.org/html/2504.01137#S0.F1)\)\. Surprisingly, ablating the non\-representative tokens not only does not hurt performance, but actually*improves alignment*by 21% relatively\. We further show that representative tokens can be identified efficiently without image generation, opening the door to pruning non\-representative tokens directly within T2I pipelines\. For*cross\-item*interactions \(5 (https://arxiv.org/html/2504.01137#S5)\), we apply the same framework to ask whether information flows between different lexical items in a prompt\. We observe cross\-item flow in 11% of cases\. Interestingly, this flow does not always follow syntactic structure, and can result in incorrect item resolution—especially with polysemous words\. For instance, in the prompt “apoolby atable”,poolcan wrongly suggesttablerefers to a billiard table, an instance ofsemantic leakage\(Rassinet al\.,2022 (https://arxiv.org/html/2504.01137#bib.bib27)\)\. Overall, our analysis shows that token\-level encoding is typically concentrated, largely item\-isolated, yet vulnerable tosemantic leakagein cases such as polysemous words, which can lead to misinterpretations\. Across our experiments, we also demonstrate simple interventions for improving alignment, highlighting the importance of further investigating the text encoder’s role in T2I\. ## 2Methodology We introduce a method for causal intervention by generating images from arbitrary subsets of token representations while masking the others\.444While tokens interact throughout encoding, their final representations can be examined in isolation\. This allows us to visualize how each token—following the encoding step—contributes to the diffusion process\.We use it to analyze two phenomena: \(1\) how information is distributed across tokens within a single lexical item \(4 (https://arxiv.org/html/2504.01137#S4)\), and \(2\) how different lexical items influence one another \(5 (https://arxiv.org/html/2504.01137#S5)\)\. Refer to captionFigure 2:Evaluating in\-item information flow\.Our proposed framework interprets the information flow within a lexical item\. We generate images from each token comprising the lexical item \(left\) and analyze them with a VLM \(right\)\. In this example, only the tokenlicrepresents the concept “pelican”, whereaspeandando not\.Our method builds on the framework proposed byTokeret al\.\(2025 (https://arxiv.org/html/2504.01137#bib.bib3)\)\. Given a prompt withNNtokenst1,t2,...,tNt\_\{1\},t\_\{2\},\\ldots,t\_\{N\}, our goal is to isolate and interpret the information encoded by a subset of these tokens\. LetS⊂\{1,...,N\}S\\subset\\\{1,\\ldots,N\\\}be the indeces of a given subset, where0<\|S\| "In Yes, No and maybe\. Does every image match one of those descriptions: \(description string\)? Answer Yes if all images match or relate to at least one description, Maybe if only some match, otherwise No\." Here, thetextual descriptioncan be either a single lexical item or a complete textual prompt\. ### A\.3Evaluating Generated Textual Descriptions\. We employ GPT\-4o\(OpenAIet al\.,2024 (https://arxiv.org/html/2504.01137#bib.bib15)\)to evaluate the textual interpretations produced by Patchscopes\. We use the following prompt: > “In Yes, No and Maybe\. Does every image match the description: \{Patchscopes\_description\} ? Answer Yes if all images match or relate to the description, Maybe if only some match, otherwise No\.” ### A\.4Evaluating Relations Between Items\. We enhance our leakage validation by distinguishing between cases where two lexical items exhibiting semantic leakage are perceptually bound together—for example, “old” and “man” in the prompt “a portrait of an old man”—and cases where they are not as “cone hat” and “eating” in the prompt “A person wearing a cone hat is eating” \(seeFig\. ̃5 (https://arxiv.org/html/2504.01137#S5.F5)\)\. To achieve this, we use a large language model \(LLM\) as a judge\. Specifically, we use GPT\-4o and employ the following prompt: > “In Yes or No: in this prompt: \{input\_prompt\}, are \{item\_1\} and \{item\_2\} perceptually bound together?” We then filter out all cases where the lexical items are perceptually bound together and find that only 6\.5% instances exhibit unintentional leakage\. ### A\.5Intended Item Evaluation For each lexical\-item, we manually create two interpretations \- one in the intended interpretation from the prompt, and another is a possible wrong interpretation of the word in other contexts\. For example, given the prompt “A standing zebra to the right of a city bus station”, the current interpretations would be “the animal zebra”, while the incorrect interpretations would be “A zebra crossing”\. We then ask a VLM to evaluate the generated image, and asses if the first or the latter interpretations exists in the images\. To evaluate the model’s capacity for contextual disambiguation, we manually define two interpretations for each lexical item in each prompt\. The first is the intended semantic meaning derived from the prompt’s context, and the second is an alternative interpretation, that is wrong in this context\. For instance, in the prompt “A standing zebra to the right of a city bus station”, the intended meaning of “zebra is the animal”, whereas the wrong interpretation is a “zebra crossing”\. Subsequently, a VLM analyzes the generated image to determine if it depicts the intended interpretation or the wrong interpretation as we define it\. ### A\.6Resources Our computational experiments involved inference with four distinct text\-to\-image models: Flux\-dev, Flux\-schnell, sdxl\-turbo, and Sana\. The parameter sizes for these models are approximately 12 billion for Flux\-dev and Flux\-schnell, 3\.1 billion for sdxl\-turbo, and a range of 0\.6 to 4\.8 billion for the Sana models, with our experiments utilizing a 1\.6 billion parameter version\. The total computational budget for these experiments is estimated to be approximately 480 GPU hours\. The computing infrastructure consisted of a cluster of eight NVIDIA A100 GPUs\. This configuration provided the necessary computational power for the large number of inference tasks performed\. ### A\.7Use of AI Assistants\. We utilized AI assistants to support this research\. For coding the experiments, we used Microsoft’s Copilot and Anthropic’s Claude 3; all generated code was manually reviewed and validated by us to ensure it aligned with our requirements\. For the paper, Google’s Gemini models \(Pro and Flash\) were used to improve the writing and clarity\. We have carefully reviewed all content to ensure it accurately reflects our intentions\. ## Appendix BData DrawBench\(Sahariaet al\.,2022 (https://arxiv.org/html/2504.01137#bib.bib18)\): We include all categories except for “misspelling”, “rare words”, and “text”\. Overall we extract 134 prompts from DrawBench\. In total, we obtain 1,053 prompts\. #### Extended dataset of leakage prompts\. Our augmentation process incorporates two components\. First, we generate variations of existing prompts fromRassinet al\.\(2022 (https://arxiv.org/html/2504.01137#bib.bib27)\)\(e\.g\., modifying ‘a gentleman with a bow in the forest’ to ‘a man wearing a bow in the jungle’\)\. Second, we introduce novel prompts with potential semantic leakage\. For these prompts, we apply a one\-lexical item change test by generating an image from a similar prompt that substitutes the affected or leaked item with an alternative term \(e\.g\., replacing ‘bishops’ with ‘cardinals’ or ‘checkers’ in ‘chess in ‘2 bishops playing chess’\)\. This test ensures that minimal lexical modifications do not alter the intended semantic meaning while producing a different image due to semantic leakage from another item in the prompt \(see the first two columns inFig\. ̃8 (https://arxiv.org/html/2504.01137#A2.F8)for few visual examples\)\. Together, these methods enrich the dataset and provide a robust framework for analyzing semantic leakage\. The full list of prompts is available in our Git repository\.888https://github.com/tokeron/lens Refer to captionFigure 8:Examples from our semantic leakage method\.Left: standard generation of leakage contained prompt\.Second: generation using a one\-lexical item change test as part of the dataset creation \(a minimal substitution to verify that a slight lexical change yields a different image\)\.Third: image from the contextual representation \(misinterpreted item\)\.Fourth: image from the uncontextualized representation \(correct interpretation\)\.Right: final generation after patching the correct, uncontextualized representation into the prompt\. ## Appendix CAdditional models ### C\.1FLUX\-Dev In addition to our primary experiments with FLUX, we repeated all analyses using the Flux\-dev variant\. The redundant versus representative token experiments yielded similar trends, with 55% of tokens identified as representative and 45% as non\-representative—values closely matching those observed with FLUX\. Likewise, our inter\-item flow experiments confirmed that information flow occurred in 11% of cases \(and 3\.1% miss intended leakage\), reinforcing the overall patterns reported in the main text\. Notably, while the aggregate trends are consistent across models, the specific lexical items resolved can differ between FLUX\-schnell and Flux\-dev, indicating a potentially slightly different inner\-lexicon\(Kaplanet al\.,2025 (https://arxiv.org/html/2504.01137#bib.bib8)\)\. These findings underscore the robustness of our approach while highlighting model\-dependent nuances in token representation and information flow dynamics\. ### C\.2SDXL\-Turbo Refer to captionFigure 9:Images generated from individual subtokens in SDXL\-Turbo\. We find that, in many cases, the representation of an item is not clearly reflected in any of its subtokens—for example, in the cas