On the Diversity of Analogy Making in Large Language Models

# On the Diversity of Analogy Making in Large Language Models
Source: [https://arxiv.org/html/2608.03233](https://arxiv.org/html/2608.03233)
Daniel Xavier de SousaCaio César Sifuentes BarcelosHongyu GuoXiaodan Zhu

###### Abstract

Large Language Models \(LLMs\) have demonstrated remarkable potential for analogy making, a core cognitive capability that drives novelty and creativity\. While prior research has extensively investigated the applications and underlying mechanisms of LLM\-based analogy making, its output diversity remains largely unexplored, despite being essential for broadening cross\-domain connections and fostering scientific innovation\. In this work, we present a comprehensive evaluation of analogy diversity across ten state\-of\-the\-art open\- and closed\-source LLMs\. Our findings highlight a concerning issue of domain homogeneity, a prevalent tendency for LLMs to generate analogies from a narrow set of target domains, limiting both inter\-query and intra\-model diversity\. Furthermore, our analysis reveals a fundamental trade\-off in existing LLM diversity\-enhancement methods: increasing output diversity often comes at the expense of output quality\. Finally, our causal analysis of LLM internals uncovers substantial differences in the model\-sensitive regions governing analogy diversity across LLMs, suggesting a potential mechanistic basis for the observed diversity–quality trade\-off\. To our knowledge, this is among the first studies to systematically investigate output diversity in LLM\-based analogy making\. Our findings provide empirical insights and practical guidance for developing more diverse LLMs for analogy generation\. Our code are available at\[url\_placeholder\]\.

Machine Learning, ICML

## 1Introduction

Recent advances in Large Language Models \(LLMs\) have demonstrated remarkable potential to accelerate scientific discovery, ushering in a new era of AI\-for\-Science \(AI4S\)\. Beyond their broad range of applications in domains such as protein design\(Fry et al\.,[2026](https://arxiv.org/html/2608.03233#bib.bib8); Liu et al\.,[2025](https://arxiv.org/html/2608.03233#bib.bib20)\), sustainable energy\(Hong et al\.,[2026](https://arxiv.org/html/2608.03233#bib.bib15); Zhang et al\.,[2026](https://arxiv.org/html/2608.03233#bib.bib39)\), materials science\(Park & Walsh,[2026](https://arxiv.org/html/2608.03233#bib.bib24)\), and interdisciplinary knowledge discovery\(Shen et al\.,[2026b](https://arxiv.org/html/2608.03233#bib.bib29)\), recent research has begun to explore LLMs’ emerging capacity to generate diverse and creative analogies across scientific domains, a cognitive capability central to human intelligence that sparks scientific innovation\(Shen et al\.,[2026a](https://arxiv.org/html/2608.03233#bib.bib28)\)\.

In AI4S, where scientific innovation is driven by interdisciplinary knowledge integration, analogy diversity strengthens LLMs’ ability to uncover both explicit and implicit connections across domains and disciplines\(Shen et al\.,[2026b](https://arxiv.org/html/2608.03233#bib.bib29)\)\. A classic historical example is James Clerk Maxwell’s development of electromagnetic theory\(Clerk Maxwell,[1864](https://arxiv.org/html/2608.03233#bib.bib4)\)\. Maxwell employed fluid and mechanical systems as thesource domainto reason about electromagnetic phenomena, thetarget domain\. Although these domains are conceptually distant, their structural correspondence enabled a breakthrough in scientific understanding, highlighting the importance of diverse analogies in driving innovation, as illustrated in Figure[1](https://arxiv.org/html/2608.03233#S1.F1)\.

![Refer to caption](https://arxiv.org/html/2608.03233v1/Figures/Figure_1_maxwell.png)Figure 1:Maxwell’s development of the theory of electromagnetism through analogical reasoning across diverse domains\. The left subfigure illustrates analogies drawn from closely related domains, while the right subfigure shows more diverse cross\-domain analogies\. This example demonstrates how greater analogy diversity enables broader knowledge integration and can foster scientific discovery, motivating our study of analogy diversity in large language models\.Motivated by this observation, a fundamental question is whether current LLMs can generate the diverse analogies needed to foster cross\-domain knowledge transfer and scientific innovation\. Although LLM\-based analogy making has received increasing attention, with existing research primarily focusing on analogy quality, applications, and underlying mechanisms\(Shen et al\.,[2026a](https://arxiv.org/html/2608.03233#bib.bib28)\), the diversity of generated analogies remains largely unexplored\. Different from conventional generation tasks, analogy\-making explicitly requires a departure from the source domain and a bold selection of the target domain, where both source and target domains share similar functional abstractions\(Gentner,[1983](https://arxiv.org/html/2608.03233#bib.bib9); Turney,[2012](https://arxiv.org/html/2608.03233#bib.bib34)\)\. The overlook of diversity in analogy making further hinders a comprehensive understanding in LLM analogy\-making mechanisms, leaving a notable gap to this central research question:Are LLMs capable ofgeneratingdiverse and informative analogies?

![Refer to caption](https://arxiv.org/html/2608.03233v1/Figures/Figure_1_overview_v2.png)Figure 2:Overview of experiments and evaluation methods on LLM’s analogy making diversity\. The model is given the input of the format\[Instruction, source\]along with different inference time methods in the left column\. We then cluster the output generations into different target domains and conduct multi\-faceted evaluations on those generations\.In this work, we provide a thorough evaluation of analogy\-making diversity across multiple LLMs from two complementary perspectives\. First, we evaluate how well mainstream LLMs are able to generate analogical sentences by introducing novel and multi\-faceted metrics that focus on domain selection diversity\. Our experimental results show that, LLMs tend to generate analogous sentences closely aligned with similar target domains, exhibiting a homogeneity trend across different model families\. The excessive homogeneity in analogy generation poses a fundamental limitation when these models are applied to scientific discovery, a process that inherently relies on diversity and creativity\.

Second, to better understand the effect of existing diversity\-enhancing methods under an analogy making setting, we evaluate approaches that span three categories of inference\-time perturbation, including \(i\) diversity prompting based methods, \(ii\) probability and logits steering, and \(iii\) entropy\-gated steering\. Our experimental results show that existing methods provide only limited improvements in analogy diversity and are often accompanied by the sacrifice of analogy quality\. The trade\-off between analogy diversity and output quality suggests that techniques originally designed for general\-purpose generation may not readily address domain homogeneity in analogy\-making tasks\. Additional analysis on LLMs’ internal layers further suggests that such inefficacy might be caused by a lack of a consistent region of analogy making across different models\.

To the best of our knowledge, our work is among the first to systematically study output diversity under the context of LLM analogy generation\. We summarize our main contributions as follows:

- •Comprehensive evaluation of LLM analogy making diversity\.Our work provides a thorough evaluation of diversity in LLMs’ analogy making, using a novel and multi\-faceted metrics that provide detailed insights into the output of LLM generations\.
- •Domain homogeneity in LLM analogy making\.Our results reveal domain homogeneity in analogy making of these models based on both intra\-query and intra\-model perspective, highlighting an alarming limitation in the promotion of creativity and diversity when applying LLMs in scientific innovation\.
- •Diversity\-quality tradeoff in LLM analogy making\.Further analysis on the efficacy of applying diversity enhancing frameworks in LLM analogy making shows a trade\-off between output diversity and quality\. We also identify notable differences in perturbation sensitivity in different models, showing a divergence in models’ internal behaviors in analogy making\.

## 2Related Works

#### Diversity in Generation\.

Generation diversity in Large Language Models \(LLMs\) has been well\-studied, especially in the realm of applications in creative writing\(Chung et al\.,[2025](https://arxiv.org/html/2608.03233#bib.bib3)\)\. Prior work has demonstrated the importance of diversity in applications such as short story completion\(Tian et al\.,[2024](https://arxiv.org/html/2608.03233#bib.bib31)\)and creative writing\(Ding et al\.,[2026](https://arxiv.org/html/2608.03233#bib.bib5)\), where creative variation and exploration of alternative narratives are fundamental to high\-quality generation\. To improve diversity, early works aim to solve the mode collapse phenomenon in LLMs\(Jiang et al\.,[2025](https://arxiv.org/html/2608.03233#bib.bib17)\), where models often generate repetitive, bland, or generic outputs\. Moreover, frameworks such as beam search\(Wu et al\.,[2016](https://arxiv.org/html/2608.03233#bib.bib37)\), Top\-kkand Nucleus \(Top\-pp\) sampling\(Fan et al\.,[2018](https://arxiv.org/html/2608.03233#bib.bib6); Holtzman et al\.,[2020](https://arxiv.org/html/2608.03233#bib.bib14)\)are proposed to mitigate the issue\. Beyond inference\-time strategies to promote diversity, metrics such asDistinct\-nn\(Li et al\.,[2016](https://arxiv.org/html/2608.03233#bib.bib18)\)andSelf\-BLEU\(Zhu et al\.,[2018](https://arxiv.org/html/2608.03233#bib.bib41)\)have become standard for measuring distributional breadth, while recent efforts have begun to emphasize "effective semantic diversity" to ensure that variety does not come at the cost of task\-specific quality\(Wiher et al\.,[2022](https://arxiv.org/html/2608.03233#bib.bib36)\)\. However, these works often address surface\-level or stylistic diversity in creative writing, leaving a sparse space for the exploration of outcome diversity in cognitive tasks such as analogical reasoning\.

#### Analogical Reasoning in LLMs\.

Analogical reasoning is deemed a central faculty of human intelligence\(Minegishi et al\.,[2026](https://arxiv.org/html/2608.03233#bib.bib21); Hofstadter & Sander,[2013](https://arxiv.org/html/2608.03233#bib.bib13)\)\. Modern methods that study human analogy often refer to the process of abstraction and structure mapping that make hops across different domains\(Gentner,[2010](https://arxiv.org/html/2608.03233#bib.bib10)\)\. The recent burgeoning of mechanistic interpretability methods\(Huben et al\.,[2024](https://arxiv.org/html/2608.03233#bib.bib16); Belrose et al\.,[2023](https://arxiv.org/html/2608.03233#bib.bib1); Ghandeharioun et al\.,[2024](https://arxiv.org/html/2608.03233#bib.bib11)\)has fueled studies on LLM analogical reasoning abilities from an anatomical perspective, offering a shift towards a more comprehensive understanding within LLMs than of their human counterparts\. In light of pioneering works that treat analogy\-making as a dual interplay between domain and function\(Gentner,[1983](https://arxiv.org/html/2608.03233#bib.bib9)\), recent works make remarkable progress ranging from model behavioral studies\(Webb et al\.,[2023](https://arxiv.org/html/2608.03233#bib.bib35)\)to efforts that explore underlying mechanisms, with proposals of certain hypotheses such as structure mapping\(Gentner,[1983](https://arxiv.org/html/2608.03233#bib.bib9)\)and graph functor\(Minegishi et al\.,[2026](https://arxiv.org/html/2608.03233#bib.bib21)\)\. However, to our knowledge, all of these explorations are conducted under a classification setup, which often involves comparisons between analogous pairs, thus causing these efforts either to remain relatively superficial or to bear the risk of obliterating the expansive nature of the solution space in analogy\-making, as argued in the bookHofstadter & Sander \([2013](https://arxiv.org/html/2608.03233#bib.bib13)\)\. To bridge this gap, our work moves beyond comparing binary labels to the open\-ended generation analysis in LLM analogical reasoning\.

## 3Experiment Setup

#### Datasets\.

To provide a comprehensive evaluation of diversity in LLM\-based analogy, we conduct experiments on three recently released analogy generation datasets\. AnaloBench \(AB\)\(Ye et al\.,[2024](https://arxiv.org/html/2608.03233#bib.bib38)\)is a benchmark dedicated to story\-level analogy evaluation that contains 340 pairs of high\-quality analogous stories from human annotators\. Metaphoric Analogies \(MA\)\(Boisson et al\.,[2025](https://arxiv.org/html/2608.03233#bib.bib2)\)focuses on extracting structured analogical mappings from literary metaphors with 203 samples that build source and target concept pairs in a four\-term analogy structure\. Metaphor Understanding Challenge \(MUNCH\)\(Tong et al\.,[2024](https://arxiv.org/html/2608.03233#bib.bib33)\)evaluates whether LLMs can understand metaphors as cross\-domain mappings through paraphrasing, providing over 10K paraphrases for sentences containing metaphor use\. We use the full evaluation dataset for both AB and MA; for MUNCH, we extract 1K samples randomly\.

#### Models and Devices\.

We include five closed\-source and five open\-source models in our evaluation\. We useGPT\-5\.2,Grok\-4\.5,gemini\-2\.5\-pro,gemini\-3\.1\-pro, andclaude\-4\.6\-sonnetfor generation diversity evaluation\. We further test five open source models, includingllama\-3\.1\-8B\-instruct,gemma2\-9B\-it,qwen\-3\-8B,mistral\-7B\-Instruct, andphi\-4\-mini\-instruct\. The maximum generation length is set to 100 output tokens for all open source models\. We run our experiments using singlel40sandh100GPUs\.

#### Generation Perturbation Methods\.

We evaluate a spectrum of inference\-time perturbation methods following the categories defined inOstermann et al\. \([2026](https://arxiv.org/html/2608.03233#bib.bib23)\)to provide a comprehensive understanding of their efficacy in LLM analogy making\.

- •Diversity Prompting\.In addition to base prompting, we run diversity prompting strategies, including Logic prompts and Diversity prompts to explicitly encourage analogies from creative and diverse domains\.
- •Probabilistic and Logits Steering\.We run probabilistic methods including Top\-kk\(k=20k=20\), Top\-pp\(p=0\.9p=0\.9\), Top\-η​σ\\eta\\sigma\(η​σ=1\.0\\eta\\sigma=1\.0\), and Min\-pp\(p=0\.1p=0\.1\) in our experiments that resample the output logits to improve output diversity\. We also include G2\(Ruan et al\.,[2025](https://arxiv.org/html/2608.03233#bib.bib27)\)in our experiments, where we set the number of iteration rounds to be 3\.
- •Entropy\-gated Steering\.We also include an adaptation fromLi et al\. \([2026](https://arxiv.org/html/2608.03233#bib.bib19)\)that uses entropy as an indicator to control the output diversity\. We set the resampling thresholdτ\\tauto be larger than the 90th percentile of the whole generated token sequence\.

#### Output Evaluations\.

We run our evaluation fromK=10K=10generations for each sample\. Following previous studies that evaluate generation diversity by mapping natural language into a semantic space\(Reimers & Gurevych,[2019](https://arxiv.org/html/2608.03233#bib.bib26)\), we adopt a similar paradigm\. Specifically, we employ theall\-MiniLM\-L6\-v2sentence transformer as an encoder to obtain semantic embeddings, calculating their mean pairwise cosine distance\. Furthermore, we incorporate the MAUVE score\(Pillutla et al\.,[2022](https://arxiv.org/html/2608.03233#bib.bib25)\)to better capture distributional differences, providing a more comprehensive analysis of generation over quality and diversity\. To evaluate the coherence and quality of the generated analogies, we useGPT\-4o\-miniunder an existing LLM\-as\-a\-judge setup\(Zheng et al\.,[2023](https://arxiv.org/html/2608.03233#bib.bib40)\)\.

![Refer to caption](https://arxiv.org/html/2608.03233v1/Figures/fig1_comp_by_model.png)

![Refer to caption](https://arxiv.org/html/2608.03233v1/Figures/fig2_inter_cluster.png)

Figure 3:\(a\) Comparison of sentence diversity, domain diversity, and output quality across prompting strategies\. Solid lines denote sentence diversity, dashed lines denote domain diversity, and square markers indicate quality scores for open\-source and closed\-source models\. \(b\) Inter\-domain diversity using average linkage\. Bars denote average linkage; dashed line indicates mean clusters per sample\.

## 4Domain Homogeneity in Analogy Generation

We first conduct a study in diversity of models’ analogy\-generation capabilities, focusing on whether the generated analogous sentences fall into diverse target domains\. We assess this from two perspectives: i\) Intra\-output analogy diversity, which measures the pairwise and clustering distances among generations within a single model, and ii\) Inter\-model analogy diversity measures, which assess whether analogy generation converges to the same target domain or commonalities in sentence structure across different models\.

![Refer to caption](https://arxiv.org/html/2608.03233v1/Figures/fig2_mauve.png)Figure 4:Heatmap for Inter\-Model MAUVE scores\. MAUVE is originally designed to measure how closely machine\-generated text matches human\-written text\. We adapt MAUVE to compare how closely related the generated texts are across different close source models\.### 4\.1Intra\-Output Diversity

To obtain intra\-output diversity for each specific model, following similar evaluation for general diversity\(Ruan et al\.,[2025](https://arxiv.org/html/2608.03233#bib.bib27); Ding et al\.,[2026](https://arxiv.org/html/2608.03233#bib.bib5)\), we first measure the pairwise distances among the distinct generations to show how closely related each generation is to the others\. Specifically, we assemble prompts in the form of a binary tuple\[Instruction, Source\], where theInstructionfield asks the model to generate analogical sentences underbase,diverse, orlogicalprompting strategies, and theSourcefield contains a sentence where the model should identify its source domain and generate analogies thereafter\. Under thebasesetting, the model is simply instructed to generate an analogy\. In thediversesetting, the model is additionally encouraged to produce creative and diverse outputs\. Finally, in thelogicalsetting, the model is instructed to preserve the same relational structure while generating analogies in different target domains, without explicitly encouraging diversity\.

Figure[3](https://arxiv.org/html/2608.03233#S3.F3)\(a\) presents the pairwise cosine distances between generation embeddings from two perspectives: \(i\) solid lines \(less saturated colors\), where embeddings are computed from the complete generated analogies, and \(ii\) dashed lines, where embeddings are computed only from the extracted words describing the target domains\. For the latter, target domains are extracted usingGPT\-4o\-mini\. To evaluate the quality of the generated analogies, followingZheng et al\. \([2023](https://arxiv.org/html/2608.03233#bib.bib40)\), we employ an LLM\-as\-a\-judge approach, with the results also reported in the figure with solid lines and saturated colors\.

Our results show intuitive behavior of the tested models: when explicitly instructed to be diverse, the models generate more creative outputs, while prompts emphasizing logical behavior also improve diversity compared to naive prompting strategies \(less saturated colors in Figure[3](https://arxiv.org/html/2608.03233#S3.F3)\(a\)\. However, the quality tends to be better for base prompts \(red saturated color\)\. Comparing closed\-source and open\-source models, there is no clear evidence of superiority between them, showing that more complex post\-training may not necessarily improve diversity\. Nevertheless, even among the mainstream closed\-source models evaluated, the best\-performing model for the diversity prompt \(less saturated solid line in the figure\),Gemini3\.1\-Pro, achieves only an average intra\-output diversity of 0\.49, indicating that overall diversity remains limited\. Considering only the extracted target domains, where irrelevant lexical content is removed and the evaluation focuses solely on the target domains, all models exhibit improved diversity, showing a similar standard among the prompts\. Nevertheless, the overall diversity remains limited, with the mean pairwise diversity not exceeding 0\.63\.

Except for the target domain extraction step, the evaluation shown in Figure[3](https://arxiv.org/html/2608.03233#S3.F3)\(a\) follows a standard approach described in several works for assessing diversity in text generation\. However, a key limitation of this evaluation is that it remains difficult to interpret whether an average similarity score of 0\.6 truly reflects high or low diversity\. To provide a complementary perspective, we employGPT\-4o\-minito identify the semantic target domains of the generated analogies and group the outputs into distinct clusters\. These clusters enable us to assess diversity at the semantic level by considering both the number of discovered clusters and the hierarchical Average Linkage, i\.e\., the average similarity over all pairs of sentences belonging to different clusters111We also evaluated Single Linkage and Complete Linkage and obtained similar results\., where a greater number of clusters and higher Average Linkage values indicate greater diversity\. Similar clustering\-based strategies have been adopted in other contexts to evaluate output diversity\(Pillutla et al\.,[2022](https://arxiv.org/html/2608.03233#bib.bib25)\)\.

Figure[3](https://arxiv.org/html/2608.03233#S3.F3)\(b\) presents the clustering results obtained using the diverse prompt across all evaluated models\. Despite explicitly instructing the models to generate 10 distinct and diverse analogies, we observe an average of only two semantic clusters per model\. Among the evaluated models,Grok\-4\.5produces the highest diversity, with an average of 3\.5 clusters, whereasQwen3\-8Bgenerates fewer than two clusters on average\. Regarding the linkage analysis, both closed\-source and open\-source models exhibit comparable Average Linkage values \(0\.39–0\.65\), indicating similar levels of semantic separation between target domains\. With an average of three clusters,Gemma\-2\-9Bachieves the highest Average Linkage \(avg¯=0\.653\\overline\{\\text\{avg\}\}=0\.653\), followed by the Gemini family among the closed\-source models \(avg¯=0\.628\\overline\{\\text\{avg\}\}=0\.628\)\. In contrast,Qwen3\-8Bobtains the lowest score \(avg¯=0\.390\\overline\{\\text\{avg\}\}=0\.390\)\. To assess the reliability of the results in Figure[3](https://arxiv.org/html/2608.03233#S3.F3), we conducted a human evaluation of the generated clusters\. Based on a randomly sampled subset comprising 10% of the data, the human annotations agreed with the automatic clustering in 95% of the cases\.

These findings reveal a notable tendency toward target\-domain homogeneity in analogy generation, raising concerns about the capability of current LLMs to consistently produce diverse analogical mappings\. For instance, when the model is provided with acontextsentence starting with "Don’t judge a person…", the model has strong preferences for repeating similar target domains such as "book" or "quiet river", with the variety not following a uniform distribution forKKanalogy generations\.

### 4\.2Inter\-Model Diversity

In addition to domain homogeneity within a single model, we further explore the diversity across different models\. Using the diverse prompt settings of the previous section, Figure[4](https://arxiv.org/html/2608.03233#S4.F4)showcases the inter\-model diversity comparison heatmap reported using adapted MAUVE scores\(Pillutla et al\.,[2022](https://arxiv.org/html/2608.03233#bib.bib25)\), where a higher score means more similar and less diverse\. As described in Figure[4](https://arxiv.org/html/2608.03233#S4.F4), although some model pairs exhibit different domain selection preferences \(gemini\-2\.5\-prov\.s\. bothgpt\-5\.2andclaude\-4\.6\-sonnet\), the inter\-model MAUVE scores remain high \(less diverse\) for the remaining models in the comparison\. Notably, when comparing the generation distribution betweengrok\-4\.5andgemini\-3\.1\-pro\-preview, we obtain a MAUVE score of 0\.72, indicating an overwhelming overlap in the generation patterns between these models\.

### 4\.3Data Pollution

One potential issue that poses a threat to the accuracy of our comparison is data pollution, i\.e\., the model has already memorized the corresponding analogous pairs given thesource\. To tackle this challenge, we apply the method introduced byGolchin & Surdeanu \([2024](https://arxiv.org/html/2608.03233#bib.bib12)\)to ensure there is no data pollution issue\. Specifically, for each sample in our dataset, we mask out words at different percentage levels and check whether the model is able to recover the complete sentence measured by token exact match\. Figure[5](https://arxiv.org/html/2608.03233#S4.F5)reports the results in our analysis\. It can be seen that all the models in our experiment arenotsuffering from a data pollution issue, as the highest exact match score of 14% is achieved bygemini\-3\.1\-pro\-previewat the 80% context level\. This means that even the best\-performing model only recovers a small portion of the original sentence when 80% of the context is provided\. The analysis of data pollution further enhances the validity of our observation regarding domain homogeneity\.

![Refer to caption](https://arxiv.org/html/2608.03233v1/Figures/fig4_data_pollution.png.png)Figure 5:Analysis on data pollution\. We use the method fromGolchin & Surdeanu \([2024](https://arxiv.org/html/2608.03233#bib.bib12)\)that makes varied percentage of context and compute the recovery accuracy with the original sentence\.

## 5Evaluation of Diversity Enhancing Methods in Analogy Making

Given the domain homogeneity in LLM analogy making observed above, we further explore the efficacy of existing frameworks that aim to enhance the output diversity of LLMs\. As is mentioned inOstermann et al\. \([2026](https://arxiv.org/html/2608.03233#bib.bib23)\), existing methods that improve output diversity can be divided into three categories, including fine\-tuning, prompting, and steering\. Since the scope of our work lies at inference time, we mainly investigate the latter two methods of prompting and steering\.

For prompting, we are using the best results from our previous section,basicanddiversesettings, which are optimal in quality and diversity respectively\. We also include approaches that fall into the steering category and tend to perturb the model’s internal hidden states in order to modify and encourage output diversity\. We include approaches that span two mainstream subcategories from this perspective in our evaluation, including probabilistic / logistic resampling and entropy\-gated resampling\.

#### Probabilistic and Logistic steering\.

We employTop\-kkthat restricts sampling to thekkmost probable tokens\(Fan et al\.,[2018](https://arxiv.org/html/2608.03233#bib.bib6)\);Top\-ppthat samples from the smallest set of tokens whose cumulative probability reaches a threshold\(Holtzman et al\.,[2020](https://arxiv.org/html/2608.03233#bib.bib14)\);Top\-η​σ\\eta\\sigmathat retains tokens whose logits are withinη\\etastandard deviations of the highest logit\(Tang et al\.,[2024](https://arxiv.org/html/2608.03233#bib.bib30)\), andMin\-ppthat discards tokens whose probability falls below a minimum threshold relative to the most probable token\(Minh et al\.,[2025](https://arxiv.org/html/2608.03233#bib.bib22)\)\. We also include G2\(Ruan et al\.,[2025](https://arxiv.org/html/2608.03233#bib.bib27)\)in our evaluation222We include a reproduction of G2 following the algorithm in the github repository\.\. G2 modifies the output logits based on previous outputs from guide models to improve diversity\.

![Refer to caption](https://arxiv.org/html/2608.03233v1/Figures/radar_comparison_large.png)Figure 6:Radar chart comparing 8 decoding methods across 5 LLMs\. Each model is shown in its own polar plot with 4 metrics \(Diversity, Cluster Linkage, Quality, \#Clusters\) normalized to 0,1 via min\-max scaling \(quality capped at 1,5\)\. Tick labels on each spoke indicate the original\-scale values\.
#### Entropy\-gated steering\.

We take the motivation and insights from\(Li et al\.,[2026](https://arxiv.org/html/2608.03233#bib.bib19)\)and implement an adapted version to accommodate our task of analogy making, where we set a thresholdτ\\tauwith respect to the entropy of the next token to be generated\. The resampling operation is only triggered when the entropy is higher thanτ\\tauand the next token to be generated is identified as the domain token\. Following observations in\(Fan et al\.,[2026](https://arxiv.org/html/2608.03233#bib.bib7)\), we run an additional forward pass in the same model in order to modify the output token distribution for resampling\. The left column in Figure[2](https://arxiv.org/html/2608.03233#S1.F2)summarizes these methods in our evaluation\.

We showcase the performance comparison averaged across three datasets in a radar chart in Figure[6](https://arxiv.org/html/2608.03233#S5.F6)333Complete results are also provided in Table 1 in the Appendix\.when applying these methods introduced above\. Overall, there is no outstanding method that excels in all four evaluation dimensions across the five open\-source models tested\. For instance, despite thebaseprompting method achieving the highest output quality on all models \(red color is close to 5 in all models\), it falls short in terms of diversity metrics when compared to other methods\. The shortcoming in diversity is particularly notable in the case ofMistral\-7B\-instruct\-v0\.3: with an average output quality rating of 4\.32 inbaseprompting, it merely achieves an output diversity score of 0\.33 and an average target domain cluster number of 1\.97 on three datasets \(Table 1 in the Appendix describes this numbers\)\. In comparison, when applying the Top\-kkmethod \(green color in Figure[6](https://arxiv.org/html/2608.03233#S5.F6)\) on theMistral\-7B\-instruct\-v0\.3, with an average analogy quality of 4\.21, it achieves an average diversity score of 0\.53 as well as the average number of target domain clusters of 2\.27, indicating a compromise of diversity in thebaseprompting method when compared with probabilistic steering methods such as Top\-kk\.

In addition to the lack of a dominating method that excels in all evaluation metrics, we also observe that different models exhibit unstable rankings of these approaches even when evaluated on a single metric\. Taking the diversity score metric \(indicated at the top of the circle in Figure[6](https://arxiv.org/html/2608.03233#S5.F6)\) as an example, the best performing methods in enhancing diversity forLlama\-3\.1\-8B\-instructare Top\-ppand G2 \(which belong to the probabilistic and Logistic steering category\), both achieving an average score of 0\.61 on all three datasets\. However, when the same two methods are applied toQwen\-3\-8B, they only achieves a diversity score of 0\.28 and 0\.29, respectively\. In comparison, the entropy\-gated steering method achieves an average performance of 0\.39 in diversity score\.

## 6Detailed Analysis

![Refer to caption](https://arxiv.org/html/2608.03233v1/Figures/fig7_tradeoff.jpeg)

![Refer to caption](https://arxiv.org/html/2608.03233v1/Figures/fig8_layerwise.png)

Figure 7:\(a\) Scatter plot of the Pareto frontier across all diversity enhancing frameworks usingPhi\-4\-mini\-instruct\. \(b\) Heatmap of each layer group being selected as Pareto\-best under layer\-wise noise perturbation across different model families\.In this section, we provide an in\-depth analysis, aiming to account for the observations of divergent performance across these diversity\-enhancing methods elaborated in the previous section\. Specifically, we show that \(1\) there exists a tradeoff between the output diversity and output quality, and \(2\) the misalignment between intervention position and the models’ analogy\-making region may account for the unstable performance in analogy making\.

#### Diversity\-quality tradeoff in analogy making\.

Following the observations in the radar chat \(Figure[6](https://arxiv.org/html/2608.03233#S5.F6)\) showing that the increase in analogy diversity is often accompanied by a decrease in output quality, we further investigate this tradeoff by plotting the Pareto frontier of each model\. Specifically, we collect the average performance across three datasets on diversity and quality metrics and make a scatter plot of these methods\. Due to space limitations, we only showcase the scatter plot forPhi\-4\-mini\-instructin Figure[7](https://arxiv.org/html/2608.03233#S6.F7)\(a\)\. When compared with the base prompting \(light red dot in Figure[7](https://arxiv.org/html/2608.03233#S6.F7)\(a\)\) that yields analogies with an average quality of 4\.28, the most diverse method, Top\-η​σ\\eta\\sigma, \(light pink dot in Figure[7](https://arxiv.org/html/2608.03233#S6.F7)\(a\)\) only yields an average quality of 3\.80, indicating a compromise in the diversity enhancing approaches in output quality\. Additionally, we also observe a clustering effect of those diversity\-enhancing approaches on the Pareto frontier\. This further indicates an absence of a dominating approach in improving models’ analogy diversity meanwhile maintaining high generation quality\.

#### Misalignment between intervention position and the analogy\-making region in LLMs\.

Given the observed tradeoff between diversity and quality, one might also be curious about whether the internal computations governing analogy generation are confined to particular regions of the model, regardless of the model family\. To probe this question, we take inspiration from exiting works in causal perturbation analysis\(Todd et al\.,[2024](https://arxiv.org/html/2608.03233#bib.bib32)\)and introduce the following method that aims to identify the most fragile regions when perturbed with looped computation, a novel perturbation method suggested in\(Fan et al\.,[2026](https://arxiv.org/html/2608.03233#bib.bib7)\)\. Specifically, we blur the hidden states via looped inference and compute the indirect effect, a mainstream method introduced by\(Todd et al\.,[2024](https://arxiv.org/html/2608.03233#bib.bib32)\)to identify the most important layers within an LLM\. In our case, we define the metric of indirect effect to be the combined score of\(5−quality\)×diversity\(5\-\\text\{quality\}\)\\times\\text\{diversity\}\. The most fragile regions are thus identified by maximizing the metric when replaced with perturbations, meaning that these layers are least informative yet diverse in analogy making\. Then, we aggregate, for each model, the frequency with which each layer is selected across all evaluation samples\. Figure[7](https://arxiv.org/html/2608.03233#S6.F7)\(b\) visualizes these frequencies as a heatmap, revealing the layer positions that are most frequently identified as Pareto\-optimal under our perturbation analysis\.

From the resulting heatmap, we observe a fluid and divergent span of layer regions that are sensitive to analogy making across different models\. For instance,llamaandmistralmodels have a more widespread span of layers \(the 20th percentile to the 70th percentile among layer depth\) when perturbed with noise, while forphi\-4\-miniandqwen\-3, the sensitive regions are concentrated between certain layer positions \(the 90th percentile forqwen\-3and the 100th percentile forphi\-4\-mini\)\. The wide span along with the fluid regions across different models that are sensitive to analogy making also echos with the insights from\(Minegishi et al\.,[2026](https://arxiv.org/html/2608.03233#bib.bib21)\), where they argue that analogical reasoning is an ability emerging from the late phase of model internals\. In contrast to this observation, however, all steering methods that influence output diversity operate on the logits that are obtained from the last layer\. The misalignment between the sensitive layer regions and these perturbation methods thus offer a potential explanation for the unstable performance of methods when applied to different LLMs\.

## 7Conclusion and Outlook

We present a systematic study of output diversity in LLM\-based analogy making across ten state\-of\-the\-art open\- and closed\-source LLMs\. Our results indicate the phenomenon of domain homogeneity from both intra\-output and inter\-model perspectives, where models repeatedly generate analogies grounded in a narrow set of target domains, a concerning issue that bears the risk of undermining AI4S creativity\. Further evaluation on various inference\-time diversity\-enhancing approaches from three categories shows that their effectiveness is generally limited and strongly model\-dependent, and there exists a trade\-off in which gains in analogy making diversity are often accompanied by degradation in analogy making quality\. Our further analysis also provides a potential explanation via layer perturbation, showing a misalignment between the intervention location of these methods and the model’s sensitive region in analogy making\.

To the best of our knowledge, this work is among the first to systematically investigate output diversity in open\-ended LLM analogy generation\. Our findings may help inform future LLM model designs that are more diversity\-aware in analogy making\.

## References

- Belrose et al\. \(2023\)Belrose, N\., Furman, Z\., Smith, L\., Halawi, D\., Ostrovsky, I\., McKinney, L\., Biderman, S\., and Steinhardt, J\.Eliciting latent predictions from transformers with the tuned lens\.*ArXiv*, abs/2303\.08112, 2023\.URL[https://api\.semanticscholar\.org/CorpusID:257504984](https://api.semanticscholar.org/CorpusID:257504984)\.
- Boisson et al\. \(2025\)Boisson, J\., Siddique, Z\., Borkakoty, H\., Antypas, D\., Espinosa Anke, L\., and Camacho\-Collados, J\.Automatic extraction of metaphoric analogies from literary texts: Task formulation, dataset construction, and evaluation\.In Rambow, O\., Wanner, L\., Apidianaki, M\., Al\-Khalifa, H\., Eugenio, B\. D\., and Schockaert, S\. \(eds\.\),*Proceedings of the 31st International Conference on Computational Linguistics*, pp\. 6692–6704, Abu Dhabi, UAE, January 2025\. Association for Computational Linguistics\.URL[https://aclanthology\.org/2025\.coling\-main\.448/](https://aclanthology.org/2025.coling-main.448/)\.
- Chung et al\. \(2025\)Chung, J\. J\. Y\., Padmakumar, V\., Roemmele, M\., Sun, Y\., and Kreminski, M\.Modifying large language model post\-training for diverse creative writing, 2025\.URL[https://arxiv\.org/abs/2503\.17126](https://arxiv.org/abs/2503.17126)\.
- Clerk Maxwell \(1864\)Clerk Maxwell, J\.On faraday’s lines of force\.*Transactions of the Cambridge Philosophical Society*, 10:27, 1864\.
- Ding et al\. \(2026\)Ding, Y\., Li, M\., Garces Arias, E\., Aßenmacher, M\., Heumann, C\., and Zhang, C\.Min\-kksampling: Decoupling truncation from temperature scaling via relative logit dynamics\.In Liakata, M\., Moreira, V\. P\., Zhang, J\., and Jurgens, D\. \(eds\.\),*Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 14932–14948, San Diego, California, United States, July 2026\. Association for Computational Linguistics\.ISBN 979\-8\-89176\-390\-6\.doi:10\.18653/v1/2026\.acl\-long\.681\.URL[https://aclanthology\.org/2026\.acl\-long\.681/](https://aclanthology.org/2026.acl-long.681/)\.
- Fan et al\. \(2018\)Fan, A\., Lewis, M\., and Dauphin, Y\.Hierarchical neural story generation\.In Gurevych, I\. and Miyao, Y\. \(eds\.\),*Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 889–898, Melbourne, Australia, July 2018\. Association for Computational Linguistics\.doi:10\.18653/v1/P18\-1082\.URL[https://aclanthology\.org/P18\-1082/](https://aclanthology.org/P18-1082/)\.
- Fan et al\. \(2026\)Fan, Y\., Svete, A\., and Lee, K\.Bridging the gap between latent and explicit reasoning with looped transformers, 2026\.URL[https://arxiv\.org/abs/2606\.31779](https://arxiv.org/abs/2606.31779)\.
- Fry et al\. \(2026\)Fry, B\., Slaw, K\., and Polizzi, N\. F\.Zero\-shot design of drug\-binding proteins via neural iterative selection–expansion\.*Nature*, June 2026\.ISSN 1476\-4687\.doi:10\.1038/s41586\-026\-10670\-w\.URL[https://doi\.org/10\.1038/s41586\-026\-10670\-w](https://doi.org/10.1038/s41586-026-10670-w)\.
- Gentner \(1983\)Gentner, D\.Structure\-mapping: A theoretical framework for analogy\.*Cognitive Science*, 7\(2\):155–170, 1983\.ISSN 0364\-0213\.doi:https://doi\.org/10\.1016/S0364\-0213\(83\)80009\-3\.URL[https://www\.sciencedirect\.com/science/article/pii/S0364021383800093](https://www.sciencedirect.com/science/article/pii/S0364021383800093)\.
- Gentner \(2010\)Gentner, D\.Bootstrapping the mind: Analogical processes and symbol systems\.*Cognitive Science*, 34\(5\):752–775, 2010\.doi:https://doi\.org/10\.1111/j\.1551\-6709\.2010\.01114\.x\.URL[https://onlinelibrary\.wiley\.com/doi/abs/10\.1111/j\.1551\-6709\.2010\.01114\.x](https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1551-6709.2010.01114.x)\.
- Ghandeharioun et al\. \(2024\)Ghandeharioun, A\., Caciularu, A\., Pearce, A\., Dixon, L\., and Geva, M\.Patchscopes: A unifying framework for inspecting hidden representations of language models\.In*Forty\-first International Conference on Machine Learning*, 2024\.URL[https://arxiv\.org/abs/2401\.06102](https://arxiv.org/abs/2401.06102)\.
- Golchin & Surdeanu \(2024\)Golchin, S\. and Surdeanu, M\.Time travel in LLMs: Tracing data contamination in large language models\.In*The Twelfth International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=2Rwq6c3tvr](https://openreview.net/forum?id=2Rwq6c3tvr)\.
- Hofstadter & Sander \(2013\)Hofstadter, D\. and Sander, E\.*Surfaces and Essence: : Analogy as the Fuel and Fire of Thinking*\.04 2013\.ISBN 978\-0465018475\.
- Holtzman et al\. \(2020\)Holtzman, A\., Buys, J\., Du, L\., Forbes, M\., and Choi, Y\.The curious case of neural text degeneration\.In*International Conference on Learning Representations \(ICLR 2020\)*\. OpenReview\.net, 2020\.URL[https://openreview\.net/forum?id=rygGQyrFvH](https://openreview.net/forum?id=rygGQyrFvH)\.
- Hong et al\. \(2026\)Hong, X\., Wang, X\., Harris, S\. J\., Zhao, H\., Meng, J\., Jia, Q\., Zhao, Q\., Xu, K\., Pang, Q\., and Jiang, B\.Deep active learning and knowledge transfer for rapid discovery of lithium metal battery electrolytes\.*Nature Communications*, 17\(1\):5146, March 2026\.ISSN 2041\-1723\.doi:10\.1038/s41467\-026\-70973\-4\.URL[https://doi\.org/10\.1038/s41467\-026\-70973\-4](https://doi.org/10.1038/s41467-026-70973-4)\.
- Huben et al\. \(2024\)Huben, R\., Cunningham, H\., Smith, L\., Ewart, A\., and Sharkey, L\.Sparse autoencoders find highly interpretable features in language models\.In*International Conference on Learning Representations*, volume 2024, pp\. 7827–7845, 2024\.
- Jiang et al\. \(2025\)Jiang, L\., Chai, Y\., Li, M\., Liu, M\., Fok, R\., Dziri, N\., Tsvetkov, Y\., Sap, M\., Albalak, A\., and Choi, Y\.Artificial hivemind: The open\-ended homogeneity of language models \(and beyond\), 2025\.URL[https://arxiv\.org/abs/2510\.22954](https://arxiv.org/abs/2510.22954)\.
- Li et al\. \(2016\)Li, J\., Galley, M\., Brockett, C\., Gao, J\., and Dolan, B\.A diversity\-promoting objective function for neural conversation models\.In Knight, K\., Nenkova, A\., and Rambow, O\. \(eds\.\),*Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pp\. 110–119, San Diego, California, June 2016\. Association for Computational Linguistics\.doi:10\.18653/v1/N16\-1014\.URL[https://aclanthology\.org/N16\-1014/](https://aclanthology.org/N16-1014/)\.
- Li et al\. \(2026\)Li, X\., Callanan, E\., Ghassel, A\., and Zhu, X\.Entropy\-gated branching for efficient test\-time reasoning\.In Demberg, V\., Inui, K\., and Marquez, L\. \(eds\.\),*Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 5054–5069, Rabat, Morocco, March 2026\. Association for Computational Linguistics\.ISBN 979\-8\-89176\-380\-7\.doi:10\.18653/v1/2026\.eacl\-long\.235\.URL[https://aclanthology\.org/2026\.eacl\-long\.235/](https://aclanthology.org/2026.eacl-long.235/)\.
- Liu et al\. \(2025\)Liu, S\., Li, Y\., Li, Z\., Gitter, A\., Zhu, Y\., Lu, J\., Xu, Z\., Nie, W\., Ramanathan, A\., Xiao, C\., Tang, J\., Guo, H\., and Anandkumar, A\.A text\-guided protein design framework\.*Nature Machine Intelligence*, 7\(4\):580–591, April 2025\.ISSN 2522\-5839\.doi:10\.1038/s42256\-025\-01011\-z\.URL[https://doi\.org/10\.1038/s42256\-025\-01011\-z](https://doi.org/10.1038/s42256-025-01011-z)\.
- Minegishi et al\. \(2026\)Minegishi, G\., Feng, J\., Furuta, H\., Kojima, T\., Iwasawa, Y\., and Matsuo, Y\.Emergent analogical reasoning in transformers\.In*Forty\-third International Conference on Machine Learning*, 2026\.URL[https://openreview\.net/forum?id=GxTkgMBiz8](https://openreview.net/forum?id=GxTkgMBiz8)\.
- Minh et al\. \(2025\)Minh, N\. N\., Baker, A\., Neo, C\., Roush, A\. G\., Kirsch, A\., and Shwartz\-Ziv, R\.Turning up the heat: Min\-p sampling for creative and coherent LLM outputs\.In*The Thirteenth International Conference on Learning Representations*, 2025\.URL[https://openreview\.net/forum?id=FBkpCyujtS](https://openreview.net/forum?id=FBkpCyujtS)\.
- Ostermann et al\. \(2026\)Ostermann, S\., Gurgurov, D\., Baeumel, T\., Hedderich, M\. A\., Lapuschkin, S\., Samek, W\., and Schmitt, V\.From weights to activations: Is steering the next frontier of adaptation?In Liakata, M\., Moreira, V\. P\., Zhang, J\., and Jurgens, D\. \(eds\.\),*Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 29854–29879, San Diego, California, United States, July 2026\. Association for Computational Linguistics\.ISBN 979\-8\-89176\-390\-6\.doi:10\.18653/v1/2026\.acl\-long\.1377\.URL[https://aclanthology\.org/2026\.acl\-long\.1377/](https://aclanthology.org/2026.acl-long.1377/)\.
- Park & Walsh \(2026\)Park, H\. and Walsh, A\.Guiding generative models to uncover diverse and novel crystals via reinforcement learning\.*Nature Machine Intelligence*, pp\. 1–13, 2026\.
- Pillutla et al\. \(2022\)Pillutla, K\., Swayamdipta, S\., Zellers, R\., Thickstun, J\., Welleck, S\., Harchaoui, Z\., et al\.Mauve: Measuring the gap between neural text and human text using divergence frontiers\.*Advances in neural information processing systems*, 2022\.
- Reimers & Gurevych \(2019\)Reimers, N\. and Gurevych, I\.Sentence\-bert: Sentence embeddings using siamese bert\-networks\.In*Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing \(EMNLP\-IJCNLP\)*, pp\. 3982–3992, 2019\.
- Ruan et al\. \(2025\)Ruan, Z\., Li, Y\., Liu, Y\., Chen, Y\., Luo, W\., Li, P\., Liu, Y\., and Chen, G\.G2: Guided generation for enhanced output diversity in LLMs\.In Christodoulopoulos, C\., Chakraborty, T\., Rose, C\., and Peng, V\. \(eds\.\),*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pp\. 14116–14134, Suzhou, China, November 2025\. Association for Computational Linguistics\.ISBN 979\-8\-89176\-332\-6\.doi:10\.18653/v1/2025\.emnlp\-main\.713\.URL[https://aclanthology\.org/2025\.emnlp\-main\.713/](https://aclanthology.org/2025.emnlp-main.713/)\.
- Shen et al\. \(2026a\)Shen, A\., Druckmann, S\., and Zou, J\.Unlocking llm creativity in science through analogical reasoning\.*arXiv preprint arXiv:2605\.11258*, 2026a\.
- Shen et al\. \(2026b\)Shen, Y\., de Sousa, D\. X\., de Andrade Nascimento, R\., Asad, A\., Guo, H\., and Zhu, X\.Understanding large language models’ ability on interdisciplinary research\.In*Proceedings of the 43rd International Conference on Machine Learning \(ICML\)*, Seoul, South Korea, July 2026b\. PMLR\.July 6–11, 2026\.
- Tang et al\. \(2024\)Tang, C\., Liu, J\., Xu, H\., and Huang, L\.Top\-n​σn\\sigma: Not all logits are you need\.*arXiv preprint arXiv:2411\.07641*, 2024\.
- Tian et al\. \(2024\)Tian, Y\., Huang, T\., Liu, M\., Jiang, D\., Spangher, A\., Chen, M\., May, J\., and Peng, N\.Are large language models capable of generating human\-level narratives?In Al\-Onaizan, Y\., Bansal, M\., and Chen, Y\.\-N\. \(eds\.\),*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp\. 17659–17681, Miami, Florida, USA, November 2024\. Association for Computational Linguistics\.doi:10\.18653/v1/2024\.emnlp\-main\.978\.URL[https://aclanthology\.org/2024\.emnlp\-main\.978/](https://aclanthology.org/2024.emnlp-main.978/)\.
- Todd et al\. \(2024\)Todd, E\., Li, M\. L\., Sharma, A\. S\., Mueller, A\., Wallace, B\. C\., and Bau, D\.Function vectors in large language models\.In*The Twelfth International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=AwyxtyMwaG](https://openreview.net/forum?id=AwyxtyMwaG)\.arXiv:2310\.15213\.
- Tong et al\. \(2024\)Tong, X\., Choenni, R\., Lewis, M\., and Shutova, E\.Metaphor understanding challenge dataset for LLMs\.In Ku, L\.\-W\., Martins, A\., and Srikumar, V\. \(eds\.\),*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 3517–3536, Bangkok, Thailand, August 2024\. Association for Computational Linguistics\.doi:10\.18653/v1/2024\.acl\-long\.193\.URL[https://aclanthology\.org/2024\.acl\-long\.193/](https://aclanthology.org/2024.acl-long.193/)\.
- Turney \(2012\)Turney, P\. D\.Domain and function: a dual\-space model of semantic relations and compositions\.*J\. Artif\. Int\. Res\.*, 44\(1\):533–585, May 2012\.ISSN 1076\-9757\.
- Webb et al\. \(2023\)Webb, T\., Holyoak, K\. J\., and Lu, H\.Emergent analogical reasoning in large language models\.*Nature Human Behaviour*, 7\(9\):1526–1541, 2023\.
- Wiher et al\. \(2022\)Wiher, G\., Meister, C\., and Cotterell, R\.On decoding strategies for neural text generators\.*Transactions of the Association for Computational Linguistics*, 10:997–1012, 2022\.
- Wu et al\. \(2016\)Wu, Y\., Schuster, M\., Chen, Z\., Le, Q\. V\., Norouzi, M\., Macherey, W\., Krikun, M\., Cao, Y\., Gao, Q\., Macherey, K\., Klingner, J\., Shah, A\., Johnson, M\., Liu, X\., Łukasz Kaiser, Gouws, S\., Kato, Y\., Kudo, T\., Kazawa, H\., Stevens, K\., Kurian, G\., Patil, N\., Wang, W\., Young, C\., Smith, J\., Riesa, J\., Rudnick, A\., Vinyals, O\., Corrado, G\., Hughes, M\., and Dean, J\.Google’s neural machine translation system: Bridging the gap between human and machine translation\.*CoRR*, abs/1609\.08144, 2016\.URL[http://arxiv\.org/abs/1609\.08144](http://arxiv.org/abs/1609.08144)\.
- Ye et al\. \(2024\)Ye, X\., Wang, A\., Choi, J\., Lu, Y\., Sharma, S\., Shen, L\., Tiyyala, V\. M\., Andrews, N\., and Khashabi, D\.AnaloBench: Benchmarking the identification of abstract and long\-context analogies\.In Al\-Onaizan, Y\., Bansal, M\., and Chen, Y\.\-N\. \(eds\.\),*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp\. 13060–13082, Miami, Florida, USA, November 2024\. Association for Computational Linguistics\.doi:10\.18653/v1/2024\.emnlp\-main\.725\.URL[https://aclanthology\.org/2024\.emnlp\-main\.725/](https://aclanthology.org/2024.emnlp-main.725/)\.
- Zhang et al\. \(2026\)Zhang, Z\., Wang, Z\., Liao, Y\., Chang, Y\., Ding, L\., Luo, G\., Wang, X\., and Wang, H\.Machine learning–driven discovery of optimal designs for water electrolysis devices\.*Science Advances*, 12\(19\):eadz1865, 2026\.
- Zheng et al\. \(2023\)Zheng, L\., Chiang, W\.\-L\., Sheng, Y\., Zhuang, S\., Wu, Z\., Zhuang, Y\., Lin, Z\., Li, Z\., Li, D\., Xing, E\., et al\.Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.*Advances in neural information processing systems*, 36:46595–46623, 2023\.
- Zhu et al\. \(2018\)Zhu, Y\., Lu, S\., Zheng, L\., Guo, J\., Zhang, W\., Wang, J\., and Yu, Y\.Texygen: A benchmarking platform for text generation models\.In*The 41st international ACM SIGIR conference on research & development in information retrieval*, pp\. 1097–1100, 2018\.

## Appendix AAppendix

## Appendix BExperimental Setup

#### Datasets

Experiments are conducted on three analogy datasets:AB\(AnaloBench\_S1\), containing 340 samples;MA\(metaphoric\_analogies\), containing 262 samples; andMUNCH\(munch\_metaphors\), containing 300 samples\. For each source sentence, every model generatesK=10K=10candidate analogies\.

#### Evaluated Models

We evaluate ten large language models \(LLMs\), comprising five proprietary and five open\-weight models\. The proprietary models include Claude Sonnet \(anthropic/claude\-sonnet\-4\.5\), GPT\-5\.2 \(openai/gpt\-5\.2\), Gemini 2\.5 Pro \(google/gemini\-2\.5\-pro\), Gemini 3\.1 Pro \(google/gemini\-3\.1\-pro\-preview\), and Grok 4\.5 \(x\-ai/grok\-4\.5\)\. The open\-weight models consist of Phi\-4 \(microsoft/Phi\-4\-mini\-instruct\), Mistral \(mistralai/Mistral\-7B\-Instruct\-v0\.3\), Gemma 2 \(google/gemma\-2\-9b\-it\), Qwen 3 \(Qwen/Qwen3\-8B\), and Llama 3\.1 \(meta\-llama/Llama\-3\.1\-8B\-Instruct\)\. We access the proprietary models via the OpenRouter API, while the open\-weight models are executed locally in FP16 precision using the Hugging Facetransformerslibrary

## Appendix CExperiment 1 \(a\): Analogy Generation

The objective of this experiment is to evaluate both the diversity and analogy quality of model\-generated sentences under different prompting and decoding strategies\.

Unless otherwise specified, all models use standard sampling with a temperature ofT=0\.7T=0\.7and generateK=10K=10candidate analogies for each input sentence\. We evaluate three prompting strategies corresponding to the Base, Diversity, and Logic prompts described in the Prompts section\.

To further investigate the diversity–quality trade\-off, we additionally evaluate a higher temperature \(T=1\.5T=1\.5\) using Base Prompt combined with four decoding strategies: Top\-kk, Top\-pp, Top\-η​σ\\eta\\sigma, and Min\-pp\.

### C\.1Evaluation Pipeline

#### Generation

For every model and decoding configuration, a JSON file is produced containing the source sentences together with theKKgenerated analogies\.

#### Diversity Metrics

For each input sample, a set of analogies is generated and evaluated using three complementary diversity metrics, which capture semantic, lexical, and vocabulary\-level variation\. The final diversity score is computed as a weighted combination of these metrics:

Dtotal=0\.5​Dcos\+0\.25​DBLEU\+0\.25​DEAD\.D\_\{\\mathrm\{total\}\}=0\.5D\_\{\\mathrm\{cos\}\}\+0\.25D\_\{\\mathrm\{BLEU\}\}\+0\.25D\_\{\\mathrm\{EAD\}\}\.
Average Cosine Distance \(DcosD\_\{\\mathrm\{cos\}\}\)\.Each generated analogy is embedded using the SentenceTransformer modelall\-MiniLM\-L6\-v2\. Pairwise cosine distances are computed between all valid analogy embeddings for each sample, and their average is used as the semantic diversity score\. Higher values indicate greater semantic dispersion among the generated analogies\. Samples containing fewer than two analogies receive a score of zero\.

Div\-BLEU \(DBLEUD\_\{\\mathrm\{BLEU\}\}\)\.Lexical diversity is measured using Self\-BLEU\. Each analogy is treated once as the candidate while all remaining analogies serve as references\. Sentence\-level BLEU is computed using SacreBLEU with exponential smoothing, and Self\-BLEU is obtained by averaging across all candidates\. The diversity score is then defined as

DBLEU=1−Self​\-​BLEU,D\_\{\\mathrm\{BLEU\}\}=1\-\\mathrm\{Self\\mbox\{\-\}BLEU\},such that higher values correspond to lower lexical overlap\.

Expectation\-Adjusted Distinctness \(DEADD\_\{\\mathrm\{EAD\}\}\)\.To measure vocabulary diversity while correcting the length bias of the original Distinct\-nnmetric, we adopt Expectation\-Adjusted Distinctness \(EAD\)\.

For each generation method, the reported diversity statistics correspond to the mean, median, and standard deviation ofDtotalD\_\{\\mathrm\{total\}\}across all evaluated samples\.

#### Analogy Quality

The quality of every generated analogy is independently assessed by a judge LLM using the Analogy Quality Scoring Prompt described in the prompt section in Appendix\. The judge receives the original sentence, the generated analogy, and a formal definition of analogy based on domain mapping theory, and assigns an integer score from 1 \(poor\) to 5 \(excellent\) according to the validity of the mapping, semantic coherence, and overall meaningfulness\. To ensure deterministic evaluation, inference is performed with temperature zero, and only the integer score is extracted from the model output\. Responses outside the valid range are discarded\.

For each input sample, the quality score is computed as the mean of the scores assigned to its generated analogies\. The quality of a generation method \(i\.e\., a specific combination of model, prompt, and decoding strategy\) is then reported as the mean and standard deviation of these sample\-level scores across the entire evaluation set\.

![Refer to caption](https://arxiv.org/html/2608.03233v1/Figures/quality_vs_diversity_scatter.png)Figure 8:Quality–diversity tradeoff for each evaluated language model\. Each point corresponds to a prompting or decoding strategy, positioned according to its mean analogy quality and mean diversity\. The highlighted star denotes the selected operating point, determined by maximizing the combined objectiveQuality×Diversity\\mathrm\{Quality\}\\times\\mathrm\{Diversity\}\.

## Appendix DExperiment 1 \(b\): Domain Extraction

This experiment measures the diversity of conceptual domains employed by language models during analogy generation\. For each generated analogy, GPT\-4o\-mini \(T=0\.0T=0\.0\) extracts its source and target domains in JSON format using the Domain Extraction Prompt described in the Prompts section\.

#### Domain Diversity

The extracted target\-domain labels are treated as text samples and evaluated using the same diversity metrics described in Experiment 1\(a\)\. This allows us to quantify the semantic diversity of the conceptual domains independently of the linguistic variation in the generated analogies\. Figure[9](https://arxiv.org/html/2608.03233#A4.F9)compares domain diversity with sentence diversity across all evaluated generation methods\. The strong positive correlation indicates that methods producing more diverse analogies also tend to explore a broader range of conceptual domains\.

![Refer to caption](https://arxiv.org/html/2608.03233v1/Figures/domain_vs_sentence_diversity.png)Figure 9:Relationship between sentence diversity and domain diversity across all evaluated generation methods\. Each point corresponds to a model–method configuration, with colors indicating the prompting strategy and marker shapes distinguishing open\-source and closed\-source models\.

## Appendix EExperiment 2: MAUVE Similarity

This experiment measures distributional similarity between proprietary language models using MAUVE\.

Only generations produced using the Base prompt in the Prompts section withT=0\.7T=0\.7are considered\. Outputs from every model are pooled across all datasets before computing pairwise MAUVE scores\. GPT\-2 Large hidden representations are used for featurization, and the resulting pairwise similarities are presented as a heatmap\.

## Appendix FExperiment 3: Memorization Test

This experiment investigates whether proprietary models have memorized examples from the AnaloBench dataset\.

For every prefix ratio

r∈\{0\.1,0\.2,0\.4,0\.8\},r\\in\\\{0\.1,0\.2,0\.4,0\.8\\\},
the firstrrfraction of the reference completion is provided to the model\.

Using the Base Prompt described in Prompts section, models are instructed to reproduce the continuation from the provided prefix using few\-shot examples\. Performance is evaluated using Token Accuracy, Character Accuracy, Exact Match \(EM\), and Contains Accuracy\. Higher Exact Match scores indicate stronger evidence of memorization\.

## Appendix GExperiment 4: LLM\-based Clustering

This experiment investigates the semantic organization of generated analogies through LLM\-based clustering\.

For every source sentence, the correspondingKKgenerated analogies are submitted to GPT\-4o\-mini, which groups them into semantic categories represented as JSON objects using the Clustering prompt described in the Prompts section in Prompts section\. The prompt instructs the model to cluster analogies according to their conceptual domains while avoiding lexical variations and synonym duplication\.

#### Cluster\-Linkage Analysis

Cluster embeddings are computed using SentenceTransformer \(all\-MiniLM\-L6\-v2\)\. For each pair of clusters within a sample, we compute the single, complete, and average linkage\. The linkage score for a given method is obtained by averaging the corresponding linkage values across all samples\. Samples containing only a single cluster are assigned a linkage score of 0\. Additionally, we report the average number of semantic clusters per source sentence\.

## Appendix HExperiment 5: Fragile Region Identification

#### Noise Blur

This experiment investigates whether analogy generation is associated with specific regions of transformer models or is distributed across the network\. For each input sample, the model generatesKKanalogies by applying hidden\-state perturbations independently at transformer layers located at 10% depth intervals, from 10% to 100% of the network\. The generated analogies are evaluated using the same diversity and quality metrics described in Experiment 1\(a\), yielding a diversity score and an analogy quality score for every layer and sample\. These scores are subsequently used to identify the layers most strongly associated with the analogy\-making process through a perturbation\-based analysis\.

#### Layer Evaluation

For each input sample, every transformer layer is represented as a point in the quality–diversity space, where quality is minimized and diversity is maximized\. The Pareto frontier is then computed by retaining only the non\-dominated layers\. Among the Pareto\-optimal candidates, a single layer is selected by maximizing the combined objective

\(5−Quality\)×Diversity,\(5\-\\mathrm\{Quality\}\)\\times\\mathrm\{Diversity\},which favors layers achieving both low\-quality and diverse analogy generation\. This procedure is repeated independently for every sample\. Finally, for each language model, we report the frequency with which each layer is selected as the optimal Pareto layer across the evaluation set, providing an estimate of the model regions that are most consistently sensitive to hidden\-state perturbations during analogy generation\.

### H\.1Complete Results

This section presents the complete experimental results in Table[1](https://arxiv.org/html/2608.03233#A8.T1)\.

Model NameMethodABMAMUNCHDiv\.↑\\uparrowQual\.↑\\uparrowClus\. Linkage\#Clus\.Div\.↑\\uparrowQual\.↑\\uparrowClus\. Linkage\#Clus\.Div\.↑\\uparrowQual\.↑\\uparrowClus\. Linkage\#Clus\.Llama\-3\.1\- 8B\-instructStd\. Prompt0\.3554\.660\.3512\.100\.3784\.060\.4212\.400\.3484\.330\.2861\.97Div\. Prompt0\.5644\.180\.6393\.240\.4953\.710\.5772\.960\.5383\.800\.5903\.28Top\-kk0\.5844\.460\.4322\.180\.5823\.830\.5502\.680\.5714\.180\.3952\.16Top\-pp0\.6134\.400\.4502\.240\.6143\.700\.5712\.700\.6044\.040\.4462\.23Top\-η​σ\\eta\\sigma0\.4464\.640\.3592\.090\.4554\.010\.4592\.570\.4304\.300\.3472\.18Min\-pp0\.5424\.530\.4352\.310\.5433\.890\.5022\.630\.5234\.220\.3942\.18Entropy0\.3934\.360\.3762\.170\.4653\.540\.5262\.700\.3843\.880\.3952\.33G20\.6011\.070\.4742\.030\.6201\.100\.5032\.060\.6101\.070\.3851\.73Gemma2\-9B\-itStd\. Prompt0\.3744\.440\.4202\.150\.4223\.770\.4932\.380\.3994\.130\.4102\.27Div\. Prompt0\.5643\.850\.6903\.090\.5143\.460\.5822\.930\.5903\.370\.6863\.54Top\-kk0\.5844\.260\.5362\.530\.6093\.570\.6062\.910\.5933\.930\.5092\.62Top\-pp0\.5494\.290\.5112\.510\.5923\.590\.5812\.870\.5723\.950\.5032\.72Top\-η​σ\\eta\\sigma0\.5064\.360\.4962\.320\.5563\.660\.5612\.710\.5234\.060\.4932\.66Min\-pp0\.4934\.380\.4712\.380\.5343\.670\.5602\.610\.5114\.030\.4542\.60Entropy0\.5893\.400\.7212\.820\.6192\.880\.7512\.970\.5922\.820\.7432\.77G20\.5701\.080\.4612\.050\.5731\.120\.5062\.030\.5991\.090\.4661\.90Std\. Prompt0\.4174\.570\.3992\.300\.4233\.970\.4502\.330\.3994\.310\.3252\.15Phi\-4\-mini\- instructDiv\. Prompt0\.5434\.310\.5963\.090\.5153\.710\.6073\.280\.5503\.850\.6103\.48Top\-kk0\.6414\.190\.5202\.480\.6283\.640\.5562\.700\.6164\.000\.4532\.34Top\-pp0\.6294\.240\.4982\.430\.6283\.600\.5702\.750\.6094\.000\.4482\.42Top\-η​σ\\eta\\sigma0\.6594\.030\.5232\.570\.6383\.490\.5742\.860\.6323\.890\.4942\.49Min\-pp0\.5454\.450\.4262\.340\.5523\.830\.5172\.740\.5264\.200\.3792\.29Entropy0\.6034\.240\.4932\.010\.5603\.620\.4772\.160\.5773\.790\.4772\.04G20\.4871\.000\.1281\.270\.4811\.000\.1671\.420\.4761\.000\.1051\.21Std\. Prompt0\.3254\.560\.3171\.940\.3514\.020\.3942\.180\.3074\.370\.2301\.78Mistral\-7B\- instruct\-v0\.3Div\. Prompt0\.5184\.160\.6043\.060\.4683\.680\.5422\.950\.5203\.660\.6063\.54Top\-kk0\.5384\.490\.4042\.240\.5453\.900\.5052\.570\.5004\.230\.3022\.00Top\-pp0\.4974\.530\.3822\.130\.5183\.930\.4622\.410\.4704\.290\.3362\.05Top\-η​σ\\eta\\sigma0\.3684\.580\.3361\.990\.3864\.040\.3982\.220\.3434\.380\.2511\.83Min\-pp0\.4414\.550\.3472\.060\.4604\.010\.4542\.400\.4174\.320\.3061\.91Entropy0\.5154\.200\.3871\.880\.5653\.650\.5492\.490\.4923\.890\.3591\.97G20\.3891\.000\.1481\.380\.4161\.010\.2151\.580\.4181\.000\.1311\.30Std\. Prompt0\.1604\.610\.2031\.590\.1724\.030\.2291\.650\.1704\.260\.2061\.63Qwen3\-8BDiv\. Prompt0\.2824\.210\.4072\.240\.2633\.720\.3632\.030\.2963\.750\.3982\.31Top\-kk0\.2854\.560\.3112\.060\.3143\.980\.3612\.190\.3004\.220\.3092\.14Top\-pp0\.2664\.600\.3052\.010\.2944\.010\.3722\.230\.2814\.210\.3042\.07Top\-η​σ\\eta\\sigma0\.2634\.580\.2901\.890\.2854\.020\.3532\.110\.2744\.240\.3052\.05Min\-pp0\.2524\.600\.2861\.930\.2694\.000\.3172\.000\.2614\.240\.3062\.08Entropy0\.3773\.470\.3341\.810\.3982\.950\.3621\.780\.3943\.040\.3001\.63G20\.2871\.030\.1771\.420\.2921\.040\.2221\.610\.2901\.030\.1741\.37

Table 1:Evaluation results on three datasets using different generation perturbation methods\. AB, MA stands for AnaloBench\(Ye et al\.,[2024](https://arxiv.org/html/2608.03233#bib.bib38)\)and Meterphoric Analogy\(Tong et al\.,[2024](https://arxiv.org/html/2608.03233#bib.bib33)\), respectively\. The best results among the perturbation methods areunderlined, and the best results across different models arebolded\.

## Appendix IPrompts

This section presents all prompt templates used throughout the experiments\. In every template, text enclosed in\{curly\_braces\}denotes a placeholder that is replaced with the corresponding input at runtime\.

Generation Base PromptGenerate a sentence that is analogous to the sentence below\. Directly output the answer with no explanation\.Sentence: \{source\_story\} Answer:

Generation Logic PromptGenerate a sentence that follows the same relational structure of the sentence below, but in a completely different domain\. Directly output the sentence with no explanation\.Sentence: \{source\_story\} Answer:

Generation Diversity PromptGenerate a sentence that is analogous to the sentence below\. Be as creative and diverse as possible in your choice of domain\. Directly output the answer with no explanation\.Sentence: \{source\_story\} Answer:

Analogy Quality Scoring PromptRead the sentence and the generated analogy, and give it a score between 1 and 5 based on whether the generated analogy is a valid analogy for the sentence\. The official definition of analogy can be found below: "An analogy is composed of object mappings and shared relations\. Object mappings establish correspondences between equivalent objects across different domains, while shared relations represent equivalent relationships between these objects in both domains\." Think carefully before making your verdict\. When assigning the score, also evaluate whether the generated analogy is coherent and meaningful\. Sentence: \{sentence\} Generated Analogy: \{analogy\} Respond ONLY with a single integer from 1 to 5\.

Domain Extraction PromptYou are analyzing an analogy to identify its conceptual structure it establishes\. An analogy maps objects from a source domain to corresponding objects in a different target domain while preserving their relationships\. \*\*Input sentence:\*\* \{\{input\_sentence\}\} \*\*Provided analogy:\*\* \{\{input\_analogy\}\} \*\*Your task:\*\* Analyze the provided analogy and identify the conceptual domains involved\. Do NOT generate a new analogy or modify the provided one\. Only extract the conceptual structure already present\. \*\*Rules for extracting domains:\*\* 1\. Granularity: Use broad, standard fields of activity, industries, or areas of knowledge\. 2\. No Specific Entities: Do NOT use specific events, projects, or historical names\. 3\. Format: Use lowercase, singular noun phrases\. Max 3 words per domain\. 4\. Language: Output the domains in English\. \*\*Return a JSON object with:\*\* 1\. "source\_domain": The primary conceptual or real\-world domain represented in the original input sentence\. 2\. "target\_domain": The conceptual or real\-world domain represented in the provided analogy\. Return ONLY the JSON object, with no additional text\.

Clustering PromptYou are given \{K\} sentences\. Your task is to group them into semantic categories based on their underlying conceptual domain\. \*\*Task\*\* Analyze the meaning of each sentence and assign each sentence to exactly one category representing its primary conceptual domain\. \*\*Category Guidelines\*\* \* Categories must represent broad semantic concepts, not the exact wording used in the sentences\. \* Normalize synonyms and closely related concepts into the same category\. \* Do not create separate categories for lexical variations that refer to the same underlying domain\. Examples: \- "recipe creation", "recipe invention", "recipe development" \-\> same category \- "car repair" and "vehicle maintenance" \-\> same category \- "writing a book" and "book creation" \-\> same category \* Prefer abstract domain names that describe the shared concept\. \* Category names should not simply copy phrases from individual sentences\. \*\*Classification Rules\*\* \* Analyze the meaning of each sentence, not just keywords\. \* Identify the main subject/domain of the sentence\. \* Group sentences together when they concern the same conceptual area, even if they use different wording or describe different scenarios\. \* If a sentence is an analogy, classify it according to the domain of the objects being compared, not according to the analogy structure\. \* Do not group sentences from different domains just because they share similar relationships, emotions, or logical patterns\. \*\*Category Creation Rules\*\* \* The number of categories is not predefined\. \* Create a new category only when the underlying conceptual domain is genuinely different\. \* Before creating a new category, check whether an existing category can describe the sentence after semantic normalization\. \* Prefer fewer categories when categories would only differ by wording\. \* Prefer more categories when the real\-world domains differ\. \*\*Constraints\*\* \* Every sentence must belong to exactly one category\. \* Do not duplicate sentence indices\. \* Do not create categories that differ only by synonyms\. \*\*Input\*\* A numbered list of sentences: \{sentences\} \*\*Output Requirements\*\* Return ONLY a valid JSON object\. \* Keys must be concise category names representing semantic domains\. \* Values must be arrays containing the sentence indices belonging to that category\. \* Do not include explanations or additional text\. Example output: \{ "Recipe Development": \[1, 2, 3\], "Vehicle Maintenance": \[4, 5\] \}