Teaching Through Analogies: A Modular Pipeline for Educational Analogy Generation
Summary
This paper presents a modular pipeline for educational analogy generation, decomposing the task into four stages and evaluating 12 LLMs and 7 embedding models. Results show that sub-concept grounding improves explanation quality and retrieval precision, with a novel LLM-as-a-judge evaluation validated against human annotations.
arXiv:2605.24211v1 Announce Type: new
Abstract: Analogies help learners understand unfamiliar concepts by relating them to known concepts. Despite recent advances, large language models (LLMs) continue to struggle to generate analogies of comparable quality to those produced by humans. We present a modular pipeline for educational analogy generation, decomposing the task into four stages: source finding, sub-concept generation, explanation generation, and evaluation. Grounded in Structure Mapping Theory, the pipeline enables systematic, stage-by-stage analysis of how model choice and input configuration affect analogy quality. We evaluate 12 state-of-the-art LLMs across six model families on two datasets with structured sub-concept annotations (SCAR and ParallelPARC), alongside seven embedding models for closed-setting retrieval. Our results show that sub-concepts substantially improve explanation quality and closed setting retrieval precision but provide limited benefit in open-ended source generation. We further introduce an LLM-as-a-judge evaluation methodology and validate its scoring against human annotations from seven annotators, finding that Claude Sonnet 4.6 aligns more reliably with human rankings than with fine-grained absolute scores. Taken together, our findings reveal cross-stage interactions that isolated studies cannot capture, and highlight sub-concept grounding as a key driver of analogy quality generation.
View Cached Full Text
Cached at: 05/26/26, 09:00 AM
# A Modular Pipeline for Educational Analogy Generation
Source: [https://arxiv.org/html/2605.24211](https://arxiv.org/html/2605.24211)
## Teaching Through Analogies: A Modular Pipeline for Educational Analogy Generation
Ekaterina Kochmar Mohamed bin Zayed University of Artificial Intelligence, UAE \{mariam\.barakat, ekaterina\.kochmar\}@mbzuai\.ac\.ae
###### Abstract
Analogies help learners understand unfamiliar concepts by relating them to known concepts\. Despite recent advances, large language models \(LLMs\) continue to struggle to generate analogies of comparable quality to those produced by humans\. We present a modular pipeline for educational analogy generation, decomposing the task into four stages: source finding, sub\-concept generation, explanation generation, and evaluation\. Grounded in Structure Mapping Theory, the pipeline enables systematic, stage\-by\-stage analysis of how model choice and input configuration affect analogy quality\. We evaluate 12 state\-of\-the\-art LLMs across six model families on two datasets with structured sub\-concept annotations \(SCAR and ParallelPARC\), alongside seven embedding models for closed\-setting retrieval\. Our results show that sub\-concepts substantially improve explanation quality and closed setting retrieval precision but provide limited benefit in open\-ended source generation\. We further introduce an LLM\-as\-a\-judge evaluation methodology and validate its scoring against human annotations from seven annotators, finding that Claude Sonnet 4\.6 aligns more reliably with human rankings than with fine\-grained absolute scores\. Taken together, our findings reveal cross\-stage interactions that isolated studies cannot capture, and highlight sub\-concept grounding as a key driver of analogy quality generation\.
Teaching Through Analogies: A Modular Pipeline for Educational Analogy Generation
Mariam Barakat and Ekaterina KochmarMohamed bin Zayed University of Artificial Intelligence, UAE\{mariam\.barakat, ekaterina\.kochmar\}@mbzuai\.ac\.ae
## 1Introduction
Figure 1:Illustrative example of a system analogy between theatomand thesolar system, showing the source and target systems and their aligned sub\-concepts\.As educational systems evolve in the age of AI, memorisation is no longer sufficient\. Instead, students are required to be able to reason across domains and understand underlying structures – skills that are foundational to critical thinking in a rapidly changing world\. One of the effective ways to develop these abilities is through analogies\(Erikssonet al\.,[2024](https://arxiv.org/html/2605.24211#bib.bib25)\), which facilitate learning by mapping unfamiliar knowledge \(target system\) to prior knowledge \(source system\) through relational structures\(Yeet al\.,[2024](https://arxiv.org/html/2605.24211#bib.bib6); Czinczollet al\.,[2022](https://arxiv.org/html/2605.24211#bib.bib12); Bettin and Ott,[2023](https://arxiv.org/html/2605.24211#bib.bib5)\)\. Even when a significant cognitive gap exists, intermediate “bridging” analogies can guide students toward comprehension\(Brown and Clement,[1989](https://arxiv.org/html/2605.24211#bib.bib32)\)\. In this work, we frame analogy generation as a modular pipeline to enable systematic analysis and improvement of each stage of the process\.
Following previous research\(Yuanet al\.,[2023a](https://arxiv.org/html/2605.24211#bib.bib4)\), we use the termsystemrather thanconceptto emphasise that analogies map multiple components\. We refer to these constituent properties assub\-conceptsthroughout this paper, using "sub\-” to emphasise that they are components of the larger system rather than standalone concepts\.
As shown in Figure[1](https://arxiv.org/html/2605.24211#S1.F1), explaining theatom\(target system\) through thesolar system\(source system\) requires the explained mapping of individual sub\-concepts: thenucleusmaps to thesun,electronsmap toplanets, andenergy levelsmap toorbital distances\.
Automatically generating high\-quality analogies remains an open challenge, as poorly constructed analogies can introduce or reinforce misconceptions through incorrect mappings or oversimplification of underlying concepts\(Zhouet al\.,[2025](https://arxiv.org/html/2605.24211#bib.bib19); Shaoet al\.,[2025](https://arxiv.org/html/2605.24211#bib.bib18)\)\.
Despite recent progress, analogy generation research remains fragmented: prior work addresses retrieval, generation, or evaluation in isolation, without connecting them into a unified pipeline\(Yuanet al\.,[2023a](https://arxiv.org/html/2605.24211#bib.bib4),[b](https://arxiv.org/html/2605.24211#bib.bib13); Zhaiet al\.,[2024](https://arxiv.org/html/2605.24211#bib.bib15); Sultanet al\.,[2024](https://arxiv.org/html/2605.24211#bib.bib33)\)\. Source finding relies predominantly on embedding\-based retrieval over closed candidate pools, favouring surface similarity over structural alignment\(Yuanet al\.,[2023a](https://arxiv.org/html/2605.24211#bib.bib4); Czinczollet al\.,[2022](https://arxiv.org/html/2605.24211#bib.bib12); Jiayanget al\.,[2023](https://arxiv.org/html/2605.24211#bib.bib14)\), while sub\-concept annotations in SCAR\(Yuanet al\.,[2023a](https://arxiv.org/html/2605.24211#bib.bib4)\)and ParallelPARC\(Sultanet al\.,[2024](https://arxiv.org/html/2605.24211#bib.bib33)\)are used for dataset construction rather than to guide generation\. Evaluation remains under\-addressed, as neither lexical metrics nor LLM\-as\-a\-judge approaches align reliably with human judgments\(Zhaiet al\.,[2024](https://arxiv.org/html/2605.24211#bib.bib15); Bhavyaet al\.,[2024](https://arxiv.org/html/2605.24211#bib.bib9)\), and most studies evaluate GPT\-4 variants exclusively\(Yuanet al\.,[2023a](https://arxiv.org/html/2605.24211#bib.bib4); Kimet al\.,[2023](https://arxiv.org/html/2605.24211#bib.bib1); Sultanet al\.,[2024](https://arxiv.org/html/2605.24211#bib.bib33); Shaoet al\.,[2025](https://arxiv.org/html/2605.24211#bib.bib18); Liet al\.,[2024](https://arxiv.org/html/2605.24211#bib.bib7)\), leaving cross\-model generalisability unexplored\.
In this paper, we make three contributions: \(1\) a modular pipeline for educational analogy generation, grounded in Structure Mapping Theory\(Gentner,[1981](https://arxiv.org/html/2605.24211#bib.bib21)\), decomposed into source finding, sub\-concept generation, explanation generation, and evaluation; \(2\) a systematic cross\-model evaluation across 12 LLMs from six families, finding that sub\-concepts improve retrieval and explanation quality but offer limited benefit in open\-ended source generation; and \(3\) a closed\-to\-open evaluation methodology, moving from controlled evaluation against gold annotations on SCAR\(Yuanet al\.,[2023a](https://arxiv.org/html/2605.24211#bib.bib4)\)and ParallelPARC\(Sultanet al\.,[2024](https://arxiv.org/html/2605.24211#bib.bib33)\)to LLM\-generated candidates with baseline, embedding\-based, and re\-ranking selection strategies\.
Each stage in our proposed pipeline corresponds to a research question:
RQ1: Source Finding\.Can LLMs identify suitable source systems for a given target in both closed and open settings, moving beyond predefined candidate pools?
RQ2: Sub\-concept Generation\.Can LLMs reliably generate sub\-concept mappings given a target\-source pair, and at which pipeline stages do sub\-concepts improve performance?
RQ3: Explanation Generation\.How well can LLMs produce full analogical explanations, and what input configurations yield the highest quality?
RQ4: Evaluation\.How well does LLM\-as\-a\-judge evaluation align with human judgments of analogy quality? Moreover, which generation model is the best at source finding in open settings?
## 2Related Work
#### Source Finding\.
Prior approaches to source finding rely primarily on embedding\-based retrieval within closed candidate pools\.Yuanet al\.\([2023a](https://arxiv.org/html/2605.24211#bib.bib4)\)use greedy and Kuhn\-Munkres matching but focus on mapping given concepts rather than discovering new sources\. Similarly,Yuanet al\.\([2023b](https://arxiv.org/html/2605.24211#bib.bib13)\)employ embedding similarity, but focus only on simple one\- or two\-relation analogies\.Jiayanget al\.\([2023](https://arxiv.org/html/2605.24211#bib.bib14)\)show that LLMs are easily distracted by surface similarity, achieving only 30% accuracy compared to 85% for humans, whileYeet al\.\([2024](https://arxiv.org/html/2605.24211#bib.bib6)\)report substantial performance gaps on longer analogies\.Liet al\.\([2024](https://arxiv.org/html/2605.24211#bib.bib7)\)further show that free generation outperforms embedding\-based retrieval for analogies in the history domain, suggesting fundamental limitations of similarity\-based methods\. However, open\-ended source discovery, where models must identify suitable sources without predefined candidates, remains largely unexplored\.
#### Sub\-concept Generation\.
Datasets such as SCAR\(Yuanet al\.,[2023a](https://arxiv.org/html/2605.24211#bib.bib4)\)and ParallelPARC\(Sultanet al\.,[2024](https://arxiv.org/html/2605.24211#bib.bib33)\)provide rich sub\-concept annotations linking source and target systems\. However, these annotations are primarily used for dataset construction rather than as inputs to generation models\.Yuanet al\.\([2023a](https://arxiv.org/html/2605.24211#bib.bib4)\)show that sub\-concept matching is critical for analogy quality, yet models achieve only 62% accuracy compared to 86% for humans, even when sub\-concepts on both sides are provided\. Moreover, evaluation is restricted to closed settings with fully specified inputs\. The ability of LLMs to generate sub\-concepts in open settings and their impact across pipeline stages has not been studied\.
#### Explanation Generation\.
A well\-matched source and accurate sub\-concept mappings are necessary but not sufficient, as the explanation is what ultimately determines whether a learner grasps the analogy\. Producing explanations that capturewhyan analogy holds remains particularly challenging\.Zhaiet al\.\([2024](https://arxiv.org/html/2605.24211#bib.bib15)\)separate concept generation and explanation generation, finding that while LLMs approach human performance on concept generation, they struggle to produce explanations with sufficient relational depth\.Bhavyaet al\.\([2024](https://arxiv.org/html/2605.24211#bib.bib9)\)similarly report that model\-generated explanations often lack coherence or drift from the intended analogy\. Overall, explanation generation remains underexplored, particularly regarding how input representations affect quality\.
#### Evaluation\.
Evaluating analogy quality remains challenging\. Lexical metrics such as ROUGE and BLEURT fail to capture relational soundness\(Zhaiet al\.,[2024](https://arxiv.org/html/2605.24211#bib.bib15)\), while more structured evaluation frameworks still fall short of fully capturing analogy quality\(Bhavyaet al\.,[2024](https://arxiv.org/html/2605.24211#bib.bib9)\)\. Human evaluation, although more reliable, is costly\(Jiayanget al\.,[2023](https://arxiv.org/html/2605.24211#bib.bib14); Czinczollet al\.,[2022](https://arxiv.org/html/2605.24211#bib.bib12)\)\. LLM\-as\-a\-judge methods offer a scalable alternative\(Zhaiet al\.,[2024](https://arxiv.org/html/2605.24211#bib.bib15)\), but their alignment with human judgments in analogy\-specific settings remains mediocre\.
## 3Methodology
Figure 2:Overview of the proposed modular pipeline for educational analogy generation\. Given a target system with its sub\-concepts and background description, the framework proceeds through four stages: \(1\) source finding, where a pool of candidate sources is generated and selected; \(2\) sub\-concept generation; \(3\) explanation generation; and \(4\) evaluation using an external LLM\-as\-a\-judge\. Each stage operates under input configurations tailored to its requirements, as determined by our empirical results\. The output illustrates the final generated analogy, including aligned sub\-concepts, explanation, and evaluation scores\.We present a systematic evaluation of the proposed modular pipeline for educational analogy generation, analysing how model choice and input configuration influence performance at each stage\. The pipeline is decomposed into four stages aligned with our research questions: source finding \(RQ1\), sub\-concept generation \(RQ2\), explanation generation \(RQ3\), and evaluation \(RQ4\)\. Figure[2](https://arxiv.org/html/2605.24211#S3.F2)provides an overview of the experimental setup\. To illustrate the full pipeline, consider the example in Figure[1](https://arxiv.org/html/2605.24211#S1.F1), where the target systematomis explained using the sourcesolar system\. In the source\-finding stage, candidate source systems are retrieved based on their similarity to the target\. Next, sub\-concept generation aligns components such asnucleus→\\rightarrowsunandelectrons→\\rightarrowplanets\. The explanation generation stage then produces a natural\-language explanation linking these correspondences\. Finally, the analogy is evaluated based on coherence, mapping soundness, and explanatory power\. This example demonstrates how each stage contributes to the final analogy\.
### 3\.1Datasets
We use two datasets with explicit sub\-concept annotations, enabling controlled evaluation across pipeline stages\.111Throughout the paper, we adopt the field names from SCAR for clarity, and align corresponding fields from other datasets to match this structure\.
SCAR\(Yuanet al\.,[2023a](https://arxiv.org/html/2605.24211#bib.bib4)\)comprises 400 system\-level analogy instances across 13 scientific domains, created by human annotators\. Each instance pairs a target system with a source system and is annotated with background descriptions, multiple sub\-concept mappings, and explanations for these mappings \(1,614 total annotated mappings\)\. A representative example is provided in Appendix[A](https://arxiv.org/html/2605.24211#A1)\.
ParallelPARC\(Sultanet al\.,[2024](https://arxiv.org/html/2605.24211#bib.bib33)\)extends the ProPara dataset\(Tandonet al\.,[2018](https://arxiv.org/html/2605.24211#bib.bib34)\)by transforming scientific\-process descriptions into analogy pairs through a pipeline of LLM generation, automatic labelling, and human validation\. It introduces 310 gold analogies, each annotated with relational alignments between corresponding properties in the format\(entity, verb, entity\) mapped to \(entity, verb, entity\)\. Please refer to Appendix[1](https://arxiv.org/html/2605.24211#LST1)for an example\.
### 3\.2Models
#### Closed Setting\.
For source retrieval in the closed setting, we evaluate a diverse set of embedding models: OpenAI’sembedding\-smallandembedding\-large,222[https://platform\.openai\.com/docs/guides/embeddings](https://platform.openai.com/docs/guides/embeddings)E5\-large,333[https://huggingface\.co/intfloat/e5\-large](https://huggingface.co/intfloat/e5-large)BGE\-large,444[https://huggingface\.co/BAAI/bge\-large\-en](https://huggingface.co/BAAI/bge-large-en)MPNet\-base,555[https://huggingface\.co/sentence\-transformers/all\-mpnet\-base\-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2)MiniLM\-L6,666[https://huggingface\.co/sentence\-transformers/all\-MiniLM\-L6\-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)andSPECTER\.777[https://huggingface\.co/allenai/specter](https://huggingface.co/allenai/specter)These models cover complementary paradigms: proprietary general\-purpose embeddings \(OpenAI\), open\-source retrieval\-oriented models \(E5\-large,BGE\-large\), sentence\-similarity baselines \(MPNet\-base,MiniLM\-L6\), and a domain\-specialised scientific encoder \(SPECTER\), enabling comparison across models that differ in scale, training objective, and domain specificity\.
#### Open Setting\.
For open\-ended generation across all three stages, we use theDSPyframework\(Khattabet al\.,[2022](https://arxiv.org/html/2605.24211#bib.bib29),[2024](https://arxiv.org/html/2605.24211#bib.bib28)\)for consistent input/output interfaces\. We evaluate 12 LLMs across six families: OpenAI’sGPT\-4\.1\(mini, nano\),GPT\-OSS\(20B, 120B\), Meta’sLLaMA 3\.1\(8B, 70B, 405B\) Instruct, Google’sGemini 2\.5 Flash\-Lite, xAI’sGrok\-4\-Fast,DeepSeek\-R1, and Alibaba’sQwen3\(14B, 32B\)\. Models were selected based on OpenRouter’s public rankings in academia and science,888[https://openrouter\.ai/rankings](https://openrouter.ai/rankings)balancing benchmark performance and cost\-efficiency\. All models were run at temperature 0\.2 for consistency\. Models were accessed via OpenAI, OpenRouter, and DeepInfra APIs\.999OpenAI:[https://openai\.com](https://openai.com/), OpenRouter:[https://openrouter\.ai](https://openrouter.ai/), DeepInfra:[https://deepinfra\.com](https://deepinfra.com/)
### 3\.3Source Finding \(RQ1\): Pool&\\&Selection
#### Closed Setting: Embedding\-based Retrieval\.
Following prior work\(Czinczollet al\.,[2022](https://arxiv.org/html/2605.24211#bib.bib12); Yuanet al\.,[2023a](https://arxiv.org/html/2605.24211#bib.bib4)\), we embed all unique source systems from SCAR and ParallelPARC using the seven models from Section[3\.2](https://arxiv.org/html/2605.24211#S3.SS2)and retrieve the top\-20 candidates per target via cosine similarity\. We test four input configurations with increasing context: \(1\)target onlyto assess surface\-level retrieval; \(2\)\+backgroundto test whether definitional context helps; \(3\)\+sub\-conceptsto assess the role of structural alignment; and \(4\)sub\-concepts & backgroundto evaluate the effect of combining both\.
To go beyond similarity\-based ranking, we introduce an LLM reranking stage: the top\-10 candidates from the best\-performing embedding configuration are passed toLLaMA\-3\.1\-70B\-Instruct, which selects the top\-3 \(Appendix[B](https://arxiv.org/html/2605.24211#A2)\)\. We evaluate five reranking configurations: \(1\)target onlyas a baseline; \(2\)\+backgroundto test definitional context; \(3\)\+gold sub\-conceptsto assess structural alignment with gold annotations; \(4\)\+sub\-concepts & backgroundto combine both signals; and \(5\)\+generated sub\-conceptsto test whether generated sub\-concepts can substitute for gold annotations in downstream selection \(see Subsection[3\.4](https://arxiv.org/html/2605.24211#S3.SS4)\)\.
We evaluate retrieval usingHit@K\(Eq\.[1](https://arxiv.org/html/2605.24211#S3.E1)\), which measures whether at least one gold source appears in the top\-KKresults\. This evaluation directly measures the quality of the retrieval component by assessing whether embedding models successfully retrieve appropriate source systems for a given target\.
Hit@K=1N∑i=1N𝕀\(ranki≤K\)\\text\{Hit@K\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbb\{I\}\(\\text\{rank\}\_\{i\}\\leq K\)\(1\)
#### Open Setting: LLM\-based Generation\.
In the open setting, each of the 12 LLMs generates 20 ranked candidate sources from scratch, evaluated on SCAR only due to its educational relevance and computational constraints\. Following closed\-setting results, we test two input configurations: \(1\)target onlyand \(2\)\+sub\-concepts\.
Since multiple candidates are generated, we evaluate three selection strategies: \(1\)baseline top\-1, selecting the first output from an LLM; \(2\)embedding\-based top\-1, selecting the candidate with the highest cosine similarity to the target usingtext\-embedding\-3\-small, strong in closed\-setting retrieval and cost\-efficient; and \(3\)re\-ranker top\-1, applying the LLM\-based reranking framework from Subsection[3\.3](https://arxiv.org/html/2605.24211#S3.SS3.SSS0.Px1)with thename&\\&subconceptsconfiguration, selected based on its strong performance in the closed setting\.
We report bothexact Hit@K\(Eq\.[1](https://arxiv.org/html/2605.24211#S3.E1)\) andsemantic Hit@K\. A generated source is considered correct if its embedding similarity to any gold source exceeds 0\.374, corresponding to the upper tertile of pairwise gold similarities in SCAR \(see Appendix[C](https://arxiv.org/html/2605.24211#A3)for more details\)\. To avoid evaluation bias, we useall\-MiniLM\-L6\-v2for semantic Hit@K scoring rather thantext\-embedding\-3\-small, which is used in the embedding\-based Top\-1 selection step\. Using the same model for both selection and evaluation might unfairly favour that configuration over others\.
### 3\.4Sub\-concept Generation \(RQ2\)
Before tackling the harder generation task, we revisit sub\-concept matching – where models align given sub\-concepts rather than produce new ones – to establish how well current LLMs capture structural correspondences and how far the field has advanced sinceYuanet al\.\([2023a](https://arxiv.org/html/2605.24211#bib.bib4)\)\. Figure[3](https://arxiv.org/html/2605.24211#S3.F3)highlights the distinction between the two tasks\.
Figure 3:Sub\-concept matching and generation stage \(RQ2\)\. Given a target system \(e\.g\.,atom\) and a source system \(e\.g\.,solar system\), the model performs sub\-concept matching using gold sub\-concepts, and sub\-concept generation when source sub\-concepts are unavailable, producing aligned sub\-concept pairs\.#### Sub\-concept Matching\.
We re\-evaluate sub\-concept matching on SCAR, using its reported system accuracy \(62% for GPT\-4 vs\. 86% for human annotators\) as a reference point\. Models receive randomly shuffled lists of target and source sub\-concepts and must identify the correct correspondences\. Then, we test two scenarios: \(1\)without background, where models rely only on sub\-concept labels to recognise structural alignments; and \(2\)with background\(see Listing[1](https://arxiv.org/html/2605.24211#LST1)in Appendix[A](https://arxiv.org/html/2605.24211#A1)\)\. We compare performance across the 12 LLMs to determine if the gap between model and human performance can be narrowed\. For evaluation, we useSystem Accuracy\(Yuanet al\.,[2023a](https://arxiv.org/html/2605.24211#bib.bib4)\), which assigns 1 if all pairs are correct and 0 otherwise\.
#### Sub\-concept Generation\.
To test whether LLMs can generate source sub\-concepts in open settings, we test two configurations: given the target system, its sub\-concepts, and the source system name, models must produce the corresponding source sub\-concepts either \(1\)without backgroundor \(2\)with background\.
To evaluate, we useSemantic Match Accuracy \(SMA\)\. It uses cosine similarity between embeddings built withall\-MiniLM\-L6\-v2\.101010all\-MiniLM\-L6\-v2:[https://huggingface\.co/sentence\-transformers/all\-MiniLM\-L6\-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)A fixed 0\.7 threshold is adopted as a conservative heuristic for MiniLM embeddings, balancing semantic sensitivity with false\-positive robustness\.
SMA=count\(similarity≥0\.7\)total concepts\\text\{SMA\}=\\frac\{\\text\{count\}\(\\text\{similarity\}\\geq 0\.7\)\}\{\\text\{total concepts\}\}\(2\)
### 3\.5Explanation Generation \(RQ3\)
The third stage evaluates whether LLMs can articulate why an analogy holds, given a target and source\. Each model generates a natural\-language explanation\. To isolate the contribution of different input signals, we test six configurations: \(S1\)names only, \(S2\)\+background, \(S3\)\+unpaired sub\-concepts, \(S4\)\+unpaired sub\-concepts & background, \(S5\)\+paired mappings, and \(S6\)\+paired mappings & background\(see the prompts in Appendix[D](https://arxiv.org/html/2605.24211#A4)\)\. Throughout the results, we refer to these configurations as S1–S6 for brevity\.
We evaluate generated explanations against gold references using cosine similarity overall\-mpnet\-base\-v2\(Reimers and Gurevych,[2019](https://arxiv.org/html/2605.24211#bib.bib35)\)embeddings known for their ability to compare sentences to each other, capturing semantic equivalence beyond surface variation, and then we compare performance across input settings\.
### 3\.6Evaluation \(RQ4\)
#### LLM\-as\-a\-judge\.
To close the pipeline, we turn to evaluation – employingClaude Sonnet 4\.6111111[https://openrouter\.ai/anthropic/claude\-sonnet\-4\.6](https://openrouter.ai/anthropic/claude-sonnet-4.6)as an external LLM judge, independent of all 12 pipeline models to reduce evaluation bias\. Analogies are scored on three dimensions \(using 1–3 scale\):coherence\(does the pairing make intuitive sense?\),mapping soundness\(are source–target correspondences structurally valid?\), andexplanatory power\(does the analogy help a learner understand the target?\)\. We use few\-shot prompting with 10 calibrated examples spanning the full score range, as preliminary experiments without examples yielded less stable judgments\. The full prompt is provided in Appendix[E](https://arxiv.org/html/2605.24211#A5)\. We apply this judge across all 12 models to identify which one produces the highest\-quality analogies overall\.
#### Human Evaluation\.
To validate the LLM\-based ratings, we collect human judgments from seven graduate students in AI and Business\.121212Annotation form:[https://myriam2002\.github\.io/analogy\-annotation/](https://myriam2002.github.io/analogy-annotation/)Prior to annotation, participants were provided with detailed instructions describing the task, including definitions of the three evaluation dimensions \(coherence, mapping soundness, and explanatory power\), along with illustrative examples to calibrate their understanding\. Annotators evaluate 45 analogy pairs – 15 target concepts across five domains \(check Appendix[J](https://arxiv.org/html/2605.24211#A10)for all analogies tested\), each paired with three candidate sources – scoring each on the same three dimensions used by the LLM judge\. In addition to scoring individual dimensions, annotators rank candidate analogies based on overall learning usefulness, capturing whether the analogy is intuitive and helpful for understanding the target concept\. This ranking provides a qualitative, learner\-oriented perspective that complements the dimension\-level scores, yielding a total of 165 annotated datapoints\.131313Confidence scores were collected for ranking but not used in the current analysis; incorporating them to weigh annotator reliability is left for future work\.
We measure inter\-annotator reliability using Krippendorff’sα\\alpha\(Krippendorff,[2011](https://arxiv.org/html/2605.24211#bib.bib36)\)for ordinal ratings, Kendall’sWW\(Kendall,[1990](https://arxiv.org/html/2605.24211#bib.bib37)\)for ranking concordance across annotators, and Spearman’sρ\\rhoto compare human scores against LLM judge scores\.
## 4Results and Analysis
### 4\.1Source Finding \(RQ1\)
#### Closed Setting\.
Embedding\-based retrieval shows consistent but dataset\-dependent trends\. On SCAR, adding sub\-concepts yields the largest gains:OpenAI\-Largeimproves from 0\.710 to 0\.792 andOpenAI\-Smallfrom 0\.677 to 0\.780, with the average increase from 0\.557 to 0\.673\. Adding background provides no benefit \(avg\. 0\.629\), likely because sub\-concepts already capture structural alignment, and extra context introduces noise\. On ParallelPARC, performance is near\-saturated across settings \(name\-only avg\. 0\.876\), with sub\-concepts \+ background performing best \(OpenAI\-Large: 0\.942,OpenAI\-Small: 0\.932, avg\. 0\.908\), as its backgrounds provide richer relational signal than SCAR descriptions\. Across both datasets,OpenAIembeddings consistently lead, whileSPECTERlags \(SCAR avg\. 0\.437; ParallelPARC avg\. 0\.781\), reflecting weaker alignment with the task\. Full results are in Appendix[I](https://arxiv.org/html/2605.24211#A9)\.
For selection, we compare five configurations: re\-ranking with names only, background, gold sub\-concepts, gold sub\-concepts \+ background, generated sub\-concepts, and embedding\-only retrieval\. Re\-ranking with gold sub\-concepts achieves the best Hit@1 \(24\.5%\), followed by background \(22\.2%\), gold \+ background \(20\.8%\), embedding\-only \(19\.5%\), generated sub\-concepts \(18\.2%\), and names only \(17\.5%\)\. At Hit@2 and Hit@3, embedding\-only performs best \(30\.5%, 40\.0%\), while gold sub\-concepts rank third \(31\.8%, 35\.8%\), indicating that embeddings better retrieve candidates, whereas LLM re\-ranking excels at top\-1 selection\. Generated sub\-concepts outperform the name\-only baseline \(18\.2% vs 17\.5% at Hit@1\), showing that the pipeline produces meaningful structure without gold annotations, though a gap remains\. Overall, embeddings narrow the search space, while LLM reasoning over sub\-concepts is key for precise selection\.
#### Open Setting\.
Adding sub\-concepts to the generation prompt yields only marginal gains \(avg\. 0\.4\-1\.6 pp across all K and models\), suggesting limited benefit compared to their substantial impact in retrieval, while incurring additional prompt cost, making target\-only prompting the more practical default\. The two exceptions areLLaMA\-405BandQwen3\-32B, which show a slight decrease when sub\-concepts are added\. In absolute terms, the strongest Hit@20 results are achieved under the target with sub\-concepts mode, withGemini\-2\.5\(75\.4%\) andGPT\-OSS\-120B\(75\.3%\) leading\. However, top models cluster within a few percentage points of each other, suggesting that scale alone does not determine generation quality\. Per\-model results are in Appendix[H](https://arxiv.org/html/2605.24211#A8)\.
### 4\.2Sub\-concept Generation \(RQ2\)
Table 1:System accuracy \(Matching\) and semantic match accuracy \(Generation\) with \(\+BG\) and without \(–BG\) background descriptions\. Bold indicates the best score per column\. Gray shading indicates the higher\-scoring mode per task for each model\. Models are sorted by matching accuracy\.Table[1](https://arxiv.org/html/2605.24211#S4.T1)contrasts sub\-concept matching and generation across models\. For matching, performance is slightly higher*without*background descriptions \(avg\. 0\.650 vs\. 0\.603\), suggesting additional context hinders alignment\.Grok\-4leads at 0\.78, surpassing the prior state\-of\-the\-art of 0\.62Yuanet al\.\([2023a](https://arxiv.org/html/2605.24211#bib.bib4)\), with gains attributed to model choice and prompt design \(Appendix[F](https://arxiv.org/html/2605.24211#A6)\)\.GPT\-4\.1\-nanoperforms worst \(0\.37\) despite not being the smallest model\. These results motivate the harder generation setting explored next\.
For generation, background descriptions provide a clear benefit, raising average semantic match accuracy from 0\.298 to 0\.412\. The LLaMA family leads, withLLaMA\-3\.1\-70BandLLaMA\-3\.1\-405Bboth reaching 0\.47, while GPT\-based variants consistently occupy the lower half\. Notably,GPT\-4\.1\-nanoandGPT\-OSS\-120B\(0\.34\) underperform the smallerGPT\-OSS\-20B\(0\.39\), confirming that scale alone does not guarantee stronger generation\.
Based on these findings, we adoptLLaMA\-3\.1\-70Bwith background descriptions for all subsequent generation experiments\.
### 4\.3Explanation generation \(RQ3\)
The most apparent pattern is the large jump in similarity when sub\-concepts are introduced: average scores rise from 0\.724 in S1–S2 to 0\.854 with unpaired sub\-concepts \(S3–S4\) and 0\.830 with paired sub\-concepts \(S5–S6\)\. By contrast, adding background descriptions has a negligible effect across all three setting pairs \(at most\+\+0\.006\), consistent with our earlier matching findings\. The best setting overall is S5 \(systems \+ paired sub\-concepts\)\.
GPT\-4\.1\-Miniis the strongest model overall \(mean 0\.827\) and is the best performer in four of the six settings \(S3–S6\), indicating strong generalisation across sub\-concept conditions\. In the system\-only settings \(S1–S2\), where scores are uniformly low and differences between models are small \(range≈\\approx0\.03\), no single model stands out, withGPT\-OSS\-120BandLlama\-70Bmarginally leading in S1 and S2, respectively\. Check Appendix[G](https://arxiv.org/html/2605.24211#A7)for the detailed heatmap\.
### 4\.4Evaluation \(RQ4\)
#### LLM\-as\-a\-judge\.
Figure 4:Per\-criterion scores assigned by Claude Sonnet 4\.6 acting as an LLM judge\. Scores are on a 1–3 scale across three criteria: Coherence, Mapping, and Explanatory power\. Black borders indicate the top\-performing model per criterion\.Figure[4](https://arxiv.org/html/2605.24211#S4.F4)shows LLM\-as\-a\-judge scores across generation models\.DeepSeek\-R1andGrok\-4achieve the highest overall performance \(2\.12 and 2\.10\), whileGPT\-4\.1\-NanoandLLaMA\-3\.1\-8Bperform the worst \(1\.71 and 1\.68\)\. The remaining models cluster closely in the middle, indicating relatively small differences across most systems\. Across all models, coherence scores are consistently higher than mapping and explanatory scores, suggesting that models often produce intuitively plausible analogies even when structural mappings are incomplete or explanations are less informative\.
Table 2:Comparison of analogy quality across selection strategies using LLM\-as\-a\-judge scores \(1–3 scale\)\.Table[2](https://arxiv.org/html/2605.24211#S4.T2)compares selection strategies under LLM\-as\-a\-judge evaluation\. The baseline and re\-ranking approaches achieve close average scores \(2\.068 vs\. 2\.069\), but differ in their strengths: the baseline yields slightly higher coherence, while re\-ranking improves mapping quality\. This indicates that LLM\-based selection refines structural alignment without substantially affecting overall performance\.
#### Human Evaluation\.
Inter\-annotator agreement is measured using Krippendorff’sα\\alpha\. Agreement is fair forcoherence\(0\.382\) andexplanatory power\(0\.381\), and lower formapping soundness\(0\.342\) – the most technically demanding dimension\. Agreement is highest for the overall ranking \(0\.479\) – where annotators rank the three candidate sources per target by overall learning usefulness – reflecting that relative ordering is easier to agree on than assigning consistent absolute scores across all three dimensions; for instance, two annotators may agree that source A is best, yet still differ on whether to rate its coherence a 2 or a 3\. Appendix[K](https://arxiv.org/html/2605.24211#A11)provides detailed ranking statistics\.
Spearman correlation \(ρ\\rho\) between Claude and human annotators varies across both annotators and dimensions\. Agreement is highest forexplanatory power\(meanρ=0\.511\\rho=0\.511, range0\.2690\.269–0\.6780\.678\) and lowest formapping soundness\(meanρ=0\.342\\rho=0\.342, range0\.1240\.124–0\.5070\.507\), consistent with LLM\-as\-a\-judge trends\. Coherence falls between these extremes \(meanρ=0\.368\\rho=0\.368, range0\.1300\.130–0\.5810\.581\), reflecting moderate agreement on whether analogies are intuitively meaningful\. Annotator\-level variability is also notable: the weakest alignment consistently comes from the annotator whose academic background differed most from the rest of the group, which may be partially explained by the domain composition of the evaluation set, where five of the fifteen target concepts were drawn from the computer science domain\.
Table 3:Spearman correlation between Claude’s implicit ranking and human rankings\.Table[3](https://arxiv.org/html/2605.24211#S4.T3)compares Claude’s implicit ranking of analogies with human rankings\. In contrast to the dimension\-level results, ranking agreement is generally stronger, with most annotators showing moderate\-to\-high correlation \(ρ\\rhobetween 0\.46 and 0\.70\) and statistically significant results \(p < 0\.05\)\. The strongest alignment is observed for ANN\-6LG7WY \(ρ=0\.705\\rho=0\.705\), while ANN\-Q777H6 again shows no significant agreement\. These results indicate that Claude more reliably captures relative preferences between analogies than assigning consistent absolute scores, suggesting that LLM\-as\-a\-judge is better suited for comparative evaluation settings than for fine\-grained scoring\.
Figure 5:Hierarchical clustering of human annotators and Claude based on their scoring patterns across evaluation dimensions\. Distances reflect dissimilarity in judgments \(computed from Spearman correlations\)\. Claude clusters closely with a subset of annotators \(e\.g\., ANN\-VM9J6N\), while other annotators \(e\.g\., ANN\-Q777H6\) form more distant branches\.Figure[5](https://arxiv.org/html/2605.24211#S4.F5)further examines agreement patterns through hierarchical clustering of annotators and Claude\. The dendrogram reveals that Claude groups closely with a subset of annotators, suggesting similar evaluation behavior, while others form more distant clusters, indicating systematic differences in scoring\. In particular, annotators with lower correlation scores \(e\.g\., ANN\-Q777H6\) appear as outliers, consistent with the results in Table[3](https://arxiv.org/html/2605.24211#S4.T3)\. This analysis reinforces that, although Claude aligns well with the majority of annotators, human judgments themselves are not fully consistent, highlighting the inherent subjectivity of analogy evaluation\.
## 5Conclusion
We presented a modular pipeline for educational analogy generation decomposed into source finding, sub\-concept generation, explanation generation, and evaluation, systematically evaluated across 12 LLMs and 7 embedding models on SCAR and ParallelPARC\. A central finding is that the best model and configuration differ across stages:Grok\-4\-Fastleads sub\-concept matching without background descriptions, whileLLaMA\-3\.1\-70Bwith background is best for generation;GPT\-4\.1\-Miniwith unpaired sub\-concept mappings achieves the strongest explanation quality; and OpenAI large embeddings with sub\-concepts lead closed\-setting retrieval, whereas the target\-only configuration is more practical in the open setting\. Sub\-concept grounding consistently improves retrieval precision and explanation quality but provides limited benefit in open\-ended source generation\. For evaluation,DeepSeek\-R1andGrok\-4\-Fastproduce the highest\-quality analogies, andClaude Sonnet 4\.6aligns more reliably with human rankings than absolute scores, suggesting LLM\-as\-a\-judge is better suited to comparative settings\. Taken together, these findings demonstrate that no single model dominates across all stages, reinforcing the necessity of a modular, stage\-aware approach to analogy generation\.
## Limitations
Several limitations of this work warrant acknowledgment\. First, all experiments are conducted exclusively on English\-language data, leaving the pipeline’s behaviour on morphologically richer or structurally different languages unexplored\. In such settings, both embedding\-based retrieval and generation quality may differ, and evaluation metrics such as Semantic Match Accuracy \(SMA\) may require recalibration\.
Second, the 0\.7 cosine similarity threshold used for SMA is a fixed heuristic selected forMiniLM\-L6embeddings rather than empirically validated\. Its suitability for more abstract or diverse sub\-concepts remains unclear, and alternative thresholds could meaningfully affect generation results\.
Third, while each pipeline stage is evaluated independently, we do not evaluate the pipeline end\-to\-end\. In a full pipeline setting, errors are likely to accumulate and propagate across stages\. For example, an incorrect source selected during the retrieval stage would lead to flawed sub\-concept mappings and explanations downstream\. This makes source finding a particularly critical stage, as early errors can disproportionately affect final analogy quality\. Future work should investigate strategies to mitigate such compounding effects, such as iterative refinement, confidence\-aware filtering, or alternative candidate selection mechanisms\.
Fourth, the current architecture relies primarily on dense embedding models for retrieval and does not explore hybrid search strategies\. Combining dense and sparse retrieval methods, or incorporating re\-ranking models more extensively, could improve the robustness and quality of source selection, particularly in open\-ended settings\.
Fifth, analogy quality is inherently subjective: an analogy that is effective for one learner may not be equally meaningful for another\. While this subjectivity is partially reflected in moderate inter\-annotator agreement, our evaluation does not explicitly model user\-dependent preferences\. This limits the ability to assess how well generated analogies generalise across diverse learner populations\. Future work should investigate how to construct learner profiles and incorporate them into the pipeline to generate personalised analogies tailored to factors such as cultural background, age, prior knowledge, and individual interests or hobbies\.
Sixth, all pipeline stages are evaluated in a single\-pass setting\. We do not measure output variance across runs, which means that reported results reflect point estimates rather than stable averages\. Although a low temperature \(0\.2\) is used, some variability is expected, particularly for smaller models\. Moreover, since ParallelPARC was used in only one stage of the pipeline \(embedding\-based retrieval\) and the remaining stages relied solely on SCAR, the generalizability of the results across datasets may be limited\.
Finally, the human evaluation study involves six out of seven graduate\-level annotators from a single institution, which may introduce demographic and disciplinary homogeneity\. Inter\-annotator agreement is moderate, reflecting the inherent subjectivity of analogy evaluation and suggesting that a larger, more diverse annotator pool could yield different conclusions\. In addition, open\-setting source finding is evaluated only on SCAR due to computational constraints, limiting the generalisability of these findings to other domains of different difficulty\.
## Ethical Considerations
We do not anticipate any ethical risks related to this study\.
## References
- B\. Bettin and L\. Ott \(2023\)Pedagogical prisms: toward domain isomorphic analogy design for relevance and engagement in computing education\.InProceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V\. 1,pp\. 410–416\.Cited by:[§1](https://arxiv.org/html/2605.24211#S1.p1.1)\.
- B\. Bhavya, C\. Palaguachi, Y\. Zhou, S\. Bhat, and C\. Zhai \(2024\)Long\-form analogy evaluation challenge\.InProceedings of the 17th International Natural Language Generation Conference: Generation Challenges,pp\. 1–16\.Cited by:[§1](https://arxiv.org/html/2605.24211#S1.p5.1),[§2](https://arxiv.org/html/2605.24211#S2.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2605.24211#S2.SS0.SSS0.Px4.p1.1)\.
- D\. E\. Brown and J\. Clement \(1989\)Overcoming misconceptions via analogical reasoning: abstract transfer versus explanatory model construction\.Instructional science18\(4\),pp\. 237–261\.Cited by:[§1](https://arxiv.org/html/2605.24211#S1.p1.1)\.
- T\. Czinczoll, H\. Yannakoudakis, P\. Mishra, and E\. Shutova \(2022\)Scientific and creative analogies in pretrained language models\.arXiv preprint arXiv:2211\.15268\.Cited by:[§1](https://arxiv.org/html/2605.24211#S1.p1.1),[§1](https://arxiv.org/html/2605.24211#S1.p5.1),[§2](https://arxiv.org/html/2605.24211#S2.SS0.SSS0.Px4.p1.1),[§3\.3](https://arxiv.org/html/2605.24211#S3.SS3.SSS0.Px1.p1.1)\.
- S\. Eriksson, N\. Gericke, and K\. Thörne \(2024\)Analogy competence for science teachers\.Studies in Science Education,pp\. 1–29\.Cited by:[§1](https://arxiv.org/html/2605.24211#S1.p1.1)\.
- D\. Gentner \(1981\)Are scientific analogies metaphors?\.Technical reportCited by:[§1](https://arxiv.org/html/2605.24211#S1.p6.1)\.
- C\. Jiayang, L\. Qiu, T\. H\. Chan, T\. Fang, W\. Wang, C\. Chan, D\. Ru, Q\. Guo, H\. Zhang, Y\. Song,et al\.\(2023\)StoryAnalogy: deriving story\-level analogies from large language models to unlock analogical understanding\.arXiv preprint arXiv:2310\.12874\.Cited by:[§1](https://arxiv.org/html/2605.24211#S1.p5.1),[§2](https://arxiv.org/html/2605.24211#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2605.24211#S2.SS0.SSS0.Px4.p1.1)\.
- M\. G\. Kendall \(1990\)Rank correlation methods\.5th edition,Oxford University Press\.Cited by:[§3\.6](https://arxiv.org/html/2605.24211#S3.SS6.SSS0.Px2.p2.3)\.
- O\. Khattab, K\. Santhanam, X\. L\. Li, D\. Hall, P\. Liang, C\. Potts, and M\. Zaharia \(2022\)Demonstrate\-search\-predict: composing retrieval and language models for knowledge\-intensive NLP\.arXiv preprint arXiv:2212\.14024\.Cited by:[§3\.2](https://arxiv.org/html/2605.24211#S3.SS2.SSS0.Px2.p1.1)\.
- O\. Khattab, A\. Singhvi, P\. Maheshwari, Z\. Zhang, K\. Santhanam, S\. Vardhamanan, S\. Haq, A\. Sharma, T\. T\. Joshi, H\. Moazam, H\. Miller, M\. Zaharia, and C\. Potts \(2024\)DSPy: compiling declarative language model calls into self\-improving pipelines\.Cited by:[§3\.2](https://arxiv.org/html/2605.24211#S3.SS2.SSS0.Px2.p1.1)\.
- J\. Kim, S\. Suh, L\. B\. Chilton, and H\. Xia \(2023\)Metaphorian: leveraging large language models to support extended metaphor creation for science writing\.InProceedings of the 2023 ACM Designing Interactive Systems Conference,pp\. 115–135\.Cited by:[§1](https://arxiv.org/html/2605.24211#S1.p5.1)\.
- K\. Krippendorff \(2011\)Computing krippendorff’s alpha\-reliability\.Annenberg School for Communication Departmental Papers\.Cited by:[§3\.6](https://arxiv.org/html/2605.24211#S3.SS6.SSS0.Px2.p2.3)\.
- N\. Li, S\. Yuan, J\. Chen, J\. Liang, F\. Wei, Z\. Liang, D\. Yang, and Y\. Xiao \(2024\)Past meets present: creating historical analogy with large language models\.arXiv preprint arXiv:2409\.14820\.Cited by:[§1](https://arxiv.org/html/2605.24211#S1.p5.1),[§2](https://arxiv.org/html/2605.24211#S2.SS0.SSS0.Px1.p1.1)\.
- N\. Reimers and I\. Gurevych \(2019\)Sentence\-bert: sentence embeddings using siamese bert\-networks\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing,External Links:[Link](https://arxiv.org/abs/1908.10084)Cited by:[§3\.5](https://arxiv.org/html/2605.24211#S3.SS5.p2.1)\.
- Z\. Shao, S\. Yuan, L\. Gao, Y\. He, D\. Yang, and S\. Chen \(2025\)Unlocking scientific concepts: how effective are llm\-generated analogies for student understanding and classroom practice?\.InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems,pp\. 1–19\.Cited by:[§1](https://arxiv.org/html/2605.24211#S1.p4.1),[§1](https://arxiv.org/html/2605.24211#S1.p5.1)\.
- O\. Sultan, Y\. Bitton, R\. Yosef, and D\. Shahaf \(2024\)Parallelparc: a scalable pipeline for generating natural\-language analogies\.arXiv preprint arXiv:2403\.01139\.Cited by:[§1](https://arxiv.org/html/2605.24211#S1.p5.1),[§1](https://arxiv.org/html/2605.24211#S1.p6.1),[§2](https://arxiv.org/html/2605.24211#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2605.24211#S3.SS1.p3.1)\.
- N\. Tandon, B\. D\. Mishra, J\. Grus, W\. Yih, A\. Bosselut, and P\. Clark \(2018\)Reasoning about actions and state changes by injecting commonsense knowledge\.arXiv preprint arXiv:1808\.10012\.Cited by:[§3\.1](https://arxiv.org/html/2605.24211#S3.SS1.p3.1)\.
- X\. Ye, A\. Wang, J\. Choi, Y\. Lu, S\. Sharma, L\. Shen, V\. Tiyyala, N\. Andrews, and D\. Khashabi \(2024\)AnaloBench: benchmarking the identification of abstract and long\-context analogies\.arXiv preprint arXiv:2402\.12370\.Cited by:[§1](https://arxiv.org/html/2605.24211#S1.p1.1),[§2](https://arxiv.org/html/2605.24211#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Yuan, J\. Chen, X\. Ge, Y\. Xiao, and D\. Yang \(2023a\)Beneath surface similarity: large language models make reasonable scientific analogies after structure abduction\.arXiv preprint arXiv:2305\.12660\.Cited by:[Figure 7](https://arxiv.org/html/2605.24211#A6.F7),[§1](https://arxiv.org/html/2605.24211#S1.p2.1),[§1](https://arxiv.org/html/2605.24211#S1.p5.1),[§1](https://arxiv.org/html/2605.24211#S1.p6.1),[§2](https://arxiv.org/html/2605.24211#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2605.24211#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2605.24211#S3.SS1.p2.1),[§3\.3](https://arxiv.org/html/2605.24211#S3.SS3.SSS0.Px1.p1.1),[§3\.4](https://arxiv.org/html/2605.24211#S3.SS4.SSS0.Px1.p1.1),[§3\.4](https://arxiv.org/html/2605.24211#S3.SS4.p1.1),[§4\.2](https://arxiv.org/html/2605.24211#S4.SS2.p1.1)\.
- S\. Yuan, J\. Chen, C\. Sun, J\. Liang, Y\. Xiao, and D\. Yang \(2023b\)Analogykb: unlocking analogical reasoning of language models with a million\-scale knowledge base\.arXiv preprint arXiv:2305\.05994\.Cited by:[§1](https://arxiv.org/html/2605.24211#S1.p5.1),[§2](https://arxiv.org/html/2605.24211#S2.SS0.SSS0.Px1.p1.1)\.
- C\. Zhai, S\. Bhat, and H\. Ji \(2024\)Long\-form analogy extraction, generation, and evaluation with large language models\.Cited by:[§1](https://arxiv.org/html/2605.24211#S1.p5.1),[§2](https://arxiv.org/html/2605.24211#S2.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2605.24211#S2.SS0.SSS0.Px4.p1.1)\.
- X\. Zhou, Y\. Gong, Y\. Zhou, Y\. Jiang, and Z\. Bai \(2025\)Co\-design of analogical and embodied representations with children for child\-centered ai learning experiences\.International Journal of Human\-Computer Studies199,pp\. 103462\.Cited by:[§1](https://arxiv.org/html/2605.24211#S1.p4.1)\.
## Appendix ADetailed Record of Datasets
Listing[1](https://arxiv.org/html/2605.24211#LST1)and[2](https://arxiv.org/html/2605.24211#LST2)shows a representative JSON example of the two datasets\.
\{
"id":3,
"lang":"en",
"system\_a":"Respiratorysystem",
"system\_b":"Engine",
"mappings":\[
\["oxygen","fuel"\],
\["lungs","combustionchamber"\],
\["breathingmuscles","piston"\]
\],
"system\_a\_domain":"Biology",
"system\_b\_domain":"Physics",
"system\_a\_background":"Therespiratorysystem\.\.\.",
"system\_b\_background":"Anengineormotor\.\.\.",
"Explanation":\[
"Oxygencorrespondstofuel:\.\.\.",
"Lungscorrespondtothecombustionchamber:\.\.\.",
"Breathingmusclescorrespondtothepiston:\.\.\."
\]
\}
Listing 1:Representative JSON example from SCAR\{
"sample\_id":1,
"source\_subject":"Whatcausesavolcanotoerupt?",
"source\_domain":"NaturalSciences",
"target\_domain":"Engineering",
"target\_subject":"Whatcausesaboilertoexplode?",
"target\_field":"ChemicalEngineering",
"relations":\[
"\(magma,heats,undergroundwater\)like\(steam,heats,liquid\)",
"\(pressure,builds,insidethevolcano\)like\(pressure,builds,insidetheboiler\)",
"\(magma,pushes,againstthewallsofthevolcano\)like\(steam,pushes,againstthewallsoftheboiler\)"
\],
"source\_paragraph":"Whenmagmaheatsupundergroundwater,pressurebeginstobuildupinsidethevolcano,leadingtoaneruption\.",
"target\_paragraph":"Steamheatstheliquidinsidetheboiler,causingthepressuretobuildupuntiltheboilercannolongercontainit\.",
"analogy\_type":"faranalogy",
"sum\_vote\_analogy":3\.0
\}
Listing 2:Representative JSON from ParallelPARC
## Appendix BPrompt for LLM Re\-ranker
Youareselectingthebestanalogoussourceconceptsforascientificanalogy\.
Yourtask:
1\.Analyzethetargetconceptanditsproperties
2\.Revieweachcandidatesourceanditsgeneratedanalogousproperties
3\.Selectthe3candidateswhosepropertiesBESTmaptothetargetproperties
Selectioncriteria:
\-Strongstructural/functionalcorrespondencebetweensourceandtargetproperties
\-Thesourceconceptshouldbefamiliarandhelpexplaintheunfamiliartarget
\-Prefersourceswithclear,well\-mappedpropertiesovervagueones
ReturntheEXACTnamesofyourtop3selectedcandidates\.
""""
Listing 3:Prompt used to select the top\-3 analogies from the top\-10 potential sources\. This was wrapped in DSPY function as instruction, alongside specific input and output variables\.
## Appendix CThreshold Decision for Semantic Hit@K
To define a threshold for semantic Hit@K, we analyse the distribution of source\-to\-source similarity scores in the SCAR dataset\. Specifically, we compare similarity values betweencorrectsource pairs \(i\.e\., gold analogies\) andincorrectpairs obtained by randomly shuffling sources\.
Figure[6](https://arxiv.org/html/2605.24211#A3.F6)shows that correct source pairs tend to exhibit higher similarity than incorrect ones, but with substantial overlap between the two distributions\. This overlap indicates that a fixed high threshold \(e\.g\., 0\.5\) would be overly restrictive, excluding many valid analogies that are structurally sound but not highly similar at the surface level\.
To balance precision and coverage, we adopt a data\-driven threshold of 0\.374, corresponding to the upper tertile \(top one\-third\) of similarity scores among gold source pairs\. This choice captures a substantial portion of valid analogies while filtering out a large fraction of incorrect ones\. As shown in the figure, this threshold lies above the mean similarity of incorrect pairs \(0\.180\) and close to the mean of correct pairs \(0\.368\), providing a principled trade\-off between false positives and false negatives\.
Figure 6:Distribution of source\-to\-source similarity scores in SCAR for correct \(gold\) and incorrect \(shuffled\) pairs\. Dashed lines indicate the mean similarity for each group, while the dotted line shows the chosen threshold \(0\.374\), corresponding to the upper tertile of gold similarities\. A higher fixed threshold \(e\.g\., 0\.5\) would exclude many valid analogies, motivating the use of a data\-driven cutoff\.
## Appendix DExplanation Generation Prompts
This appendix lists theDSPysignatures used for the six explanation\-generation settings\. Each signature corresponds to one input configuration\.
#### Setting 1: System Names Only
This configuration provides only the target and source system names\.
classExplGen\_Base\(dspy\.Signature\):
"""Generateananalogyexplanationusingsystemnamesonly\."""
target:str=dspy\.InputField\(
desc="Targetsystem"
\)
source:str=dspy\.InputField\(
desc="Sourcesystem"
\)
explanation:str=dspy\.OutputField\(
desc="Explanationoftheanalogy"
\)
#### Setting 2: System Names with Background Descriptions
This setting augments system names with background descriptions for both systems\.
classExplGen\_BaseDesc\(dspy\.Signature\):
"""Generateananalogyexplanationusingnamesanddescriptions\."""
target:str=dspy\.InputField\(
desc="Targetsystem"
\)
target\_desc:str=dspy\.InputField\(
desc="Backgrounddescriptionofthetarget"
\)
source:str=dspy\.InputField\(
desc="Sourcesystem"
\)
source\_desc:str=dspy\.InputField\(
desc="Backgrounddescriptionofthesource"
\)
explanation:str=dspy\.OutputField\(
desc="Explanationoftheanalogy"
\)
#### Setting 3: System Names with Unpaired Sub\-concepts
This configuration provides both systems with unpaired lists of sub\-concepts\.
classExplGen\_Sub\(dspy\.Signature\):
"""Generateananalogyexplanationusingunpairedsub\-concepts\."""
target:str=dspy\.InputField\(
desc="Targetsystem"
\)
target\_sub:list\[str\]=dspy\.InputField\(
desc="Targetsub\-concepts"
\)
source:str=dspy\.InputField\(
desc="Sourcesystem"
\)
source\_sub:list\[str\]=dspy\.InputField\(
desc="Sourcesub\-concepts"
\)
explanation:list\[str\]=dspy\.OutputField\(
desc="Explanationafterimplicitlypairingsub\-concepts"
\)
#### Setting 4: System Names with Unpaired Sub\-concepts and Descriptions
This setting extends Setting 3 by adding background descriptions for both systems\.
classExplGen\_SubDesc\(dspy\.Signature\):
"""Generateananalogyexplanationusingunpairedsub\-conceptsanddescriptions\."""
target:str=dspy\.InputField\(
desc="Targetsystem"
\)
target\_desc:str=dspy\.InputField\(
desc="Backgrounddescriptionofthetarget"
\)
target\_sub:list\[str\]=dspy\.InputField\(
desc="Targetsub\-concepts"
\)
source:str=dspy\.InputField\(
desc="Sourcesystem"
\)
source\_desc:str=dspy\.InputField\(
desc="Backgrounddescriptionofthesource"
\)
source\_sub:list\[str\]=dspy\.InputField\(
desc="Sourcesub\-concepts"
\)
explanation:list\[str\]=dspy\.OutputField\(
desc="Explanationafterimplicitlypairingsub\-concepts"
\)
#### Setting 5: Paired Sub\-concept Mappings
This configuration provides explicit one\-to\-one sub\-concept alignments between the target and source systems\.
classExplGen\_Pair\(dspy\.Signature\):
"""Generateananalogyexplanationusingpairedsub\-concepts\."""
target:str=dspy\.InputField\(
desc="Targetsystem"
\)
source:str=dspy\.InputField\(
desc="Sourcesystem"
\)
pairs:list\[list\[str\]\]=dspy\.InputField\(
desc="Pairedtarget\-sourcesub\-concepts"
\)
explanation:list\[str\]=dspy\.OutputField\(
desc="Explanationforeachpairedsub\-concept"
\)
#### Setting 6: Paired Sub\-concept Mappings with Descriptions
This setting combines explicit sub\-concept pairings with background descriptions for both systems\.
classExplGen\_PairDesc\(dspy\.Signature\):
"""Generateananalogyexplanationusingpairedsub\-conceptsanddescriptions\."""
target:str=dspy\.InputField\(
desc="Targetsystem"
\)
target\_desc:str=dspy\.InputField\(
desc="Backgrounddescriptionofthetarget"
\)
source:str=dspy\.InputField\(
desc="Sourcesystem"
\)
source\_desc:str=dspy\.InputField\(
desc="Backgrounddescriptionofthesource"
\)
pairs:list\[list\[str\]\]=dspy\.InputField\(
desc="Pairedtarget\-sourcesub\-concepts"
\)
explanation:list\[str\]=dspy\.OutputField\(
desc="Explanationforeachpairedsub\-concept"
\)
## Appendix EPrompt for the LLM\-as\-a\-Judge
Youareanexpertevaluatorofscientificanalogies\.
Givenatargetconceptandachosensourceanalogy,evaluatewhetherthisisagoodanalogy\.
AgoodanalogyusesaFAMILIARsourceconcepttoexplainanUNFAMILIARtargetconceptthrough
meaningfulstructuralorfunctionalparallels\.
Foreachofthethreedimensionsbelow,firstprovideabriefreasoningexplainingyour
assessment,thengivethenumericscore\(1,2,or3\)\.
ANALOGY\_COHERENCE:Doesthepairingmakeintuitivesense?
\-3:Theconnectionisimmediatelyclearandnatural\-\-mostpeoplewouldseeitwithoutexplanation\.
Thesourceandtargetshareanobviousstructuralorfunctionalparallel\.
\-2:Ameaningfulconnectionexistsbutrequiressomeexplanationtosee\.
Thelinkisrealbutnotself\-evident;asentenceortwoisneededtoestablishit\.
\-1:Nomeaningfulconnectionexists,orthepairingisrandom,forced,ormisleading\.
MAPPING\_SOUNDNESS:Couldproperties/mechanismsofthesourcemaptothetarget?
\-3:Rich,consistentstructuralorfunctionalcorrespondencesexist\.
Multiplesourcepropertiesmappreciselyontotargetproperties\.
Themappingholdsacrossthemaincomponentsofbothdomains\.
\-2:Somevalidmappingsexist,butcoverageispartial\.
Corecorrespondenceswork,butimportantaspectsofthetargetarenotrepresented,
orsomemappingsareapproximateorstrained\.
\-1:Novalidmappingsarepossible,ortheapparentmappingsareentirelysuperficial
ormisleading\.Sourceandtargetarefundamentallyincompatible\.
EXPLANATORY\_POWER:Wouldthisanalogyhelpalearnerunderstandthetarget?
\-3:Theanalogyclearlyilluminatesthetargetconceptandsupportscorrectreasoning\.
Alearnercoulduseittopredictorexplaintargetbehavior\.
\-2:Theanalogyprovidespartialinsightwithnotablelimitations\.
Itconveysthegeneralideabutcannotsupportdeeperreasoning,
oritriskscreatingminormisconceptions\.
\-1:Theanalogyfailstoaidunderstandingandwouldlikelyconfuseormisleadalearner\.
CALIBRATIONEXAMPLES\-\-usethesetoanchoryourscoring\.
Scoreeachdimensionindependently\.Thescaleis1\-3only:
3=strong/clearlyworks
2=partial/workswithcaveats
1=doesn’twork/poorormisleading
Example1\(scores:3/3/3\)
target:"electriccircuit"\|analogy:"waterflowingthroughpipes"
coherence=3:
Theconnectionisimmediatelyclearwithoutanyexplanationneeded\.
Adrivingforcepushesasubstancethroughaconstrainedpathwayinbothcases\-\-
thisstructuralparallelisinstantlyvisible\.
mapping=3:
Multipleprecisecorrespondencesholdconsistently:
voltage\-\>pressure\|current\-\>flowrate\|resistance\-\>piperestriction
battery\-\>pump\|closedloop\-\>closedpipesystem
Relationshipsbetweenvariables\(increasepressure\-\>increaseflow\)arepreserved\.
explanatory=3:
AlearnercanusetheanalogytoreasonaboutOhm’slawqualitatively
andmakecorrectpredictionsaboutcircuitbehavior\.
Example2\(scores:3/2/3\)
target:"cell"\|analogy:"factory"
coherence=3:
Theanalogycapturesorganizeddivisionoflaborwithinaboundedsystem\.
Mostpeopleimmediatelyseewhyacellresemblesafactory\.
mapping=2:
nucleus\-\>controlcenter\|ribosomes\-\>productionworkers
mitochondria\-\>energysupply\|membrane\-\>boundary/security
explanatory=3:
Helpslearnersgraspcoordinationandspecializationclearly\.
Example3\(scores:2/2/2\)
target:"mathematicalfunction"\|analogy:"machine"
coherence=2:
Thesharedidea\(input\-\>transformation\-\>output\)isrealbutrequiresexplanation\.
mapping=2:
input\-\>rawmaterial,rule\-\>mechanism,output\-\>product
explanatory=2:
Usefulforinitialintuitionbutlimitedfordeeperreasoning\.
Example4\(scores:1/1/1\)
target:"neuralnetwork"\|analogy:"humanbrain"
coherence=1:
Superficialsimilarity\("neurons"\)withoutstructuralalignment\.
mapping=1:
Novalidcorrespondencesacrosscoremechanisms\.
explanatory=1:
Likelytomisleadunderstanding\.
Example5\(scores:1/1/1\)
target:"chemicalreaction"\|analogy:"anovel"
coherence=1:
Nomeaningfulconnection\.
mapping=1:
Nosystematiccorrespondences\.
explanatory=1:
Noinstructionalvalue\.
Example6\(scores:3/1/2\)
target:"democracy"\|analogy:"majorityvoteinaclassroom"
coherence=3:
Bothinvolvemajority\-baseddecisions\.
mapping=1:
Institutionalaspectshavenocounterpart\.
explanatory=2:
Partialunderstandingonly\.
Example7\(scores:2/3/3\)
target:"atom"\|analogy:"solarsystem"
coherence=2:
Spatialsimilarityexistsbutrequiresexplanation\.
mapping=3:
nucleus\-\>sun\|electrons\-\>planets
explanatory=3:
Supportsinitialreasoningaboutstructure\.
Example8\(scores:3/2/1\)
target:"photosynthesis"\|analogy:"solar\-poweredfactory"
coherence=3:
Energyconversionisclearlyanalogous\.
mapping=2:
sunlight\-\>energysupply\|chloroplast\-\>factory
explanatory=1:
Oversimplifiesprocesses\.
Example9\(scores:1/3/3\)
target:"compoundinterest"\|analogy:"asnowballrollingdownhill"
coherence=1:
Domainsdifferfundamentally\.
mapping=3:
principal\-\>snowball\|growth\-\>accumulation
explanatory=3:
Clearlyconveysacceleratinggrowth\.
Example10\(scores:3/1/2\)
target:"ecosystem"\|analogy:"afamily"
coherence=3:
Interdependenceisclear\.
mapping=1:
Nostructuralecologicalcorrespondences\.
explanatory=2:
Partialintuitiononly\.
Listing 4:Prompt used to evaluate the analogies from different settings on different dimensions\. This was wrapped in DSPY function as instruction, alongside specific input and output variables\.
## Appendix FAblation results for Sub\-concept Matching
Figure 7:Ablation study on sub\-concept matching system accuracy, isolating the contribution of each component relative to prior workYuanet al\.\([2023a](https://arxiv.org/html/2605.24211#bib.bib4)\)\. Upgrading the model to our best preforming Grok\-4 alone \(B\) yields the largest gain \(\+10\.8%\), followed by prompt redesign \(C, \+5\.5%\)\. Wrapping with DSPy Chain\-of\-Thought \(D, our final system\) introduces a marginal drop \(−\-1\.5%\), confirming that automated prompt optimisation does not improve over the hand\-crafted prompt for this task\.Figure[7](https://arxiv.org/html/2605.24211#A6.F7)shows the ablation study to figure out what aspect actually led to the improved results\.
## Appendix GExplanation Heatmap Results
Figure 8:Average SBERT similarity scores across twelve models and six experimental settings\. Settings S1\-S2 provide only system\-level context \(without and with background descriptions, respectively\), while S3\-S4 additionally supply unpaired sub\-concepts and S5\-S6 supply paired sub\-concepts\. Black borders indicate the best\-performing model per setting\. GPT\-4\.1\-Mini consistently leads in sub\-concept\-enriched settings \(S3\-S5\), while DeepSeek\-R1 and Qwen3 variants are competitive in paired settings \(S5\-S6\)\. Scores rise substantially when sub\-concepts are introduced \(S3\-S6 vs\. S1\-S2\), suggesting that grounding the comparison in explicit sub\-concept structure is a stronger driver of similarity than background context alone\.Figure[8](https://arxiv.org/html/2605.24211#A7.F8)presents SBERT similarity scores across twelve LLMs and six experimental settings for the explanation generation stage\.
## Appendix HOpen Setting Source Finding: Per\-Model Hit@20 Comparison
Figure 9:Per\-model Hit@20 comparison in the open setting under Target\-Only and Target \+ Sub\-Concept modes\. Results are reported using semantic matching and highlight the effect of incorporating sub\-concepts on top\-20 retrieval performance\.Figure[9](https://arxiv.org/html/2605.24211#A8.F9)presents per\-model Hit@20 scores in the open setting, comparing target\-only and target with sub\-concept prompting across all twelve models\.
## Appendix IEmbedding\-based Retrieval Results
Figure 10:Hit@20 performance on the SCAR dataset across embedding models and input configurations\. Incorporating sub\-concepts consistently improves retrieval compared to name\-only inputs, while adding background provides mixed benefits\. OpenAI embeddings achieve the strongest overall performance\.Figure 11:Hit@20 performance on the ParallelPARC dataset\. Retrieval performance is generally higher than SCAR, reflecting the more structured nature of process\-based analogies\. Combining sub\-concepts with background yields the best results across most models, indicating the value of richer contextual representations\.Figure[10](https://arxiv.org/html/2605.24211#A9.F10)and[11](https://arxiv.org/html/2605.24211#A9.F11)show the heatmap of the 7 embedding models across 4 input configurations\.
## Appendix JAnnotation Dataset
Table[4](https://arxiv.org/html/2605.24211#A10.T4)presents the annotation dataset, comprising 15 target concepts spanning five domains, each paired with three source analogies evaluated under target\-only and target with sub\-concept conditions\. Furthermore, Figures[12](https://arxiv.org/html/2605.24211#A10.F12)\-[21](https://arxiv.org/html/2605.24211#A10.F21)provide an overview of the web\-based annotation form used in our human evaluation study\. The form guides annotators through rating each of three candidate source analogies per target concept on three dimensions – coherence, mapping soundness, and explanatory power – on a 1–3 scale, before ranking the three sources by overall learning usefulness and reporting their confidence in the ranking\.
Table 4:Annotation dataset: 15 target concepts across five domains, each paired with three source analogies\. T = Target\-only, T\+S = Target \+ Sub\-concepts\.Figure 12:\(a\) Welcome screen introducing the task and study background\.Figure 13:\(b\) Key definitions and rating scale for the three evaluation dimensions\.Figure 14:\(c\) Explanatory power dimension definition and full rating scale\.Figure 15:\(d\) Ranking confidence scale and worked examples panel\.Figure 16:\(e\) Worked example for theAtomtarget with scored candidate sources\.Figure 17:\(f\) Annotator onboarding screen with anonymous session ID and task reminders\.Figure 18:\(g\) Rating interface forGas Diffusion— Source Analogy 1\.Figure 19:\(h\) Rating interface forGas Diffusion— Source Analogy 2\.Figure 20:\(i\) Rating interface forGas Diffusion— Source Analogy 3\.Figure 21:\(j\) Drag\-to\-rank interface for ordering sources by learning usefulness with confidence selection\.
## Appendix KHuman Evaluation Rank Agreement
Table 5:Inter\-annotator agreement for analogy ranking using Kendall’s W\. Each row corresponds to a target concept with three candidate analogies\. HigherWWindicates stronger agreement among annotators, while the p\-value tests statistical significance\.Table[5](https://arxiv.org/html/2605.24211#A11.T5)reports inter\-annotator agreement using Kendall’s coefficient of concordance \(W\) for each target concept\. Annotators ranked three candidate analogies per target\. Higher values ofWWindicate stronger agreement\.
## Appendix LFull Human and LLM\-as\-a\-judge Results Heatmap
Figure 22:Mean coherence scores per analogy across eight annotators \(seven human annotators and Claude\), sorted by inter\-annotator standard deviation \(highest disagreement at top\)\. Scores are on a 1–3 scale\. High\-disagreement analogies such asisotope dating \-\> half\-lifeandplatelets \-\> band\-aidreveal cases where structural clarity is perceived differently across annotators\. Low\-disagreement analogies at the bottom \(e\.g\.,platelets \-\> tiny repair crew,bill of rights \-\> rulebook\) receive consistently high scores, indicating clear and intuitive mappings\. Claude’s ratings \(rightmost column\) generally align with the human majority, with occasional deviations in mid\-range cases\.Figure 23:Mean mapping quality scores per analogy across eight annotators, sorted by inter\-annotator standard deviation\. Mapping quality—capturing how well source sub\-concepts correspond to target sub\-concepts—shows greater disagreement than coherence, particularly for analogies such asrenaissance \-\> spring bloomandindustrial revolution \-\> automation\. Low\-disagreement cases \(e\.g\.,platelets \-\> beads,platelets \-\> tiny repair crew\) tend to receive uniformly low scores, suggesting that weak mappings are easier to agree on than strong ones\. Claude’s scores broadly follow human judgments, with a slight tendency toward mid\-range values in ambiguous cases\.Figure 24:Mean explanatory power scores per analogy across eight annotators, sorted by inter\-annotator standard deviation\. Explanatory power—measuring how well an analogy supports understanding—exhibits the highest disagreement, especially for analogies such asisotope dating \-\> half\-lifeandencryption \-\> sealed envelope\. Consistently high\-scoring analogies \(e\.g\.,platelets \-\> tiny repair crew,adaboost algorithm \-\> teamwork\) indicate strong pedagogical value\. Analogies involving less familiar concepts \(e\.g\.,gas diffusion \-\> gasoline,random forest \-\> tree\) cluster in lower\-score regions, suggesting more limited explanatory effectiveness\.Figures[22](https://arxiv.org/html/2605.24211#A12.F22),[23](https://arxiv.org/html/2605.24211#A12.F23)and[24](https://arxiv.org/html/2605.24211#A12.F24)show the full human and Claude sonnet 4\.6 agreement\.
## Appendix MWord Overlap Analysis in Embedding\-based Retrieval
Figure 25:Impact of lexical overlap on embedding\-based source retrieval\.This figure shows how the number of overlapping words between target and source names affects retrieval performance, measured by the percentage of times the gold source is retrieved and its average rank \(lower is better\)\. As lexical overlap increases, both retrieval success and ranking quality improve, indicating that embedding\-based retrieval is strongly influenced by surface\-level word overlap\.Figure[25](https://arxiv.org/html/2605.24211#A13.F25)shows the impact of lexical overlap on embedding\-based source retrieval\.Similar Articles
Elmes*: Automated Construction of Fine-Grained Evaluation Rubrics for Large Language Models in Long-Tail Educational Scenarios
This paper introduces Elmes+, an automated framework for constructing fine-grained evaluation rubrics for LLMs in long-tail educational scenarios, and presents the Edu-330 benchmark covering 330 scenarios across 11 subjects. The framework uses a multi-agent engine and self-evolving module to co-optimize evaluation criteria and test data, revealing multidimensional educational capability differences among top LLMs.
LLM-as-Judge in Education: A Curriculum-Grounded Marking Pipeline
This paper presents a curriculum-grounded LLM-as-Judge pipeline for automated question-level marking in high-stakes exam preparation, using syllabus artefacts and marking guidelines to improve consistency and transparency, with preliminary evaluation showing outcomes comparable to human tutors.
Beyond LLMs: A Linguistic Approach to Causal Graph Generation from Narrative Texts
This paper proposes a hybrid framework combining LLM summarization, a linguistically grounded Expert Index, and STAC classification to generate causal graphs from narrative texts, outperforming GPT-4o and Claude 3.5 on benchmark stories.
Error as a Lens: Probing LLM Reasoning through Synthetic Misconception Generation
This paper presents a framework using LLMs to generate targeted synthetic misconceptions aligned to a five-class taxonomy adapted from Bloom's taxonomy, addressing the scarcity of labeled student error data in education research.
OmniThoughtVis: A Scalable Distillation Pipeline for Deployable Multimodal Reasoning Models
This paper introduces OmniThoughtVis, a scalable pipeline for distilling multimodal reasoning capabilities from large teacher models to smaller, deployment-oriented MLLMs. The method uses curated chain-of-thought data to significantly improve reasoning performance on benchmarks like MathVerse and MMMU-Pro for models ranging from 2B to 8B parameters.