From Reasoning Depth to Reasoning Breadth: Evaluating Multi-Point Associative Reasoning in Large Language Models

# From Reasoning Depth to Reasoning Breadth: Evaluating Multi-Point Associative Reasoning in Large Language Models
Source: [https://arxiv.org/html/2608.10444](https://arxiv.org/html/2608.10444)
Si’an Xie1,∗, Jiaxun Liu2,∗, Biao Yang3, Wei Yuan3,†, Fan Yang3, Tingting Gao3, Ming Wu1,† 1Beijing University of Posts and Telecommunications 2Peking University 3Kuaishou Technology ∗Equal contribution\.†Corresponding authors

###### Abstract

Large language models \(LLMs\) have made substantial progress on reasoning tasks that require increasingly long and complex inferential chains\. This progress primarily reflectsreasoning depth\. A complementary and comparatively unexamined capability isreasoning breadth: exploring multiple semantic directions in parallel and integrating the resulting clues into one coherent answer\. We introduceMPAR\-Bench, a bilingual \(English–Chinese\) benchmark that isolates reasoning breadth through multi\-point associative reasoning\. Inspired by the cooperative gameJust One, each item asks a model to recover a hidden target from several independently generated, semantically diverse clues\. We construct 1,000 items using a multi\-agent clue\-generation pipeline, embedding\-based diversity filtering, and human verification\. Crucially, only the answer space is drawn from public word lists, whereas every clue set is generated from scratch\. Beyond exact\-match accuracy, we evaluate models along a coarse\-to\-fine protocol \(accuracy, ANLS, embedding similarity, and reasoning\-trace verification\) and a perturbation suite that probes four distinct robustness axes: clue masking, order shuffling, distractor injection, and multi\-step clues\. Across evaluated models, perturbations reduce accuracy by 9–18 points in English and 5–12 points in Chinese\. Enabling thinking mode improves standard\-setting accuracy—especially in English—but does not consistently reduce sensitivity to perturbations, and case\-level trace analysis shows that extended reasoning can overturn an initially correct hypothesis\. These results indicate that greater reasoning depth does not automatically confer robust reasoning breadth, and that breadth is a measurement axis that current benchmarks largely leave uncovered\.

## Introduction

Large language models \(LLMs\) have rapidly evolved from early neural language models to highly capable Transformer\-based systems\[[43](https://arxiv.org/html/2608.10444#bib.bib56)\]such as GPT, Gemini, Qwen, and others\[[35](https://arxiv.org/html/2608.10444#bib.bib21),[17](https://arxiv.org/html/2608.10444#bib.bib20),[18](https://arxiv.org/html/2608.10444#bib.bib19),[2](https://arxiv.org/html/2608.10444#bib.bib22),[50](https://arxiv.org/html/2608.10444#bib.bib24),[31](https://arxiv.org/html/2608.10444#bib.bib23),[4](https://arxiv.org/html/2608.10444#bib.bib25),[8](https://arxiv.org/html/2608.10444#bib.bib26)\]\. Through techniques such as reinforcement learning\[[36](https://arxiv.org/html/2608.10444#bib.bib27)\], supervised finetuning\[[47](https://arxiv.org/html/2608.10444#bib.bib48)\], Chain\-of\-Thought \(CoT\)\[[48](https://arxiv.org/html/2608.10444#bib.bib28)\]and Retrieval\-Augmented Generation\[[27](https://arxiv.org/html/2608.10444#bib.bib29)\], modern LLMs have achieved remarkable success and demonstrated near\-human performance in solving practical problems, which depends on their step\-by\-step linear reasoning\. Humans possess another important capability: multi\-point associative reasoning, which enables the structured integration of a wide range of concepts\. This ability allows humans not only to explore problems in depth, but also to bridge existing concepts and synthesize them into novel ideas\. These ability differences are shown in Fig\.[1](https://arxiv.org/html/2608.10444#Sx1.F1)\. An important question remains unresolved: do current LLMs possess the non\-linear, cross\-domain associative reasoning abilities?

![Refer to caption](https://arxiv.org/html/2608.10444v1/x1.png)Figure 1:Linear Reasoning and Multi\-Point Associative Reasoning Flow Chart\.![Refer to caption](https://arxiv.org/html/2608.10444v1/x2.png)Figure 2:A brief introduction of MPAR\-Bench\.Existing LLM benchmarks\[[21](https://arxiv.org/html/2608.10444#bib.bib32),[20](https://arxiv.org/html/2608.10444#bib.bib30),[13](https://arxiv.org/html/2608.10444#bib.bib31),[39](https://arxiv.org/html/2608.10444#bib.bib33),[29](https://arxiv.org/html/2608.10444#bib.bib55)\]primarily emphasize reasoning “depth,” evaluating step\-by\-step logical deduction and procedural reasoning\. In contrast, the evaluation of reasoning “breadth”—the ability to aggregate dispersed semantic signals and perform abstract conceptual convergence—remains largely unexplored\. This capability matters whenever the relevant evidence is distributed across different semantic perspectives rather than arranged as a single derivation\. Multi\-document synthesis, cross\-domain analogy, hypothesis generation, and reasoning under incomplete or distracting evidence all require a model to hold several partial relations in view and reconcile them into a final prediction\[[42](https://arxiv.org/html/2608.10444#bib.bib17)\]\. A model may therefore reason deeply along one path while still failing to combine information available across several paths\.

To fill this gap, we introduceMPAR\-Bench, a cognitively inspired benchmark designed to systematically evaluate multi\-point associative reasoning in LLMs\. To improve evaluation reliability and robustness, we further construct high\-quality bilingual test sets with 1,000 questions through a carefully designed multi\-agent generation and verification pipeline\[[28](https://arxiv.org/html/2608.10444#bib.bib36),[46](https://arxiv.org/html/2608.10444#bib.bib34),[16](https://arxiv.org/html/2608.10444#bib.bib35)\]\.

Beyond exact\-match metrics, we propose a fine\-grained evaluation framework that analyzes model behavior from reasoning perspectives\. This framework enables a more comprehensive investigation of how LLMs “reason broad”, offering deeper insight into the current capabilities and limitations of associative reasoning in modern language models\. A general overview of MPAR\-Bench is shown in Fig\.[2](https://arxiv.org/html/2608.10444#Sx1.F2)\.

In general, our contributions are summarized as follows:

- •A benchmark targeting reasoning breadth\.MPAR\-Bench operationalizes many\-to\-one integration from multiple, semantically diverse clues\. Unlike RAT\-style tests \(three fixed compound\-word cues\) and existing game benchmarks \(clue giving, or grouping a fixed word set\), MPAR\-Bench isolates the guesser\-side integration of an open number of free\-form clues, and pairs it with a controlled perturbation suite\.
- •An innovative multi\-agent clue\-synthesis pipeline\. We propose a multi\-agent collaborative clue\-synthesis framework with embedding\-based filtering and human verification, enabling the construction of semantically diverse, high\-difficulty evaluation instances while substantially reducing memorization risk\.
- •A coarse\-to\-fine evaluation protocol\.Beyond exact match, we combine ANLS, embedding similarity, and reasoning\-trace verification, and analyze robustness per perturbation type rather than as a single aggregate, exposing a blank space in reasoning breadth\.

## Related Work

### LLM in Reasoning Depth

The dominant trajectory of LLM reasoning research extends inferential depth\. Chain\-of\-thought prompting\[[48](https://arxiv.org/html/2608.10444#bib.bib28)\], tree\- and plan\-structured search\[[51](https://arxiv.org/html/2608.10444#bib.bib49),[45](https://arxiv.org/html/2608.10444#bib.bib45)\], self\-verification\[[49](https://arxiv.org/html/2608.10444#bib.bib46)\], and process supervision\[[30](https://arxiv.org/html/2608.10444#bib.bib38)\]all lengthen or stabilize a single reasoning trajectory, and reinforcement\-learned thinking modes push test\-time computation further\[[14](https://arxiv.org/html/2608.10444#bib.bib40)\]\. A parallel line of work documents the failure mode of overthinking, in which additional reasoning steps degrade rather than improve answers\[[12](https://arxiv.org/html/2608.10444#bib.bib41),[41](https://arxiv.org/html/2608.10444#bib.bib42)\]\. Depth\-oriented benchmarks—mathematical\[[21](https://arxiv.org/html/2608.10444#bib.bib32),[13](https://arxiv.org/html/2608.10444#bib.bib31)\], knowledge\-intensive\[[20](https://arxiv.org/html/2608.10444#bib.bib30)\], and broad\-coverage suites\[[39](https://arxiv.org/html/2608.10444#bib.bib33),[29](https://arxiv.org/html/2608.10444#bib.bib55)\]—are increasingly saturated for frontier models\. These tasks answer how far a model can push one chain; they do not answer whether a model can integrate evidence across chains\. Reasoning breadth is thus an orthogonal axis, and one on which depth\-oriented benchmarks provide little discrimination\.

### Associative Reasoning

Associative reasoning has long been studied in cognitive psychology through convergent\-thinking instruments, most notably the Remote Associates Test \(RAT\)\[[33](https://arxiv.org/html/2608.10444#bib.bib2)\]\. RAT has recently been repurposed to probe LLMs:Schonet al\.\[[38](https://arxiv.org/html/2608.10444#bib.bib3)\]model associative reasoning processes,Kumaret al\.\[[25](https://arxiv.org/html/2608.10444#bib.bib4)\]study human–AI convergent and divergent thinking, and generative models have been reported to match or exceed humans on such tests\[[9](https://arxiv.org/html/2608.10444#bib.bib57),[3](https://arxiv.org/html/2608.10444#bib.bib58)\]—though such results are hard to interpret, since the test items are publicly available and may have been seen during pre\-training\[[15](https://arxiv.org/html/2608.10444#bib.bib59),[11](https://arxiv.org/html/2608.10444#bib.bib16)\]\.

Related open\-ended formulations argue for process\-based rather than multiple\-choice evaluation\[[23](https://arxiv.org/html/2608.10444#bib.bib5)\]and for generating explicit associative paths\[[44](https://arxiv.org/html/2608.10444#bib.bib6)\]\. We inherit the construct of convergent association from this tradition but deliberately depart from its instrument: RAT items are public, largely three\-cue, and dominated by fixed phrasal collocations that next token prediction learns readily\. MPAR\-Bench instead poses an open, variable number of free\-form semantic clues that must be integrated, with every clue set synthesized de novo to substantially reduce overlap with public clue–target pairings and lower the risk of memorization\. This is what MPAR\-Bench adds over prior RAT\-on\-LLM evaluations: a breadth\-oriented benchmark with reduced contamination risk, rather than a re\-run of a public convergent\-thinking test\.

![Refer to caption](https://arxiv.org/html/2608.10444v1/x3.png)Figure 3:Introduction of Board GameJust One\.
### Boardgame\-Based Benchmarks

Cooperative and word\-association games offer constrained rules with large state spaces, which helps mitigate contamination and yields human\-aligned semantic tasks\. Codenames has been used to evaluate one\-to\-many clue giving and ad\-hoc concept forming\[[40](https://arxiv.org/html/2608.10444#bib.bib13),[19](https://arxiv.org/html/2608.10444#bib.bib14)\]; the NYT Connections game requires partitioning a fixed set of words into latent groups\[[34](https://arxiv.org/html/2608.10444#bib.bib1)\]; and the Word Synchronization Challenge measures two agents converging on a shared word without communication\[[10](https://arxiv.org/html/2608.10444#bib.bib15)\]\. MPAR\-Bench differs along three axes:\(i\) clue cardinality—models must jointly integrate an open, variable number of clues rather than fixed cues;\(ii\) association type—clues are free\-form semantic descriptions spanning lexical, cultural, phonetic, and world\-knowledge relations, not compound\-word completions or fixed candidate pools; and\(iii\) item availability—all clue sets are synthesized from scratch, leaving no public clue–target pairing to memorize\.

## Methodology

### Task Definition

Given a clue setC=\{c1,c2,…,cn\}C=\\\{c\_\{1\},c\_\{2\},\\ldots,c\_\{n\}\\\}, the task is to recover a targetyysuch that each cluecic\_\{i\}contributes an*independently informative*semantic relation toyy\. Reasoning breadth, in this setting, is the ability to integrate multiple semantically distinct and non\-redundant clues into a single coherent answer\.

We ensure that each item genuinely requires breadth through two construction\-side safeguards\. First, clues are generated from diverse semantic angles to maximize the range of associations a model must reconcile\. Second, a judge agent and an embedding\-based filter remove synonyms, paraphrases, and near\-duplicates, so that each retained clue carries non\-overlapping information\. At evaluation time, we measure not only whether a model’s prediction matches the target, but also whether that prediction remains stable under perturbation—clue masking, order shuffling, distractor injection, and multi\-step inference—which probes whether the integration is robust or merely superficial\.

![Refer to caption](https://arxiv.org/html/2608.10444v1/x4.png)Figure 4:Word Cloud of MPAR\-Bench\.
### Just One

The design borrows the constraint structure of the cooperative gameJust One, in which players give single\-word hints to help a guesser infer a hidden target, while direct synonyms, translations, homophones, and duplicate clues are forbidden\. These constraints are what make the game a clean instrument for breadth\. Rather than rewarding the most obvious lexical association, they force clue writers to approach the target from distinct, indirect angles; the guesser must then integrate fragmented, non\-overlapping signals rather than pattern\-match a single cue\. This yields a constrained multi\-point associative reasoning task emphasizing semantic abstraction, conceptual bridging, and integration\. An illustrative round is shown in Fig\.[3](https://arxiv.org/html/2608.10444#Sx2.F3)\.

### Dataset Construction

Answer space\.Target words are drawn from public word lists—RAT\-derived vocabulary \(collected on the internet\) andJust Oneword cards\.

Multi\-agent clue generation\.Given a target and the clues already accepted, LLM\-based agents iteratively propose new clues that remain semantically relevant to the target while minimizing redundancy with existing clues\. Each agent is assigned a distinct association angle to encourage coverage across semantic directions\. A judge agent then removes clues that are the answer itself, direct synonyms/translations/homophones/morphological variants, exact or near\-duplicates of accepted clues, or genuinely low quality\[[32](https://arxiv.org/html/2608.10444#bib.bib62),[5](https://arxiv.org/html/2608.10444#bib.bib61)\]\. This division of labor mirrors the independent clue\-provider and arbiter structure ofJust One\. Notably, all agents’ prompt are provided in Appendix\.

Embedding\-based diversity filtering\.We use Qwen3\-Embedding\-8B\[[52](https://arxiv.org/html/2608.10444#bib.bib12)\]to score clue–answer and clue–clue similarity, discarding clues that are either trivially close to the answer or too weakly related to be informative, and clue pairs that are near\-duplicates\[[1](https://arxiv.org/html/2608.10444#bib.bib60)\]\. An experimental embedding similarity threshold ranges from 0\.3 to 0\.8, which serves as a primary filtering step in benchmark construction\.

Answer uniqueness and graded acceptability\.A central concern is that a clue set may admit more than one reasonable target\. We address this in two ways\. \(i\)*Construction:*the judge stage filters out clue sets that jointly under\-determine the target, and problematic items are reconstructed\. \(ii\)*Human verification:*on a randomly sampled subset of 250 items, two master’s students majoring in NLP independently assess whether each item admits a unique, unambiguous target\. Table[1](https://arxiv.org/html/2608.10444#Sx3.T1)reports the results with 95% Wilson CI, indicating that 92\.8% of the items were judged as having a unique answer\.

CategoryCountRate95% Wilson CIUnique23292\.8%\[88\.6%, 95\.4%\]Ambiguity187\.2%\[4\.6%, 11\.1%\]Table 1:Answer uniqueness with 95% Wilson CIBilingual design\.MPAR\-Bench includes English and Chinese subsets built with the same pipeline \(500 validated items each\), combining synthesized and native\-speaker\-authored samples\. The English subset emphasizes lexical and abstract associations; the Chinese subset additionally incorporates idioms, character\-level and pictographic properties, and contemporary cultural memes\. Word clouds for each subset are shown in Fig\.[4](https://arxiv.org/html/2608.10444#Sx3.F4)\.

Benchmark rationale\.MPAR\-Bench is intentionally constructed to emphasize long\-range, multi\-point association rather than lexical overlap or frequent collocations\. By synthesizing clues from complementary perspectives while enforcing low redundancy, each item requires integrating sparse and semantically distant evidence into a single target, making successful prediction less dependent on next token co\-occurrence patterns and more on associative reasoning\.

![Refer to caption](https://arxiv.org/html/2608.10444v1/x5.png)Figure 5:Examples of Enhanced settings in MPAR\-Bench\.
### Difficulty Settings and Perturbation

We distinguish two complementary facets of reasoning breadth\. The Standard setting measures baseline breadth: given complete, well\-formed clues under ideal conditions, can the model integrate multiple semantic signals into a correct answer? To further evaluate whether this integration capability remains reliable under more realistic conditions, we introduce an Enhanced setting that systematically perturbs the standard test protocol to simulate information\-restricted or noisy environments\. These two settings provide a complete picture: the Standard setting establishes what a model can achieve under favorable conditions, while the Enhanced setting reveals whether that capability is resilient enough to matter in practice\[[53](https://arxiv.org/html/2608.10444#bib.bib63)\]\.

Specifically, we implement the following enhanced transformations and perturbations:

- •Clue Masking:Randomly masking clues to evaluate model reasoning ability under information deficiency\.
- •Order Shuffling:Shuffling the clues order to find whether the model’s reasoning process is sensitive to order\.
- •Distractors:Injecting semantically misleading or irrelevant cue words to test model resistance to noisy contexts and spurious correlations\.
- •Multi\-step Inferring:Increasing the associative semantic distance between clues and the mystery word, forcing models to generate intermediate latent connections rather than relying on direct surface co\-occurrence\.

Each enhanced setting evenly distributes words from the standard task\. We refer to this setting as the Enhanced MPAR\-Bench in the remainder of the paper\. As shown in Fig\.[5](https://arxiv.org/html/2608.10444#Sx3.F5), each question has a corresponding enhanced variant, posing a greater challenge to LLMs across multiple dimensions of robustness\.

## Evaluation

Associative reasoning cannot be fully captured by a single exact\-match metric, since semantically reasonable predictions may differ lexically from the ground truth, and some correct predictions may come from flawed reasoning processes\. To obtain a more comprehensive understanding of LLM behavior, we evaluate models at three progressively finer granularities: accuracy, word\-based similarity, and the validity of reasoning trace, i\.e\., the explanation of how the answer and clues are connecting with each other\.

### Accuracy

We first evaluate model performance using exact\-match accuracy\. A prediction is considered correct if and only if it exactly matches the ground\-truth answer\. We report accuracy across different subsets, including multilingual Standard and Enhanced settings, to measure models’ associative retrieval capability under varying semantic and linguistic conditions\.

ModelEnglishChineseAccANLSEmbTraceAccANLSEmbTraceGPT\-5\.277\.60\.7920\.8480\.93364\.40\.7400\.8230\.954Gemini\-3\.1pro86\.80\.8840\.9150\.94172\.20\.8020\.8690\.957Sonnet\-4\.579\.00\.8030\.8560\.86567\.40\.7670\.8390\.927Qwen3\-max73\.20\.7430\.8000\.87865\.00\.7330\.8130\.928Kimi\-k271\.60\.7290\.7950\.87157\.40\.6750\.7700\.888Deepseek\-v3\.269\.80\.7160\.7840\.84861\.20\.6860\.7930\.893Seed\-2\-pro71\.40\.7240\.8020\.87564\.60\.7390\.8200\.937Table 2:Standard MPAR\-Bench Results on Thinking ModelsModelEnglishChineseAccANLSEmbTraceAccANLSEmbTraceGPT\-5\.266\.60\.6810\.7640\.90258\.80\.6690\.7720\.932Gemini\-3\.1pro76\.90\.7780\.8420\.90664\.00\.7220\.8150\.926Sonnet\-4\.563\.40\.6460\.7390\.83661\.00\.6890\.7890\.873Qwen3\-max56\.80\.5910\.6790\.84154\.80\.6210\.7320\.911Kimi\-k257\.60\.5960\.6870\.82745\.80\.5470\.6830\.850Deepseek\-v3\.252\.00\.5310\.6390\.79452\.40\.5960\.7210\.842Seed\-2\-pro56\.60\.5790\.6820\.81758\.20\.6660\.7680\.887Table 3:Enhanced MPAR\-Bench Results on Thinking Models
### Word\-based Evaluation

To further evaluate semantic proximity between predictions and target concepts, we adopt both Average Normalized Levenshtein Similarity \(ANLS\) and word embedding similarity to judge at the semantic level\.

ANLS\[[6](https://arxiv.org/html/2608.10444#bib.bib37)\]is computed using normalized Levenshtein edit distance:

ANLS​\(y^,y\)=1−dlev​\(y^,y\)max⁡\(\|y^\|,\|y\|\),\\text\{ANLS\}\(\\hat\{y\},y\)=1\-\\frac\{d\_\{\\text\{lev\}\}\(\\hat\{y\},y\)\}\{\\max\(\|\\hat\{y\}\|,\|y\|\)\},\(1\)wherey^\\hat\{y\}andyydenote the model prediction and ground truth respectively,dlev​\(⋅,⋅\)d\_\{\\text\{lev\}\}\(\\cdot,\\cdot\)denotes the Levenshtein edit distance, and\|⋅\|\|\\cdot\|denotes string length\.

Moreover, we compute word embedding similarity usingfastText\[[7](https://arxiv.org/html/2608.10444#bib.bib18)\]as a word embedding model:

Sim​\(y^e​m​b,ye​m​b\)=y^e​m​b⊤​ye​m​b‖y^e​m​b‖2​‖ye​m​b‖2\.\\text\{Sim\}\(\\hat\{y\}\_\{emb\},y\_\{emb\}\)=\\frac\{\\hat\{y\}^\{\\top\}\_\{emb\}y\_\{emb\}\}\{\\\|\\hat\{y\}\_\{emb\}\\\|\_\{2\}\\\|y\_\{emb\}\\\|\_\{2\}\}\.\(2\)wherey^e​m​b\\hat\{y\}\_\{emb\}andye​m​by\_\{emb\}denote the word embedding of the model prediction and ground truth infastTextrespectively, and∥⋅∥2\\\|\\cdot\\\|\_\{2\}denotes their Euclidean \(ℓ2\\ell\_\{2\}\) norms\.

### Reasoning Trace Evaluation

Beyond final\-answer accuracy, we assess the validity of intermediate processes via reasoning trace evaluation, decomposed into two dimensions: logical verification and factual verification\[[26](https://arxiv.org/html/2608.10444#bib.bib39),[49](https://arxiv.org/html/2608.10444#bib.bib46),[30](https://arxiv.org/html/2608.10444#bib.bib38)\]\. Logical verification examines whether reasoning trajectories follow coherent inferential steps from clues to predictions, while factual verification checks whether intermediate claims are factually grounded\. This dual assessment distinguishes valid associative reasoning from spurious correlations and hallucinated paths\. We manually review a randomly sampled subset of 300 reasoning trace predictions, confirming high consistency \(98\.7% on factual verification, 94\.7% on logical verification\) between human and LLM judgement\. Evaluation prompts are provided in Appendix\.

## Experiments and Results

ModelEnglishChineseAccANLSEmbTraceAccANLSEmbTraceGPT\-5\.259\.60\.6140\.7050\.81161\.80\.7180\.8040\.943Gemini\-3flash70\.00\.7170\.7870\.83167\.00\.7650\.8360\.920Sonnet\-4\.570\.40\.7160\.7910\.83868\.80\.7760\.8430\.921Qwen3\-max55\.40\.5720\.6690\.79264\.40\.7480\.8200\.930DeepSeek\-v3\.251\.40\.5360\.6440\.73060\.60\.7020\.7880\.906Seed\-2\-pro59\.80\.6190\.7110\.74364\.20\.7260\.8170\.910Table 4:Standard MPAR\-Bench Results on Non\-thinking ModelsModelEnglishChineseAccANLSEmbTraceAccANLSEmbTraceGPT\-5\.244\.20\.4600\.5860\.75454\.40\.6320\.7490\.906Gemini\-3flash56\.00\.5800\.6750\.76361\.40\.6950\.7910\.870Sonnet\-4\.555\.80\.5720\.6790\.77462\.00\.7070\.7970\.865Qwen3\-max44\.80\.4640\.5860\.75156\.00\.6620\.7580\.874Deepseek\-v3\.242\.60\.4440\.5660\.68554\.00\.6180\.7380\.851Seed\-2\-pro43\.40\.4520\.5860\.69356\.00\.6460\.7610\.866Table 5:Enhanced MPAR\-Bench Results on Non\-thinking Models### Implementation Details

We benchmark a diverse set of representative LLM families—including GPT, Gemini, Sonnet, Qwen, Kimi, DeepSeek, and Seed series\[[35](https://arxiv.org/html/2608.10444#bib.bib21),[17](https://arxiv.org/html/2608.10444#bib.bib20),[18](https://arxiv.org/html/2608.10444#bib.bib19),[2](https://arxiv.org/html/2608.10444#bib.bib22),[50](https://arxiv.org/html/2608.10444#bib.bib24),[31](https://arxiv.org/html/2608.10444#bib.bib23),[4](https://arxiv.org/html/2608.10444#bib.bib25),[8](https://arxiv.org/html/2608.10444#bib.bib26)\]; the concrete information is shown in Appendix\. Evaluations are conducted across both thinking and non\-thinking modes, under standard and enhanced settings, on the bilingual subsets of MPAR\-Bench\.

We also report fine\-grained metrics including ANLS, word embedding similarity, and reasoning trace failure analysis to dissect the reasoning behaviors of LLMs from lexical, semantic, and reasoning process perspectives\.

### Main Results

![Refer to caption](https://arxiv.org/html/2608.10444v1/x6.png)Figure 6:Challenging Cases in MPAR\-Bench\. Examples where most models fail to identify the correct answer\.We analyze the results from three key perspectives: cross\-model comparison, the effect of thinking mode, and robustness under perturbation\. Challenging cases in MPAR\-Bench are shown in Fig\.[6](https://arxiv.org/html/2608.10444#Sx5.F6), and more detailed experimental results are provided in Appendix\.

#### Model Comparisons\.

Table[2](https://arxiv.org/html/2608.10444#Sx4.T2)and Table[4](https://arxiv.org/html/2608.10444#Sx5.T4)report the standard MPAR\-Bench results under thinking and non\-thinking modes\. Under the thinking mode, Gemini\-3\.1pro leads on both English \(86\.8%\) and Chinese \(72\.2%\), followed by GPT\-5\.2 and Sonnet\-4\.5 in English\. Under the non\-thinking mode, Sonnet\-4\.5 achieves the highest accuracy in both languages\.

#### Standard vs\. Enhanced\.

Comparing standard \(Tables[2](https://arxiv.org/html/2608.10444#Sx4.T2),[4](https://arxiv.org/html/2608.10444#Sx5.T4)\) with enhanced settings \(Tables[3](https://arxiv.org/html/2608.10444#Sx4.T3),[5](https://arxiv.org/html/2608.10444#Sx5.T5)\) reveals consistent degradation under perturbation\. For instance, in thinking mode, Deepseek\-v3\.2 exhibits the largest decline in English, while Kimi\-k2 drops the most in Chinese\. In non\-thinking mode, Seed\-2\-pro shows the largest decline in English, whereas Qwen3\-max drops the sharpest in Chinese\. These contrasts indicate that robustness varies substantially across models: some suffer pronounced degradation under perturbation while others remain comparatively stable\. Notably, this robustness is strongly model\- and language\-dependent\. Detailed information and analyses are shown in Appendix\.

#### Thinking vs\. Non\-Thinking\.

Comparing thinking\-mode results \(Tables[2](https://arxiv.org/html/2608.10444#Sx4.T2),[3](https://arxiv.org/html/2608.10444#Sx4.T3)\) against their non\-thinking counterparts \(Tables[4](https://arxiv.org/html/2608.10444#Sx5.T4),[5](https://arxiv.org/html/2608.10444#Sx5.T5)\), we find that thinking mode\[[30](https://arxiv.org/html/2608.10444#bib.bib38),[14](https://arxiv.org/html/2608.10444#bib.bib40)\]consistently improves most indicators, but the magnitude of the gain is markedly larger on English than on Chinese: averaged across models, thinking lifts English accuracy by a substantially wider margin and produces clear, stable gains for every model, whereas its effect on Chinese is much smaller and model\-dependent \(Sonnet\-4\.5 even shows a slight regression\), and the improvement under perturbation is non\-monotonic\. Moreover, thinking mode is not always reliable: models occasionally overthink and override correct intermediate answers, as discussed in the next section and Appendix\. This suggests a crucial distinction: while thinking mode improves reasoning depth, it may not necessarily enhance reasoning breadth\.

## Discussion

The benchmark results above establish that current LLMs exhibit measurable but imperfect reasoning breadth, and that this capability degrades under perturbation\. A natural follow\-up question is: under what conditions does reasoning breadth improve, and what mechanisms cause it to fail? We investigate this question through four complementary lenses\. First, we examine overthinking, a failure mode in which extended reasoning actively harms breadth by overriding correct intermediate answers\. Second, we characterize information gain curve, asking how breadth scales as more clues become available\. Third, we explore scaling laws to determine whether larger models inherently develop broader reasoning\. Fourth, we test whether semantic feedback can steer models toward correct answers across multiple refinement rounds\. We additionally try to create a structured reasoning skill as an intervention strategy; results are reported in Appendix\.

### Overthinking

A notable failure pattern we observe is overthinking\[[12](https://arxiv.org/html/2608.10444#bib.bib41),[41](https://arxiv.org/html/2608.10444#bib.bib42)\]: models initially arrive at the correct answer but subsequently override it during extended reasoning, often drifting toward a semantically related but incorrect concept\. This behavior is particularly pronounced in Qwen3\-max and Kimi\-k2\. For instance, given the answer wordPhilosophy, the model outputsPlato, over\-focusing on a representative entity implied by the clues rather than the academic discipline itself\.

Table[6](https://arxiv.org/html/2608.10444#Sx6.T6)reports a detailed overthinking analysis\. “Wrong Ex\.” is the total count of incorrect predictions\. “Ans\. Mention” \(ratio\) is the proportion of those incorrect cases in which the correct answer appeared in the reasoning trace but was subsequently overridden\. “Token Len\.” \(ratio\) is the proportion of incorrect cases whose reasoning length exceeded the model’s average reasoning length on correct samples\.

ModelEnglishChineseWrongEx\.Ans\.MentionTokenLen\.WrongEx\.Ans\.MentionTokenLen\.Sonnet\-4\.51050\.4570\.3711630\.7060\.571Qwen3\-max1340\.5900\.5151740\.8740\.690Kimi\-k21420\.5850\.4152130\.8640\.366Deepseek\-v3\.21510\.6230\.5101940\.8300\.634Seed\-2\-pro1430\.4970\.4061770\.7510\.599Table 6:Overthinking Results on models\.![Refer to caption](https://arxiv.org/html/2608.10444v1/x7.png)Figure 7:Information Gain Curve of Seed\-2\-pro\. As number of words increases, accuracy rises but at a decreasing rate\.
### Information Gain Curve

The information gain curve characterizes how a model’s reasoning accuracy scales with the number of provided clue words; we utilize it to evaluate Seed\-2\-pro’s capacity to leverage incremental semantic evidence for multi\-point associative reasoning\. As shown in Fig\.[7](https://arxiv.org/html/2608.10444#Sx6.F7), accuracy consistently improves as the number of clue words increases, suggesting that the model is able to accumulate and integrate incremental semantic information across multiple clues\. This provides preliminary evidence that LLMs possess multi\-point associative reasoning capability—the ability to jointly combine several semantically distinct clues into a coherent answer\. However, the marginal gain progressively slows down, indicating that while models benefit from richer semantic context, they saturate beyond a certain evidence threshold\.

### Scaling Law

![Refer to caption](https://arxiv.org/html/2608.10444v1/x8.png)Figure 8:Scaling Law of Qwen3 family in Chinese MPAR\-Bench\. As model scales up, all metrics improve, except for Qwen3\-32B, which has a more severe overthinking issue\.We investigate how model scale affects performance on our benchmark\. In particular, Fig\.[8](https://arxiv.org/html/2608.10444#Sx6.F8)reports the scaling behavior of the locally deployed Qwen3 family in thinking mode \(0\.6B to 32B parameters\), evaluated on accuracy, ANLS, and fastText embedding similarity across both English and Chinese datasets under Standard and Enhanced settings\. Except for Qwen3\-32B, accuracy consistently improves with increasing model size\[[24](https://arxiv.org/html/2608.10444#bib.bib43),[22](https://arxiv.org/html/2608.10444#bib.bib44),[37](https://arxiv.org/html/2608.10444#bib.bib53)\]\. Case\-level inspection finds that Qwen3\-32B suffers from overthinking, causing it to reject correct answers during an extended reasoning process\.

### Feedback

The proposed feedback method iteratively delivers semantic similarity metrics \(ANLS and average word embedding similarity\), guiding the model to refine its responses toward the answer across multiple rounds\. Fig\.[9](https://arxiv.org/html/2608.10444#Sx6.F9)visualizes the corresponding trajectories \(rounds 3–6\) where the mechanism successfully corrected Qwen3\-max’s outputs\. The results show that LLMs are not fully sensitive to word embedding similarity as a guidance signal; instead, they rely on other internal strategies\. While feedback provides opportunities for exploration and sometimes enables recovery of the answer, the revision trajectory remains weakly aligned with semantic indicators, suggesting that models do not naturally exploit surface\-level semantic proximity for iterative refinement\.

![Refer to caption](https://arxiv.org/html/2608.10444v1/x9.png)Figure 9:Chinese MPAR\-Bench feedback results for Qwen3\-max in thinking mode, rounds 3–6\. Blue lines show individual cases; green lines show average trends\.

## Conclusion

We introducedMPAR\-Bench, a bilingual benchmark that evaluates multi\-point associative reasoning—reasoningbreadth—in LLMs through 1,000 boardgame\-rule\-based questions and a coarse\-to\-fine evaluation protocol spanning accuracy, ANLS, embedding similarity, and reasoning\-trace verification, complemented by a four\-axis perturbation suite\.

Our experiments yield three findings\. First, reasoning breadth remains far from solved: the best models reach 86\.8%/72\.2% accuracy in English/Chinese, with perturbations causing 9–18/5–12 point drops\. Second, greater reasoning depth does not automatically confer breadth: thinking mode improves standard\-setting accuracy but does not consistently reduce perturbation sensitivity, and case\-level analysis reveals that extended reasoning can override correct answers through overthinking\. Third, improving breadth appears challenging: scaling model size, adding reasoning strategies, and iterative feedback each bring only partial gains, suggesting that reasoning breadth may be a capability that current training paradigms do not naturally optimize for\. We release MPAR\-Bench and its pipeline to encourage the community to move beyond depth\-oriented evaluation and toward a more complete picture of reasoning—one that values breadth as much as depth\.

## References

- \[1\]\(2021\)Semdedup: data\-efficient learning at web\-scale through semantic deduplication, 2023\.URL https://arxiv\. org/abs/2303\.09540\.Cited by:[Dataset Construction](https://arxiv.org/html/2608.10444#Sx3.SSx3.p3.1)\.
- \[2\]Anthropic\(2025\)Introducing claude sonnet 4\.5\.Note:[https://www\.anthropic\.com/news/claude\-sonnet\-4\-5](https://www.anthropic.com/news/claude-sonnet-4-5)Cited by:[Introduction](https://arxiv.org/html/2608.10444#Sx1.p1.1),[Implementation Details](https://arxiv.org/html/2608.10444#Sx5.SSx1.p1.1)\.
- \[3\]V\. Arora, A\. Thabane, S\. Parpia, G\. Calic, and M\. Bhandari\(2025\)Generative artificial intelligence models outperform students on divergent and convergent thinking assessments\.Scientific Reports15\(1\),pp\. 36987\.Cited by:[Associative Reasoning](https://arxiv.org/html/2608.10444#Sx2.SSx2.p1.1)\.
- \[4\]Y\. Bai, Y\. Bao, Y\. Charles, C\. Chen, G\. Chen, H\. Chen, H\. Chen, J\. Chen, N\. Chen,et al\.\(2025\)Kimi k2: open agentic intelligence\.arXiv preprint arXiv:2507\.20534\.Cited by:[Introduction](https://arxiv.org/html/2608.10444#Sx1.p1.1),[Implementation Details](https://arxiv.org/html/2608.10444#Sx5.SSx1.p1.1)\.
- \[5\]A\. Bavaresco, R\. Bernardi, L\. Bertolazzi, D\. Elliott, R\. Fernández, A\. Gatt, E\. Ghaleb, M\. Giulianelli, M\. Hanna, A\. Koller,et al\.\(2025\)Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),pp\. 238–255\.Cited by:[Dataset Construction](https://arxiv.org/html/2608.10444#Sx3.SSx3.p2.1)\.
- \[6\]A\. F\. Biten, R\. Tito, A\. Mafla, L\. Gomez, M\. Rusiñol, E\. Valveny, C\. V\. Jawahar, and D\. Karatzas\(2019\)Scene text visual question answering\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 4291–4301\.Cited by:[Word\-based Evaluation](https://arxiv.org/html/2608.10444#Sx4.SSx2.p2.5)\.
- \[7\]P\. Bojanowski, E\. Grave, A\. Joulin, and T\. Mikolov\(2017\)Enriching word vectors with subword information\.Transactions of the Association for Computational Linguistics5,pp\. 135–146\.External Links:ISSN 2307\-387XCited by:[Word\-based Evaluation](https://arxiv.org/html/2608.10444#Sx4.SSx2.p3.5)\.
- \[8\]Bytedance\(2026\)SEED 2\.0\.Note:[https://seed\.bytedance\.com/en/blog/seed\-2\-0\-official\-launch](https://seed.bytedance.com/en/blog/seed-2-0-official-launch)Cited by:[Introduction](https://arxiv.org/html/2608.10444#Sx1.p1.1),[Implementation Details](https://arxiv.org/html/2608.10444#Sx5.SSx1.p1.1)\.
- \[9\]A\. Carolus, M\. J\. Koch, and S\. Feng\(2025\)Time\-on\-task and instructions help humans to keep up with ai: replication and extension of a comparison of creative performances\.Scientific reports15\(1\),pp\. 20173\.Cited by:[Associative Reasoning](https://arxiv.org/html/2608.10444#Sx2.SSx2.p1.1)\.
- \[10\]T\. Cazalets and J\. Dambre\(2025\)Word synchronization challenge: a benchmark for word association responses for large language models\.InInternational Conference on Human\-Computer Interaction,pp\. 3–19\.Cited by:[Boardgame\-Based Benchmarks](https://arxiv.org/html/2608.10444#Sx2.SSx3.p1.1)\.
- \[11\]S\. Chen, Y\. Chen, Z\. Li, Y\. Jiang, Z\. Wan, Y\. He, D\. Ran, T\. Gu, H\. Li, T\. Xie,et al\.\(2025\)Benchmarking large language models under data contamination: a survey from static to dynamic evaluation\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 10091–10109\.Cited by:[Associative Reasoning](https://arxiv.org/html/2608.10444#Sx2.SSx2.p1.1)\.
- \[12\]X\. Chen, J\. Xu, T\. Liang, Z\. He, J\. Pang, D\. Yu, L\. Song, Q\. Liu, M\. Zhou, Z\. Zhang, R\. Wang, Z\. Tu, H\. Mi, and D\. Yu\(2025\)Do NOT think that much for 2\+3=? on the overthinking of o1\-like LLMs\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[LLM in Reasoning Depth](https://arxiv.org/html/2608.10444#Sx2.SSx1.p1.1),[Overthinking](https://arxiv.org/html/2608.10444#Sx6.SSx1.p1.1)\.
- \[13\]K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[Introduction](https://arxiv.org/html/2608.10444#Sx1.p2.1),[LLM in Reasoning Depth](https://arxiv.org/html/2608.10444#Sx2.SSx1.p1.1)\.
- \[14\]DeepSeek\-AI\(2025\)DeepSeek\-R1: incentivizing reasoning capability in LLMs via reinforcement learning\.Nature\.Cited by:[LLM in Reasoning Depth](https://arxiv.org/html/2608.10444#Sx2.SSx1.p1.1),[Thinking vs\. Non\-Thinking\.](https://arxiv.org/html/2608.10444#Sx5.SSx2.SSS0.Px3.p1.1)\.
- \[15\]C\. Deng, Y\. Zhao, Y\. Heng, Y\. Li, J\. Cao, X\. Tang, and A\. Cohan\(2024\)Unveiling the spectrum of data contamination in language model: a survey from detection to remediation\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 16078–16092\.Cited by:[Associative Reasoning](https://arxiv.org/html/2608.10444#Sx2.SSx2.p1.1)\.
- \[16\]Y\. Du, S\. Li, A\. Torralba, J\. B\. Tenenbaum, and I\. Mordatch\(2024\)Improving factuality and reasoning in language models through multiagent debate\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),Cited by:[Introduction](https://arxiv.org/html/2608.10444#Sx1.p3.1)\.
- \[17\]Google\(2025\)Gemini 3 flash: frontier intelligence built for speed\.Note:[https://blog\.google/products\-and\-platforms/products/gemini/gemini\-3\-flash/](https://blog.google/products-and-platforms/products/gemini/gemini-3-flash/)Cited by:[Introduction](https://arxiv.org/html/2608.10444#Sx1.p1.1),[Implementation Details](https://arxiv.org/html/2608.10444#Sx5.SSx1.p1.1)\.
- \[18\]Google\(2026\)Gemini 3\.1 pro: a smarter model for your most complex tasks\.Note:[https://blog\.google/innovation\-and\-ai/models\-and\-research/gemini\-models/gemini\-3\-1\-pro/](https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-pro/)Cited by:[Introduction](https://arxiv.org/html/2608.10444#Sx1.p1.1),[Implementation Details](https://arxiv.org/html/2608.10444#Sx5.SSx1.p1.1)\.
- \[19\]S\. Hakimov, L\. Pfennigschmidt, and D\. Schlangen\(2025\)Ad\-hoc concept forming in the game codenames as a means for evaluating large language models\.InProceedings of the Fourth Workshop on Generation, Evaluation and Metrics \(GEM2\),pp\. 728–740\.Cited by:[Boardgame\-Based Benchmarks](https://arxiv.org/html/2608.10444#Sx2.SSx3.p1.1)\.
- \[20\]D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt\(2021\)Measuring massive multitask language understanding\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[Introduction](https://arxiv.org/html/2608.10444#Sx1.p2.1),[LLM in Reasoning Depth](https://arxiv.org/html/2608.10444#Sx2.SSx1.p1.1)\.
- \[21\]D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt\(2021\)Measuring mathematical problem solving with the MATH dataset\.InAdvances in Neural Information Processing Systems Track on Datasets and Benchmarks \(NeurIPS\),Cited by:[Introduction](https://arxiv.org/html/2608.10444#Sx1.p2.1),[LLM in Reasoning Depth](https://arxiv.org/html/2608.10444#Sx2.SSx1.p1.1)\.
- \[22\]J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. de Las Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark,et al\.\(2022\)Training compute\-optimal large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.35,pp\. 30016–30030\.Cited by:[Scaling Law](https://arxiv.org/html/2608.10444#Sx6.SSx3.p1.1)\.
- \[23\]Z\. Huang, J\. Ke, X\. Fan, Y\. Yang, Y\. Liu, L\. Zhonghan, Z\. Wang, J\. Dai, H\. Jiang, Y\. Zhou,et al\.\(2025\)MM\-opera: benchmarking open\-ended association reasoning for large vision\-language models\.arXiv preprint arXiv:2510\.26937\.Cited by:[Associative Reasoning](https://arxiv.org/html/2608.10444#Sx2.SSx2.p2.1)\.
- \[24\]J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. Amodei\(2020\)Scaling laws for neural language models\.arXiv preprint arXiv:2001\.08361\.Cited by:[Scaling Law](https://arxiv.org/html/2608.10444#Sx6.SSx3.p1.1)\.
- \[25\]H\. Kumar, J\. Vincentius, E\. Jordan, and A\. Anderson\(2025\)Human creativity in the age of llms: randomized experiments on divergent and convergent thinking\.InProceedings of the 2025 CHI conference on human factors in computing systems,pp\. 1–18\.Cited by:[Associative Reasoning](https://arxiv.org/html/2608.10444#Sx2.SSx2.p1.1)\.
- \[26\]T\. Lanham, A\. Chen, A\. Radhakrishnan, B\. Steiner, C\. Denison, D\. Hernandez, D\. Li, E\. Durmus, E\. Hubinger, J\. Kernion,et al\.\(2023\)Measuring faithfulness in chain\-of\-thought reasoning\.arXiv preprint arXiv:2307\.13702\.Cited by:[Reasoning Trace Evaluation](https://arxiv.org/html/2608.10444#Sx4.SSx3.p1.1)\.
- \[27\]P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. Kiela\(2020\)Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.33,pp\. 9459–9474\.Cited by:[Introduction](https://arxiv.org/html/2608.10444#Sx1.p1.1)\.
- \[28\]G\. Li, H\. A\. A\. K\. Hammoud, H\. Itani, D\. Khizbullin, and B\. Ghanem\(2023\)CAMEL: communicative agents for “mind” exploration of large language model society\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.36\.Cited by:[Introduction](https://arxiv.org/html/2608.10444#Sx1.p3.1)\.
- \[29\]P\. Liang, R\. Bommasani, T\. Lee,et al\.\(2023\)Holistic evaluation of language models\.Transactions on Machine Learning Research \(TMLR\)\.Cited by:[Introduction](https://arxiv.org/html/2608.10444#Sx1.p2.1),[LLM in Reasoning Depth](https://arxiv.org/html/2608.10444#Sx2.SSx1.p1.1)\.
- \[30\]H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe\(2024\)Let’s verify step by step\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[LLM in Reasoning Depth](https://arxiv.org/html/2608.10444#Sx2.SSx1.p1.1),[Reasoning Trace Evaluation](https://arxiv.org/html/2608.10444#Sx4.SSx3.p1.1),[Thinking vs\. Non\-Thinking\.](https://arxiv.org/html/2608.10444#Sx5.SSx2.SSS0.Px3.p1.1)\.
- \[31\]A\. Liu, A\. Mei, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong,et al\.\(2025\)Deepseek\-v3\. 2: pushing the frontier of open large language models\.arXiv preprint arXiv:2512\.02556\.Cited by:[Introduction](https://arxiv.org/html/2608.10444#Sx1.p1.1),[Implementation Details](https://arxiv.org/html/2608.10444#Sx5.SSx1.p1.1)\.
- \[32\]L\. Long, R\. Wang, R\. Xiao, J\. Zhao, X\. Ding, G\. Chen, and H\. Wang\(2024\)On llms\-driven synthetic data generation, curation, and evaluation: a survey\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 11065–11082\.Cited by:[Dataset Construction](https://arxiv.org/html/2608.10444#Sx3.SSx3.p2.1)\.
- \[33\]S\. Mednick\(1962\)The associative basis of the creative process\.\.Psychological review69\(3\),pp\. 220\.Cited by:[Associative Reasoning](https://arxiv.org/html/2608.10444#Sx2.SSx2.p1.1)\.
- \[34\]T\. Merino, S\. Earle, R\. Sudhakaran, S\. Sudhakaran, and J\. Togelius\(2024\)Making new connections: llms as puzzle generators for the new york times’ connections word game\.InProceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment,Vol\.20,pp\. 87–96\.Cited by:[Boardgame\-Based Benchmarks](https://arxiv.org/html/2608.10444#Sx2.SSx3.p1.1)\.
- \[35\]OpenAI\(2025\)GPT\-5\.2\.Note:[https://openai\.com/index/introducing\-gpt\-5\-2/](https://openai.com/index/introducing-gpt-5-2/)Cited by:[Introduction](https://arxiv.org/html/2608.10444#Sx1.p1.1),[Implementation Details](https://arxiv.org/html/2608.10444#Sx5.SSx1.p1.1)\.
- \[36\]L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. Christiano, J\. Leike, and R\. Lowe\(2022\)Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.35,pp\. 27730–27744\.Cited by:[Introduction](https://arxiv.org/html/2608.10444#Sx1.p1.1)\.
- \[37\]R\. Schaeffer, B\. Miranda, and S\. Koyejo\(2023\)Are emergent abilities of large language models a mirage?\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.36\.Cited by:[Scaling Law](https://arxiv.org/html/2608.10444#Sx6.SSx3.p1.1)\.
- \[38\]C\. Schon, U\. Furbach, and M\. Ragni\(2022\)Modeling associative reasoning processes\.arXiv preprint arXiv:2201\.00716\.Cited by:[Associative Reasoning](https://arxiv.org/html/2608.10444#Sx2.SSx2.p1.1)\.
- \[39\]A\. Srivastavaet al\.\(2023\)Beyond the imitation game: quantifying and extrapolating the capabilities of language models\.Transactions on Machine Learning Research \(TMLR\)\.Cited by:[Introduction](https://arxiv.org/html/2608.10444#Sx1.p2.1),[LLM in Reasoning Depth](https://arxiv.org/html/2608.10444#Sx2.SSx1.p1.1)\.
- \[40\]M\. Stephenson, M\. Sidji, and B\. Ronval\(2025\)Codenames as a benchmark for large language models\.IEEE Transactions on Games\.Cited by:[Boardgame\-Based Benchmarks](https://arxiv.org/html/2608.10444#Sx2.SSx3.p1.1)\.
- \[41\]Y\. Sui, Y\. Chuang, G\. Wang, J\. Zhang, T\. Zhang, J\. Yuan, H\. Liu, A\. Wen, S\. Zhong, H\. Chen, and X\. Hu\(2025\)Stop overthinking: a survey on efficient reasoning for large language models\.arXiv preprint arXiv:2503\.16419\.Cited by:[LLM in Reasoning Depth](https://arxiv.org/html/2608.10444#Sx2.SSx1.p1.1),[Overthinking](https://arxiv.org/html/2608.10444#Sx6.SSx1.p1.1)\.
- \[42\]J\. Treutlein, D\. Choi, J\. Betley, S\. Marks, C\. Anil, R\. Grosse, and O\. Evans\(2024\)Connecting the dots: llms can infer and verbalize latent structure from disparate training data\.Advances in Neural Information Processing Systems37,pp\. 140667–140730\.Cited by:[Introduction](https://arxiv.org/html/2608.10444#Sx1.p2.1)\.
- \[43\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.Advances in neural information processing systems30\.Cited by:[Introduction](https://arxiv.org/html/2608.10444#Sx1.p1.1)\.
- \[44\]M\. Wadhwa, T\. S\. Roy, H\. Lederman, J\. J\. Li, and G\. Durrett\(2026\)CREATE: testing llms for associative creativity\.arXiv preprint arXiv:2603\.09970\.Cited by:[Associative Reasoning](https://arxiv.org/html/2608.10444#Sx2.SSx2.p2.1)\.
- \[45\]L\. Wang, W\. Xu, Y\. Lan, Z\. Hu, Y\. Lan, R\. K\. Lee, and E\. Lim\(2023\)Plan\-and\-solve prompting: improving zero\-shot chain\-of\-thought reasoning by large language models\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 2609–2634\.Cited by:[LLM in Reasoning Depth](https://arxiv.org/html/2608.10444#Sx2.SSx1.p1.1)\.
- \[46\]Y\. Wang, Y\. Kordi, S\. Mishra, A\. Liu, N\. A\. Smith, D\. Khashabi, and H\. Hajishirzi\(2023\)Self\-instruct: aligning language models with self\-generated instructions\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 13484–13508\.Cited by:[Introduction](https://arxiv.org/html/2608.10444#Sx1.p3.1)\.
- \[47\]J\. Wei, M\. Bosma, V\. Y\. Zhao, K\. Guu, A\. W\. Yu, B\. Lester, N\. Du, A\. M\. Dai, and Q\. V\. Le\(2022\)Finetuned language models are zero\-shot learners\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[Introduction](https://arxiv.org/html/2608.10444#Sx1.p1.1)\.
- \[48\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. Zhou\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.35,pp\. 24824–24837\.Cited by:[Introduction](https://arxiv.org/html/2608.10444#Sx1.p1.1),[LLM in Reasoning Depth](https://arxiv.org/html/2608.10444#Sx2.SSx1.p1.1)\.
- \[49\]Y\. Weng, M\. Zhu, F\. Xia, B\. Li, S\. He, S\. Liu, B\. Sun, K\. Liu, and J\. Zhao\(2023\)Large language models are better reasoners with self\-verification\.InFindings of the Association for Computational Linguistics: EMNLP,pp\. 2550–2575\.Cited by:[LLM in Reasoning Depth](https://arxiv.org/html/2608.10444#Sx2.SSx1.p1.1),[Reasoning Trace Evaluation](https://arxiv.org/html/2608.10444#Sx4.SSx3.p1.1)\.
- \[50\]A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[Introduction](https://arxiv.org/html/2608.10444#Sx1.p1.1),[Implementation Details](https://arxiv.org/html/2608.10444#Sx5.SSx1.p1.1)\.
- \[51\]S\. Yao, D\. Yu, J\. Zhao, I\. Shafran, T\. Griffiths, Y\. Cao, and K\. Narasimhan\(2023\)Tree of thoughts: deliberate problem solving with large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.36\.Cited by:[LLM in Reasoning Depth](https://arxiv.org/html/2608.10444#Sx2.SSx1.p1.1)\.
- \[52\]Y\. Zhang, M\. Li, D\. Long, X\. Zhang, H\. Lin, B\. Yang, P\. Xie, A\. Yang, D\. Liu, J\. Lin,et al\.\(2025\)Qwen3 embedding: advancing text embedding and reranking through foundation models\.arXiv preprint arXiv:2506\.05176\.Cited by:[Dataset Construction](https://arxiv.org/html/2608.10444#Sx3.SSx3.p3.1)\.
- \[53\]G\. Zhao, S\. Hu, X\. Jian, J\. Wu, Y\. Wu, L\. Sun, and X\. Zhang\(2025\)Stress testing generalization: how minor modifications undermine large language model performance\.arXiv e\-prints,pp\. arXiv–2502\.Cited by:[Difficulty Settings and Perturbation](https://arxiv.org/html/2608.10444#Sx3.SSx4.p1.1)\.

## Limitations

While this benchmark provides an initial step toward evaluating the multi\-point associative reasoning capabilities of LLMs, there remains some room to further broaden its coverage and ecological validity\. Future work may extend the benchmark with more diverse and interactive settings to better capture associative reasoning behaviors that arise in real\-world environments\. In addition, we hope this benchmark can serve as a foundation for systematically studying non\-linear associative and creative capabilities in LLMs, as well as for developing evaluation protocols and modeling principles that more closely align with practical applications of multi\-point associative reasoning\.

## Ethical Considerations

MPAR\-Bench is built upon lexical materials sourced from cognitive psychology \(the Remote Associates Test\) and cooperative gaming \(Just One\)\. Throughout dataset construction, we adopted a series of safeguards to preempt potential ethical risks\. The benchmark consists solely of word\-level clues and semantic associations, carrying no personally identifiable information and requiring no collection of human subject data\. All clues were either drawn from publicly available game corpora or synthesized through an LLM\-based multi\-agent pipeline, after which embedding\-based filtering and manual verification were applied to exclude content that could introduce demographic, cultural, or other unintended biases, in accordance with established ethical standards\. The bilingual design of our benchmark further ensures equitable treatment of English and Chinese linguistic contexts, with dedicated attention to cultural appropriateness within each subset\. Beyond its primary evaluation purpose, MPAR\-Bench also serves as a diagnostic instrument for probing model reasoning behaviors and uncovering failure modes, thereby supporting the broader goal of identifying and mitigating ethical vulnerabilities in deployed AI systems\.

## Appendix AMulti\-agent Generation Prompts

Prompts are used in English and Chinese in generating, evaluating, and judging models separately\.

### Questioner Prompt

You are playing the board game “Just One”\.Secret answer: \{word\}Requirements:1\. Give exactly 1 English clue word\.2\. The clue must be a common English word or a widely recognized proper noun\.3\. The clue must not be the answer itself, a translation, a synonym, a homophone, a made\-up word, an obvious morphological variant, or contain the answer as a substring\.4\. The clue should be indirect and moderately difficult, but useful when combined with other clues\.5\. You MUST approach the answer from this specific association angle: \{angle\}

### Judger Prompt

You are the judge for the board game “Just One”\.Secret answer: \{word\}Already approved clues \(LOCKED \- do NOT remove or re\-evaluate these\): \{locked\_clues\}New candidate clues to evaluate: \{clue\_list\}\.Your tasks \(apply only to the NEW candidates above\):1\. Remove any new clue that IS the answer itself, a direct synonym, a translation, a homophone, an obvious morphological variant, or contains the full answer as a substring\.2\. Remove EXACT duplicates among the new candidates\.3\. Remove any new clue that refers to the SAME specific concept, entity, or phrase as an already\-approved clue or another new candidate\. Clues that merely belong to the same broad category \(e\.g\., two different fictional characters, two different countries\) are NOT duplicates \- keep both\.4\. Remove genuinely LOW\-QUALITY new clues: completely obscure, made\-up, grammatically wrong, or with no logical connection to the answer\. Do NOT remove a clue just because it requires one step of reasoning\.5\. Do NOT remove a clue simply because it seems “too direct” unless using it would immediately give away the answer with zero reasoning required\.

### Player Prompt

You are the one who guess the mystery word for “Just One”\.Based on the clue words given, guess the mystery word and explain the logical connection between each related clue word and the mystery word\.The clue words are “\{clue\_word1\}”, “\{clue\_word2\}”, “\{clue\_word3\}”Requirements:1\. The mystery word is only one word\.2\. Provide the connection between each clue word and the mystery word\.

ModelsModel SizeAccessVersionProviderGPT\-5\.2undisclosedapigpt\-5\.2\-2025\-12\-11OpenAIGemini\-3\.1proundisclosedapiGemini\-3\.1pro\-previewGoogleGemini\-3flashundisclosedapiGemini\-3flash\-previewSonnet\-4\.5undisclosedapiclaude\-sonnet\-4\-5\-20250929AnthropicQwen3\-maxundisclosedapiqwen3\-max\-2026\-01\-23QwenQwen3\-0\.6B0\.6Bweights\-Qwen3\-1\.7B1\.7Bweights\-Qwen3\-4B4Bweights\-Qwen3\-8B8Bweights\-Qwen3\-14B14Bweights\-Qwen3\-32B32Bweights\-Kimi\-k21Tapikimi\-k2\-thinking\-251104Moonshot AIDeepseek\-v3\.2671BapiDeepseek\-v3\.2DeepSeekSeed\-2\-proundisclosedapidoubao\-seed\-2\-0\-pro\-260215ByteDanceTable 7:Summary of Evaluated ModelsComponentVersionPython3\.11\.14PyTorch2\.6\.0transformers4\.57\.1accelerate1\.10\.1vllm0\.16\.0rc2openai1\.96\.1tiktoken0\.9\.0fastText0\.9\.3gensim4\.3\.0Table 8:Software versions used in our experimentsModelAvg Token\(English\)Avg Token\(Chinese\)GPT\-5\.21355\.11263\.6Gemini\-3\.1pro3551\.11883\.3Sonnet\-4\.51468\.91729\.2Qwen3\-max9983\.3777\.9Kimi\-k24659\.53290\.7Deepseek\-v3\.23091\.02194\.4Seed\-2\-pro1865\.21563\.8Table 9:Token Usage in Thinking Mode LLMs \(Standard\)ModelAvg Token\(English\)Avg Token\(Chinese\)GPT\-5\.21741\.21338\.9Gemini\-3\.1pro4940\.72409\.8Sonnet\-4\.51577\.41838\.7Qwen3\-max11743\.44782\.4Kimi\-k24790\.14120\.7Deepseek\-v3\.23255\.02307\.8Seed\-2\-pro2140\.71858\.6Table 10:Token Usage in Thinking Mode LLMs \(Enhanced\)

## Appendix BEvaluation Prompt

You are a rigorous AI logic auditor\. Your task is to evaluate each reasoning step produced by an Agent in a word\-guessing game\.Break the reasoning into independent steps \(atoms\) and judge each atom on two dimensions: “Factual Accuracy” and “Logical Soundness”\.For each step:1\. Fact\_Check: Are the objective claims in this step \(numbers, ingredients, mechanisms, historical origins, etc\.\) factually correct? \(Pass / Fail\)2\. Logic\_Check: Is the reasoning chain from the clue to the predicted answer natural and sound? Are there signs of over\-generalization, edge\-case pandering, or multi\-layer reinterpretation? \(Pass / Fail\)Judging Principles \(you MUST follow these\):\- Scrutinize specific claims \(numbers, ingredients, mechanisms, definitions\) — any factual error → Fact Fail\- If a step cherry\-picks a marginal meaning of the clue to fit the answer, ignoring a more obvious association → Logic Fail\- If the clue clearly points to a more specific/precise word, but the step generalizes it to a broad concept → Logic Fail\- If the step requires more than two layers of inference or reinterpretation to connect the clue to the answer → Logic Fail\- Only when the reasoning feels natural, direct, and requires no mental gymnastics for an ordinary person should it be judged Logic Pass

## Appendix CAsset Distribution and Compliance

To ensure reproducibility, both the English and Chinese subsets of MPAR\-Bench, along with our evaluation scripts, will be publicly released under the MIT License upon publication\. Our evaluation items are generated via frontier LLM APIs \(e\.g\., GPT, Gemini, Qwen\), and we have verified that our pipeline adheres to the respective terms of service of these model providers, restricting the usage of our dataset strictly to non\-commercial academic benchmarking\.

ModelEnglish Acc\(%\)Chinese Acc\(%\)MaskShuf\.Dis\.MultistepMaskShuf\.Dis\.MultistepGPT\-5\.256\.880\.860\.868\.048\.868\.067\.251\.2Gemini\-3\.1pro67\.286\.376\.777\.655\.273\.666\.460\.8Sonnet\-4\.553\.673\.656\.869\.649\.670\.467\.256\.8Qwen3\-max56\.068\.843\.259\.248\.063\.258\.449\.6Kimi\-k249\.669\.649\.661\.635\.257\.648\.841\.6Deepseek\-v3\.240\.864\.844\.857\.644\.860\.057\.647\.2Seed\-2\-pro47\.267\.243\.268\.851\.268\.060\.852\.8Table 11:Detailed Results of Enhanced Benchmark on Thinking ModelsModelEnglish Acc\(%\)Chinese Acc\(%\)MaskShuf\.Dis\.MultistepMaskShuf\.Dis\.MultistepGPT\-5\.232\.054\.539\.251\.242\.467\.264\.843\.2Gemini\-3flash43\.268\.851\.260\.849\.676\.051\.268\.8Sonnet\-4\.545\.672\.851\.253\.653\.671\.271\.252\.0Qwen3\-max35\.255\.244\.044\.844\.065\.667\.247\.2Deepseek\-v3\.232\.056\.839\.242\.447\.261\.662\.444\.8Seed\-2\-pro31\.256\.033\.652\.847\.264\.858\.453\.6Table 12:Detailed Results of Enhanced Benchmark on Non\-thinking Models
## Appendix DExperiment Setup

For all evaluated models we keep the official default sampling parameters specified in each provider’s release\. For reasoning models, we additionally set reasoning\_effort=high where the API supports it\. We do not perform any per\-model hyper\-parameter tuning\. Detailed model configurations for all experiments are provided in Table[7](https://arxiv.org/html/2608.10444#A1.T7)\. And detailed software versions are provided in Table[8](https://arxiv.org/html/2608.10444#A1.T8)\.

ModelEnglishChineseFact\(%\)Logic\(%\)Fact\(%\)Logic\(%\)✓\\checkmark×\\times✓\\checkmark×\\times✓\\checkmark×\\times✓\\checkmark×\\timesGPT\-5\.21\.756\.616\.3924\.640\.872\.255\.5911\.69Gemini\-3\.1pro2\.5811\.526\.0820\.612\.162\.455\.717\.77Sonnet\-4\.54\.3017\.148\.7131\.623\.445\.288\.5514\.72Qwen3\-max4\.3718\.968\.0338\.212\.224\.949\.2215\.30Kimi\-k23\.9115\.779\.6640\.564\.327\.6111\.0124\.13Deepseek\-v3\.26\.0213\.6012\.8943\.603\.597\.7310\.7824\.85Seed\-2\-pro5\.2720\.147\.9034\.692\.485\.087\.0613\.33Table 13:Detailed Fact Fail and Logic Fail of Reasoning Trace on Standard MPAR\-Bench on Thinking ModelsModelEnglishChineseFact\(%\)Logic\(%\)Fact\(%\)Logic\(%\)✓\\checkmark×\\times✓\\checkmark×\\times✓\\checkmark×\\times✓\\checkmark×\\timesGPT\-5\.25\.3024\.069\.9347\.031\.753\.876\.3412\.88Gemini\-3flash4\.7435\.337\.8947\.734\.246\.798\.7815\.52Sonnet\-4\.55\.409\.9427\.3045\.413\.847\.189\.2414\.10Qwen3\-max4\.5531\.308\.5245\.652\.614\.498\.3215\.28Deepseek\-v3\.28\.9529\.7115\.1055\.972\.188\.838\.8421\.83Seed\-2\-pro10\.0337\.5115\.6552\.143\.438\.278\.4720\.34Table 14:Detailed Fact Fail and Logic Fail of Reasoning Trace on Standard MPAR\-Bench on Non\-thinking ModelsModelEnglishChineseFact\(%\)Logic\(%\)Fact\(%\)Logic\(%\)✓\\checkmark×\\times✓\\checkmark×\\times✓\\checkmark×\\times✓\\checkmark×\\timesGPT\-5\.22\.526\.2210\.3826\.852\.272\.309\.9413\.31Gemini\-3\.1pro4\.0710\.719\.8523\.513\.113\.5910\.9712\.56Sonnet\-4\.56\.7916\.4912\.9339\.216\.118\.3515\.7322\.32Qwen3\-max4\.7415\.7811\.8635\.903\.154\.3011\.8317\.13Kimi\-k25\.9614\.0314\.0240\.255\.6110\.1518\.1825\.07Deepseek\-v3\.27\.0914\.3717\.1145\.325\.979\.8017\.1131\.04Seed\-2\-pro7\.0721\.6012\.2137\.825\.837\.0014\.8718\.23Table 15:Detailed Fact Fail and Logic Fail of Reasoning Trace on Enhanced MPAR\-Bench on Thinking ModelsModelEnglishChineseFact\(%\)Logic\(%\)Fact\(%\)Logic\(%\)✓\\checkmark×\\times✓\\checkmark×\\times✓\\checkmark×\\times✓\\checkmark×\\timesGPT\-5\.27\.1624\.1913\.9747\.393\.385\.7310\.1119\.29Gemini\-3flash7\.2233\.0013\.6248\.066\.3811\.2213\.6024\.33Sonnet\-4\.58\.5226\.7014\.7946\.045\.9710\.4415\.7325\.45Qwen3\-max6\.2525\.5613\.2148\.913\.329\.2914\.4725\.21Deepseek\-v3\.211\.7929\.6320\.0956\.304\.5811\.5014\.4930\.75Seed\-2\-pro12\.9733\.9518\.0350\.885\.3611\.3614\.0824\.81Table 16:Detailed Fact Fail and Logic Fail of Reasoning Trace on Enhanced MPAR\-Bench on Non\-thinking Models
## Appendix EDetailed Results

### Token Usage

Tables[9](https://arxiv.org/html/2608.10444#A1.T9)and[10](https://arxiv.org/html/2608.10444#A1.T10)report the average token consumption of models operating in thinking mode, encompassing prompt tokens, Chain\-of\-Thought reasoning tokens, and output tokens\.

Token consumption increases consistently from the Standard to the Enhanced setting for nearly all models, reflecting the greater reasoning demand imposed by the four perturbation conditions\. The magnitude of this increase, however, varies markedly across models, pointing to fundamentally different strategies for allocating reasoning computation\.

The most pronounced outlier is Qwen3\-max, which exhibits disproportionately high token consumption in English across both settings, averaging roughly 10,000 tokens in Standard MPAR\-Bench and 11,700 in Enhanced MPAR\-Bench\. Qualitative inspection of its generated reasoning trace reveals a consistent tendency toward repetitive self\-verification: the model frequently revisits intermediate conclusions, generates multiple redundant candidate answers, and enters circular deliberation loops before converging on a final prediction\. This overthinking behavior substantially inflates token usage\.

### Enhanced MPAR\-Bench Result Analysis

Tables[11](https://arxiv.org/html/2608.10444#A3.T11)and[12](https://arxiv.org/html/2608.10444#A3.T12)present the detailed accuracy of model across four perturbation variants in the Enhanced MPAR\-Bench setting: Clue Masking, Order Shuffling, Distractor Injection, and Multi\-step Inferring\.

The four perturbation types exhibit different impacts on model performance\. Order Shuffling consistently yields the highest accuracy across all models, some of which are even higher than Standard setting\. In contrast, Clue Masking causes the most severe degradation in English, with an average drop of 20\.0% from Order Shuffling\. Distractor Injection also substantially reduces performance, particularly for Qwen3\-max and Seed\-2\-pro, both dropping by over 28% from their respective Standard accuracies\. This indicates that spurious semantic correlations introduced by irrelevant clue words can effectively derail the model’s reasoning trajectory\. Multi\-step Inferring occupies an intermediate difficulty level, suggesting that extending the associative chain length moderately taxes the model’s long\-range semantic mapping capacity but does not fundamentally break the reasoning process\.

### Reasoning Trace Error Analysis

Tables[13](https://arxiv.org/html/2608.10444#A4.T13)through[16](https://arxiv.org/html/2608.10444#A4.T16)present the reasoning trace evaluation results, decomposing reasoning failures into “fact fail” and “logic fail”\.

Across all models, logical error rates substantially exceed factual error rates\. In the English thinking standard setting, Deepseek\-v3\.2 exhibits a logical error rate of 43\.60%, Kimi\-k2 40\.56%, and Qwen3\-max 38\.21%\. Even the best\-performing model, Gemini\-3\.1pro, shows a logical error rate of 20\.61%\. By contrast, factual error rates in the same setting are considerably lower: Deepseek\-v3\.2 at 13\.60%, Kimi\-k2 at 15\.77%, and Gemini\-3\.1pro at 11\.52%\. This indicates that the primary failure in multi\-point associative reasoning is not incorrect factual knowledge but rather invalid inferential jumps in constructing the reasoning chain from clues to the answer\. Moreover, logical error rates in Chinese reasoning are consistently lower than in English\.

Furthermore, in the non\-thinking mode, factual error rates increase substantially, which suggests that without an explicit reasoning process, models are more susceptible to factual hallucination\.

### Structured Reasoning Skill Experiment

We designed a structured three\-step reasoning skill to test whether explicit multi\-step prompting can enhance reasoning breadth beyond what thinking mode alone achieves\. The skill instructs the model to: \(1\) examine all clues comprehensively and identify their underlying associations; \(2\) prioritize specific concepts over abstract hypernyms; and \(3\) perform reverse verification by reasoning backward from the predicted answer to the provided clues\. We evaluate this skill on Seed\-2\-pro \(thinking mode, standard setting\)\.

On the English subset, accuracy rises marginally from 71\.4% to 72\.4% \(\+1\.0pp\), ANLS from 0\.724 to 0\.734, and embedding similarity from 0\.802 to 0\.807\. On the Chinese subset, gains are somewhat larger: accuracy from 64\.6% to 67\.8% \(\+3\.2pp\), ANLS from 0\.739 to 0\.765, and embedding similarity from 0\.820 to 0\.838\. The modest magnitude of improvement—particularly on English—suggests that thinking\-mode models may already perform these associative integration steps implicitly during extended reasoning, and that prompt\-level interventions alone offer limited leverage on the core challenge of multi\-source evidence integration\.