EDMA：熵驱动的多模态问答

# EDMA: Entropy-Driven Multimodal Answering
Source: [https://arxiv.org/html/2610.27120](https://arxiv.org/html/2610.27120)
Gertjan BurghoutsAffiliation:TNOEmail:[gertjan\.burghouts@tno\.nl](mailto:)Fabio MassacciAffiliation:Vrije Universiteit AmsterdamAffiliation:University of TrentoEmail:[fabio\.massacci@ieee\.org](mailto:)Mengyuan ZhangAffiliation:Vrije Universiteit AmsterdamEmail:[m\.zhang@vu\.nl](mailto:)

###### Abstract

Multimodal question answering \(MMQA\) is based on integrating heterogeneous data sources, selectively leveraging relevant modalities, and ignoring distractors\. Existing approaches based on Multimodal Large Language Models \(MLLMs\) improve performance by splitting the task into intermediate steps, but do not quantify how each modality contributes to reaching the final answer\. We argue that quantifying the effect that each modality has on the answering process improves process traceability and final performance\. Thus, we propose Entropy\-Driven Multimodal Answering \(EDMA\), which uses logical entropy to formalise multimodal QA as an iterative process of entropy reduction and to quantify the contribution of each modality in distinguishing correct from incorrect candidate answers\. This enables the construction of*quantifiable answering trajectories*, where each step is associated with a measurable entropy reduction\. EDMA introduces \(i\) modality\-conditioned partitioning to estimate unimodal entropy reduction, \(ii\) entropy\-driven multimodal fusion to capture complementary information across modalities, and \(iii\) entropy\-based selection of the answering trajectory that minimises the most logical entropy\. Evaluated across standard MMQA benchmarks, EDMA outperforms competitive prompting baselines, with average gains of 12\.8 F1 and 4\.6 LLM\-J points over Planning Prompting and 18\.6 F1 and 8\.3 LLM\-J points over Chain\-of\-Thought\. Moreover, EDMA outperforms self\-consistency baselines and is competitive with the multi\-agent system baseline\. Moreover, we show that higher entropy correlates with more false positives and, through controlled trajectory selection, demonstrate that logical entropy serves as an effective, monotonic selector for correct reasoning paths, validating entropy level as a reliable proxy for performance\.111Code will be made publicly available with the final camera\-ready version\.

![Refer to caption](https://arxiv.org/html/2610.27120v1/us_and_them_new.png)Figure 1:Comparison between CoT and EDMA\. While they both structure QA in intermediate steps, EDMA generates quantifiable answering trajectories, allowing for improved performance and quantified modality contribution\.## 1Introduction

Multimodal question answering \(MMQA\) requires the integration of heterogeneous data sources such as text, images, and tables[Talmor et al\. \(2021\)](https://arxiv.org/html/2610.27120#bib.bib2);[Luo et al\. \(2023\)](https://arxiv.org/html/2610.27120#bib.bib3);[Foroutan et al\. \(2025\)](https://arxiv.org/html/2610.27120#bib.bib4)\. For example, answering the question*“What is the state in the Contour Airlines city destinations, that has a tall metal arch on a river?”*requires identifying visual evidence of a metal arch in images while simultaneously retrieving structured information from tables to determine the destinations of the airline, and combining the evidence\.

Recent Multimodal Large Language Models \(MLLMs\) have shown strong performance on such tasks by leveraging unified representations across modalities[Gao et al\. \(2023\)](https://arxiv.org/html/2610.27120#bib.bib1);[Zhang et al\. \(2025a\)](https://arxiv.org/html/2610.27120#bib.bib8), and are currently being employed to solve domain\-specific problems[Li et al\. \(2025a\)](https://arxiv.org/html/2610.27120#bib.bib40);[Yang et al\. \(2025\)](https://arxiv.org/html/2610.27120#bib.bib41);[Xia et al\. \(2024\)](https://arxiv.org/html/2610.27120#bib.bib12);[Wang et al\. \(2024a\)](https://arxiv.org/html/2610.27120#bib.bib38);[Zhao et al\. \(2025\)](https://arxiv.org/html/2610.27120#bib.bib10)\. Several techniques have been proposed to improve multimodal inference, most notably Chain\-of\-Thought \(CoT\) prompting[Wei et al\. \(2022\)](https://arxiv.org/html/2610.27120#bib.bib29);[Zhang et al\. \(2023\)](https://arxiv.org/html/2610.27120#bib.bib5);[Zheng et al\. \(2023\)](https://arxiv.org/html/2610.27120#bib.bib34), which decomposes complex tasks into intermediate steps\. Building on this paradigm, extensions such as compositional prompting[Mitra et al\. \(2024\)](https://arxiv.org/html/2610.27120#bib.bib6), knowledge\-augmented reasoning[Mondal et al\. \(2024\)](https://arxiv.org/html/2610.27120#bib.bib32), PEAR\-CoT[Li et al\. \(2025b\)](https://arxiv.org/html/2610.27120#bib.bib9), frameworks such as Chain\-of\-Action \(CoA\)[Pan et al\. \(2024\)](https://arxiv.org/html/2610.27120#bib.bib33)and Visualization\-of\-Thought \(VoT\)[Wu et al\. \(2024\)](https://arxiv.org/html/2610.27120#bib.bib39)have been proposed to address limitations in compositionality, grounding, and spatial awareness\. To further improve the inference process, recent work has incorporated uncertainty quantification, enabling models to adaptively seek information or refine intermediate steps[Hu et al\. \(2024\)](https://arxiv.org/html/2610.27120#bib.bib45);[Yin et al\. \(2024\)](https://arxiv.org/html/2610.27120#bib.bib46);[Liang et al\. \(2024\)](https://arxiv.org/html/2610.27120#bib.bib47)\. However, while these approaches improve robustness by structuring the inference process in intermediate steps, augmenting contextual information, and guaranteeing higher robustness, they do not provide explicit stepwise quantification of how individual modalities and their fusion contribute to reducing uncertainty over the space of candidate answers\. As a result, it remains unclear how different modalities contribute to the inference process, limiting interpretability and controllability in multimodal settings\.

To address these limitations, we propose Entropy\-Driven Multimodal Answering \(EDMA\), a methodology that models QA as a process of iterative entropy reduction over candidate answers\. As Figure[1](https://arxiv.org/html/2610.27120#S0.F1)shows, EDMA quantifies the impact of incorporating a specific modality and fusing modalities by measuring how this refines the space of possible answers\. Compared to CoT, which splits the answering process in steps without quantifying the contribution of each modality, EDMA builds*quantifiable answering trajectories*, where each step and each modality is associated with a measurable contribution in the reduction of entropy\. At the core of EDMA lies logical entropy[Ellerman \(2021\)](https://arxiv.org/html/2610.27120#bib.bib7), which measures distinctions between possible outcomes\. This formulation is particularly well\-suited for multimodal QA, where the objective is to progressively separate correct from incorrect candidate answers based on evidence from different modalities\. By framing answering as partition refinement, EDMA treats each piece of evidence as a condition that splits the current answer space into two subsets of answers: those that satisfy the evidence and those that do not\. This systematic partitioning provides a natural and interpretable mechanism for tracking exactly how information from each modality contributes to narrowing down the final prediction\.

We evaluate EDMA on challenging multimodal QA benchmarks[Talmor et al\. \(2021\)](https://arxiv.org/html/2610.27120#bib.bib2);[Hannan et al\. \(2020\)](https://arxiv.org/html/2610.27120#bib.bib24)and show that it outperforms best\-practice prompting baselines and, on average, state\-of\-the\-art multi\-agent systems, while providing traceable uncertainty estimates that guide modality selection and the answering process\. Our contributions are as follows:

- •EDMA, a methodology that formalizes multimodal QA as modality\-conditioned entropy minimization over the space of alternatives\.
- •A modality\-conditioned entropy reduction objective that enables explicit measurement of each modality’s contribution\.
- •We establish the relation between entropy minimization and lower false positives \(FPs\) levels, thus validating entropy minimization as a proxy for performance\.

## 2Entropy\-Driven Multimodal Answering \(EDMA\)

### 2\.1Problem formulation

The problem setting involves one questionQQ, the set of modalities available to answer the question denoted byMMand, for eachm∈Mm\\in M, the elements associated with that modality denoted byΩm\\Omega\_\{m\}, and one Answerer denoted byL​L​MLLM\. To answer each question, theL​L​MLLMmust find the modalities of interest, find the elements within those modalities that contain the necessary information to answer the question, and then combine the information to provide a cross\-modal answer, denoted bya∗a^\{\*\}\.

### 2\.2Overview of EDMA

Figure[2](https://arxiv.org/html/2610.27120#S2.F2)shows that EDMA is composed of four stages: \(a\) prediction of the modalities to be used and extraction of answer requirements from the question, \(b\) generation of unimodal partitions and calculation of unimodal logical entropy, \(c\) generation of the multimodal partitions with paired multimodal logical entropy, and \(d\) answer generation by selecting the answering trajectory that minimized the most the logical entropy\. The process allows to reduce at each step the number of elements which represent the possible alternatives by checking which ones fit the requirements extracted from the question, returning an answer based on the answering trajectory that minimised the most the logical entropy\. In Algorithm[1](https://arxiv.org/html/2610.27120#alg1)in Appendix[B](https://arxiv.org/html/2610.27120#A2), we describe EDMA’s answering process\.

![Refer to caption](https://arxiv.org/html/2610.27120v1/methodology_new.png)Figure 2:EDMA is composed of four stages: \(a\) modality prediction and extraction of answer requirements, \(b\) generation of unimodal partitions and calculation of unimodal logical entropy, \(c\) generation of multimodal partitions and calculation of multimodal logical entropy, \(d\) selection of the answering trajectory which minimises the logical entropy and generation of the answer\.
### 2\.3Extraction of Answer Requirements

Supposing we want to answer the question*“What is the state in the Contour Airlines city destinations, that has a tall metal arch on a river?”*, EDMA begins by extracting the set of answer requirements, representing the characteristics that the final answera∗a^\{\*\}must satisfy\. We denote the answer requirements asRR\. As shown in Figure[2](https://arxiv.org/html/2610.27120#S2.F2), theL​L​MLLMgeneratesRRby identifying the key elements of the question\. In this specific case, the final answer has to: \(i\) be a*State*, \(ii\) be a*destination of Contour Airlines*, and \(iii\) be characterized by a*Metal Arch*\. We denote the extraction ofRRasR=L​L​M​\(P​r​o​m​p​tR,Q\)R=LLM\(Prompt\_\{R\},Q\), where theL​L​MLLMreceives in input the user instructionsP​r​o​m​p​tRPrompt\_\{R\}and the questionQQ\. In this case,R=\{S,C,A\}R=\\\{S,C,A\\\}, where we useSSto point to State,CCfor Contour Destination, andAAfor Metal Arch\.

### 2\.4Prediction of Starting Modalities

EDMA predicts the subset of modalitiesMs⊆MM\_\{s\}\\subseteq Mrepresenting the modalities with which to start the analysis\. This phase does not exclude the possibility of incorporating additional modalities later in the answering process\. TheL​L​MLLMreceives the input, the user instructionsP​r​o​m​p​tMsPrompt\_\{M\_\{s\}\}, the questionQQ, and for each modalitymm, the corresponding set of elementsΩm\\Omega\_\{m\}\. It is then tasked with predicting the combination of modalities that is most likely to contain the elements needed to reach the correct answer\. This operation can be summarized asMs=L​L​M​\(P​r​o​m​p​tMs,Q,\{Ωm,m∈M\}\)M\_\{s\}=LLM\(Prompt\_\{M\_\{s\}\},Q,\\\{\\Omega\_\{m\},m\\in M\\\}\)\. The modalitiesMsM\_\{s\}are then added toMaM\_\{a\}, which is the list that keeps track of the modalities used for the analysis\. In Figure[2](https://arxiv.org/html/2610.27120#S2.F2), the predicted modalities are*Table \(T\)*and*Image \(I\)*\.

### 2\.5Entropy\-Driven Answer Search

To identify the correct answer in a multimodal setting, EDMA models inference as an entropy\-driven search over the space of candidate answers\. It quantifies how individual modalities and their interactions reduce uncertainty over the answer space\. To this end, EDMA first induces partitions over the answer space using unimodal evidence and then refines these partitions through sequential cross\-modal conditioning\. This process enables systematic measurement of entropy reduction from both unimodal and multimodal evidence\.

##### Quantifying Uncertainty of Answer Partitions

We adopt logical entropy[Ellerman \(2021\)](https://arxiv.org/html/2610.27120#bib.bib7), which measures uncertainty in terms of distinctions induced by a partition of a candidate set\. LetSSdenote a finite set of candidate elements and letPS=\{B1,…,Bn\}P\_\{S\}=\\\{B\_\{1\},\\dots,B\_\{n\}\\\}be a partition ofSSinto disjoint non\-empty blocks\. To ensure compatibility across open\-source and closed\-source models without requiring calibrated probability estimates from the underlying model, we assume a uniform distribution overSS, such that each block has probability masspi=\|Bi\|\|S\|p\_\{i\}=\\frac\{\|B\_\{i\}\|\}\{\|S\|\}\. Logical entropy is defined asH⁡\(PS\)=1−∑i=1npi2H\(P\_\{S\}\)=1\-\\sum\_\{i=1\}^\{n\}p\_\{i\}^\{2\}, and can be interpreted as the probability that two independently sampled elements fromSSbelong to different blocks\.

For example, consider\|S\|=10\|S\|=10and a binary partitionPS=\{Sr,S¬r\}P\_\{S\}=\\\{S\_\{r\},S\_\{\\neg r\}\\\}induced by a requirementrr\. The setSrS\_\{r\}contains elements whose information overlaps withrr, whileS¬rS\_\{\\neg r\}contains elements that do not\. If\|Sr\|=5\|S\_\{r\}\|=5and\|S¬r\|=5\|S\_\{\\neg r\}\|=5, thenH⁡\(PS\)=0\.50H\(P\_\{S\}\)=0\.50, corresponding to maximal entropy for a binary partition under a uniform prior\. If instead\|Sr\|=1\|S\_\{r\}\|=1and\|S¬r\|=9\|S\_\{\\neg r\}\|=9, thenH⁡\(PS\)=1−\(0\.12\+0\.92\)=0\.18H\(P\_\{S\}\)=1\-\(0\.1^\{2\}\+0\.9^\{2\}\)=0\.18, reflecting reduced uncertainty due to the increased concentration of mass\.

In our multimodal QA setting, the answering process is modeled as a sequence of partition refinements overSS, where information extracted from the elements of each modality introduces distinctions that separate relevant from irrelevant candidates\. We restrict attention to valid modality\-induced refinements where the predicate\-satisfying subset is non\-empty,Sr≠∅S\_\{r\}\\neq\\emptyset, ensuring that entropy reduction is grounded in non\-vacuous evidence\.

Logical entropy thus quantifies the effectiveness of each modality in refining the answer space, allowing to prioritize modalities that maximize entropy reduction, guiding the system toward increasingly discriminative evidence\.

##### Generation of Unimodal Partitions

EDMA analyses the elementsΩm\\Omega\_\{m\}of each modalitym∈Mam\\in M\_\{a\}\. For each elementω∈Ωm\\omega\\in\\Omega\_\{m\}, theL​L​MLLMevaluates, for each answer requirementr∈Rr\\in R, whetherω\\omegacontains information overlapping withrr\. This operation can be formally expressed asL​L​M​\(P​r​o​m​p​tC​h​e​c​k,ω,r\)LLM\(Prompt\_\{Check\},\\omega,r\)\. Based on this evaluation, EDMA constructs, for each setΩm\\Omega\_\{m\}and requirementrr, a partitionPmr=\{Ωmr,Ωm¬r\}P\_\{m\}^\{r\}=\\\{\\Omega\_\{m\}^\{r\},\\Omega\_\{m\}^\{\\neg r\}\\\}, whereΩmr\\Omega\_\{m\}^\{r\}contains elements whose information overlaps withrr, andΩm¬r\\Omega\_\{m\}^\{\\neg r\}contains elements that do not\. Under the assumption of uniform distribution over elements inΩm\\Omega\_\{m\}, the logical entropy ofPmrP\_\{m\}^\{r\}is:

H⁡\(Pmr\)=1−\(\(\|Ωmr\|\|Ωm\|\)2\+\(\|Ωm¬r\|\|Ωm\|\)2\)H\(P\_\{m\}^\{r\}\)=1\-\\left\(\\left\(\\frac\{\|\\Omega\_\{m\}^\{r\}\|\}\{\|\\Omega\_\{m\}\|\}\\right\)^\{2\}\+\\left\(\\frac\{\|\\Omega\_\{m\}^\{\\neg r\}\|\}\{\|\\Omega\_\{m\}\|\}\\right\)^\{2\}\\right\)\(1\)
This formulation expresses logical entropy as a function of partition cardinalities, enabling estimation without relying on confidence estimates from theL​L​MLLM\. Logical entropy measures the distinction between elements that satisfy a given condition and those that do not\. It attains its global minimum of00when eitherΩmr=∅\\Omega\_\{m\}^\{r\}=\\emptysetorΩm¬r=∅\\Omega\_\{m\}^\{\\neg r\}=\\emptyset\. Otherwise, it is strictly positive and is minimized when one subset is highly concentrated \(i\.e\., when a single candidate is clearly distinguished\)\.

In Figure[2](https://arxiv.org/html/2610.27120#S2.F2), unimodal partitions illustrate how different modalities contribute to distinguishing relevant from irrelevant candidates\. The image\-based partitionPIAP\_\{I\}^\{\\text\{$A$\}\}yields a relatively low entropy \(0\.380\.38\), indicating that the visual modality already provides a strong separation by isolating the candidate satisfying the requirementAA\. In contrast, the table\-based partitions exhibit higher entropy values \(e\.g\.,0\.490\.49for theSSrequirement\), reflecting a weaker discriminative effect when considered in isolation\. Since lower entropy corresponds to sharper distinctions among candidates, this comparison indicates that the image modality is more informative at this stage for narrowing the answer space\. EDMA leverages this signal to prioritize modalities that induce stronger reductions in uncertainty, guiding the answering process toward partitions that more effectively constrain the set of plausible answers\.

##### Generation of Multimodal Partitions

Once unimodal partitionsPIAP\_\{I\}^\{\\text\{$A$\}\},PTSP\_\{T\}^\{\\text\{$S$\}\},PTCP\_\{T\}^\{\\text\{$C$\}\}are constructed, EDMA proceeds to generate multimodal partitions\. This step leverages information extracted from multiple modalities to refine candidate elements, progressively reducing logical entropy, with the aim of isolating the correct candidate\(s\), filtering out the alternatives that are inconsistent with cross\-modal evidence, which is information extracted from one modality used to reduce the answer space in another modality\. In a multimodal setting, the final answer typically resides in a specific modality, referred to as the*target modality*, denoted bymtm\_\{t\}\. However, the target modality is not known in advance\. To identify it, EDMA systematically treats each modality in turn as a candidate target modalitymtm\_\{t\}, while the remaining modalitiesMc=Ma∖\{mt\}M\_\{c\}=M\_\{a\}\\setminus\\\{m\_\{t\}\\\}serve as conditioning modalities\.

For each candidatemtm\_\{t\}, EDMA applies information from all conditioning modalities sequentially\. For each conditioning modalitym∈Mcm\\in M\_\{c\}and the subsetΩmr′\\Omega\_\{m\}^\{r^\{\\prime\}\}containing elements that satisfy a requirementr′∈Rr^\{\\prime\}\\in R, EDMA extracts the cross\-modal evidence set:

Em=\{e\|e=LLM\(PromptE​x​t​r​a​c​t,Q,ω\),ω∈Ωmr′\}E\_\{m\}=\\Big\\\{\\,e\\;\\big\|\\;e=LLM\(Prompt\_\{Extract\},Q,\\omega\),\\ \\omega\\in\\Omega\_\{m\}^\{r^\{\\prime\}\}\\,\\Big\\\}\(2\)
Each piece of evidence represents information from the conditioning modalitymmand has the same function as the answer requirements\. EDMA applies the elements of this set sequentially to iteratively refine the candidates in the target modalitymtm\_\{t\}that satisfy requirementsr∈Rr\\in R, wherer≠r′r\\neq r^\{\\prime\}\. This allows evidence from one modality \(e\.g\., visual\) that satisfies one requirement to be used to filter candidates in another modality \(e\.g\., tabular\) that satisfy a distinct requirement\.

Specifically, for a given requirementr∈Rr\\in R, the target subsetΩmtr\\Omega\_\{m\_\{t\}\}^\{r\}is updated sequentially for each conditioning modality\. At each step, EDMA generates a partitionPmt\|mr,Em=\{Ωmtr,Em,Ωmtr,¬Em\}P\_\{m\_\{t\}\|m\}^\{r,E\_\{m\}\}=\\\{\\Omega\_\{m\_\{t\}\}^\{r,E\_\{m\}\},\\Omega\_\{m\_\{t\}\}^\{r,\\neg E\_\{m\}\}\\\}of the elements of modalitymtm\_\{t\}consistent with requirementrr, whereΩmtr,Em\\Omega\_\{m\_\{t\}\}^\{r,E\_\{m\}\}contains the elements that are also consistent with the evidenceEmE\_\{m\}from the current conditioning modality, andΩmtr,¬Em\\Omega\_\{m\_\{t\}\}^\{r,\\neg E\_\{m\}\}contains those that are not\. Only the subsetΩmtr,Em\\Omega\_\{m\_\{t\}\}^\{r,E\_\{m\}\}is carried forward to the next iteration, whileΩmtr,¬Em\\Omega\_\{m\_\{t\}\}^\{r,\\neg E\_\{m\}\}is excluded from further conditioning, as it cannot lead to the correct answer\. This sequential conditioning progressively narrows the answer space ofmtm\_\{t\}, increasing the distinction among the remaining candidates, thus decreasing logical entropy\.

By evaluating the multimodal logical entropy after each step, EDMA identifies whichmtm\_\{t\}ultimately yields the partition with the lowest entropy, where distinctions among candidates are maximized, and uncertainty is minimized\. Conditional or multimodal logical entropy is computed analogously to the unimodal case:

H⁡\(Pmt\|mr,Em\)=1−\(\(\|Ωmtr,Em\|\|Ωmt\|\)2\+\(\|Ωmtr,¬Em\|\|Ωmt\|\)2\)H\\Big\(P\_\{m\_\{t\}\|m\}^\{r,E\_\{m\}\}\\Big\)=1\-\\left\(\\left\(\\frac\{\|\\Omega\_\{m\_\{t\}\}^\{r,E\_\{m\}\}\|\}\{\|\\Omega\_\{m\_\{t\}\}\|\}\\right\)^\{2\}\+\\left\(\\frac\{\|\\Omega\_\{m\_\{t\}\}^\{r,\\neg E\_\{m\}\}\|\}\{\|\\Omega\_\{m\_\{t\}\}\|\}\\right\)^\{2\}\\right\)\(3\)
In the example in Figure[2](https://arxiv.org/html/2610.27120#S2.F2), cross\-modal conditioning yields asymmetric effects depending on the direction of information flow\. When table\-derived constraints \(ETE\_\{\\text\{$T$\}\}\) are applied to the image modality \(PIAP\_\{I\}^\{\\text\{$A$\}\}\), entropy remains unchanged \(0\.380\.38\), indicating limited additional discriminative power beyond visual cues\. In contrast, conditioning the table modality \(PTCP\_\{T\}^\{\\text\{$C$\}\}\) using visual evidence \(EIE\_\{\\text\{$I$\}\}\) significantly reduces entropy \(0\.44→0\.200\.44\\rightarrow 0\.20\), showing that the image provides high\-value filtering information that effectively disambiguates table entries, and showing that EDMA does not treat modalities symmetrically, but instead identifies directional entropy reduction, where certain modalities act as stronger selectors of relevant candidates if the derived information is applied in the right order\.

### 2\.6Answer Generation

##### Trajectory Selection via Entropy Minimization

The systematic generation of unimodal and multimodal partitions effectively maps a set of*quantifiable answering trajectories*\(𝒯\)\(\\mathcal\{T\}\), representing paths in the search process\. Each trajectory provides a quantifiable history of how information is fused and how logical entropy changed after the modality fusion\. This allows for a modality\-level attribution of the logical entropy variation\. A trajectory may remain within a unimodal setting \(m→mm\\to m\) or bridge modalities to form a multimodal trajectory \(m→mtm\\to m\_\{t\}\), where the first modality conditions the second\(mt\|m\)\(m\_\{t\}\|m\)\. We denoteΔ​H=H⁡\(Pmtr\)−H⁡\(Pmt\|mr,Em\)H⁡\(Pmtr\)\\Delta H=\\frac\{H\(P\_\{m\_\{t\}\}^\{r\}\)\-H\(P\_\{m\_\{t\}\|m\}^\{r,E\_\{m\}\}\)\}\{H\(P\_\{m\_\{t\}\}^\{r\}\)\}, the relative change between unimodal entropy and multimodal entropy for modalitymtm\_\{t\}\. IfΔ​H\>0\\Delta H\>0, modality fusion increases the distinguishability among candidates, thereby reducing uncertainty\. IfΔ​H≤0\\Delta H\\leq 0, the conditional modality provides redundant or non\-discriminative information for that specific answering path\.

In Figure[2](https://arxiv.org/html/2610.27120#S2.F2), EDMA identifies four answering trajectories, of which two are unimodal and two are multimodal\. The unimodal trajectories are represented byPIAP\_\{I\}^\{\\text\{$A$\}\}\(initialH=0\.38H=0\.38\) andPTCP\_\{T\}^\{\\text\{$C$\}\}\(initialH=0\.44H=0\.44\)\. The Table\-Image Trajectory starts with partitionPIAP\_\{I\}^\{\\text\{$A$\}\}and ends with partitionPI\|TA,ETP\_\{\\text\{$I$\|$T$\}\}^\{\\text\{$A$\},E\_\{\\text\{$T$\}\}\}, where table data regarding destinations conditions the image set, resulting in a final entropy ofH=0\.38H=0\.38withΔ​H=0%\\Delta H=0\\%\. In contrast, the Image\-Table Trajectory, where visual evidence of the*Metal Arch*conditions the table rows, starts with partitionPTCP\_\{T\}^\{\\text\{$C$\}\}and ends with partitionPT\|IC,EIP\_\{\\text\{$T$\|$I$\}\}^\{\\text\{$C$\},E\_\{\\text\{$I$\}\}\}, reducing logical entropy fromH=0\.44H=0\.44toH=0\.20H=0\.20, achieving a significant entropy decrease ofΔ​H=54%\\Delta H=54\\%, a reduction attributed to the image modality’s influence on the table modality\.

As EDMA prioritizes the trajectory whose final partition yields the lowest logical entropy\(t∗\)\(t^\{\*\}\), it selects the Image–Table Trajectory, isolating the table entry corresponding to the city satisfying requirementCC, which is also associated with an image from which EDMA extracts visual evidence of the arch\. The multimodal partition at the endpoint oft∗t^\{\*\}is chosen as the basis for the final answer\.

##### Final Inference

Once the optimal trajectory, modality, and partition are selected, theL​L​MLLMreceives the questionQQ, the satisfied requirements, the cross\-modal evidenceE∗E^\{\*\}\(e\.g\., the context of the arch\), and the relevant elementsΩ∗\\Omega^\{\*\}that satisfy both the requirements and the cross\-modal evidence \(e\.g\., the specific row for St\. Louis\)\. Based on this information, the system returns the final answera∗a^\{\*\}\(e\.g\.,*Missouri*\)\. If theL​L​MLLMcannot produce an answer \(i\.e\.,a∗=Nonea^\{\*\}=\\text\{None\}\), EDMA selects a new set of modalitiesMs⊆M∖MaM\_\{s\}\\subseteq M\\setminus M\_\{a\}, where the newly selected modalities are not contained in the already analysed modality set\. EDMA then repeats the process: generating new partitions, recomputing entropy deltas, and attempting to produce the answer with the newly acquired information\. The iteration continues until a valid answer is obtained or all modalities are exhausted\.

Crucially, by constructing and selecting among quantifiable answering trajectories, EDMA provides a principled mechanism to guide cross\-modal inference\. Even when SOTA approaches arrive at the same final answer, EDMA makes explicit how that answer is obtained, enabling quantifiable modality\-level attribution\.

## 3Experiments

### 3\.1Experimental Setup

We evaluate our methodology across a diverse set of SOTA LLMs, including both closed source models, such as GPT\-5\.2[OpenAI \(2025\)](https://arxiv.org/html/2610.27120#bib.bib13), Kimi\-K2\.5[Team et al\. \(2026\)](https://arxiv.org/html/2610.27120#bib.bib17), Nova\-2\-Lite\-v1 \(Nova\-2\-Lite\)[Services \(2025\)](https://arxiv.org/html/2610.27120#bib.bib18), and Claude\-Sonnet\-4\.6 \(Claude\-4\.6\)[Anthropic \(2026\)](https://arxiv.org/html/2610.27120#bib.bib14), and open source models, such as Qwen3\-VL\-235B\-A22B \(Qwen3\)[Bai et al\. \(2025\)](https://arxiv.org/html/2610.27120#bib.bib16), and Nemotron\-Nano\-12B\-v2 \(Nemotron\)[NVIDIA \(2025\)](https://arxiv.org/html/2610.27120#bib.bib15)\. Appendix[E](https://arxiv.org/html/2610.27120#A5)reports how to access models and the resources needed to run them, while Appendix[K](https://arxiv.org/html/2610.27120#A11)reports the prompts to implement EDMA’s operations\.

##### Baselines

We compare our approach against SOTA multimodal QA system MAMMQA[Anvekar et al\. \(2025\)](https://arxiv.org/html/2610.27120#bib.bib42)and against several common best\-practice prompting strategies: Direct Prompting \(DP\), where the model generates an answer without explicit intermediate steps, Chain\-of\-Thought \(CoT\)[Wei et al\. \(2022\)](https://arxiv.org/html/2610.27120#bib.bib29), which encourages step\-by\-step generation of intermediate steps within a single generation, and Planning Prompting \(PP\)[Wang et al\. \(2023\)](https://arxiv.org/html/2610.27120#bib.bib31), where the model first generates a plan and subsequently executes it, CoT\-SC[Wang et al\. \(2022\)](https://arxiv.org/html/2610.27120#bib.bib43)and PP\-SC which improve the greedy versions of CoT and PP by generating multiple reasoning chains and returning the majority\-vote final answer\.

##### Datasets

We evaluate our methodology on MultiModalQA[Talmor et al\. \(2021\)](https://arxiv.org/html/2610.27120#bib.bib2)and ManyModalQA[Hannan et al\. \(2020\)](https://arxiv.org/html/2610.27120#bib.bib24)datasets which remain, to the best of our knowledge, among the most complex benchmarks for multimodal QA, requiring cross\-modal integration across text, tables, and images\. They both require deep joint inference over diverse data formats to arrive at a final answer, and thus provide a rigorous testbed for evaluating a model’s ability to integrate and reason over heterogeneous information sources\.

##### Evaluation Metrics

We evaluate all methods using Exact Match \(EM\) and F1 score, following standard practice in multimodal QA[Talmor et al\. \(2021\)](https://arxiv.org/html/2610.27120#bib.bib2);[Yu et al\. \(2023\)](https://arxiv.org/html/2610.27120#bib.bib26);[Lim et al\. \(2024\)](https://arxiv.org/html/2610.27120#bib.bib27)\. EM is a strict binary metric that assigns a score of 1 if the predicted answer exactly matches the ground\-truth string and 0 otherwise\. The F1 score measures token\-level overlap between the predicted and reference answers, balancing precision and recall\. Because EM and F1 can be overly penalizing for semantically correct open\-ended generations, in our evaluation we also include an LLM\-Judge \(LLM\-J\) metric[Majumdar et al\. \(2024\)](https://arxiv.org/html/2610.27120#bib.bib44)\.

### 3\.2EDMA Performance Evaluation

Here we report the performance of EDMA across benchmarks\. We first compare EDMA to standard prompting baselines \(DP, CoT, and PP\) and then we evaluate EDMA against budget\-matched baselines\.

#### 3\.2\.1Comparison between EDMA and Prompting Baselines

Table[1](https://arxiv.org/html/2610.27120#S3.T1)summarises the performance of EDMA alongside standard prompting baselines across six representative MLLMs\. EDMA consistently outperforms Direct Prompting \(DP\), Chain\-of\-Thought \(CoT\), and Planning Prompting \(PP\) across all evaluated backbones and datasets\. On MultiModalQA, EDMA achieves an average score of80\.180\.1EM,81\.181\.1F1, and79\.379\.3LLM\-J, outperforming the strongest prompting baseline \(PP\) by\+22\.7\+22\.7EM,\+14\.0\+14\.0F1, and\+8\.5\+8\.5LLM\-J points\. On ManyModalQA, EDMA reaches83\.083\.0EM,83\.383\.3F1, and89\.189\.1LLM\-J, improving over PP by\+23\.0\+23\.0EM,\+11\.5\+11\.5F1, and\+0\.6\+0\.6LLM\-J\. Across both benchmarks, EDMA achieves average gains of\+22\.9\+22\.9EM,\+12\.8\+12\.8F1, and\+4\.6\+4\.6LLM\-J over the strongest prompting baseline\. The smaller LLM\-Judge gain on ManyModalQA \(\+0\.6 vs \+8\.5 on MultiModalQA\) stems from lower benchmark complexity, limiting EDMA’s possible improvement space\. Nevertheless, EDMA’s strong leads in EM \(\+23\.0\) and F1 \(\+11\.5\) on ManyModalQA confirm that entropy\-guided decomposition remains essential for precise entity extraction even on simpler queries\.

Table 1:EDMA achieves the highest average EM, F1, and LLM\-judge scores on MultiModalQA, ManyModalQA across all evaluated models, outperforming best\-practice prompting strategies: Direct Prompting \(DP\), Chain\-of\-Thought \(CoT\), and Planning Prompting \(PP\)\.In Appendix[F](https://arxiv.org/html/2610.27120#A6)and Appendix[G](https://arxiv.org/html/2610.27120#A7), we respectively report detailed analyses regarding EDMA’s intermediate operation accuracy and statistical variance for EDMA and baselines results\.

#### 3\.2\.2Comparison between EDMA and Budget Matched Baselines

To determine whether EDMA’s gains result merely from increased inference computation, we evaluate EDMA on MultiModalQA against budget\-matched self\-consistency baselines \(CoT\-SC and PP\-SC withN=6N=6samples\) and the multi\-agent framework MAMMQA\. As shown in Table[5](https://arxiv.org/html/2610.27120#A5.T5)\(Appendix[E](https://arxiv.org/html/2610.27120#A5)\), EDMA is approximately6×6\\timescomputationally more expensive than single\-pass CoT and PP, makingN=6N=6a direct budget\-matched baseline\. Furthermore, EDMA’s computational footprint is comparable to MAMMQA, establishing it as an additional budget\-matched baseline\. We evaluate MAMMQA using exclusively LLM\-J, as its verbose answer style would unfairly penalize it under exact\-matching metrics like EM and F1\.

Table 2:Comparison of EDMA with MAMMQA and self\-consistency \(SC\) baselines \(N=6N=6samples\) across EM, F1, and LLM\-J metrics\.As detailed in Table[2](https://arxiv.org/html/2610.27120#S3.T2), the effect of self\-consistency depends on model capacity\. Under PP\-SC, Nova\-2\-Lite gains\+5\.5\+5\.5EM,\+9\.5\+9\.5F1, and\+17\.8\+17\.8LLM\-J over single\-pass PP, whereas Nemotron improves slightly on EM \(\+0\.7\+0\.7\) and F1 \(\+1\.6\+1\.6\) but drops−4\.4\-4\.4LLM\-J\. In contrast, the larger backbones degrade: Claude\-4\.6 drops−4\.9\-4\.9EM,−3\.2\-3\.2F1, and−3\.1\-3\.1LLM\-J, and Kimi\-K2\.5 drops−3\.5\-3\.5EM,−0\.3\-0\.3F1, and−1\.2\-1\.2LLM\-J\. Under CoT\-SC, Nova\-2\-Lite again benefits across all metrics \(\+3\.1\+3\.1EM,\+5\.4\+5\.4F1,\+9\.8\+9\.8LLM\-J over single\-pass CoT\), while Nemotron gains\+3\.9\+3\.9LLM\-J but loses−3\.9\-3\.9EM and−1\.5\-1\.5F1\. The larger backbones again degrade on all metrics: Claude\-4\.6 drops−2\.9\-2\.9EM,−0\.3\-0\.3F1, and−3\.1\-3\.1LLM\-J, and Kimi\-K2\.5 drops−5\.3\-5\.3EM,−3\.1\-3\.1F1, and−3\.5\-3\.5LLM\-J\.

Across the four backbones, SC yields only modest average LLM\-J gains over single\-pass prompting \(\+2\.3\+2\.3for PP\-SC and\+1\.8\+1\.8for CoT\-SC\) and does not improve EM\. In contrast, EDMA \(78\.278\.2LLM\-J,80\.380\.3EM,81\.481\.4F1\) outperforms CoT\-SC \(71\.271\.2LLM\-J\) and PP\-SC \(70\.870\.8LLM\-J\) by\+7\.0\+7\.0and\+7\.4\+7\.4LLM\-J, respectively, and MAMMQA \(75\.775\.7LLM\-J\) by\+2\.5\+2\.5LLM\-J on average, although MAMMQA is stronger on Kimi\-K2\.5 \(88\.488\.4vs\.83\.283\.2LLM\-J\)\. The importance of EDMA’s core mechanisms is further supported by two ablation studies, which isolate the contributions of entropy minimization and cross\-modal conditioning \(Tables[10](https://arxiv.org/html/2610.27120#A8.T10)–[11](https://arxiv.org/html/2610.27120#A8.T11), Appendix[H](https://arxiv.org/html/2610.27120#A8)\)\.

### 3\.3Qualitative Example

![Refer to caption](https://arxiv.org/html/2610.27120v1/qualitative_results.png)Figure 3:Qualitative comparison of CoT and EDMA\. EDMA answers both questions correctly, isolating information in the unimodal setting before bridging modalities in the multimodal step, while CoT returns incorrect answers\.We attribute the performance gains achieved by EDMA to its staged reasoning process\. Specifically, EDMA first decomposes the answer space into unimodal partitions before extracting explicit cross\-modal evidence to construct combined multimodal partitions\. This progressive refinement allows the model to isolate and verify relevant information within individual modalities prior to cross\-modal integration and trajectory selection, which may improve the reliability of evidence aggregation and reduce ambiguity in cross\-modal alignment\. In contrast, standard prompting paradigms such as CoT and PP depend on the model capacity to implicitly manage decomposition and multimodal integration within a single inference stream, which may limit their ability to consistently isolate and align evidence across modalities\.

Figure[3](https://arxiv.org/html/2610.27120#S3.F3)illustrates this contrast on two multi\-hop examples\. In the first, CoT misidentifies the winner of the 33rdTCA Award for Outstanding New Program\. EDMA instead isolates the tabular context to identify “This Is Us” as the winner, and then queries the secondary modality to retrieve the correct broadcasting network \(NBC\)\. In the second example, CoT retrieves the tabular context \(the series “Talking Heads”\) but its prediction is not grounded in the visual modality\. EDMA conditions its visual query on the tabular entry and correctly aligns the image, identifying the depicted activity as writing\.

### 3\.4False Positives Correlate with Higher Entropy

Here we analyse the relationship between entropy and prediction errors to assess the effectiveness of EDMA, examining how entropy correlates with the occurrence of FPs across target modalities\. Figure[4\(a\)](https://arxiv.org/html/2610.27120#S3.F4.sf1)shows that across all modalities, FPs are consistently associated with higher entropy values compared to true positives \(TPs\)\. This indicates that incorrect predictions tend to occur in regions of higher uncertainty, where the candidate answer space remains insufficiently constrained\. In contrast, correct predictions are more likely to arise when entropy is lower, reflecting a more decisive partitioning of the answer space\. See Table[14](https://arxiv.org/html/2610.27120#A9.T14)in Appendix[I](https://arxiv.org/html/2610.27120#A9)for data regarding statistical significance\.

![Refer to caption](https://arxiv.org/html/2610.27120v1/boxplot_tp_fp.png)\(a\)Across all three target modalities, the mean entropy of FPs is higher than for TPs\.
![Refer to caption](https://arxiv.org/html/2610.27120v1/interventions_graph.png)\(b\)Selecting higher\-entropy trajectories is associated with a higher FDR\.

Figure 4:Wrong answers are associated with higher entropy values, as shown by a correlational analysis \(a\) and a controlled trajectory\-selection analysis \(b\)\.
### 3\.5Answering Trajectories with Lower Entropy Yield Better Answers

To further show the connection between entropy level and FPs, we perform a controlled intervention analysis to assess whether entropy directly impacts prediction performance\. We generate three candidate answers per question by selecting answering trajectories corresponding to the lowest, median, and highest entropy levels\. The low\-entropy trajectory corresponds to the one selected by EDMA in the main experiments\. This design keeps the input question fixed while varying the selected trajectory, and hence its entropy\. We evaluate prediction reliability using the False Discovery Rate \(FDR\), defined as the proportion of FPs among all positive predictions\. We compute FDR separately for each entropy group across modalities\. Figure[4\(b\)](https://arxiv.org/html/2610.27120#S3.F4.sf2)shows a consistent monotonic trend: low\-entropy trajectories yield the lowest FDR, while medium\- and high\-entropy trajectories exhibit progressively higher error rates\. This pattern is consistent across all target modalities, indicating a strong association between higher logical entropy and reduced prediction reliability\. By systematically varying the selected trajectory while keeping the input query fixed, we observe a robust degradation in performance as entropy increases\. These results support the importance of entropy minimization in EDMA: lower\-entropy trajectories are associated with fewer false positives and improved prediction performance\. Tables[15](https://arxiv.org/html/2610.27120#A9.T15)–[16](https://arxiv.org/html/2610.27120#A9.T16)in Appendix[I](https://arxiv.org/html/2610.27120#A9), report data regarding statistical significance\.

## 4Related work

##### MultimodalQA

Multimodal question answering \(MMQA\) has been introduced as a challenging task requiring the integration of heterogeneous data sources such as text, tables, and images[Talmor et al\. \(2021\)](https://arxiv.org/html/2610.27120#bib.bib2)\. Subsequent work has focused on improving representation and integration across modalities, including unified architectures such as UniMMQA[Luo et al\. \(2023\)](https://arxiv.org/html/2610.27120#bib.bib3), Solar[Yu et al\. \(2023\)](https://arxiv.org/html/2610.27120#bib.bib26), retrieval\-augmented approaches such as UniRAG[Lim et al\. \(2024\)](https://arxiv.org/html/2610.27120#bib.bib27), and conversational extensions like MMConvQA[Li et al\. \(2022b\)](https://arxiv.org/html/2610.27120#bib.bib25)\. More recent approaches leverage LLMs and MLLMs to enhance multi\-step inference capabilities, for instance by combining language and vision models[Liu et al\. \(2025\)](https://arxiv.org/html/2610.27120#bib.bib30), improving cross\-modal alignment through similarity computation[Zhang et al\. \(2025b\)](https://arxiv.org/html/2610.27120#bib.bib11), or framing information synthesis as a multi\-agent process, as in MAMMQA[Anvekar et al\. \(2025\)](https://arxiv.org/html/2610.27120#bib.bib42)\.

##### LLMs in MultimodalQA

LLMs have been widely adopted to enhance multimodal QA, often through extensions of Chain\-of\-Thought \(CoT\) prompting\. Prior work includes Chain\-of\-Action \(CoA\)[Pan et al\. \(2024\)](https://arxiv.org/html/2610.27120#bib.bib33), knowledge\-augmented CoT \(KAM\-CoT\)[Mondal et al\. \(2024\)](https://arxiv.org/html/2610.27120#bib.bib32), duty\-distinct CoT \(DDCoT\)[Zheng et al\. \(2023\)](https://arxiv.org/html/2610.27120#bib.bib34), and frameworks for structured CoT such as UniCoTT[Zhuang et al\. \(2025\)](https://arxiv.org/html/2610.27120#bib.bib35)\. These approaches improve inference by decomposing complex tasks into intermediate steps or incorporating external knowledge\. Additional adaptations target specific modalities, such as AVQA\-CoT for audio\-visual QA[Li et al\. \(2024\)](https://arxiv.org/html/2610.27120#bib.bib36)and methods that leverage textual signals to guide multimodal understanding[Wang et al\. \(2024b\)](https://arxiv.org/html/2610.27120#bib.bib37)\. Despite these advances, existing CoT\-based approaches do not provide explicit mechanisms to quantify the extent to which each modality contributes to the final prediction\.

##### Uncertainty as Guidance

Recent work has explored incorporating uncertainty quantification into LLM reasoning to improve robustness and decision\-making\. Hu et al\.[Hu et al\. \(2024\)](https://arxiv.org/html/2610.27120#bib.bib45)propose Uncertainty of Thoughts, which evaluates alternative planning paths based on their expected uncertainty reduction\. Yin et al\.[Yin et al\. \(2024\)](https://arxiv.org/html/2610.27120#bib.bib46)introduce uncertainty\-aware adaptive guidance, an approach that relies on uncertainty signals from the LLM to guide the reasoning process\. Liang et al\.[Liang et al\. \(2024\)](https://arxiv.org/html/2610.27120#bib.bib47)propose introspective planning to align model uncertainty with task ambiguity, while Bayat et al\.[Bayat et al\. \(2024\)](https://arxiv.org/html/2610.27120#bib.bib48)leverage uncertainty estimates to guide answer selection\. While these approaches treat uncertainty as a global signal for improving reasoning, they do not provide fine\-grained, modality\-level attribution of uncertainty reduction, limiting their effectiveness in multimodal QA settings\.

## 5Limitations

EDMA’s worst\-case complexity isO⁡\(n3\)O\(n^\{3\}\)when modalities, requirements, and elements grow together\. In practice this translates into many LLM calls per question, costing between55and6\.4×6\.4\\timesmore and taking between2020and46×46\\timeslonger than the prompting baselines \(Appendix[E](https://arxiv.org/html/2610.27120#A5)\)\. Our budget\-matched baselines match its monetary cost, but not its latency or number of calls, and a full accuracy–cost analysis is left to future work\.

Logical entropy is computed under a uniform distribution over elements, which avoids calibrated probabilities but ignores differences in the relevance of individual elements\.

Finally, our methodology does not explicitly address scenarios in which different modalities provide conflicting information\. One possible extension would be to partition the answer space based on parallel interpretations of the available evidence, treating them as distinct answering trajectories, which would let the model explore potentially conflicting paths and evaluate their outcomes independently\.

## 6Conclusion

This paper introduces Entropy\-Driven Multimodal Answering \(EDMA\), a novel methodology for multimodal QA based on logical entropy minimization and the construction of quantifiable answering trajectories\.

By measuring how each step reduces logical entropy, EDMA quantifies the contribution of every modality, improving traceability\. Our analyses show that false positives are associated with higher entropy, and that choosing the lowest\-entropy trajectory tends to lead to better answers, making entropy a useful proxy for performance\. Overall, on the same backbones, EDMA improves over the baselines it is compared against\.

## Acknowledgments and Disclosures

This work was partially supported by theNederlandse Organisatie voor Wetenschappelijk Onderzoek \(NWO\)under the KIC HEWSTI Project under grant no\. KIC1\.VE01\.20\.004\.

The authors declare that they have no competing financial interests or personal relationships that could have appeared to influence the work reported in this paper\.

## References

- \[1\]A\. Talmor, O\. Yoran, A\. Catav, D\. Lahav, Y\. Wang, A\. Asai, G\. Ilharco, H\. Hajishirzi, and J\. Berant\(2021\)Multimodalqa: complex question answering over text, tables and images\.arXiv preprint arXiv:2104\.06039\.Cited by:[§C\.1\.1](https://arxiv.org/html/2610.27120#A3.SS1.SSS1.Px2),[§C\.2](https://arxiv.org/html/2610.27120#A3.SS2.p1.1),[Table 4](https://arxiv.org/html/2610.27120#A4.T4.5.1.2.1.1),[Table 4](https://arxiv.org/html/2610.27120#A4.T4.5.1.7.1),[§1](https://arxiv.org/html/2610.27120#S1.p1.1),[§1](https://arxiv.org/html/2610.27120#S1.p4.1),[§3\.1](https://arxiv.org/html/2610.27120#S3.SS1.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2610.27120#S3.SS1.SSS0.Px3.p1.1),[Table 1](https://arxiv.org/html/2610.27120#S3.T1.5.1.4.1.1),[§4](https://arxiv.org/html/2610.27120#S4.SS0.SSS0.Px1.p1.1)\.
- \[2\]H\. Luo, Y\. Shen, and Y\. Deng\(2023\)Unifying text, tables, and images for multimodal question answering\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 9355–9367\.Cited by:[§C\.1\.1](https://arxiv.org/html/2610.27120#A3.SS1.SSS1.Px3),[Table 4](https://arxiv.org/html/2610.27120#A4.T4.5.1.10.1),[Table 4](https://arxiv.org/html/2610.27120#A4.T4.5.1.4.1),[§1](https://arxiv.org/html/2610.27120#S1.p1.1),[§4](https://arxiv.org/html/2610.27120#S4.SS0.SSS0.Px1.p1.1)\.
- \[3\]N\. Foroutan, A\. Romanou, M\. Ansaripour, J\. M\. Eisenschlos, K\. Aberer, and R\. Lebret\(2025\)Wikimixqa: a multimodal benchmark for question answering over tables and charts\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 24941–24958\.Cited by:[§1](https://arxiv.org/html/2610.27120#S1.p1.1)\.
- \[4\]J\. Gao, R\. Pi, J\. Zhang, J\. Ye, W\. Zhong, Y\. Wang, L\. Hong, J\. Han, H\. Xu, Z\. Li,et al\.\(2023\)G\-llava: solving geometric problem with multi\-modal large language model\.arXiv preprint arXiv:2312\.11370\.Cited by:[§1](https://arxiv.org/html/2610.27120#S1.p2.1)\.
- \[5\]H\. Zhang, Z\. Meng, M\. Luo, H\. Han, L\. Liao, E\. Cambria, and H\. Fei\(2025\)Towards multimodal empathetic response generation: a rich text\-speech\-vision avatar\-based benchmark\.InProceedings of the ACM on Web Conference 2025,pp\. 2872–2881\.Cited by:[§1](https://arxiv.org/html/2610.27120#S1.p2.1)\.
- \[6\]J\. Li, D\. Zhang, X\. Wang, Z\. Hao, J\. Lei, Q\. Tan, C\. Zhou, W\. Liu, Y\. Yang, X\. Xiong,et al\.\(2025\)Chemvlm: exploring the power of multimodal large language models in chemistry area\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 415–423\.Cited by:[§1](https://arxiv.org/html/2610.27120#S1.p2.1)\.
- \[7\]S\. Yang, J\. Liu, R\. Zhang, M\. Pan, Z\. Guo, X\. Li, Z\. Chen, P\. Gao, H\. Li, Y\. Guo,et al\.\(2025\)Lidar\-llm: exploring the potential of large language models for 3d lidar understanding\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 9247–9255\.Cited by:[§1](https://arxiv.org/html/2610.27120#S1.p2.1)\.
- \[8\]H\. Xia, Z\. Yang, J\. Zou, R\. Tracy, Y\. Wang, C\. Lu, C\. Lai, Y\. He, X\. Shao, Z\. Xie,et al\.\(2024\)Sportu: a comprehensive sports understanding benchmark for multimodal large language models\.arXiv preprint arXiv:2410\.08474\.Cited by:[§1](https://arxiv.org/html/2610.27120#S1.p2.1)\.
- \[9\]L\. Wang, Y\. Hu, J\. He, X\. Xu, N\. Liu, H\. Liu, and H\. T\. Shen\(2024\)T\-sciq: teaching multimodal chain\-of\-thought reasoning via large language model signals for science question answering\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 19162–19170\.Cited by:[§1](https://arxiv.org/html/2610.27120#S1.p2.1)\.
- \[10\]J\. Zhao, X\. Wei, and L\. Bo\(2025\)R1\-omni: explainable omni\-multimodal emotion recognition with reinforcement learning\.arXiv preprint arXiv:2503\.05379\.Cited by:[§1](https://arxiv.org/html/2610.27120#S1.p2.1)\.
- \[11\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[§C\.1\.2](https://arxiv.org/html/2610.27120#A3.SS1.SSS2.Px2),[§1](https://arxiv.org/html/2610.27120#S1.p2.1),[§3\.1](https://arxiv.org/html/2610.27120#S3.SS1.SSS0.Px1.p1.1)\.
- \[12\]Z\. Zhang, A\. Zhang, M\. Li, H\. Zhao, G\. Karypis, and A\. Smola\(2023\)Multimodal chain\-of\-thought reasoning in language models\.arXiv preprint arXiv:2302\.00923\.Cited by:[§1](https://arxiv.org/html/2610.27120#S1.p2.1)\.
- \[13\]G\. Zheng, B\. Yang, J\. Tang, H\. Zhou, and S\. Yang\(2023\)Ddcot: duty\-distinct chain\-of\-thought prompting for multimodal reasoning in language models\.Advances in Neural Information Processing Systems36,pp\. 5168–5191\.Cited by:[§1](https://arxiv.org/html/2610.27120#S1.p2.1),[§4](https://arxiv.org/html/2610.27120#S4.SS0.SSS0.Px2.p1.1)\.
- \[14\]C\. Mitra, B\. Huang, T\. Darrell, and R\. Herzig\(2024\)Compositional chain\-of\-thought prompting for large multimodal models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 14420–14431\.Cited by:[§1](https://arxiv.org/html/2610.27120#S1.p2.1)\.
- \[15\]D\. Mondal, S\. Modi, S\. Panda, R\. Singh, and G\. S\. Rao\(2024\)Kam\-cot: knowledge augmented multimodal chain\-of\-thoughts reasoning\.InProceedings of the AAAI conference on artificial intelligence,Vol\.38,pp\. 18798–18806\.Cited by:[§1](https://arxiv.org/html/2610.27120#S1.p2.1),[§4](https://arxiv.org/html/2610.27120#S4.SS0.SSS0.Px2.p1.1)\.
- \[16\]Y\. Li, X\. Lan, H\. Chen, K\. Lu, and D\. Jiang\(2025\)Multimodal pear chain\-of\-thought reasoning for multimodal sentiment analysis\.ACM Transactions on Multimedia Computing, Communications and Applications20\(9\),pp\. 1–23\.Cited by:[§1](https://arxiv.org/html/2610.27120#S1.p2.1)\.
- \[17\]Z\. Pan, H\. Luo, M\. Li, and H\. Liu\(2024\)Chain\-of\-action: faithful and multimodal question answering through large language models\.arXiv preprint arXiv:2403\.17359\.Cited by:[§1](https://arxiv.org/html/2610.27120#S1.p2.1),[§4](https://arxiv.org/html/2610.27120#S4.SS0.SSS0.Px2.p1.1)\.
- \[18\]W\. Wu, S\. Mao, Y\. Zhang, Y\. Xia, L\. Dong, L\. Cui, and F\. Wei\(2024\)Mind’s eye of llms: visualization\-of\-thought elicits spatial reasoning in large language models\.Advances in Neural Information Processing Systems37,pp\. 90277–90317\.Cited by:[§1](https://arxiv.org/html/2610.27120#S1.p2.1)\.
- \[19\]Z\. Hu, C\. Liu, X\. Feng, Y\. Zhao, S\. Ng, A\. T\. Luu, J\. He, P\. W\. W\. Koh, and B\. Hooi\(2024\)Uncertainty of thoughts: uncertainty\-aware planning enhances information seeking in llms\.Advances in Neural Information Processing Systems37,pp\. 24181–24215\.Cited by:[§1](https://arxiv.org/html/2610.27120#S1.p2.1),[§4](https://arxiv.org/html/2610.27120#S4.SS0.SSS0.Px3.p1.1)\.
- \[20\]Z\. Yin, Q\. Sun, Q\. Guo, Z\. Zeng, X\. Li, J\. Dai, Q\. Cheng, X\. Huang, and X\. Qiu\(2024\)Reasoning in flux: enhancing large language models reasoning through uncertainty\-aware adaptive guidance\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 2401–2416\.Cited by:[§1](https://arxiv.org/html/2610.27120#S1.p2.1),[§4](https://arxiv.org/html/2610.27120#S4.SS0.SSS0.Px3.p1.1)\.
- \[21\]K\. Liang, Z\. Zhang, and J\. F\. Fisac\(2024\)Introspective planning: aligning robots’ uncertainty with inherent task ambiguity\.Advances in Neural Information Processing Systems37,pp\. 71998–72031\.Cited by:[§1](https://arxiv.org/html/2610.27120#S1.p2.1),[§4](https://arxiv.org/html/2610.27120#S4.SS0.SSS0.Px3.p1.1)\.
- \[22\]D\. Ellerman\(2021\)Introduction to logical entropy and its relationship to shannon entropy\.arXiv preprint arXiv:2112\.01966\.Cited by:[§1](https://arxiv.org/html/2610.27120#S1.p3.1),[§2\.5](https://arxiv.org/html/2610.27120#S2.SS5.SSS0.Px1.p1.1)\.
- \[23\]D\. Hannan, A\. Jain, and M\. Bansal\(2020\)Manymodalqa: modality disambiguation and qa over diverse inputs\.InProceedings of the AAAI conference on artificial intelligence,Vol\.34,pp\. 7879–7886\.Cited by:[§C\.1\.1](https://arxiv.org/html/2610.27120#A3.SS1.SSS1.Px6),[§C\.2](https://arxiv.org/html/2610.27120#A3.SS2.p2.1),[Table 4](https://arxiv.org/html/2610.27120#A4.T4.5.1.11.1),[Table 4](https://arxiv.org/html/2610.27120#A4.T4.5.1.8.1.1),[§1](https://arxiv.org/html/2610.27120#S1.p4.1),[§3\.1](https://arxiv.org/html/2610.27120#S3.SS1.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2610.27120#S3.T1.5.1.9.1.1)\.
- \[24\]OpenAI\(2025\)Update to gpt\-5 system card: gpt\-5\.2\.External Links:[Link](https://openai.com/index/gpt-5-system-card-update-gpt-5-2/)Cited by:[§3\.1](https://arxiv.org/html/2610.27120#S3.SS1.p1.1)\.
- \[25\]K\. Team, T\. Bai, Y\. Bai, Y\. Bao, S\. Cai, Y\. Cao, Y\. Charles, H\. Che, C\. Chen, G\. Chen,et al\.\(2026\)Kimi k2\. 5: visual agentic intelligence\.arXiv preprint arXiv:2602\.02276\.Cited by:[§3\.1](https://arxiv.org/html/2610.27120#S3.SS1.p1.1)\.
- \[26\]A\. W\. Services\(2025\)Amazon nova 2 lite – overview\.External Links:[Link](https://docs.aws.amazon.com/ai/responsible-ai/nova-2-lite/overview.html)Cited by:[§3\.1](https://arxiv.org/html/2610.27120#S3.SS1.p1.1)\.
- \[27\]Anthropic\(2026\)Introducing claude sonnet 4\.6\.External Links:[Link](https://www.anthropic.com/news/claude-sonnet-4-6/)Cited by:[§3\.1](https://arxiv.org/html/2610.27120#S3.SS1.p1.1)\.
- \[28\]S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge,et al\.\(2025\)Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[§3\.1](https://arxiv.org/html/2610.27120#S3.SS1.p1.1)\.
- \[29\]NVIDIA\(2025\)Nemotron nano 12b v2 vl model card\.External Links:[Link](https://build.nvidia.com/nvidia/nemotron-nano-12b-v2-vl/modelcard)Cited by:[§3\.1](https://arxiv.org/html/2610.27120#S3.SS1.p1.1)\.
- \[30\]T\. Anvekar, K\. S\. Rajput, C\. Baral, and V\. Gupta\(2025\)Rethinking information synthesis in multimodal question answering a multi\-agent perspective\.InProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics,pp\. 3674–3686\.Cited by:[§C\.1\.1](https://arxiv.org/html/2610.27120#A3.SS1.SSS1.Px1),[§3\.1](https://arxiv.org/html/2610.27120#S3.SS1.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2610.27120#S4.SS0.SSS0.Px1.p1.1)\.
- \[31\]L\. Wang, W\. Xu, Y\. Lan, Z\. Hu, Y\. Lan, R\. K\. Lee, and E\. Lim\(2023\)Plan\-and\-solve prompting: improving zero\-shot chain\-of\-thought reasoning by large language models\.InProceedings of the 61st annual meeting of the association for computational linguistics \(volume 1: long papers\),pp\. 2609–2634\.Cited by:[§C\.1\.2](https://arxiv.org/html/2610.27120#A3.SS1.SSS2.Px3),[§3\.1](https://arxiv.org/html/2610.27120#S3.SS1.SSS0.Px1.p1.1)\.
- \[32\]X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou\(2022\)Self\-consistency improves chain of thought reasoning in language models\.arXiv preprint arXiv:2203\.11171\.Cited by:[§C\.1\.2](https://arxiv.org/html/2610.27120#A3.SS1.SSS2.Px4),[§C\.1\.2](https://arxiv.org/html/2610.27120#A3.SS1.SSS2.Px5.p1.1),[§3\.1](https://arxiv.org/html/2610.27120#S3.SS1.SSS0.Px1.p1.1)\.
- \[33\]B\. Yu, C\. Fu, H\. Yu, F\. Huang, and Y\. Li\(2023\)Unified language representation for question answering over text, tables, and images\.InFindings of the Association for Computational Linguistics: ACL 2023,pp\. 4756–4765\.Cited by:[§C\.1\.1](https://arxiv.org/html/2610.27120#A3.SS1.SSS1.Px4),[Table 4](https://arxiv.org/html/2610.27120#A4.T4.5.1.6.1),[§3\.1](https://arxiv.org/html/2610.27120#S3.SS1.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2610.27120#S4.SS0.SSS0.Px1.p1.1)\.
- \[34\]Q\. Z\. Lim, C\. P\. Lee, K\. M\. Lim, and A\. K\. Samingan\(2024\)Unirag: unification, retrieval, and generation for multimodal question answering with pre\-trained language models\.IEEE Access12,pp\. 71505–71519\.Cited by:[§C\.1\.1](https://arxiv.org/html/2610.27120#A3.SS1.SSS1.Px5),[Table 4](https://arxiv.org/html/2610.27120#A4.T4.5.1.5.1),[§3\.1](https://arxiv.org/html/2610.27120#S3.SS1.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2610.27120#S4.SS0.SSS0.Px1.p1.1)\.
- \[35\]A\. Majumdar, A\. Ajay, X\. Zhang, P\. Putta, S\. Yenamandra, M\. Henaff, S\. Silwal, P\. Mcvay, O\. Maksymets, S\. Arnaud,et al\.\(2024\)Openeqa: embodied question answering in the era of foundation models\.In2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 16488–16498\.Cited by:[§3\.1](https://arxiv.org/html/2610.27120#S3.SS1.SSS0.Px3.p1.1)\.
- \[36\]Y\. Li, W\. Li, and L\. Nie\(2022\)Mmcoqa: conversational question answering over text, tables, and images\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 4220–4231\.Cited by:[§4](https://arxiv.org/html/2610.27120#S4.SS0.SSS0.Px1.p1.1)\.
- \[37\]J\. Liu, L\. Li, S\. Rao, X\. Gao, W\. Guan, B\. Li, and C\. Ma\(2025\)Union is strength\! unite the power of llms and mllms for chart question answering\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 5487–5495\.Cited by:[§4](https://arxiv.org/html/2610.27120#S4.SS0.SSS0.Px1.p1.1)\.
- \[38\]X\. Zhang, F\. Zeng, Y\. Quan, Z\. Hui, and J\. Yao\(2025\)Enhancing multimodal large language models complex reason via similarity computation\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 10203–10211\.Cited by:[§4](https://arxiv.org/html/2610.27120#S4.SS0.SSS0.Px1.p1.1)\.
- \[39\]X\. Zhuang, Z\. Zhu, Z\. Wang, X\. Cheng, and Y\. Zou\(2025\)UnicoTT: a unified framework for structural chain\-of\-thought distillation\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§4](https://arxiv.org/html/2610.27120#S4.SS0.SSS0.Px2.p1.1)\.
- \[40\]G\. Li, H\. Du, and D\. Hu\(2024\)AVQA\-cot: when cot meets question answering in audio\-visual scenarios\.Cited by:[§4](https://arxiv.org/html/2610.27120#S4.SS0.SSS0.Px2.p1.1)\.
- \[41\]Y\. Wang, P\. Sun, Y\. Li, H\. Zhang, and D\. Hu\(2024\)Can textual semantics mitigate sounding object segmentation preference?\.InEuropean Conference on Computer Vision,pp\. 340–356\.Cited by:[§4](https://arxiv.org/html/2610.27120#S4.SS0.SSS0.Px2.p1.1)\.
- \[42\]F\. F\. Bayat, X\. Liu, H\. Jagadish, and L\. Wang\(2024\)Enhanced language model truthfulness with learnable intervention and uncertainty expression\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 12388–12400\.Cited by:[§4](https://arxiv.org/html/2610.27120#S4.SS0.SSS0.Px3.p1.1)\.
- \[43\]C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. Liu\(2020\)Exploring the limits of transfer learning with a unified text\-to\-text transformer\.Journal of machine learning research21\(140\),pp\. 1–67\.Cited by:[Table 4](https://arxiv.org/html/2610.27120#A4.T4.5.1.10.2),[Table 4](https://arxiv.org/html/2610.27120#A4.T4.5.1.4.2),[Table 4](https://arxiv.org/html/2610.27120#A4.T4.5.1.6.2)\.
- \[44\]H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee\(2023\)Visual instruction tuning\.Advances in neural information processing systems36,pp\. 34892–34916\.Cited by:[Table 4](https://arxiv.org/html/2610.27120#A4.T4.5.1.5.2)\.
- \[45\]J\. Li, D\. Li, C\. Xiong, and S\. Hoi\(2022\)Blip: bootstrapping language\-image pre\-training for unified vision\-language understanding and generation\.InInternational conference on machine learning,pp\. 12888–12900\.Cited by:[Table 4](https://arxiv.org/html/2610.27120#A4.T4.5.1.6.2)\.
- \[46\]Y\. Liu, M\. Ott, N\. Goyal, J\. Du, M\. Joshi, D\. Chen, O\. Levy, M\. Lewis, L\. Zettlemoyer, and V\. Stoyanov\(2019\)Roberta: a robustly optimized bert pretraining approach\.arXiv preprint arXiv:1907\.11692\.Cited by:[Table 4](https://arxiv.org/html/2610.27120#A4.T4.5.1.7.2)\.
- \[47\]J\. Lu, D\. Batra, D\. Parikh, and S\. Lee\(2019\)Vilbert: pretraining task\-agnostic visiolinguistic representations for vision\-and\-language tasks\.Advances in neural information processing systems32\.Cited by:[Table 4](https://arxiv.org/html/2610.27120#A4.T4.5.1.7.2)\.
- \[48\]M\. E\. Peters, M\. Neumann, M\. Iyyer, M\. Gardner, C\. Clark, K\. Lee, and L\. Zettlemoyer\(2018\)Deep contextualized word representations\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long Papers\),pp\. 2227–2237\.Cited by:[Table 4](https://arxiv.org/html/2610.27120#A4.T4.5.1.11.2)\.
- \[49\]J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova\(2019\)Bert: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 \(long and short papers\),pp\. 4171–4186\.Cited by:[Table 4](https://arxiv.org/html/2610.27120#A4.T4.5.1.11.2)\.

## Appendix ANotation

Table 3:Summary of notation used\.
## Appendix BAlgorithm

The EDMA algorithm has a worst\-case computational complexity ofO⁡\(M⋅R⋅K\+M2⋅R\)O\(M\\cdot R\\cdot K\+M^\{2\}\\cdot R\), whereMMdenotes the number of modalities,RRthe number of requirements, andKKthe maximum number of elements per modality\. Stage 1 \(Unimodal Partitioning\) contributesO⁡\(M⋅R⋅K\)O\(M\\cdot R\\cdot K\), evaluating each of theKKelements per modality against each requirement, and also caches the cross\-modal evidence extracted for each\(m,r\)\(m,r\)pair\. Stage 2 \(Cross\-Modal Sequential Conditioning\) contributesO⁡\(M2⋅R\)O\(M^\{2\}\\cdot R\), refining each of theM⋅RM\\cdot Rtarget pairs by conditioning on the remainingMMmodalities viaO⁡\(1\)O\(1\)lookups into this cache\. UnderM∼R∼K∼nM\\sim R\\sim K\\sim n, the overall complexity isO⁡\(n3\)O\(n^\{3\}\)\.

Algorithm 1Entropy\-Driven Multimodal Answering \(EDMA\)1:

QQ,

MM,

\{Ωm,m∈M\}\\\{\\Omega\_\{m\},m\\in M\\\}
2:Final answer

a∗a^\{\*\}
3:

R←L​L​M​\(P​r​o​m​p​tR,Q\)R\\leftarrow LLM\(Prompt\_\{R\},Q\)// Extraction of answer requirements

4:

Ms←L​L​M​\(P​r​o​m​p​tMs,Q,\{Ωm,m∈M\}\)M\_\{s\}\\leftarrow LLM\(Prompt\_\{M\_\{s\}\},Q,\\\{\\Omega\_\{m\},m\\in M\\\}\)// Prediction of Starting Modalities

5:

a∗←Nonea^\{\*\}\\leftarrow\\text\{None\}
6:

Ma←∅M\_\{a\}\\leftarrow\\emptyset
7:while

a∗=Nonea^\{\*\}=\\text\{None\}and

Ms≠∅M\_\{s\}\\neq\\emptysetdo

8:

Ma←Ma∪MsM\_\{a\}\\leftarrow M\_\{a\}\\cup M\_\{s\}
9:

M←M∖MsM\\leftarrow M\\setminus M\_\{s\}
10:// Stage 1: Unimodal Partitioning \(with Extraction Caching\)

11:for

\(m,r\)∈Ma×R\(m,r\)\\in M\_\{a\}\\times Rdo

12:Generate

Pmr=\{Ωmr,Ωm¬r\}P\_\{m\}^\{r\}=\\\{\\Omega\_\{m\}^\{r\},\\Omega\_\{m\}^\{\\neg r\}\\\}and compute

H⁡\(Pmr\)H\(P\_\{m\}^\{r\}\)
13:Store

H⁡\(Pmr\)H\(P\_\{m\}^\{r\}\)and

Ωmr\\Omega\_\{m\}^\{r\}in

ℋu​n​i​\[m,r\]\\mathcal\{H\}\_\{uni\}\[m,r\]
14:

E⁡\[m,r\]←\{L​L​M​\(P​r​o​m​p​tE​x​t​r​a​c​t,Q,ω\)∣ω∈Ωmr\}E\[m,r\]\\leftarrow\\\{LLM\(Prompt\_\{Extract\},Q,\\omega\)\\mid\\omega\\in\\Omega\_\{m\}^\{r\}\\\}// Cached once per\(m,r\)\(m,r\)

15:endfor

16:// Stage 2: Cross\-Modal Sequential Conditioning

17:for

\(mt,r\)∈Ma×R\(m\_\{t\},r\)\\in M\_\{a\}\\times Rdo

18:

Mc←Ma∖\{mt\},Ωc​u​r​r←Ωmtr,Ec​u​r​r←∅M\_\{c\}\\leftarrow M\_\{a\}\\setminus\\\{m\_\{t\}\\\},\\,\\Omega\_\{curr\}\\leftarrow\\Omega\_\{m\_\{t\}\}^\{r\},\\,E\_\{curr\}\\leftarrow\\emptyset
19:for

m∈Mcm\\in M\_\{c\}do

20:

Em←E⁡\[m,r\]E\_\{m\}\\leftarrow E\[m,r\]//O⁡\(1\)O\(1\)lookup

21:Generate

Pmt\|mr,Em=\{Ωmtr,Em,Ωmtr,¬Em\}P\_\{m\_\{t\}\|m\}^\{r,E\_\{m\}\}=\\\{\\Omega\_\{m\_\{t\}\}^\{r,E\_\{m\}\},\\Omega\_\{m\_\{t\}\}^\{r,\\neg E\_\{m\}\}\\\}using

Ωc​u​r​r\\Omega\_\{curr\}and

EmE\_\{m\}
22:

Ωc​u​r​r←Ωmtr,Em\\Omega\_\{curr\}\\leftarrow\\Omega\_\{m\_\{t\}\}^\{r,E\_\{m\}\}
23:

Ec​u​r​r←Ec​u​r​r∪EmE\_\{curr\}\\leftarrow E\_\{curr\}\\cup E\_\{m\}
24:endfor

25:

H⁡\(Pmt\|mr,Em\)←H⁡\(Pmt\|mr,Em\)H\(P\_\{m\_\{t\}\|m\}^\{r,E\_\{m\}\}\)\\leftarrow H\(P\_\{m\_\{t\}\|m\}^\{r,E\_\{m\}\}\)
26:Store

H⁡\(Pmt\|mr,Em\),Ωc​u​r​r,Ec​u​r​rH\(P\_\{m\_\{t\}\|m\}^\{r,E\_\{m\}\}\),\\Omega\_\{curr\},E\_\{curr\}in

ℋm​u​l​t​i​\[mt,r\]\\mathcal\{H\}\_\{multi\}\[m\_\{t\},r\]
27:endfor

28:// Stage 3: Trajectory Selection

29:

𝒯←∅\\mathcal\{T\}\\leftarrow\\emptyset
30:for

\(mt,r\)∈Ma×R\(m\_\{t\},r\)\\in M\_\{a\}\\times Rdo

31:

Hu,Ωu←ℋu​n​i​\[mt,r\]H\_\{u\},\\Omega\_\{u\}\\leftarrow\\mathcal\{H\}\_\{uni\}\[m\_\{t\},r\]
32:

Hm,Ωm,Em←ℋm​u​l​t​i​\[mt,r\]H\_\{m\},\\Omega\_\{m\},E\_\{m\}\\leftarrow\\mathcal\{H\}\_\{multi\}\[m\_\{t\},r\]
33:

𝒯←𝒯∪\{\(mt,r,Hu,Ωu,∅\)\}\\mathcal\{T\}\\leftarrow\\mathcal\{T\}\\cup\\\{\(m\_\{t\},r,H\_\{u\},\\Omega\_\{u\},\\emptyset\)\\\}// Unimodal path

34:if

Hu\>0H\_\{u\}\>0then

Δ​H←\(Hu−Hm\)/Hu\\Delta H\\leftarrow\(H\_\{u\}\-H\_\{m\}\)/H\_\{u\}else

Δ​H←0\\Delta H\\leftarrow 0
35:

𝒯←𝒯∪\{\(mt,r,Hm,Ωm,Em,Δ​H\)\}\\mathcal\{T\}\\leftarrow\\mathcal\{T\}\\cup\\\{\(m\_\{t\},r,H\_\{m\},\\Omega\_\{m\},E\_\{m\},\\Delta H\)\\\}// Multimodal path

36:endfor

37:Select

t∗=\(m∗,r∗,H∗,Ω∗,E∗\)t^\{\*\}=\(m^\{\*\},r^\{\*\},H^\{\*\},\\Omega^\{\*\},E^\{\*\}\)from

𝒯\\mathcal\{T\}that minimizes

HH
38:// Stage 4: Final Inference

39:

a∗←L​L​M​\(Q,Ω∗,E∗\)a^\{\*\}\\leftarrow LLM\(Q,\\Omega^\{\*\},E^\{\*\}\)// Inference from optimal subset and evidence

40:if

a∗=Nonea^\{\*\}=\\text\{None\}and

M≠∅M\\neq\\emptysetthen

41:

Ms←L​L​M​\(P​r​o​m​p​tMs,Q,\{Ωm,m∈M\}\)M\_\{s\}\\leftarrow LLM\(Prompt\_\{M\_\{s\}\},Q,\\\{\\Omega\_\{m\},m\\in M\\\}\)
42:else

43:

Ms←∅M\_\{s\}\\leftarrow\\emptyset
44:endif

45:endwhile

46:return

a∗a^\{\*\}

## Appendix CExperimental Setup

### C\.1Baselines

In this section, we provide detailed descriptions of the baselines used in our experiments\. We categorize these into multimodal architectures and prompting\-based strategies\.

#### C\.1\.1Multimodal QA Systems

##### MAMMQA\[[30](https://arxiv.org/html/2610.27120#bib.bib42)\]

A multi\-agent QA framework for multimodal inputs spanning text, tables, and images\. The system comprises two Vision\-Language Model \(VLM\) agents and one text\-based LLM agent\. The first VLM decomposes the query into sub\-questions to sequentially retrieve partial evidence across modalities, the second VLM synthesizes intermediate insights via cross\-modal reasoning, and the LLM aggregates the findings into a cohesive final answer\.

##### MultiModalQA\[[1](https://arxiv.org/html/2610.27120#bib.bib2)\]

The baseline pipeline introduced alongside the MultiModalQA benchmark\. It employs modality\-specific information extractors combined with anImplicitDecompmodule to perform multi\-step question decomposition and answer synthesis across text, tables, and images\.

##### UniMMQA\[[2](https://arxiv.org/html/2610.27120#bib.bib3)\]

A unified multimodal QA framework designed to reduce the modality gap by projecting heterogeneous data \(text, tables, and images\) into a shared representation space, enabling integrated cross\-modal retrieval and answer generation\.

##### Solar\[[33](https://arxiv.org/html/2610.27120#bib.bib26)\]

A sequence\-to\-sequence multimodal framework that unifies visual and textual token processing within a single transformer architecture, optimized for cross\-modal alignment through joint pre\-training objectives\.

##### UniRAG\[[34](https://arxiv.org/html/2610.27120#bib.bib27)\]

A retrieval\-augmented generation \(RAG\) framework that unifies evidence retrieval across textual, visual, and tabular corpora before feeding the aggregated context into a Large Multimodal Model \(LMM\) for answer generation\.

##### ManyModalQA\[[23](https://arxiv.org/html/2610.27120#bib.bib24)\]

The baseline framework introduced with the ManyModalQA benchmark\. It incorporates an explicit modality routing mechanism to determine the most relevant data source \(text, image, or table\) prior to applying specialized extraction modules to return the final answer\.

#### C\.1\.2Prompting Strategies

##### Direct Prompting \(DP\)

A standard zero\-shot prompting approach where the model receives the input context and query, and directly outputs the final answer without explicit intermediate reasoning steps\.

##### Chain\-of\-Thought \(CoT\)\[[11](https://arxiv.org/html/2610.27120#bib.bib29)\]

A prompting method that prompts the model to generate a sequence of intermediate natural language reasoning steps before outputting the final answer within a single generation stream\.

##### Planning Prompting \(PP\)\[[31](https://arxiv.org/html/2610.27120#bib.bib31)\]

A two\-stage prompting technique that decouples reasoning planning from execution: the model first generates a high\-level plan of sub\-tasks, and subsequently executes each sub\-task to derive the final answer\.

##### CoT Self\-Consistency \(CoT\-SC\)\[[32](https://arxiv.org/html/2610.27120#bib.bib43)\]

An ensemble strategy that samplesNNindependent Chain\-of\-Thought reasoning paths and selects the final answer via majority voting\.

##### PP Self\-Consistency \(PP\-SC\)

An extension of self\-consistency\[[32](https://arxiv.org/html/2610.27120#bib.bib43)\]applied to Planning Prompting, samplingNNindependent plan\-and\-execute reasoning trajectories and selecting the answer via majority voting\.

### C\.2Datasets

MultiModalQA\[[1](https://arxiv.org/html/2610.27120#bib.bib2)\]is a comprehensive benchmark designed to evaluate a model’s ability to perform joint integration across three distinct data modalities: structured tables, natural language text, and images\. Unlike earlier multimodal datasets where the answer usually resides within a single source \(e\.g\., just an image or just a paragraph\), MultiModalQA is specifically engineered to require multi\-hop reasoning\. We report results on the validation set since ground\-truth annotations for the test set are not publicly accessible\.

To further show the generalizability of our methodology, we test it also onManyModalQA\[[23](https://arxiv.org/html/2610.27120#bib.bib24)\], which is a multimodal question\-answering benchmark designed to evaluate an agent’s ability to reason across three distinct data types: textual passages, structured tables, and visual images\. Unlike standard multimodal datasets where the target modality is often implied by the question, ManyModalQA contains ambiguous questions where the source of the answer cannot be determined by the question text alone\.

## Appendix DComparison with Previous Multimodal QA Systems

Table[4](https://arxiv.org/html/2610.27120#A4.T4)compares EDMA with prior multimodal QA systems\. EDMA achieves the best performance across both datasets\. On MultiModalQA, it obtains82\.882\.8EM and83\.583\.5F1, improving over the strongest prior system \(UniMMQA\) by\+9\.1\+9\.1EM and\+3\.4\+3\.4F1\. On ManyModalQA, the gains are substantially larger, with improvements of\+35\.9\+35\.9EM and\+36\.1\+36\.1F1 over UniMMQA\. We note that these comparisons reflect both methodological improvements and differences in underlying foundation models\.

Table 4:Comparison of EDMA against prior specialized multimodal QA frameworks on MultiModalQA and ManyModalQA datasets\.
## Appendix EModels

### E\.1Access to Models

We access GPT\-5\.2 via its official API and all remaining models through Amazon Bedrock\. For all experiments, we use greedy decoding \(temperature=0\\text\{temperature\}=0\) with a maximum generation length of10001000tokens\.

### E\.2Computational Resources

Table[5](https://arxiv.org/html/2610.27120#A5.T5)details the raw average token consumption, API call counts, monetary cost, and latency per model across evaluated approaches \(Nova\-2\-Lite, GPT\-5\.2, Kimi\-K2\.5, Nemotron, Qwen3, and Claude\-4\.6\)\. Furthermore, Table[6](https://arxiv.org/html/2610.27120#A5.T6)summarizes the relative computational overhead of EDMA compared to each baseline strategy\.

Table 5:Average computational cost and latency across models and approaches\.As shown in Table[6](https://arxiv.org/html/2610.27120#A5.T6), EDMA is on average22\.83×22\.83\\timesslower than CoT,45\.79×45\.79\\timesslower than DP,19\.92×19\.92\\timesslower than PP, and11\.05×11\.05\\timesslower than MAMMQA\. This reflects the higher computational complexity of the algorithm\. Regarding financial cost, EDMA is on average5\.06×5\.06\\timesmore expensive than PP,5\.91×5\.91\\timesmore expensive than CoT, and6\.40×6\.40\\timesmore expensive than DP\. However, its cost is directly comparable to MAMMQA \(1\.14×1\.14\\times\)\. Although MAMMQA executes only seven API calls per query, it repeatedly inputs all data modalities \(text, images, and tables\) into the LLM context, making its token consumption and cost comparable to EDMA\. Note that estimates for MAMMQA were computed across all models except GPT\-5\.2 and Qwen3, where evaluations were omitted due to budget restrictions\.

Table 6:Average cost and latency overhead of EDMA compared with other approaches\.

## Appendix FAnalysis of Intermediate Operations

Table[7](https://arxiv.org/html/2610.27120#A6.T7)reports the average accuracy across each intermediate phase of EDMA, consisting in modality selection, criteria extraction, intermediate answer extraction, and multimodal fusion, evaluated specifically on questions where EDMA produced an incorrect final answer\.

Table 7:Intermediate\-step accuracy for questions incorrectly answered by EDMA\.Across most models, errors primarily originate during multimodal fusion, where the system must cross\-reference information extracted from one modality \(e\.g\., text\) with another \(e\.g\., images\)\. In these cases, incorrect final answers stem from over\-filtered or wrongly filtered partitions, resulting in the selection of a low\-entropy yet incorrect multimodal partition\. Nemotron is the sole exception: its failures are driven by low accuracy at the earlier Intermediate Answers stage \(30%30\\%\), causing trajectories to fail before reaching the fusion step\.

Table[8](https://arxiv.org/html/2610.27120#A6.T8)presents the corresponding accuracy metrics across all evaluated questions, both correctly and incorrectly answered\. Early\-stage accuracies \(modality selection, criteria extraction, and intermediate answer extraction\) remain consistently high across the full dataset\. The critical divergence appears in the multimodal fusion phase, confirming that cross\-modal fusion constitutes the main point of failure in the methodology\.

Table 8:Intermediate\-step accuracy across all questions, both correctly and incorrectly answered\.
## Appendix GEDMA Statistical Variance

Table[9](https://arxiv.org/html/2610.27120#A7.T9)reports the mean and standard deviation calculated across ten independent runs\. The analysis confirms that the performance advantages observed in single\-point comparisons remain stable under repeated evaluation: EDMA’s lead over MAMMQA on Claude\-4\.6, Nemotron, and Nova\-2\-Lite, as well as MAMMQA’s advantage on Kimi\-K2\.5, hold consistently across runs\.

Table 9:Mean±\\pmstandard deviation of EM, F1, and LLM\-judge score \(LLM\-J\) over 10 runs at temperature 0\. MAMMQA is only evaluated with LLM\-J\.
## Appendix HAblation Studies

This section presents two ablation studies on the MultiModalQA dataset to isolate the contributions of EDMA’s key components: entropy minimization and cross\-modal conditioning\.

### H\.1Ablation 1: Random Trajectory Selection

Table[10](https://arxiv.org/html/2610.27120#A8.T10)shows the performance impact of disabling entropy minimization during trajectory selection\. Instead of choosing the candidate trajectory that minimizes logical entropy, a trajectory is selected uniformly at random while keeping all other pipeline components identical\. To account for the stochasticity of random selection, results are averaged over 10 runs using the LLM\-Judge metric\. Across all evaluated models, selecting trajectories uniformly at random causes a severe performance drop \(−32\.2\-32\.2percentage points on average\) compared to entropy\-guided selection\. This shows that EDMA’s performance gains do not merely stem from task decomposition into intermediate partitions, but critically rely on selecting the trajectory that minimizes logical entropy\.

Table 10:Performance comparison between full EDMA and random trajectory selection\.
### H\.2Ablation 2: Removing Cross\-Modal Conditioning

To isolate the contribution of cross\-modal conditioning, we disable multimodal partitioning\. Specifically, we execute requirement extraction, unimodal partitioning, and entropy\-based trajectory selection, but omit the construction of multimodal partitions via cross\-modal conditioning\. Final answers are thus generated solely from the lowest\-entropy unimodal partition\.

Table 11:Performance comparison between full EDMA and unimodal\-only partitioning\.As shown in Table[11](https://arxiv.org/html/2610.27120#A8.T11), removing cross\-modal conditioning leads to an average performance drop of14\.414\.4percentage points across all models\. This drop is of the same order as the share of questions that require cross\-modal reasoning: approximately35%35\\%of questions in MultiModalQA strictly require it, whereas∼65%\\sim 65\\%can be answered using a single modality\. This is consistent with the framework failing to resolve cross\-modal dependencies without cross\-modal conditioning, and confirms that multimodal partitioning is fundamental for handling multimodal queries\.

## Appendix IStatistical significance

### I\.1EDMA vs PP

We establish the statistical reliability of the performance gains achieved by EDMA over PP by conducting paired statistical significance testing across all evaluated backbone architectures\.

Statistical significance is assessed using both a parametric test \(paired two\-tailedtt\-test\) and a non\-parametric test \(two\-tailed Wilcoxon signed\-rank test\) at a significance threshold ofα=0\.05\\alpha=0\.05\.

Table[12](https://arxiv.org/html/2610.27120#A9.T12)reports the statistical comparison between EDMA and PP across backbones\. EDMA achieves statistically significant improvements \(p<0\.05p<0\.05\) over PP across every model and metric evaluated under both parametric and non\-parametric tests, demonstrating robust gains across diverse model families\.

Table 12:Statistical comparison of EDMA against PP across model backbones\.Δ\\Deltarepresents the mean score difference \(EDMA−PP\\text\{EDMA\}\-\\text\{PP\}\)\. Tick \(✓\) indicates significance atp<0\.05p<0\.05\.
### I\.2Modality Fusion Decreases Entropy

We analyse the role of modality fusion in EDMA through the lens of logical entropy reduction over the candidate answer space\. Figure[5](https://arxiv.org/html/2610.27120#A9.F5)compares the distribution of entropy values before and after modality fusion, which occurs when cross\-modal evidence is extracted from one modality and used to condition the target modality\.

![Refer to caption](https://arxiv.org/html/2610.27120v1/entropy_change.png)Figure 5:Multimodal entropy distribution consistently shows lower values compared to unimodal entropy distributions\.Conditioning the target modality with complementary information from a secondary modality decreases logical entropy, thereby reducing uncertainty\. This shows that modality fusion is associated with lower entropy, mitigating uncertainty arising from noisy or ambiguous unimodal signals\. Table[13](https://arxiv.org/html/2610.27120#A9.T13)reports the statistical significance of these reductions across all directional modality pairs\.

Table 13:Statistical Evaluation of Entropy Reduction via Information Fusion\.H¯\\overline\{H\}represents mean entropy;ttrepresents the t\-statistic; Point\-Biserial \(PB\) is the correlation between a binary modality indicator \(unimodal vs\. multimodal\) and entropy values\. Negative PB values reflect lower entropy under multimodal representations\. Tick \(✓\) indicates significance atp<0\.05p<0\.05\.
### I\.3Higher Entropy Is Associated with More False Positives

To investigate the relationship between entropy and error rates, we compare entropy levels between correct answers \(True Positives, TP\) and incorrect answers \(False Positives, FP\)\. As detailed in Table[14](https://arxiv.org/html/2610.27120#A9.T14), false positives consistently exhibit higher mean entropy across all modalities\.

Table 14:Statistical comparison of entropy in True Positives \(TP\) vs\. False Positives \(FP\)\.H¯\\overline\{H\}denotes mean entropy;ttdenotes the t\-test statistic; PB denotes the point\-biserial correlation between entropy and a binary outcome variable \(TP = 0, FP = 1\)\. Tick \(✓\) indicates significance atp<0\.05p<0\.05\.We further assess the impact of entropy on False Discovery Rates \(FDR\) by grouping entropy into Low, Medium, and High level\. Table[15](https://arxiv.org/html/2610.27120#A9.T15)confirms significance via the Friedman test, while Table[16](https://arxiv.org/html/2610.27120#A9.T16)provides pairwise Wilcoxon signed\-rank comparisons across all entropy levels\.

Table 15:Statistical comparison across modalities using Mean False Discovery Rates \(FDR¯\\overline\{\\text\{FDR\}\}\) and the Friedman Test for Related Samples\.QQdenotes the Friedman test statistic\. Tick \(✓\) indicates significance atp<0\.05p<0\.05\.Table 16:Pairwise comparisons of Mean False Discovery Rates \(FDR¯\\overline\{\\text\{FDR\}\}\) across entropy levels within each modality using the Wilcoxon Signed\-Rank Test for paired samples\. Tick \(✓\) indicates significance atp<0\.05p<0\.05\.Table[17](https://arxiv.org/html/2610.27120#A9.T17)breaks down the False Discovery Rate \(FDR\) controlling for modality paths and candidate set size\. We split candidate set sizes \(ranging from 13 to 26\) into three buckets containing approximately 33% of the queries each\. Even after controlling for modality path and candidate set size,FDRincreases monotonically moving from low to medium and high entropy\.

Table 17:False Discovery Rate across modality paths, candidate set size buckets, and entropy levels\. FDR increases consistently as entropy increases from Low to Medium and High\.

## Appendix JImpact Assessment

The introduction of the EDMA methodology enhances the ability of LLMs to solve cross\-modal problems by guiding the answering process through entropy minimization and producing a quantifiable answer trajectory\. This traceability allows users to understand which modalities contributed most to the final answer, improving the interpretability and reliability of LLM predictions in real\-world scenarios\. By providing structured answering trajectories, EDMA also makes LLM\-based systems more robust and trustworthy\. While privacy and trustworthiness are general concerns associated with LLMs, our methodology does not introduce additional risks beyond those already inherent to LLMs\.

## Appendix KPrompts

### K\.1Prompts EDMA

#### K\.1\.1Prompt Modality Decision

Table 18:Prompt used to determine which modalities are required to answer a query\.
#### K\.1\.2Prompt Extraction of Answer Requirements

Table 19:Prompt used to extract structured answer requirements from natural\-language questions\.
#### K\.1\.3Prompt Checking Answer Requirement Paragraph

Table 20:Prompt to determine the presence of content overlap between a paragraph and an answer requirement\.
#### K\.1\.4Prompt Checking Answer Requirement Image

Table 21:Prompt to determine the presence of content overlap between an image and an answer requirement\.
#### K\.1\.5Prompt Checking Answer Requirement Table Row

Table 22:Prompt to determine the presence of content overlap between a table row and an answer requirement\.
#### K\.1\.6Prompt to Extract Information Text

Table 23:Prompt used for extracting question\-relevant evidence from a paragraph\.
#### K\.1\.7Prompt to Extract Information Image

Table 24:Prompt used for extracting question\-relevant evidence from an image\.
#### K\.1\.8Prompt to Extract Information Table Row

Table 25:Prompt used for extracting question\-relevant evidence from a table row\.
#### K\.1\.9Prompt to Check Mutual Information Between Paragraphs

Table 26:Prompt to determine the presence of mutual information between two paragraphs\.
#### K\.1\.10Prompt Check Mutual Information Between Textual Description and Image

Table 27:Prompt to check the presence of mutual information between an image and a paragraph\.
#### K\.1\.11Prompt to Check Mutual Information Between Textual Description and Table Row

Table 28:Prompt to check the presence of mutual information between a table row and a paragraph\.
#### K\.1\.12Check Answer in Paragraph

Table 29:Prompt to check whether a paragraph contains an entity that matches the answer type\.
#### K\.1\.13Check Answer in Image

Table 30:Prompt to check if the caption or the image contains an entity that matches the answer type\.
#### K\.1\.14Check Answer in Table Row

Table 31:Prompt to verify whether a table row contains a value that directly answers a question\.

### K\.2Baselines Prompts

#### K\.2\.1Prompt for Direct Prompting

Table 32:Direct prompting baseline used in our experiments\.
#### K\.2\.2Prompt for Chain\-of\-Thought

Table 33:Chain\-of\-Thought \(CoT\) prompt used in our experiments\.
#### K\.2\.3Prompt for Planning Prompting

Table 34:Planning\-based prompting baseline used in our experiments\.