LEDGERMIND: Provenance-Constrained Multimodal Agentic Reasoning with a Structured Evidence Ledger

Hugging Face Daily Papers Papers
multimodal agentic-reasoning provenance evidence-ledger visual-question-answering faithfulness reasoning

Summary

This paper introduces LedgerMind, a provenance-constrained multimodal agentic reasoning framework that uses a Structured Evidence Ledger to ensure grounded, faithful reasoning in visual question answering, addressing failure patterns like hallucination and over-reasoning.

Multimodal agents for visual question answering increasingly operate as multi-step trajectories that interleave perception, retrieval, and reasoning, yet evaluation still largely reduces to final-answer accuracy. This aggregate signal cannot tell whether a correct answer was reached through grounded evidence, language priors, or accidental error cancellation. We propose to treat a multimodal agent trajectory as a provenance-constrained state machine: tool outputs are normalized into a Structured Evidence Ledger that serves as the trajectory state, downstream reasoning and decision claims may cite only active ledger entries, grounding is checked at the entity and numeric level, and repair is realized as typed state transitions that cannot introduce content without tool-produced provenance. We instantiate this design as LedgerMind (Provenance-Constrained Multimodal Agentic Reasoning with a Structured Evidence Ledger), augmented by a Three-Layer Grounding Protocol, an Adaptive Dual-Path Dispatcher that matches reasoning depth to question complexity, and an Event-Triggered Verification-and-Repair engine with a formal provenance non-amplification guarantee. We use LedgerMind to target four recurring failure patterns that final-answer accuracy tends to obscure: unsupported intermediate reasoning, citation-backed entity hallucination (Phantom Grounding), over-reasoning on simple queries, and repair-time amplification. Experiments across multiple multimodal reasoning benchmarks and backbone MLLMs show that LedgerMind improves both answer accuracy and trajectory-level faithfulness.
Original Article
View Cached Full Text

Cached at: 07/31/26, 05:52 AM

Paper page - LEDGERMIND: Provenance-Constrained Multimodal Agentic Reasoning with a Structured Evidence Ledger

Source: https://huggingface.co/papers/2607.28374

Abstract

Multimodalagentsforvisualquestionansweringincreasinglyoperateasmulti-steptrajectoriesthatinterleaveperception,retrieval,andreasoning,yetevaluationstilllargelyreducestofinal-answeraccuracy.Thisaggregatesignalcannottellwhetheracorrectanswerwasreachedthroughgroundedevidence,languagepriors,oraccidentalerrorcancellation.Weproposetotreatamultimodalagenttrajectoryasaprovenance-constrainedstatemachine:tooloutputsarenormalizedintoaStructuredEvidenceLedgerthatservesasthetrajectorystate,downstreamreasoninganddecisionclaimsmayciteonlyactiveledgerentries,groundingischeckedattheentityandnumericlevel,andrepairisrealizedastypedstatetransitionsthatcannotintroducecontentwithouttool-producedprovenance.WeinstantiatethisdesignasLedgerMind(Provenance-ConstrainedMultimodalAgenticReasoningwithaStructuredEvidenceLedger),augmentedbyaThree-LayerGroundingProtocol,anAdaptiveDual-PathDispatcherthatmatchesreasoningdepthtoquestioncomplexity,andanEvent-TriggeredVerification-and-Repairenginewithaformalprovenancenon-amplificationguarantee.WeuseLedgerMindtotargetfourrecurringfailurepatternsthatfinal-answeraccuracytendstoobscure:unsupportedintermediatereasoning,citation-backedentityhallucination(PhantomGrounding),over-reasoningonsimplequeries,andrepair-timeamplification.ExperimentsacrossmultiplemultimodalreasoningbenchmarksandbackboneMLLMsshowthatLedgerMindimprovesbothansweraccuracyandtrajectory-levelfaithfulness.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.28374

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.28374 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.28374 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.28374 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Beacon: Knowing When and How to Perform Agentic Visual Reasoning

Hugging Face Daily Papers

This paper introduces Beacon, an agentic visual reasoning model that improves multimodal LLMs' ability to decide when to use tools and benefit from tool use, using Necessity-Aware Adaptive Reward and Hint-Guided Capability Expansion in reinforcement learning.