Paper page - AI, Take the Wheel: What Drives Delegation and Trust in Human-Computer Cooperative Question Answering?

Source: https://huggingface.co/papers/2605.28255

Abstract

Human-AI collaboration in question-answering tasks reveals suboptimal reliance decisions where humans under-rely on correct AI suggestions and over-rely when AI misleads them, with confirmation bias contributing to reduced trust in conflicting AI outputs.

AI systems are fallible, and humans can make mistakes in deciding whether totrustAI over their own judgment. Thus, improvinghuman-AI collaborationrequires understanding when, why, and how humans decide to rely on AI. We study two distinct reliance decisions: thedelegation choice-- deciding when to let AI act autonomously without knowing its output, and theadoption choice-- evaluating AI suggestions and deciding how to use them. Both of these decoupled reliance patterns shape collaboration, but prior work rarely studies them together in realistic settings with the same users. We address this gap by studying collaborative human--AI teams competing in aquestion-answering gamein which humans can choose when and how to work withAI agentsto win. Our 24 matches pair 23 expert humans with 16AI agents, capturing 387 delegation and 1440 adoption decisions. While human--AI collaboration performs better than either AI or humans alone, humans make suboptimal collaboration decisions, both under-relying on correct AI suggestions (3.9% of opportunities missed) and over-relying when AI misleads them (1.7%). Both parties contribute wrong answers: reported model confidence is near chance when humans and AI disagree, whileconfirmation biasdrives higher under-reliance (64.5%) when an AI suggestion agrees with humans’ initial incorrect answer. To close this gap, we recommendcalibrated confidence,evidence-grounded explanations, and mechanisms that help users refinetrust.

View arXiv pageView PDFAdd to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.28255 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.28255 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.28255 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.