When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs

# When Seeing Is Not Enough:Benchmarking Interactive Visual Grounding in LVLMs
Source: [https://arxiv.org/html/2608.23978](https://arxiv.org/html/2608.23978)
Zhengxiang WangAffiliation:Department of Linguistics & IACSAffiliation:Stony Brook UniversityEmail:[zhengxiang\.wang@stonybrook\.edu](mailto:)Owen RambowAffiliation:Department of Linguistics & IACSAffiliation:Stony Brook UniversityEmail:[owen\.rambow@stonybrook\.edu](mailto:)

###### Abstract

Visual grounding is typically evaluated as a one\-shot mapping from an informative referring expression to a visual target\. This formulation misses a central property of real\-world reference: target information is often incomplete, ambiguous, and established through interaction\. We introduce a controlled evaluation framework for interactive visual grounding in large vision\-language models \(LVLMs\), varying how much target information is provided upfront and how much must be acquired through dialogue\. Across four human\-grounded visual contexts and four interaction protocols, current LVLMs perform significantly below task\-level human baselines\. Interaction can help when follow\-up questions refine or repair an initial target description\. Performance is lowest when no initial description is provided and target information must be acquired through questions, indicating that proactive question\-driven grounding remains difficult\. LVLMs are also poorly calibrated, often reporting confidence that exceeds their empirical accuracy\. Follow\-up studies confirm these patterns across varied description sources \(human versus AI\), reasoning efforts, repeated interactions, description providers, and visual contexts\. Overall, interactive visual grounding remains an important challenge, requiring visual matching, information seeking and synthesis\.111GitHub project:[https://github\.com/jaaack\-wang/benchmarking\-interative\-visual\-grounding](https://github.com/jaaack-wang/benchmarking-interative-visual-grounding)\.

## 1Introduction

Visual grounding is a core vision\-language capability that requires a system to identify and localize regions or objects in a visual environment from a textual description[Xiao et al\. \(2026\)](https://arxiv.org/html/2608.23978#bib.bib26)\. It underlies applications such as visual question answering, visual retrieval, human–AI interaction, and embodied AI agents\. Enabled by large\-scale cross\-modal training and alignment[Radford et al\. \(2021\)](https://arxiv.org/html/2608.23978#bib.bib6);[Alayrac et al\. \(2022\)](https://arxiv.org/html/2608.23978#bib.bib7);[Liu et al\. \(2023\)](https://arxiv.org/html/2608.23978#bib.bib8), recent large vision\-language models \(LVLMs\) have demonstrated strong general\-purpose vision\-language capabilities[Wang et al\. \(2024\)](https://arxiv.org/html/2608.23978#bib.bib12);[OpenAI et al\. \(2024\)](https://arxiv.org/html/2608.23978#bib.bib9);[Gemini Team et al\. \(2025\)](https://arxiv.org/html/2608.23978#bib.bib10); in particular, they have largely saturated standard visual grounding benchmarks[Chen et al\. \(2024\)](https://arxiv.org/html/2608.23978#bib.bib11);[Dong et al\. \(2026\)](https://arxiv.org/html/2608.23978#bib.bib23)\.

![Refer to caption](https://arxiv.org/html/2608.23978v1/spiritFigure.png)Figure 1:Traditional visual grounding performs one\-shot matching, assuming an informative input description\. Interactive visual grounding removes this assumption, requiring the Matcher \(M\) to interact with the Director \(D\) to seek more information\. Examples are adapted from[Hawkins et al\. \(2020\)](https://arxiv.org/html/2608.23978#bib.bib36)to be illustrative\.However, traditional visual grounding \(Figure[1](https://arxiv.org/html/2608.23978#S1.F1)upper part\) relies on a restrictive assumption: the model is given a sufficiently informative referring expression before grounding begins\. Most benchmarks therefore cast grounding as a one\-shot, non\-interactive task, where the model maps an already available description to a visual target[Mao et al\. \(2016\)](https://arxiv.org/html/2608.23978#bib.bib27)\. This setup primarily evaluatesstatic visual matching\. It does not test whether a model can recognize that a description is incomplete, ask for missing information, or update its grounding decision as new evidence becomes available\.

This assumption departs from how reference works in real\-world human communication \(Figure[1](https://arxiv.org/html/2608.23978#S1.F1), lower part\)\. Referring expressions are frequently underspecified, ambiguous, or sufficient only relative to a particular visual context\. Speakers and listeners establish reference collaboratively, using clarification questions and feedback to build common ground over time[Clark and Wilkes\-Gibbs \(1986\)](https://arxiv.org/html/2608.23978#bib.bib4);[Brennan and Clark \(1996\)](https://arxiv.org/html/2608.23978#bib.bib5)\.

We therefore studyinteractive visual grounding, where target information may be ambiguous or underspecified at first and must be established through interaction\. We argue that interactive visual grounding requires two broad capabilities\. First, models needvisual matchingability: mapping a referring expression, or an accumulated target representation, to the intended visual item\. Second, models needinformation seeking and synthesisability: assessing whether the current information is sufficient, asking effective questions when it is not, and synthesizing answers across turns\. Traditional grounding benchmarks primarily test the former; interactive grounding requires both\.

Prior studies evaluate non\-interactive referring\-expression comprehension or object matching[Tang et al\. \(2024\)](https://arxiv.org/html/2608.23978#bib.bib18);[Wang et al\. \(2025a\)](https://arxiv.org/html/2608.23978#bib.bib37), interactive reference games, and human–AI communication[Hua and Artzi \(2024\)](https://arxiv.org/html/2608.23978#bib.bib19);[Imai et al\. \(2025\)](https://arxiv.org/html/2608.23978#bib.bib20);[Zeng et al\. \(2026\)](https://arxiv.org/html/2608.23978#bib.bib38);[Jones et al\. \(2026\)](https://arxiv.org/html/2608.23978#bib.bib21)\. They reveal important limitations but do not systematically vary whether target information is supplied or must be acquired, leaving visual\-matching failures difficult to distinguish from information\-seeking failures\.

To address this gap, we introduce a controlled evaluation framework for interactive visual grounding\. Our framework varies how much target information is available upfront and how much must be acquired through dialogue, thereby testing whether models can ground informative descriptions, use optional clarification, recover from underspecification, and construct target representations through their own questions\. Across 38,848 target selections from eight LVLMs under diverse experimental conditions, current models perform significantly below task\-level human baselines even when additional target information is available\. They can use follow\-up questions to refine or repair an initial description, but struggle to ground targets through proactive question\-driven dialogue despite asking the most questions\. Their reported confidence also frequently exceeds empirical accuracy\. Together, these findings show that interactive visual grounding remains an important challenge for LVLMs beyond static visual matching\.

## 2Related Work

#### Static Visual Grounding

Most visual grounding benchmarks evaluate grounding as a static identification or localization problem: given an image and a referring expression, a model selects the corresponding region or object[Lin et al\. \(2014\)](https://arxiv.org/html/2608.23978#bib.bib28);[Kazemzadeh et al\. \(2014\)](https://arxiv.org/html/2608.23978#bib.bib29);[Plummer et al\. \(2015\)](https://arxiv.org/html/2608.23978#bib.bib30);[Yu et al\. \(2016\)](https://arxiv.org/html/2608.23978#bib.bib31);[Chen et al\. \(2024\)](https://arxiv.org/html/2608.23978#bib.bib11);[Dong et al\. \(2026\)](https://arxiv.org/html/2608.23978#bib.bib23)\. This formulation has been central to measuring visual\-language alignment, but it assumes that the referring expression is already sufficiently informative\. As a result, it primarily tests visual matching, rather than whether a model can recognize missing information or seek clarification\.

#### Interactive Visual Grounding

Prior work has introduced interaction into visual grounding through goal\-oriented visual dialogue and clarification\-based object identification\. GuessWhat?\! frames object identification as a single\-target game in which a questioner asks yes/no questions to identify an object[de Vries et al\. \(2017\)](https://arxiv.org/html/2608.23978#bib.bib32)\. INGRESS studies situated referring\-expression grounding, where a robot asks clarification questions when a referent is ambiguous[Shridhar et al\. \(2020\)](https://arxiv.org/html/2608.23978#bib.bib34)\. InViG benchmarks open\-ended multi\-turn grounding for object disambiguation[Zhang et al\. \(2023\)](https://arxiv.org/html/2608.23978#bib.bib33)\. These settings show that interaction is important for grounding, but they typically focus on single\-target identification or constrained question\-answering\. In contrast, we study multi\-target grounding under controlled information conditions, allowing us to distinguish visual matching from information\-seeking ability\.

![Refer to caption](https://arxiv.org/html/2608.23978v1/methodology_fiugure.png)Figure 2:Overview of our proposed framework for controlled interactive visual grounding evaluation\. \(A\) and \(C\) show the visual contexts and metrics used in our main experiments, respectively; \(B\) summarizes the interaction protocols and grounding process\. The “Others” entries in \(A\) and \(C\) highlight the framework’s extensibility\.
#### LVLMs in Reference Games

Recent studies have evaluated LVLMs in repeated, multi\-target reference games, building on classic work on referential communication and common ground[Krauss and Glucksberg \(1969\)](https://arxiv.org/html/2608.23978#bib.bib35);[Clark and Wilkes\-Gibbs \(1986\)](https://arxiv.org/html/2608.23978#bib.bib4);[Schober and Clark \(1989\)](https://arxiv.org/html/2608.23978#bib.bib3)\. Some work examines whether LVLMs can comprehend referring expressions in human\-human dialogues that become more efficient over repeated interaction[Wang et al\. \(2025a\)](https://arxiv.org/html/2608.23978#bib.bib37)\. Other work studies LVLM behavior in self\-play[Hua and Artzi \(2024\)](https://arxiv.org/html/2608.23978#bib.bib19);[Imai et al\. \(2025\)](https://arxiv.org/html/2608.23978#bib.bib20)or human–AI interaction[Zeng et al\. \(2026\)](https://arxiv.org/html/2608.23978#bib.bib38);[Jones et al\. \(2026\)](https://arxiv.org/html/2608.23978#bib.bib21), revealing non\-human\-like patterns in both generation and comprehension\. Our work differs by systematically controlling how much target information is provided upfront and how much must be acquired through dialogue\. This lets us test whether model failures arise from visual matching or information seeking or their combination\.

#### Multi\-Turn Evaluation

Multi\-turn evaluation has been increasingly used to evaluate LLMs beyond isolated single\-turn prompts[Zheng et al\. \(2023\)](https://arxiv.org/html/2608.23978#bib.bib42);[Bai et al\. \(2024\)](https://arxiv.org/html/2608.23978#bib.bib40);[Kwan et al\. \(2024\)](https://arxiv.org/html/2608.23978#bib.bib41);[Wang et al\. \(2025b\)](https://arxiv.org/html/2608.23978#bib.bib47);[Han et al\. \(2025\)](https://arxiv.org/html/2608.23978#bib.bib43)\. Recent work shows that LLMs can degrade across turns and often fail to seek clarification, causing misunderstandings to compound over interaction[Laban et al\. \(2026\)](https://arxiv.org/html/2608.23978#bib.bib48);[Shaikh et al\. \(2025\)](https://arxiv.org/html/2608.23978#bib.bib22)\. Multimodal multi\-turn evaluation extends these challenges to LVLMs, where models must track visual inputs, dialogue history, and evolving user intent[Liu et al\. \(2024\)](https://arxiv.org/html/2608.23978#bib.bib45);[Lee et al\. \(2025\)](https://arxiv.org/html/2608.23978#bib.bib44);[Xue et al\. \(2025\)](https://arxiv.org/html/2608.23978#bib.bib46);[Feng et al\. \(2023\)](https://arxiv.org/html/2608.23978#bib.bib24);[Tian et al\. \(2025\)](https://arxiv.org/html/2608.23978#bib.bib25)\. Our work instead tests whether models can use dialogue to determine and acquire missing target information and improve grounding accuracy\.

## 3Interactive Visual Grounding

Figure[2](https://arxiv.org/html/2608.23978#S2.F2)visualizes our proposed evaluation framework, connecting the visual contexts \(Panel A\), multi\-target visual grounding \(Panel B\), and evaluation metrics \(Panel C\)\. This section details the core designs for the overall task, the interaction protocols, and the evaluation metrics\.

#### Task Formalization

A*Director*sees an ordered target setD=\(d1,…,dk\)D=\(d\_\{1\},\\ldots,d\_\{k\}\); a*Matcher*sees a separately indexed candidate setM=\(m1,…,mn\)M=\(m\_\{1\},\\ldots,m\_\{n\}\)containing the targets and possibly additional distractors\. The players do not see each other’s view, and indices are randomized independently\.

For each target positionjj, the players exchange utterances until the Matcher selectsd^j∈M\\hat\{d\}\_\{j\}\\in M\. The Matcher may first ask clarification questions and may later revise a selection\. The final output is the ordered sequenceD^=\(d^1,…,d^k\)\\hat\{D\}=\(\\hat\{d\}\_\{1\},\\ldots,\\hat\{d\}\_\{k\}\)\. Thus, the task adds clarification, cross\-turn integration, and revision to one\-shot matching under partial observability[Clark and Wilkes\-Gibbs \(1986\)](https://arxiv.org/html/2608.23978#bib.bib4)\.

Table 1:Dataset statistics and key properties of the four visual contexts used in our interactive visual grounding experiments\. Source: W =[Wang et al\. \(2025a\)](https://arxiv.org/html/2608.23978#bib.bib37); Z =[Zeng et al\. \(2026\)](https://arxiv.org/html/2608.23978#bib.bib38); and H =[Hawkins et al\. \(2020\)](https://arxiv.org/html/2608.23978#bib.bib36)\. In the source human tasks, free\-form matching permitted revisions, whereas sequential matching proceeded one target at a time without revision; our simulated interaction rules are described in Section[2](https://arxiv.org/html/2608.23978#S2.F2)\.
#### Interaction Protocols

We instantiate four protocols that vary how much target information is provided upfront and how much the Matcher must acquire through interaction\.

InStatic, the Matcher receives one human\-produced target description at a time and selects without interaction\. “Full” means that the description comes from a high\-accuracy human dialogue, not that it exhaustively specifies the target or was individually matched correctly\.Fullprovides the same description but permits optional follow\-up questions, testing whether clarification can verify, refine, or repair an initially informative input\.Partialinstead begins with an underspecified attribute, requiring the Matcher to seek missing information\.Noneprovides no initial description; the Director returns only yes/no/unclear answers, so the Matcher must decide what to request and construct a target representation through questions[de Vries et al\. \(2017\)](https://arxiv.org/html/2608.23978#bib.bib32)\. Together, the protocols progress from matching an informative description \(Static\) to optional clarification \(Full\), recovery from underspecification \(Partial\), and fully question\-driven grounding \(None\)\.

A description drawn from a high\-performing human dialogue may still leave an LVLM or even another human uncertain or incorrect, motivating optional interaction even in theFullcondition\.

#### Evaluation Metrics

We primarily consideraccuracyandquestion rateas our main metrics, reflecting models’ visual matching ability and tendency to seek information in interactive settings\. Accuracy is the percentage of correctly matched items within a task round\. Question rate is the average number of questions asked per item; by definition, it is 0 underStatic\. As a baseline, random accuracy is1/n1/nfor a Matcher view withnncandidates\.

We also analyzecalibrationto assess whether models know when they have enough information to make a selection\. Each Matcher is prompted to report a confidence score with its final selection, and we compare these stated confidence scores against empirical accuracy using calibration plots\.222We also compute Expected Calibration Error \(ECE\), which is strongly correlated with accuracy in our setting \(Spearman’sρ=−0\.96\\rho=\-0\.96\), making it less informative as a separate measure of calibration\. See Appendix[B](https://arxiv.org/html/2608.23978#A2)for details\.

Finally, we considerself\-repairsas a secondary behavioral signal\. Self\-repairs occur when a Matcher revises an earlier selection after receiving additional information, indicating whether models can recognize and correct earlier grounding decisions during interaction\.

## 4Experimental Setup

### 4\.1Datasets

We evaluate interactive visual grounding across four human\-grounded, object\-level visual contexts \(Table[1](https://arxiv.org/html/2608.23978#S3.T1)\): dogs and baskets from[Wang et al\. \(2025a\)](https://arxiv.org/html/2608.23978#bib.bib37), baskets from[Zeng et al\. \(2026\)](https://arxiv.org/html/2608.23978#bib.bib38), and tangrams from[Hawkins et al\. \(2020\)](https://arxiv.org/html/2608.23978#bib.bib36)\. Appendix[H](https://arxiv.org/html/2608.23978#A8)provides more details on the four image sets, and Panel A of Figure[2](https://arxiv.org/html/2608.23978#S2.F2)shows representative examples\.

We select these datasets for two reasons\. First, they provide human\-to\-human reference data over the same visual objects, allowing us to simulate human Directors and compare model behavior against human\-grounded baselines\. Second, they cover several dimensions central to referential communication: real versus abstract objects, spoken versus text\-based references, free matching versus sequential target selection, and objects with versus without conventional names\. The less nameable contexts are particularly challenging: baskets require fine\-grained attribute descriptions, while tangrams often elicit highly variable descriptions across speakers\. We also include two basket image sets with different levels of initial\-round human performance: 80% in[Zeng et al\. \(2026\)](https://arxiv.org/html/2608.23978#bib.bib38)and 100% in[Wang et al\. \(2025a\)](https://arxiv.org/html/2608.23978#bib.bib37)\. Together, these image sets provide a diverse, human\-grounded testbed for evaluating LVLMs under varying degrees of visual ambiguity, nameability, and referential difficulty\.

All four datasets contain repeated\-round human reference data, where a round denotes one pass through the same image set by the same dyad\. By default, we use only the initial round, where referring expressions are more self\-contained and less dependent on dyad\-specific common ground\.

### 4\.2Referring Expression Processing

We reuse the referring expressions from the four datasets in Table[1](https://arxiv.org/html/2608.23978#S3.T1), which contain human\-produced descriptions for the same visual objects across multiple discourse participants\. This supports a realistic and diverse evaluation setting, given the open\-ended nature of referring expression generation\.

#### Filtering

To ensure that the retained descriptions are accurate and sufficiently informative, we keep only initial\-round referring expressions from dialogues in which human participants achieved at least 80% matching accuracy for[Zeng et al\. \(2026\)](https://arxiv.org/html/2608.23978#bib.bib38)and 90% for[Hawkins et al\. \(2020\)](https://arxiv.org/html/2608.23978#bib.bib36), balancing description quality against sample size\. For the two image sets from[Wang et al\. \(2025a\)](https://arxiv.org/html/2608.23978#bib.bib37), initial\-round accuracy is 100%, so we retain all dialogues\. Although item\-level success could provide a more fine\-grained filter, we use dialogue\-level accuracy because the referring expressions are drawn from complete interaction transcripts and may therefore depend on the broader dialogue context\.

We further control description quality by selecting only participants whose matching performance does not decrease across rounds within each dataset\. Since repeated reference games typically become easier as participants establish common ground, this filter removes dialogues with unstable engagement or inconsistent task behavior\. It also ensures that the selected dialogues are suitable for our multi\-round follow\-up studies, where we examine whether LVLMs can handle realistic long\-horizon interactive visual grounding\. This filtering yields 48 high\-quality human dialogues\.

#### Full Target Item Description Extraction

Following[Zeng et al\. \(2026\)](https://arxiv.org/html/2608.23978#bib.bib38), we use GPT\-5\.4[OpenAI \(2025\)](https://arxiv.org/html/2608.23978#bib.bib1)to automatically extract object\-level referring expressions for each image set in Table[1](https://arxiv.org/html/2608.23978#S3.T1), except for Tangrams, which already includes manually curated object\-level descriptions\. This process yields 536 diverse, high\-quality referring expressions across the four image sets, which we use inStaticandFull\.

#### Partial Target Item Description Extraction

To construct underspecified referring expressions, we prompt GPT\-5\.4 with the full descriptions for all target items in the same image set and ask it to extract the least informative attribute from each description\. We use these extracted attributes as the initial descriptions inPartial\. This LLM\-as\-a\-judge procedure is designed not to find optimal shortened descriptions, but to minimize initial informativeness so that the Matcher must decide whether clarification is needed\. We verify that the extracted attributes are lexically supported by the source full expressions \(see Appendix[C](https://arxiv.org/html/2608.23978#A3)\)\. Our experiments further validate this design: all models ask questions underPartial, indicating that the extracted attributes are underspecified enough to elicit clarification \(e\.g\., Figure[3](https://arxiv.org/html/2608.23978#S4.F3)in Section[5](https://arxiv.org/html/2608.23978#S5)\)\.

### 4\.3Director Simulation

Because our focus is the Matcher, we use GPT\-5\.4 as a fixed simulated Director\. This choice is motivated by prior evidence that GPT\-5\.2 reaches about 85% initial\-round accuracy with human Matchers[Zeng et al\. \(2026\)](https://arxiv.org/html/2608.23978#bib.bib38)\. Fixing the Director makes information access more comparable across Matchers and protocols; Section[6](https://arxiv.org/html/2608.23978#S6)tests sensitivity to an alternative Director, showing comparable results\.

The Director is always conditioned on the same human\-produced full target description, defined in Section[4\.2](https://arxiv.org/html/2608.23978#S4.SS2)\. InStatic, we programmatically present this preconfigured full description to the Matcher one target at a time\. InFullandPartial, the Matcher first receives a preconfigured full or underspecified description, respectively, and the GPT\-5\.4 Director is called only when the Matcher asks a clarification question\. InNone, no initial description is provided; instead, the Director answers the Matcher’s questions while being prompted with the corresponding full target description\. When a question asks about information not explicitly stated in the description, the prompt instructs the Director to answer based only on the provided description and visual input\. See Appendix[F](https://arxiv.org/html/2608.23978#A6)for the full Director prompts\.

![Refer to caption](https://arxiv.org/html/2608.23978v1/figures/overall_results.png)Figure 3:Overall results across the four image sets\. Top: average matcher accuracy\. Bottom: average questions per target item\. Colors denote the four interaction protocols; horizontal lines mark human and random baselines\. The human line is atask\-levelreference, but not a protocol\-matched baseline, since the interaction conditions differ\.
### 4\.4Matcher Simulation

#### Models

We evaluate LVLMs from different model families and scales to simulate matchers: \(1\) five proprietary LVLMs, namely GPT\-5\.4, GPT\-5\.4\-Mini, and GPT\-5\.5[OpenAI \(2026a\)](https://arxiv.org/html/2608.23978#bib.bib13);[OpenAI \(2026b\)](https://arxiv.org/html/2608.23978#bib.bib14), and Gemini\-3\.1\-Pro and Gemini\-3\.1\-Flash\-Lite[Google DeepMind \(2026\)](https://arxiv.org/html/2608.23978#bib.bib15); and \(2\) three open\-weight LVLMs, namely Gemma\-4\-31B\-IT[Google \(2026\)](https://arxiv.org/html/2608.23978#bib.bib16)and Qwen3\-VL\-8B/32B\-Instruct[Bai et al\. \(2025\)](https://arxiv.org/html/2608.23978#bib.bib17)\.

#### Prompting

We use protocol\-specific system prompts while keeping the task overview fixed for both the Director and Matcher across conditions\. In interactive settings, the Matcher returns a JSON object containing a natural\-language utterance and any selections it makes, with a 100\-point confidence score for each selection\. In the static setting, the Matcher returns a JSON selection with a confidence score\. See Appendix[F](https://arxiv.org/html/2608.23978#A6)for full prompts\.

#### Simulation Scale

Across the four datasets, we obtain 536 item–description pairs\. Under the four interaction protocols in the main experiments, each LVLM matcher makes 2,144 target selections based on information from the simulated Director\. Across eight LVLM matchers, this yields 17,152 selections\. Including the follow\-up studies in Section[6](https://arxiv.org/html/2608.23978#S6), which add another 21,696 selections, our experiments produce 38,848 total target selections for analysis\.

### 4\.5Multi\-Turn Dialogue Simulation

#### Visual Input Format

For each image set, the Director is shown only the target items, while the Matcher is shown the full candidate set, which may include distractors\. We concatenate the individual item images into a single indexed grid and provide the grid as one image input\. Director indices follow the item\-description order in the original dataset, while Matcher indices are randomized to prevent trivial index matching or data contamination\.

#### Turn Taking

The Director speaks first inStatic,Full, andPartial, whereas the Matcher speaks first by initiating a yes/no question inNone\. In the interactive protocols, Matchers are allowed—but not required—to ask clarification questions and to revise earlier selections when later information changes their decision\. Each turn contains one natural\-language utterance capped at 50 words, and a Matcher may ask as many questions as needed across the 10\-turn budget, which is empirically sufficient\. If the turn budget is exhausted, the Matcher is prompted to make a target selection\. The Director advances to the next target only after the Matcher selects\. The final prediction sequence is computed from the Matcher’s finalized selections\.

Table 2:Best paired mean LVLM–human matcher accuracy gap \(%\) across datasets, selected from the best\-performing interaction protocol for each model–dataset pair\. The All column is computed over all examples from the four datasets\. Darker red denotes larger deficits relative to human matchers, and bold marks the smallest gap within each dataset\. Significance is assessed with pairedtt\-tests:∗p≤\.05\{\}^\{\*\}p\\leq\.05,p∗⁣∗≤\.01\{\}^\{\*\*\}p\\leq\.01, and∗∗∗p≤\.001\{\}^\{\*\*\*\}p\\leq\.001\. See Appendix[D\.1](https://arxiv.org/html/2608.23978#A4.SS1)for protocol\-specific results\.Table 3:Micro\-averaged accuracy \(%\) by model and interaction protocol\. Subscripts report paired mean differences relative toStatic;greenindicates gains andredindicates losses\. The best\-performing model within each protocol isbolded, and the second\-best model isunderlined\. Significance is assessed with pairedtt\-tests:∗p≤\.05\{\}^\{\*\}p\\leq\.05,p∗⁣∗≤\.01\{\}^\{\*\*\}p\\leq\.01, and∗∗∗p≤\.001\{\}^\{\*\*\*\}p\\leq\.001\.

## 5Results

### 5\.1Interactive Visual Grounding Poses a Challenging Task

#### LVLM matchers significantly lag behind human matchers in interactive visual grounding\.

The upper panels of Figure[3](https://arxiv.org/html/2608.23978#S4.F3)show that nearly all tested LVLM matchers underperform the corresponding task\-level human baselines across the four image sets, with the largest gaps on the Tangram Image Set\. Table[2](https://arxiv.org/html/2608.23978#S4.T2)further shows that these gaps are often substantial and statistically significant under pairedtt\-tests, especially when results are pooled across all datasets\. The only case of human parity is Gemini\-3\.1\-Pro on Basket Image Set 2 underPartial, but no model outperforms the human baselines\. Moreover, the consistent gaps underStaticandFullsuggest that current LVLMs still struggle to comprehend descriptions drawn from high\-accuracy human dialogues[Tang et al\. \(2024\)](https://arxiv.org/html/2608.23978#bib.bib18);[Wang et al\. \(2025a\)](https://arxiv.org/html/2608.23978#bib.bib37);[Zeng et al\. \(2026\)](https://arxiv.org/html/2608.23978#bib.bib38)\.

#### The task is challenging because it requires both visual matching and effective information seeking\.

GPT\-5\.4\-Mini consistently asks the fewest questions\. It outperforms Qwen3\-VL\-8B underStaticandFull, but its advantage nearly disappears underPartialand reverses underNone\(Table[3](https://arxiv.org/html/2608.23978#S4.T3)\), where question asking matters more\. This suggests that limited question asking can hurt when the task requires acquiring missing information\. However, asking more questions alone is not sufficient\. Gemini\-3\.1\-Pro asks more questions than GPT\-5\.5, yet GPT\-5\.5 achieves higher overall accuracy underNone\. These contrasts suggest that success requires not only visual matching or more questions, but useful questions and effective integration of answers across turns\.

### 5\.2Stronger Models Perform Better, but Question\-Only Grounding Remains Hard

#### Model performance exhibits a clear hierarchy\.

Figure[3](https://arxiv.org/html/2608.23978#S4.F3)and Table[3](https://arxiv.org/html/2608.23978#S4.T3)show that larger models generally outperform their smaller counterparts from the same model family, suggesting benefits from scaling, and proprietary models tend to outperform open\-weight models\. Gemini\-3\.1\-Pro is strongest overall, achieving the best performance underStatic,Full, andPartial, with especially large gains on the Tangram Image Set\. GPT\-5\.5 is also competitive, particularly underNone, suggesting stronger question\-driven information seeking and integration capabilities\.

#### LVLMs can benefit from interaction, but question\-driven grounding remains challenging\.

Table[3](https://arxiv.org/html/2608.23978#S4.T3)shows thatFullimproves overStaticfor seven of the eight models, suggesting that optional follow\-up questions can help refine or repair initial descriptions\. UnderPartial, results are mixed: some models recover from underspecification, while others decline\. The clearest limitation appears underNone, where all models perform belowStaticdespite asking the most questions\. This suggests that current LVLMs struggle to construct and ground target representations through proactive question\-driven dialogue\.

![Refer to caption](https://arxiv.org/html/2608.23978v1/figures/protocol_level_confidence_calibration.png)Figure 4:Protocol\-level calibration\. Points show model\-specific confidence bins from 10 evenly spaced bins between 0 and 100; point size is proportional to bin count\. The dashed diagonal denotes perfect calibration\.

### 5\.3LVLMs Are Not Yet Calibrated for Interactive Visual Grounding

#### Current LVLMs are sensitive to information availability, but remain overconfident\.

All models ask more questions when less initial information is available:None\>\>Partial\>\>Full\>\>Static\(lower panels of Figure[3](https://arxiv.org/html/2608.23978#S4.F3)\)\. Figure[4](https://arxiv.org/html/2608.23978#S5.F4)shows a similar shift in confidence: models tend to report lower confidence in more interaction\-heavy protocols\. However, most points still fall below the diagonal, indicating that reported confidence often exceeds empirical accuracy\. This overconfidence is especially pronounced underPartialandNone, suggesting that LVLMs are less reliable at judging information sufficiency when the initial referring expression is underspecified or must be constructed through interaction\. Overall, this suggests that LVLMs adjust their behavior to the amount of available information, but their confidence does not yet reliably reflect whether they have enough evidence for accurate grounding\.

#### Self\-repairs are rare and inconsistently reliable\.

Revising a selection could indicate that a model recognizes and acts on uncertainty, but repairs occur only underFullandPartialand account for under 2% of matches\. Among the 49 observed repairs, 26 \(53\.1%\) correct a wrong selection, 5 \(10\.2%\) change a correct selection to a wrong one, and 18 \(36\.7%\) replace one wrong answer with another\. Outcomes are strongly model\-dependent: Gemini\-3\.1\-Pro corrects 13 of its 17 revisions \(76\.5%\), whereas GPT\-5\.4 corrects none of six and nevertheless raises its confidence by 11\.2 points on average\. Thus, models can occasionally exploit later evidence to repair an error, but revision remains too sparse and its confidence changes too inconsistent to serve as reliable error awareness\. Full model\- and protocol\-level results appear in Appendix[D\.2](https://arxiv.org/html/2608.23978#A4.SS2)\.

### 5\.4Summary

Overall, interactive visual grounding remains challenging for current LVLMs\. Models ask more questions when information is limited and can sometimes benefit from interaction, especially when follow\-up questions refine or repair an initial description\. However, they still lag behind human matchers, struggle with proactive question\-driven dialogue, and remain poorly calibrated for the task\.

## 6Follow\-Up Studies

![Refer to caption](https://arxiv.org/html/2608.23978v1/figures/overall_human_versus_ai_accuracy.png)Figure 5:Accuracy \(%\) on AI\-generated and human\-generated descriptions across models and interaction protocols, averaged over all four datasets\.Table 4:Accuracy \(%\) under different reasoning\-effort settings on tangrams\. Subscripts report differences relative to default no reasoning\.To test whether our conclusions depend on description source, test\-time reasoning, repeated\-reference structure, Director choice, or the original visual contexts, we conduct five follow\-up studies\. We \(i\) replace human descriptions with GPT\-5\.4\-generated descriptions across all Matchers and protocols; \(ii\) vary reasoning effort for GPT\-5\.4 and GPT\-5\.4\-Mini on tangrams; \(iii\) evaluate three LVLMs over four rounds of repeated reference; \(iv\) replace GPT\-5\.4 with Gemini\-3\.1\-Flash as the Director on Basket Set 2 and Tangrams; and \(v\) evaluate five LVLM Matchers \(i\.e\., GPT\-5\.4, GPT\-5\.4\-Mini, Gemini\-3\.1\-Flash, Gemma\-4\-31B\-IT, and Qwen3\-VL\-8B\) on six public face image sets with GPT\-5\.4\-generated descriptions[Holland et al\. \(2018\)](https://arxiv.org/html/2608.23978#bib.bib39), each containing 12 images of one individual with varied facial expressions\.

#### LVLMs perform better with AI\-generated descriptions, but result patterns remain comparable\.

Figure[5](https://arxiv.org/html/2608.23978#S6.F5)shows that LVLMs generally achieve higher accuracy with AI\-generated descriptions than with human\-generated descriptions, especially underStaticandFull\. This suggests that AI\-generated descriptions are easier for LVLMs to interpret, while human referring expressions pose a distinct challenge\. Importantly, however, the overall protocol\-level patterns remain similar:StaticandFullare strongest,Partialis more variable, andNoneremains most difficult\.

![Refer to caption](https://arxiv.org/html/2608.23978v1/figures/full_multi_round_accuracy_by_dataset_2x2.png)Figure 6:Accuracy \(%\) across rounds on all four image sets underFull\. See Appendix[E\.3](https://arxiv.org/html/2608.23978#A5.SS3)forStaticresults\.
#### Reasoning helps in interactive settings, but scaling plateaus quickly\.

Reasoning yields little benefit inStatic, but substantially improves accuracy inFull\(Table[4](https://arxiv.org/html/2608.23978#S6.T4)\)\. This suggests that stronger reasoning helps models gather and synthesize visual evidence through conversation\. However, gains largely plateau from medium to high reasoning, indicating that additional test\-time compute alone is not sufficient\. Reasoning also modestly improves calibration, with similar plateaus from medium to high effort, as discussed in Appendix[E\.2](https://arxiv.org/html/2608.23978#A5.SS2)\.

#### LVLMs can benefit from repeated interaction, but the gap to human matchers remains\.

Figure[6](https://arxiv.org/html/2608.23978#S6.F6)shows that models can improve over rounds underFull\. Results forStaticshow similar patterns and are reported in Appendix[E\.3](https://arxiv.org/html/2608.23978#A5.SS3)for space reasons\. This suggests that LVLMs can handle increasingly efficient referring expressions that emerge through entrainment\. However, these improvements are not consistent across datasets, and human matchers continue to outperform LVLMs\. While Gemini\-3\.1\-Pro approaches human baselines in some settings, long\-horizon interactive visual grounding remains a broad challenge for LVLMs\.

Table 5:Aggregate alternative\-Director \(A\) and face\-domain \(B\) results\. Panel A averages across Matchers, datasets, and description sources; Panel B across Matchers and six face sets\.
#### Matcher rankings are stable across Director models\.

Table[5](https://arxiv.org/html/2608.23978#S6.T5)A shows that replacing the GPT\-5\.4 Director with Gemini\-3\.1\-Flash reduces mean Matcher accuracy by only 1\.1 points inFull\(51\.5% to 50\.4%\), but by 3\.5 points inPartial\(48\.3% to 44\.8%\)\. The largerPartialdecrease is consistent with the Director carrying more responsibility when the initial description is underspecified\. Despite these absolute changes, Matcher rankings remain highly correlated across Directors \(Spearman’sρ=\.929\\rho=\.929forFulland\.976\.976forPartial\)\. Thus, interaction quality affects accuracy, especially under underspecification, but the central Matcher hierarchy is not an artifact of a single simulated Director \(Appendix[E\.4](https://arxiv.org/html/2608.23978#A5.SS4)\)\.

#### The protocol\-level pattern extends to fine\-grained face matching\.

Across six face sets with AI\-generated descriptions, Table[5](https://arxiv.org/html/2608.23978#S6.T5)B shows thatFullremains within 0\.9 points ofStatic\(47\.8% vs\. 48\.7%\), whereasPartialandNonefall 6\.8 and 18\.0 points belowStatic, respectively\. Thus, optional clarification largely preserves performance when an informative description is available, but interaction does not fully compensate as initial information is removed\. This graded pattern matches Figure[5](https://arxiv.org/html/2608.23978#S6.F5)and extends the main protocol\-level observation to a distinct fine\-grained domain, confirming the core finding in Section[5](https://arxiv.org/html/2608.23978#S5)\(Appendix[E\.5](https://arxiv.org/html/2608.23978#A5.SS5)\)\.

## 7Conclusion

We introduce a controlled framework for evaluating interactive visual grounding beyond one\-shot matching\. Across four human\-grounded visual contexts, current LVLMs perform significantly below task\-level human references\. Interaction can help when follow\-up questions refine or repair an initial description, but models consistently struggle with proactive question\-driven grounding\. They are also poorly calibrated, often reporting confidence that exceeds accuracy\. These findings are further validated across follow\-up studies using AI\-generated descriptions, test\-time scaling, exposure to repeated interaction, an alternative simulated Director, and a fine\-grained face context\. Overall, interactive visual grounding remains an important challenge for LVLMs, with potential implications for future embodied human–AI interaction\.

## Acknowledgments

This material is based upon work supported by the National Science Foundation \(NSF\) under Grant No\. 2125295 and by a seed grant from Stony Brook University\. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author\(s\) and do not necessarily reflect the views of the NSF\.

We are grateful for support from the Institute for Advanced Computational Science \(IACS\) at Stony Brook University\. We also thank the three anonymous reviewers for their valuable feedback\.

## Limitations

#### Simulated Directors and Director Dependence\.

We use simulated rather than human Directors for scale and experimental control\. RepeatingFullandPartialwith Gemini\-3\.1\-Flash produces highly stable Matcher rankings, reducing concern that the main conclusions are specific to GPT\-5\.4\. That said, this robustness check does not establish equivalence to human Directors, and individual model–protocol outcomes can still vary with the Director\.

#### Attribution\.

The four protocols vary how much information must be acquired interactively\. They are designed to create a controlled way to separate visual matching from information seeking and question\-driven grounding, but do not uniquely isolate question selection, answer integration, dialogue memory, and visual matching etc\. Our claim is also limited: these protocols increase the need for information seeking and synthesis beyond static visual matching, rather than attributing each failure to one precise subcomponent, which we leave to future studies\.

#### Visual Context Coverage\.

The main testbed contains four object\-level contexts, and the face follow\-up adds another fine\-grained within\-category setting\. Neither covers cluttered scenes, spatial or relational grounding, or open\-world visual search\. Our conclusions therefore concern proactive interactive grounding in controlled, fine\-grained matching contexts rather than visual grounding in all settings\.

#### Constrained Interaction underNone\.

The Director provides yes/no answers underNoneto create a controlled information\-seeking setting, following[de Vries et al\. \(2017\)](https://arxiv.org/html/2608.23978#bib.bib32)\. Open\-ended Director answers may support different questioning strategies and should be studied separately\.

## Ethical Considerations

#### Data Use and Privacy\.

This work evaluates LVLMs on interactive visual grounding using existing research datasets and public or previously released visual materials\. We do not collect new human\-subject data\. The face\-domain robustness study uses public research images only for within\-set visual matching; we neither infer identity nor predict sensitive attributes\. The main evaluated images involve object\-level visual contexts such as dogs, baskets, and abstract tangrams, which reduces risks related to privacy or demographic stereotyping\.

## References

- J\. Alayrac, J\. Donahue, P\. Luc, A\. Miech, I\. Barr, Y\. Hasson, K\. Lenc, A\. Mensch, K\. Millican, M\. Reynolds, R\. Ring, E\. Rutherford, S\. Cabi, T\. Han, Z\. Gong, S\. Samangooei, M\. Monteiro, J\. Menick, S\. Borgeaud, A\. Brock, A\. Nematzadeh, S\. Sharifzadeh, M\. Binkowski, R\. Barreira, O\. Vinyals, A\. Zisserman, and K\. SimonyanFlamingo: a visual language model for few\-shot learning\.InAdvances in Neural Information Processing Systems,A\. H\. Oh, A\. Agarwal, D\. Belgrave, and K\. Cho \(Eds\.\),External Links:[Link](https://openreview.net/forum?id=EbMuimAbPbs)Cited by:[§1](https://arxiv.org/html/2608.23978#S1.p1.1)\.
- Baiet al\.\(2024\)G\. Bai, J\. Liu, X\. Bu, Y\. He, J\. Liu, Z\. Zhou, Z\. Lin, W\. Su, T\. Ge, B\. Zheng, and W\. OuyangMT\-bench\-101: a fine\-grained benchmark for evaluating large language models in multi\-turn dialogues\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 7421–7454\.External Links:[Link](https://aclanthology.org/2024.acl-long.401/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.401)Cited by:[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px4.p1.1)\.
- Baiet al\.\(2025\)S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge, W\. Ge, Z\. Guo, Q\. Huang, J\. Huang, F\. Huang, B\. Hui, S\. Jiang, Z\. Li, M\. Li, M\. Li, K\. Li, Z\. Lin, J\. Lin, X\. Liu, J\. Liu, C\. Liu, Y\. Liu, D\. Liu, S\. Liu, D\. Lu, R\. Luo, C\. Lv, R\. Men, L\. Meng, X\. Ren, X\. Ren, S\. Song, Y\. Sun, J\. Tang, J\. Tu, J\. Wan, P\. Wang, P\. Wang, Q\. Wang, Y\. Wang, T\. Xie, Y\. Xu, H\. Xu, J\. Xu, Z\. Yang, M\. Yang, J\. Yang, A\. Yang, B\. Yu, F\. Zhang, H\. Zhang, X\. Zhang, B\. Zheng, H\. Zhong, J\. Zhou, F\. Zhou, J\. Zhou, Y\. Zhu, and K\. ZhuQwen3\-vl technical report\.External Links:2511\.21631,[Link](https://arxiv.org/abs/2511.21631)Cited by:[Table 6](https://arxiv.org/html/2608.23978#Ax1.T6.2.1.8.1),[Table 6](https://arxiv.org/html/2608.23978#Ax1.T6.2.1.9.1),[§4\.4](https://arxiv.org/html/2608.23978#S4.SS4.SSS0.Px1.p1.1)\.
- Brennan and Clark \(1996\)S\. E\. Brennan and H\. H\. ClarkConceptual pacts and lexical choice in conversation\.\.Journal of Experimental Psychology: Learning, Memory, and Cognition22\(6\),pp\. 1482–1493\.External Links:ISSN 0278\-7393,[Link](http://dx.doi.org/10.1037/0278-7393.22.6.1482),[Document](https://dx.doi.org/10.1037/0278-7393.22.6.1482)Cited by:[§1](https://arxiv.org/html/2608.23978#S1.p3.1)\.
- Chenet al\.\(2024\)J\. Chen, F\. Wei, J\. Zhao, S\. Song, B\. Wu, Z\. Peng, S\. \-H\. G\. Chan, and H\. ZhangRevisiting referring expression comprehension evaluation in the era of large multimodal models\.External Links:2406\.16866,[Link](https://arxiv.org/abs/2406.16866)Cited by:[§1](https://arxiv.org/html/2608.23978#S1.p1.1),[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px1.p1.1)\.
- Clark and Wilkes\-Gibbs \(1986\)H\. H\. Clark and D\. Wilkes\-GibbsReferring as a collaborative process\.Cognition22\(1\),pp\. 1–39\.External Links:ISSN 0010\-0277,[Link](http://dx.doi.org/10.1016/0010-0277(86)90010-7),[Document](https://dx.doi.org/10.1016/0010-0277%2886%2990010-7)Cited by:[§1](https://arxiv.org/html/2608.23978#S1.p3.1),[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px3.p1.1),[§3](https://arxiv.org/html/2608.23978#S3.SS0.SSS0.Px1.p2.1)\.
- de Vrieset al\.\(2017\)H\. de Vries, F\. Strub, S\. Chandar, O\. Pietquin, H\. Larochelle, and A\. CourvilleGuessWhat?\! visual object discovery through multi\-modal dialogue\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2608.23978#S3.SS0.SSS0.Px2.p2.1),[Constrained Interaction underNone\.](https://arxiv.org/html/2608.23978#Sx2.SS0.SSS0.Px4.p1.1)\.
- Donget al\.\(2026\)Q\. Dong, K\. Yang, L\. Ju, H\. Zhao, Y\. Zhang, Y\. Wang, H\. Zeng, J\. Lu, and Y\. FuRef\-adv: exploring MLLM visual reasoning in referring expression tasks\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=iEBgrepR9i)Cited by:[§1](https://arxiv.org/html/2608.23978#S1.p1.1),[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px1.p1.1)\.
- Fenget al\.\(2023\)J\. Feng, Q\. Sun, C\. Xu, P\. Zhao, Y\. Yang, C\. Tao, D\. Zhao, and Q\. LinMMDialog: a large\-scale multi\-turn dialogue dataset towards multi\-modal open\-domain conversation\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 7348–7363\.External Links:[Link](https://aclanthology.org/2023.acl-long.405/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.405)Cited by:[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px4.p1.1)\.
- Gemini Teamet al\.\(2025\)Gemini Team, R\. Anil, S\. Borgeaud, J\. Alayrac, J\. Yu, R\. Soricut, J\. Schalkwyk, A\. M\. Dai, A\. Hauth, K\. Millican, D\. Silver, M\. Johnson, I\. Antonoglou, J\. Schrittwieser, A\. Glaese, J\. Chen, E\. Pitler, T\. Lillicrap, and Angeliki Lazaridou and 1332 othersGemini: a family of highly capable multimodal models\.External Links:2312\.11805,[Link](https://arxiv.org/abs/2312.11805)Cited by:[§1](https://arxiv.org/html/2608.23978#S1.p1.1)\.
- Google DeepMind \(2026\)Google DeepMindGemini 3\.1 Pro model card\.Note:[https://deepmind\.google/models/model\-cards/gemini\-3\-1\-pro/](https://deepmind.google/models/model-cards/gemini-3-1-pro/)Accessed: 2026\-05\-19Cited by:[Table 6](https://arxiv.org/html/2608.23978#Ax1.T6.2.1.5.1),[Table 6](https://arxiv.org/html/2608.23978#Ax1.T6.2.1.6.1),[§4\.4](https://arxiv.org/html/2608.23978#S4.SS4.SSS0.Px1.p1.1)\.
- Google \(2026\)GoogleGemma 4 model overview\.Note:[https://ai\.google\.dev/gemma/docs/core/model\_card\_4](https://ai.google.dev/gemma/docs/core/model_card_4)Accessed: 2026\-05\-19Cited by:[Table 6](https://arxiv.org/html/2608.23978#Ax1.T6.2.1.7.1),[§4\.4](https://arxiv.org/html/2608.23978#S4.SS4.SSS0.Px1.p1.1)\.
- Hanet al\.\(2025\)C\. Han, X\. Liu, H\. Wang, S\. Li, J\. Yang, H\. Jiang, Z\. Wang, Q\. Yin, L\. Qiu, C\. Yu, Y\. Gao, Z\. Li, B\. Yin, J\. Shang, and H\. JiCan language models follow multiple turns of entangled instructions?\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 25445–25460\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.1387/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.1387),ISBN 979\-8\-89176\-335\-7Cited by:[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px4.p1.1)\.
- Hawkinset al\.\(2020\)R\. D\. Hawkins, M\. C\. Frank, and N\. D\. GoodmanCharacterizing the dynamics of learning in repeated reference games\.Cognitive Science44\(6\)\.External Links:ISSN 1551\-6709,[Link](http://dx.doi.org/10.1111/cogs.12845),[Document](https://dx.doi.org/10.1111/cogs.12845)Cited by:[Figure 17](https://arxiv.org/html/2608.23978#A8.F17),[Figure 1](https://arxiv.org/html/2608.23978#S1.F1),[Table 1](https://arxiv.org/html/2608.23978#S3.T1),[§4\.1](https://arxiv.org/html/2608.23978#S4.SS1.p1.1),[§4\.2](https://arxiv.org/html/2608.23978#S4.SS2.SSS0.Px1.p1.1)\.
- Hollandet al\.\(2018\)C\. A\. C\. Holland, N\. C\. Ebner, T\. Lin, and G\. R\. Samanez\-LarkinEmotion identification across adulthood using the dynamic faces database of emotional expressions in younger, middle aged, and older adults\.Cognition and Emotion33\(2\),pp\. 245–257\.External Links:ISSN 1464\-0600,[Link](http://dx.doi.org/10.1080/02699931.2018.1445981),[Document](https://dx.doi.org/10.1080/02699931.2018.1445981)Cited by:[§E\.5](https://arxiv.org/html/2608.23978#A5.SS5.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2608.23978#S6.p1.1)\.
- Hua and Artzi \(2024\)Y\. Hua and Y\. ArtziTalk less, interact better: evaluating in\-context conversational adaptation in multimodal LLMs\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=lVOw78nYXS)Cited by:[§1](https://arxiv.org/html/2608.23978#S1.p5.1),[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px3.p1.1)\.
- Imaiet al\.\(2025\)S\. Imai, M\. Inan, A\. B\. Sicilia, and M\. AlikhaniMeasuring how \(not just whether\) VLMs build common ground\.InProceedings of the 15th International Conference on Recent Advances in Natural Language Processing \- Natural Language Processing in the Generative AI Era,G\. Angelova, M\. Kunilovskaya, M\. Escribe, and R\. Mitkov \(Eds\.\),Varna, Bulgaria,pp\. 441–451\.External Links:[Link](https://aclanthology.org/2025.ranlp-1.53/)Cited by:[§1](https://arxiv.org/html/2608.23978#S1.p5.1),[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px3.p1.1)\.
- Joneset al\.\(2026\)C\. R\. Jones, A\. Lombardi, K\. Mahowald, and B\. K\. BergenLLMs and people both learn to form conventions – just not with each other\.External Links:2602\.08208,[Link](https://arxiv.org/abs/2602.08208)Cited by:[§1](https://arxiv.org/html/2608.23978#S1.p5.1),[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px3.p1.1)\.
- Kazemzadehet al\.\(2014\)S\. Kazemzadeh, V\. Ordonez, M\. Matten, and T\. BergReferItGame: referring to objects in photographs of natural scenes\.InProceedings of the 2014 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),A\. Moschitti, B\. Pang, and W\. Daelemans \(Eds\.\),Doha, Qatar,pp\. 787–798\.External Links:[Link](https://aclanthology.org/D14-1086/),[Document](https://dx.doi.org/10.3115/v1/D14-1086)Cited by:[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px1.p1.1)\.
- Krauss and Glucksberg \(1969\)R\. M\. Krauss and S\. GlucksbergThe development of communication: competence as a function of age\.Child Development40\(1\),pp\. 255\.External Links:ISSN 0009\-3920,[Link](http://dx.doi.org/10.2307/1127172),[Document](https://dx.doi.org/10.2307/1127172)Cited by:[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px3.p1.1)\.
- Kwanet al\.\(2024\)W\. Kwan, X\. Zeng, Y\. Jiang, Y\. Wang, L\. Li, L\. Shang, X\. Jiang, Q\. Liu, and K\. WongMT\-eval: a multi\-turn capabilities evaluation benchmark for large language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 20153–20177\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.1124/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.1124)Cited by:[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px4.p1.1)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. StoicaEfficient memory management for large language model serving with pagedattention\.InProceedings of the 29th Symposium on Operating Systems Principles,SOSP ’23,New York, NY, USA,pp\. 611–626\.External Links:ISBN 9798400702297,[Link](https://doi.org/10.1145/3600006.3613165),[Document](https://dx.doi.org/10.1145/3600006.3613165)Cited by:[Appendix A](https://arxiv.org/html/2608.23978#A1.p1.1)\.
- Labanet al\.\(2026\)P\. Laban, H\. Hayashi, Y\. Zhou, and J\. NevilleLLMs get lost in multi\-turn conversation\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=VKGTGGcwl6)Cited by:[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px4.p1.1)\.
- Leeet al\.\(2025\)Y\. Lee, B\. Lee, J\. Zhang, Y\. Hwang, B\. Ko, H\. Kim, D\. Yao, X\. Rong, E\. Joo, S\. Han, B\. Ko, and H\. ChoiMultiVerse: a multi\-turn conversation benchmark for evaluating large vision and language models\.External Links:2510\.16641,[Link](https://arxiv.org/abs/2510.16641)Cited by:[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px4.p1.1)\.
- Linet al\.\(2014\)T\. Lin, M\. Maire, S\. Belongie, J\. Hays, P\. Perona, D\. Ramanan, P\. Dollár, and C\. L\. ZitnickMicrosoft coco: common objects in context\.InComputer Vision – ECCV 2014,pp\. 740–755\.External Links:ISBN 9783319106021,ISSN 1611\-3349,[Link](http://dx.doi.org/10.1007/978-3-319-10602-1_48),[Document](https://dx.doi.org/10.1007/978-3-319-10602-1%5F48)Cited by:[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2023\)H\. Liu, C\. Li, Q\. Wu, and Y\. J\. LeeVisual instruction tuning\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 34892–34916\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/6dcf277ea32ce3288914faf369fe6de0-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2608.23978#S1.p1.1)\.
- Liuet al\.\(2024\)Z\. Liu, T\. Chu, Y\. Zang, X\. Wei, X\. Dong, P\. Zhang, Z\. Liang, Y\. Xiong, Y\. Qiao, D\. Lin, and J\. WangMMDU: a multi\-turn multi\-image dialog understanding benchmark and instruction\-tuning dataset for LVLMs\.InThe Thirty\-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=s8h2jSN6a6)Cited by:[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px4.p1.1)\.
- Maoet al\.\(2016\)J\. Mao, J\. Huang, A\. Toshev, O\. Camburu, A\. Yuille, and K\. MurphyGeneration and comprehension of unambiguous object descriptions\.In2016 IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),Vol\.,pp\. 11–20\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2016.9)Cited by:[§1](https://arxiv.org/html/2608.23978#S1.p2.1)\.
- OpenAIet al\.\(2024\)OpenAI, J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat, R\. Avila, I\. Babuschkin, S\. Balaji, V\. Balcom, P\. Baltescu, H\. Bao, M\. Bavarian, and Jeff Belgum and 262 othersGPT\-4 technical report\.External Links:2303\.08774,[Link](https://arxiv.org/abs/2303.08774)Cited by:[§1](https://arxiv.org/html/2608.23978#S1.p1.1)\.
- OpenAI \(2025\)OpenAIGPT\-5 is here\.Note:[https://openai\.com/gpt\-5/](https://openai.com/gpt-5/)Accessed: 2026\-05\-20Cited by:[§4\.2](https://arxiv.org/html/2608.23978#S4.SS2.SSS0.Px2.p1.1)\.
- OpenAI \(2026a\)OpenAIIntroducing GPT\-5\.4\.Note:[https://openai\.com/index/introducing\-gpt\-5\-4/](https://openai.com/index/introducing-gpt-5-4/)Accessed: 2026\-05\-19Cited by:[Table 6](https://arxiv.org/html/2608.23978#Ax1.T6.2.1.3.1),[Table 6](https://arxiv.org/html/2608.23978#Ax1.T6.2.1.4.1),[§4\.4](https://arxiv.org/html/2608.23978#S4.SS4.SSS0.Px1.p1.1)\.
- OpenAI \(2026b\)OpenAIIntroducing GPT\-5\.5\.Note:[https://openai\.com/index/introducing\-gpt\-5\-5/](https://openai.com/index/introducing-gpt-5-5/)Accessed: 2026\-05\-19Cited by:[Table 6](https://arxiv.org/html/2608.23978#Ax1.T6.2.1.2.1),[§4\.4](https://arxiv.org/html/2608.23978#S4.SS4.SSS0.Px1.p1.1)\.
- Plummeret al\.\(2015\)B\. A\. Plummer, L\. Wang, C\. M\. Cervantes, J\. C\. Caicedo, J\. Hockenmaier, and S\. LazebnikFlickr30k entities: collecting region\-to\-phrase correspondences for richer image\-to\-sentence models\.In2015 IEEE International Conference on Computer Vision \(ICCV\),Vol\.,pp\. 2641–2649\.External Links:[Document](https://dx.doi.org/10.1109/ICCV.2015.303)Cited by:[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px1.p1.1)\.
- Radfordet al\.\(2021\)A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark, G\. Krueger, and I\. SutskeverLearning transferable visual models from natural language supervision\.InProceedings of the 38th International Conference on Machine Learning,M\. Meila and T\. Zhang \(Eds\.\),Proceedings of Machine Learning Research, Vol\.139,pp\. 8748–8763\.External Links:[Link](https://proceedings.mlr.press/v139/radford21a.html)Cited by:[§1](https://arxiv.org/html/2608.23978#S1.p1.1)\.
- Schober and Clark \(1989\)M\. F\. Schober and H\. H\. ClarkUnderstanding by addressees and overhearers\.Cognitive Psychology21\(2\),pp\. 211–232\.External Links:ISSN 0010\-0285,[Link](http://dx.doi.org/10.1016/0010-0285(89)90008-X),[Document](https://dx.doi.org/10.1016/0010-0285%2889%2990008-x)Cited by:[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px3.p1.1)\.
- Shaikhet al\.\(2025\)O\. Shaikh, H\. Mozannar, G\. Bansal, A\. Fourney, and E\. HorvitzNavigating rifts in human\-LLM grounding: study and benchmark\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 20832–20847\.External Links:[Link](https://aclanthology.org/2025.acl-long.1016/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1016),ISBN 979\-8\-89176\-251\-0Cited by:[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px4.p1.1)\.
- Shridharet al\.\(2020\)M\. Shridhar, D\. Mittal, and D\. HsuINGRESS: interactive visual grounding of referring expressions\.Int\. J\. Rob\. Res\.39\(2–3\),pp\. 217–232\.External Links:ISSN 0278\-3649,[Link](https://doi.org/10.1177/0278364919897133),[Document](https://dx.doi.org/10.1177/0278364919897133)Cited by:[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px2.p1.1)\.
- Tanget al\.\(2024\)Z\. Tang, L\. Mao, and A\. SuhrGrounding language in multi\-perspective referential communication\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 19727–19741\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.1100/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.1100)Cited by:[§1](https://arxiv.org/html/2608.23978#S1.p5.1),[§5\.1](https://arxiv.org/html/2608.23978#S5.SS1.SSS0.Px1.p1.1)\.
- Tianet al\.\(2025\)Y\. Tian, T\. Ma, L\. Xie, and Q\. YeChatterBox: multimodal referring and grounding with chain\-of\-questions\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 7401–7409\.Cited by:[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px4.p1.1)\.
- Wanget al\.\(2024\)W\. Wang, Q\. Lv, W\. Yu, W\. Hong, J\. Qi, Y\. Wang, J\. Ji, Z\. Yang, L\. Zhao, X\. Song, J\. Xu, K\. Chen, B\. Xu, J\. Li, Y\. Dong, M\. Ding, and J\. TangCogVLM: visual expert for pretrained language models\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 121475–121499\.External Links:[Document](https://dx.doi.org/10.52202/079017-3860),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/dc06d4d2792265fb5454a6092bfd5c6a-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2608.23978#S1.p1.1)\.
- Wanget al\.\(2025a\)Z\. Wang, W\. Li, P\. Kaliosis, O\. Rambow, and S\. BrennanLVLMs are bad at overhearing human referential communication\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 16758–16782\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.849/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.849),ISBN 979\-8\-89176\-332\-6Cited by:[Figure 14](https://arxiv.org/html/2608.23978#A8.F14),[Figure 15](https://arxiv.org/html/2608.23978#A8.F15),[§1](https://arxiv.org/html/2608.23978#S1.p5.1),[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px3.p1.1),[Table 1](https://arxiv.org/html/2608.23978#S3.T1),[§4\.1](https://arxiv.org/html/2608.23978#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2608.23978#S4.SS1.p2.1),[§4\.2](https://arxiv.org/html/2608.23978#S4.SS2.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2608.23978#S5.SS1.SSS0.Px1.p1.1)\.
- Wanget al\.\(2025b\)Z\. Wang, V\. Makarova, Z\. Li, J\. Kodner, and O\. RambowLLMs can perform multi\-dimensional analytic writing assessments: a case study of L2 graduate\-level academic English writing\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 8637–8663\.External Links:[Link](https://aclanthology.org/2025.acl-long.423/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.423),ISBN 979\-8\-89176\-251\-0Cited by:[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px4.p1.1)\.
- Xiaoet al\.\(2026\)L\. Xiao, X\. Yang, X\. Lan, Y\. Wang, and C\. XuToward visual grounding: a survey\.IEEE Transactions on Pattern Analysis and Machine Intelligence48\(3\),pp\. 2749–2771\.External Links:[Document](https://dx.doi.org/10.1109/TPAMI.2025.3630635)Cited by:[§1](https://arxiv.org/html/2608.23978#S1.p1.1)\.
- Xueet al\.\(2025\)H\. Xue, F\. Tang, M\. Hu, Y\. Liu, Q\. Huang, Y\. Li, C\. Liu, Z\. Xu, C\. Zhang, C\. Feng, Y\. Xie, I\. Razzak, Z\. Ge, J\. Su, J\. He, and Y\. QiaoMMRC: a large\-scale benchmark for understanding multimodal large language model in real\-world conversation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 22477–22503\.External Links:[Link](https://aclanthology.org/2025.acl-long.1096/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1096),ISBN 979\-8\-89176\-251\-0Cited by:[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px4.p1.1)\.
- Yuet al\.\(2016\)L\. Yu, P\. Poirson, S\. Yang, A\. C\. Berg, and T\. L\. BergModeling context in referring expressions\.External Links:1608\.00272,[Link](https://arxiv.org/abs/1608.00272)Cited by:[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px1.p1.1)\.
- Zenget al\.\(2026\)P\. Zeng, W\. Li, A\. Paige, Z\. Wang, P\. Kaliosis, D\. Samaras, G\. Zelinsky, S\. Brennan, and O\. RambowLVLMs and humans ground differently in referential communication\.External Links:2601\.19792,[Link](https://arxiv.org/abs/2601.19792)Cited by:[§G\.1](https://arxiv.org/html/2608.23978#A7.SS1.p2.1),[Figure 16](https://arxiv.org/html/2608.23978#A8.F16),[§1](https://arxiv.org/html/2608.23978#S1.p5.1),[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px3.p1.1),[Table 1](https://arxiv.org/html/2608.23978#S3.T1),[§4\.1](https://arxiv.org/html/2608.23978#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2608.23978#S4.SS1.p2.1),[§4\.2](https://arxiv.org/html/2608.23978#S4.SS2.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2608.23978#S4.SS2.SSS0.Px2.p1.1),[§4\.3](https://arxiv.org/html/2608.23978#S4.SS3.p1.1),[§5\.1](https://arxiv.org/html/2608.23978#S5.SS1.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2023\)H\. Zhang, J\. Xu, Y\. Mo, and T\. KongInViG: benchmarking interactive visual grounding with 500k human\-robot interactions\.External Links:2310\.12147,[Link](https://arxiv.org/abs/2310.12147)Cited by:[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px2.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-judge with MT\-bench and chatbot arena\.InThirty\-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=uccHPGDlao)Cited by:[§2](https://arxiv.org/html/2608.23978#S2.SS0.SSS0.Px4.p1.1)\.

## Appendix Table of Contents

Table 6:LVLM variants used in our experiments\. API models are identified by their provider\-facing model names, and open\-weight models are identified by their Hugging Face repository IDs\. Open\-weight models are run locally with vLLM\.
## Appendix ALVLM Details

Table[6](https://arxiv.org/html/2608.23978#Ax1.T6)lists the exact LVLM variants used in our experiments\. We evaluate proprietary GPT and Gemini models through their respective APIs, using the latest available variants at the time of experimentation\. For open\-weight models, we use the Hugging Face checkpoints listed in the table and run local inference with vLLM[Kwon et al\. \(2023\)](https://arxiv.org/html/2608.23978#bib.bib2)\. Unless otherwise specified, we use the default decoding configuration across models\.

## Appendix BExpected Calibration Error

We compute Expected Calibration Error \(ECE\) to summarize the alignment between the Matcher’s stated confidence and empirical accuracy\. For each final selection, the Matcher is prompted to report a confidence scorecj∈\[0,100\]c\_\{j\}\\in\[0,100\], and we define correctness as whether the selected item matches the target item at the corresponding position\. We partition predictions into 10 evenly spaced confidence bins over\[0,100\]\[0,100\]:\[0,10\),\[10,20\),…,\[90,100\]\[0,10\),\[10,20\),\\ldots,\[90,100\]\.

For each binBℓB\_\{\\ell\}, we compute the average empirical accuracy and the average stated confidence of all predictions in that bin\. ECE is then computed as the weighted average absolute gap between accuracy and confidence across bins:

ECE=∑ℓ=110\|Bℓ\|N​\|acc⁡\(Bℓ\)−conf⁡\(Bℓ\)\|,\\mathrm\{ECE\}=\\sum\_\{\\ell=1\}^\{10\}\\frac\{\|B\_\{\\ell\}\|\}\{N\}\\left\|\\mathrm\{acc\}\(B\_\{\\ell\}\)\-\\mathrm\{conf\}\(B\_\{\\ell\}\)\\right\|,whereNNis the total number of predictions,\|Bℓ\|\|B\_\{\\ell\}\|is the number of predictions in binBℓB\_\{\\ell\},acc⁡\(Bℓ\)\\mathrm\{acc\}\(B\_\{\\ell\}\)is the percentage of correct predictions in the bin, andconf⁡\(Bℓ\)\\mathrm\{conf\}\(B\_\{\\ell\}\)is the average stated confidence in the bin\. Lower ECE indicates better calibration\.

In the main text, we report calibration primarily through calibration plots rather than ECE\. This is because ECE is strongly negatively correlated with accuracy in our setting \(Spearman’sρ=−0\.96\\rho=\-0\.96\), making it less informative as a separate measure of calibration\. Intuitively, models with lower accuracy tend to have larger confidence–accuracy gaps even when their confidence patterns are qualitatively similar\. Calibration plots therefore provide a more interpretable view of whether models are overconfident or underconfident across confidence levels\.

## Appendix CExtracted Partial Description Validation

#### Metric definitions\.

Letdddenote the full description and letA=\{a1,…,an\}A=\\\{a\_\{1\},\\ldots,a\_\{n\}\\\}denote the generated attribute list for an item\. Text is normalized by lowercasing and tokenized with the regular expression\[A\-Za\-z0\-9\]\+, so punctuation is ignored and only alphanumeric tokens are retained\.

For each attributeaia\_\{i\}, we compute an attribute support scores⁡\(ai,d\)s\(a\_\{i\},d\)as the maximum of exact substring match and token overlap:

s⁡\(ai,d\)=max⁡\(substr⁡\(ai,d\),overlap⁡\(ai,d\)\)\.s\(a\_\{i\},d\)=\\max\\left\(\\mathrm\{substr\}\(a\_\{i\},d\),\\mathrm\{overlap\}\(a\_\{i\},d\)\\right\)\.An attribute is marked as supported ifs⁡\(ai,d\)≥0\.45s\(a\_\{i\},d\)\\geq 0\.45\.

The attribute support rate for an item is the fraction of generated attributes whose support score exceeds this threshold:

support​\_​rate​\(A,d\)=∑i=1n𝟙\[s\(ai,d\)≥0\.45\]n\.\\mathrm\{support\\\_rate\}\(A,d\)=\\frac\{\\sum\_\{i=1\}^\{n\}\\mathbb\{1\}\[s\(a\_\{i\},d\)\\geq 0\.45\]\}\{n\}\.
Description coverage measures how much of the original full description is recovered by the generated attributes:

coverage⁡\(A,d\)=∑w∈Vmin⁡\(cA​\(w\),cd​\(w\)\)\|T⁡\(d\)\|\.\\mathrm\{coverage\}\(A,d\)=\\frac\{\\sum\_\{w\\in V\}\\min\\left\(c\_\{A\}\(w\),c\_\{d\}\(w\)\\right\)\}\{\|T\(d\)\|\}\.
For the selected underspecified expression, we compute the least\-attribute support score ass⁡\(aleast,d\)s\(a\_\{\\mathrm\{least\}\},d\), which verifies that the initial partial expression is grounded in the original description despite being intentionally underspecified\.

#### Validation results\.

The extracted attributes are strongly supported by the original full descriptions\. Across all items, the mean attribute support rate is 0\.998, indicating that nearly all generated attributes can be matched back to the source description\. The mean attribute support score is 0\.980, and the mean description coverage is 0\.895, suggesting that the decomposition preserves most of the descriptive content while keeping the attributes grounded in the original expression\. The selected least\-informative attributes are also well supported, with a mean least\-attribute support score of 0\.981\.

These results suggest that our procedure produces source\-grounded partial descriptions: the initial expressions used inPartialare intentionally underspecified, but not hallucinated or detached from the original human descriptions\.

## Appendix DResults

### D\.1LVLM and Human Matcher Performance Gap across Interaction Protocols

Tables[7](https://arxiv.org/html/2608.23978#A4.T7)–[10](https://arxiv.org/html/2608.23978#A4.T10)report the full LVLM–human matcher accuracy gaps for each interaction protocol, complementing the best\-over\-protocol summary in Table[2](https://arxiv.org/html/2608.23978#S4.T2)\. AcrossStatic,Full, andPartial, nearly all LVLMs significantly lag behind human matchers, although the size of the gap varies by model, dataset, and protocol\. The only non\-significant cases occur for Gemini\-3\.1\-Pro, including one case in which it reaches parity with human matchers\.

UnderNone, all models significantly underperform human matchers\. This comparison should be interpreted with caution, however, because the human reference data were not collected under an explicitNonesetting\. Instead, the human interactions more closely resemble a mixture of the other protocols, depending on how much detail human Directors provided in their object descriptions and how well Matchers understood those descriptions\.

Table 7:Mean LVLM–human matcher accuracy gap \(%\) under theStaticprotocol\. Darker red denotes larger deficits relative to human matchers, and bold marks the smallest gap within each dataset\. Significance is assessed with pairedtt\-tests:∗p≤\.05\{\}^\{\*\}p\\leq\.05,p∗⁣∗≤\.01\{\}^\{\*\*\}p\\leq\.01,∗∗∗p≤\.001\{\}^\{\*\*\*\}p\\leq\.001\.Table 8:Mean LVLM–human matcher accuracy gap \(%\) under theFullprotocol\. Darker red denotes larger deficits relative to human matchers, and bold marks the smallest gap within each dataset\. Significance is assessed with pairedtt\-tests:∗p≤\.05\{\}^\{\*\}p\\leq\.05,p∗⁣∗≤\.01\{\}^\{\*\*\}p\\leq\.01,∗∗∗p≤\.001\{\}^\{\*\*\*\}p\\leq\.001\.Table 9:Mean LVLM–human matcher accuracy gap \(%\) under thePartialprotocol\. Darker red denotes larger deficits relative to human matchers, and bold marks the smallest gap within each dataset\. Significance is assessed with pairedtt\-tests:∗p≤\.05\{\}^\{\*\}p\\leq\.05,p∗⁣∗≤\.01\{\}^\{\*\*\}p\\leq\.01,∗∗∗p≤\.001\{\}^\{\*\*\*\}p\\leq\.001\.Table 10:Mean LVLM–human matcher accuracy gap \(%\) under theNoneprotocol\. Darker red denotes larger deficits relative to human matchers, and bold marks the smallest gap within each dataset\. Significance is assessed with pairedtt\-tests:∗p≤\.05\{\}^\{\*\}p\\leq\.05,p∗⁣∗≤\.01\{\}^\{\*\*\}p\\leq\.01,∗∗∗p≤\.001\{\}^\{\*\*\*\}p\\leq\.001\.Table 11:Distribution of self\-repair cases by model and interaction protocol\. Counts indicate the number of observed self\-repairs underFullandPartial\. We did not observe self\-repairs underNone\.Table 12:Self\-repair outcomes by model\. Count denotes the number of self\-repair cases\. Transition columns report the percentage of repairs that change an initially wrong selection to correct, correct selection to wrong, or wrong selection to another wrong answer\.Δ\\DeltaConfidence denotes the mean change in confidence after repair; positive values indicate increased confidence, and negative values indicate decreased confidence\.
### D\.2Self\-Repair Analysis

Tables[12](https://arxiv.org/html/2608.23978#A4.T12)and[11](https://arxiv.org/html/2608.23978#A4.T11)summarize self\-repair behavior for models that revise their selections at least once\. Self\-repairs are relatively rare, accounting for no more than 2% of the 536 matches in any interaction protocol, and occur only underFullandPartial, with no observed cases underNone\. When models do revise, outcomes vary substantially: some repairs change initially wrong selections to correct ones, especially for Gemini\-3\.1\-Pro, while others introduce errors or replace one wrong answer with another, as in about half of GPT\-5\.5 cases\. Confidence changes are also mixed, suggesting that revisions do not always correspond to increased certainty\. Overall, self\-repair is observable but limited in current LVLM matchers\.

![Refer to caption](https://arxiv.org/html/2608.23978v1/figures/overall_results_ai_generated_descriptions.png)Figure 7:Overall results across the four image sets from the follow\-up experiments with AI\-generated descriptions\. Top: average matcher accuracy\. Bottom: average questions per target item\. Colors denote the four interaction protocols; horizontal lines mark human and random baselines\.![Refer to caption](https://arxiv.org/html/2608.23978v1/figures/dataset_level_confidence_calibration_ai_generated_descriptions.png)Figure 8:Protocol\-level calibration from the follow\-up experiments with AI\-generated descriptions\. Points show model\-specific confidence bins from 10 evenly spaced bins between 0 and 100; point size is proportional to bin count\. The dashed diagonal denotes perfect calibration\.![Refer to caption](https://arxiv.org/html/2608.23978v1/figures/overall_human_versus_ai_all_datasets.png)Figure 9:Differences in LVLM outcomes and question\-asking behavior between AI\- and human\-generated descriptions, averaged across all four datasets\. Bars showΔ=\\Delta=AI\-generated−\-human\-generated descriptions for accuracy and questions per item\.

## Appendix EFollow\-Up Studies

This section provides additional implementation details and results for the follow\-up studies conducted in Section[6](https://arxiv.org/html/2608.23978#S6)\.

### E\.1AI\-Generated Descriptions

#### Experimental Procedure

For experiments on AI\-generated descriptions, we replace human\-generated descriptions with GPT\-5\.4\-generated ones and rerun the main experiments across all models and interaction protocols\. We repeat each experiment three times using the same description map \(see below\) but with shuffled description order\. This controls for variation in item ordering while isolating the effect of description source, allowing us to test whether the main results depend on the use of human referring expressions\.

#### Additional Result Discussion

Figure[7](https://arxiv.org/html/2608.23978#A4.F7)reports the full results for each dataset\. The overall patterns are comparable to those in the main experiments with human\-generated descriptions \(Figure[3](https://arxiv.org/html/2608.23978#S4.F3); Section[5](https://arxiv.org/html/2608.23978#S5)\)\. LVLMs also remain overconfident and undercalibrated even with AI\-generated descriptions, as shown in Figure[8](https://arxiv.org/html/2608.23978#A4.F8)\.

For easier comparison, we aggregate results across the four datasets and visualize the differences in accuracy and question\-asking behavior between AI\- and human\-generated descriptions in Figure[9](https://arxiv.org/html/2608.23978#A4.F9)\. Figure[9](https://arxiv.org/html/2608.23978#A4.F9)shows that LVLMs generally achieve higher accuracy with AI\-generated descriptions than with human\-generated descriptions, especially underStaticandFull\. Models also tend to ask fewer questions underFull, suggesting that AI\-generated descriptions are easier for LVLMs to interpret\. This contrast indicates that human referring expressions pose a distinct challenge, even when they are sufficiently informative for human matchers\. Importantly, however, the overall protocol\-level patterns remain similar:StaticandFullare strongest,Partialis more variable, andNoneremains difficult\.

Table 13:Average number of questions per target item underFullfor different reasoning\-effort settings\. Subscripts report differences relative to the default no\-reasoning setting;greenindicates increases andredindicates decreases\.![Refer to caption](https://arxiv.org/html/2608.23978v1/figures/reasoning_effort_confidence_calibration.png)Figure 10:Calibration under different reasoning\-effort settings for GPT\-5\.4 and GPT\-5\.4\-Mini\. Each point represents a confidence bin, with thexx\-axis showing mean reported confidence and theyy\-axis showing empirical accuracy\. Point size indicates bin size, colors denote reasoning effort, and the dashed diagonal marks perfect calibration\.

### E\.2Reasoning Effort

#### Reasoning\-effort setup\.

Default reasoning effort is model\-specific rather than uniform across GPT models\. GPT\-5\.5 uses medium reasoning effort by default, whereas GPT\-5\.4 defaults to no reasoning\. We therefore vary the reasoning effort of GPT\-5\.4 and GPT\-5\.4\-Mini through the official Responses API to isolate the effect of test\-time reasoning\.

#### Reasoning effort changes question asking modestly\.

Table[13](https://arxiv.org/html/2608.23978#A5.T13)shows that higher reasoning effort slightly increases question asking for GPT\-5\.4 underFull, while GPT\-5\.4\-Mini asks very few questions across settings\. Together with Table[4](https://arxiv.org/html/2608.23978#S6.T4), this suggests that reasoning improves interactive performance partly by supporting information seeking, but not simply by increasing question volume\.

#### Reasoning effort slightly improves calibration, but gains plateau from medium to high\.

Figure[10](https://arxiv.org/html/2608.23978#A5.F10)shows that higher reasoning effort generally brings more calibration bins closer to the diagonal, especially when accounting for bin sizes, indicating slightly better alignment between confidence and accuracy\. However, the shift from medium to high reasoning is small, suggesting that scaling reasoning effort does not reliably improve calibration\. Most bins still fall below the diagonal, so both models remain overconfident\.

![Refer to caption](https://arxiv.org/html/2608.23978v1/figures/static_multi_round_accuracy_by_dataset_2x2.png)Figure 11:Accuracy \(%\) across rounds on the four image sets underStatic\.![Refer to caption](https://arxiv.org/html/2608.23978v1/figures/full_multi_round_questions_by_dataset_2x2.png)Figure 12:Average number of questions per item across rounds on the tangrams underFull\.![Refer to caption](https://arxiv.org/html/2608.23978v1/figures/multi_round_dataset_round_calibration_4x4.png)Figure 13:Round\-level calibration across datasets in the four\-round follow\-up experiments\. Points show model\-specific confidence bins from 10 evenly spaced bins between 0 and 100; point size is proportional to bin count\. The dashed diagonal denotes perfect calibration\.

### E\.3Multi\-Round Repeated References

#### Per\-dataset accuracy results underStatic\.

Figure[11](https://arxiv.org/html/2608.23978#A5.F11)shows model performance on tangrams across rounds underStatic\. Models improve matching performance even without the ability to ask questions, confirming that they can comprehend the increasingly efficient referring expressions generated by humans over repeated interaction\.

#### Question asking underFull\.

Figure[12](https://arxiv.org/html/2608.23978#A5.F12)shows the average number of questions per item across rounds underFull\. Question rates are generally stable or decrease slightly over rounds for most models and datasets\.

#### Model calibration across rounds\.

Figure[13](https://arxiv.org/html/2608.23978#A5.F13)shows round\-level calibration across datasets, aggregating results fromStaticandFullbecause the two protocols exhibit broadly similar patterns\. Models remain imperfectly calibrated across rounds: many confidence bins fall below the diagonal, indicating overconfidence\. Calibration does not consistently improve with repeated exposure, even when accuracy increases over rounds\. This suggests that LVLMs can sometimes benefit from repeated references without reliably improving their uncertainty estimates\.

### E\.4Alternative Director Robustness

#### Experimental procedure\.

To test whether Matcher results depend on one simulated information source, we replace GPT\-5\.4 with Gemini\-3\.1\-Flash as the Director and rerunFullandPartialon Basket Image Set 2 and Tangrams\. We evaluate both human\- and GPT\-5\.4\-generated initial descriptions and otherwise preserve the original prompts, candidate views, and turn budget\.

Table 14:Accuracy \(%\) with GPT\-5\.4 and Gemini\-3\.1\-Flash Directors, aggregated across Basket Image Set 2 and Tangrams and across human\- and AI\-generated descriptions\. Matcher rankings across Directors remain highly correlated \(ρ=\.929\\rho=\.929forFulland\.976\.976forPartial\)\.
#### Results\.

Table[14](https://arxiv.org/html/2608.23978#A5.T14)shows that individual model–protocol combinations vary with the Director, as expected in a collaborative task, but Matcher rankings remain highly stable\. Rank correlations between results with GPT\-5\.4 and Gemini\-3\.1\-Flash Directors areρ=\.929\\rho=\.929underFullandρ=\.976\\rho=\.976underPartial\. GPT\-5\.4 is generally the stronger Director, yet the main protocol\-level patterns and Matcher hierarchy do not depend on using GPT\-5\.4 alone\. The correlations are computed over Matcher\-level accuracies aggregated across the two datasets and description sources\.

### E\.5Face\-Domain Robustness

#### Experimental procedure\.

We evaluate five representative Matchers on six image sets from the public FACES database\([Holland et al\., 2018](https://arxiv.org/html/2608.23978#bib.bib39)\)\. Each set contains 12 images of one individual displaying varied facial expressions\. Because these image sets do not include human interaction data, we generate initial descriptions with GPT\-5\.4 and evaluate the same four protocols\. Director and Matcher indices are randomized independently as in the main study\. This experiment contains no protocol\-matched human baseline and is intended as a robustness check in another fine\-grained within\-category context\.

Table 15:Accuracy \(%\) on six face image sets with AI\-generated descriptions\. Each image set contains 12 images\. There is no protocol\-matched human baseline for this robustness experiment\.
#### Results and scope\.

Table[15](https://arxiv.org/html/2608.23978#A5.T15)reproduces the main aggregate pattern:StaticandFullare comparable \(48\.7% and 47\.8%\),Partialis lower \(41\.9%\), andNoneis most difficult \(30\.7%\)\. The result extends the protocol\-level observation beyond dogs, baskets, and tangrams, but faces remain a controlled within\-category matching context; this study does not establish generality to cluttered scenes, spatial relations, or open\-world visual search\.

## Appendix FPrompts for Visual Grounding Experiments

This section reports the prompts used in the study for the visual grounding experiments\.

### F\.1Task Overview

We reuse the same task overview prompt for both the Director and Matcher across all interaction protocols to maximize the comparability across the four interaction protocols\.

Youwillparticipateinavisualreferencetaskinvolvingtworoles:DirectorandMatcher\.Thetwoparticipantsmaynotsharethesamevisualcontext\.

Bothparticipantsareshownrandomizedsetsofindexedvisualitems\.Thesevisualitemsmaybeobjects,scenes,orotherimage\-basedstimuli\.Theparticipantsshareasubsetoftargetitems,buttheMatchermayalsoseeadditionaldistractoritemsthatarenotvisibletotheDirector\.

ThegoalisfortheDirectorandMatchertocollaboratesothattheMatchercanidentifyeachtargetitemasaccuratelyaspossible\.

Thetaskmayconsistofmultiplerounds\.Pleasestrictlyfollowtheinstructionsforyourassignedroleacrossallrounds\.

### F\.2Matcher Output Format Instructions

Across all interaction protocols, Matcher is prompted to produce a JSON object containing the target item index and its confidence level \(%\) rated from 0 to 100\.

The output format instruction for theStaticprotocol is as follows\.

OutputyourresponseinJSONformatusingthefollowingstructure:

\{

"target\_item\_index":5,

"confidence":75

\}

\*\*FieldDefinitions\*\*

\-"target\_item\_index":theindexoftheitemyouareselectingasthematchfortheidentifiedtargetitem\.

\-"confidence":yourconfidencelevel\(%\)intheselection,ratedfrom0\(notconfidentatall\)to100\(mostconfident\)\.

The output format instruction for the other three interactive protocols \(Full,Partial, andNone\) is as follows\.

OutputyourresponseinJSONformatusingthefollowingstructure:

\{

"utterance":"Yournatural,conciseutterancetotheDirector\.",

"selections":\[

\{

"nth\_item":2,

"target\_item\_index":5,

"confidence":75

\},

…

\]

\}

\*\*FieldDefinitions\*\*

\-"utterance"shouldcontainyournaturallanguageutteranceyousaytotheDirector\.Donotmentionindicesorpositionalcueshere\.

\-"selections"shouldbealistofitemsyouareselectingbasedontheDirector’sresponses\.Eachitemselectionshouldinclude:

\-"nth\_item":theorderofthetargetitembeingidentified\(startingfrom1forthefirstitemidentifiedinthecurrentround\)\.

\-"target\_item\_index":theindexoftheitemyouareselectingasthematchfortheidentifiedtargetitem\.

\-"confidence":yourconfidencelevel\(%\)intheselection,ratedfrom0\(notconfidentatall\)to100\(mostconfident\)\.

\*\*OutputRules\*\*

\-Ifyoudonothaveenoughinformationtomakeaselection,set"selections"toanemptylist\.

\-Ifyouhaveenoughinformationtomakeaselectionforthecurrentitem,includethatitemin"selections"\.

\-Ifyouarerevisingoneormoreearlierselections,includeeachreviseditemasaseparateentryin"selections"\.

\-Ifyouarebothrevisingearlierselectionsandselectingthecurrentitem,includeallofthoseupdatesin"selections"inthesameresponse\.

\-Unlessyouarerevisinganearlierchoice,"selections"shouldjustcontainonlyoneentryforthemostrecentlyidentifieditem\.

\-Donotincludeduplicateentriesforthesamenth\_iteminasingleresponse\.

\-Ifyoureviseapreviousselection,your"utterance"maybrieflyacknowledgethecorrection,butitmustnotdiscloseanyitemindexorpositionalinformation\.

### F\.3StaticProtocol

Since there is no interaction between the Director and Matcher under this protocol, we simulate the Director model by programmatically sending the Matcher a full item description at a time without actually prompting a Director LVLM\. The Matcher system prompt is given below\.

\{TaskOverview\}

\#\#\#RoleInstructions

YouaretheMatcher\.Inthisrole,youmustcarefullyanalyzetheDirector’sdescriptionsandchoosethemostlikelymatchforeachtargetitembasedonthevisualitemsvisibletoyou\.

\#\#\#OutputFormatInstruction

\{MatcherOutputFormatInstruction\}

### F\.4FullProtocol

Director system prompt\.

\{TaskOverview\}

\#\#\#RoleInstructions

YouaretheDirector\.Inthisrole,youmustdescribeeachvisualitemvisibletoyoufromthelowestindextothehighestindexoneatatimeandansweranyquestionstheMatchermayhave\.

AnalyzetheMatcher’squestioncarefullybasedonyourvisualcontextandanswerthequestiontothebestofyourability\.

\#\#\#CommunicationGuidelines

\-Describeexactlyonetargetitematatime\.

\-DonotmoveontothenextitemuntiltheMatchersignalstoproceed\.Waitforthemtosignalwhentheyareready\.

\-Usenatural,human\-like,simple,andclearlanguagethatsupportsefficientcommunication\.

\-TryyourbesttomakeeachdescriptionunambiguousandsufficientlyinformativefortheMatchertoidentifytherelatedtargetitem\.

\-Keepeachdescriptionwithin50words,favoringshorterdescriptionswhenclarityismaintained\.

\-Donotreferencedisplayposition,displayorder,neighboring\-itemrelations,orexplicititemindices\.Youmaydescribewithin\-itemspatialstructurewhenitispartoftheitem’sappearance\.

Matcher system prompt\.

\{TaskOverview\}

\#\#\#RoleInstructions

YouaretheMatcher\.Inthisrole,youmustcarefullyanalyzetheDirector’sdescriptionsandaskanyclarificationquestionsifneededtoidentifyeachtargetitembasedonthevisualitemsvisibletoyou\.

\#\#\#CommunicationGuidelines

\-Ineachturn,youareallowedtoaskaquestionorsignaltoproceedtothenexttargetitem,butkeepitnolongerthan50words\.

\-Ifyouaskaquestionoraconfirmation,youmustnotinternallyselectanytargetitemuntiltheDirectorprovidesananswertoyourquestion\.

\-Ifyousignaltoproceed,youmusthaveinternallyselectedatargetiteminyouroutput\.

\-Useconcise,natural,human\-likelanguageandnevermentionitemindicesorpositionalcuesinyournatural\-languageutterance\.

\-Youmayaskasmanyquestionsasneededtoclarifyyourunderstandingofthetargetitems,butavoidaskingredundantorirrelevantquestions\.

\-Onceyouhaveidentifiedatargetitem,youcanmoveontothenextoneuntilallthetargetitemshavebeenidentified\.

\-Youcanaskquestionsaboutyourpreviousselection\(s\),incaseyouwanttodoublecheckorrevisethem\.

\-Donotreferencedisplayposition,displayorder,neighboring\-itemrelations,orexplicititemindices\.Youmaydescribewithin\-itemspatialstructurewhenitispartoftheitem’sappearance\.

\#\#\#OutputFormatInstruction

\{MatcherOutputFormatInstructions\}

### F\.5PartialProtocol

Director system prompt\.

\{TaskOverview\}

\#\#\#RoleInstructions

YouaretheDirector\.Inthisrole,youmustdescribeeachvisualitemvisibletoyoufromthelowestindextothehighestindexoneatatimeandansweranyquestionstheMatchermayhave\.

AnalyzetheMatcher’squestioncarefullybasedonyourvisualcontextandanswerthequestiontothebestofyourability\.

\#\#\#CommunicationGuidelines

\-Describeexactlyonetargetitematatime\.

\-DonotmoveontothenextitemuntiltheMatchersignalstoproceed\.Waitforthemtosignalwhentheyareready\.

\-Usenatural,human\-like,simple,andclearlanguagethatsupportsefficientcommunication\.

\-Foreachnewtarget,beginwithanunderspecifieddescriptiontoencouragetheMatchertoaskclarifyingquestions\.

\-WhenrespondingtotheMatcher’squestions,behelpfulandinformative,butalsobeincrementalandpartialdescriptionsatatime,insteadofover\-sharing\.

\-Donotreferencedisplayposition,displayorder,neighboring\-itemrelations,orexplicititemindices\.Youmaydescribewithin\-itemspatialstructurewhenitispartoftheitem’sappearance\.

Matcher system prompt\.

\{TaskOverview\}

\#\#\#RoleInstructions

YouaretheMatcher\.Inthisrole,youmustcarefullyanalyzetheDirector’sdescriptionsandaskanyclarificationquestionsifneededtoidentifyeachtargetitembasedonthevisualitemsvisibletoyou\.

\#\#\#CommunicationGuidelines

\-Ineachturn,youareallowedtoaskaquestionorsignaltoproceedtothenexttargetitem,butkeepitnolongerthan50words\.

\-Ifyouaskaquestionoraconfirmation,youmustnotinternallyselectanytargetitemuntiltheDirectorprovidesananswertoyourquestion\.

\-Ifyousignaltoproceed,youmusthaveinternallyselectedatargetiteminyouroutput\.

\-Useconcise,natural,human\-likelanguageandnevermentionitemindicesorpositionalcuesinyournatural\-languageutterance\.

\-Youmayaskasmanyquestionsasneededtoclarifyyourunderstandingofthetargetitems,butavoidaskingredundantorirrelevantquestions\.

\-Onceyouhaveidentifiedatargetitem,youcanmoveontothenextoneuntilallthetargetitemshavebeenidentified\.

\-Youcanaskquestionsaboutyourpreviousselection\(s\),incaseyouwanttodoublecheckorrevisethem\.

\-Donotreferencedisplayposition,displayorder,neighboring\-itemrelations,orexplicititemindices\.Youmaydescribewithin\-itemspatialstructurewhenitispartoftheitem’sappearance\.

\#\#\#OutputFormatInstruction

\{MatcherOutputFormatInstruction\}

### F\.6NoneProtocol

Director system prompt\.

\{TaskOverview\}

\#\#\#RoleInstructions

YouaretheDirector\.Inthisrole,youmustanswertheMatcher’squestionsusingonlyoneofthefollowingresponses:"Yes,""No,"or"Unclear\."

AnalyzetheMatcher’squestioncarefullybasedonyourvisualcontextandanswerthequestiontothebestofyourability\.

\#\#\#CommunicationGuidelines

\-Justoutput"Yes,""No,"or"Unclear"withoutanyadditionalexplanationorinformationineachturn\.

\-IftheMatcher’sresponseindicatesthetargetitemindextheyselected,justrespondwith"Unclear\."

Matcher system prompt\.

\{TaskOverview\}

\#\#\#RoleInstructions

YouaretheMatcher\.Inthisrole,youmustaskonlyyesornoquestionstotheDirectortoidentifyeachtargetitembasedonthevisualitemsvisibletoyou\.Thereareatotalof\\$num\_targetstargetitemsforyoutoidentifyineachround\.

\#\#\#CommunicationGuidelines

\-Ineachturn,youareonlyallowedtoaskaclearandconcisequestionatatime,nolongerthan50words\.

\-AvoidaskingquestionsthatrequiretheDirectortoprovideinformationbeyond"Yes,""No,"or"Unclear\."

\-Useconcise,natural,human\-likelanguageandnevermentionitemindicesorpositionalcuesinyournatural\-languageutterance\.

\-Youmayaskupto\\$max\_questionsquestionsasneededtoclarifyyourunderstandingofthetargetitems,andavoidaskingredundantorirrelevantquestions\.

\-Onceyouhaveidentifiedatargetitem,youcanmoveontothenextoneuntilallthe\\$num\_targetstargetitemshavebeenidentified\.

\-Youcanaskquestionsaboutyourpreviousselection\(s\),incaseyouwanttodoublecheckorrevisethem\.

\-Foreachquestion,makeclearwhichtargetitemitrefersto,suchas"thefirsttargetitem,""thefifthtargetitem,"andsoon\.

\#\#\#OutputFormatInstruction

\{MatcherOutputFormatInstruction\}

## Appendix GPrompts for Referring Expression Processing

### G\.1Referring Expression Extraction

Prompt used for the automatic extraction of referring expressions for each target visual referent produced in a referential conversation\. Words within “<\>" denote placeholders\.

This prompt is from[Zeng et al\. \(2026\)](https://arxiv.org/html/2608.23978#bib.bib38)and we only use it for the dog and basket datasets\. The tangrams dataset already comes with manually curated object descriptions for each target object\.

Youwillbegivenatranscriptofaconversationbetweentwoparticipantsengagedinacollaborativeobject\-matchingtask\.Thereareexactly<num\_objects\>targetobjects\.Oneparticipant\(thedescriber\)describeseachtargetobject,andtheotherparticipant\(thematcher\)attemptstoidentifythem\.

Yourtaskistoextractthedescriptivephrasesusedbythedescriberforeachtargetobject\.

\-Extractphrasesverbatimfromthetranscript\.

\-Donotextractthewholeutterance,onlythedescriptivephrases\.

\-Excludedisfluencies,fillers,andfalsestarts\(e\.g\.,"um","uh","like"\)\.

\-Donotparaphraseorinfermissinginformation\.

\-Eachobjectmayhaveoneormultipledescriptivephrases\.

ReturntheresultsinthefollowingJSONformat:

\{

"object\_\#1":"descriptivephrasesforobject1",

"object\_\#2":"descriptivephrasesforobject2",

…

"object\_\#<num\_objects\>":"descriptivephrasesforobject<num\_objects\>"

\}

Exampledescriptionphrases:

\-doesn’thavehandle,tipofitisthickerthanrestofbody,brownishcolor,weavesareinsquaresifyoulookatitdirectly

\-halfcircle,nohandles,toptipofitisalittlebitthickerthanrestofbody

\-tipwhichisalittlebitthickerthanrestofbody

\-tipthatisalittlebitlargerthanbody,looksalittlebitthicker

Transcript:

<transcript\>

OutputonlytheJSONobject\.Donotincludeanyadditionaltextorexplanations\.

### G\.2Partial Description Extraction

1\."full":theoriginalfulldescriptionasastring,unchanged\.

2\."attributes":alistofatomicattributesextractedfromthefulldescription\.

3\."least":theleastinformativeattributeforthisvisualitem,selectedfromits"attributes"list\.

Attributeextractionrules:

\-Eachattributeshouldbeexpressedinnaturallanguage\.

\-Eachattributemustdescribeonlyonevisualfeature\.

\-Keepattributesascloseaspossibletotheoriginalwording;verbatimifpossible\.

\-Donotinferattributesthatarenotexplicitlystatedinthedescription\.

\-Donottreatmissingattributesasnegativeevidence\.

\-Removeduplicateattributeswithinthesameitem\.

\-Splitcompounddescriptionsintoseparateatomicattributes\.

\-Preserverelative/comparativeattributeswhenexplicitlystated,suchas"tallest","shortest","secondshortest",or"shorterthan9"\.

\-Resolvecoreferenceswhenextractingattributes,sothateachattributeisself\-containedanddoesnotrelyonpronounsorotherreferencestootherattributes\.

\-Ifthereisonlyoneattributeinthedescription,simplyreusethefulldescriptionasthesingleattribute\.

Leastattributeselectionrules:

\-Foreachitem,chooseexactlyoneattributefromthatitem’s"attributes"list\.

\-Choosetheattributethatisleastinformativeforidentifyingtheitem,takingintoaccountallattributesacrossallitemsintheinputmap\.

\-Donotchooseanattributethatgivesthegestaltimpressionofthewholeitem;instead,chooseanattributethatdescribesaspecificvisualfeature\.

\-Preferageneric,common,orweaklydistinguishingattributeoveranattributethatisdistinctiveorrareacrossitems\.

\-Ifseveralattributesaresimilarlygeneric,choosetheonethatwouldbeleasthelpfulonitsownfordistinguishingtheitemfromtheotheritems\.

\-Donotinvent,rewrite,orcombineattributesfor"least";itmustexactlymatchonestringinthatitem’s"attributes"list\.

ReturnonlyvalidJSON\.Donotincludeexplanations,markdown,orcomments\.

Inputformat:

\{

"image\_id":"fullnatural\-languagedescription"

\}

Outputformat:

\{

"image\_id":\{

"full":"originalfullnatural\-languagedescription",

"attributes":\[

"attribute1",

"attribute2"

\],

"least":"attribute2"

\}

\}

### G\.3Object Description Generation

We use the following prompt to generate full descriptions for each object across the four image sets with GPT\-5\.4\. To mirror the human Director setting, the model is shown only the target images\. We prompt it to produce a description map that can be used directly in our experimental pipeline\.

Thetaskinvolvestworoles:aDirectorandaMatcher\.TheDirectorseesasetoftargetvisualitemsandmustdescribethemsotheMatchercanidentifythesameitemsamongpossibledistractors\.YourjobistowritetheDirector\-styleinitialdescriptionsforoneround\.

Youwillreceivealldirectorimagesfortheroundinasinglemessage\.Eachimageislabeledwithitsfilenameonlysotheoutputcanbekeyedcorrectly\.

ReturnonlyvalidJSON:anobjectmappingeachexactfilenametooneshortnatural\-languagedescription\.

Descriptionrules:

\-Describeeveryprovidedimageexactlyonce\.

\-UsetheexactfilenamesasJSONkeys\.

\-Eachdescriptionmustbefewerthan50tokens\.

\-Usenatural,human\-like,simple,andclearlanguage\.

\-MakeeachdescriptionunambiguousandsufficientlyinformativeforaMatchertoidentifytheitemamongsimilaritems\.

\-Describetheitemitself:shape,pose,orientation,silhouette,distinctiveparts,andwithin\-itemspatialstructurewhenuseful\.

\-Donotreferencedisplayposition,displayorder,neighboring\-itemrelations,explicititemindices,filepaths,orfilenamesinthedescriptions\.

\-Donotsay"image","picture","itemnumber",orsimilardisplay\-orientedlabelsinthedescriptions\.

\-Donotincludemarkdown,comments,explanations,orextrakeys\.

## Appendix HDataset Details

Figures[14](https://arxiv.org/html/2608.23978#A8.F14),[15](https://arxiv.org/html/2608.23978#A8.F15),[16](https://arxiv.org/html/2608.23978#A8.F16), and[17](https://arxiv.org/html/2608.23978#A8.F17)show the four object image sets used in our experiments\. These sets span real\-world objects \(dogs and baskets\) and abstract tangrams, and differ in the number of target and distractor items\. For the basket and dog sets, the Director sees only the target items, while the Matcher sees the full candidate set\. For the tangram set, both players see the full set\. In all experiments, item orders for Director follow the item\-description order in the original dataset, while Matcher’s item orders are randomized to prevent trivial index matching or data contamination\.

We obtain the image sets and corresponding human–human interaction data from the GitHub repositories associated with the cited papers\.

![Refer to caption](https://arxiv.org/html/2608.23978v1/figures/baskets_emnlp2025.png)Figure 14:Basket image set from[Wang et al\. \(2025a\)](https://arxiv.org/html/2608.23978#bib.bib37)\. The first 10 baskets are target items visible to the Director; the Matcher sees the full set\.![Refer to caption](https://arxiv.org/html/2608.23978v1/figures/dogs_emnlp2025.png)Figure 15:Dog image set from[Wang et al\. \(2025a\)](https://arxiv.org/html/2608.23978#bib.bib37)\. The first 10 dogs are target items visible to the Director; the Matcher sees the full set\.![Refer to caption](https://arxiv.org/html/2608.23978v1/figures/baskets_acl2026.png)Figure 16:Basket image set from[Zeng et al\. \(2026\)](https://arxiv.org/html/2608.23978#bib.bib38)\. The first 12 baskets are target items visible to the Director; the Matcher sees the full set\.![Refer to caption](https://arxiv.org/html/2608.23978v1/figures/tangrams_hawkins2020.png)Figure 17:Tangram image set from[Hawkins et al\. \(2020\)](https://arxiv.org/html/2608.23978#bib.bib36)\. Both the Director and Matcher see the full set; item orders are randomized independently for the two views in our experiments\.