Learning to Refer from Estimated Listener Gaze

# Learning to Refer from Estimated Listener Gaze
Source: [https://arxiv.org/html/2609.14207](https://arxiv.org/html/2609.14207)
Téa Wright & Alane SuhrAffiliation:Department of Computer ScienceAffiliation:University of California, BerkeleyAffiliation:Berkeley, CA 94704, USAAffiliation:\{teaywright, suhr\}@berkeley\.edu

###### Abstract

We propose to finetune vision\-language models to generate more pragmatically optimal referring expressions by transforming observations of incremental listener comprehension, in the form of gaze scanpaths, into learning signals\. During training, referring expressions are sampled from the speaker policy being optimized, conditioned on images and target referents; then, a neural listener estimating human gaze behavior maps from images and sampled referring expressions to scanpaths, each represented by a sequence of fixations, with each fixation corresponding to a word in the referring expression\. We experiment with several approaches to convert fixation sequences and target referents into token\- and sequence\-level rewards, which are used to optimize policy parameters\. Through evaluation with human listeners, we find that speaker policies trained with gaze\-estimating listeners result in significantly more pragmatically\-optimal references than base models, reducing sequence length from 15\.4 down to 4\.0 words while increasing referential success from 75\.2 up to 80\.0%\. Our work demonstrates a promising opportunity for learning to generate utterances through language\-based interaction, not only from the explicit signal of communicative success, but also from implicitly\-available observations of a listener’s process of comprehension\.111Code and model weights are available at:[https://github\.com/Berkeley\-NLP/reg\-from\-gaze](https://github.com/Berkeley-NLP/reg-from-gaze)

## 1Introduction

Language\-based interaction is collaborative\. People collaborate through language use to come to a shared understanding of the world, and they do so efficiently by minimizing the total combined effort necessary to reach mutual understanding\([Clark and Wilkes\-Gibbs, 1986](https://arxiv.org/html/2609.14207#bib.bib18)\)\. In natural human interaction, we minimize joint effort through communication signals beyond the textual content of our utterances, including backchanneling, paralanguage, and gaze\. Reference games are an ideal testbed to understand and measure these phenomena\. Here, a speaker and a listener collaborate to establish joint attention to a part of their shared environment\([Dale and Reiter, 1995](https://arxiv.org/html/2609.14207#bib.bib19)\)\. The speaker’s task, known as referring expression generation \(REG\), is to produce an utterance that uniquely identifies the region of interest to their listener\. The listener’s task is to identify the target based on the speaker’s description, called referring expression comprehension \(REC\)\. In reference games and elsewhere, people design their utterances by taking into account the shared interaction situation and their knowledge of their audience so that their contributions to the interaction are as informative as necessary, but not overly verbose\([Grice, 1975](https://arxiv.org/html/2609.14207#bib.bib17);[Kapur et al\., 2026](https://arxiv.org/html/2609.14207#bib.bib31)\)\.

On the other hand, neural language models do not exhibit this level of pragmatic optimality\.[Ma et al\. \(2025\)](https://arxiv.org/html/2609.14207#bib.bib4)finds that referring expressions generated by modern vision\-language models \(VLMs\) in reference games differ greatly from human\-written references; they tend to include unnecessary information and often fail to uniquely identify referents by taking into account the interaction context\. Recent work has approached the problem of improving referential success through learning from interaction, where a VLM policy is rewarded for successfully describing an intended referent to a listener\([Gul and Artzi, 2024](https://arxiv.org/html/2609.14207#bib.bib36);[Tang et al\., 2024](https://arxiv.org/html/2609.14207#bib.bib2)\)\. These policies are trained only from the outcome of reference games\. Instead,[Jara\-Ettinger and Rubio\-Fernandez \(2022\)](https://arxiv.org/html/2609.14207#bib.bib16)argue that REG is best understood as a pragmatic effort to guide the listener’s visual search process, observations of which are used by speakers to check listener understanding during language production\([Clark and Krych, 2004](https://arxiv.org/html/2609.14207#bib.bib12)\)\.

We use observations of the process of visual search during reference comprehension as a learning signal for improving VLMs as speakers in reference games\. Gaze provides a quantifiable signal that allows us to directly relate a speaker’s language to a listener’s visual search process; we can test exactly which tokens meaningfully contribute to the listener’s understanding\. To this end, we design a learning setup where VLMs are finetuned as speakers in interaction with gaze\-estimating listeners\. First, we train a VLM to estimate listener gaze\. Then, during training of the VLM speaker, we iterate between deploying it in reference games with the gaze\-estimating listener \(Figure[1](https://arxiv.org/html/2609.14207#S1.F1)a\), and optimizing the speaker policy using rewards derived from listener gaze estimates \(Figure[1](https://arxiv.org/html/2609.14207#S1.F1)b\)\.

We experiment with a variety of reward functions for converting gaze estimates to learning signals, and compare our approach to learning from non\-gaze\-estimating listeners\. Our results show that using gaze\-estimating listeners during training can both boost referential success in reference games with human listeners from 75\.2 up to 80\.0%, while also reducing the verbosity of such references from 15\.4 down to 4\.0 words on average\. Our results also reveal a clear tradeoff in current VLMs between accuracy and verbosity: while speakers trained with non\-gaze\-estimating listeners achieve even higher referential success \(up to 86%\), this comes at a cost of generating increasingly longer references, up to an average of 24\.4 words\. Overall, we find that learning from estimated observations of theprocessof reference comprehension, rather than theoutcomeof a reference game alone, results in speaker policies that generate much more pragmatically optimal references\.

![Refer to caption](https://arxiv.org/html/2609.14207v1/gazefig.png)Figure 1:Learning to refer from interaction with incremental listeners\. During training, we iterate between \(a\) deploying a speaker policyπθS\\pi^\{\\text\{S\}\}\_\{\\theta\}in a reference game with a listener policyπG\\pi^\{\\text\{G\}\}, which is trained to estimate gaze behavior of human listeners, and \(b\) updating the speaker policy by transforming observations of listener gaze behavior to rewards\.
## 2Technical overview

Problem statementLetx=⟨x1,…,xT⟩∈𝒱\+x=\\langle x\_\{1\},\\dots,x\_\{T\}\\rangle\\in\\mathcal\{V\}^\{\+\}be a natural language referring expression comprised of tokens from a vocabulary𝒱\\mathcal\{V\},ℐ∈\{0,…,255\}M×N×3\\mathcal\{I\}\\in\\\{0,\\dots,255\\\}^\{M\\times N\\times 3\}an image of sizeMMbyNN, andb=\(bx,by,bw,bh\)∈ℤ4b=\(b\_\{x\},b\_\{y\},b\_\{w\},b\_\{h\}\)\\in\\mathbb\{Z\}^\{4\}a bounding box circumscribing a target referent\.2220≤bx≤M0\\leq b\_\{x\}\\leq Mand0≤by≤N0\\leq b\_\{y\}\\leq Nrepresent the anchor point of the bounding box at its top left corner;0<bw≤M−bx0<b\_\{w\}\\leq M\-b\_\{x\}and0<bh≤N−by0<b\_\{h\}\\leq N\-b\_\{y\}represent the width and height of the bounding box\.We study the problem of referring expression generation, where a speaker policyπθS\\pi^\{\\text\{S\}\}\_\{\\theta\}maps from an imageℐ\\mathcal\{I\}and target bounding boxbbto a referring expressionxx\. The task of referring expression comprehension \(REC\) is executed by a listener policyπREC\\pi^\{\\text\{REC\}\}, which maps from the imageℐ\\mathcal\{I\}and the generated referring expressionxxto a coordinater=\(rx,ry\)∈\{0,…,M\}×\{0,…,N\}r=\(r\_\{x\},r\_\{y\}\)\\in\\\{0,\\dots,M\\\}\\times\\\{0,\\dots,N\\\}\. Communicative success is achieved if the listener correctly identifies the target referent circumscribed bybb; i\.e\.,r∈br\\in b\.333We say a coordinate\(x,y\)∈b=\(bx,by,bw,bh\)\(x,y\)\\in b=\(b\_\{x\},b\_\{y\},b\_\{w\},b\_\{h\}\)\(a “hit”\) ifbx≤x≤bx\+bwb\_\{x\}\\leq x\\leq b\_\{x\}\+b\_\{w\}andby≤y≤by\+bhb\_\{y\}\\leq y\\leq b\_\{y\}\+b\_\{h\}\.Success requires thatxxdescribes the target referent accurately and unambiguously with respect to the entire image\. For every image and target referent, there are a large number of possible referring expressions that satisfy these criteria; one open challenge, then, is achieving referential efficiency by identifying the shortest such referring expression\. This is the focus of our study\. We extend this classical formulation of referring expression comprehension, where a listener maps from an entire referring expression to a single candidate coordinate, to anincrementalcomprehension task\. Let a scanpathg=⟨g1,…,gT⟩g=\\langle g\_\{1\},\\dots,g\_\{T\}\\ranglebe a sequence of gaze fixations, with eachgig\_\{i\}taking on either a null value∅\\emptyset, or a coordinate value\(gx\(i\),gy\(i\)\)∈\{0,…,M\}×\{0,…,N\}\\left\(g^\{\(i\)\}\_\{x\},g^\{\(i\)\}\_\{y\}\\right\)\\in\\\{0,\\dots,M\\\}\\times\\\{0,\\dots,N\\\}\. Here, a listener policyπG\\pi^\{\\text\{G\}\}maps from an imageℐ\\mathcal\{I\}, \(possibly partial\) referring expressionxxof lengtht≤Tt\\leq T, and a scanpath prefixggof lengtht−1t\-1, to the next fixationgtg\_\{t\}\. Reference comprehension is performed iteratively; each fixationgt∼πG\(⋅∣ℐ,x1:t,g1:t−1\)g\_\{t\}\\sim\\pi^\{\\text\{G\}\}\(\\cdot\\mid\\mathcal\{I\},x\_\{1:t\},g\_\{1:\{t\-1\}\}\)corresponds to an input wordxtx\_\{t\}\.

ApproachOur goal is to learn a speaker policyπθS\\pi^\{\\text\{S\}\}\_\{\\theta\}that generates referring expressions which are botheffectiveandefficient\.πS\\pi^\{\\text\{S\}\}is instantiated as a large VLM, with initial parametersθ0\\theta\_\{0\}equivalent to the pretrained Molmo\-7B\-D model\([Deitke et al\., 2024](https://arxiv.org/html/2609.14207#bib.bib3)\)\.πθ0S\\pi^\{\\text\{S\}\}\_\{\\theta\_\{0\}\}is neither efficient nor effective as a speaker; references generated from this policy are overly verbose and result in low communicative success\. We finetune this policy through reinforcement learning to maximize reward \(Section[5](https://arxiv.org/html/2609.14207#S5)\)\. During training, we simulate REC with a listener modelπϕG\\pi^\{\\text\{G\}\}\_\{\\phi\}that we have trained to estimate human gaze scanpaths \(Section[4](https://arxiv.org/html/2609.14207#S4)\)\. Speaker reward is a function not only of final communicative success, but also of the incremental process of reference comprehension as estimated by the listener modelg∼πϕG\(⋅∣ℐ,x\)g\\sim\\pi^\{\\text\{G\}\}\_\{\\phi\}\(\\cdot\\mid\\mathcal\{I\},x\)\.

## 3Related works

Referring expression generation \(REG\) is often framed as an optimization problem where a speaker must identify the lowest\-cost \(e\.g\., shortest\) utterance which uniquely distinguishes a target referent in the context of distractors\. While this optimization is typically modeled at a whole\-utterance level, e\.g\., via the rational speech acts \(RSA\) model\([Frank and Goodman, 2012](https://arxiv.org/html/2609.14207#bib.bib33);[Goodman and Frank, 2016](https://arxiv.org/html/2609.14207#bib.bib28)\),[Cohn\-Gordon et al\. \(2018\)](https://arxiv.org/html/2609.14207#bib.bib32)achieve improved communicative success with LSTM\-based speakers using incremental character\-level RSA\. Modern VLMs are typically able to accurately describe target referents in isolation, but they very often generate pragmatically suboptimal utterances that are unnecessarily verbose\([Kapur et al\., 2026](https://arxiv.org/html/2609.14207#bib.bib31)\)or ambiguous with respect to the entire image, resulting in both low communicative success and inefficient interaction\([Ma et al\., 2025](https://arxiv.org/html/2609.14207#bib.bib4)\)\.

Recent work finetunes existing VLMs to produce more pragmatically optimal references by learning from communicative success through interaction with listeners\([Willemsen and Skantze, 2024](https://arxiv.org/html/2609.14207#bib.bib22);[Tang et al\., 2024](https://arxiv.org/html/2609.14207#bib.bib2);[Gul and Artzi, 2024](https://arxiv.org/html/2609.14207#bib.bib36)\)\. During learning, references are sampled from the speaker policy \(i\.e\., REG\), then sent to a listener, who selects the point or region of the image that the sampled reference most likely refers to \(i\.e\., REC\)\. References that result in communicative success are kept as positive examples, while references for which the listener failed to identify the target are kept as negative examples\. This prior work varies in how positive and negative examples are converted to learning signals; for example,[Gul and Artzi \(2024\)](https://arxiv.org/html/2609.14207#bib.bib36)cast learning as a contextual bandit problem, assigning positive rewards to positive examples and negative rewards to negative ones\. However, in all existing work, the learning signal is sparse: each reference is paired with only one piece of feedback, dependent on the finaloutcomeof a reference game\. Here, we use observations of theprocessof comprehension as a learning signal, as evidenced by the trajectory of listener gaze during reference comprehension\.

In machine learning, human gaze data has been modeled and used as a behavior of interest both as a communication device and learning signal, particularly in natural language processing \(NLP\) and human\-robot interaction \(HRI\)\. Gaze and attention data have also been used to guide language generation in descriptive visual tasks like image captioning, video captioning, and video summarization\([Yu et al\., 2017](https://arxiv.org/html/2609.14207#bib.bib44);[Chen and Zhao, 2018](https://arxiv.org/html/2609.14207#bib.bib45);[Xu et al\., 2015](https://arxiv.org/html/2609.14207#bib.bib43)\)\. In HRI, gaze of human operators in jointly embodied environments with robots has been studied as communicative signal for establishing joint attention\([Huang and Mutlu, 2016](https://arxiv.org/html/2609.14207#bib.bib30);[Rüppel et al\., 2025](https://arxiv.org/html/2609.14207#bib.bib29)\), but embodied gaze has not been explored as a signal from which general language\-using agents can learn\. In NLP, human gaze patterns during reading of written text have been used as a learning signal to improve performance on text\-related tasks\([Mishra et al\., 2016](https://arxiv.org/html/2609.14207#bib.bib6);[Barrett and Hollenstein, 2020](https://arxiv.org/html/2609.14207#bib.bib23);[Kiegeland et al\., 2024](https://arxiv.org/html/2609.14207#bib.bib5);[Papadopoulos, 2025](https://arxiv.org/html/2609.14207#bib.bib24);[Bondar et al\., 2025](https://arxiv.org/html/2609.14207#bib.bib25);[Lopez\-Cardona et al\., 2025](https://arxiv.org/html/2609.14207#bib.bib26)\)\. However, this work is limited to non\-embodied settings, where gaze provides a signal more about human processing of the text itself rather than its grounding to the world\.

Existing work on gaze estimation focuses mainly on free\-viewing tasks\([Kümmerer et al\., 2022](https://arxiv.org/html/2609.14207#bib.bib8);[Kümmerer and Bethge, 2023](https://arxiv.org/html/2609.14207#bib.bib7);[Yin et al\., 2024](https://arxiv.org/html/2609.14207#bib.bib15)\)and visual search\([Hu et al\., 2021b](https://arxiv.org/html/2609.14207#bib.bib9);[Cartella et al\., 2025b](https://arxiv.org/html/2609.14207#bib.bib27);[Cartella et al\., 2025a](https://arxiv.org/html/2609.14207#bib.bib13);[Mondal et al\., 2025](https://arxiv.org/html/2609.14207#bib.bib14)\)conditioned on simple single\-word references\. Some recent works study gaze estimation during more complex language understanding tasks, such as visual question answering\([Chen et al\., 2021](https://arxiv.org/html/2609.14207#bib.bib34)\)\. Most similar to our approach,[Mondal et al\. \(2024\)](https://arxiv.org/html/2609.14207#bib.bib1)study estimating human gaze during REC by collecting scanpaths over RefCOCO\([Kazemzadeh et al\., 2014](https://arxiv.org/html/2609.14207#bib.bib20)\)and training a corresponding gaze\-prediction model with a single\-stream VLM architecture\.

## 4Estimating listener gaze

We train an incremental listener policy, which we refer to as Molmo\-REC\-Gaze, to estimate human gaze scanpaths during reference comprehension\. We initialize the policy’s parameters as Molmo\-7B\-D\([Deitke et al\., 2024](https://arxiv.org/html/2609.14207#bib.bib3)\)and finetune it using examples adapted from the RefCOCO\-Gaze dataset\([Mondal et al\., 2024](https://arxiv.org/html/2609.14207#bib.bib1)\)\. This approach results in gaze scanpath estimation superior to previous approaches such as ART\([Mondal et al\., 2024](https://arxiv.org/html/2609.14207#bib.bib1)\)\.

### 4\.1Data

RefCOCO\([Kazemzadeh et al\., 2014](https://arxiv.org/html/2609.14207#bib.bib20)\)is a popular corpus used for the study of multimodal reference\.[Mondal et al\. \(2024\)](https://arxiv.org/html/2609.14207#bib.bib1)introduce RefCOCO\-Gaze, which labels a subset of examples from RefCOCO with scanpaths of human participants’ gaze during reference comprehension\. We adapt this dataset to train a neural model that estimates the gaze scanpath of a human listener for arbitrary images and referring expressions\. To acquire each scanpath, they display the image \(without the bounding box annotation\) to the participant, and instruct the participant to focus their attention to the center of the image\. The human\-written referring expression is then verbalized with word onset timestamps recorded\. During reference comprehension, an infrared video\-based eye\-tracker records participant scanpaths, each of which comprises a sequence of fixations paired with timestamps\. Each scanpath in RefCOCO\-Gaze demonstrates successful referential comprehension\.

Word and fixation alignmentTo create demonstration data for our incremental listener policy, we must pair each word in the referring expression to a single fixation\. Word onset and fixation timestamps included in RefCOCO\-Gaze allow us to identify, for each fixation, which word was most recently heard by the participant\. We apply a 200ms delay to the timestamp of each word to account for auditory processing time\([Kirchner and Thorpe, 2006](https://arxiv.org/html/2609.14207#bib.bib35)\)\. There may be multiple fixations per word; when this happens, we select the final fixation before the subsequent word onset\. In some cases, there are no corresponding fixations for a word, for example if the word is very short, resulting in a null point∅\\emptyset\. We convert the RefCOCO\-Gaze training set, which containsDT=D\_\{T\}=16,982 scanpaths across 1,799 RefCOCO examples, into a set of demonstrations𝒟Gtrain=\{x\(i\),g\(i\)\}i=1DT\\mathcal\{D\}^\{\\text\{train\}\}\_\{\\text\{G\}\}=\\left\\\{x^\{\(i\)\},g^\{\(i\)\}\\right\\\}\_\{i=1\}^\{D\_\{T\}\}, where\|x\(i\)\|=\|g\(i\)\|\\left\|x^\{\(i\)\}\\right\|=\\left\|g^\{\(i\)\}\\right\|\. For each example,x0=BOSx\_\{0\}=\\texttt\{BOS\},g0g\_\{0\}indicates listener gaze before the audio is played, andxT=EOSx\_\{T\}=\\texttt\{EOS\}\. The final fixationgT∈bg\_\{T\}\\in bis always in the target bounding box\. We also convert the validation set into a dataset𝒟Gval\\mathcal\{D\}\_\{\\text\{G\}\}^\{\\text\{val\}\}comprising 869 scanpaths across 92 RefCOCO examples\.

### 4\.2Finetuning for gaze estimation

Most existing VLMs are trained to perform REC by mapping from an image and a referring expression to bounding box coordinates\. The Molmo family of VLMs\([Deitke et al\., 2024](https://arxiv.org/html/2609.14207#bib.bib3)\)are instead trained to perform REC by mapping form an image and referring expression to a set of point coordinates, which makes it an ideal initialization for estimating gaze\. We initialize our gaze listener policy as Molmo\-7B\-D and optimize it to minimize the token\-level loss of gaze sequences included in𝒟G\\mathcal\{D\}\_\{\\text\{G\}\}, i\.e\., we approximate

ϕ=argmaxϕ′\(−∑i=1DT∑t=0\|x\(i\)\|logπϕ′G\(gt\(i\)∣x≤t\(i\),g<t\(i\)\)\)\.\\phi=\\arg\\max\_\{\\phi^\{\\prime\}\}\\left\(\-\\sum\_\{i=1\}^\{D\_\{T\}\}\\sum\_\{t=0\}^\{\\left\|x^\{\(i\)\}\\right\|\}\\log\\pi^\{\\text\{G\}\}\_\{\\phi^\{\\prime\}\}\\left\(g^\{\(i\)\}\_\{t\}\\mid x^\{\(i\)\}\_\{\\leq t\},g^\{\(i\)\}\_\{<t\}\\right\)\\right\)\\;\\;\.Words inxxmay be tokenized into multiple subwords; we label each token with the same fixation label\. We train for 4 epochs and select the best checkpoint based on loss on the validation set𝒟Gval\\mathcal\{D\}\_\{\\text\{G\}\}^\{\\text\{val\}\}\. We refer to the final trained policy, which selectsgt≈argmaxπϕG\(⋅∣x≤t,g<t\)g\_\{t\}\\approx\\arg\\max\\pi^\{\\text\{G\}\}\_\{\\phi\}\(\\cdot\\mid x\_\{\\leq t\},g\_\{<t\}\), as Molmo\-REC\-Gaze\. Figure[5](https://arxiv.org/html/2609.14207#A1.F5)compares scanpaths from human participants from the validation set𝒟Gval\\mathcal\{D\}\_\{\\text\{G\}\}^\{\\text\{val\}\}with scanpaths estimated by Molmo\-REC\-Gaze\.

Results[Mondal et al\. \(2024\)](https://arxiv.org/html/2609.14207#bib.bib1)release ART, a VLM trained on RefCOCO\-Gaze to estimate human listener gaze, which we evaluate as a baseline to Molmo\-REC\-Gaze\. We report two metrics: \(a\) path distance, based on dynamic time warping\([Senin, 2008](https://arxiv.org/html/2609.14207#bib.bib21)\), compares a human gaze scanpath with an estimated scanpath; and \(b\) REC accuracy, the frequency at which an estimated scanpath ends in the target bounding box \(REC accuracy\)\. On the validation set𝒟Gval\\mathcal\{D\}^\{\\text\{val\}\}\_\{\\text\{G\}\},444The test set of RefCOCO\-Gaze is not publicly available\.Molmo\-REC\-Gaze significantly outperforms ART, reducing path distance to the closest human scanpath from 55\.9 to 39\.9 and increasing REC accuracy from 66\.7 to 84\.4%\. See Appendix[A](https://arxiv.org/html/2609.14207#A1)for implementation details, formatting changes, and qualitative examples of gaze prediction\.

## 5Learning from estimated gaze

Our overall learning setup follows existing work on continual learning from interaction\([Kojima et al\., 2021](https://arxiv.org/html/2609.14207#bib.bib37);[Suhr and Artzi, 2023](https://arxiv.org/html/2609.14207#bib.bib38);[Gul and Artzi, 2024](https://arxiv.org/html/2609.14207#bib.bib36)\), where training iterates between deploying the policy being optimized in interaction with another language\-using agent \(Figure[1](https://arxiv.org/html/2609.14207#S1.F1)a\), and optimizing the policy based on observations of the interaction \(Figure[1](https://arxiv.org/html/2609.14207#S1.F1)b\)\. In our work, the policy being optimized is a VLMπθS\\pi^\{\\text\{S\}\}\_\{\\theta\}that maps from images and bounding boxes to a referring expression \(i\.e\., REG\), and the listener agent is Molmo\-REC\-Gaze, which maps from images and referring expressions to an estimated word\-aligned scanpath\.

Deploying the speaker policyWe assume access to a dataset𝒟I\\mathcal\{D\}\_\{\\text\{I\}\}of images, where each image is annotated with object bounding boxes\. In practice,𝒟I\\mathcal\{D\}\_\{\\text\{I\}\}is the validation split of COCO\-2014\([Lin et al\., 2015](https://arxiv.org/html/2609.14207#bib.bib41)\)\.555We use the validation set to avoid overlap with images on which Molmo\-REC\-Gaze was trained, as RefCOCO images \(thus, RefCOCO\-Gaze images\) were taken from the COCO\-2014 training set\.We randomly split𝒟I\\mathcal\{D\}\_\{\\text\{I\}\}into training𝒟Itrain\\mathcal\{D\}^\{\\text\{train\}\}\_\{\\text\{I\}\}\(95%, 38,131 images\) and validation𝒟Ival\\mathcal\{D\}^\{\\text\{val\}\}\_\{\\text\{I\}\}\(5%, 2,006\) splits\. During each training interaction, we sample an imageℐ\\mathcal\{I\}from𝒟Itrain\\mathcal\{D\}^\{\\text\{train\}\}\_\{\\text\{I\}\}, then sample a bounding boxbbwhose size is between 1 and 30% of the image’s area\. This removes target referents that are trivially easy to describe \(too large\), and ones which are too difficult to perceive \(too small\)\. We then sample a referring expression from the current speaker policy; i\.e\.,x∼τπθS\(⋅∣ℐ,b\)x\\sim^\{\\tau\}\\pi^\{\\text\{S\}\}\_\{\\theta\}\(\\cdot\\mid\\mathcal\{I\},b\)\. Bounding boxes are annotated on the image as a transparent rectangle with a 3\-pixel red border\. We subsequently estimate a gaze scanpathggfrom Molmo\-REC\-Gaze, conditioned on the imageℐ\\mathcal\{I\}and sampled referring expressionxx\. To avoid a listener spuriously identifying the target referent, we set the initial pre\-reference fixationg0g\_\{0\}to a random coordinate outside of the bounding boxbb\. Each interaction results in an example\(ℐ,b,x,g\)\(\\mathcal\{I\},b,x,g\)such that\|x\|=\|g\|\|x\|=\|g\|\.

Optimizing policy from interaction observationsWe experiment with four reward functions that transform our interaction observation\(ℐ,b,x,g\)\(\\mathcal\{I\},b,x,g\)into a learning signal\. Each reward function assigns a rewardrtr\_\{t\}to each of the generated tokensxt≥1x\_\{t\\geq 1\}, taking into account whether fixations in the estimated gaze scanpath are in the bounding box\.

rtSeqLPHit=𝕀\[gT′∈b\]

Sequence\-level last\-point hit\(SeqLPHit\)\. Each tokenxtx\_\{t\}receives a positive reward only if the final non\-null fixationgT′g\_\{T^\{\\prime\}\}is in the bounding box, whereT′=max⁡\{t≤T:gt≠∅\}T^\{\\prime\}=\\max\\\{t\\leq T:g\_\{t\}\\neq\\emptyset\\\}\. This function rewards all tokens only if the listener’s gaze settles on the intended target at the end of the reference\.

rtSeqAnyHit=𝕀\[∃k∈\{1,…,T\}:gk∈b\]

Sequence\-level any hit\(SeqAnyHit\)\. Each tokenxtx\_\{t\}receives a positive reward if any fixations in the gaze scanpath are within the bounding boxbb\. We use this reward to identify if the speaker provided sufficient information to ground the referent at any point during the utterance\.

rtBeforeFirstHit=\{1if​1≤t≤τ1if​t=T​and​τ\>00otherwise

Before first hit\(BeforeFirstHit\)\. If the interaction demonstrates communicative success, we identify the timestep of the first hitτ=min⁡\{k:gk∈b\}\\tau=\\min\\\{k:g\_\{k\}\\in b\\\}; otherwise, we setτ=0\\tau=0\. Then, we assign a positive reward toxτx\_\{\\tau\}and all tokens in its prefixx1≤t<τx\_\{1\\leq t<\\tau\}\. For successful interactions, we assign a positive reward to theEOStokenxTx\_\{T\}\. The goal of this function is to only reinforce the tokens necessary to reach the earliest point of understanding\.

rtShaping=d⁡\(gt−1,b\)−d⁡\(gt,b\)d⁡\(g0,b\)

Distance\-based shaping\(Shaping\)\. We experiment with reward shaping\([Ng et al\., 1999](https://arxiv.org/html/2609.14207#bib.bib39)\)using a potential function based on the distance of each fixationgtg\_\{t\}from the bounding boxbbrelative to the initial fixationg0g\_\{0\}\. We define the distance metricd⁡\(g,b\)d\(g,b\)to be the Euclidean distance between the fixation pointggand the closest point in the bounding boxbb\. Thus, the reward for each tokenxtx\_\{t\}is the change in relative distance from the fixation associated with the previous token\. This rewards tokens that progressively guide the listener’s gaze closer to the target, even before a ”hit” occurs\.

After assigning rewards, we use REINFORCE\([Sutton et al\., 1999](https://arxiv.org/html/2609.14207#bib.bib40)\)to compute and apply gradients to policy parameters:

∇θ𝒥​\(θ\)=𝔼\(ℐ,b,x,t\)​\(rt−β​KL​\(πθS,ℐ,b,x,t\)⏟KL penalty\)​∇θ\(log⁡πθS​\(xt∣ℐ,b,x<t\)\)⏟Policy gradient\+α​∇θH​\(πθS​\(xt∣ℐ,b,x<t\)\)⏟Entropy bonus,\\nabla\_\{\\theta\}\\mathcal\{J\}\(\\theta\)=\\mathbb\{E\}\_\{\(\\mathcal\{I\},b,x,t\)\}\(r\_\{t\}\-\\underbrace\{\\beta\\text\{KL\}\\left\(\\pi^\{\\text\{S\}\}\_\{\\theta\},\\mathcal\{I\},b,x,t\\right\)\}\_\{\\text\{KL penalty\}\}\)\\underbrace\{\\nabla\_\{\\theta\}\(\\log\\pi^\{\\text\{S\}\}\_\{\\theta\}\\left\(x\_\{t\}\\mid\\mathcal\{I\},b,x\_\{<t\}\)\\right\)\}\_\{\\text\{Policy gradient\}\}\+\\underbrace\{\\alpha\\nabla\_\{\\theta\}H\\left\(\\pi^\{\\text\{S\}\}\_\{\\theta\}\(x\_\{t\}\\mid\\mathcal\{I\},b,x\_\{<t\}\)\\right\)\}\_\{\\text\{Entropy bonus\}\}\\;\\;,whereKL\(πθS,ℐ,b,x,t\)=DKL\(πθS\(xt∣ℐ,b,x<t\)∣∣πθ0S\(xt∣ℐ,b,x<t\)\)\\text\{KL\}\\left\(\\pi^\{\\text\{S\}\}\_\{\\theta\},\\mathcal\{I\},b,x,t\\right\)=D\_\{\\text\{KL\}\}\\left\(\\pi^\{\\text\{S\}\}\_\{\\theta\}\(x\_\{t\}\\mid\\mathcal\{I\},b,x\_\{<t\}\)\\mid\\mid\\pi^\{\\text\{S\}\}\_\{\\theta\_\{0\}\}\(x\_\{t\}\\mid\\mathcal\{I\},b,x\_\{<t\}\)\\right\)is the Kullback\-Leibler distance between the current policy with parametersθ\\thetaand the initial policyθ0\\theta\_\{0\}\.

## 6Experimental Setup

Our experiments focus on fine\-tuning a Molmo\([Deitke et al\., 2024](https://arxiv.org/html/2609.14207#bib.bib3)\)model as a speaker\. Evaluation targets two metrics: reference accuracy and efficiency\. We compare our approach with learning from interaction with a non\-gaze listener model, referred to as Molmo\-REC\.

Speaker model trainingWe initialize the speaker policyπS\\pi^\{\\text\{S\}\}withθ0\\theta\_\{0\}as Molmo\-7B\-D\-0924\([Deitke et al\., 2024](https://arxiv.org/html/2609.14207#bib.bib3)\)\. When sampling, we use temperatureτ=\\tau=0\.8, and randomly select a prompt from the brief prompts in[Ma et al\. \(2025\)](https://arxiv.org/html/2609.14207#bib.bib4)for each episode\. We linearly normalize rewards within\-batch to span\[−1,1\]\[\-1,1\]\. We use KL penalty coefficient ofβ=\\beta=0\.02 and an entropy regularization coefficient ofα=\\alpha=0\.01\. In all experiments, we finetune for 1,000 optimizer steps using an effective batch size of 64 and learning rate of 1e\-5\. We apply LoRA finetuning\([Hu et al\., 2021a](https://arxiv.org/html/2609.14207#bib.bib42)\)\(rank 8; alpha 16\) and checkpoint parameters every 2,000 episodes, selecting the best checkpoint for each run using the reward on the held\-out validation set𝒟Ival\\mathcal\{D\}^\{\\text\{val\}\}\_\{\\text\{I\}\}\.

System variationsIn addition to the models trained in the main experiments using the four different reward functions \(SeqLPHit,SeqAnyHit,BeforeFirstHit, andShaping; see Section[5](https://arxiv.org/html/2609.14207#S5)\), we compare with \(a\) the pre\-trained Molmo model without any fine\-tuning \(Molmo\), which serves as a baseline, and \(b\) human\-written references \(Human\), which serve as an upper bound\. We also experiment with two additional interaction settings during learning, where the listener is a pre\-trained Molmo\-7B\-D model\. As with our proposed models, we compute a scalar reward for each tokenxtx\_\{t\}based on observations of listener comprehension, and apply REINFORCE\. Interaction data and rewards for examples\(ℐ,b,x\)\(\\mathcal\{I\},b,x\)are computed as follows:

rtREC\-Success=𝕀\[gREC∈b\]

Communicative success\(REC\-Success\)\. We condition a Molmo model on imageℐ\\mathcal\{I\}and referring expressionxxand instruct it to point at the object being referred to, resulting in a single coordinategRECg\_\{\\text\{REC\}\}\. This method removes any signals of incremental comprehension\.

Iterative prompting\(REC\-Iter\)\. For each prefix in a referring expressionx≤tx\_\{\\leq t\}, we condition a Molmo model on imageℐ\\mathcal\{I\}and prefixx≤tx\_\{\\leq t\}, and instruct it to point at the object being referred to\. This results in a sequence of coordinatesg~\\tilde\{g\}\. This allows us to use the same reward functions we use withπG\\pi^\{G\}in an effort to disentangle effects of the listener policy and reward design\. Policies using this listener are indicated by the prefixREC\-\.

Table 1:Human evaluation averaged over all datasets withdπsd\_\{\\pi^\{s\}\}from Qwen2\.5\-VL for comparison\. Time to target refers to the average time after speech onset when the human evaluators can accurately identify the target by clicking on it\.### 6\.1Evaluation

Our goal is to train VLMs to generate referring expressions that are botheffectiveat communicating an intended referent, andefficientin doing so\. We report two core metrics: rate of communicative success and reference length\. We evaluate across four referring expression datasets, using both human and neural model listeners to measure communicative success\.

MetricsFor each test dataset𝒟Itest=\{ℐ\(i\),b\(i\),x^\(i\)\}i=1DI\\mathcal\{D\}^\{\\text\{test\}\}\_\{\\text\{I\}\}=\\left\\\{\\mathcal\{I\}^\{\(i\)\},b^\{\(i\)\},\\hat\{x\}^\{\(i\)\}\\right\\\}\_\{i=1\}^\{D\_\{I\}\}withDID\_\{I\}examples \(each also containing a human\-written referencex^\(i\)\\hat\{x\}^\{\(i\)\}\), a speaker policyπS\\pi^\{\\text\{S\}\}, and listener policyπREC\\pi^\{\\text\{REC\}\}, we report therate of communicative successas the proportion of examples in the test set for which the sampled referring expressionx\(i\)x^\{\(i\)\}results in the listener policyπREC\\pi^\{\\text\{REC\}\}identifying the target reference circumscribed byb\(i\)b^\{\(i\)\}in imageℐ\(i\)\\mathcal\{I\}^\{\(i\)\}\. We compute the averagereference lengthas the mean number of words\. During development of system variants and when choosing hyperparameters, and before deploying models as speakers in interaction with human listeners, we target a metricdπSd\_\{\\pi^\{\\text\{S\}\}\}that combines both core metrics using their distance to the results of human\-written references relative to the base model,Molmo\. Metrics for human\-written references are computed as:

AHuman=1DI∑i=1DI𝕀\[argmaxπREC\(x^\(i\),ℐ\(i\)\)∈b\(i\)\]\\displaystyle A\_\{\\textsc\{Human\}\}=\\dfrac\{1\}\{D\_\{I\}\}\\sum\_\{i=1\}^\{D\_\{I\}\}\\mathbb\{I\}\\left\[\\arg\\max\\pi^\{\\text\{REC\}\}\\left\(\\hat\{x\}^\{\(i\)\},\\mathcal\{I\}^\{\(i\)\}\\right\)\\in b^\{\(i\)\}\\right\]LHuman=1DI​∑i=1DI\|x^\(i\)\|\.\\displaystyle L\_\{\\textsc\{Human\}\}=\\dfrac\{1\}\{D\_\{I\}\}\\sum\_\{i=1\}^\{D\_\{I\}\}\\left\|\\hat\{x\}^\{\(i\)\}\\right\|\\;\\;\.The normalized distance to the upper\-bound performance for some policyπS\\pi^\{\\text\{S\}\}is then

dπS=\(max⁡\(0,AHuman−AπSAHuman−AMolmo\)\)2\+\(max⁡\(0,LπS−LHumanLMolmo−LHuman\)\)2\.d\_\{\\pi^\{\\text\{S\}\}\}=\\sqrt\{\\left\(\\max\\left\(0,\\frac\{A\_\{\\textsc\{Human\}\}\-A\_\{\\pi^\{\\text\{S\}\}\}\}\{A\_\{\\textsc\{Human\}\}\-A\_\{\\textsc\{Molmo\}\}\}\\right\)\\right\)^\{2\}\+\\left\(\\max\\left\(0,\\frac\{L\_\{\\pi^\{\\text\{S\}\}\}\-L\_\{\\textsc\{Human\}\}\}\{L\_\{\\textsc\{Molmo\}\}\-L\_\{\\textsc\{Human\}\}\}\\right\)\\right\)^\{2\}\}\\;\\;\.When reporting results, we use three random seeds to train three models per system and report the average on all metrics across the three trials\.

DatasetsWe evaluate generated references on four different test sets𝒟Itest\\mathcal\{D\}^\{\\text\{test\}\}\_\{\\text\{I\}\}: \(a\) RefCOCO Test A \(1,975 examples\), a subset of the RefCOCO test set whose target referents are people; \(b\) RefCOCO Test B \(1,810 examples\), a subset of the RefCOCO test set whose target referents are not people; \(c\) RefOI Co\-Occurrence \(993 examples\)\([Ma et al\., 2025](https://arxiv.org/html/2609.14207#bib.bib4)\), a benchmark designed to elicit referring expressions that successfully distinguish between multiple objects resembling the target referent; and \(d\) RefOI Single Presence \(492 examples\), which contains simple images that contain a single salient target referent and few to no distractors\.

Listener policiesWe evaluate speaker policy performance in interaction with two listener policiesπREC\\pi^\{\\text\{REC\}\}:QwenVLandHuman\.QwenVLuses the Qwen\-VL\-2\.5\-7B model\([Bai et al\., 2025](https://arxiv.org/html/2609.14207#bib.bib11)\)to map from an image and referring expression to a bounding boxb^\\hat\{b\}\.666Although CogVLM\-Grounding was found to be the best\-performing REC model in[Ma et al\. \(2025\)](https://arxiv.org/html/2609.14207#bib.bib4), we find it underperforms compared to Qwen\-VL \(see[B\.3](https://arxiv.org/html/2609.14207#A2.SS3)\)\.We consider the proposed bounding boxb^\\hat\{b\}to be successful if its intersection over union \(IoU\) with the target bounding boxbbis greater than 0\.5\.Humanreflects the performance of speaker models through interaction with human listeners\. First, we sample 200 examples from each of the four test sets\. In each trial, the image is displayed, and the referring expression is verbalized using Web Speech API’s Speech Synthesis777[https://developer\.mozilla\.org/en\-US/docs/Web/API/SpeechSynthesis](https://developer.mozilla.org/en-US/docs/Web/API/SpeechSynthesis)\. Participants are instructed to click on the region of the image they believe to be described by what they hear\. If a participant clicks an image multiple times, we take their final click to be their guess at the target referent\. Participants are also given the option to reveal the text of the reference after the audio has played\. We collect three responses𝒫=\{p1,p2,p3\}\\mathcal\{P\}=\\\{p\_\{1\},p\_\{2\},p\_\{3\}\\\}from three participants per image\-reference pair; we consider the set of responses𝒫∈b\\mathcal\{P\}\\in bif the majority of guesses are in the bounding box; i\.e\.,∑p∈𝒫𝕀⁡\(p∈b\)≥2\\sum\_\{p\\in\\mathcal\{P\}\}\\mathbb\{I\}\(p\\in b\)\\geq 2\. All three annotators for an example agreed on 77\.4% of examples\. Statistics on the rate of early clicks and text reveals can be found in Appendix[B](https://arxiv.org/html/2609.14207#A2)and the full details and annotator instructions are in Appendix[E](https://arxiv.org/html/2609.14207#A5)\.

## 7Results

Figure 2:Comparison of communicative success and average reference length in words with human and Qwen2\.5\-VL listeners with 95% confidence ellipses\.Table[1](https://arxiv.org/html/2609.14207#S6.T1)shows results for all system variations\. Models trained with our gaze\-based listener Molmo\-REC\-Gaze significantly reduce the gapdπSd\_\{\\pi^\{\\text\{S\}\}\}fromMolmoto the performance ofHumanreferences\. Almost all gaze\-based models result in references significantly shorter and referential success around or higher thanMolmo, demonstrating that learning from listeners that estimate the process of reference comprehension yields improvements in pragmatic competence\. When paired with human listeners, these models achieve up to an 4\.8% absolute improvement in communicative success \(Gaze\-SeqAnyHit\) and a reduction from 15\.4 down to 4\.0 words per reference \(Gaze\-Shaping\)\. On the other hand, while speakers trained with non\-gaze\-based listeners achieve even higher boosts in communicative success up to a 10\.8% absolutely improvement \(REC\-Success\), this comes at a serious cost to reference efficiency, up to references with an average of 24\.4 words \(REC\-SeqAnyHit\)\. This is also reflected in absolute time for human listeners to identify target referents; whileREC\-SeqAnyHitoutperformsGaze\-SeqAnyHitin communicative success by 5%, identifying target referents takes more than 4 seconds on average longer thanGaze\-SeqAnyHit\. AlthoughREC\-Shapingperforms competitively with the machine listener, the communicative success suffers with human listeners, indicating the presence of some behavior that is less intelligible to human listeners\. Further discussion of such behaviors can be found in Appendix[D](https://arxiv.org/html/2609.14207#A4)\. Our results illustrate a tradeoff between efficiency and efficacy in reference generation \(Figure[2](https://arxiv.org/html/2609.14207#S7.F2)\)\. Overall, we find that training with listeners that estimate the process of reference comprehension, rather than only the outcome, results in speakers that generate much more humanlike references\.888Full results per dataset and all model\-based evaluation can be found in Appendix[B](https://arxiv.org/html/2609.14207#A2)

![Refer to caption](https://arxiv.org/html/2609.14207v1/examples_comparison_fig_pretty.png)Figure 3:Examples of generated referring expressions conditioned on an image and bounding box from several different speaker policies\. See Appendix[D](https://arxiv.org/html/2609.14207#A4)for examples from all speaker models\.Figure 4:Comparison of referential strategies utilized by different speakers\. The keywords used for this analysis can be found in Appendix[D\.1](https://arxiv.org/html/2609.14207#A4.SS1)\.Beyond main results, we analyze the shifts in referential behavior induced by gaze\-based and REC\-based learning\. We consider the referential strategy of each speaker model using a list of keywords to tag each generated reference as utilizing spatial reference, attribute\-based reference, or both \(Figure[4](https://arxiv.org/html/2609.14207#S7.F4)\)\. We find that theGaze\-Shapingpolicy is the only machine speaker to use mostly spatial reference; its distribution of referential strategies most closely resembles the human distribution\. While the accuracy and length of references fromGaze\-ShapingandREC\-Shapingare similar for machine listeners, human evaluation and the differences in referential strategy demonstrate that gaze\-based rewards result in more humanlike references\.

We also conduct an analysis of the patterns observed in[Ma et al\. \(2025\)](https://arxiv.org/html/2609.14207#bib.bib4), noting what portions of references are redundant, or if the reference is ambiguous or misleading for a subset of 50 examples \(Appendix[D\.2](https://arxiv.org/html/2609.14207#A4.SS2)\)\. Our analysis reveals that gaze\-based optimization reduces redundancy, but also produces more ambiguous expressions\. While theMolmobaseline andREC\-Successpolicies generate an average of 13\.35 and 15\.01 unnecessary words per reference, respectively,Gaze\-SeqAnyHitnearly halves this redundancy to 7\.85 words, andGaze\-Shapingfurther minimizes it to just 3\.26 words\.

## 8Conclusion

In this work, we demonstrate that VLMs can be finetuned to generate more pragmatically optimal referring expressions using gaze as an implicit signal of listener comprehension\. Policies that learn from gaze reduce the gap between base VLM outputs and human\-like performance, reducing average length by as much as 11\.4 words while maintaining or improving communicative success rates\. Furthermore, theGaze\-Shapingpolicy is the only speaker policy to prioritize spatial reference, mirroring human behavior without explicit feedback\. These results demonstrate that learning from signals provided from a human\-like listener result in more human\-like generations\. Nonetheless, a tradeoff remains; speaker policies trained with REC listeners achieve the highest accuracy rates, but do so by producing verbose references\. Overall, our findings establish that gaze\-based insights into the process of comprehension offer a valuable implicit signal for advancing pragmatic language generation\.

## Acknowledgments

This material is based upon work supported by the National Science Foundation Graduate Research Fellowship Program under Grant No\. DGE 2146752\. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author\(s\) and do not necessarily reflect the views of the National Science Foundation\. This research was supported by an Amazon Research Award, a gift from Google, a Technical AI Safety Research award from Coefficient Giving, and an NVIDIA Academic Grant Program award\.

## Ethics Statement

This work focuses exclusively on English, potentially reinforcing linguistic patterns and cultural norms from high\-resource languages; our approach may inadvertently marginalize low\-resource or non\-standard dialects\. We acknowledge that generating human\-like expressions requires ongoing oversight to prevent the reinforcement of systemic linguistic biases\.

All human evaluation followed ethical research practices, with 586 participants completing the study after quality filtering\. Participants were compensated at an average rate of $19\.67 USD per hour\. Interaction data, including clicks and timestamps, was fully anonymized to protect participant privacy\.

## References

- S\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, S\. Song, K\. Dang, P\. Wang, S\. Wang, J\. Tang, H\. Zhong, Y\. Zhu, M\. Yang, Z\. Li, J\. Wan, P\. Wang, W\. Ding, Z\. Fu, Y\. Xu, J\. Ye, X\. Zhang, T\. Xie, Z\. Cheng, H\. Zhang, Z\. Yang, H\. Xu, and J\. LinQwen2\.5\-vl technical report\.External Links:2502\.13923,[Link](https://arxiv.org/abs/2502.13923)Cited by:[§6\.1](https://arxiv.org/html/2609.14207#S6.SS1.p4.1)\.
- Barrett and Hollenstein \(2020\)M\. Barrett and N\. HollensteinSequence labelling and sequence classification with gaze: novel uses of eye\-tracking data for natural language processing\.Language and Linguistics Compass14\(11\),pp\. 1–16\.Cited by:[§3](https://arxiv.org/html/2609.14207#S3.p3.1)\.
- Beyeret al\.\(2024\)L\. Beyer, A\. Steiner, A\. S\. Pinto, A\. Kolesnikov, X\. Wang, D\. Salz, M\. Neumann, I\. Alabdulmohsin, M\. Tschannen, E\. Bugliarello, T\. Unterthiner, D\. Keysers, S\. Koppula, F\. Liu, A\. Grycner, A\. Gritsenko, N\. Houlsby, M\. Kumar, K\. Rong, J\. Eisenschlos, R\. Kabra, M\. Bauer, M\. Bošnjak, X\. Chen, M\. Minderer, P\. Voigtlaender, I\. Bica, I\. Balazevic, J\. Puigcerver, P\. Papalampidi, O\. Henaff, X\. Xiong, R\. Soricut, J\. Harmsen, and X\. ZhaiPaliGemma: a versatile 3b vlm for transfer\.External Links:2407\.07726,[Link](https://arxiv.org/abs/2407.07726)Cited by:[§C\.5](https://arxiv.org/html/2609.14207#A3.SS5.p1.1)\.
- Bondaret al\.\(2025\)A\. Bondar, D\. R\. Reich, and L\. A\. JägerAlEYEgnment: leveraging Eye\-Tracking\-While\-Reading to align language models with human preferences\.InProceedings of the First International Workshop on Gaze Data and Natural Language Processing,C\. Acarturk, J\. Nasir, B\. Can, and C\. Coltekin \(Eds\.\),Varna, Bulgaria,pp\. 58–70\.External Links:[Link](https://aclanthology.org/2025.gaze4nlp-1.8/)Cited by:[§3](https://arxiv.org/html/2609.14207#S3.p3.1)\.
- Cartellaet al\.\(2025a\)G\. Cartella, V\. Cuculo, A\. D’Amelio, M\. Cornia, G\. Boccignone, and R\. CucchiaraModeling human gaze behavior with diffusion models for unified scanpath prediction\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 16206–16216\.Cited by:[§3](https://arxiv.org/html/2609.14207#S3.p4.1)\.
- Cartellaet al\.\(2025b\)G\. Cartella, V\. Cuculo, A\. D’Amelio, M\. Cornia, G\. Boccignone, and R\. CucchiaraModeling human gaze behavior with diffusion models for unified scanpath prediction\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 16206–16216\.Cited by:[§3](https://arxiv.org/html/2609.14207#S3.p4.1)\.
- Chen and Zhao \(2018\)S\. Chen and Q\. ZhaoBoosted attention: leveraging human attention for image captioning\.InComputer Vision – ECCV 2018,Lecture Notes in Computer Science,pp\. 72–88\(en\)\.Cited by:[§3](https://arxiv.org/html/2609.14207#S3.p3.1)\.
- Chenet al\.\(2021\)X\. Chen, M\. Jiang, and Q\. ZhaoPredicting human scanpaths in visual question answering\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 10876–10885\.Cited by:[§3](https://arxiv.org/html/2609.14207#S3.p4.1)\.
- Clark and Krych \(2004\)H\. H\. Clark and M\. A\. KrychSpeaking while monitoring addressees for understanding\.Journal of memory and language50\(1\),pp\. 62–81\.Cited by:[§1](https://arxiv.org/html/2609.14207#S1.p2.1)\.
- Clark and Wilkes\-Gibbs \(1986\)H\. H\. Clark and D\. Wilkes\-GibbsReferring as a collaborative process\.Cognition22\(1\),pp\. 1–39\.Cited by:[§1](https://arxiv.org/html/2609.14207#S1.p1.1)\.
- Cohn\-Gordonet al\.\(2018\)R\. Cohn\-Gordon, N\. Goodman, and C\. PottsPragmatically informative image captioning with character\-level inference\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 \(Short Papers\),M\. Walker, H\. Ji, and A\. Stent \(Eds\.\),New Orleans, Louisiana,pp\. 439–443\.External Links:[Link](https://aclanthology.org/N18-2070/),[Document](https://dx.doi.org/10.18653/v1/N18-2070)Cited by:[§3](https://arxiv.org/html/2609.14207#S3.p1.1)\.
- Dale and Reiter \(1995\)R\. Dale and E\. ReiterComputational interpretations of the gricean maxims in the generation of referring expressions\.Cognitive science19\(2\),pp\. 233–263\.Cited by:[§1](https://arxiv.org/html/2609.14207#S1.p1.1)\.
- Deitkeet al\.\(2024\)M\. Deitke, C\. Clark, S\. Lee, R\. Tripathi, Y\. Yang, J\. S\. Park, M\. Salehi, N\. Muennighoff, K\. Lo, L\. Soldaini, J\. Lu, T\. Anderson, E\. Bransom, K\. Ehsani, H\. Ngo, Y\. Chen, A\. Patel, M\. Yatskar, C\. Callison\-Burch, A\. Head, R\. Hendrix, F\. Bastani, E\. VanderBilt, N\. Lambert, Y\. Chou, A\. Chheda, J\. Sparks, S\. Skjonsberg, M\. Schmitz, A\. Sarnat, B\. Bischoff, P\. Walsh, C\. Newell, P\. Wolters, T\. Gupta, K\. Zeng, J\. Borchardt, D\. Groeneveld, C\. Nam, S\. Lebrecht, C\. Wittlif, C\. Schoenick, O\. Michel, R\. Krishna, L\. Weihs, N\. A\. Smith, H\. Hajishirzi, R\. Girshick, A\. Farhadi, and A\. KembhaviMolmo and pixmo: open weights and open data for state\-of\-the\-art vision\-language models\.External Links:2409\.17146,[Link](https://arxiv.org/abs/2409.17146)Cited by:[§2](https://arxiv.org/html/2609.14207#S2.p2.1),[§4\.2](https://arxiv.org/html/2609.14207#S4.SS2.p1.1),[§4](https://arxiv.org/html/2609.14207#S4.p1.1),[§6](https://arxiv.org/html/2609.14207#S6.p1.1),[§6](https://arxiv.org/html/2609.14207#S6.p2.1)\.
- Frank and Goodman \(2012\)M\. C\. Frank and N\. D\. GoodmanPredicting pragmatic reasoning in language games\.Science336\(6084\),pp\. 998–998\.Cited by:[§3](https://arxiv.org/html/2609.14207#S3.p1.1)\.
- Goodman and Frank \(2016\)N\. D\. Goodman and M\. C\. FrankPragmatic language interpretation as probabilistic inference\.Trends in cognitive sciences20\(11\),pp\. 818–829\.Cited by:[§3](https://arxiv.org/html/2609.14207#S3.p1.1)\.
- Grice \(1975\)H\. P\. GriceLogic and conversation\.InSpeech acts,pp\. 41–58\.Cited by:[§1](https://arxiv.org/html/2609.14207#S1.p1.1)\.
- Gul and Artzi \(2024\)M\. O\. Gul and Y\. ArtziCoGen: learning from feedback with coupled comprehension and generation\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 12966–12982\.Cited by:[§1](https://arxiv.org/html/2609.14207#S1.p2.1),[§3](https://arxiv.org/html/2609.14207#S3.p2.1),[§5](https://arxiv.org/html/2609.14207#S5.p1.1)\.
- Huet al\.\(2021a\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.External Links:2106\.09685,[Link](https://arxiv.org/abs/2106.09685)Cited by:[§6](https://arxiv.org/html/2609.14207#S6.p2.1)\.
- Huet al\.\(2021b\)Z\. Hu, A\. Bulling, S\. Li, and G\. WangFixationNet: forecasting eye fixations in task\-oriented virtual environments\.IEEE Trans\. Vis\. Comput\. Graph\.27\(5\),pp\. 2681–2690\(en\)\.Cited by:[§3](https://arxiv.org/html/2609.14207#S3.p4.1)\.
- Huang and Mutlu \(2016\)C\. Huang and B\. MutluAnticipatory robot control for efficient human\-robot collaboration\.In2016 11th ACM/IEEE international conference on human\-robot interaction \(HRI\),pp\. 83–90\.Cited by:[§3](https://arxiv.org/html/2609.14207#S3.p3.1)\.
- Jara\-Ettinger and Rubio\-Fernandez \(2022\)J\. Jara\-Ettinger and P\. Rubio\-FernandezThe social basis of referential communication: speakers construct physical reference based on listeners’ expected visual search\.\.Psychological review129\(6\),pp\. 1394\.Cited by:[§1](https://arxiv.org/html/2609.14207#S1.p2.1)\.
- Kapuret al\.\(2026\)R\. Kapur, R\. Hawkins, and E\. KreissWhen more words say less: decoupling length and specificity in image description evaluation\.External Links:2601\.04609,[Link](https://arxiv.org/abs/2601.04609)Cited by:[§1](https://arxiv.org/html/2609.14207#S1.p1.1),[§3](https://arxiv.org/html/2609.14207#S3.p1.1)\.
- Kazemzadehet al\.\(2014\)S\. Kazemzadeh, V\. Ordonez, M\. Matten, and T\. BergReferItGame: referring to objects in photographs of natural scenes\.InProceedings of the 2014 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),A\. Moschitti, B\. Pang, and W\. Daelemans \(Eds\.\),Doha, Qatar,pp\. 787–798\.External Links:[Link](https://aclanthology.org/D14-1086/),[Document](https://dx.doi.org/10.3115/v1/D14-1086)Cited by:[§3](https://arxiv.org/html/2609.14207#S3.p4.1),[§4\.1](https://arxiv.org/html/2609.14207#S4.SS1.p1.1)\.
- Kiegelandet al\.\(2024\)S\. Kiegeland, D\. R\. Reich, R\. Cotterell, L\. A\. J¨ager, and E\. WilcoxThe pupil becomes the master: eye\-tracking feedback for tuning LLMs\.LLMs and Cognition at ICML\.Cited by:[§3](https://arxiv.org/html/2609.14207#S3.p3.1)\.
- Kirchner and Thorpe \(2006\)H\. Kirchner and S\. J\. ThorpeUltra\-rapid object detection with saccadic eye movements: visual processing speed revisited\.Vision research46\(11\),pp\. 1762–1776\.Cited by:[§4\.1](https://arxiv.org/html/2609.14207#S4.SS1.p2.1)\.
- Kojimaet al\.\(2021\)N\. Kojima, A\. Suhr, and Y\. ArtziContinual learning for grounded instruction generation by observing human following behavior\.Transactions of the Association for Computational Linguistics9,pp\. 1303–1319\.External Links:[Link](https://aclanthology.org/2021.tacl-1.77/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00428)Cited by:[§5](https://arxiv.org/html/2609.14207#S5.p1.1)\.
- Kümmereret al\.\(2022\)M\. Kümmerer, M\. Bethge, and T\. S\. A\. WallisDeepGaze III: modeling free\-viewing human scanpaths with deep learning\.J\. Vis\.22\(5\),pp\. 7\(en\)\.Cited by:[§3](https://arxiv.org/html/2609.14207#S3.p4.1)\.
- Kümmerer and Bethge \(2023\)M\. Kümmerer and M\. BethgePredicting visual fixations\.Annu\. Rev\. Vis\. Sci\.9\(Volume 9, 2023\),pp\. 269–291\(en\)\.Cited by:[§3](https://arxiv.org/html/2609.14207#S3.p4.1)\.
- Linet al\.\(2015\)T\. Lin, M\. Maire, S\. Belongie, L\. Bourdev, R\. Girshick, J\. Hays, P\. Perona, D\. Ramanan, C\. L\. Zitnick, and P\. DollárMicrosoft coco: common objects in context\.External Links:1405\.0312,[Link](https://arxiv.org/abs/1405.0312)Cited by:[§5](https://arxiv.org/html/2609.14207#S5.p2.1)\.
- Liuet al\.\(2024\)H\. Liu, C\. Li, Y\. Li, and Y\. J\. LeeImproved baselines with visual instruction tuning\.External Links:2310\.03744,[Link](https://arxiv.org/abs/2310.03744)Cited by:[§C\.5](https://arxiv.org/html/2609.14207#A3.SS5.p1.1)\.
- Lopez\-Cardonaet al\.\(2025\)A\. Lopez\-Cardona, C\. Segura, A\. Karatzoglou, S\. Abadal, and I\. ArapakisSeeing eye to ai: human alignment via gaze\-based response rewards for large language models\.External Links:2410\.01532,[Link](https://arxiv.org/abs/2410.01532)Cited by:[§3](https://arxiv.org/html/2609.14207#S3.p3.1)\.
- Maet al\.\(2025\)Z\. Ma, J\. Ding, X\. Zhang, D\. Luo, J\. Ding, S\. Xu, Y\. Huang, R\. Peng, and J\. ChaiVision\-language models are not pragmatically competent in referring expression generation\.arXiv \[cs\.CL\]\.Cited by:[§B\.3](https://arxiv.org/html/2609.14207#A2.SS3.p1.1),[§1](https://arxiv.org/html/2609.14207#S1.p2.1),[§3](https://arxiv.org/html/2609.14207#S3.p1.1),[§6\.1](https://arxiv.org/html/2609.14207#S6.SS1.p3.1),[§6](https://arxiv.org/html/2609.14207#S6.p2.1),[§7](https://arxiv.org/html/2609.14207#S7.p3.1),[footnote 6](https://arxiv.org/html/2609.14207#footnote6)\.
- Mishraet al\.\(2016\)A\. Mishra, D\. Kanojia, and P\. BhattacharyyaPredicting readers’ sarcasm understandability by modeling gaze behavior\.Proceedings of the AAAI Conference on Artificial Intelligence30\(1\)\.External Links:[Link](https://ojs.aaai.org/index.php/AAAI/article/view/9884),[Document](https://dx.doi.org/10.1609/aaai.v30i1.9884)Cited by:[§3](https://arxiv.org/html/2609.14207#S3.p3.1)\.
- Mondalet al\.\(2024\)S\. Mondal, S\. Ahn, Z\. Yang, N\. Balasubramanian, D\. Samaras, G\. Zelinsky, and M\. HoaiLook hear: gaze prediction for speech\-directed human attention\.arXiv \[cs\.CV\]\.Cited by:[§A\.3](https://arxiv.org/html/2609.14207#A1.SS3.p1.1),[§3](https://arxiv.org/html/2609.14207#S3.p4.1),[§4\.1](https://arxiv.org/html/2609.14207#S4.SS1.p1.1),[§4\.2](https://arxiv.org/html/2609.14207#S4.SS2.p2.1),[§4](https://arxiv.org/html/2609.14207#S4.p1.1)\.
- Mondalet al\.\(2025\)S\. Mondal, N\. Sendhilnathan, T\. Zhang, Y\. Liu, M\. Proulx, M\. L\. Iuzzolino, C\. Qin, and T\. R\. JonkerGaze\-language alignment for zero\-shot prediction of visual search targets from human gaze scanpaths\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 2738–2749\.Cited by:[§3](https://arxiv.org/html/2609.14207#S3.p4.1)\.
- Nget al\.\(1999\)A\. Y\. Ng, D\. Harada, and S\. J\. RussellPolicy invariance under reward transformations: theory and application to reward shaping\.InProceedings of the Sixteenth International Conference on Machine Learning,ICML ’99,San Francisco, CA, USA,pp\. 278–287\.External Links:ISBN 1558606122Cited by:[§5](https://arxiv.org/html/2609.14207#S5.p7.1)\.
- Papadopoulos \(2025\)N\. PapadopoulosEye\-tracking as implicit feedback for aligning large language models and enhancing human\-ai teaming\.InProceedings of the 2025 Symposium on Eye Tracking Research and Applications,ETRA ’25,New York, NY, USA\.External Links:ISBN 9798400714870,[Link](https://doi.org/10.1145/3715669.3727334),[Document](https://dx.doi.org/10.1145/3715669.3727334)Cited by:[§3](https://arxiv.org/html/2609.14207#S3.p3.1)\.
- Rüppelet al\.\(2025\)J\. V\. Rüppel, A\. Rudenko, T\. Schreiter, M\. Magnusson, and A\. J\. LilienthalGaze\-supported large language model framework for bi\-directional human\-robot interaction\.In2025 34th IEEE International Conference on Robot and Human Interactive Communication \(RO\-MAN\),pp\. 677–684\.Cited by:[§3](https://arxiv.org/html/2609.14207#S3.p3.1)\.
- Senin \(2008\)P\. SeninDynamic time warping algorithm review\.Information and Computer Science Department University of Hawaii at Manoa Honolulu, USA855\(1\-23\),pp\. 40\.Cited by:[§4\.2](https://arxiv.org/html/2609.14207#S4.SS2.p2.1)\.
- Suhr and Artzi \(2023\)A\. Suhr and Y\. ArtziContinual learning for instruction following from realtime feedback\.InProceedings of the 37th International Conference on Neural Information Processing Systems,NIPS ’23,Red Hook, NY, USA\.Cited by:[§5](https://arxiv.org/html/2609.14207#S5.p1.1)\.
- Suttonet al\.\(1999\)R\. S\. Sutton, D\. McAllester, S\. Singh, and Y\. MansourPolicy gradient methods for reinforcement learning with function approximation\.InProceedings of the 13th International Conference on Neural Information Processing Systems,NIPS’99,Cambridge, MA, USA,pp\. 1057–1063\.Cited by:[§5](https://arxiv.org/html/2609.14207#S5.p8.1)\.
- Tanget al\.\(2024\)Z\. Tang, L\. Mao, and A\. SuhrGrounding language in multi\-perspective referential communication\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 19727–19741\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.1100/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.1100)Cited by:[§1](https://arxiv.org/html/2609.14207#S1.p2.1),[§3](https://arxiv.org/html/2609.14207#S3.p2.1)\.
- Willemsen and Skantze \(2024\)B\. Willemsen and G\. SkantzeReferring expression generation in visually grounded dialogue with discourse\-aware comprehension guiding\.InProceedings of the 17th International Natural Language Generation Conference,S\. Mahamood, N\. L\. Minh, and D\. Ippolito \(Eds\.\),Tokyo, Japan,pp\. 453–469\.External Links:[Link](https://aclanthology.org/2024.inlg-main.38/),[Document](https://dx.doi.org/10.18653/v1/2024.inlg-main.38)Cited by:[§3](https://arxiv.org/html/2609.14207#S3.p2.1)\.
- Xuet al\.\(2015\)J\. Xu, L\. Mukherjee, Y\. Li, J\. Warner, J\. M\. Rehg, and V\. SinghGaze\-enabled egocentric video summarization via constrained submodular maximization\.InProceedings of the IEEE conference on computer vision and pattern recognition,pp\. 2235–2244\.Cited by:[§3](https://arxiv.org/html/2609.14207#S3.p3.1)\.
- Yinet al\.\(2024\)P\. Yin, J\. Wang, G\. Zeng, D\. Xie, and J\. ZhuLg\-gaze: learning geometry\-aware continuous prompts for language\-guided gaze estimation\.InEuropean Conference on Computer Vision,pp\. 1–17\.Cited by:[§3](https://arxiv.org/html/2609.14207#S3.p4.1)\.
- Yuet al\.\(2017\)Y\. Yu, J\. Choi, Y\. Kim, K\. Yoo, S\. Lee, and G\. KimSupervising neural attention models for video captioning by human gaze data\.In2017 IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§3](https://arxiv.org/html/2609.14207#S3.p3.1)\.

## Appendix AGaze prediction details and evaluation

### A\.1Point format augmentation

For the Molmo\-based architectures, we augment the standard point format to exclude alternate text and quotation marks to ensure simplicity during reward calculation\. The Molmo point formatting

> <point x=”50\.0” y=”50\.0” alt=”cat”\>cat</point\><\\text\{point x="50\.0" y="50\.0" alt="cat"\}\>\\text\{cat\}<\\text\{/point\}\>

is simplified during finetuning the gaze predictor to:

> <x=50\.0 y=50\.0\><\\text\{x=50\.0 y=50\.0\}\>

This formatting change also applies to other finetuned listener models trained for ablations in following appendices\.

### A\.2Example of gaze prediction

![Refer to caption](https://arxiv.org/html/2609.14207v1/gazepred_0330.png)Figure 5:Example of the iterative gaze prediction for gold and finetuned Molmo\. The orange sequence represents the human scanpath and the blue is that produced by our gaze predictor model\. Each point indicates the location of the gaze and the word heard at the time the fixation was recorded\.
### A\.3Full gaze prediction results

Below are the full evaluation results for our gaze predictor and ART\([Mondal et al\., 2024](https://arxiv.org/html/2609.14207#bib.bib1)\)\. We include the distance to the gold point both averaged over all gold scanpaths and using the closest gold scanpath at each word index and total\. In both cases, the VLM\-based predictor is closer to the human scanpath than ART\.

Table 2:Average Euclidean distance per word index and total sum of distances, averaged over all gold sequences\.Table 3:Average Euclidean distance per word index and total sum of distances, using the closest gold path \(Best Match\)\.
### A\.4Predicting gaze with human and machine\-generated references

Since the gaze prediction model is trained only on human\-generated references, we analyze differences in gaze prediction accuracy between the gold references and machine\-generated references resembling those the model would see at the start of training\. To do so, we sample 100 examples from each of the four evaluation datasets and generate scanpaths for these 400 examples using the gold references for each as well as references generated by Molmo\-7B with no additional training\.

Across human and machine\-generated references, the gaze predictor achieves higher accuracy and fewer null points on shorter references, although it is difficult to disentangle the length or speaker from the difficulty of the produced references\. The difference in performance definitely influences the resulting speaker policies trained with the gaze listener\. However, it is intentional that the listener used during speaker training reflects a human listener’s reference resolution process, in order to result in a speaker that generates references that are easy for a human listener to understand\. The Molmo REC listener excels at reference resolution, in that it is trained to understand references that humans cannot \(i\.e\., coordinates\)\. Our results show that a listener trained on the kinds of expressions people use, and the processes involved in resolving such references, results in the production of more humanlike references\.

Figure 6:\(A\) Comparison of length distributions of human and Molmo\-generated references, \(B\) accuracy of the gaze predictor on human\-generated \(blue\) and Molmo\-generated references \(red\) over different reference lengths, and \(C\) comparison of the proportion of words not assigned fixations\.

## Appendix BFull REG results and evaluator choice

### B\.1Full REG results

In this section, we report the complete dataset\-level evaluation for all speaker variants\. Table[4](https://arxiv.org/html/2609.14207#A2.T4)details the REF performance metrics: accuracy, average utterance length, average time for a human listener to resolve the expression, anddπsd\_\{\\pi^\{s\}\}\. Additionally, Table[5](https://arxiv.org/html/2609.14207#A2.T5)presents behavioral metrics— resolution time, reveal rate, and early click rate\. Finally, we present the averaged and per\-dataset results using the QwenVL listener in Tables[6](https://arxiv.org/html/2609.14207#A2.T6)and[7](https://arxiv.org/html/2609.14207#A2.T7), respectively\.

SpeakerAcc\.↑\\uparrowLengthTime \(s\)↓\\downarrowdπsd\_\{\\pi^\{s\}\}↓\\downarrowRefCOCO testAHuman \(Gold\)95\.673\.593\.970\.00Molmo84\.1715\.157\.991\.41REC\-BeforeFirstHit68\.502\.014\.232\.37REC\-SeqAnyHit93\.1725\.2711\.401\.89REC\-Success93\.5018\.589\.211\.31REC\-Shaping81\.175\.275\.301\.27Gaze\-SeqAnyHit87\.839\.876\.350\.87Gaze\-SeqLPHit85\.0011\.538\.941\.15Gaze\-Shaping79\.834\.364\.521\.38Gaze\-BeforeFirstHit79\.004\.485\.121\.45RefCOCO testBHuman \(Gold\)96\.173\.734\.450\.00Molmo65\.5013\.848\.191\.41REC\-BeforeFirstHit58\.171\.933\.991\.25REC\-SeqAnyHit83\.8323\.5611\.982\.00REC\-Success83\.6716\.908\.761\.37REC\-Shaping67\.003\.785\.200\.95Gaze\-SeqAnyHit77\.338\.847\.130\.80Gaze\-SeqLPHit71\.3310\.088\.041\.02Gaze\-Shaping67\.334\.065\.440\.94Gaze\-BeforeFirstHit66\.834\.045\.170\.96RefOI Co\-occurrenceHuman \(Gold\)84\.835\.024\.070\.00Molmo62\.8317\.009\.451\.41REC\-BeforeFirstHit52\.001\.984\.081\.51REC\-SeqAnyHit72\.5024\.7011\.501\.74REC\-Success74\.6718\.368\.661\.20REC\-Shaping59\.004\.215\.511\.18Gaze\-SeqAnyHit65\.839\.247\.240\.93Gaze\-SeqLPHit65\.1710\.487\.781\.00Gaze\-Shaping64\.333\.635\.350\.94Gaze\-BeforeFirstHit58\.333\.865\.281\.21RefOI Single PresenceHuman \(Gold\)93\.501\.393\.280\.00Molmo88\.3315\.577\.571\.41REC\-BeforeFirstHit88\.331\.983\.731\.00REC\-SeqAnyHit90\.6724\.0010\.261\.69REC\-Success92\.0018\.688\.511\.25REC\-Shaping88\.334\.454\.141\.02Gaze\-SeqAnyHit89\.009\.946\.671\.06Gaze\-SeqLPHit89\.3312\.078\.241\.10Gaze\-Shaping90\.333\.894\.440\.64Gaze\-BeforeFirstHit88\.834\.294\.120\.93Table 4:Full human evaluation results by dataset\.
### B\.2Training curves

Fig[B\.2](https://arxiv.org/html/2609.14207#A2.SS2)shows the train and validation reward, success rate of the reference \(i\.e\., whether any point in the set of produced point was within the target region\), and generated reference length in tokens\. The shaded area represents the range of values over three random seeds\. The rewards and success rates for gaze\-based rewards tend to be lower than their REC counterparts\. TheREC\-BeforeFirstHittraining is noticeably the most unstable relative to the other reward configurations\.

![[Uncaptioned image]](https://arxiv.org/html/2609.14207v1/main_results_4subplots.png)Table 5:Behavioral efficiency metrics grouped by dataset, including overall averages\.SpeakerAcc\.↑\\uparrowLengthdπsd\_\{\\pi^\{s\}\}↓\\downarrowRefCOCO testAHuman \(Gold\)94\.603\.510\.00Molmo48\.2015\.711\.41REC\-BeforeFirstHit40\.073\.031\.18REC\-SeqAnyHit60\.2025\.811\.97REC\-Success62\.8520\.271\.53REC\-Shaping51\.465\.630\.95Gaze\-SeqAnyHit49\.059\.181\.09Gaze\-SeqLPHit52\.8812\.351\.16Gaze\-Shaping48\.575\.191\.00Gaze\-BeforeFirstHit51\.935\.330\.93RefCOCO testBHuman \(Gold\)88\.003\.780\.00Molmo33\.5015\.961\.41REC\-BeforeFirstHit30\.873\.141\.05REC\-SeqAnyHit46\.3224\.521\.87REC\-Success50\.0219\.061\.44REC\-Shaping36\.674\.980\.95Gaze\-SeqAnyHit37\.828\.691\.01Gaze\-SeqLPHit43\.0911\.271\.03Gaze\-Shaping39\.014\.730\.90Gaze\-BeforeFirstHit38\.995\.010\.90RefOI Co\-occurrenceHuman \(Gold\)92\.905\.340\.00Molmo33\.2016\.461\.41REC\-BeforeFirstHit35\.623\.080\.98REC\-SeqAnyHit49\.2225\.811\.98REC\-Success54\.2819\.421\.42REC\-Shaping40\.554\.970\.88Gaze\-SeqAnyHit42\.408\.530\.89Gaze\-SeqLPHit44\.8111\.140\.96Gaze\-Shaping43\.474\.670\.83Gaze\-BeforeFirstHit44\.314\.940\.81RefOI Single PresenceHuman \(Gold\)80\.401\.450\.00Molmo67\.7017\.271\.41REC\-BeforeFirstHit78\.182\.950\.20REC\-SeqAnyHit76\.0825\.041\.53REC\-Success80\.6220\.071\.18REC\-Shaping79\.815\.090\.23Gaze\-SeqAnyHit77\.859\.090\.52Gaze\-SeqLPHit76\.2212\.840\.79Gaze\-Shaping74\.394\.940\.52Gaze\-BeforeFirstHit81\.575\.330\.26Table 6:QwenVL\-based evaluation results by dataset\.Table 7:QwenVL\-based evaluation results averaged across all datasets\.
### B\.3Comparison of machine listener models

We choose to use QwenVL for our main evaluation despite[Ma et al\. \(2025\)](https://arxiv.org/html/2609.14207#bib.bib4)finding CogVLM\-Grounding to be the best model for REC\. As shown in[7](https://arxiv.org/html/2609.14207#A2.F7), the reference resolution rate of CogVLM\-Grounding becomes extremely low as references get longer, whereas QwenVL maintains higher success rates across reference lengths\. We hypothesize that this difference reflects a training bias in CogVLM\-Grounding toward concise, human\-written references\.

![Refer to caption](https://arxiv.org/html/2609.14207v1/img/efficiency_qwen_vs_cogvlm.png)Figure 7:Comparison of REC results with QwenVL and CogVLM\-Grounding\.

## Appendix CSpeaker model ablations, baselines, and comparisons

### C\.1KL penalty ablation

We demonstrate the utility of the KL penalty on the highest performing speaker model with the QwenVL listener \(Gaze\-BFH\) by ablating it\. Although training without the KL penalty yields higher validation rewards late in training, it causes entropy collapse and degrades overall evaluation performance \(Figure[8](https://arxiv.org/html/2609.14207#A3.F8)\)\. With reference length collapsing to just 1–2 tokens on average, these results confirm that the KL penalty is necessary to mitigate reward hacking\. As a result, we apply the KL penalty to the training of all speaker models\.

![Refer to caption](https://arxiv.org/html/2609.14207v1/img/kl_comparison_plots.png)Figure 8:Comparison ofGaze\-BFHtraining performance with and without the KL penalty\. Red shows training with a KL penalty \(β=0\.02\\beta=0\.02\) and blue shows the ablated training run\. For comparison, we include Molmo and human speakers in the rightmost chart\.
### C\.2REC\-Success with length penalty

While the goal of this work is not to demonstrate that learning from gaze results in the most efficient and successful references, we compare toREC\-Successincorporating a length penalty to control for verbosity\. Because policy optimization can favor longer sequences to maximize reward, penalizing length allows for a fair comparison at closer sequence lengths\. We implement the length penalty as a negative reward,λ=0\.05\\lambda=0\.05\. The penalty is applied after batch normalization to each generated token with the exception of the end\-of\-sequence token\. The training objective becomes:

∇θ𝒥​\(θ\)=𝔼\(ℐ,b,x,t\)​\(rt−β​KL​\(πθS,ℐ,b,x,t\)⏟KL penalty−λ​𝕀​\(xt≠eos\)⏟Length penalty\)×∇θ\(log⁡πθS​\(xt∣ℐ,b,x<t\)\)⏟Policy gradient\+α​∇θH​\(πθS​\(xt∣ℐ,b,x<t\)\)⏟Entropy bonus,\\nabla\_\{\\theta\}\\mathcal\{J\}\(\\theta\)=\\mathbb\{E\}\_\{\(\\mathcal\{I\},b,x,t\)\}\\Big\(r\_\{t\}\-\\underbrace\{\\beta\\text\{KL\}\\left\(\\pi^\{\\text\{S\}\}\_\{\\theta\},\\mathcal\{I\},b,x,t\\right\)\}\_\{\\text\{KL penalty\}\}\-\\underbrace\{\\lambda\\mathbb\{I\}\(x\_\{t\}\\neq\\text\{\{eos\}\}\)\}\_\{\\text\{Length penalty\}\}\\Big\)\\\\ \\times\\underbrace\{\\nabla\_\{\\theta\}\\left\(\\log\\pi^\{\\text\{S\}\}\_\{\\theta\}\\left\(x\_\{t\}\\mid\\mathcal\{I\},b,x\_\{<t\}\\right\)\\right\)\}\_\{\\text\{Policy gradient\}\}\+\\underbrace\{\\alpha\\nabla\_\{\\theta\}H\\left\(\\pi^\{\\text\{S\}\}\_\{\\theta\}\(x\_\{t\}\\mid\\mathcal\{I\},b,x\_\{<t\}\)\\right\)\}\_\{\\text\{Entropy bonus\}\}\\;,\(1\)where𝕀⁡\(⋅\)\\mathbb\{I\}\(\\cdot\)is the indicator function\.

SpeakerAcc\.↑\\uparrowLengthdπsd\_\{\\pi^\{s\}\}↓\\downarrowOverall \(Averaged\)Molmo45\.6516\.351\.41REC\-BeforeFirstHit46\.183\.050\.99REC\-SeqAnyHit57\.9625\.291\.84REC\-Success61\.9419\.701\.41REC\-Shaping52\.125\.170\.86Gaze\-SeqAnyHit51\.788\.870\.95Gaze\-SeqLPHit54\.2511\.901\.03Gaze\-Shaping51\.364\.880\.87Gaze\-BeforeFirstHit54\.205\.150\.81Human \(Gold\)88\.973\.520\.00Length Penalty \(λ=0\.05\\lambda=0\.05\)REC\-Success \(LPλ=0\.05\\lambda=0\.05\)61\.089\.500\.83Table 8:QwenVL\-based evaluation comparing standard models against the length\-penalizedREC\-Successvariant\.The speaker trained withREC\-Succesand a length penalty outperforms almost all other configurations, but falls just short ofGaze\-BeforeFirstHit\. Explicit length penalization successfully reduces verbosity, but gaze\-based feedback can also encourage concise, pragmatically optimal expressions without relying on manually tuned penalty weights\.

### C\.3Listener variant vaselines

#### C\.3\.1Final gaze point listener

To ensure the brevity of gaze\-trained speakers is driven by the gaze signal and reward design rather than training data bias, we evaluate a speaker trained with a variant ofπG\\pi^\{G\}that predicts only the final fixation point\. We then integrate this model with theREC\-Successtraining configuration\. The resulting speaker model results in length and accuracy similar to that ofREC\-Successand even slightly longer, demonstrating that the gaze listener’s training distribution is not the cause of the shorter references\.

Table 9:Evaluation comparing speakers trained on full incremental gaze versus models predicting only the final gaze point\.
#### C\.3\.2RefCOCO finetuned listener

To ensure the gaze listener does not outperform the VLM listener solely through indirect exposure to RefCOCO grounding data, we finetune Molmo\-7B on the RefCOCO subset within the RefCOCO\-Gaze training set for a fair comparison\. Then, similarly to the prior baselines, we train a speaker against this listener withREC\-Successrewards\. Once again, we find that exposure to RefCOCO does not result in higher accuracy or shorter references\.

Table 10:Performance comparison when training the speaker against a listener finetuned directly on RefCOCO\.

### C\.4Comparison to SFT

To establish an upper bound, we finetune Molmo\-7B\-D directly on the RefCOCO validation set \(8\.8k examples\)\. We consider 1%, 10%, and 100% of this train set\. Each model is trained for 4 epochs at learning rate 5e\-5\. As expected, direct supervision on human\-written references yields strong performance\. We additionally try training from the RefCOCO 1% checkpoint with theGaze\-ShapingandGaze\-SeqAnyHitreward setups; however, this degrades performance from the SFT checkpoint\.

SpeakerAcc\.↑\\uparrowLengthdπsd\_\{\\pi^\{s\}\}↓\\downarrowOverall \(Averaged\)Molmo45\.6516\.351\.41REC\-BeforeFirstHit46\.183\.050\.99REC\-SeqAnyHit57\.9625\.291\.84REC\-Success61\.9419\.701\.41REC\-Shaping52\.125\.170\.86Gaze\-SeqAnyHit51\.788\.870\.95Gaze\-SeqLPHit54\.2511\.901\.03Gaze\-Shaping51\.364\.880\.87Gaze\-BeforeFirstHit54\.205\.150\.81Human \(Gold\)88\.983\.520\.00Comparison to SFT BaselinesRefCOCO SFT 100%64\.443\.340\.59RefCOCO SFT 10%64\.563\.170\.59RefCOCO SFT 1%60\.273\.280\.71RefCOCO SFT 1% \+ Shaping Gaze55\.213\.790\.84RefCOCO SFT 1% \+ Binary Gaze59\.547\.340\.78Table 11:Comparison of Supervised Finetuning \(SFT\) baselines and SFT\+RL hybrid models against the default RL/Gaze speakers\.
### C\.5Generalization to other VLMs

We recreate a subset of the main results, instead training PaliGemma\-3B\-pt\-448\([Beyer et al\., 2024](https://arxiv.org/html/2609.14207#bib.bib10)\)and LLaVA\-1\.5\-7B\([Liu et al\., 2024](https://arxiv.org/html/2609.14207#bib.bib46)\)as speaker models\. This subset includesGaze\-Shaping,Gaze\-BeforeFirstHit,Gaze\-SeqAnyHit,REC\-Shaping,REC\-BeforeFirstHit,REC\-SeqAnyHit, andREC\-Success\.

Table[12](https://arxiv.org/html/2609.14207#A3.T12)reports results; we observe that gains do not cleanly generalize across VLMs\. Because hyperparameters were held constant from the Molmo\-7B experiments, we attribute this to a lack of model\-specific tuning, especially for the KL penalty coefficient \(β\\beta\), and differences in base models and their initial policy entropy\.

SpeakerAcc\.↑\\uparrowLengthdπsd\_\{\\pi^\{s\}\}↓\\downarrowHuman \(Gold\)88\.983\.520\.00Molmo\-7BMolmo45\.6516\.351\.41REC\-BeforeFirstHit46\.183\.050\.99REC\-SeqAnyHit57\.9625\.291\.84REC\-Success61\.9419\.701\.41REC\-Shaping52\.125\.170\.86Gaze\-SeqAnyHit51\.788\.870\.95Gaze\-SeqLPHit54\.2511\.901\.03Gaze\-Shaping51\.364\.880\.87Gaze\-BeforeFirstHit54\.205\.150\.81LLaVA\-1\.5\-7BLLaVA\-1\.528\.326\.161\.59REC\-BeforeFirstHit37\.081\.921\.34REC\-Shaping44\.402\.041\.14REC\-Success53\.0312\.061\.12Gaze\-BeforeFirstHit37\.081\.921\.31Gaze\-Shaping20\.513\.191\.79Gaze\-SeqAnyHit39\.483\.021\.27PaliGemma\-3BPaliGemma31\.803\.081\.48REC\-BeforeFirstHit41\.290\.961\.24REC\-Shaping41\.080\.961\.24REC\-SeqAnyHit34\.1726\.092\.28REC\-Success42\.921\.551\.19Gaze\-BeforeFirstHit32\.841\.051\.46Gaze\-Shaping29\.380\.921\.56Gaze\-SeqAnyHit35\.118\.001\.43Table 12:Cross\-architecture evaluation using LLaVA\-1\.5\-7B and PaliGemma\-3B as speaker backbones\.

## Appendix DQuality and qualitative analyses

### D\.1Strategy analysis

To consider not only the similarity of length and accuracy to human\-generated references, but also stylistic choices, we perform a simple analysis of referring expression strategy\. Referring expressions are often separated into spatial or attribute based references; using keyword lists, we show the proportion of references that use spatial or attribute based keywords, both, or neither\. The keywords used for each category were:

- •Spatial Keywords:left, right, top, bottom, middle, center, above, below, next, near, far, between, behind, front, centered, side, corner, upper, lower, background, foreground, back, overhead, touching, overlapping, contacting, abutting, alongside, bordering, nearest, farthest, furthest, closest, distant, adjacent, beside, inside, outside, within, interior, exterior, embedded, surrounded, among, amongst, amid, under, underneath, beneath, atop, upon, over, edge, margin, boundary, perimeter, diagonal, aligned, stacked, facing, opposite, rear, first, second, third, last, outer, inner, innermost, outermost
- •Attribute Keywords:red, blue, green, yellow, orange, purple, pink, brown, black, white, gray, grey, silver, gold, tan, beige, maroon, navy, teal, cyan, magenta, violet, dark, light, bright, pale, deep, vivid, transparent, translucent, opaque, multicolored, colorful, neon, turquoise, indigo, cream, charcoal, blonde, brunette, auburn, large, small, big, little, tall, short, long, round, circular, square, rectangular, triangular, oval, spherical, cylindrical, flat, curved, pointed, straight, crooked, hollow, solid, narrow, wide, thick, thin, slender, heavy, metal, metallic, wooden, plastic, glass, leather, denim, cotton, woolen, ceramic, stone, brick, concrete, paper, smooth, rough, soft, hard, fluffy, hairy, furry, shiny, glossy, matte, rusty, wet, dry, striped, checkered, patterned, spotted, dotted, printed, wearing, holding, carrying, sitting, standing, walking, running, smiling, lying, reclining, leaning, crouching, kneeling, jumping, riding, flying, swimming, eating, drinking, looking, pointing, reading, sleeping, driving, hanging, attached, mounted, folded, elderly, young, old, new, broken, dirty, clean, damaged, stained, burnt, full, empty, open, closed, bare, decorated, painted, covered, single, double, pair, couple, group, crowd, several, few, many

A qualitative observation of references produced by Molmo without any additional training was that many of them started by restating the prompt by mentioning the target region identifier — in this case, the ”red box”\. This is a clear indicator that the models are not pragmatic, as they can not distinguish whether this information is useful for a listener\. We investigate which reward setups successfully decrease the usage of such prefixes in Fig[9](https://arxiv.org/html/2609.14207#A4.F9)\.

Figure 9:Mention frequency of the red bounding box across speaker models\.
### D\.2Reference quality annotations

We perform a small scale analysis of the quality and ambiguity of references\. On a subset of reward configurations, one author annotated a subset of 50 image\-reference pairs, sampled from each dataset as either well\-specified, ambiguous, or incorrect\. Configurations that encourage longer references such asREC\-SuccessandGaze\-SeqAnyHitwere more likely to be well\-specified\.Gaze\-Shapingencourages shorter references, but also produces more ambiguous references, demonstrating the tradeoff between accuracy and efficiency\.

![Refer to caption](https://arxiv.org/html/2609.14207v1/img/information_analysis_plot.png)Figure 10:Analysis of ambiguity and incorrect reference rates across four selected speaker models\.
### D\.3Qualitative behavior and error discussion

![Refer to caption](https://arxiv.org/html/2609.14207v1/examples_comparison_full_page.png)Figure 11:Examples of generated references from all speaker models\.
### D\.4Gaze retention analysis

Gaze\-BFHis the best performing model with machine listener, but it falls short with human listeners\. Thus, we investigate why this might be the case\. While not displayed in the examples in Fig[11](https://arxiv.org/html/2609.14207#A4.F11), we suspect theGaze\-BFHconfiguration underperforms with human listeners due to its tendency to reward false positives\. To corroborate this hypothesis, we analyze the post\-hit target retention rate across speaker models, defined as the conditional probability that a gaze trajectory \(or sequence of REC points\) remains within the target bounding box after initial entry\. Analysis results are shown in Table[13](https://arxiv.org/html/2609.14207#A4.T13)\. Specifically, we evaluate the generated point sequences from the validation episodes of each model’s best checkpoint \(the checkpoint selected for evaluation\)\. Overall, the gaze predictor is more prone to exiting the target region than the REC model \(Molmo\) since Molmo is trained for REC rather than realistic scanpaths\.Gaze\-BFHdisplays the worst retention rate of all speaker models\. The reward setup does not encourage references that keep the listener’s gaze in the bounding box\.Gaze\-SeqAnyHitalso displays very poor retention rate, but likely does not suffer during human evaluation due to longer generations including more information\.

Table 13:Gaze trajectory retention metrics across speaker model configurations, demonstrating post\-hit retention across reward setups\.
### D\.5Punctuation and spacing errors

We observe that some of the configurations result in list\-like references of many short sentences \(e\.g\.,Gaze\-SeqLPHitin Fig[11](https://arxiv.org/html/2609.14207#A4.F11)\)\. Additionally,REC\-Shapinginserts extra spaces\. To quantify these phenomena, we record the double\-space rate of each speaker model along with the average number of periods in each reference\. The latter metric is based on the assumption that most references are single sentences or phrases\. In Table[14](https://arxiv.org/html/2609.14207#A4.T14),Gaze\-SeqLPHit,Gaze\-SeqAnyHit, andREC\-SeqAnyHitare particularly prone to inserting extra periods\. OnlyREC\-Shapingdevelops the double space behavior; sometimes, the speaker generates spaces between tokens that make up a single word, affecting the speech synthesis during human evaluation\. This exemplifies how the human evaluation accounts for behaviors that are otherwise difficult to quantify\. We hypothesize both of these behaviors develop to exploit the listener model by giving it extra chances to resolve the reference through adding extra tokens without any additional semantic value\.

Speaker ModelMid\-Ref Period Rate \(%\)↓\\downarrowDouble Space Rate \(%\)↓\\downarrowMean PeriodsMolmo18\.580\.001\.22REC\-BeforeFirstHit0\.780\.000\.02REC\-Shaping6\.9353\.170\.84REC\-Success17\.120\.001\.20REC\-SeqAnyHit39\.310\.001\.46Gaze\-BeforeFirstHit0\.520\.000\.04Gaze\-Shaping4\.670\.000\.42Gaze\-SeqLPHit53\.540\.002\.87Gaze\-SeqAnyHit28\.570\.001\.82Human0\.460\.320\.02Table 14:Frequency of mid\-reference period punctuation, double spacing, and mean period counts across speaker model configurations\.
### D\.6Coordinate generation

Finally, we occasionally observe references that are simply coordinates inREC\-BFH\(Table[15](https://arxiv.org/html/2609.14207#A4.T15)\)\. Although not incredibly prevalent, this case highlights how using non human\-like listeners can result in reward\-hacking by creating references that exploit the machine listener\.

Table 15:Frequency of coordinate artifact occurrences across speaker model outputs\.

## Appendix EHuman evaluation details

We recruit 751 participants on Prolific who are fluent English speakers in the United States\. Participants receive $2 USD per session, each of which takes an average of 6\.1 minutes\. In each session, a participant completes up to 40 trials, with each trial corresponding to an image paired with a referring expression\. We record all mouse movements, including clicks; only after the audio has finished playing will a click on the image move the participant to the next trial\. Five of the 40 trials include human\-written referring expressions; participants are provided a bonus of $0\.50 USD if they successfully identify the target referent for all 5 human\-written references without revealing the reference text\. This bonus structure both incentivizes participants to perform well as a listener, and discourages them from revealing the reference text when unnecessary\. To reduce spam and low\-effort sessions, we discard all data from participants who do not successfully identify at least 4 of the 5 human\-written references, resulting in a final pool of 633 participants\.

### E\.1Experimental instructions

The following instructions were provided to Prolific participants prior to beginning the reference game:

Experimental Instructions \(Verbatim\)Welcome\!This task consists of 40 or less trials\. When each trial begins, you will see an image, and hear a description read aloud\. Your job is to click on the region of the image that best matches the description\.Bonus\!We will check your answers on a subset of trials with unambiguous descriptions\. Participants who get 100% on this subset will receive a $0\.50 bonus\.Need to see the description?If the speech description is unclear, you have the option to reveal the description text with a button below the image\. This button will only appear after the speech description finishes\.However, if you reveal the description for examples from the unambiguous subset, you will not receive the bonus, so only reveal it when you truly need it\.What if you don’t know?There will be some ambiguous descriptions\. If you are unsure, just make your best guess\!Ok, ready to begin?Please make sure your volume is turned up\. The task will begin on the next page\.