Kolab：长周期工作流中的人类与智能体协作基准评测

# Kolab: Benchmarking Human-Agent Collaboration in Long-Horizon Workflows
Source: [https://arxiv.org/html/2610.21109](https://arxiv.org/html/2610.21109)
Patricia Anugrah SetianiAffiliation:Department of Computer ScienceEmail:[dongyeop@umn\.edu](mailto:)Ruizi WangAffiliation:Department of Computer ScienceZhouyi LiAffiliation:Department of EconomicsUniversity of MinnesotaKhanh Chi LeAffiliation:Department of Computer ScienceYoung\-Jun Lee & Dongyeop KangAffiliation:Department of Computer Science

###### Abstract

When humans collaborate with AI agents on long\-horizon tasks, what happens to the agents’ contributions? For solving long\-horizon tasks, humans place greater importance on ongoing interaction with AI agents \(e\.g\.,revising plans\) than on an agent’s ability to complete the task in a single attempt\. However, most agentic benchmarks have focused more on automation\. Strong performance on such benchmarks does not imply effective real\-world collaboration\. Professional workflows offer a useful setting, as they often involve interdependent steps\. To address this gap, we introduceKolab, a new dataset of 115\+ hours of screen recordings of 50 professionals from 40 occupations collaborating with agents on realistic, long\-horizon problems\.Kolabalso contains participants’ collaboration and output quality judgments\. Our analysis reveals diverse collaboration patterns, highlighting the importance of when and how professionals coordinate with agents\. A collaborative agent must do two things that automation benchmarks never test: anticipate whether experts will adopt, modify, or reject its contribution once it leaves the chat window to guide its next steps, and assess collaboration quality from the unfolding workflow without constant expert feedback\. Our benchmark operationalizes both: \(1\) forecasting how professionals use agent contributions at diverse workflows, and \(2\) verifying their judgments of collaboration satisfaction and output quality\. Findings show that frontier models, both agentic and non\-agentic, overestimate collaboration with experts and struggle to distinguish between high\- and low\-quality collaboration\. Longer workflow context does not consistently improve these predictions\. As frontier agents cannot yet understand long\-horizon collaboration with experts, we positionKolabas a valuable dataset for studying human–agent collaboration grounded in real professional workflows\.111Code and data are available at[https://minnesotanlp\.github\.io/kolab/](https://minnesotanlp.github.io/kolab/)

## 1Introduction

As AI agents become more capable of complex reasoning and knowledge work, recent research has examined whether they can complete such tasks autonomously in a single attempt, given a single instruction\([Patwardhan et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib20);[Vidgen et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib24);[Zhu et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib2);[Sun et al\., 2026b](https://arxiv.org/html/2610.21109#bib.bib7);[Merrill et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib8);[Xu et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib16);[Xie et al\., 2024](https://arxiv.org/html/2610.21109#bib.bib26)\)\. However, real\-world problem solving may not follow this autonomous setting: human goals can evolve, earlier decisions shape subsequent work, and errors may require intervention before they propagate\([Tack et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib30)\)\. In professional work, for example, workers formulate requirements, produce intermediate artifacts, revise decisions, and move between agents and domain\-specific tools\. Thus, strong autonomous task performance may not translate into effective human\-agent collaboration\([Zou and others, 2025](https://arxiv.org/html/2610.21109#bib.bib21);[Mozannar et al\., 2025](https://arxiv.org/html/2610.21109#bib.bib23);[Shen et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib5)\)\. Despite the need for studying effective human\-agent collaboration, existing datasets provide limited grounding in real collaborative workflows\. They are often not publicly released, rely on predefined tasks constructed or sourced by researchers, or focus on a single professional domain such as software development\([Wang et al\., 2025](https://arxiv.org/html/2610.21109#bib.bib15);[Chen et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib1);[Shao et al\., 2026b](https://arxiv.org/html/2610.21109#bib.bib6)\)\. As a result, benchmarking long\-horizon human\-agent collaboration workflows remains underexplored\.

To address this gap, we introduceKolab, a dataset of screen recordings capturing human\-agent collaboration on long\-horizon professional problems\. We represent each workflow as a graph of semantically meaningfulhumanandagentactions and their dependencies \(Figure[1](https://arxiv.org/html/2610.21109#S1.F1)\), allowing us to understand when and how collaboration occurs in the workflow\. In this example, an architect completes a long\-horizon task in five steps, and agent outputs can influence later steps\. A task delegated in Step 2 \(source node\) produces information \(target AI node\) that is later modified and used in Step 3 \(Modify\)\. Human responses to agent outputs may also occur outside the chat interface\. The architect ultimately adopts and executes the resulting plan in Revit, an action not directly observable by the agent\.

![Refer to caption](https://arxiv.org/html/2610.21109v1/workflow_graph_main_fig.png)Figure 1:An example human\-agent collaborative workflow for an architect\. Each step is a subgraph ofhumanandagentactions; annotations link a source node, the target AI output it triggers, and the human response to it \(here,ModifyandAdopt\)\.Our analysis reveals that human\-agent collaboration workflow varies in when and how professionals engage with AI agents\. Burstier workflows are associated with longer periods of independent work before interacting with AI, suggesting that professionals may organize their collaboration around periods of independent work rather than continuous AI interaction\. Workflows with more concurrent human\-agent activities are associated with lower collaboration satisfaction, suggesting that parallel activities may introduce additional coordination costs\.

Beyond characterizing collaboration, we ask whether frontier multimodal large language models \(MLLMs\) and agents can*understand their own place*in a professional’s workflow\. We frame this as two capabilities that a deployed collaborative agent needs but that autonomous benchmarks never measure\.

Anticipating the fate of a contribution \(RQ1\)\.In long\-horizon work, an agent’s output is rarely the end of the story: it is adopted, modified, or rejected, often outside the chat interface \(e\.g\.,an architect incorporating an agent’s plan in Revit\)\. An agent that can*recognize*what happened to its output after the fact \(1a\) could turn naturalistic screen recordings into feedback signals for training and for auditing deployed agents, without asking the expert to label anything\. An agent that can*forecast*this response at delivery time \(1b\)—before the human reacts—could act proactively: asking a clarifying question when rejection is likely, flagging the parts it expects to be edited, or escalating to the human when its contribution carries risk that would propagate to later steps\([Oh et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib40)\)\. Because such forecasts depend on how much of the collaboration the agent remembers, we vary the available history from the preceding steps \(1c\) to the entire workflow so far \(1d\), which directly informs how much context a long\-running or computer\-use agent must retain to model its collaborator\.

Judging the collaboration itself \(RQ2\)\.Expert ratings of collaboration satisfaction \(2a\) and output quality \(2b\) are the ground truth for whether human\-agent collaboration works, yet they are expensive to collect and unavailable at deployment\. If an MLLM could recover these ratings from the workflow alone, it could serve as a process\-level reward model for training collaborative agents, as an automatic monitor that detects friction or low\-quality outcomes in deployed systems, and as a scalable evaluator for professional tasks where no reference answer exists\. Crucially, such a verifier must separate genuinely successful interactions from dissatisfying ones, rather than echoing the predominantly positive ratings\.

We observe that frontier multimodal agents still struggle to anticipate how professionals use agent outputs and to recover their judgments of collaboration quality\. For example, Gemini 3\.1 Pro and GPT\-6 achieve only 39\.1% and 51\.3% F1 in forecasting adoption behavior, while their predictions of collaboration satisfaction correlate weakly with professional ratings \(Spearman’sρ=\.08\\rho=\.08and\.38\.38\)\. Providing longer workflow context or additional modalities does not consistently improve these predictions\. Further analysis reveals systematic biases: models often favor certain adoption labels and overpredict high ratings, limiting their ability to distinguish successful from poor collaboration\. These findings suggest that current models may rely on their own prediction tendencies rather than reliably understanding professionals’ behaviors and judgments from the workflow\. These results highlight an important challenge for developing collaborative agents that can anticipate human decisions and adapt to how their contributions are actually used\.

Our main contributions are as follows\.

- •We presentKolab, a new dataset of 115 hours of screen\-recorded human\-agent workflows from 50 professionals solving their own real\-world problems, comprising 508 steps across 90\+ softwares\. Each workflow includes a final deliverable with step\-level human ratings of collaboration experience and output quality\.
- •We analyze human\-agent collaboration dynamics using three metrics: collaboration burstiness, workflow topology, and AI retention rate\.
- •We benchmark frontier MLLMs on their ability to understand human\-agent collaborative workflows when solving professional problems\.
- •We uncover systematic MLLM failure modes, revealing prediction biases that vary by model and input modality\.

## 2Related Work

Table 1:Comparison ofKolabwith long\-horizon, multi\-domain workflow benchmarks\. “H\-AI Collab\.” indicates the presence of human\-agent collaborative workflows; “Software Diversity” refers to tasks spanning multiple software applications; “Modalities” lists types of publicly released data\.##### Long\-horizon agent and workflow benchmarks\.

Autonomous agent benchmarks cover desktop applications\([Xie et al\., 2024](https://arxiv.org/html/2610.21109#bib.bib26);[Yuan et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib27)\), enterprise software\([Dai et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib25)\), hybrid GUI/CLI and scientific computer use\([Li et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib13);[Jang et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib14);[Wang et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib18);[Zhu et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib2);[Sun et al\., 2026a](https://arxiv.org/html/2610.21109#bib.bib37)\), professional deliverables\([Patwardhan et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib20);[Sun et al\., 2026b](https://arxiv.org/html/2610.21109#bib.bib7);[Xu et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib16)\), and long\-running agent behavior\([Maldaner et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib19)\)\. Some include human baselines but do not release human action trajectories\. Workflow\-GYM\([Zhu et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib2)\)collects expert workflow trajectories but uses them as demonstrations to evaluate autonomous agents\. Human\-only process datasets such as ScholaWrite\([Le et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib32)\)and LawFlow\([Das et al\., 2025](https://arxiv.org/html/2610.21109#bib.bib31)\)capture expert work in a single domain \(scholarly writing and legal reasoning\) without agent collaboration\. Among the benchmarks in Table[1](https://arxiv.org/html/2610.21109#S2.T1),Kolabuniquely combines multimodal trajectories of professionals collaborating with agents on self\-defined problems with step\-level human ratings\.

##### Human\-agent collaborative workflows\.

Prior work argues that autonomous task performance does not ensure effective collaboration\([Shen et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib5)\), and that the role humans play relative to an AI system shapes the interaction\([Wan et al\., 2022](https://arxiv.org/html/2610.21109#bib.bib43)\); indeed, varying LLM access measurably changes how people work\([Christenson et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib42)\)\. Existing evaluations often rely on simulated users\([Wu et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib4);[Yao et al\., 2024](https://arxiv.org/html/2610.21109#bib.bib10);[Vijayvargiya et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib11);[Qian et al\., 2025](https://arxiv.org/html/2610.21109#bib.bib12)\)or human intervention in short shopping and web browsing tasks\([Huq et al\., 2025](https://arxiv.org/html/2610.21109#bib.bib3);[Mozannar et al\., 2025](https://arxiv.org/html/2610.21109#bib.bib23)\)\. Collaborative Gym\([Shao et al\., 2026a](https://arxiv.org/html/2610.21109#bib.bib17)\)supports real or simulated users but covers only three predefined tasks\. CollabSkill\([Shao et al\., 2026b](https://arxiv.org/html/2610.21109#bib.bib6)\)assigns professionals problems from existing datasets;[Chen et al\. \(2026\)](https://arxiv.org/html/2610.21109#bib.bib1)study interaction\-level satisfaction with software\-engineering agents; and[Wang et al\. \(2025\)](https://arxiv.org/html/2610.21109#bib.bib15)compare separate human and agent workflows\. Our study setup is most similar to that of[Hayati et al\. \(2026\)](https://arxiv.org/html/2610.21109#bib.bib28)who investigate human\-agent collaboration in interdisciplinary research using only chat logs\. Meanwhile, our work also captures the complete workflow, including how professionals act on agent contributions in their work that may happen outside the chat interface\.

## 3Kolab: A Multimodal, Multidomain Dataset of Human\-Agent Collaborative Workflows on Long\-Horizon Problems

Kolabis built to understand human\-agent collaborative workflows when humans solve long\-horizon problems\.

Problem criteria\.To ensure that the collected workflows represent substantive professional problem solving rather than generic computer\-use activities, we required each problem to satisfy four criteria:

- •Real\-world\.The problem must originate from the professional’s actual work\. Professionals were not given tasks from existing workflow datasets or scenarios\.
- •Domain\-specific\.The problem must require the application of professional knowledge, skills, or judgment from the participant’s field\. Tasks that could be completed without domain expertise, such as routine personal activities or generic information seeking, were excluded\.
- •Complex and long\-horizon\.The problem must require a sequence of interdependent steps to achieve its final goal\. Participants were asked to plan a workflow requiring at least one hour of work and consisting of at least five steps, with each step involving multiple computer actions or decisions\.
- •Deliverable\-oriented\.The problem must result in a concrete final deliverable\. Each step must also produce an intermediate output that can be evaluated\.

![Refer to caption](https://arxiv.org/html/2610.21109v1/occupation_bars_arxiv.png)Figure 2:Kolabdomain distributions\.Data collection\.We recruited 50 industry professionals via word of mouth, Prolific, and Upwork\. Each participant received a compensation of $100 with a bonus for extra time\. Figure[2](https://arxiv.org/html/2610.21109#S3.F2)shows our participants’ domains with details in Appendix[A\.1](https://arxiv.org/html/2610.21109#A1.SS1)\. Professionals interacted with an agentic AI, Claude\([Anthropic, 2026](https://arxiv.org/html/2610.21109#bib.bib29)\), for two 1\-hour recording sessions\.222They were free to use other AI agents when appropriate for their work\. For example, an architect might use Claude for planning and reasoning while using ChatGPT to generate or render architectural images\.Each participant planned their workflow beforehand to ensure that their problem satisfied our criteria\. Then they updated the workflow document, rated their collaboration experience and output quality, and filled out the post\-interaction survey in the Appendix[A\.12](https://arxiv.org/html/2610.21109#A1.SS12)\.

Dataset statistics\.Kolabcovers 100 problem sets and human\-agent collaborative workflows conducted by professionals from 10 sectors of 40 occupations \(Figure[2](https://arxiv.org/html/2610.21109#S3.F2)\)\. The post\-interaction survey also measured collaboration satisfaction and output quality on a 1 \(low\) to 5 \(high\) scale\.333Details in Table[8](https://arxiv.org/html/2610.21109#A1.T8)in the Appendix\.Collaboration satisfaction was rated 5 in 54\.9% of sessions and 4 in 29\.7%, with 15\.3% rated 3 or below\. Similarly, output quality was rated 5 in 53\.5% of sessions and 4 in 31\.0%\.

Data release\.We publicly releaseKolabwith seven components: \(1\) screen\-recording videos for all sessions; \(2\) the workflow documents written by the professionals, including goal, context, and step names/actions; \(3\) the human–agent chat logs, including agent tool calls; \(4\) optional input and output files of each workflow; \(5\) human ratings for collaboration experience and output quality, along with step\-level output rubrics written by professionals; \(6\) the workflow graph per session, with step alignment, timestamps, and adoption labels; and \(7\) post\-session survey responses\.

Table 2:Real examples ofAdopt,Modify, andRejectresponses fromKolab, drawn from a music producer workflow, architecture workflow, and a video\-production workflow respectively\.Table 3:Summary statistics ofKolabfor videos, workflow graphs, and human action annotations\.
## 4Understanding Human\-Agent Collaboration Dynamics

### 4\.1Data Processing and Annotation

To make the workflow structured, each step in the workflow is represented as a temporal graph, where each nodev∈Vv\\in Vcorresponds to a human or agent action at a time intervalτ⁡\(v\)=\[ts​\(v\),te​\(v\)\]⊂ℝ≥0\\tau\(v\)=\[t\_\{s\}\(v\),t\_\{e\}\(v\)\]\\subset\\mathbb\{R\}\_\{\\geq 0\}\. The research team and professionals refined these graphs together\. An example graph for one step is shown in Figure[12](https://arxiv.org/html/2610.21109#A1.F12)in the Appendix\. A node can branch when activities occur at overlapping timestamps and loops happen when the same action is repeated\. Formally, each workflow is represented as follows:

G=\(V,E,a,w,τ\),G=\(V,E,a,w,\\tau\),\(1\)
whereVVis a set of human \(H\) or AI \(A\) action nodes;E⊆V×VE\\subseteq V\\times Vis the set of directed edges, with\(u,v\)∈E\(u,v\)\\in Eindicating that actionvvdirectly follows from actionuu\.

After graph construction, two human annotators labeled how humans incorporated agent outputs\. A human node is identified as a*source node*if its action triggered the agent’s generation \(e\.g\.,“design a floor plan”\)\. An agent node is the*target AI node*if it generates an output that is subsequently acted on by the participant\. The human node that acts on this AI output is labeled as follows\.

- •Adopt: The agent output is observably incorporated into the workflow as\-is, such as directly using an agent\-generated design or adding a suggested paper to a literature review\.
- •Modify: The agent output is incorporated with modifications,e\.g\.,the human revises agent\-generated text before using it\.
- •Reject: The agent output is not incorporated after being reviewed, such as when the human discards it or clearly expresses a negative sentiment and proceeds with a different approach\.

As seen from real examples in Table[2](https://arxiv.org/html/2610.21109#S3.T2), professionals evaluate agent contributions and act on softwares outside the AI interface\. A summary ofKolabis provided in Table[3](https://arxiv.org/html/2610.21109#S3.T3)\. In total,Kolabcontains 100 workflow videos\. Each video has corresponding human\-curated temporal graphs annotated with AI adoption labels, yielding a total of 692 instances for benchmarking MLLMs\.

### 4\.2Collaboration Dynamics Metrics

Collaboration burstiness\.Human\-agent interactions are not uniformly distributed over time\. Humans may work with an agent steadily or in short, intensive episodes separated by longer periods of independent work\. To capture this temporal rhythm, following[Goh and Barabási \(2008\)](https://arxiv.org/html/2610.21109#bib.bib39), we estimate collaboration burstiness asB=στ−μτστ\+μτ∈\[−1,1\]\\displaystyle B=\\frac\{\\sigma\_\{\\tau\}\-\\mu\_\{\\tau\}\}\{\\sigma\_\{\\tau\}\+\\mu\_\{\\tau\}\}\\in\[\-1,1\], whereμτ\\mu\_\{\\tau\}andστ\\sigma\_\{\\tau\}are the mean and standard deviation of the inter\-event intervals\.B→1B\\to 1indicates bursty interactions,B→−1B\\to\-1indicates regular interactions, andB=0B=0corresponds to a Poisson\-like process\.

Workflow topology\.Motivated by graph\-theoretic representations of complex systems\([Bondy and Murty, 2008](https://arxiv.org/html/2610.21109#bib.bib35);[Boccaletti et al\., 2006](https://arxiv.org/html/2610.21109#bib.bib38);[Newman et al\., 2011](https://arxiv.org/html/2610.21109#bib.bib33)\), we characterize workflow topology to capture the extent to which humans work concurrently while the agent generates an output\. We operationalize this as the proportion of*branch*nodes with out\-degree\>1\>1\(B​rBr\) relative to the number of edges \(\|E\|\|E\|\) as a workflow topologyW=B​r\|E\|W=\\frac\{Br\}\{\|E\|\}\.

AI retention rate\.To measure how agent outputs are retained in subsequent work\([Solé et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib36)\), we define AI retention through adoption and modification rates:

RA=NAdoptNAdopt\+NModify\+NReject,RM=NModifyNAdopt\+NModify\+NReject\.R\_\{A\}=\\frac\{N\_\{\\textsc\{Adopt\}\}\}\{N\_\{\\textsc\{Adopt\}\}\+N\_\{\\textsc\{Modify\}\}\+N\_\{\\textsc\{Reject\}\}\},\\qquad R\_\{M\}=\\frac\{N\_\{\\textsc\{Modify\}\}\}\{N\_\{\\textsc\{Adopt\}\}\+N\_\{\\textsc\{Modify\}\}\+N\_\{\\textsc\{Reject\}\}\}\.These capture the proportion of agent outputs incorporated directly or with modification\.

![Refer to caption](https://arxiv.org/html/2610.21109v1/collab_burstiness.png)\(a\)Collaboration burstiness
\(b\)Workflow topology\(c\)AI retention rate
Figure 3:Collaboration dynamics inKolab\. \(a\) Burstiness vs\. solo lead\-in; \(b\) workflow branching vs\. collaboration rating; \(c\) timing ofAdopt,Modify, andRejectresponses across sectors\. \*\*\* refers top<0\.001p<0\.001and \* isp<0\.05p<0\.05\.
### 4\.3Analysis

Using these three metrics, we analyze how human\-agent collaboration varies in its temporal rhythm \(BB\), topology of concurrent work \(WW\), and integration of agent contributions \(RR\)\. Our analyses are intended to describe the collaborative patterns observed inKolab, rather than as population\-level comparisons across professional domains\. Figure[3](https://arxiv.org/html/2610.21109#S4.F3)shows the results\.

\#1: Burstier workflows reflect more planning before interacting with the agent\.Collaboration burstiness was relatively consistent within individuals across two sessions \(Spearmanρ=0\.59\\rho=0\.59\)\. We define solo lead\-in as the proportion of a step completed before the first AI interaction, averaged over all steps in a session\. This reflects the proportion of time a person spends working independently before first interacting with AI\. For example, if the first AI interaction occurs 4 minutes into a 10\-minute step, the solo lead\-in is 40%\. As shown in Figure[3\(a\)](https://arxiv.org/html/2610.21109#S4.F3.sf1), burstier workflows had longer solo lead\-ins \(ρ=0\.35\\rho=0\.35,p<0\.001p<0\.001\)\. Consistent with this pattern, participants who reported having a clearer plan before using AI in the post\-interaction survey also had longer solo lead\-ins \(ρ=0\.53\\rho=0\.53,p<0\.001p<0\.001\)\.

\#2: Workflows with branches are associated with lower perceived collaboration experience\.Workflow branching \(WW\) is negatively associated with collaboration ratings \(Figure[3\(b\)](https://arxiv.org/html/2610.21109#S4.F3.sf2)\) \(ρ=−0\.24\\rho=\-0\.24,p<0\.05p<0\.05\)\. Although modest, this association suggests that concurrent human and agent activity may introduce coordination costs\.

\#3: AI retention patterns vary across professional domains\.As shown in Figure[3\(c\)](https://arxiv.org/html/2610.21109#S4.F3.sf3), most domains predominantly adopt agent outputs as\-is, with adoption rates ranging from 57% in Legal to 79% in Sciences and Computer & Mathematical work\. Legal workflows stand out for modification: 41% of agent outputs were modified, about twice the rate of any other domain \(Arts & Sports, the next highest, is 22%\), consistent with lawyers’ reluctance to adopt AI outputs as\-is\([Schwarcz et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib41)\)\. This shows work style variation in how professional domains integrate agent contributions\.

## 5Experiments and Results

### 5\.1Benchmark Tasks

Our benchmark tasks aim to answer the following research questions as summarized in Table[4](https://arxiv.org/html/2610.21109#S5.T4)\.

- •RQ1:\(AI Adoption\)How well can frontier MLLMs forecast how humans use agent contributions in long\-horizon professional workflows?
- •RQ2:\(Human Rating\)How well can frontier MLLMs predict human ratings of collaboration satisfaction and output quality given long\-horizon human\-agent collaboration workflows?

Table 4:Benchmark tasks\.TxstartT\_\{x\}^\{\\mathrm\{start\}\}andTxendT\_\{x\}^\{\\mathrm\{end\}\}denote the start and end timestamps of event or segmentxx, respectively;xxcan refer to thesource\\mathrm\{source\}node, targetAI\\mathrm\{AI\}node, humanadoption\\mathrm\{adoption\}node, or the fullworkflow\\mathrm\{workflow\}\.RQ1\(AI Adoption\)\.We evaluate whether MLLMs can predict whether humansAdopt,Modify, orRejectagent contributions under two settings—recognition \(1a\) and forecasting \(1b\) —which differ in whether the human response is observable or not \(Table[4](https://arxiv.org/html/2610.21109#S5.T4)\)\. Each input begins at the source node, where the human delegates to the agent \(TsourcestartT\_\{\\mathrm\{source\}\}^\{\\mathrm\{start\}\}\)\. Forecasting clips average 14\.0 minutes \(S​D=13\.7SD=13\.7; max=73\.0\), while prediction clips average 8\.8 minutes \(S​D=11\.7SD=11\.7; max=72\.0\) with more details in Table[10](https://arxiv.org/html/2610.21109#A1.T10)\.

We are also curious whether longer context improves or degrades model performance\. Thus, for Experiment 1c \(step\-N forecasting\), the model must forecast the AI adoption label for each target AI node in the last step, while providing the preceding 1, 2, or 3 steps as additional context\. For the longest context setting, motivated by how human annotators often need to watch the entire video context to determine the adoption label, we introduce Experiment 1d \(workflow\-level forecasting\)\. In this setting, we provide the entire workflow up to the current target AI node\.

RQ2\(Human Rating\)\.An evaluator should distinguish genuinely successful interactions from those in which professionals were dissatisfied or produced lower\-quality outcomes, rather than simply reflecting the predominantly positive ratings\. Thus, we investigate whether MLLMs can similarly recover professionals’ ratings from multimodal, long\-horizon human\-agent collaboration workflows\. Given the full workflow video for each step, we evaluate whether frontier MLLMs can predict the professional’s 1\-5 rating of collaboration satisfaction \(Experiment 2a\) and output quality \(Experiment 2b\)\.

Experimental SetupWe evaluate four leading multimodal agentic models, Gemini 3\.8 Flash, Gemini 3\.1 Pro, GPT\-6 Astra444\+GPT was evaluated on fewer instances and steps \(279 instances for 1a and 349 for 1b instead of 692 instances, 276 steps for 2a and 2b\) due to cost limitation\., and Claude Code Opus 5\.5, and two open\-weight video\-capable MLLMs: Qwen3\-VL\-30B\-A3B\-Instruct \(MoE\) and Qwen3\-VL\-32B\-Instruct \(dense\)\([Bai et al\., 2025](https://arxiv.org/html/2610.21109#bib.bib9)\)\. We compare against majority\-class \(Adopt\) and mode\-rating baselines for RQ1 and RQ2, respectively\. Our primary input is workflow video, which captures actions inside and outside the AI interface\. We additionally evaluate all seven combinations of video, graph, and chat to assess the contribution of each modality\.

### 5\.2Results and Discussion

Performance on forecasting AI adoption\.Table[5](https://arxiv.org/html/2610.21109#S5.T5)shows each MLLM’s performance with video only as input\. Claude performs the best for both recognizing and forecasting human adoption behavior, yet even the upper\-bound only reaches an F1\-score of 0\.676 aside from GPT\-6, which was evaluated on fewer instances than other models\. Full details of model comparison with the common instance sets are shown in the Table[15](https://arxiv.org/html/2610.21109#A1.T15)and Table[16](https://arxiv.org/html/2610.21109#A1.T16)in the Appendix\. Since the default AI agent used in this study is Claude, we hypothesize that it has a strong advantage in understanding interactions within its own interface\. Even so, Claude’s F1 for forecasting human adoption is only 0\.492, dominated byAdopt\. It has substantially lower performance onModifyandReject\. Among the remaining models, Gemini 3\.1 Pro performs the best when the human’s actual response is observable, with an F1 score of 0\.617\. Gemini models performs particularly poorly at forecastingReject, even worse than the open\-weight Qwen models\. Gemini 3\.1 Pro is very biased toward generatingAdopt\. 52% of trueModifyand 71% of trueRejectare predicted asAdoptby Gemini 3\.1 Pro \(Figure[7](https://arxiv.org/html/2610.21109#S5.F7)\)\.

Why should an AI model care to understand human collaboration with other AI models? Humans orchestrate various software tools in our dataset, and their workflows are not necessarily limited to a single AI model\. Even within our study, professionals used other AI systems, such as Gemini, ChatGPT, and Grok, in addition to Claude\. This makes it increasingly important for agents to understand human collaboration with other models, rather than only interactions involving themselves\.

Table 5:Benchmarking performance for AI adoption recognition \(Experiment 1a\) and for forecasting \(Experiment 1b\), for video only, mean±\\pmstd over 3 runs per model\. The best result in each column isshadedand the second best result isbolded\.Figure 4:Change in macro\-F1 as forecasting context grows\. X\-axis shows how many N previous steps are included as a context\. Full refers to the start of the whole workflow\.Does longer context help models anticipate human adoption behavior?We investigate whether providing more workflow context improves MLLMs’ performance on forecasting human’s behavior for AI adoption\. Using 114 instances in which both the source and target AI nodes appear in the last step of each session, we extend the context to include up to three preceding steps \(Experiment 1c\) or the entire workflow up to the target AI node \(Experiment 1d\)\. Longer context seems to provide limited improvements, with macro\-F1 changes below0\.070\.07for most video and graph settings even though none of the improvements with video are statistically significant \(Figure[4](https://arxiv.org/html/2610.21109#S5.F4)\)\. Chat logs benefit the most from additional context, with Gemini 3\.1 Pro improving by0\.120\.12F1 when given three preceding steps \(p<0\.01p<0\.01\)\. However, this improvement does not persist when the entire workflow is provided\. Longer context can also hurt performance, as Qwen\-32B drops from0\.430\.43to0\.320\.32F1 with the full graph \(p<0\.01p<0\.01\)\. These findings suggest that limited context alone may not explain models’ difficulty in anticipating human adoption behavior as providing the entire workflow does not consistently improve adoption forecasting\.

Table 6:Rating prediction for Experiments 2a and 2b, mean±\\pmstd over 3 runs\. The best result in each column isshaded; the second best result isbolded\.Performance on predicting collaboration satisfaction and output quality\.We evaluate alignment with human ratings using mean absolute error \(MAE\), within\-1 accuracy, and Spearman’sρ\\rho\. The mode baseline outperforms every MLLM in MAE of 0\.682 for collaboration satisfaction and 0\.735 for output quality, reflecting the strong skew of human ratings toward 5\. Gemini 3\.1 Pro achieves the lowest MAE of 0\.685 for collaboration satisfaction and 0\.764 for output quality, but even the best\-performing model fails to outperform the baseline\. Moreover, rank agreement with human judgments remains weak\. GPT\-6 achieves the highest Spearman correlation for both collaboration satisfaction \(ρ=0\.379\\rho=\{0\.379\}\) and output quality \(ρ=0\.352\\rho=\{0\.352\}\), while several models exhibit near\-zero correlations\. Thus, when disregarding differences in absolute rating scale, current frontier multimodal agents and LLMs still struggle to recover how professionals comparatively judge their collaboration and resulting outputs\.

\(a\)1a: Recognition \(F1\)
\(b\)1b: Forecasting \(F1\)
\(c\)2a: Collaboration satisfaction \(ρ\\rho\)
\(d\)2b: Output quality \(ρ\\rho\)

Figure 5:Full results for seven input\-modality combinations across the four benchmark tasks\. V stands for video, G: graph, and C: chat log; “\+” is a combination of modalities, “All” refers to using video, graph, and chat as the input\. The best score in each row is in bold\.Figure 6:How each modality helps\.
### 5\.3Modality Analysis

We examine how video, graph, and chat, individually and in combination, affect models’ understanding of human\-agent collaboration \(Figures[5](https://arxiv.org/html/2610.21109#S5.F5)and[6](https://arxiv.org/html/2610.21109#S5.F6)\)\. No modality consistently performs best across models or experiment tasks\. Video alone yields the highest recognition performance with GPT\-6 \(F1=0\.71=0\.71, Experiment 1a\)\. Graphs, either alone or combined with video or chat, tend to yield the best forecasting performance, with GPT\-6 achieving an F1 score of 0\.52 in Experiment 1b\. However, chat alone particularly hurts recognition ofRejectcases: relative to video, F1 changes by\+0\.07\+0\.07forAdopt,\+0\.01\+0\.01forModify, and−0\.07\-0\.07forReject\(Figure[6](https://arxiv.org/html/2610.21109#S5.F6)b\), likely because chat does not capture how agent outputs are subsequently used outside the AI interface\. Yet video is not consistently optimal\. Human\-curated workflow graphs, which capture high\-level professional activities, particularly benefit the non\-agentic Qwen models; for Qwen\-32B, adding graphs to video improves forecasting F1 from0\.230\.23to0\.370\.37\.

For human\-rating prediction, performance remains limited regardless of modality \(Figure[5](https://arxiv.org/html/2610.21109#S5.F5)c–d\)\. Correlations peak at onlyρ=0\.38\\rho=0\.38for collaboration satisfaction \(GPT\-6, video only\) andρ=0\.42\\rho=0\.42for output quality \(GPT\-6, video\+graph\)\. Graphs are particularly informative for output quality: video alone yields near\-zero correlations for Gemini 3\.8 Flash and Qwen\-30B \(ρ=0\.04\\rho=0\.04–0\.050\.05\), while adding graphs raises them toρ=0\.23\\rho=0\.23–0\.300\.30\. Overall, providing additional modalities does not resolve MLLMs’ difficulty in approximating professionals’ judgments of collaboration experience and output quality\.

Figure 7:Distributions of model\-generated labels forModifyandRejectcases in AI adoption forecasting\.Figure 8:Mean predicted rating vs\. human rating for collaboration \(Exp 2a\) and output quality \(Exp 2b\)\.
### 5\.4Failure Cases

For forecasting whether humans adopt agent contributions, models often strongly prefer certain labels although the true labels areModifyorReject\(Figure[7](https://arxiv.org/html/2610.21109#S5.F7)\)\. Gemini 3\.1 Pro and Flash overpredictAdopt\. With video inputs, Gemini 3\.8 Flash forecastsAdoptfor 83\.8% of examples compared with a 65\.9% ground\-truth rate, and Qwen3\-32B instead favorsModify, predicting it in 87\.3% of cases compared with 19\.1% in the ground truth\. These biases largely persist with graph and chat inputs as well \(e\.g\., Flash generatesAdoptfor 82\.3% and 74\.0%, and Qwen3\-32B outputsModifyfor 67\.7% and 62\.0%\)\. Models therefore exhibit distinct forecasting biases, which vary with both the model and input modality\.

For human\-rating prediction \(Figure[8](https://arxiv.org/html/2610.21109#S5.F8)\), some models predict similar ratings all the time\. With video, Gemini 3\.1 Pro predicts 4\.83 for a true rating of 1 and 4\.95 for a 5 \(Experiment 2a\); in Experiment 2b, it predicts 5\.00 and 4\.94\. Graph input improves this separation, with gaps between true ratings of 1 and 5 reaching 1\.95 and 2\.45 for Experiments 2a and 2b\. Claude and Qwen variants also show a similar pattern\. Claude’s predictions range only from 3\.3 to 3\.9 across the entire 1–5 rating scale with video input in both experiments, while the ratings of Qwen variants range from 3\.04 to 4\.15\. However, GPT\-6 already separates low and high ratings with video and remains the most stable across modalities\.

These systematic biases raise a more fundamental question: to what extent are MLLMs actually grounding their predictions in the observed workflow? If models understand the workflow context, their predictions should vary with the human behaviors and outcomes they observe\. Some models repeatedly produce similar labels or ratings despite different inputs and ground\-truths\. It suggests that their predictions may be driven by model\-specific priors instead of the workflow itself\.

## 6Conclusion and Future Work

We have introducedKolabas a new dataset to benchmark frontier multimodal agentic models on their capabilities to support human\-agent collaboration better on long\-horizon workflows\. Our results show that even frontier MLLMs, including GPT\-6, struggle to predict and understand human–AI collaboration\. As a future work, we will explore ways to improve these capabilities by constructing training datasets or developing sandbox environments where models can practice collaborating with simulated users before real\-world deployment\.

### Ethics statement

Understanding how humans collaborate with AI agents is important for developing AI systems that better support human needs and decisions\. However, evaluating human\-agent collaboration also raises ethical concerns\. First, models that inaccurately predict whether humans will adopt, modify, or reject AI contributions may misrepresent human intentions and preferences\. Our findings highlight the limitations of current MLLMs in understanding professional human\-agent collaborative workflows\. Relying on these models as proxies for human judgments could lead to misleading assessments, particularly in professional domains where decisions require specialized expertise and contextual understanding\. Second, collecting and releasing real\-world professional workflows raises privacy concerns, as screen recordings and interaction logs may contain sensitive information\. To mitigate these risks, we remove all personally identifiable information \(PII\) in our data\. Our work aims to support the development of human\-centered collaborative agents while emphasizing that model predictions should not replace human judgments or undermine human agency\. This study was approved by our institution’s IRB\. All participant data are anonymized and securely stored\.

### Reproducibility statement

Code and data are released in our project webpage[https://minnesotanlp\.github\.io/kolab/](https://minnesotanlp.github.io/kolab/)\. Details about data collection details are explained in[A\.2](https://arxiv.org/html/2610.21109#A1.SS2)in the Appendix\. Data processing and annotation details are explained in[A\.4](https://arxiv.org/html/2610.21109#A1.SS4)\. All model prompts and model settings are elaborated in the Appendix\.

#### Acknowledgments

We are grateful to Jingcheng Liang, Risako Owan, and Su Yeon Han for helping with data cleaning and annotation\. This research was supported in part by the Interdisciplinary Doctoral Fellowship from the University of Minnesota for the first author\.

## References

- Anthropic \(2026\)AnthropicClaude\.Note:[https://claude\.ai](https://claude.ai/)Accessed: 2026\-09\-25Cited by:[§3](https://arxiv.org/html/2610.21109#S3.p4.1)\.
- Baiet al\.\(2025\)S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge,et al\.Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[§A\.7](https://arxiv.org/html/2610.21109#A1.SS7.p2.1),[§5\.1](https://arxiv.org/html/2610.21109#S5.SS1.p5.1)\.
- Boccalettiet al\.\(2006\)S\. Boccaletti, V\. Latora, Y\. Moreno, M\. Chavez, and D\. HwangComplex networks: structure and dynamics\.Physics reports424\(4\-5\),pp\. 175–308\.Cited by:[§4\.2](https://arxiv.org/html/2610.21109#S4.SS2.p2.1)\.
- Bondy and Murty \(2008\)J\. A\. Bondy and U\. S\. R\. MurtyGraph theory\.Springer Publishing Company, Incorporated\.Cited by:[§4\.2](https://arxiv.org/html/2610.21109#S4.SS2.p2.1)\.
- Chenet al\.\(2026\)V\. Chen, R\. Malhotra, X\. Wang, J\. Michelini, X\. Zhou, A\. B\. Soni, H\. H\. Tran, C\. Smith, A\. Talwalkar, and G\. NeubigHow can we assess human\-agent interactions? case studies in software agent design\.InInternational Conference on Machine Learning \(ICML\),Seoul, South Korea\.External Links:[Link](https://arxiv.org/abs/2510.09801)Cited by:[§1](https://arxiv.org/html/2610.21109#S1.p1.1),[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px2.p1.1)\.
- Christensonet al\.\(2026\)J\. Christenson, K\. J\. D\. de Langis, S\. A\. Hayati, and D\. KangEffects of varying LLM access on essay writing behavior\.InProceedings of the Workshop on Innovative Use of NLP for Building Educational Applications \(BEA\),External Links:[Link](https://arxiv.org/abs/2606.00250)Cited by:[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px2.p1.1)\.
- Daiet al\.\(2026\)Y\. Dai, K\. Ramakrishnan, J\. Gu, M\. Fernandez, Y\. Luo, V\. Prabhu, Z\. Hu, C\. Xiong, Z\. Chen, R\. Xu,et al\.Scuba: salesforce computer use benchmark\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 24363–24386\.Cited by:[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px1.p1.1)\.
- Daset al\.\(2025\)D\. Das, K\. C\. Le, R\. S\. Parkar, K\. De Langis, B\. Madson, C\. M\. Berryman, R\. M\. Willis, D\. H\. Moses, B\. McDonnell, D\. Schwarcz, and D\. KangLawFlow: collecting and simulating lawyers’ thought processes on business formation case studies\.InConference on Language Modeling \(COLM\),External Links:[Link](https://openreview.net/forum?id=MsgdEkcLRz)Cited by:[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px1.p1.1)\.
- Goh and Barabási \(2008\)K\. Goh and A\. BarabásiBurstiness and memory in complex systems\.Europhysics Letters81\(4\),pp\. 48002\.Cited by:[§4\.2](https://arxiv.org/html/2610.21109#S4.SS2.p1.1)\.
- Hayatiet al\.\(2026\)S\. A\. Hayati, M\. Zhou, P\. A\. Setiani, R\. Wang, J\. C\. Chang, and D\. KangOrchestrating genai for interdisciplinary research\.External Links:2609\.30588,[Link](https://arxiv.org/abs/2609.30588)Cited by:[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px2.p1.1)\.
- Huqet al\.\(2025\)F\. Huq, Z\. Z\. Wang, F\. F\. Xu, T\. Ou, S\. Zhou, J\. P\. Bigham, and G\. NeubigCowpilot: a framework for autonomous and human\-agent collaborative web navigation\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(System Demonstrations\),pp\. 163–172\.Cited by:[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px2.p1.1)\.
- Janget al\.\(2026\)L\. K\. Jang, A\. K\. Jang, J\. Y\. Koh, and R\. SalakhutdinovMyPCBench: a benchmark for personally intelligent computer\-use agents\.arXiv preprint arXiv:2606\.16748\.Cited by:[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px1.p1.1)\.
- Leet al\.\(2026\)K\. C\. Le, L\. Wang, M\. Lee, R\. Volkov, L\. T\. Chau, and D\. KangScholaWrite: a dataset of end\-to\-end scholarly writing\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 34755–34788\.External Links:[Link](https://aclanthology.org/2026.acl-long.1606/),[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.1606),ISBN 979\-8\-89176\-390\-6Cited by:[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2026\)W\. Li, B\. Zhou, Y\. Yu, Z\. Xu, Y\. Yang, D\. Li, and C\. ShanWeaveBench: a long\-horizon, real\-world benchmark for computer\-use agents with hybrid interfaces\.arXiv preprint arXiv:2606\.09426\.Cited by:[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2610.21109#S2.T1.4.4.1)\.
- Maldaneret al\.\(2026\)M\. K\. Maldaner, A\. Fourney, A\. Swearngin, H\. Mozannar, G\. Bansal, M\. Murad, R\. Hosn, and S\. AmershiSentinelBench: a benchmark for long\-running monitoring agents\.arXiv preprint arXiv:2606\.05342\.Cited by:[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px1.p1.1)\.
- Merrillet al\.\(2026\)M\. Merrill, A\. Shaw, N\. Carlini, B\. Li, H\. Raj, I\. Bercovich, L\. Shi, J\. Shin, T\. Walshe, E\. K\. Buchanan,et al\.Terminal\-bench: benchmarking agents on hard, realistic tasks in command line interfaces\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 40903–40986\.Cited by:[§1](https://arxiv.org/html/2610.21109#S1.p1.1)\.
- Miserendinoet al\.\(2025\)S\. Miserendino, M\. Wang, T\. Patwardhan, and J\. HeideckeSWE\-lancer: can frontier LLMs earn $ 1 million from real\-world freelance software engineering?\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 44412–44450\.External Links:[Link](https://proceedings.mlr.press/v267/miserendino25a.html)Cited by:[§A\.1](https://arxiv.org/html/2610.21109#A1.SS1.p1.1)\.
- Mozannaret al\.\(2025\)H\. Mozannar, G\. Bansal, C\. Tan, A\. Fourney, V\. Dibia, J\. Chen, J\. Gerrits, T\. Payne, M\. K\. Maldaner, M\. Grunde\-McLaughlin,et al\.Magentic\-ui: towards human\-in\-the\-loop agentic systems\.arXiv preprint arXiv:2507\.22358\.Cited by:[§1](https://arxiv.org/html/2610.21109#S1.p1.1),[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px2.p1.1)\.
- Newmanet al\.\(2011\)M\. Newman, A\. Barabási, and D\. J\. WattsThe structure and dynamics of networks\.Princeton university press\.Cited by:[§4\.2](https://arxiv.org/html/2610.21109#S4.SS2.p2.1)\.
- Ohet al\.\(2026\)J\. Oh, S\. Do, Y\. Lee, S\. E\. Whang, and D\. KangFoundations of proactive agents: principles, technical layers, and proactivity\-gym\.arXiv preprint arXiv:2609\.37267\.Cited by:[§1](https://arxiv.org/html/2610.21109#S1.p5.1)\.
- Patwardhanet al\.\(2026\)T\. Patwardhan, R\. Dias, E\. Proehl, G\. Kim, M\. Wang, O\. Watkins, S\. Fishman, M\. Aljubeh, P\. Thacker, L\. Fauconnet,et al\.Gdpval: evaluating ai model performance on real\-world economically valuable tasks\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 24005–24040\.Cited by:[§A\.1](https://arxiv.org/html/2610.21109#A1.SS1.p1.1),[§1](https://arxiv.org/html/2610.21109#S1.p1.1),[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2610.21109#S2.T1.4.6.1)\.
- Qianet al\.\(2025\)C\. Qian, Z\. Liu, A\. Prabhakar, Z\. Liu, J\. Zhang, H\. Chen, H\. Ji, W\. Yao, S\. Heinecke, S\. Savarese,et al\.Userbench: an interactive gym environment for user\-centric agents\.arXiv preprint arXiv:2507\.22034\.Cited by:[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px2.p1.1)\.
- Schwarczet al\.\(2026\)D\. Schwarcz, D\. Das, D\. Kang, and B\. McDonnellThinking like a lawyer in the age of generative AI: cognitive limits on AI adoption among lawyers\.Journal of Institutional and Theoretical Economics\.External Links:[Link](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5260645)Cited by:[§4\.3](https://arxiv.org/html/2610.21109#S4.SS3.p4.1)\.
- Shaoet al\.\(2026a\)Y\. Shao, V\. Samuel, Y\. Jiang, J\. Yang, and D\. YangCollaborative gym: a framework for enabling and evaluating human\-agent collaboration\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 99616–99649\.Cited by:[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px2.p1.1)\.
- Shaoet al\.\(2026b\)Y\. Shao, Z\. Z\. Wang, N\. Ahuja, Y\. Wang, B\. Liu, and D\. YangCollabSkill: evaluating human\-agent collaboration on real\-world tasks\.arXiv preprint arXiv:2606\.09833\.Cited by:[§1](https://arxiv.org/html/2610.21109#S1.p1.1),[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px2.p1.1)\.
- Shaoet al\.\(2025\)Y\. Shao, H\. Zope, Y\. Jiang, J\. Pei, D\. Nguyen, E\. Brynjolfsson, and D\. YangFuture of work with ai agents: auditing automation and augmentation potential across the us workforce\.arXiv preprint arXiv:2506\.06576\.Cited by:[§A\.1](https://arxiv.org/html/2610.21109#A1.SS1.p1.1)\.
- Shenet al\.\(2026\)S\. Z\. Shen, V\. Chen, K\. Gu, A\. Ross, Z\. Ma, J\. Ross, A\. Gu, C\. Si, W\. Chi, A\. Peng, J\. J\. Shen, A\. Talwalkar, T\. Wu, and D\. SontagScaling collaborative effort with agents\.InFindings of the Association for Computational Linguistics: ACL 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 12254–12271\.External Links:[Link](https://aclanthology.org/2026.findings-acl.595/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.595),ISBN 979\-8\-89176\-395\-1Cited by:[§1](https://arxiv.org/html/2610.21109#S1.p1.1),[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px2.p1.1)\.
- Soléet al\.\(2026\)R\. Solé, G\. Ruffini, F\. Castaldo, M\. Tuccio, L\. F\. Seoane, M\. de Domenico, S\. F\. Elena, D\. C\. Krakauer, and M\. LevinLarge\-language models as a cognitive virus\.arXiv preprint arXiv:2609\.03344\.Cited by:[§4\.2](https://arxiv.org/html/2610.21109#S4.SS2.p3.1)\.
- Sunet al\.\(2026a\)Q\. Sun, Z\. Liu, C\. Ma, Z\. Ding, F\. Xu, Z\. Yin, H\. Zhao, Z\. Wu, K\. Cheng, Z\. Liu,et al\.Scienceboard: evaluating multimodal autonomous agents in realistic scientific workflows\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 75694–75731\.Cited by:[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px1.p1.1)\.
- Sunet al\.\(2026b\)Y\. Sun, X\. Han, W\. Zhang, Y\. Pang, T\. Wang, Y\. Cao, Y\. Huang, C\. Duroiu, H\. Zhang, J\. Lin,et al\.Agents’ last exam\.arXiv preprint arXiv:2606\.05405\.Cited by:[§A\.1](https://arxiv.org/html/2610.21109#A1.SS1.p1.1),[§1](https://arxiv.org/html/2610.21109#S1.p1.1),[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2610.21109#S2.T1.4.2.1)\.
- Tacket al\.\(2026\)J\. Tack, P\. Laban, and J\. NevilleLLMs get lost in evolving user intent\.arXiv preprint arXiv:2607\.20734\.Cited by:[§1](https://arxiv.org/html/2610.21109#S1.p1.1)\.
- Vidgenet al\.\(2026\)B\. Vidgen, A\. Mann, A\. Fennelly, J\. W\. Stanly, L\. Rothman, M\. Burstein, J\. Benchek, D\. Ostrofsky, A\. Ravichandran, D\. Sur,et al\.APEX\-agents\.arXiv preprint arXiv:2601\.14242\.Cited by:[§1](https://arxiv.org/html/2610.21109#S1.p1.1)\.
- Vijayvargiyaet al\.\(2026\)S\. Vijayvargiya, X\. Zhou, A\. Yerukola, M\. Sap, and G\. NeubigAmbig\-swe: interactive agents to overcome underspecificity in software engineering\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 146716–146737\.Cited by:[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px2.p1.1)\.
- Wanet al\.\(2022\)R\. Wan, N\. Etori, K\. Badillo\-Urquiola, and D\. KangUser or labor: an interaction framework for human\-machine relationships in NLP\.InProceedings of the Workshop on Data Science with Human\-in\-the\-Loop \(DaSH\),External Links:[Link](https://arxiv.org/abs/2211.01553)Cited by:[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2026\)W\. Wang, T\. Xiong, J\. Ni, Y\. Bao, X\. Li, T\. Liu, H\. Guo, Z\. Huang, and S\. ZhangDeskCraft: benchmarking desktop agents on professional workflows and human\-in\-the\-loop collaboration\.arXiv preprint arXiv:2606\.03103\.Cited by:[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2610.21109#S2.T1.4.7.1)\.
- Wanget al\.\(2025\)Z\. Z\. Wang, Y\. Shao, O\. Shaikh, D\. Fried, G\. Neubig, and D\. YangHow do ai agents do human work? comparing ai and human workflows across diverse occupations\.arXiv preprint arXiv:2510\.22780\.Cited by:[§1](https://arxiv.org/html/2610.21109#S1.p1.1),[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px2.p1.1)\.
- Wuet al\.\(2026\)Y\. Wu, W\. Huang, J\. Guo, D\. Li, R\. Jiang, H\. P\. Zou, C\. Miao, S\. Li, W\. Zhang, W\. Ye,et al\.HAS\-bench: evaluating llm\-based human\-agent systems under configurable human participation\.arXiv preprint arXiv:2607\.04329\.Cited by:[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px2.p1.1)\.
- Xieet al\.\(2024\)T\. Xie, D\. Zhang, J\. Chen, X\. Li, S\. Zhao, R\. Cao, T\. J\. Hua, Z\. Cheng, D\. Shin, F\. Lei,et al\.Osworld: benchmarking multimodal agents for open\-ended tasks in real computer environments\.Advances in Neural Information Processing Systems37,pp\. 52040–52094\.Cited by:[§1](https://arxiv.org/html/2610.21109#S1.p1.1),[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px1.p1.1)\.
- Xuet al\.\(2026\)F\. F\. Xu, Y\. Song, B\. Li, Y\. Tang, K\. Jain, M\. Bao, Z\. Wang, X\. Zhou, Z\. Guo, M\. Cao,et al\.Theagentcompany: benchmarking llm agents on consequential real world tasks\.Advances in Neural Information Processing Systems38\.Cited by:[§1](https://arxiv.org/html/2610.21109#S1.p1.1),[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2610.21109#S2.T1.4.5.1)\.
- Yaoet al\.\(2024\)S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhant​a​utau\-Bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.arXiv preprint arXiv:2406\.12045\.Cited by:[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px2.p1.1)\.
- Yuanet al\.\(2026\)M\. Yuan, Z\. Zhou, X\. Xiong, W\. Wu, J\. Sun, J\. Song, K\. Cui, B\. Wang, H\. Wu, Y\. Li, D\. Lu, H\. Lu, Q\. Zhen, X\. Wang, J\. Deng, Y\. Yang, C\. Chen, B\. Zheng, A\. Su, X\. Yu, H\. Zou, S\. Agashe, X\. H\. Lu, M\. Kaur, Z\. Qi, V\. S\. Chen, F\. Sala, D\. Liu, J\. Lin, Z\. Yu, Y\. Su, S\. Reddy, X\. E\. Wang, P\. Qi, T\. Xie, and T\. YuOSWorld2\.0: benchmarking computer use agents on long\-horizon real\-world tasks\.External Links:2606\.29537,[Link](https://arxiv.org/abs/2606.29537)Cited by:[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px1.p1.1)\.
- Zhuet al\.\(2026\)L\. Zhu, J\. Ding, J\. Zhang, J\. Xue, S\. Liang, G\. Zhang, Y\. Zhu, D\. Zeng, X\. Gao, Q\. Gu,et al\.Workflow\-gym: towards long\-horizon evaluation of computer\-use agentic tasks in real\-world professional fields\.arXiv preprint arXiv:2606\.11042\.Cited by:[§1](https://arxiv.org/html/2610.21109#S1.p1.1),[§2](https://arxiv.org/html/2610.21109#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2610.21109#S2.T1.4.3.1)\.
- Zouet al\.\(2025\)H\. P\. Zouet al\.LLM\-based human\-agent collaboration and interaction systems: a survey\.arXiv preprint arXiv:2505\.00753\.Note:Findings of ACL 2026\. 20 total co\-authors, abbreviated here\.Cited by:[§1](https://arxiv.org/html/2610.21109#S1.p1.1)\.

## Appendix AAppendix

### A\.1Professionals Background and Software Usage

The 50 study participants span 10 O\*NET sectors and 40 distinct occupations, ranging from film editors, graphic designers, and audio engineers to aerospace engineers, GIS specialists, attorneys, and bioinformaticians\. We prioritize non\-computer science professions, which remain comparatively under\-resourced with real\-world task data relative to software engineering\([Patwardhan et al\., 2026](https://arxiv.org/html/2610.21109#bib.bib20);[Shao et al\., 2025](https://arxiv.org/html/2610.21109#bib.bib22);[Sun et al\., 2026b](https://arxiv.org/html/2610.21109#bib.bib7);[Miserendino et al\., 2025](https://arxiv.org/html/2610.21109#bib.bib34)\)\. Across these roles, participants reported using 90 distinct profession\-specific non\-AI softwares \(excluding Google or Microsoft office products\) and 9 AI tools in their day\-to\-day work, as shown in Figure[9](https://arxiv.org/html/2610.21109#A1.F9)\. This mix of shared productivity tools and domain\-specific software illustrates the breadth of professional practice that the benchmark’s workflows are grounded in\.

Table 7:Summary statistics of the 50 professionals\.
![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/software_categories_arxiv.png)Figure 9:Software used by participants in their day\-to\-day work, grouped by category\.Figure 10:Occupation distributions ofKolab\. “Other” refers to three O\*NET sectors: Life & Social Science, Sales & Related, and Healthcare\.
### A\.2Data Collection Details

We collectedKolabthrough two sessions per participant, in which we recorded the participant’s computer screen\. Each session lasted approximately one hour or longer and centered on a problem chosen by the participant\. After data processing and due to time constraints, some problems contained fewer than five steps; however, each interaction session still lasted approximately one hour\. To capture different patterns of human\-agent collaboration, we used two protocols\. In Session 1, participants were encouraged to interact with the agent at least five times while completing their workflow\. In Session 2, participants were encouraged to interact with the agent at each step of the workflow\. The two sessions did not have to address the same problem\. However, participants could use the second session to continue or extend the work from the first session\. For example, a participant might use the first session to develop a branding logo and use the second session to create a marketing advertisement based on the resulting brand identity\. Participants were required to interact with agentic AI, Claude, using our credentials during each session and were free to use other AI agents when appropriate for their work\. For example, an architect might use Claude for planning and reasoning while using ChatGPT to generate or render architectural images\. At the time of data collection, most participants used the default Claude model \(Sonnet 5\.5\), while a few used Opus 5\. Each participant planned their workflow before the recording sessions to ensure that their problem satisfied our task criteria\. After the recording sessions, they updated the workflow document and completed surveys about their collaboration experience and output quality\. During each recording session, participants were encouraged to think aloud; we used the transcriptions to construct the workflow graphs and to guide the annotation of response labels\.

Figure 11:Time spent in each recording session working with agentic AI versus the participant’s estimated time to solve the same problem without agentic AI\.![Refer to caption](https://arxiv.org/html/2610.21109v1/images/p4_example.png)Figure 12:Example workflow graph for one step of a participant’s session\.
### A\.3Graph as a Collaboration Workflow Representation

Actions and timestamps from workflow video are aligned with steps documented by each professional\. We then represented each workflow as a directed graph, where each node corresponds to a human or agent action\. The research team and professionals jointly reviewed and refined these graphs\. Graphs can capture non\-linear workflows; for instance, a node branches when activities occur in parallel and loops happen when the same action is repeated\.

Formally, each session is represented as a directed temporal graph

Gt​e​m​p​o​r​a​l=\(V,E,a,w,τ\),G\_\{temporal\}=\(V,E,a,w,\\tau\),\(2\)
whereVVis the set of nodes, each corresponding to a discrete human or AI action, andE⊆V×VE\\subseteq V\\times Vis the set of directed edges, with\(u,v\)∈E\(u,v\)\\in Eindicating that actionvvdirectly follows from actionuuwithin the workflow\. Each nodev∈Vv\\in Vis annotated by three functions: an actor label

a:V→\{H,A\},a:V\\to\\\{\{\\color\[rgb\]\{0\.2461,0\.4961,0\.7773\}\\textbf\{H\}\},\{\\color\[rgb\]\{0\.8242,0\.4609,0\.1836\}\\textbf\{A\}\}\\\},\(3\)indicating whether the action was performed by the human or the agent; a workflow\-step label

w:V→\{1,…,K\},w:V\\to\\\{1,\\dots,K\\\},\(4\)
indicating which of theKKsteps of the task the action belongs to; and a time interval

τ⁡\(v\)=\[ts​\(v\),te​\(v\)\]⊂ℝ≥0\.\\tau\(v\)=\[t\_\{s\}\(v\),t\_\{e\}\(v\)\]\\subset\\mathbb\{R\}\_\{\\geq 0\}\.\(5\)recording when the action began and ended\.

After graph construction, two human annotators annotated how each AI output was incorporated into the workflow\. For every observed user response to an AI output, annotators identified the*prompt node*that triggered generation, the*AI\-content node*being acted on, and a*response label*:Adopt\(incorporated without modification\),Modify\(incorporated with changes\), orReject\(explicitly rejected or not incorporated\)\. This is an annotation layer on top ofGt​e​m​p​o​r​a​lG\_\{temporal\}: each noden∈Vn\\in Vrepresenting a user response carries a set of response triples as follows\.

U⁡\(n\)=\{\(pi,ci,ℓi\)\}i=1mn,U\(n\)=\\\{\(p\_\{i\},c\_\{i\},\\ell\_\{i\}\)\\\}\_\{i=1\}^\{m\_\{n\}\},\(6\)wherepi∈V∪\{∅\}p\_\{i\}\\in V\\cup\\\{\\emptyset\\\}is the prompt node \(absent if the user acts on existing AI content without a new prompt\),ci∈Vc\_\{i\}\\in Vis the AI\-content node \(a⁡\(ci\)=Aa\(c\_\{i\}\)=\{\\color\[rgb\]\{0\.8242,0\.4609,0\.1836\}\\textbf\{A\}\}\), andℓi∈\{adopt,modify,reject\}\\ell\_\{i\}\\in\\\{\\text\{adopt\},\\text\{modify\},\\text\{reject\}\\\}is the response label\.

The following is an example workflow document that an architect filled out to plan their workflow before each recording session\. After the session, they revised the content and filled out the assessment questions about collaboration experience and output quality ratings\. Other step details are omitted due to space\.

Goal:Developing an architectural plan for Residential HouseContext:I have chosen this project because it requires a set of expertise to layout an architectural plan\. Conditions including design brief, site and weather analysis, building codes, area and zoning standards to evaluate and design according to that as an expert\.Did you achieve your goal after ~1\-hour interaction with Claude?Yes Why or why not?Claude was able to produce layoutsStep 1 name:Summary of design brief, a set of requirements to start the projectStep action:Prepare the design brief and define scope of project\. Input:client brief shared with me Output:Revise details when design brief in comprehensive format is generated\. Softwares:ClaudeRubric:1\.Each design improvement includes a clear architectural reason2\.Each revision idea includes a rationale\.3\.The revised layout remains consistent with the client brief\.4\.Suggestions are consistent with the requirements\.Output Quality and Collaboration Assessment1\.Rate the output quality from 1–5 \(1 not satisfy, 5 satisfy\)\.42\.Reason for your rating?“It created a set of pointers in design brief from the narrative, and curated a long brief in detail\. Added extra things also that were not questioned\.”3\.Which ones of the rubrics above are satisfied in this output?2, 3, and 4Human\-Agent Collaboration1\.Did the AI Agent successfully complete the task?Yes2\.Why?It created a set of pointers in design brief from the narrative, and curated a long brief in detail\. Added extra things also that were not questioned\.3\.Collaboration satisfaction — write a number \(1–5\) to the left of each statement, where 1 = not satisfied and 5 = satisfied:•5— The agent understood the task, context, and instructions provided\.•4— The agent responded appropriately to requests and adapted when given feedback or corrections\.•4— The agent reduced effort, time, or cognitive load during the task\.•4— The agent proactively suggested useful ideas, alternatives, or next steps when appropriate\.•5— The agent produced accurate, relevant, and trustworthy outputs that required minimal correction\.•2— The interaction felt smooth, with the agent contributing at the right moments without creating unnecessary work\.•3— I was able to guide, redirect, and override the agent’s actions when desired\.4\.Rate the overall collaboration experience from 1–5 \(1 not satisfy, 5 satisfy\)\.4Step 2:area codes to use in projectStep 3:plan zoning and divisions of spacesStep 4:Arrangement of spaces and Finalizing layoutStep 5:Architectural floor Plan finalization

Another example of workflow document by a different participant \(sound designer and composer\) is shown below\. In this case, the agentic AI failed to help and actually slowed down the process\.

Goal:Sound Design a promotional video for a real estate sales pitchContext:This goal will allow me to evaluate how AI can enhance my work as a sound designer\. The plan is to work on a promotional sales pitch for apartments in cozumel, which was recorded by a production company, but they lack sound for many of the camera shots they use \(I\.E: Shots of the beach, of the town and its localities, etc\.\) This lack of sound makes the project seem empty, therefore, a sound designer is needed to enhance the immersion of what is shown by adding the ambience noise, sound of actions \(Walking, placing, etc\) while ensuring audio files don’t overlap and work in symphony\.Did you achieve your goal after ~1\-hour interaction with Claude?NoWhy or why not?“Although we did advance on the video, It was not finished in the 1 hour mark\. I felt like I had to keep up with Claude at the start, rather than being benefited by it while working\. This gradually improved when I used it for smaller activities like finding specific SFX or a good usage of effect plugins, but when I initially tried to use it as an analysis assistant, it took almost 20 minutes just analyzing the video, followed by 10 more minutes of analyzing just a fragment of it and providing ideas, which had slowed me down significantly, as I was already done with most of the other things I had not delegated to Claude\.”Step 1 name:Analyze the video and understand key moments from itStep action:Understand key moments where sound design can exponentially increase quality and immersion for the video, uploading the video to Claude for it to create this in depth analysis\.Input:MP4 video with dialogue and visual insights to the required SFXOutput:A pdf or DOCX document with key moments, time stamps and descriptions of the required sound effects\.Softwares:FL Studio, Artlist, Claude MaxRubric:1\.A score of 5 means that Claude was able to analyze the video and describe with detail what it is, what its selling, what key visuals and moments it has and how to interpret it as a client\.2\.A score of 5 means the analysis included accurate, verifiable timestamps for each key moment, ready to use directly without needing to re\-watch the video\.Output Quality and Collaboration Assessment1\.Rate the output quality from 1–5 \(1 not satisfy, 5 satisfy\)\.12\.Reason for your rating?“Claude was not able to fully analyze the video, due to a runout of time\.”3\.Which ones of the rubrics above are satisfied in this output?N/AHuman\-Agent Collaboration1\.Did the AI Agent successfully complete the task?No\.2\.Why?“Claude mentioned a timeout when trying to analyze the video\.”3\.Collaboration satisfaction — write a number \(1–5\) to the left of each statement, where 1 = not satisfied and 5 = satisfied:•3— The agent understood the task, context, and instructions provided\.•1— The agent responded appropriately to requests and adapted when given feedback or corrections\.•1— The agent reduced effort, time, or cognitive load during the task\.•1— The agent proactively suggested useful ideas, alternatives, or next steps when appropriate\.•1— The agent produced accurate, relevant, and trustworthy outputs that required minimal correction\.•1— The interaction felt smooth, with the agent contributing at the right moments without creating unnecessary work\.•2— I was able to guide, redirect, and override the agent’s actions when desired\.4\.Rate the overall collaboration experience from 1–5 \(1 not satisfy, 5 satisfy\)\.1Step 2:Develop the sound design conceptStep 3:Find and select appropriate sound effectsStep 4:Generate a sound effect from scratchStep 5:Create a step by step guide to improve the realism of a sound effect using effects \(EQ, Reverb, etc\.\)

Table 8:Distribution of collaboration satisfaction and output quality ratings, by session\.Table[8](https://arxiv.org/html/2610.21109#A1.T8)shows the rating distribution by participants for collaboration satisfaction \(1 low, 5 high\) and output quality \(1 low, 5 high\) for each step\.

![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/adoption_chain_diagram.png)Figure 13:How three professionals \(musician, marketing specialist, and attorney\) adopt, modify, and reject agent outputs\.This is based on the real data point inKolab\.
### A\.4Data Processing and Annotation Details

The actions from workflow video are aligned with steps documented by each professional\. Claude was used to map video timestamps to the corresponding steps, after which professionals reviewed and corrected the alignments\. We then represented each workflow as a directed graph, where each node corresponds to a human or AI action and is associated with a workflow step\. Nodes with out\-degree greater than one denote a branch into concurrent sub\-paths, and nodes with in\-degree greater than one denote a merge of previously concurrent branches back into a single line of work\. The authors and professionals jointly reviewed and refined these graphs\. Graphs can include branches when activities occur in parallel and loops when the same action is repeated, allowing the representation to capture non\-linear workflow structures as shown in Figure[12](https://arxiv.org/html/2610.21109#A1.F12)\. After graph construction, two human annotators annotated how agent outputs were incorporated into the workflow\. For each observed response to an agent output, annotators identified the source node, corresponding to the human action that initiated the request, and the agent node, corresponding to the resulting agent response\. They then identified the subsequent human node in which the response was acted upon and labeled the user’s response asAdopt,Modify, orReject\.Adoptindicates that the AI output was incorporated without modification;Modifyindicates that it was incorporated with changes; andRejectindicates that the user explicitly rejected the output or did not incorporate it into the workflow\. Because an AI output may be acted upon later in the workflow rather than immediately after it is produced, these annotations explicitly link each user response to the AI output it concerns\.

Table 9:Real examples ofAdopt,Modify, andRejectresponses fromKolab, drawn from a music producer workflow, architecture workflow, and a video\-production workflow respectively\.#### A\.4\.1Annotation Guidelines

Each workflow graph records not only the sequence of user prompts and Claude responses, but also how each Claude response was incorporated into the user’s workflow\. This is captured by theuser\_responsefield, which is added only to the user node that represents the adopt, modify, or reject decision\.

user\_responsetakes the following form:

```
"user_response": [["source_node_id", "claude_node_id", "label",
                    "source_node_time", "claude_node_time",
                    "user_response_time"]]
```

This ensures that each adopt, modify, or reject annotation is linked to the correct occurrence of the repeated node, rather than being ambiguous across its several instances\.

##### Field definitions\.

- •source\_node\_id— The user node ID that initiated the Claude request for this step of the workflow\. This node may originate from a previous step: all subsequent user responses within the same workflow continue to reference it as the source, even as the user issues additional requests\.source\_node\_idis left empty when the user reuses a prior Claude response within a new step whose narrative is not coherently linked back to the node from that earlier step\.
- •claude\_node\_id— The Claude node ID containing the response to the request\.
- •label— One ofadopt,modify, orreject, indicating how the user incorporated the Claude response into the workflow\. On rare occasions a single node carries more than one label; in that case, each label is recorded as a separate tuple with its ownsource\_node\_idandclaude\_node\_id\.

##### Label definitions\.

- •Adopt: The user incorporates Claude’s response into the workflow without changes\.
- •Modify: The user incorporates Claude’s response into the workflow after modifying it\.
- •Reject: The user does not incorporate Claude’s response into the workflow\. This label is applied only when the user explicitly states that they are rejecting the response\.

Examples of how professionals adopt, modify, or reject agent outputs are shown in Figure[13](https://arxiv.org/html/2610.21109#A1.F13)\. A request from a previous step can undergo multiple refinements before being adopted in a later step\.

### A\.5Details for Constructing and Annotating Graphs from Videos

Figure[14](https://arxiv.org/html/2610.21109#A1.F14)shows our workflow annotation interface\. Professionals first align their documented workflow steps with the corresponding video timestamps \(Figure[14\(a\)](https://arxiv.org/html/2610.21109#A1.F14.sf1)\)\. Step names are taken from their workflow documents and initially mapped to the video by Claude Code Sonnet 5\.5\. After professionals correct the step boundaries, we extract a workflow graph for each step and present it for verification \(Figure[14\(b\)](https://arxiv.org/html/2610.21109#A1.F14.sf2)\), where professionals confirm whether the high\-level workflow structure is consistent with their actual process\. The prompt for generating the initial graph by Claude is shown below\.

Four members of the research team then reconstruct and annotate the full workflow graphs, as the automatically extracted graphs frequently contain errors in action labels, edges, timestamps, and actor assignments\. Annotators use the screen recordings, think\-aloud transcripts, agent chat logs, and input/output files to ground these corrections\. When an action or dependency remains ambiguous, the research team consults the corresponding professional for clarification\.

Workflow Graph ExtractionYou are given:1\.A workflow video\.2\.A list of workflow steps with timestamps\.3\.The workflow goal\.4\.A transcript of the video \(\.srt\)\.5\.A workflow planning document \(\.docx\)\.6\.A chat log between human user and ClaudeYour task is to extract a behavioral workflow graph, capturing goal\-directed activities performed by the human user and AI throughout the workflow\.Behavioral nodes A behavioral node represents what the actor is trying to accomplish, not the computer use actions\. You must focus on the underlying work activity, intention, or objective and include relevant context, such as the object being worked on or the purpose of the activity\. Examples: \* Define objective \* Plan approach \* Seek information \* Gather evidence \* Analyze information \* Compare alternatives \* Refine requirements \* Verify findings \* Make decision \* Implement solution \* Validate outcome \* Revise solution These are examples only, not a fixed taxonomy\. Use a more specific label when the context supports it\. For example, prefer: \> “Evaluate candidate methods for the analysis” over: \> “Read information” Do not create nodes for low\-level actions such as clicking, typing, scrolling, switching tabs, opening files, copying text, or reading an AI response\. Similarly, do not create nodes simply because the actor is using a particular software application\. Only include a software interaction when it represents a meaningful work activity\. A behavioral graph corresponds to the workflow step at one time\-stamp range\. For example, if the workflow is “Step 2” →\\rightarrow“Step 3”→\\rightarrow“Step 2”, you must have 3 different behavioral graphs\.Use the video and workflow context to infer the actor’s behavior\. Do not invent intentions that are unsupported by the evidence\. Actors Include meaningful behavioral activities performed by both: \* \`user\` \* \`ai brand\` For AI, indicate whether it is \`claude\`, \`grok\`, or other types of AI\. Most of AI here are Claude, but some users may use other AI too, so please check carefully\. For AI behavior, describe the meaningful contribution to the user’s workflow rather than individual tool calls\. Softwares Infer the softwares being used in the screen\. Graph structure Construct a directed graph: \* Nodes represent behavioral activities\. \* Edges represent transitions between activities\. \* Include loops when the workflow revisits an earlier behavioral activity or repeatedly follows a behavioral cycle\. \* Use edge types such as \`sequential\` or \`loop\` when appropriate\. Output Return only valid JSON following the format below: ``` { "session_id": "...", "participant_id": "...", "title": "...", "goal": "...", "steps": [ { "key": "seg0a", "badge": "Step ID (e.g., Step-1, Custom-1, and so on", "step_name": "Step name from steps.json", "step_action": "Step action from steps.json", "time_range": "Step time range from steps.json", "start_seconds": 0, "end_seconds": 600, "nodes": [ { "id": "n1", "actor": "user", "label": "The behavioral node label", "detail": "A brief description of the behavioral activity.", "time": ["HH:MM:SS1-HH:MM:SS2", ...], "actor_type": "user", "softwares": "...", "source_steps": "..." }, { "id": "n2", "actor": "claude or grok or gemini or chatgpt", "label": "The behavioral node label for an AI-authored action", "detail": "A brief description of the behavioral activity.", "time": ["HH:MM:SS1-HH:MM:SS2", ...], "actor_type": "ai", "softwares": "...", "source_steps": "..." } ], "edges": [ { "from": "node_id", "to": "node_id", "type": "sequential" } ] } ] } ``` Note on \`actor\` vs\. \`actor\_type\`: \`actor\` is the specific actor performing the node — \`user\`, or the AI’s actual name \(\`claude\`, \`grok\`, \`chatgpt\`, \`gemini\`, etc\.\)\. \`actor\_type\` is always just the coarse category — \`user\` or \`ai\` — never an AI brand name\. Do not swap these: an AI node must have \`actor\_type: "ai"\` \(with the brand name in \`actor\`\), not \`actor\_type: "claude"\`\.For each node: \* \`label\`: the behavioral node label \* \`detail\`: brief contextual description of the activity\. Only include 2–3 short sentences\! \* \`time\`: \[the start and ending timestamps of that behavior\] \* \`source\_steps\`: workflow step\(s\) supporting the node Before returning the result, verify that every node represents a meaningful goal\-directed activity rather than a low\-level interface action, and that the graph accurately reflects the workflow’s behavioral sequence and iterations\. You must watch the video to correctly create the graph\! Don’t rely on the \.srt transcript since it can be wrong\! Save the video frames for later use\. Double\-check if there are some loops in the graph\. If there are loops, make sure that it is presented in the JSON file correctly\. For example, if the loop is node A →\\rightarrownode B→\\rightarrownode A, there should be two edges \(from node A to node B and node B back to node A\)\. A loop can also be A→\\rightarrowB→\\rightarrowC→\\rightarrowA→\\rightarrowD, so please check carefully\.

![Refer to caption](https://arxiv.org/html/2610.21109v1/images/annot_webpage1.png)\(a\)The interface for professionals to align steps and timestamps\.
![Refer to caption](https://arxiv.org/html/2610.21109v1/images/annot_webpage2.png)\(b\)The interface for professionals to verify and revise workflow\.

Figure 14:A webpage for professionals to align timestamps with steps and verify the generated graphs\. Professionals first verify workflow steps and their corresponding video timestamps \(top\), and subsequently inspect the workflow graph for each step \(bottom\)\.#### A\.5\.1How Humans Construct the Graph

While an LLM was used to generate initial workflow graphs from the video, think\-aloud transcript, workflow document, step mappings, and AI chat logs collected from professionals, the research team manually reviewed and substantially reconstructed the graphs, often effectively from scratch\.

### A\.6Prompts for Evaluating MLLMs

We run each experiment under seven input conditions, one for every non\-empty combination of video, workflow graph, and chat\. The combined conditions reuse the parts of the single\-source prompts, so we show only the video\-only, graph\-only, and chat\-only prompts\.

#### A\.6\.1Experiment 1a: Adoption Behavior Prediction Prompt \(with target timestamp\)

Experiment 1a: video\-onlyYou are given a video snippet of \{PROFESSION\_PHRASE\} working with an AI agent toward the following goal:Goal: \{GOAL\} The workflow consists of multiple steps that contribute to achieving this goal\. The AI may produce an output in one step, and the user may act on that output either within the same step or in a later step\. AI Action Step: \{AI\_ACTION\_STEP\} User Action Step: \{USER\_ACTION\_STEP\} Target AI Action Timestamp: \{START\_TIME\}–\{END\_TIME\} You must predict how the user will respond to the output of the target AI action: \* ADOPT: Use the AI output as provided, without any modification\. \* MODIFY: Use the AI output but modify it in any way, including minor edits, formatting changes, or other adjustments\. \* REJECT/BYPASS: Do not use the AI output; reject, ignore, replace, or proceed without it\. Use the workflow context available to make your prediction\. Output exactly one label: \`ADOPT\`, \`MODIFY\`, or \`REJECT/BYPASS\`

Experiment 1a: graph\-onlyYou are given the workflow graph of \{PROFESSION\_PHRASE\} working with an AI agent toward the following goal \(no video is provided for this task – rely only on the workflow graph below\):Goal: \{GOAL\} The workflow consists of multiple steps that contribute to achieving this goal\. The AI may produce an output in one step, and the user may act on that output either within the same step or in a later step\. AI Action Step: \{AI\_ACTION\_STEP\} User Action Step: \{USER\_ACTION\_STEP\} Workflow graph for this portion of the workflow is defined as follows \(each node is one action taken by the human or the agent; edges show the order they happened in\): \[n0\]\(\{actor\}\) \{label\}: \{details\} \[n1\]\(\{actor\}\) \{label\}: \{details\} … A directed edge X →\\rightarrowY means that action Y follows from or depends on action X\. The graph is directed and may contain branches\.Edges \(order of actions\): n0 \-\> n1 n1 \-\> n2 … You must predict how the user will respond to the output of the target AI action \(the node above matching the “AI Action Step” described earlier\): \* ADOPT: Use the AI output as provided, without any modification\. \* MODIFY: Use the AI output but modify it in any way, including minor edits, formatting changes, or other adjustments\. \* REJECT/BYPASS: Do not use the AI output; reject, ignore, replace, or proceed without it\. Use the workflow context available to make your prediction\. Output exactly one label: \`ADOPT\`, \`MODIFY\`, or \`REJECT/BYPASS\`

Experiment 1a: chat\-onlyYou are given the chat of \{PROFESSION\_PHRASE\} working with an AI agent toward the following goal \(no video is provided for this task – rely only on the chat below\):Goal: \{GOAL\} The workflow consists of multiple steps that contribute to achieving this goal\. The AI may produce an output in one step, and the user may act on that output either within the same step or in a later step\. AI Action Step: \{AI\_ACTION\_STEP\} User Action Step: \{USER\_ACTION\_STEP\} Chat between the human and the AI agent during this portion of the workflow, in chronological order \(each message from the human is wrapped in \` <<human\>\>\` tags, each message from the AI agent in \`<<ai\>\>\` tags\):<human\> \{PROMPT\} </human\> <ai\> \{AI\_RESPONSE\} </ai\> \{…ADDITIONAL TURNS…\} You must predict how the user will respond to the output of the target AI action: \* ADOPT: Use the AI output as provided, without any modification\. \* MODIFY: Use the AI output but modify it in any way, including minor edits, formatting changes, or other adjustments\. \* REJECT/BYPASS: Do not use the AI output; reject, ignore, replace, or proceed without it\. Use the workflow context available to make your prediction\. Output exactly one label: \`ADOPT\`, \`MODIFY\`, or \`REJECT/BYPASS\`

#### A\.6\.2Experiment 1b & 1c & 1d: Adoption Behavior Forecasting Prompt \(clip trimmed to AI action\)

Experiment 1b & 1c & 1d: video\-onlyYou are given a video snippet of \{PROFESSION\_PHRASE\} working with an AI agent toward the following goal:Goal: \{GOAL\} The workflow consists of multiple steps that contribute to achieving this goal\. The AI may produce an output in one step, and the user may act on that output either within the same step or in a later step\. AI Action Step: \{AI\_ACTION\_STEP\} You must predict how the user will respond to the output of this AI action: \* ADOPT: Use the AI output as provided, without any modification\. \* MODIFY: Use the AI output but modify it in any way, including minor edits, formatting changes, or other adjustments\. \* REJECT/BYPASS: Do not use the AI output; reject, ignore, replace, or proceed without it\. Use the workflow context available to make your prediction\. Output exactly one label: \`ADOPT\`, \`MODIFY\`, or \`REJECT/BYPASS\`

Experiment 1b & 1c & 1d: graph\-onlyYou are given the workflow graph of \{PROFESSION\_PHRASE\} working with an AI agent toward the following goal \(no video is provided for this task – rely only on the workflow graph below\):Goal: \{GOAL\} The workflow consists of multiple steps that contribute to achieving this goal\. The AI may produce an output in one step, and the user may act on that output either within the same step or in a later step\. AI Action Step: \{AI\_ACTION\_STEP\} Workflow graph for this portion of the workflow is defined as follows \(each node is one action taken by the human or the agent; edges show the order they happened in\): \[n0\]\(\{actor\}\) \{label\}: \{details\} \[n1\]\(\{actor\}\) \{label\}: \{details\} … A directed edge X →\\rightarrowY means that action Y follows from or depends on action X\. The graph is directed and may contain branches\.Edges \(order of actions\): n0 \-\> n1 n1 \-\> n2 … You must predict how the user will respond to the output of the target AI action \(the node above matching the “AI Action Step” described earlier\): \* ADOPT: Use the AI output as provided, without any modification\. \* MODIFY: Use the AI output but modify it in any way, including minor edits, formatting changes, or other adjustments\. \* REJECT/BYPASS: Do not use the AI output; reject, ignore, replace, or proceed without it\. Use the workflow context available to make your prediction\. Output exactly one label: \`ADOPT\`, \`MODIFY\`, or \`REJECT/BYPASS\`

Experiment 1b & 1c & 1d: chat\-onlyYou are given the chat of \{PROFESSION\_PHRASE\} working with an AI agent toward the following goal \(no video is provided for this task – rely only on the chat below\):Goal: \{GOAL\} The workflow consists of multiple steps that contribute to achieving this goal\. The AI may produce an output in one step, and the user may act on that output either within the same step or in a later step\. AI Action Step: \{AI\_ACTION\_STEP\} Chat between the human and the AI agent during this portion of the workflow, in chronological order \(each message from the human is wrapped in \` <<human\>\>\` tags, each message from the AI agent in \`<<ai\>\>\` tags\):<human\> \{PROMPT\} </human\> <ai\> \{AI\_RESPONSE\} </ai\> \{…ADDITIONAL TURNS…\} You must predict how the user will respond to the output of the target AI action: \* ADOPT: Use the AI output as provided, without any modification\. \* MODIFY: Use the AI output but modify it in any way, including minor edits, formatting changes, or other adjustments\. \* REJECT/BYPASS: Do not use the AI output; reject, ignore, replace, or proceed without it\. Use the workflow context available to make your prediction\. Output exactly one label: \`ADOPT\`, \`MODIFY\`, or \`REJECT/BYPASS\`

#### A\.6\.3Experiment 2a: Collaboration Rating Prediction Prompt

Experiment 2a: video\-onlyYou are given a video snippet of \{PROFESSION\_PHRASE\} working with an AI agent toward the following goal:Goal: \{GOAL\} The video shows a step of a multi\-step workflow for accomplishing the goal: Step: \{STEP\_NAME\} Watch how the human and the AI agent interact throughout this step\. Examine whether the agent understood the task, context, and instructions; whether it adapted appropriately when given feedback or corrections; whether it reduced the human’s effort, time, or cognitive load; whether it proactively suggested useful ideas or next steps; whether its output was accurate, relevant, and trustworthy with minimal correction needed; whether the interaction felt smooth, with the agent contributing at the right moments without creating unnecessary work; and whether the human was able to guide, redirect, or override the agent’s actions when desired\. Based on this, rate the overall collaboration experience for this step on a scale from 1 to 5, where 1 means that the user is not satisfied with the collaboration experience with the AI agent and 5 means that the user is satisfied with the collaboration experience with the AI agent\. Output exactly one integer, with no other text: 1, 2, 3, 4, or 5

Experiment 2a: graph\-onlyYou are given a video snippet of \{PROFESSION\_PHRASE\} working with an AI agent toward the following goal:Goal: \{GOAL\} The video shows a step of a multi\-step workflow for accomplishing the goal: Step: \{STEP\_NAME\} The workflow consists of multiple steps that contribute to achieving this goal\. The AI may produce an output in one step, and the user may act on that output either within the same step or in a later step\. Workflow graph for this portion of the workflow is defined as follows \(each node is one action taken by the human or the agent; edges show the order they happened in\): \[n0\]\(\{actor\}\) \{label\}: \{details\} \[n1\]\(\{actor\}\) \{label\}: \{details\} … A directed edge X →\\rightarrowY means that action Y follows from or depends on action X\. The graph is directed and may contain branches\.Edges \(order of actions\): n0 \-\> n1 n1 \-\> n2 … Read the graph above\. Examine whether the agent understood the task, context, and instructions; whether it adapted appropriately when given feedback or corrections; whether it reduced the human’s effort, time, or cognitive load; whether it proactively suggested useful ideas or next steps; whether its output was accurate, relevant, and trustworthy with minimal correction needed; whether the interaction felt smooth, with the agent contributing at the right moments without creating unnecessary work; and whether the human was able to guide, redirect, or override the agent’s actions when desired\. Based on this, rate the overall collaboration experience for this step on a scale from 1 to 5, where 1 means that the user is not satisfied with the collaboration experience with the AI agent and 5 means that the user is satisfied with the collaboration experience with the AI agent\. Output exactly one integer, with no other text: 1, 2, 3, 4, or 5

Experiment 2a: chat\-onlyYou are given a video snippet of \{PROFESSION\_PHRASE\} working with an AI agent toward the following goal:Goal: \{GOAL\} The video shows a step of a multi\-step workflow for accomplishing the goal: Step: \{STEP\_NAME\} Chat between the human and the AI agent during this portion of the workflow, in chronological order \(each message from the human is wrapped in \` <<human\>\>\` tags, each message from the AI agent in \`<<ai\>\>\` tags\):<human\> \{PROMPT\} </human\> <ai\> \{AI\_RESPONSE\} </ai\> \{…ADDITIONAL TURNS…\} Read the chat above\. Examine whether the agent understood the task, context, and instructions; whether it adapted appropriately when given feedback or corrections; whether it reduced the human’s effort, time, or cognitive load; whether it proactively suggested useful ideas or next steps; whether its output was accurate, relevant, and trustworthy with minimal correction needed; whether the interaction felt smooth, with the agent contributing at the right moments without creating unnecessary work; and whether the human was able to guide, redirect, or override the agent’s actions when desired\. Based on this, rate the overall collaboration experience for this step on a scale from 1 to 5, where 1 means that the user is not satisfied with the collaboration experience with the AI agent and 5 means that the user is satisfied with the collaboration experience with the AI agent\. Output exactly one integer, with no other text: 1, 2, 3, 4, or 5

#### A\.6\.4Experiment 2b: Output Quality Rating Prediction Prompt

Experiment 2b: video\-onlyYou are given a video snippet of \{PROFESSION\_PHRASE\} working with an AI agent toward the following goal:Goal: \{GOAL\} The video shows a step of a multi\-step workflow for accomplishing the goal: Step: \{STEP\_NAME\} Watch the output the AI agent produced and how it was used during this step\. Examine whether the output was accurate, relevant, and directly usable for the task; whether it required significant correction, rework, or was discarded; and whether it met the specific requirements of this step\. Based on this, rate the output quality for this step on a scale from 1 to 5, where 1 means that the user is not satisfied with the output quality and 5 means that the user is satisfied with the output quality\. Output exactly one integer, with no other text: 1, 2, 3, 4, or 5

Experiment 2b: graph\-onlyYou are given a video snippet of \{PROFESSION\_PHRASE\} working with an AI agent toward the following goal:Goal: \{GOAL\} The video shows a step of a multi\-step workflow for accomplishing the goal: Step: \{STEP\_NAME\} Workflow graph for this portion of the workflow is defined as follows \(each node is one action taken by the human or the agent; edges show the order they happened in\): \[n0\]\(\{actor\}\) \{label\}: \{details\} \[n1\]\(\{actor\}\) \{label\}: \{details\} … A directed edge X →\\rightarrowY means that action Y follows from or depends on action X\. The graph is directed and may contain branches\.Edges \(order of actions\): n0 \-\> n1 n1 \-\> n2 … Read the graph above, focusing on the output the AI agent produced and how it was used during this step\. Examine whether the output was accurate, relevant, and directly usable for the task; whether it required significant correction, rework, or was discarded; and whether it met the specific requirements of this step\. Based on this, rate the output quality for this step on a scale from 1 to 5, where 1 means that the user is not satisfied with the output quality and 5 means that the user is satisfied with the output quality\. Output exactly one integer, with no other text: 1, 2, 3, 4, or 5

Experiment 2b: chat\-onlyYou are given a video snippet of \{PROFESSION\_PHRASE\} working with an AI agent toward the following goal:Goal: \{GOAL\} The video shows a step of a multi\-step workflow for accomplishing the goal: Step: \{STEP\_NAME\} Chat between the human and the AI agent during this portion of the workflow, in chronological order \(each message from the human is wrapped in \` <<human\>\>\` tags, each message from the AI agent in \`<<ai\>\>\` tags\):<human\> \{PROMPT\} </human\> <ai\> \{AI\_RESPONSE\} </ai\> \{…ADDITIONAL TURNS…\} Read the chat above, focusing on the output the AI agent produced and how it was used during this step\. Examine whether the output was accurate, relevant, and directly usable for the task; whether it required significant correction, rework, or was discarded; and whether it met the specific requirements of this step\. Based on this, rate the output quality for this step on a scale from 1 to 5, where 1 means that the user is not satisfied with the output quality and 5 means that the user is satisfied with the output quality\. Output exactly one integer, with no other text: 1, 2, 3, 4, or 5

### A\.7Model Selection

For proprietary models, we evaluate leading video\-capable multimodal models from major providers:Gemini 3\.1 ProandGemini 3\.8 Flash\(Google\),GPT\-6 Astra\(OpenAI\), andClaude Opus 5\.5\. Gemini natively accepts video as an input\. GPT\-6 Astra has no native video modality, so we sample evenly\-spaced frames from each clip as its input; this also lets GPT\-6 Astra serve as a frame\-sampled point of comparison against Gemini’s genuinely video\-native processing\. We additionally evaluate Claude despite it likewise lacking native video input \(also requiring frame sampling\), because Claude is the specific agent our study participants interacted with during their recorded sessions, so evaluating it tests whether the very model embedded in the human\-AI collaboration can itself judge or predict the outcome of that collaboration\.

For open\-weight models, we evaluate two representative video\-capable Qwen3\-VL variants\([Bai et al\., 2025](https://arxiv.org/html/2610.21109#bib.bib9)\):Qwen3\-VL\-30B\-A3B\-Instruct, a mixture\-of\-experts model \(30B total parameters, 3B active per token\) included to test whether MoE\-style efficiency scaling preserves video\-temporal understanding, andQwen3\-VL\-32B\-Instruct, a dense model of comparable total scale that instead invests its full parameter budget in deeper visual\-language fusion at every token\. Comparing the two isolates the effect of the MoE\-vs\-dense architectural choice at a fixed parameter scale\.

### A\.8Sample Size and Selection

Claude evaluation subsetsClaude was queried manually through its web interface, so for Experiments 1a, 2a and 2b we evaluated it on fixed subsets; Experiment 1b uses all instances\. For 1a, we sampled 25 instances per label \(N = 75\), because the labels are imbalanced \(13% reject\), and a random sample would leave too few rejects for per\-class F1\. For 2a, we kept every step rated 1–4 and sampled 30 of the steps rated 5 \(N = 78\), since 5s make up 60% of the data\. 2b uses the same steps, except six with no output\-quality rating \(N = 72\)\. All subsets were drawn with a fixed seed, spread across participants

### A\.9Experiment Results

Table 10:Length \(in minutes\) of the video clips given to the models in each experiment\.Results for Experiment 1a \(recognizing the user’s response to the AI’s action\) are shown in Table[11](https://arxiv.org/html/2610.21109#A1.T11), and results for Experiment 1b \(forecasting the user’s response\) are shown in Table[12](https://arxiv.org/html/2610.21109#A1.T12)\. Results for Experiments 2a and 2b are given in Table[13](https://arxiv.org/html/2610.21109#A1.T13)and Table[14](https://arxiv.org/html/2610.21109#A1.T14), respectively\.

MacroPer\-class F1ModelSettingPRF1AdoptModifyRejectGPT\-6\-Astravideo\-only0\.7420\.7250\.7130\.8040\.6020\.732chat\-only0\.6480\.6730\.6580\.7770\.6440\.552video\-graph\-chat0\.6990\.7420\.6730\.8120\.4800\.727Claudevideo\-only0\.7360\.6480\.6760\.8430\.5560\.630chat\-only0\.6850\.5840\.6180\.8310\.4490\.574video\-graph\-chat0\.6880\.6410\.6590\.8350\.5050\.638Gemini\-3\.1\-Pro\-Previewvideo\-only0\.616±\\pm0\.0050\.622±\\pm0\.0090\.617±\\pm0\.0070\.795±\\pm0\.0050\.479±\\pm0\.0140\.577±\\pm0\.008graph\-only0\.683±\\pm0\.0030\.702±\\pm0\.0050\.689±\\pm0\.0040\.823±\\pm0\.0020\.533±\\pm0\.0040\.711±\\pm0\.007chat\-only0\.589±\\pm0\.0040\.617±\\pm0\.0060\.600±\\pm0\.0050\.764±\\pm0\.0050\.494±\\pm0\.0180\.543±\\pm0\.007video\-graph0\.690±\\pm0\.0100\.712±\\pm0\.0080\.697±\\pm0\.0090\.830±\\pm0\.0070\.557±\\pm0\.0110\.703±\\pm0\.017video\-chat0\.618±\\pm0\.0090\.633±\\pm0\.0030\.621±\\pm0\.0060\.784±\\pm0\.0080\.493±\\pm0\.0070\.587±\\pm0\.014graph\-chat0\.691±\\pm0\.0090\.693±\\pm0\.0090\.682±\\pm0\.0070\.804±\\pm0\.0050\.525±\\pm0\.0160\.716±\\pm0\.006video\-graph\-chat0\.675±\\pm0\.0070\.673±\\pm0\.0060\.665±\\pm0\.0050\.804±\\pm0\.0060\.518±\\pm0\.0060\.671±\\pm0\.008Gemini\-3\.8\-Flashvideo\-only0\.588±\\pm0\.0090\.588±\\pm0\.0090\.587±\\pm0\.0100\.809±\\pm0\.0020\.425±\\pm0\.0200\.527±\\pm0\.008graph\-only0\.699±\\pm0\.0110\.700±\\pm0\.0070\.699±\\pm0\.0090\.849±\\pm0\.0030\.566±\\pm0\.0060\.684±\\pm0\.018chat\-only0\.625±\\pm0\.0120\.632±\\pm0\.0100\.628±\\pm0\.0110\.803±\\pm0\.0070\.509±\\pm0\.0250\.571±\\pm0\.010video\-graph0\.690±\\pm0\.0060\.700±\\pm0\.0060\.694±\\pm0\.0060\.849±\\pm0\.0040\.545±\\pm0\.0130\.689±\\pm0\.002video\-chat0\.612±\\pm0\.0040\.629±\\pm0\.0070\.620±\\pm0\.0050\.799±\\pm0\.0060\.479±\\pm0\.0160\.580±\\pm0\.006graph\-chat0\.700±\\pm0\.0100\.706±\\pm0\.0120\.702±\\pm0\.0120\.842±\\pm0\.0050\.560±\\pm0\.0200\.702±\\pm0\.013video\-graph\-chat0\.689±\\pm0\.0140\.696±\\pm0\.0150\.691±\\pm0\.0150\.834±\\pm0\.0080\.565±\\pm0\.0240\.675±\\pm0\.020Qwen3\-VL\-30B\-A3B\-Instructvideo\-only0\.500±\\pm0\.0170\.557±\\pm0\.0230\.511±\\pm0\.0190\.680±\\pm0\.0090\.371±\\pm0\.0350\.481±\\pm0\.025graph\-only0\.549±\\pm0\.0010\.605±\\pm0\.0010\.542±\\pm0\.0050\.723±\\pm0\.0120\.409±\\pm0\.0130\.494±\\pm0\.013chat\-only0\.540±\\pm0\.0090\.518±\\pm0\.0020\.502±\\pm0\.0020\.773±\\pm0\.0010\.306±\\pm0\.0040\.427±\\pm0\.002video\-graph0\.593±\\pm0\.0110\.611±\\pm0\.0100\.589±\\pm0\.0100\.791±\\pm0\.0050\.426±\\pm0\.0180\.550±\\pm0\.013video\-chat0\.583±\\pm0\.0260\.541±\\pm0\.0180\.536±\\pm0\.0190\.788±\\pm0\.0070\.327±\\pm0\.0330\.494±\\pm0\.018graph\-chat0\.586±\\pm0\.0090\.607±\\pm0\.0110\.570±\\pm0\.0120\.785±\\pm0\.0060\.382±\\pm0\.0090\.544±\\pm0\.024video\-graph\-chat0\.611±\\pm0\.0070\.611±\\pm0\.0010\.591±\\pm0\.0050\.806±\\pm0\.0040\.401±\\pm0\.0120\.568±\\pm0\.002Qwen3\-VL\-32B\-Instructvideo\-only0\.524±\\pm0\.0090\.529±\\pm0\.0100\.337±\\pm0\.0070\.123±\\pm0\.0070\.361±\\pm0\.0040\.526±\\pm0\.014graph\-only0\.589±\\pm0\.0030\.558±\\pm0\.0010\.476±\\pm0\.0020\.430±\\pm0\.0040\.365±\\pm0\.0010\.634±\\pm0\.005chat\-only0\.492±\\pm0\.0050\.476±\\pm0\.0060\.393±\\pm0\.0010\.424±\\pm0\.0090\.351±\\pm0\.0070\.403±\\pm0\.003video\-graph0\.597±\\pm0\.0040\.615±\\pm0\.0020\.559±\\pm0\.0040\.621±\\pm0\.0070\.414±\\pm0\.0020\.643±\\pm0\.004video\-chat0\.481±\\pm0\.0050\.492±\\pm0\.0060\.416±\\pm0\.0040\.462±\\pm0\.0050\.364±\\pm0\.0060\.423±\\pm0\.012graph\-chat0\.564±\\pm0\.0010\.585±\\pm0\.0010\.528±\\pm0\.0020\.609±\\pm0\.0030\.405±\\pm0\.0050\.570±\\pm0\.001video\-graph\-chat0\.533±\\pm0\.0110\.570±\\pm0\.0090\.529±\\pm0\.0100\.664±\\pm0\.0070\.388±\\pm0\.0070\.537±\\pm0\.017

Table 11:Result for Experiment 1a\. Mean±\\pmstd over runs where a model has several; bold marks the best score within each model\.MacroPer\-class F1ModelSettingPRF1AdoptModifyRejectGPT\-6\-Astravideo\-only0\.539±\\pm0\.0030\.524±\\pm0\.0130\.513±\\pm0\.0030\.721±\\pm0\.0230\.582±\\pm0\.0980\.235±\\pm0\.070chat\-only0\.3650\.3710\.3680\.6950\.4080\.000video\-graph\-chat0\.5460\.5120\.5120\.6960\.5920\.250Claudevideo\-only0\.6100\.4830\.4920\.7900\.4340\.250chat\-only0\.4280\.4140\.4080\.7700\.3780\.076video\-graph\-chat0\.6230\.4940\.5140\.8090\.4610\.272Gemini\-3\.1\-Pro\-Previewvideo\-only0\.461±\\pm0\.0260\.402±\\pm0\.0110\.391±\\pm0\.0130\.710±\\pm0\.0120\.333±\\pm0\.0180\.129±\\pm0\.031graph\-only0\.523±\\pm0\.0170\.478±\\pm0\.0160\.492±\\pm0\.0170\.775±\\pm0\.0060\.390±\\pm0\.0370\.311±\\pm0\.015chat\-only0\.399±\\pm0\.0110\.387±\\pm0\.0160\.390±\\pm0\.0140\.694±\\pm0\.0070\.313±\\pm0\.0100\.162±\\pm0\.027video\-graph0\.522±\\pm0\.0080\.469±\\pm0\.0030\.482±\\pm0\.0040\.781±\\pm0\.0050\.404±\\pm0\.0130\.261±\\pm0\.022video\-chat0\.431±\\pm0\.0070\.415±\\pm0\.0040\.420±\\pm0\.0050\.743±\\pm0\.0030\.320±\\pm0\.0280\.195±\\pm0\.018graph\-chat0\.488±\\pm0\.0310\.446±\\pm0\.0230\.459±\\pm0\.0260\.760±\\pm0\.0120\.357±\\pm0\.0370\.259±\\pm0\.033video\-graph\-chat0\.509±\\pm0\.0250\.458±\\pm0\.0140\.472±\\pm0\.0170\.773±\\pm0\.0090\.374±\\pm0\.0250\.269±\\pm0\.018Gemini\-3\.8\-Flashvideo\-only0\.437±\\pm0\.0160\.383±\\pm0\.0080\.383±\\pm0\.0090\.758±\\pm0\.0090\.218±\\pm0\.0270\.172±\\pm0\.013graph\-only0\.557±\\pm0\.0130\.458±\\pm0\.0110\.475±\\pm0\.0140\.797±\\pm0\.0050\.333±\\pm0\.0310\.293±\\pm0\.008chat\-only0\.387±\\pm0\.0270\.357±\\pm0\.0200\.351±\\pm0\.0230\.725±\\pm0\.0080\.163±\\pm0\.0400\.164±\\pm0\.027video\-graph0\.526±\\pm0\.0200\.454±\\pm0\.0070\.467±\\pm0\.0100\.779±\\pm0\.0060\.319±\\pm0\.0130\.302±\\pm0\.020video\-chat0\.465±\\pm0\.0300\.408±\\pm0\.0180\.414±\\pm0\.0210\.767±\\pm0\.0090\.263±\\pm0\.0210\.212±\\pm0\.033graph\-chat0\.532±\\pm0\.0220\.450±\\pm0\.0160\.465±\\pm0\.0180\.786±\\pm0\.0090\.337±\\pm0\.0200\.272±\\pm0\.029video\-graph\-chat0\.522±\\pm0\.0120\.445±\\pm0\.0010\.458±\\pm0\.0010\.783±\\pm0\.0050\.319±\\pm0\.0070\.273±\\pm0\.002Qwen3\-VL\-30B\-A3B\-Instructvideo\-only0\.389±\\pm0\.0100\.404±\\pm0\.0110\.364±\\pm0\.0070\.501±\\pm0\.0040\.305±\\pm0\.0190\.285±\\pm0\.012graph\-only0\.478±\\pm0\.0140\.448±\\pm0\.0140\.458±\\pm0\.0140\.748±\\pm0\.0060\.246±\\pm0\.0220\.381±\\pm0\.019chat\-only0\.474±\\pm0\.0160\.387±\\pm0\.0070\.385±\\pm0\.0080\.746±\\pm0\.0020\.203±\\pm0\.0100\.205±\\pm0\.016video\-graph0\.520±\\pm0\.0170\.426±\\pm0\.0080\.439±\\pm0\.0110\.772±\\pm0\.0060\.252±\\pm0\.0260\.295±\\pm0\.006video\-chat0\.396±\\pm0\.0380\.358±\\pm0\.0130\.343±\\pm0\.0180\.769±\\pm0\.0020\.105±\\pm0\.0160\.155±\\pm0\.036graph\-chat0\.507±\\pm0\.0190\.439±\\pm0\.0080\.444±\\pm0\.0110\.781±\\pm0\.0060\.193±\\pm0\.0190\.358±\\pm0\.013video\-graph\-chat0\.447±\\pm0\.0070\.389±\\pm0\.0030\.383±\\pm0\.0040\.782±\\pm0\.0030\.142±\\pm0\.0080\.227±\\pm0\.006Qwen3\-VL\-32B\-Instructvideo\-only0\.436±\\pm0\.0120\.400±\\pm0\.0120\.235±\\pm0\.0100\.092±\\pm0\.0060\.337±\\pm0\.0070\.274±\\pm0\.022graph\-only0\.494±\\pm0\.0120\.395±\\pm0\.0020\.316±\\pm0\.0070\.454±\\pm0\.0020\.335±\\pm0\.0040\.158±\\pm0\.023chat\-only0\.339±\\pm0\.0030\.344±\\pm0\.0050\.266±\\pm0\.0040\.371±\\pm0\.0030\.314±\\pm0\.0070\.115±\\pm0\.012video\-graph0\.524±\\pm0\.0380\.428±\\pm0\.0150\.370±\\pm0\.0140\.523±\\pm0\.0120\.358±\\pm0\.0100\.229±\\pm0\.039video\-chat0\.358±\\pm0\.0020\.375±\\pm0\.0050\.315±\\pm0\.0020\.480±\\pm0\.0140\.336±\\pm0\.0120\.129±\\pm0\.021graph\-chat0\.445±\\pm0\.0090\.440±\\pm0\.0040\.395±\\pm0\.0060\.616±\\pm0\.0080\.380±\\pm0\.0030\.188±\\pm0\.010video\-graph\-chat0\.428±\\pm0\.0090\.425±\\pm0\.0040\.396±\\pm0\.0050\.629±\\pm0\.0050\.356±\\pm0\.0050\.204±\\pm0\.013

Table 12:Result for Experiment 1b\. Mean±\\pmstd over runs where a model has several; bold marks the best score within each model\.OverallMAE by true ratingModelSettingMAE↓\\downarrowExact acc\.Within\-1Spearmanρ\\rhoLow 1–2Moderate 3Mod\-high 4High 5GPT\-6\-Astravideo\-only0\.945±\\pm0\.0700\.244±\\pm0\.0310\.841±\\pm0\.0320\.379±\\pm0\.1301\.775±\\pm0\.3910\.684±\\pm0\.0310\.577±\\pm0\.1500\.988±\\pm0\.062chat\-only1\.2460\.1830\.6110\.1071\.7140\.6360\.7591\.480video\-graph\-chat0\.7400\.3650\.8870\.2631\.4290\.9170\.3730\.802Claudevideo\-only0\.8580\.2660\.8650\.1361\.6750\.7420\.2751\.131chat\-only0\.8570\.2760\.8670\.2441\.7780\.6330\.3461\.048video\-graph\-chat0\.6920\.4070\.8950\.2991\.3530\.9330\.3710\.735Gemini\-3\.1\-Pro\-Previewvideo\-only0\.685±\\pm0\.0060\.527±\\pm0\.0030\.850±\\pm0\.0030\.078±\\pm0\.0573\.175±\\pm0\.0501\.968±\\pm0\.0320\.961±\\pm0\.0090\.048±\\pm0\.006graph\-only0\.632±\\pm0\.0120\.509±\\pm0\.0030\.872±\\pm0\.0030\.273±\\pm0\.0161\.563±\\pm0\.0271\.699±\\pm0\.0490\.855±\\pm0\.0050\.221±\\pm0\.020chat\-only0\.992±\\pm0\.0180\.471±\\pm0\.0020\.752±\\pm0\.0030\.078±\\pm0\.0272\.611±\\pm0\.0561\.856±\\pm0\.0381\.131±\\pm0\.0320\.576±\\pm0\.030video\-graph0\.635±\\pm0\.0180\.525±\\pm0\.0030\.862±\\pm0\.0100\.264±\\pm0\.0201\.642±\\pm0\.0761\.731±\\pm0\.0490\.952±\\pm0\.0090\.161±\\pm0\.021video\-chat0\.768±\\pm0\.0290\.530±\\pm0\.0110\.813±\\pm0\.0050\.215±\\pm0\.0242\.627±\\pm0\.0341\.811±\\pm0\.0381\.051±\\pm0\.0480\.232±\\pm0\.057graph\-chat0\.631±\\pm0\.0020\.559±\\pm0\.0030\.851±\\pm0\.0020\.295±\\pm0\.0081\.852±\\pm0\.1161\.778±\\pm0\.0190\.984±\\pm0\.0330\.100±\\pm0\.007video\-graph\-chat0\.609±\\pm0\.0190\.550±\\pm0\.0150\.864±\\pm0\.0050\.261±\\pm0\.0481\.784±\\pm0\.2381\.822±\\pm0\.0190\.962±\\pm0\.0150\.079±\\pm0\.020Gemini\-3\.8\-Flashvideo\-only0\.800±\\pm0\.0290\.503±\\pm0\.0070\.819±\\pm0\.0140\.011±\\pm0\.0033\.108±\\pm0\.2891\.914±\\pm0\.0191\.015±\\pm0\.0450\.245±\\pm0\.026graph\-only0\.607±\\pm0\.0030\.521±\\pm0\.0040\.882±\\pm0\.0020\.276±\\pm0\.0101\.738±\\pm0\.0481\.688±\\pm0\.0190\.870±\\pm0\.0090\.149±\\pm0\.008chat\-only0\.892±\\pm0\.0180\.480±\\pm0\.0040\.777±\\pm0\.0040\.085±\\pm0\.0112\.537±\\pm0\.0321\.644±\\pm0\.0381\.076±\\pm0\.0190\.471±\\pm0\.030video\-graph0\.611±\\pm0\.0070\.525±\\pm0\.0070\.876±\\pm0\.0030\.255±\\pm0\.0151\.825±\\pm0\.0871\.849±\\pm0\.0490\.925±\\pm0\.0180\.094±\\pm0\.008video\-chat0\.731±\\pm0\.0450\.543±\\pm0\.0240\.826±\\pm0\.0070\.159±\\pm0\.0402\.843±\\pm0\.1801\.811±\\pm0\.0190\.975±\\pm0\.0320\.180±\\pm0\.053graph\-chat0\.607±\\pm0\.0070\.537±\\pm0\.0020\.878±\\pm0\.0030\.276±\\pm0\.0081\.926±\\pm0\.0321\.700±\\pm0\.0330\.924±\\pm0\.0320\.094±\\pm0\.004video\-graph\-chat0\.589±\\pm0\.0110\.559±\\pm0\.0080\.868±\\pm0\.0020\.299±\\pm0\.0271\.980±\\pm0\.0681\.789±\\pm0\.0510\.924±\\pm0\.0190\.048±\\pm0\.007Qwen3\-VL\-30B\-A3B\-Instructvideo\-only0\.898±\\pm0\.0660\.330±\\pm0\.0200\.814±\\pm0\.0330\.102±\\pm0\.0752\.225±\\pm0\.1001\.043±\\pm0\.0930\.635±\\pm0\.0860\.887±\\pm0\.083graph\-only0\.770±\\pm0\.0050\.295±\\pm0\.0060\.930±\\pm0\.0020\.194±\\pm0\.0181\.770±\\pm0\.0550\.989±\\pm0\.0370\.278±\\pm0\.0190\.910±\\pm0\.003chat\-only0\.827±\\pm0\.0310\.438±\\pm0\.0300\.844±\\pm0\.0040\.145±\\pm0\.0472\.500±\\pm0\.0561\.467±\\pm0\.0330\.764±\\pm0\.0740\.549±\\pm0\.032video\-graph0\.622±\\pm0\.0170\.482±\\pm0\.0100\.900±\\pm0\.0060\.275±\\pm0\.0461\.992±\\pm0\.0291\.387±\\pm0\.1120\.609±\\pm0\.0190\.358±\\pm0\.009video\-chat0\.721±\\pm0\.0190\.519±\\pm0\.0140\.835±\\pm0\.0070\.170±\\pm0\.0292\.765±\\pm0\.1561\.778±\\pm0\.0510\.908±\\pm0\.0460\.211±\\pm0\.031graph\-chat0\.684±\\pm0\.0110\.431±\\pm0\.0090\.902±\\pm0\.0070\.277±\\pm0\.0251\.907±\\pm0\.0321\.167±\\pm0\.0330\.562±\\pm0\.0150\.516±\\pm0\.008video\-graph\-chat0\.634±\\pm0\.0160\.520±\\pm0\.0110\.875±\\pm0\.0070\.285±\\pm0\.0332\.059±\\pm0\.0591\.544±\\pm0\.0840\.811±\\pm0\.0530\.222±\\pm0\.000Qwen3\-VL\-32B\-Instructvideo\-only1\.194±\\pm0\.0180\.193±\\pm0\.0160\.634±\\pm0\.0040\.130±\\pm0\.0281\.425±\\pm0\.0870\.333±\\pm0\.1450\.739±\\pm0\.0331\.578±\\pm0\.005graph\-only0\.692±\\pm0\.0130\.415±\\pm0\.0120\.891±\\pm0\.0030\.236±\\pm0\.0181\.754±\\pm0\.0271\.043±\\pm0\.0190\.479±\\pm0\.0240\.640±\\pm0\.009chat\-only0\.803±\\pm0\.0050\.476±\\pm0\.0030\.830±\\pm0\.0050\.187±\\pm0\.0142\.519±\\pm0\.0321\.544±\\pm0\.0380\.899±\\pm0\.0070\.421±\\pm0\.004video\-graph0\.657±\\pm0\.0120\.426±\\pm0\.0040\.914±\\pm0\.0080\.243±\\pm0\.0131\.825±\\pm0\.0501\.226±\\pm0\.0560\.493±\\pm0\.0140\.535±\\pm0\.025video\-chat0\.719±\\pm0\.0040\.496±\\pm0\.0070\.840±\\pm0\.0040\.172±\\pm0\.0072\.588±\\pm0\.0001\.567±\\pm0\.0880\.861±\\pm0\.0370\.289±\\pm0\.016graph\-chat0\.615±\\pm0\.0060\.524±\\pm0\.0060\.878±\\pm0\.0030\.297±\\pm0\.0122\.259±\\pm0\.0851\.489±\\pm0\.0380\.777±\\pm0\.0070\.184±\\pm0\.013video\-graph\-chat0\.604±\\pm0\.0120\.532±\\pm0\.0080\.876±\\pm0\.0020\.282±\\pm0\.0282\.196±\\pm0\.0341\.589±\\pm0\.0190\.781±\\pm0\.0130\.159±\\pm0\.011

Table 13:Result for Experiment 2a\. Mean±\\pmstd over runs where a model has several; bold marks the best score within each model\. Band columns give the MAE within each true\-rating band\.OverallMAE by true ratingModelSettingMAE↓\\downarrowExact acc\.Within\-1Spearmanρ\\rhoLow 1–2Moderate 3Mod\-high 4High 5GPT\-6\-Astravideo\-only0\.912±\\pm0\.0840\.332±\\pm0\.0670\.805±\\pm0\.0300\.352±\\pm0\.1011\.853±\\pm0\.2550\.711±\\pm0\.0770\.710±\\pm0\.1570\.855±\\pm0\.105chat\-only1\.1340\.2210\.7130\.1631\.5000\.7270\.7881\.320video\-graph\-chat0\.7650\.4380\.8540\.3111\.5000\.7500\.5000\.741Claudevideo\-only0\.8170\.3440\.8490\.2081\.5680\.6250\.4051\.018chat\-only0\.9730\.2670\.8090\.1371\.7891\.0590\.6101\.052video\-graph\-chat0\.7410\.3620\.9170\.3221\.5560\.7650\.2800\.897Gemini\-3\.1\-Pro\-Previewvideo\-only0\.764±\\pm0\.0160\.508±\\pm0\.0020\.821±\\pm0\.0060\.046±\\pm0\.0393\.225±\\pm0\.0502\.000±\\pm0\.0001\.023±\\pm0\.0180\.062±\\pm0\.017graph\-only0\.750±\\pm0\.0220\.471±\\pm0\.0080\.836±\\pm0\.0050\.299±\\pm0\.0211\.428±\\pm0\.0501\.458±\\pm0\.0140\.891±\\pm0\.0300\.440±\\pm0\.030chat\-only1\.069±\\pm0\.0290\.436±\\pm0\.0080\.729±\\pm0\.0130\.093±\\pm0\.0212\.561±\\pm0\.1101\.676±\\pm0\.0511\.248±\\pm0\.0630\.621±\\pm0\.066video\-graph0\.763±\\pm0\.0050\.478±\\pm0\.0040\.825±\\pm0\.0040\.280±\\pm0\.0131\.514±\\pm0\.1091\.558±\\pm0\.0290\.974±\\pm0\.0170\.384±\\pm0\.033video\-chat0\.895±\\pm0\.0320\.485±\\pm0\.0040\.776±\\pm0\.0130\.139±\\pm0\.0383\.123±\\pm0\.0301\.765±\\pm0\.0511\.142±\\pm0\.0250\.246±\\pm0\.056graph\-chat0\.734±\\pm0\.0310\.485±\\pm0\.0080\.845±\\pm0\.0140\.287±\\pm0\.0371\.789±\\pm0\.0531\.598±\\pm0\.0900\.951±\\pm0\.0240\.261±\\pm0\.035video\-graph\-chat0\.759±\\pm0\.0100\.479±\\pm0\.0090\.834±\\pm0\.0040\.253±\\pm0\.0101\.842±\\pm0\.0531\.706±\\pm0\.0290\.976±\\pm0\.0320\.262±\\pm0\.018Gemini\-3\.8\-Flashvideo\-only0\.792±\\pm0\.0060\.503±\\pm0\.0040\.811±\\pm0\.0020\.042±\\pm0\.0123\.167±\\pm0\.0501\.975±\\pm0\.0251\.049±\\pm0\.0300\.114±\\pm0\.036graph\-only0\.652±\\pm0\.0160\.513±\\pm0\.0090\.870±\\pm0\.0070\.286±\\pm0\.0281\.659±\\pm0\.0661\.517±\\pm0\.0380\.810±\\pm0\.0230\.260±\\pm0\.005chat\-only1\.113±\\pm0\.0040\.431±\\pm0\.0040\.697±\\pm0\.0040\.094±\\pm0\.0082\.596±\\pm0\.1321\.804±\\pm0\.0451\.313±\\pm0\.0190\.633±\\pm0\.025video\-graph0\.643±\\pm0\.0030\.524±\\pm0\.0040\.869±\\pm0\.0030\.303±\\pm0\.0111\.877±\\pm0\.0251\.667±\\pm0\.0380\.891±\\pm0\.0130\.138±\\pm0\.009video\-chat0\.777±\\pm0\.0200\.509±\\pm0\.0060\.811±\\pm0\.0090\.187±\\pm0\.0142\.842±\\pm0\.0911\.853±\\pm0\.0291\.012±\\pm0\.0530\.108±\\pm0\.008graph\-chat0\.669±\\pm0\.0150\.513±\\pm0\.0130\.864±\\pm0\.0020\.306±\\pm0\.0171\.807±\\pm0\.0301\.618±\\pm0\.0510\.919±\\pm0\.0250\.146±\\pm0\.020video\-graph\-chat0\.658±\\pm0\.0110\.525±\\pm0\.0020\.862±\\pm0\.0060\.320±\\pm0\.0221\.772±\\pm0\.1321\.647±\\pm0\.0290\.907±\\pm0\.0250\.129±\\pm0\.022Qwen3\-VL\-30B\-A3B\-Instructvideo\-only1\.281±\\pm0\.0110\.278±\\pm0\.0190\.641±\\pm0\.0100\.048±\\pm0\.0341\.978±\\pm0\.1901\.125±\\pm0\.0250\.971±\\pm0\.0331\.415±\\pm0\.019graph\-only0\.767±\\pm0\.0060\.384±\\pm0\.0070\.876±\\pm0\.0100\.206±\\pm0\.0281\.558±\\pm0\.0660\.975±\\pm0\.0500\.437±\\pm0\.0130\.829±\\pm0\.008chat\-only0\.926±\\pm0\.0190\.439±\\pm0\.0170\.783±\\pm0\.0130\.133±\\pm0\.0082\.702±\\pm0\.0611\.529±\\pm0\.0780\.923±\\pm0\.0600\.546±\\pm0\.026video\-graph0\.731±\\pm0\.0080\.445±\\pm0\.0140\.868±\\pm0\.0040\.225±\\pm0\.0321\.529±\\pm0\.1091\.300±\\pm0\.0660\.652±\\pm0\.0570\.566±\\pm0\.021video\-chat0\.884±\\pm0\.0400\.473±\\pm0\.0190\.781±\\pm0\.0100\.113±\\pm0\.0662\.877±\\pm0\.1521\.706±\\pm0\.0781\.008±\\pm0\.0190\.348±\\pm0\.072graph\-chat0\.711±\\pm0\.0040\.469±\\pm0\.0060\.866±\\pm0\.0040\.268±\\pm0\.0111\.789±\\pm0\.1391\.363±\\pm0\.0450\.772±\\pm0\.0140\.376±\\pm0\.007video\-graph\-chat0\.699±\\pm0\.0230\.499±\\pm0\.0080\.858±\\pm0\.0180\.323±\\pm0\.0271\.667±\\pm0\.1321\.588±\\pm0\.1350\.915±\\pm0\.0320\.232±\\pm0\.021Qwen3\-VL\-32B\-Instructvideo\-only1\.223±\\pm0\.0130\.202±\\pm0\.0070\.633±\\pm0\.0190\.153±\\pm0\.0051\.442±\\pm0\.0660\.408±\\pm0\.1230\.828±\\pm0\.0461\.601±\\pm0\.026graph\-only0\.775±\\pm0\.0150\.385±\\pm0\.0120\.849±\\pm0\.0030\.254±\\pm0\.0091\.558±\\pm0\.0250\.858±\\pm0\.0140\.509±\\pm0\.0150\.825±\\pm0\.016chat\-only0\.925±\\pm0\.0160\.413±\\pm0\.0150\.794±\\pm0\.0000\.165±\\pm0\.0252\.456±\\pm0\.0301\.373±\\pm0\.0170\.935±\\pm0\.0310\.607±\\pm0\.011video\-graph0\.732±\\pm0\.0030\.425±\\pm0\.0040\.858±\\pm0\.0070\.252±\\pm0\.0111\.674±\\pm0\.0871\.008±\\pm0\.0520\.520±\\pm0\.0250\.689±\\pm0\.040video\-chat0\.804±\\pm0\.0280\.436±\\pm0\.0230\.817±\\pm0\.0080\.199±\\pm0\.0442\.404±\\pm0\.0301\.284±\\pm0\.0450\.813±\\pm0\.0070\.470±\\pm0\.048graph\-chat0\.686±\\pm0\.0130\.452±\\pm0\.0060\.898±\\pm0\.0090\.317±\\pm0\.0291\.982±\\pm0\.0301\.118±\\pm0\.0880\.703±\\pm0\.0140\.399±\\pm0\.008video\-graph\-chat0\.661±\\pm0\.0100\.478±\\pm0\.0080\.893±\\pm0\.0080\.319±\\pm0\.0222\.000±\\pm0\.0001\.216±\\pm0\.0450\.703±\\pm0\.0070\.324±\\pm0\.014

Table 14:Result for Experiment 2b\. Mean±\\pmstd over runs where a model has several; bold marks the best score within each model\. Band columns give the MAE within each true\-rating band\.
### A\.10Output Quality×\\timesCollaboration Rating

As shown in Figure[15](https://arxiv.org/html/2610.21109#A1.F15), output quality and the collaboration experience perceived by professionals are correlated\.

Figure 15:Professionals’ ratings of collaboration satisfaction and output quality across all steps\.
### A\.11Model Comparison on Common Instance Sets

Due to resource limitations, GPT\-6\-Astra was evaluated on fewer instances than the other models\. For a fair comparison, we also evaluate all models on the same instances used for GPT\-6\-Astra\. Scores therefore differ from those in the full tables \(Tables[11](https://arxiv.org/html/2610.21109#A1.T11)and[12](https://arxiv.org/html/2610.21109#A1.T12)\)\.

Table 15:F1 for Experiment 1a on the common set of 279 instances from 32 participants\. The common set is limited by GPT\-6\-Astra, which has 279 valid outputs\.Table 16:F1 for Experiment 1b on the common set of 349 instances from 37 participants\. The common set is limited by GPT\-6\-Astra, which has 349 valid outputs\. GPT\-6\-Astra’s score is based on its single run covering all 349 instances\.
### A\.12Post\-interaction Questionnaire

#### Overview: All Behavioral Likert Items \(Q13–Q38\)

![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/likert_overview.png)Figure 16:Mean rating \(1–10\) for each of the 26 AI\-collaboration behavior items, sorted low to high \(error bars: 1 SD\)\. The dashed line marks the scale midpoint\.
#### Session 1 & Session 2 Reflection

![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q1.png)\(a\)Q1\. After watching the session 1 recording again and looking at the extracted workflow, how natural do you think your workflow there compared to your general workflow?##### Q2\. If you answered not natural at all in the previous question, why do you think so?

Open\-ended response

![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q3.png)\(a\)Q3\. How difficult was the problem you worked on in Session 1?
![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q4.png)\(b\)Q4\. Have you ever done problem 1 manually without agentic AI’s help?

![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q5.png)\(a\)Q5\. How many hours on average did it take or do you think it takes for you to finish the problem for session 1 without agentic AI’s help?
![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q6.png)\(b\)Q6\. After watching the session 2 recording again and looking at the extracted workflow, how natural do you think your workflow there compared to your general workflow?

##### Q7\. If you answered not natural at all in the previous question, why do you think so?

Open\-ended response

![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q8.png)\(a\)Q8\. How difficult was the problem you worked on in Session 2?
![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q9.png)\(b\)Q9\. Have you ever done problem 2 manually without agentic AI’s help?

![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q10.png)\(a\)Q10\. How many hours on average did it take or do you think it takes for you to finish the problem for session 2 without agentic AI’s help?
##### Q11\. For either or both session 1 and 2, were there any points where you changed your plan or workflow? What prompted that?

Open\-ended response

##### Q12\. If you did a similar task again, would you approach it differently? If so, what would you change?

Open\-ended response\.

#### AI Reliance / Delegation

![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q13.png)\(a\)Q13\. I prefer letting AI complete substantial portions of my work before reviewing it\.
![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q14.png)\(b\)Q14\. I frequently ask AI to generate complete drafts instead of working incrementally\.

![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q15.png)\(a\)Q15\. When possible, I let AI handle routine subtasks\.
![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q16.png)\(b\)Q16\. I prefer completing most work myself before involving AI\.

#### Verification Behavior

![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q17.png)\(a\)Q17\. I usually verify AI\-generated information using external sources\.
![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q18.png)\(b\)Q18\. I rarely use AI output without checking it\.

![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q19.png)\(a\)Q19\. I like understanding how AI reached its conclusions\.
![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q20.png)\(b\)Q20\. If the answer looks reasonable, I usually move on without further verification\.

#### Control & Redirection

![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q21.png)\(a\)Q21\. I often redirect AI when it takes an unhelpful direction\.
![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q22.png)\(b\)Q22\. I refine my requests multiple times until the AI matches my intent\.

![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q23.png)\(a\)Q23\. I actively shape how the collaboration unfolds\.
![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q24.png)\(b\)Q24\. I usually follow the direction suggested by the AI\.

#### Exploration of Alternatives

![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q25.png)\(a\)Q25\. I use AI to explore multiple perspectives before deciding\.
![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q26.png)\(b\)Q26\. I intentionally ask AI for alternative viewpoints\.

![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q27.png)\(a\)Q27\. I enjoy exploring ideas with AI before committing to one\.
![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q28.png)\(b\)Q28\. I usually stop searching once I find one satisfactory answer\.

#### External / Cross\-checking

![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q29.png)\(a\)Q29\. I frequently combine information from AI with external documents\.
![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q30.png)\(b\)Q30\. I compare AI responses with academic papers or other trusted sources\.

#### Tool\-Switching Behavior

![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q31.png)\(a\)Q31\. My workflow often involves moving between AI and other tools\.
![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q32.png)\(b\)Q32\. Most of my work happens within a single AI chat\.

#### Planning Behavior

![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q33.png)\(a\)Q33\. Before using AI, I usually have a clear plan\.
![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q34.png)\(b\)Q34\. I break complex tasks into smaller steps before asking AI\.

![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q35.png)\(a\)Q35\. I use AI to execute a plan rather than determine one\.
#### Reflection & Reframing

![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q36.png)\(a\)Q36\. I often reconsider my approach after interacting with AI\.
![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q37.png)\(b\)Q37\. Working with AI changes how I think about the problem\.

![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q38.png)\(a\)Q38\. AI helps me discover assumptions I hadn’t considered\.
#### Error Cost & Risk Perception

![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q39.png)\(a\)Q39\. What is the average hourly wage \(in US Dollar\) for a job similar to yours?![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q40.png)Figure 37:Q40\. How serious would an undetected material error be in the tasks you did for this study?![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q41.png)Figure 38:Q41\. Estimated financial/resource cost of an undetected error:![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q42.png)Figure 39:Q42\. Did you notice any issues in the AI content during the tasks you did for this study?![Refer to caption](https://arxiv.org/html/2610.21109v1/appendix_figures/q43.png)\(a\)Q43\. Estimated chance your final output still contains a material error: