SERUM: State Extraction and Refinement for User Modeling
Summary
Presents SERUM, a multi-pass framework that extracts structured behavioral models of user actions and intents from raw egocentric video using hierarchical VLM annotation, reducing hallucinations and producing interpretable process models without manual annotation.
arXiv:2607.29181v1 Announce Type: new
Abstract: Agentic assistants capable of proactive, personalized interactions require structured models of user intent and workflow. However, building these models from raw, unstructured screen activity remains an open challenge. We present SERUM, a multi-pass framework that extracts finite-state behavioral models directly from unstructured egocentric video using hierarchical VLM annotation. Processing screen recordings through a sliding window, SERUM alternates between activity-recognition and intent-inference passes, with each pass refining labels using accumulated prior context to reduce hallucination and temporal conflation seen in single-pass annotation. Synonymous states are then merged via sentence embeddings and human-calibrated thresholds into a compact, coherent taxonomy. We evaluate behavioral structure by fitting first-order Markov models over the resulting label sequences (both actions and intents) and measuring predictive accuracy against frequency baselines. Across 61 egocentric videos in four domains (coding, cooking, physical activities, and daily life), we find: (1) iterative label refinement converges to a stable state vocabulary, which we term schematic equilibrium, after several passes; (2) normalized Markov models achieve substantially lower perplexity and higher action predictions than frequency baselines, with the largest gains on structured tasks like coding; and (3) human annotators rate final-pass labels as accurate and meaningfully improved over first-pass labels. To our knowledge, SERUM is the first system to produce interpretable process models from unstructured egocentric screen video without manual annotation, opening a scalable pathway for user modeling and behavioral understanding in the wild. Our demo, code, and results are publicly available
View Cached Full Text
Cached at: 08/03/26, 07:37 AM
# Serum: State Extraction and Refinement for User Modeling
Source: [https://arxiv.org/html/2607.29181](https://arxiv.org/html/2607.29181)
Andy J\. Phu, James Mooney, Karin de Langis, Khanh Chi Le, Dongyeop Kang Minnesota NLP Lab University of Minnesota Minneapolis, MN 55455, USA \{phu00003,moone174,dento019,le000422,dongyeop\}@umn\.edu
###### Abstract
Agentic assistants capable of proactive, personalized interactions require structured models of user intent and workflow\. However, building these models from raw, unstructured screen activity remains an open challenge\. We presentSerum, a multi\-pass framework that extracts finite\-state behavioral models directly from unstructured egocentric video using hierarchical VLM annotation\. Processing screen recordings through a sliding window,Serumalternates between activity\-recognition and intent\-inference passes, with each pass refining labels using accumulated prior context to reduce hallucination and temporal conflation seen in single\-pass annotation\. Synonymous states are then merged via sentence embeddings and human\-calibrated thresholds into a compact, coherent taxonomy\. We evaluate behavioral structure by fitting first\-order Markov models over the resulting label sequences \(both actions and intents\) and measuring predictive accuracy against frequency baselines\. Across 61 egocentric videos in four domains \(coding, cooking, physical activities, and daily life\), we find: \(1\) iterative label refinement converges to a stable state vocabulary, which we termschematic equilibrium, after several passes; \(2\) normalized Markov models achieve substantially lower perplexity and higher action predictions than frequency baselines, with the largest gains on structured tasks like coding; and \(3\) human annotators rate final\-pass labels as accurate and meaningfully improved over first\-pass labels\. To our knowledge,Serumis the first system to produce interpretable process models from unstructured egocentric screen video without manual annotation, opening a scalable pathway for user modeling and behavioral understanding in the wild\. Our demo, code, and results are publicly available111[https://minnesotanlp\.github\.io/SERUM\-web/](https://minnesotanlp.github.io/SERUM-web/)
## 1Introduction
Proactive AI assistants need structured models of user behavior — compact representations of how people move through goal\-directed activity over time\. Rich egocentric footage has been scarce before platforms like YouTube, and Twitch, \. converting raw video into structured behavioral models is non\-trivial\. Existing activity recognition methods either depend on fixed hand\-crafted taxonomies\(Damenet al\.,[2022](https://arxiv.org/html/2607.29181#bib.bib9); Graumanet al\.,[2022](https://arxiv.org/html/2607.29181#bib.bib10)\)or require expensive frame\-level annotation\. Process mining produces elegant behavioral models from event logs\(van der Aalstet al\.,[2012](https://arxiv.org/html/2607.29181#bib.bib5); van der Aalst,[2016](https://arxiv.org/html/2607.29181#bib.bib6)\), but assumes activity labels already exist\. Neither path applies to unstructured, open\-ended video\.
We ask:*can we extract interpretable, structured models of user behavior directly from raw egocentric video, without a predefined ontology and without manual annotation?*
Figure 1:\(Top\) Current standard methods process each frame independently, producing isolated activity descriptions and coarse intent estimates\. \(Bottom\) SERUM’s multi\-pass pipeline revisits prior context across frames, enabling the construction of a refined user model for both user actions and intents, enabling better informed proactive suggestions\.A simple answer is to prompt a vision\-language model \(VLM\) to label each frame, similar to recent work on general user modelsShaikhet al\.\([2025](https://arxiv.org/html/2607.29181#bib.bib1)\)\. But single\-pass annotation fails in two ways: VLMs hallucinate, and they suffer from*temporal conflation*— collapsing semantically distinct activities into generic labels because they lack surrounding context\.
We introduceSERUM\(StateExtraction andRefinement forUserModeling\), a multi\-pass pipeline that addresses these limitations through alternating rounds of*activity recognition*and*intent inference*, each grounded in the accumulated context of prior passes\. SERUM operates at two complementary levels:*user actions*\(directly observable behaviors, e\.g\., pulling a git repository”\) and*user intents*\(intermediate goals, e\.g\., setting up a development environment”\)\. A sliding context window provides each pass with a run\-length encoding of surrounding frames\. After annotation, a*label normalization*step merges synonymous labels into a compact vocabulary\. SERUM outputs activity and intent models implemented as first\-order Markov chains, capturing the probabilistic transition structure of user behavior\.
We apply SERUM to 61 egocentric YouTube videos spanning coding, cooking, physical activity, and daily life\. Our experiments show that: \(1\) the extracted label vocabulary reliably converges to a stable taxonomy by pass 8 — a phenomenon we term*schematic equilibrium*; \(2\) label normalization compresses the vocabulary and sharpens transition structure, yielding better predictive models; \(3\) normalized Markov models outperform frequency baselines on both action and intent sequences; and \(4\) annotators judge final\-pass labels as correct 88\.3% of the time \(α\\alpha=0\.40\) and prefer them over first\-pass labels 82\.8% of the time \(α\\alpha=0\.41\)\.
SERUM is, to our knowledge, the first framework for extracting structured user activity and intent models directly from unstructured egocentric video — requiring no logs, no predefined taxonomies, and no labeled data\. The resulting models represent an early validation for downstream applications such as proactive agentic assistance and personalization\. Code and data are publicly available\.222https://github\.com/minnesotanlp/SERUM
## 2Related Work
Egocentric Video Understanding and Action Anticipation\.Benchmarks such as EPIC\-KITCHENS\(Damenet al\.,[2018](https://arxiv.org/html/2607.29181#bib.bib8);[2022](https://arxiv.org/html/2607.29181#bib.bib9)\)and Ego4D\(Graumanet al\.,[2022](https://arxiv.org/html/2607.29181#bib.bib10)\)have established that predicting what a user will do next requires reasoning at two levels: the immediate action and the underlying goal\. Mascaro et al\.\(Mascaroet al\.,[2023](https://arxiv.org/html/2607.29181#bib.bib17)\)exploit this hierarchy by conditioning low\-level action predictions on inferred high\-level intentions for long\-term anticipation\.Furnari and Farinella \([2020](https://arxiv.org/html/2607.29181#bib.bib18)\)show that rolling\-unrolling recurrent representations further improve anticipation on EPIC\-KITCHENS\. SERUM is complementary: rather than operating on labeled benchmark data, it infers both action and intent labels from scratch using raw, unannotated video\.
VLMs as Video Annotators and User Modelers\.Foundation vision\-language models such as the Qwen\-VL family\(Baiet al\.,[2023](https://arxiv.org/html/2607.29181#bib.bib11); Wanget al\.,[2024](https://arxiv.org/html/2607.29181#bib.bib12); Baiet al\.,[2025](https://arxiv.org/html/2607.29181#bib.bib13)\)have lowered the cost of video annotation, but single\-pass inference on long videos suffers from hallucination and temporal conflation\. ROVER\(Schroederet al\.,[2025](https://arxiv.org/html/2607.29181#bib.bib21)\)and VideoNarrator\(Wuet al\.,[2025](https://arxiv.org/html/2607.29181#bib.bib22)\)address these failures through recursive decomposition and multi\-component verification pipelines, respectively; LLMs more broadly have been shown to match or surpass crowd workers when labels are iteratively verified\(Gilardiet al\.,[2023](https://arxiv.org/html/2607.29181#bib.bib19); Heet al\.,[2024](https://arxiv.org/html/2607.29181#bib.bib20)\)\. SERUM shares the insight that iterative, context\-aware annotation suppresses hallucination, but goes further: each pass re\-annotates frames conditioned on all prior\-pass labels, enabling the model to revise earlier judgments as context accumulates and driving convergence toward a self\-consistent vocabulary without any predefined ontology\. Most closely related in motivation is GUM\(Shaikhet al\.,[2025](https://arxiv.org/html/2607.29181#bib.bib1)\), which builds user models from computer\-use screenshots by inferring and revising confidence\-weighted propositions about user preferences and knowledge\. While GUM targets*who*the user is, SERUM targets*what*the user is doing and intends to do next, producing activity and intent transition models via multi\-pass re\-annotation rather than propositions about user traits\.
Process Mining and Behavioral Sequence Models\.Process mining recovers structured process models from event logs\(van der Aalstet al\.,[2012](https://arxiv.org/html/2607.29181#bib.bib5); van der Aalst,[2016](https://arxiv.org/html/2607.29181#bib.bib6)\), but these algorithms assume clean logs of known event types\. SERUM, while conceptually related, assumes no ontology of event types and uses unstructured video as input\. Process\-mining quality criteria \(i\.e\., fitness, precision, and generalization\(Buijset al\.,[2012](https://arxiv.org/html/2607.29181#bib.bib7)\)\) motivate our use of next\-action prediction accuracy and perplexity as evaluation metrics\.
Semantic Label Normalization\.Open\-vocabulary annotation produces synonymous labels that inflate state\-space size and degrade model quality\. We consolidate them via pairwise embedding similarity using Sentence\-BERT\(Reimers and Gurevych,[2019a](https://arxiv.org/html/2607.29181#bib.bib15)\), with a human\-calibrated merging threshold \(t∗=0\.43t^\{\*\}=0\.43\)\. This is analogous to entity resolution and ontology alignment in knowledge\-base construction\. We treat normalization as a design component rather than a post\-hoc fix, and empirically show it improves both vocabulary compactness and predictive accuracy of the resulting Markov models\.
## 3Serum:StateExtraction andRefinement forUserModeling
Serumis a multi\-pass framework for extracting structured behavioral models from raw egocentric video—no predefined ontology, no manual annotation required\. Given a sequence ofTTsampled frames\{f1,…,fT\}\\\{f\_\{1\},\\ldots,f\_\{T\}\\\},Serumalternates between grounded activity recognition and intent inference, progressively refining coarse perceptual observations into temporally coherent behavioral descriptions\. The final output is a pair ofUser Models\(UMs\)—one over actions, one over intents—that compactly represent the user’s behavioral dynamics and directly support downstream applications such as proactive next\-action prediction\. Figure[2](https://arxiv.org/html/2607.29181#S3.F2)provides an overview of the full pipeline\.
Figure 2:TheSerumpipeline applied to an HVAC repair video\. Frames are annotated through alternating activity \(orange\) and intent \(blue\) passes\. Early passes yield generic labels \(e\.g\., “holding drill”\); later passes produce fine\-grained, context\-aware labels \(e\.g\., “testing HVAC repair”\)\. Labels are normalized before constructing the final User Models\.### 3\.1Actions and Intents
Serumrepresents behavior at two distinct levels\.Actions\(ata\_\{t\}\) are mid\-level natural\-language descriptors of directly observable behavior at framett\(e\.g\., “washing vegetables,” “pulling a git repository”\)\.Intents\(iti\_\{t\}\) are latent goal\-directed states, that may not be directly observable, but are inferred from sequences of actions \(e\.g\., “preparing dinner,” “setting up a dev environment”\)\.
The two levels are*mutually informative*: action evidence anchors intent inference; intent context disambiguates ambiguous actions\. For example, “looking at a phone” is labeled “checking map directions” once intent context establishes “navigating to a destination\.” This bidirectionality motivatesSerum’s alternating design—running separate independent passes for each level underperforms because neither level grounds the other\. Prompt templates are provided in Appendix[G](https://arxiv.org/html/2607.29181#A7)\.
### 3\.2Multi\-Pass Annotation Pipeline
A natural baseline is single\-pass VLM annotation: prompt the model once per frame for both labels\. This fails for two reasons: intent inference is inherently*retrospective*\(the meaning of an action often only becomes clear after observing subsequent frames\), and without shared context across frames, VLMs produce*semantically inconsistent*labels \(e\.g\., “rinsing produce” vs\. “cleaning vegetables” for the same activity\)\. Running separate independent passes for each level does not help either\. Actions and intents are*mutually informative*—intent context disambiguates ambiguous actions, while action evidence anchors intent inference\. Only by*alternating*the two—each pass conditioning on the outputs of the last—can they ground each other in a feedback loop converging toward coherent, disambiguated descriptions\.
Serumimplements this via alternating passes over frames\{f1,…,fT\}\\\{f\_\{1\},\\ldots,f\_\{T\}\\\}usingQwen3\-VL\-8B\-Instruct, where odd passes annotate actions and even passes annotate intents\. Pass 1 produces unconditioned action labels\{at\(1\)\}\\\{a\_\{t\}^\{\(1\)\}\\\}in free\-form natural language\. Pass 2 infers intent labels\{it\(1\)\}\\\{i\_\{t\}^\{\(1\)\}\\\}conditioned on\{at\(1\)\}\\\{a\_\{t\}^\{\(1\)\}\\\}; Pass 3 refines action labels conditioned on\{it\(1\)\}\\\{i\_\{t\}^\{\(1\)\}\\\}; and so on\. Each pass \(from Pass 2 onward\) receives two context signals\. First, atemporal context windowofw=20w\{=\}20neighboring frames, encoded as a run\-length encoding \(RLE\) that collapses consecutive identical states into count\-weighted entries, providing dense local context without exceeding the model’s token limit\. Second, aninter\-pass summary—a natural\-language summary generated from the full\-pass RLE and the prior summary—that propagates global narrative context forward as compressed episodic memory\. For long videos exceeding2152^\{15\}tokens, a map\-reduce procedure summarizes fragments independently before merging into a coherent global summary\. Passes continue until the label vocabulary stabilizes—empirically by pass 8—a convergence we term*schematic equilibrium*\(§[4\.2](https://arxiv.org/html/2607.29181#S4.SS2)\)\.
### 3\.3Label Normalization
Free\-form annotation produces surface synonyms that inflate vocabulary size and degrade model quality\. We resolve these via Sentence\-BERT\(Reimers and Gurevych,[2019a](https://arxiv.org/html/2607.29181#bib.bib15)\)cosine similarity with a human\-calibrated merging thresholdt∗=0\.43t^\{\*\}\{=\}0\.43, chosen to maximize F1 on human\-judged synonym pairs\.333Calibration procedure detailed in Appendix[F](https://arxiv.org/html/2607.29181#A6)\.This reduces vocabulary size by 46\.0% on average and measurably improves predictive accuracy downstream\.
### 3\.4Output: User Models
The final output is a pair ofUser Models\(UMs\): directed weighted graphsℳ=\(𝒮,ℰ\)\\mathcal\{M\}\{=\}\(\\mathcal\{S\},\\mathcal\{E\}\)where𝒮\\mathcal\{S\}is the canonicalized state vocabulary and each edge\(s,s′\)\(s,s^\{\\prime\}\)is weighted by observed transition frequency\. One UM is built over actions, one over intents, yielding complementary views of behavior \(Figure[3](https://arxiv.org/html/2607.29181#S3.F3)\)\. UMs support proactive assistance by surfacing probable next states given the user’s current state\. We evaluate predictive utility via anext\-action predictiontask: UMs trained on the first 60% of frames predict the held\-out final 40%, measuring whether captured behavioral dynamics generalize to unseen activity\.
\(\(a\)\)Action\-level UM
\(\(b\)\)Intent\-level UM
Figure 3:Example User Models from a single video\. Action\-level states \(a\) capture observable behaviors; intent\-level states \(b\) capture inferred goals\.
## 4Evaluation
We evaluateSerumon 61 egocentric videos spanning coding, cooking, physical activity, and daily life, addressing four research questions:RQ1\(§[4\.2](https://arxiv.org/html/2607.29181#S4.SS2)\) doesSerum’s iterative annotation converge to a stable label vocabulary \(*schematic equilibrium*\)?RQ2\(§[4\.3](https://arxiv.org/html/2607.29181#S4.SS3)\) do the resulting user models usefully predict next user states?RQ3\(§[4\.4](https://arxiv.org/html/2607.29181#S4.SS4)\) areSerum’s labels aligned with human judgment?RQ4\(§[4\.5](https://arxiv.org/html/2607.29181#S4.SS5)\) how do label normalization and intent passes each contribute to model quality?
### 4\.1Experimental Setup
DatasetWe evaluate on 61 egocentric videos sampled at 5\-second intervals, yielding 11,125 total frames across four domains \(Table[1](https://arxiv.org/html/2607.29181#S4.T1)\)\.
Table 1:Per\-domain dataset statistics\. Vocabulary and accuracy from the final activity \(P11\) and intent \(P12\) passes\. Values are mean±\\pmstd\. Act\. = Activity, Int\. = Intent\. We find that coding videos tend to consist of repetitive actions \(writing code\) focused on a singular goal \(releasing a snake game\), resulting in higher accuracy\.61 videos, 11,125 frames, 927 min \(15\.5 hrs\), 12 passes each, 133,500 total state extractions\. A generalization study on EPIC\-KITCHENS\-100 \(366 videos, 37 participants\) is reported in Appendix[C](https://arxiv.org/html/2607.29181#A3)\.
Model and inference\.All annotation passes use Qwen3\-VL\-8B\-Instruct\(Baiet al\.,[2025](https://arxiv.org/html/2607.29181#bib.bib13)\)444Preliminary study on model choice in Appendix[H](https://arxiv.org/html/2607.29181#A8)in BF16, served via vLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2607.29181#bib.bib16)\)with tensor parallelism across two GPUs per node\. We distribute inference across two nodes \(2×\\timesNVIDIA A5000, 2×\\timesNVIDIA A6000\), each running an independent vLLM server, achieving a combined throughput of 1\.3 inferences/sec and processing 12 passes for a 10\-minute video in≈\\approx17 minutes per node\.
Label Normalization\.We apply pairwise semantic merging using SentenceBERT embeddings\(Reimers and Gurevych,[2019b](https://arxiv.org/html/2607.29181#bib.bib2)\)with cosine\-distance thresholdt∗=0\.43t^\{\*\}=0\.43, selected to maximize F1 on a human\-annotated calibration set of 100 activities and 100 intents randomly sampled from all passes\. \(§[6\(a\)](https://arxiv.org/html/2607.29181#S4.F6.sf1)\)\.
### 4\.2RQ1: Schematic Equilibrium
Setup\.To determine how many annotation passes are needed, we ran a pilot study on 13 videos for 30 passes, tracking vocabulary size per pass before and after label normalization\.
Results\.As shown in Figure[4](https://arxiv.org/html/2607.29181#S4.F4), the average raw activity vocabulary drops from∼\\sim28 to∼\\sim18 unique states by pass 8, while the average raw intent vocabulary drops more steeply from∼\\sim54 to∼\\sim24\. Label normalization compresses both further to∼\\sim10 states and remains stable thereafter\. Per\-video vocabulary curves \(Figure[4](https://arxiv.org/html/2607.29181#S4.F4)c\) confirm that all videos individually stabilize by pass 8 — a convergence we term*schematic equilibrium*\. Based on this finding, we run all large\-scale evaluations at 12 passes \(6 activity and 6 intent, interleaved\), providing a margin beyond the observed convergence point\.
\(\(a\)\)Action: raw vs\. normalized
\(\(b\)\)Intent: raw vs\. normalized
\(\(c\)\)Per\-video vocabulary across passes\. Left: Activity, Right: Intent
Figure 4:30\-pass pilot study \(13 videos\)\. \(a\) Activity vocabulary drops from 28 to 18 raw states by pass 8, with normalization compressing further to 10\. \(b\) Intent vocabulary shows a steeper decline from 54 to 24 raw states, with normalization consistently reducing to 10 across all passes\. \(c\) Per\-video vocabulary stabilizes by pass 8 \(*schematic equilibrium*\)\.
### 4\.3RQ2: Next\-State Prediction
Setup\.We construct Markov user models from the first 60% of frames per video and evaluate on the remaining 40%, applying add\-one Laplace smoothing to transition counts\. We compare against three baselines:Majority\(always predict the most frequent training state\),Weighted Random\(sample proportional to marginal frequency\), andUniform\(sample uniformly over observed states\)\. Performance is measured bytop\-1 accuracyandperplexity\(exponentiated cross\-entropy over held\-out transitions; lower is better\)\.
Table 2:Mean top\-1 accuracy and perplexity at the final pass\.ndenotes normalized labels\.Main Result\.Table[2](https://arxiv.org/html/2607.29181#S4.T2)shows that at the final annotation pass, Markov user models outperform naive baselines in both top\-1 accuracy and perplexity\.555Top\-3 and Top\-5 show similar tendencies but more strongly favor Markov\.Normalized variants \(superscriptnn\) apply post\-hoc label normalization merging prior to model construction\. Normalization benefits the Markov model disproportionately, since fewer labels reduces sparsity in its transition matrix, whereas the Majority baseline only tracks label frequencies\.
Table 3:Markov top\-1 accuracy and perplexity by domain \(final pass, normalized labels\)\.Domain\-level results \(Table[3](https://arxiv.org/html/2607.29181#S4.T3)\) show largest gains on structured tasks: Coding achieves 76\.4% normalized activity accuracy, reflecting the rich, repetitive transition structure of coding workflows\. Cooking and physical tasks see smaller absolute accuracy but substantial relative gains from normalization\. We further find preliminary evidence that SERUM\-produced markov models can transfer to similar but unseen workflow videos\.666Preliminary study detailed in Appendix[I](https://arxiv.org/html/2607.29181#A9)\.
### 4\.4RQ3: Human Assessment of Label Quality
Setup\.We recruited five colleagues with domain expertise in human workflow research to evaluate label quality\. Annotators assessed 180 uniformly sampled frames from 9 videos \(10 activity, 10 intent per video\), of a 26 video pre\-vetted set777See the Ethics Statement for pre\-vetting criteria\., presented via a web application embedding the source video at the relevant timestamp\. For each frame, annotators judged: \(1\)label accuracy— whether the final\-pass label correctly describes the observed activity or intent; and \(2\)pass preference— whether the final\-pass or first\-pass label is better\. Responses are aggregated by majority vote; inter\-annotator agreement \(IAA\) is measured via Krippendorff’sα\\alpha\.
Results\.By majority vote,88\.3%of labels were rated accurate \(α=0\.40\\alpha=0\.40\) and final\-pass labels were preferred over first\-pass labels82\.8%of the time \(α=0\.41\\alpha=0\.41\), suggesting that iterative annotation produces meaningfully better labels \(Figure[5](https://arxiv.org/html/2607.29181#S4.F5)\)\. Intent labels were slightly more accurate than Activity labels \(90% vs\. 86\.7%\), yet preference rates were comparable across both types \(81\.1% vs\. 82\.8%\), indicating that multi\-pass refinement improves intent inference similarly to activity recognition despite intent being harder to verify from a single frame\.
Figure 5:Human annotation results and error analysis\(a\) Per\-annotator accuracy and pass preference\.
\(b\) Inter\-annotator agreement by label type\.
\(c\) Distribution of inaccuracy causes, VLM \(blue\) was responsible for 62% of issues, annotator labeling \(orange\) caused 38% of misjudgements
### 4\.5RQ4: Ablation Study
Setup\.We isolate the contribution of the three core design choices inSerum\. Forlabel norm, we compare Markov models built on raw vs\. normalized label sequences at the final pass, measuring vocabulary reduction, top\-1 accuracy, and perplexity\. Forintent passes, We compare the fullSerumpipeline against an activity\-only baseline pipeline in three scenarios\. \(1\) both pipelines infer and evaluate on their own freely generated vocabularies, \(2\) both pipelines initially infer on open vocabularies, but project activity\-only labels on to the intent\-conditioned vocabulary before evaluation, \(3\) Project in reverse direction\. Fortemporal window size, we compare SERUM’s default temporal window of n=20 with n=10 and n=0 on accuracy and perplexity against respective baselines\.
Label normalization\.Normalization reduces the state vocabulary by46\.0%on average \(±\\pm21\.1%\) while improving Markov top\-1 accuracy by\+\+18\.2 pp\(±\\pm21\.2 pp\) and reducing perplexity by 14\.9 points Gains are consistent across domains and largest for intent models, where surface\-synonym proliferation is most severe\. This validates open\-vocabulary annotation followed by principled merging as better than a fixed ontology: the former preserves fine\-grained behavioral distinctions that the latter would collapse, and normalization then recovers the compact transition structure needed for reliable Markov estimation\. The calibrated SentenceBERT threshold \(t∗=0\.43t^\{\*\}\\\!=\\\!0\.43, F1 = 0\.822; Figure[6\(b\)](https://arxiv.org/html/2607.29181#S4.F6.sf2)and[6\(c\)](https://arxiv.org/html/2607.29181#S4.F6.sf3)\) separates synonymous from distinct labels with high precision \(0\.768\) and recall \(0\.883\)\.
\(\(a\)\)Effect of label normalization on vocabulary and Markov model quality\.
\(\(b\)\)Calibration statistics
\(\(c\)\)Distribution for synonymous \(blue\) and distinct \(red\) pairs\. Dashed line markst∗=0\.43t^\{\*\}\\\!=\\\!0\.43
Figure 6:Label normalization summary \(left\) and semantic threshold calibration on 200 human\-annotated label pairs \(right\)\.Value of intent passes\.To isolate the contribution of intent gathering annotation passes, we compare the full pipeline against an activity\-only baseline using 12 activity passes with no intent inference\. The preference study below tests whether this predictability reflects genuine label quality\. Across 61 videos, 47% of frames received different activity labels between the two conditions\. We sampled 30 of these divergent frames \(10 per domain, stratified across 3 videos\) and presented each as a blinded A/B pair\.
By majority vote, annotators preferred labels from the full pipeline73%of the time \(α=0\.726\\alpha=0\.726; Table[4](https://arxiv.org/html/2607.29181#S4.T4)\)\. Without intent context, activity labels collapse to uninformative dominant states:typing\_on\_keyboardfor 92–96% of coding domain frames \(vs\. the intent\-informed pipeline’sediting\_css\_style,editing\_html\_code,debugging\_code\)\.
Table 4:Ablation preference study: full pipeline \(activity\+intent\) vs\. activity\-only labels\. Three annotators evaluated 30 blinded A/B pairs across three domains\.
Value of intent passes with frozen vocabularies\.To study the contribution of intent passes, notwithstanding differences in vocabulary size produced by the activity\-only and the full pipelines, we project the vocabulary produced by the full pipeline onto the activity\-only pipeline vocabulary, project the activity\-only pipeline onto the full pipeline vocabulary, and test the respective performances of both vocabularies\. Both normalizations support the conclusion that vocabulary size differences do not significantly impact results\.
Table 5:full vs\. activity\-only vocabulary before and after projection\. OOV measures percent of labels that have no match in target vocabulary\.ConditionVocabMarkovMajorityPPLOOVRaw \(own vocabulary\)Intent\-conditioned29\.9±27\.429\.9\{\\pm\}27\.436\.4±30\.836\.4\{\\pm\}30\.836\.8±32\.136\.8\{\\pm\}32\.122\.5±23\.122\.5\{\\pm\}23\.1—Activity\-only28\.0±26\.828\.0\{\\pm\}26\.842\.6±34\.042\.6\{\\pm\}34\.043\.6±34\.243\.6\{\\pm\}34\.220\.7±21\.720\.7\{\\pm\}21\.7—Normalized→\\tointent vocabIntent\-conditioned18\.4±14\.818\.4\{\\pm\}14\.845\.9±28\.945\.9\{\\pm\}28\.942\.6±30\.442\.6\{\\pm\}30\.411\.9±10\.911\.9\{\\pm\}10\.91\.2Activity\-only17\.2±14\.517\.2\{\\pm\}14\.550\.2±31\.050\.2\{\\pm\}31\.046\.5±33\.546\.5\{\\pm\}33\.511\.1±10\.811\.1\{\\pm\}10\.82\.0Normalized→\\toactivity\-only vocabIntent\-conditioned16\.9±13\.616\.9\{\\pm\}13\.648\.4±28\.048\.4\{\\pm\}28\.044\.6±29\.944\.6\{\\pm\}29\.910\.6±10\.010\.6\{\\pm\}10\.07\.2Activity\-only17\.8±15\.217\.8\{\\pm\}15\.250\.0±31\.250\.0\{\\pm\}31\.246\.8±33\.346\.8\{\\pm\}33\.311\.3±11\.511\.3\{\\pm\}11\.51\.2Effect of temporal window size\.To study the effects of various temporal window sizes, we evaluate over the same 12 randomly chosen videos at window sizes n = \{0,10,20\}\. Markov \- Majority gap increases at higher window size \(4\.4 vs 8\.0\), suggesting prediction structures become more prominent in produced Markov models at higher window sizes\.
Table 6:Temporal\-window sensitivity \(w∈\{0,10,20\}w\\in\\\{0,10,20\\\}\),n=12n=12videos\. Markov / Majority in %\. Normalized: all conditions projected onto thew=20w=20vocabulary\.Qualitative Study\.Figure[7](https://arxiv.org/html/2607.29181#S4.F7)shows multi\-pass refinement and its predictive consequence on a daily life video\.888Additional examples in Appendix, Figure[8](https://arxiv.org/html/2607.29181#A4.F8)\.
\(\(a\)\)Pass 1 \(Activity\)
\(\(b\)\)Pass 11 \(Activity\)
Activity:raising PR Next Act:raising PR
\(\(c\)\)att=7t\{=\}7:05
Figure 7:Activity refinement and next\-state prediction forbehindP12\. \(a\) Pass 1 produces generic labels\. \(b\) By pass 11, task\-specific states emerge\. \(c\) The Markov model correctly predicts state persistence by conditioning on the current state, while the majority baseline erroneously predicts the three globally most frequent states regardless of context\.
## 5Conclusion and Discussion
We presentedSerum, a multi\-pass VLM framework that extracts structured activity and intent models from egocentric video without a predefined ontology or manual annotation\. Alternating activity and intent passes converge to a stable vocabulary \(*schematic equilibrium*\) by pass 8; subsequent label normalization compresses it by 46%, yielding Markov user models that outperform frequency baselines on next\-state prediction\. Human annotators rate 88\.3% of final\-pass labels accurate and prefer them over first\-pass labels 82\.8% of the time, suggesting that iterative refinement produces meaningful, recognizable improvements\. This work has the following limitations and interesting directions for future work:
Evaluation protocol\.Split\-half evaluation penalizes Markov models on videos whose content progresses linearly without revisiting earlier states, since training and test vocabularies become largely disjoint\. This effect can be seen with several videos achieving near\-zero accuracy before normalization \(§LABEL:tab:markov\-final\)\. Future work could address this through cross\-video evaluation, where models trained on one user’s videos predict states in another’s\.
Downstream applications\.An important open question is whetherSerum’s user models can drive proactive agentic assistance — anticipating recurring errors or context switches before they occur\. AlthoughSerum’s computational complexity presents a challenge in latency to its feasibility in live settings, the majority of compute will be front loaded into a startup cost asSerumlearns a user’s workflow, with minor revisions after the incubation period\. This frees up compute for live suggestions\. Evaluation in live assistive settings and scaling to larger video corpora are the highest\-priority directions for future work\.
Counterfactual scenarios\.Future work could explore reversingSerumto allow video generation models to imagine counterfactual scenarios\. WhileSeruminfers actions and intentions from video, the reverse would useSerum’s action and intent labels to generate video\. This would enable generating counterfactual videos through perturbing inferred actions and intentions\.
Hallucinations\.OverSerum’s iterative annotation passes, we observe two main sources of hallucinations: 1\. The image is not clear \(e\.g\., due to motion blur, occlusion\)\. 2\. The VLM confuses whether an action is starting or ending due to limited temporal granularity\. For most hallucinationsSerumself\-corrects by re\-examining the original frame in each pass and by attaining neighbor consensus via the temporal context window to normalize inconsistent cases\.
## Acknowledgements\.
We thank the members of the Minnesota NLP group for giving feedback on initial drafts and, crucially, our colleague\-annotators \(Khanh Chi Le, Ruizi Wang, Jingcheng Liang\) who dedicated significant time annotatingSerum’s results over several trials\.
## Ethics Statement
This work analyzes publicly available YouTube videos and does not involve human subjects research\. We acknowledge that behavioral modeling from screen recordings could be misused for unauthorized surveillance; our work is intended for user\-initiated workflow analysis and support\. We release our code to promote reproducibility and encourage its responsible use\. Annotations are generated by a vision\-language model and may reflect biases present in its training data\.
Human annotation\.Two forms of human annotation supported this work: \(1\) a calibration set of 200 label pairs \(100 activities, 100 intents\) was hand\-rated by one of the authors to fit the SentenceBERT semantic\-merge thresholdt∗t^\{\*\}\(§[6\(a\)](https://arxiv.org/html/2607.29181#S4.F6.sf1)\); \(2\) five members of our research lab rated final\-pass labels for accuracy on a set of 26 videos pre\-vetted by the authors \(§[4\.4](https://arxiv.org/html/2607.29181#S4.SS4)\)\. Annotators were uncompensated lab volunteers, viewed only the pre\-vetted videos, and agreed to participate and to the use of their judgements in this research\. No personally identifying information was collected from annotators, and the videos contained no third\-party private data\. We did not seek formal Institutional Review Board approval, treating the rating task as internal validation by research collaborators; we acknowledge this is a limitation of the human evaluation and that a small, in\-lab annotator pool may bias results toward positive judgments\.
## LLM Disclosure
In accordance with the COLM 2026 policy on LLM usage, we disclose the following\. LLM\-assisted coding tools were used during software development and infrastructure management\. An LLM was also used to proofread drafts and assist with an initial literature survey; all references were verified by the authors\.
LLMs were not used to generate experimental results, figures, datasets, or quantitative analysis\. The research ideas, experimental design, implementation, analysis, and paper content are the work of the authors\.
## References
- Qwen\-VL: a versatile vision\-language model for understanding, localization, text reading, and beyond\.arXiv preprint arXiv:2308\.12966\.Cited by:[§2](https://arxiv.org/html/2607.29181#S2.p2.1)\.
- S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge, W\. Ge, Z\. Guo, Q\. Huang, J\. Huang, F\. Huang, B\. Hui, S\. Jiang, Z\. Li, M\. Li, M\. Li, K\. Li, Z\. Lin, J\. Lin, X\. Liu, J\. Liu, C\. Liu, Y\. Liu, D\. Liu, S\. Liu, D\. Lu, R\. Luo, C\. Lv, R\. Men, L\. Meng, X\. Ren, X\. Ren, S\. Song, Y\. Sun, J\. Tang, J\. Tu, J\. Wan, P\. Wang, P\. Wang, Q\. Wang, Y\. Wang, T\. Xie, Y\. Xu, H\. Xu, J\. Xu, Z\. Yang, M\. Yang, J\. Yang, A\. Yang, B\. Yu, F\. Zhang, H\. Zhang, X\. Zhang, B\. Zheng, H\. Zhong, J\. Zhou, F\. Zhou, J\. Zhou, Y\. Zhu, and K\. Zhu \(2025\)Qwen3\-vl technical report\.External Links:2511\.21631,[Link](https://arxiv.org/abs/2511.21631)Cited by:[§2](https://arxiv.org/html/2607.29181#S2.p2.1),[§4\.1](https://arxiv.org/html/2607.29181#S4.SS1.p2.3)\.
- J\. C\. A\. M\. Buijs, B\. F\. van Dongen, and W\. M\. P\. van der Aalst \(2012\)On the role of fitness, precision, generalization and simplicity in process discovery\.InOn the Move to Meaningful Internet Systems: OTM 2012 \(CoopIS\),Lecture Notes in Computer Science, Vol\.7565,pp\. 305–322\.Cited by:[§2](https://arxiv.org/html/2607.29181#S2.p3.1)\.
- D\. Damen, H\. Doughty, G\. M\. Farinella, S\. Fidler, A\. Furnari, E\. Kazakos, D\. Moltisanti, J\. Munro, T\. Perrett, W\. Price, and M\. Wray \(2018\)Scaling egocentric vision: the EPIC\-KITCHENS dataset\.InEuropean Conference on Computer Vision \(ECCV\),Cited by:[§2](https://arxiv.org/html/2607.29181#S2.p1.1)\.
- D\. Damen, H\. Doughty, G\. M\. Farinella, A\. Furnari, E\. Kazakos, J\. Ma, D\. Moltisanti, J\. Munro, T\. Perrett, W\. Price, and M\. Wray \(2022\)Rescaling egocentric vision: collection, pipeline and challenges for EPIC\-KITCHENS\-100\.International Journal of Computer Vision130,pp\. 33–55\.Cited by:[§1](https://arxiv.org/html/2607.29181#S1.p1.1),[§2](https://arxiv.org/html/2607.29181#S2.p1.1)\.
- A\. Furnari and G\. M\. Farinella \(2020\)Rolling\-unrolling LSTMs for action anticipation from first\-person video\.IEEE Transactions on Pattern Analysis and Machine Intelligence43\(11\),pp\. 4021–4036\.External Links:[Document](https://dx.doi.org/10.1109/TPAMI.2020.2992889)Cited by:[§2](https://arxiv.org/html/2607.29181#S2.p1.1)\.
- F\. Gilardi, M\. Alizadeh, and M\. Kubli \(2023\)ChatGPT outperforms crowd\-workers for text\-annotation tasks\.Proceedings of the National Academy of Sciences120\(30\),pp\. e2305016120\.Note:arXiv:2303\.15056External Links:[Document](https://dx.doi.org/10.1073/pnas.2305016120)Cited by:[§2](https://arxiv.org/html/2607.29181#S2.p2.1)\.
- K\. Grauman, A\. Westbury, E\. Byrne, Z\. Chavis, A\. Furnari, R\. Girdhar, J\. Hamburger, H\. Jiang, M\. Liu, X\. Liu,et al\.\(2022\)Ego4D: around the world in 3,000 hours of egocentric video\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 18995–19012\.Cited by:[§1](https://arxiv.org/html/2607.29181#S1.p1.1),[§2](https://arxiv.org/html/2607.29181#S2.p1.1)\.
- X\. He, Z\. Lin, Y\. Gong, A\. Jin, H\. Zhang, C\. Lin, J\. Jiao, S\. M\. Yiu, N\. Duan, and W\. Chen \(2024\)AnnoLLM: making large language models to be better crowdsourced annotators\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\),Note:arXiv:2303\.16854Cited by:[§2](https://arxiv.org/html/2607.29181#S2.p2.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with PagedAttention\.InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles \(SOSP\),Cited by:[§4\.1](https://arxiv.org/html/2607.29181#S4.SS1.p2.3)\.
- E\. V\. Mascaro, H\. Ahn, and D\. Lee \(2023\)Intention\-conditioned long\-term human egocentric action forecasting\.InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision \(WACV\),Note:arXiv:2207\.12080Cited by:[§2](https://arxiv.org/html/2607.29181#S2.p1.1)\.
- N\. Reimers and I\. Gurevych \(2019a\)Sentence\-BERT: sentence embeddings using siamese BERT\-networks\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing \(EMNLP\-IJCNLP\),pp\. 3982–3992\.Cited by:[§2](https://arxiv.org/html/2607.29181#S2.p4.1),[§3\.3](https://arxiv.org/html/2607.29181#S3.SS3.p1.1)\.
- N\. Reimers and I\. Gurevych \(2019b\)Sentence\-bert: sentence embeddings using siamese bert\-networks\.External Links:1908\.10084,[Link](https://arxiv.org/abs/1908.10084)Cited by:[§4\.1](https://arxiv.org/html/2607.29181#S4.SS1.p3.1)\.
- P\. Schroeder, O\. Biza, T\. Weng, H\. Luo, and J\. Glass \(2025\)ROVER: recursive reasoning over videos with vision\-language models for embodied tasks\.InarXiv preprint,Note:arXiv:2508\.01943Cited by:[§2](https://arxiv.org/html/2607.29181#S2.p2.1)\.
- O\. Shaikh, S\. Sapkota, S\. Rizvi, E\. Horvitz, J\. S\. Park, D\. Yang, and M\. S\. Bernstein \(2025\)Creating general user models from computer use\.External Links:2505\.10831,[Link](https://arxiv.org/abs/2505.10831)Cited by:[§1](https://arxiv.org/html/2607.29181#S1.p3.1),[§2](https://arxiv.org/html/2607.29181#S2.p2.1)\.
- W\. van der Aalst, A\. Adriansyah, A\. K\. A\. de Medeiros, F\. Arcieri, T\. Baier, T\. Blickle, J\. C\. Bose, P\. van den Brand, R\. Brandtjen, J\. Buijs,et al\.\(2012\)Process mining manifesto\.InBusiness Process Management Workshops \(BPM 2011\),Lecture Notes in Business Information Processing, Vol\.99,pp\. 169–194\.Cited by:[§1](https://arxiv.org/html/2607.29181#S1.p1.1),[§2](https://arxiv.org/html/2607.29181#S2.p3.1)\.
- W\. M\. P\. van der Aalst \(2016\)Process mining: data science in action\.2nd edition,Springer\-Verlag,Berlin\.Cited by:[§1](https://arxiv.org/html/2607.29181#S1.p1.1),[§2](https://arxiv.org/html/2607.29181#S2.p3.1)\.
- P\. Wang, S\. Bai, S\. Tan, S\. Wang, Z\. Fan, J\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge,et al\.\(2024\)Qwen2\-VL: enhancing vision\-language model’s perception of the world at any resolution\.arXiv preprint arXiv:2409\.12191\.Cited by:[§2](https://arxiv.org/html/2607.29181#S2.p2.1)\.
- T\. Wu, T\. Trigui, S\. N\. Sridhar, A\. Bodas, and S\. Tripathi \(2025\)Toward scalable video narration: a training\-free approach using multimodal large language models\.InProceedings of the International Conference on Computer Vision \(ICCV\) Workshop on CVAM,Note:arXiv:2507\.17050Cited by:[§2](https://arxiv.org/html/2607.29181#S2.p2.1)\.
## Appendix AFull Data Collection
Table 7:Dataset OverviewVideoCategoryFramesPassesIntervalSource URLACS\_salestrainingP12Daily Life55125s[youtu\.be/ZG4ExqMVA7w](https://arxiv.org/html/2607.29181v1/youtu.be/ZG4ExqMVA7w)AC\_leetcode2P12Coding80125s[youtu\.be/vRAK2YnFr1o](https://arxiv.org/html/2607.29181v1/youtu.be/vRAK2YnFr1o)AC\_leetcodeP12Coding352125s[youtu\.be/zeLZuhi6eYU](https://arxiv.org/html/2607.29181v1/youtu.be/zeLZuhi6eYU)AC\_pizzaP12Cooking86125s[youtu\.be/Q9j6HhF0tGE](https://arxiv.org/html/2607.29181v1/youtu.be/Q9j6HhF0tGE)AC\_profreactsP12Daily Life108125s[youtu\.be/3mRvCF4qyTA](https://arxiv.org/html/2607.29181v1/youtu.be/3mRvCF4qyTA)AC\_sandwichP12Cooking120125s[youtu\.be/ad8TWumCSnY](https://arxiv.org/html/2607.29181v1/youtu.be/ad8TWumCSnY)AC\_studrecordingP12Daily Life100125s[youtu\.be/eB54LIupAhU](https://arxiv.org/html/2607.29181v1/youtu.be/eB54LIupAhU)AC\_ukdayinlifeP12Daily Life187125s[youtu\.be/BvWnEiOoAEk](https://arxiv.org/html/2607.29181v1/youtu.be/BvWnEiOoAEk)AC\_waiterP12Cooking133125s[youtu\.be/w4pGt\-iGpBI](https://arxiv.org/html/2607.29181v1/youtu.be/w4pGt-iGpBI)BC\_dunkinhelpP12Cooking213125s[youtu\.be/j\_gUBLwxG1U](https://arxiv.org/html/2607.29181v1/youtu.be/j_gUBLwxG1U)BC\_nycswevlogP12Daily Life114125s[youtu\.be/4lo81zt7HK8](https://arxiv.org/html/2607.29181v1/youtu.be/4lo81zt7HK8)BC\_pizzarushP12Cooking252125s[youtu\.be/S5ltPbUur38](https://arxiv.org/html/2607.29181v1/youtu.be/S5ltPbUur38)BC\_swevlogP12Daily Life121125s[youtu\.be/b\_eeMSNO97U](https://arxiv.org/html/2607.29181v1/youtu.be/b_eeMSNO97U)BC\_vibecodingP12Coding157125s[youtu\.be/P3JA7MTiGg8](https://arxiv.org/html/2607.29181v1/youtu.be/P3JA7MTiGg8)CC\_baristaP12Cooking429125s[youtu\.be/jdguVU0F7fs](https://arxiv.org/html/2607.29181v1/youtu.be/jdguVU0F7fs)CC\_swisssweP12Daily Life327125s[youtu\.be/\_GSI2RaiV0s](https://arxiv.org/html/2607.29181v1/youtu.be/_GSI2RaiV0s)DC\_calcappcodingP12Coding409125s[youtu\.be/sBJmRD7kNTk](https://arxiv.org/html/2607.29181v1/youtu.be/sBJmRD7kNTk)DC\_snakecodingP12Coding289125s[youtu\.be/Wlu4MsBnjuk](https://arxiv.org/html/2607.29181v1/youtu.be/Wlu4MsBnjuk)PERS\_coinflipP12Coding137125s[youtu\.be/\-o\-H1Ecqo\_M](https://arxiv.org/html/2607.29181v1/youtu.be/-o-H1Ecqo_M)PERS\_movieP12Coding200125s[youtu\.be/J6uam9jEmDU](https://arxiv.org/html/2607.29181v1/youtu.be/J6uam9jEmDU)PERS\_weatherP12Coding246125s[youtu\.be/iILFBGm\_I9M](https://arxiv.org/html/2607.29181v1/youtu.be/iILFBGm_I9M)bartenderP12Cooking212125s[youtu\.be/1G\-9Pibx5JI](https://arxiv.org/html/2607.29181v1/youtu.be/1G-9Pibx5JI)basketballP12Physical50125s[youtu\.be/N7RNoleA7Sk](https://arxiv.org/html/2607.29181v1/youtu.be/N7RNoleA7Sk)behindP12Daily Life124125s[youtu\.be/h4exLX8Wz4E](https://arxiv.org/html/2607.29181v1/youtu.be/h4exLX8Wz4E)carrepair2P12Physical113125s[youtu\.be/o0OBJCfAfOY](https://arxiv.org/html/2607.29181v1/youtu.be/o0OBJCfAfOY)carrepair3P12Physical171125s[youtu\.be/VdR5zPyqp\_4](https://arxiv.org/html/2607.29181v1/youtu.be/VdR5zPyqp_4)carrepairP12Physical223125s[youtu\.be/vHdz74orr1Q](https://arxiv.org/html/2607.29181v1/youtu.be/vHdz74orr1Q)cashboothP12Cooking50125s[youtu\.be/9lNBUsF4WRU](https://arxiv.org/html/2607.29181v1/youtu.be/9lNBUsF4WRU)coding2P12Coding185125s[youtu\.be/gRyvG7PZ4m0](https://arxiv.org/html/2607.29181v1/youtu.be/gRyvG7PZ4m0)coding3P12Coding226125s[youtu\.be/825u2Puaej0](https://arxiv.org/html/2607.29181v1/youtu.be/825u2Puaej0)codingP12Coding179125s[youtu\.be/DfDPJqD3FjI](https://arxiv.org/html/2607.29181v1/youtu.be/DfDPJqD3FjI)codinglogoP12Coding234125s[youtu\.be/B\_puD1rTsOQ](https://arxiv.org/html/2607.29181v1/youtu.be/B_puD1rTsOQ)codingqrcodeP12Coding198125s[youtu\.be/I50Xwve6QW4](https://arxiv.org/html/2607.29181v1/youtu.be/I50Xwve6QW4)competitiveP12Coding206125s[youtu\.be/uGrBHohIgQY](https://arxiv.org/html/2607.29181v1/youtu.be/uGrBHohIgQY)compgamingP12Coding127125s[youtu\.be/yCezqhatLV8](https://arxiv.org/html/2607.29181v1/youtu.be/yCezqhatLV8)construction2P12Physical81125s[youtu\.be/GlsCRChrdfU](https://arxiv.org/html/2607.29181v1/youtu.be/GlsCRChrdfU)constructionP12Physical260125s[youtu\.be/2avgoVsQ\_og](https://arxiv.org/html/2607.29181v1/youtu.be/2avgoVsQ_og)csscodingP12Coding42125s[youtu\.be/EZhPsuIXawk](https://arxiv.org/html/2607.29181v1/youtu.be/EZhPsuIXawk)dayinthelifesweP12Daily Life102125s[youtu\.be/aTHBJwVgu3I](https://arxiv.org/html/2607.29181v1/youtu.be/aTHBJwVgu3I)drivingP12Physical158125s[youtu\.be/iSnP5c997Uk](https://arxiv.org/html/2607.29181v1/youtu.be/iSnP5c997Uk)dunkinP12Cooking210125s[youtu\.be/hEJaSuDiQU8](https://arxiv.org/html/2607.29181v1/youtu.be/hEJaSuDiQU8)fluttercodingP12Coding216125s[youtu\.be/C7Kafde7gZ4](https://arxiv.org/html/2607.29181v1/youtu.be/C7Kafde7gZ4)goprochefP12Cooking299125s[youtu\.be/CBSsL4u\_nng](https://arxiv.org/html/2607.29181v1/youtu.be/CBSsL4u_nng)headchefP12Cooking350125s[youtu\.be/Ipe9xJCfuTM](https://arxiv.org/html/2607.29181v1/youtu.be/Ipe9xJCfuTM)hotdogP12Cooking265125s[youtu\.be/YMpGWAB41lI](https://arxiv.org/html/2607.29181v1/youtu.be/YMpGWAB41lI)labworkP12Daily Life164125s[youtu\.be/C3aKnhXn20U](https://arxiv.org/html/2607.29181v1/youtu.be/C3aKnhXn20U)markiplierP12Daily Life317125s[youtu\.be/Yk\-I7IVLAGo](https://arxiv.org/html/2607.29181v1/youtu.be/Yk-I7IVLAGo)mcdcookP12Cooking151125s[youtu\.be/8kcUsQdxtSs](https://arxiv.org/html/2607.29181v1/youtu.be/8kcUsQdxtSs)mcdtakingordersP12Cooking337125s[youtu\.be/\_c8PppBiMqE](https://arxiv.org/html/2607.29181v1/youtu.be/_c8PppBiMqE)microbialP12Daily Life54125s[youtu\.be/NUkrCXMdl3o](https://arxiv.org/html/2607.29181v1/youtu.be/NUkrCXMdl3o)musicplayercodingP12Coding283125s[youtu\.be/KndQpfPkOOY](https://arxiv.org/html/2607.29181v1/youtu.be/KndQpfPkOOY)paperworkP12Daily Life69125s[youtu\.be/JdkMmLhPw\_E](https://arxiv.org/html/2607.29181v1/youtu.be/JdkMmLhPw_E)phonerepairP12Physical343125s[youtu\.be/p9hA59nn7uQ](https://arxiv.org/html/2607.29181v1/youtu.be/p9hA59nn7uQ)radiatorrepairP12Physical98125s[youtu\.be/ldIo1L6S\_Sw](https://arxiv.org/html/2607.29181v1/youtu.be/ldIo1L6S_Sw)rmlineP12Physical105125s[youtu\.be/jIJTEm0qNuo](https://arxiv.org/html/2607.29181v1/youtu.be/jIJTEm0qNuo)sushiP12Cooking183125s[youtu\.be/KUzYFMgWs4w](https://arxiv.org/html/2607.29181v1/youtu.be/KUzYFMgWs4w)tractorfarmingP12Physical96125s[youtu\.be/rqA\-iT2DKO4](https://arxiv.org/html/2607.29181v1/youtu.be/rqA-iT2DKO4)tttcodingP12Coding175125s[youtu\.be/MgtGHfdpigU](https://arxiv.org/html/2607.29181v1/youtu.be/MgtGHfdpigU)tutorialP12Daily Life92125s[youtu\.be/a32fbqPNir4](https://arxiv.org/html/2607.29181v1/youtu.be/a32fbqPNir4)welshgardeningP12Physical160125s[youtu\.be/T3fgL091hXs](https://arxiv.org/html/2607.29181v1/youtu.be/T3fgL091hXs)wslinstallP12Daily Life102125s[youtu\.be/QadguqFAt\_8](https://arxiv.org/html/2607.29181v1/youtu.be/QadguqFAt_8)Table 7:Dataset Overview \(continued\)
## Appendix BVerbose Next\-Action Prediction Task Results
Table 8:Markov Prediction Accuracy \(Final Pass\)VideoTypeVocabMarkovMajorityWt\. RandUniformVocabnMarkovnMajnACS\_salestrainingP12Act761\.9%19\.0%28\.6%14\.3%661\.9%19\.0%ACS\_salestrainingP12Int244\.8%0\.0%4\.3%4\.2%79\.5%38\.1%AC\_leetcode2P12Act2100\.0%100\.0%96\.0%50\.0%2100\.0%100\.0%AC\_leetcode2P12Int883\.9%83\.9%40\.0%12\.5%5100\.0%100\.0%AC\_leetcodeP12Act692\.6%94\.3%84\.5%16\.7%593\.4%94\.3%AC\_leetcodeP12Int1163\.1%69\.7%41\.0%9\.1%665\.6%69\.7%AC\_pizzaP12Act455\.9%0\.0%2\.1%2\.2%258\.8%0\.0%AC\_pizzaP12Int430\.0%5\.9%2\.8%2\.3%1352\.9%32\.4%AC\_profreactsP12Act2211\.6%20\.9%10\.5%4\.5%1916\.3%20\.9%AC\_profreactsP12Int407\.0%23\.3%4\.4%2\.5%2614\.0%25\.6%AC\_sandwichP12Act464\.3%0\.0%1\.4%2\.2%2312\.8%12\.8%AC\_sandwichP12Int1951\.1%48\.9%30\.3%5\.3%774\.5%74\.5%AC\_studrecordingP12Act2048\.7%25\.6%8\.6%5\.0%1451\.3%7\.7%AC\_studrecordingP12Int1959\.0%23\.1%10\.4%5\.3%864\.1%28\.2%AC\_ukdayinlifeP12Act4412\.2%5\.4%3\.3%2\.3%3124\.3%5\.4%AC\_ukdayinlifeP12Int665\.4%1\.4%1\.9%1\.5%3717\.6%9\.5%AC\_waiterP12Act2420\.8%24\.5%10\.2%4\.2%1532\.1%39\.6%AC\_waiterP12Int1828\.3%39\.6%20\.4%5\.6%756\.6%47\.2%BC\_dunkinhelpP12Act6617\.6%25\.9%6\.4%1\.5%2950\.6%52\.9%BC\_dunkinhelpP12Int4215\.3%22\.4%9\.5%2\.4%1365\.9%71\.8%BC\_nycswevlogP12Act506\.7%6\.7%2\.8%2\.0%456\.7%6\.7%BC\_nycswevlogP12Int596\.7%4\.4%1\.6%1\.7%476\.7%4\.4%BC\_pizzarushP12Act6731\.0%5\.0%5\.0%1\.5%2852\.0%13\.0%BC\_pizzarushP12Int3853\.0%26\.0%15\.1%2\.6%1586\.0%87\.0%BC\_swevlogP12Act3112\.5%20\.8%7\.3%3\.2%2429\.2%22\.9%BC\_swevlogP12Int436\.2%8\.3%4\.1%2\.3%218\.3%8\.3%BC\_vibecodingP12Act943\.5%50\.0%31\.5%11\.1%862\.9%50\.0%BC\_vibecodingP12Int1840\.3%43\.5%31\.2%5\.6%848\.4%43\.5%CC\_baristaP12Act9418\.1%21\.1%6\.2%1\.1%3732\.7%28\.7%CC\_baristaP12Int837\.6%0\.0%2\.5%1\.2%2048\.5%33\.3%CC\_swisssweP12Act895\.4%7\.7%2\.2%1\.1%538\.5%8\.5%CC\_swisssweP12Int10520\.0%21\.5%2\.6%1\.0%6424\.6%21\.5%DC\_calcappcodingP12Act682\.2%82\.8%55\.2%16\.7%682\.2%82\.8%DC\_calcappcodingP12Int861\.8%0\.0%0\.5%1\.2%1990\.8%91\.4%DC\_snakecodingP12Act510\.4%11\.3%11\.5%20\.0%311\.3%11\.3%DC\_snakecodingP12Int911\.3%14\.8%13\.0%11\.1%299\.1%99\.1%PERS\_coinflipP12Act1063\.0%59\.3%45\.0%10\.0%672\.2%59\.3%PERS\_coinflipP12Int3357\.4%0\.0%4\.6%3\.0%1092\.6%94\.4%PERS\_movieP12Act6100\.0%100\.0%88\.9%16\.7%5100\.0%100\.0%PERS\_movieP12Int3643\.0%8\.9%8\.5%2\.8%972\.2%13\.9%PERS\_weatherP12Act863\.3%62\.2%33\.1%12\.5%563\.3%62\.2%PERS\_weatherP12Int2834\.7%0\.0%8\.4%3\.6%495\.9%95\.9%bartenderP12Act8510\.7%2\.4%1\.9%1\.2%4511\.9%3\.6%bartenderP12Int9610\.7%11\.9%1\.8%1\.0%4028\.6%25\.0%basketballP12Act1021\.1%21\.1%13\.8%10\.0%742\.1%52\.6%basketballP12Int989\.5%89\.5%41\.8%11\.1%5100\.0%100\.0%behindP12Act1512\.2%0\.0%3\.9%6\.7%1412\.2%0\.0%behindP12Int198\.2%0\.0%2\.2%5\.3%138\.2%0\.0%carrepair2P12Act328\.9%13\.3%4\.6%3\.1%198\.9%13\.3%carrepair2P12Int3131\.1%44\.4%12\.8%3\.2%1453\.3%57\.8%carrepair3P12Act442\.9%5\.9%2\.2%2\.3%2020\.6%22\.1%carrepair3P12Int3813\.2%11\.8%8\.5%2\.6%1629\.4%38\.2%carrepairP12Act521\.1%0\.0%1\.4%1\.9%342\.2%0\.0%carrepairP12Int3332\.6%30\.3%12\.7%3\.0%1546\.1%47\.2%cashboothP12Act1010\.5%21\.1%10\.5%10\.0%815\.8%26\.3%cashboothP12Int1431\.6%10\.5%12\.0%7\.1%642\.1%47\.4%coding2P12Act148\.2%0\.0%6\.2%7\.1%916\.4%0\.0%coding2P12Int628\.2%0\.0%2\.7%1\.6%1616\.4%0\.0%coding3P12Act1068\.9%66\.7%54\.3%10\.0%766\.7%66\.7%coding3P12Int890\.0%90\.0%77\.1%12\.5%5100\.0%100\.0%codingP12Act4100\.0%100\.0%91\.0%25\.0%3100\.0%100\.0%codingP12Int1742\.3%16\.9%16\.5%5\.9%490\.1%90\.1%codinglogoP12Act298\.9%98\.9%98\.2%50\.0%298\.9%98\.9%codinglogoP12Int2100\.0%100\.0%98\.6%50\.0%2100\.0%100\.0%codingqrcodeP12Act889\.9%93\.7%76\.6%12\.5%689\.9%93\.7%codingqrcodeP12Int4232\.9%48\.1%10\.5%2\.4%960\.8%75\.9%competitiveP12Act572\.0%76\.8%72\.8%20\.0%473\.2%76\.8%competitiveP12Int1237\.8%26\.8%21\.3%8\.3%368\.3%74\.4%compgamingP12Act396\.0%96\.0%84\.1%33\.3%396\.0%96\.0%compgamingP12Int276\.0%76\.0%62\.0%50\.0%1––construction2P12Act2421\.9%28\.1%10\.5%4\.2%1221\.9%31\.2%construction2P12Int2612\.5%34\.4%9\.2%3\.8%571\.9%71\.9%constructionP12Act1371\.8%75\.7%49\.9%7\.7%972\.8%75\.7%constructionP12Int1822\.0%26\.8%21\.9%5\.6%781\.7%81\.7%csscodingP12Act387\.5%87\.5%50\.4%33\.3%293\.8%87\.5%csscodingP12Int837\.5%0\.0%19\.1%12\.5%462\.5%62\.5%dayinthelifesweP12Act420\.0%7\.5%2\.2%2\.4%315\.0%12\.5%dayinthelifesweP12Int4812\.5%15\.0%2\.3%2\.1%3112\.5%0\.0%drivingP12Act2181\.0%84\.1%25\.5%4\.8%1485\.7%92\.1%drivingP12Int1793\.7%95\.2%46\.7%5\.9%1195\.2%95\.2%dunkinP12Act981\.2%2\.4%1\.3%1\.0%3910\.8%13\.3%dunkinP12Int746\.0%14\.5%2\.9%1\.4%2510\.8%22\.9%fluttercodingP12Act1462\.8%68\.6%43\.8%7\.1%965\.1%68\.6%fluttercodingP12Int5530\.2%37\.2%5\.9%1\.8%2245\.3%54\.7%goprochefP12Act1123\.4%10\.1%2\.3%0\.9%5321\.8%31\.9%goprochefP12Int6814\.3%21\.8%6\.5%1\.5%2523\.5%27\.7%headchefP12Act8115\.1%18\.0%4\.4%1\.2%3523\.0%19\.4%headchefP12Int5934\.5%42\.4%15\.1%1\.7%2239\.6%42\.4%hotdogP12Act7018\.1%24\.8%5\.4%1\.4%3054\.3%48\.6%hotdogP12Int4812\.4%19\.0%6\.9%2\.1%1582\.9%83\.8%labworkP12Act867\.7%67\.7%44\.8%12\.5%670\.8%70\.8%labworkP12Int946\.2%38\.5%27\.1%11\.1%247\.7%47\.7%markiplierP12Act2246\.0%49\.2%30\.1%4\.5%1566\.7%67\.5%markiplierP12Int5319\.0%23\.8%9\.2%1\.9%2639\.7%46\.0%mcdcookP12Act3025\.0%26\.7%8\.9%3\.3%980\.0%80\.0%mcdcookP12Int2128\.3%38\.3%14\.1%4\.8%495\.0%95\.0%mcdtakingordersP12Act6320\.9%16\.4%5\.1%1\.6%4229\.9%24\.6%mcdtakingordersP12Int6611\.9%1\.5%3\.5%1\.5%3521\.6%4\.5%microbialP12Act174\.8%0\.0%8\.1%5\.9%919\.0%19\.0%microbialP12Int652\.4%61\.9%45\.6%16\.7%290\.5%95\.2%musicplayercodingP12Act892\.9%94\.7%83\.5%12\.5%592\.9%94\.7%musicplayercodingP12Int4311\.5%10\.6%4\.3%2\.3%961\.9%12\.4%paperworkP12Act1511\.1%11\.1%8\.8%6\.7%833\.3%29\.6%paperworkP12Int722\.2%33\.3%27\.8%14\.3%344\.4%44\.4%phonerepairP12Act6115\.3%17\.5%5\.8%1\.6%3020\.4%19\.0%phonerepairP12Int3134\.3%10\.9%9\.6%3\.2%1556\.9%64\.2%radiatorrepairP12Act953\.8%64\.1%30\.7%11\.1%576\.9%84\.6%radiatorrepairP12Int541\.0%46\.2%36\.8%20\.0%3100\.0%100\.0%rmlineP12Act107\.3%2\.4%11\.6%10\.0%548\.8%51\.2%rmlineP12Int1065\.9%68\.3%34\.3%10\.0%382\.9%82\.9%sushiP12Act921\.4%6\.8%1\.2%1\.1%2639\.7%50\.7%sushiP12Int3821\.9%20\.5%8\.3%2\.6%1178\.1%79\.5%tractorfarmingP12Act257\.9%0\.0%4\.0%4\.0%1878\.9%84\.2%tractorfarmingP12Int342\.6%0\.0%2\.9%2\.9%1981\.6%0\.0%tttcodingP12Act672\.5%72\.5%54\.9%16\.7%672\.5%72\.5%tttcodingP12Int13100\.0%100\.0%75\.4%7\.7%4100\.0%100\.0%tutorialP12Act1347\.2%52\.8%15\.8%7\.7%1072\.2%58\.3%tutorialP12Int3150\.0%41\.7%8\.6%3\.2%1663\.9%16\.7%welshgardeningP12Act4225\.4%38\.1%7\.7%2\.4%2060\.3%68\.3%welshgardeningP12Int3944\.4%19\.0%8\.7%2\.6%1458\.7%28\.6%wslinstallP12Act2722\.5%42\.5%10\.3%3\.7%2125\.0%47\.5%wslinstallP12Int492\.5%5\.0%1\.5%2\.0%242\.5%0\.0%n= normalized labels
## Appendix CGeneralizing Procedure on EPIC\-KITCHENS\-100
Table 9:EPIC\-KITCHENS\-100 generalization \(366 videos from 37 participants\)\. Same Markov harness as Table[2](https://arxiv.org/html/2607.29181#S4.T2); final\-pass P11 \(activity\) / P12 \(intent\)\.ndenotes models built on normalized labels\.366 videos, 33,788 frames at the final activity pass\.
Table 10:Curated vs\. EPIC\-KITCHENS\-100 generalization \(61 curated videos vs\. 366 EK videos; same Markov harness\)\. Absolute Markovnaccuracy is lower on EK; the Markovn−\-Majoritynmethod gap is wider on EK\.Δ=EK−Curated\\Delta=\\text\{EK\}\-\\text\{Curated\}\.ndenotes models built on normalized labels\.Table 11:Per\-participant Markovntop\-1 accuracy and perplexity on EPIC\-KITCHENS\-100 \(final\-pass P11/P12, normalized labels\)\.ParticipantVideosAct Top\-1Act PPLInt Top\-1Int PPLP042820\.9±\\pm23\.515\.4±\\pm8\.142\.3±\\pm26\.97\.2±\\pm3\.7P222723\.3±\\pm14\.816\.4±\\pm7\.145\.3±\\pm23\.96\.6±\\pm4\.0P022328\.6±\\pm30\.815\.7±\\pm7\.859\.3±\\pm30\.25\.3±\\pm3\.5P032327\.6±\\pm22\.211\.2±\\pm5\.258\.2±\\pm26\.84\.7±\\pm3\.0P081714\.8±\\pm11\.618\.6±\\pm8\.751\.0±\\pm28\.26\.5±\\pm3\.5P281732\.8±\\pm18\.311\.9±\\pm6\.653\.1±\\pm34\.85\.1±\\pm3\.6P301715\.8±\\pm16\.518\.2±\\pm8\.036\.5±\\pm29\.67\.6±\\pm3\.9P011629\.9±\\pm26\.714\.5±\\pm7\.358\.9±\\pm27\.35\.0±\\pm3\.0P071622\.3±\\pm15\.912\.0±\\pm7\.056\.6±\\pm30\.84\.8±\\pm3\.6P261642\.2±\\pm27\.37\.0±\\pm3\.870\.9±\\pm28\.62\.7±\\pm1\.7P061334\.0±\\pm29\.713\.2±\\pm6\.548\.5±\\pm30\.95\.7±\\pm3\.2P251228\.2±\\pm22\.712\.0±\\pm5\.651\.2±\\pm22\.25\.4±\\pm2\.8P111019\.8±\\pm15\.815\.8±\\pm4\.147\.0±\\pm24\.86\.2±\\pm3\.1P121029\.7±\\pm12\.813\.3±\\pm4\.738\.7±\\pm24\.96\.8±\\pm3\.5P271032\.0±\\pm27\.613\.5±\\pm9\.551\.1±\\pm29\.16\.2±\\pm4\.4P31923\.4±\\pm29\.717\.2±\\pm9\.464\.0±\\pm24\.44\.6±\\pm2\.2P33925\.4±\\pm22\.814\.9±\\pm5\.441\.3±\\pm26\.76\.4±\\pm2\.3P35933\.6±\\pm23\.614\.0±\\pm8\.456\.3±\\pm21\.54\.7±\\pm2\.2P15824\.1±\\pm14\.515\.1±\\pm6\.956\.8±\\pm25\.74\.9±\\pm2\.9P09728\.3±\\pm26\.112\.0±\\pm6\.746\.2±\\pm29\.55\.7±\\pm3\.4P23743\.7±\\pm37\.810\.7±\\pm7\.356\.2±\\pm30\.75\.4±\\pm3\.6P24736\.4±\\pm17\.112\.0±\\pm6\.262\.3±\\pm26\.93\.9±\\pm2\.1P34742\.8±\\pm26\.28\.8±\\pm4\.461\.3±\\pm26\.74\.2±\\pm2\.4P05647\.4±\\pm29\.09\.4±\\pm5\.579\.0±\\pm16\.02\.8±\\pm1\.3P18626\.1±\\pm26\.814\.7±\\pm7\.747\.0±\\pm15\.76\.5±\\pm3\.7P20518\.2±\\pm17\.417\.8±\\pm6\.465\.4±\\pm23\.54\.1±\\pm1\.8P13464\.7±\\pm29\.07\.8±\\pm4\.766\.9±\\pm32\.84\.8±\\pm4\.4P29426\.1±\\pm34\.113\.2±\\pm8\.871\.8±\\pm26\.53\.6±\\pm3\.0P32431\.1±\\pm32\.49\.4±\\pm5\.070\.1±\\pm18\.12\.8±\\pm1\.5P10323\.4±\\pm3\.517\.2±\\pm1\.438\.3±\\pm13\.68\.6±\\pm3\.3P17316\.1±\\pm8\.718\.4±\\pm3\.246\.6±\\pm9\.46\.7±\\pm0\.6P19322\.8±\\pm23\.217\.9±\\pm8\.741\.0±\\pm35\.05\.5±\\pm3\.1P14222\.6±\\pm22\.69\.2±\\pm2\.171\.4±\\pm3\.63\.1±\\pm0\.3P16234\.0±\\pm6\.411\.4±\\pm2\.571\.3±\\pm20\.23\.1±\\pm1\.1P21223\.5±\\pm15\.021\.1±\\pm9\.057\.8±\\pm8\.96\.9±\\pm2\.9P36287\.2±\\pm2\.12\.1±\\pm0\.189\.4±\\pm0\.01\.6±\\pm0\.2P37272\.3±\\pm14\.910\.3±\\pm8\.650\.0±\\pm13\.85\.4±\\pm2\.6Overall36628\.3±\\pm25\.013\.8±\\pm7\.653\.6±\\pm28\.75\.5±\\pm3\.5
## Appendix DAdditional Qualitative Example
#### Content creation \(tutorialP12\)\.
Figure[8](https://arxiv.org/html/2607.29181#A4.F8)shows refinement on a content\-creation video\. Activity labels evolve from perceptual \(*sitting*,*browsing web*\) to task\-specific \(*preparing tutorial video on intersection observer API*,*responding to viewer comment*\) by pass 11\.
The intent graphs \(d–f\) reveal structure invisible in the activity graph\. By pass 12, two workflow clusters emerge: an audience\-facing loop \(*responding to viewer comment*→\\to*speaking into microphone*→\\to*creating digital content*→\\to*preparing tutorial video on intersection observer API*\), and a production pipeline \(*managing content schedule*→\\to*reviewing and refining video content*→\\to*managing video content pipeline*\)\. These clusters connect through*managing content creation workflow*\. An agent consuming this model could distinguish recording from planning phases — a distinction the activity graph cannot surface\.
\(\(a\)\)Pass 1 \(Activity\)
\(\(b\)\)Pass 5 \(Activity\)
\(\(c\)\)Pass 11 \(Activity\)
\(\(d\)\)Pass 2 \(Intent\)
\(\(e\)\)Pass 6 \(Intent\)
\(\(f\)\)Pass 12 \(Intent\)
Figure 8:Content creation video \(tutorialP12\)\.*Top*: activity labels refine from perceptual to task\-specific\.*Bottom*: intent graphs reveal two workflow clusters \(audience\-facing vs\. production\) connected through a management hub\.
## Appendix EPass\-by\-Pass Model Performance
Figure[9](https://arxiv.org/html/2607.29181#A5.F9)shows accuracy and perplexity across all 12 passes \(61 videos\)\. Activity accuracy \(a\): raw Markov and Majority are closely matched; normalized Markov consistently leads\. Intent accuracy \(b\): normalized models show clearer separation, reaching∼\{\\sim\}60% vs\.∼\{\\sim\}30% raw accuracy\. Perplexity \(c, d\): Markov achieves the lowest at every pass; normalization roughly halves it\. Wide standard deviation bands reflect high per\-video variance from vocabulary size and domain differences\.
\(\(a\)\)Activity accuracy vs baselines
\(\(b\)\)Intent accuracy vs baselines
\(\(c\)\)Activity perplexity vs baselines
\(\(d\)\)Intent perplexity vs baselines
Figure 9:Pass\-by\-pass top\-1 accuracy and perplexity for normalized Markov vs\. baselines across all 12 annotation passes, averaged over 61 videos with standard deviation bands\.
## Appendix FThreshold Calibration Procedure
To calibrate the semantic merging threshold, we assemble all unique activity and intent labels across every video and pass, compute pairwise SentenceBERT cosine distances, and partition the distance range into 10 equal\-width bins\. We randomly sample 10 pairs per bin per type \(activity, intent\), yielding 100 pairs per type \(200 total\) stratified across the full similarity spectrum\. A single annotator labels each pair as*same*\(semantically equivalent\),*different*\(distinct states\), or*skip*\(ambiguous\)\. The optimal thresholdt∗t^\{\*\}is selected as the cosine distance maximizing F1 score on non\-skipped pairs, treating*same*as the positive class\.
## Appendix GPrompt Templates
Table[12](https://arxiv.org/html/2607.29181#A7.T12)summarizes the four prompt templates used across annotation passes\. Pass 1 receives only the frame image; all subsequent passes additionally receive the temporal context window \(RLE of neighboring frames’ labels\) and the inter\-pass summary\.
Table 12:Prompt templates by pass type\. Each prompt instructs the VLM to output structured JSON with a state label, confidence score \(1–10\), and supporting evidence\.Full prompt text is available in the released codebase999https://github\.com/minnesotanlp/SERUM/
## Appendix HModel choice preliminary study
We investigated 3 new models over 4 videos chosen randomly from the expanded annotation round\. We find each model still reaches schematic equilibrium at every scale tested, but generally larger models took longer to reach schematic equilibrium \(Table[13](https://arxiv.org/html/2607.29181#A8.T13)\)\. There is no obvious pattern to the effectiveness of larger models in next\-state prediction\. Larger models benefit more from normalization \(32B’s Markov accuracy saw a 122% increase going from 2\.7 to 6\.0\) \(Table[14](https://arxiv.org/html/2607.29181#A8.T14)\) primarily due to larger models being more verbose and specific about label assessments, thereby inflating vocabulary sizes \(Table[13](https://arxiv.org/html/2607.29181#A8.T13)\)\.
Table 13:Schematic equilibrium across VLM scales: vocab size by pass\. 4 Qwen3\-VL variants on 4 videos\.Table 14:Model choice: Markov vs\. majority accuracy \(%\) and perplexity before/after normalization, by label type\.Δ=\\Delta=Markov−\-Majority\. n = 4 videos, 4 Qwen3\-VL variants\.
## Appendix ITransferability Study
As a preliminary check on cross\-video transferability, we ran 3 leave\-one\-out markov evaluations on two same domain video triples: car repair, and coffee shop operation each \(6 total\)\. For car repair videos, Markov improved over the Majority baseline on the held\-out video, demonstrating the transferability of learned models to new videos\. On the other hand, in the coffee shop triple, one state \(pouring\_milk\_into\_cup\) occurs very frequently \(27%\-39% frames in each video\); in this case, Majority demonstrates better transferability\.
Table 15:Cross\-video Markov transferability \(LOOCV\): train on 2 videos’ concatenated activity sequences, test on the held\-out video\. Normalized vocabulary on the union of each triple att∗=0\.43t^\{\*\}=0\.43\.We were also interested if there was significant pair\-wise video transferability and conducted a brief pairwise transferability experiment, and found that yes if two videos are similar enough they are transferable; label ontology is relatively consistent for similar videos\.
Table 16:Cross\-video Markov transferability \(pairwise\): each video used as a single training set against another single video as test\. More conservative than LOOCV \(less train data, higher novel %\)\.Similar Articles
VideoSearch-R1: Iterative Video Retrieval and Reasoning via Soft Query Refinement
VideoSearch-R1 introduces an agentic framework that iteratively retrieves videos and refines search queries using continuous latent space refinement and policy optimization, achieving state-of-the-art performance on video corpus moment retrieval and temporal grounding tasks.
Watch, Remember, Reason: Human-View Video Understanding with MLLMs
A survey presenting a human-view perspective on video understanding with multimodal large language models, organized around watching, remembering, and reasoning abilities, covering challenges, methods, and applications.
GROVE: Growing and Reasoning over Temporally Stratified Memory from Streaming Video Experience
GROVE is a training-free framework that grows a temporally stratified memory from continuous video streams, supporting both reactive QA and proactive assistance. It achieves state-of-the-art results on benchmarks like MM-lifelong and EgoServe.
EffectLearner: World-Aware Object-Effect Reasoning for Real-World Video Object Removal
EffectLearner is a semantic-reasoning-enhanced framework for real-world video object removal, combining a VLM-based Object-Effect Reasoner with a DiT-based Video Eraser and introducing the EffectWorld dataset to handle complex object-induced effects.
VideoSeeker: Incentivizing Instance-level Video Understanding via Native Agentic Tool Invocation
VideoSeeker introduces a paradigm for instance-level video understanding that integrates agentic reasoning with visual prompts, achieving superior performance through automated data synthesis and reinforcement learning, outperforming GPT-4o and Gemini-2.5-Pro.