Activity Frames: Deterministic Screen-Activity Compilation for Agent Memory and Replay

# Deterministic Screen-Activity Compilation for Agent Memory and Replay
Source: [https://arxiv.org/html/2608.05784](https://arxiv.org/html/2608.05784)
Abstract

Computer\-use agents pay full frontier inference to re\-derive routines their user has already performed, because an agent’s memory today records what the user said, not what the user did\. We compile passively captured screen activity into agent memory with a deterministic, zero\-model pipeline: it segments a local capture stream into typed*activity frames*, bounded episodes carrying application, site, timing, input volume, and evidence pointers back to the raw rows, with no model in the loop, so the output is byte\-identical, cacheable, and mechanically auditable\. On one professional’s single\-user corpus of128,756128\{,\}756frames over5151active days, the compiler reduces a day of raw capture to a prompt\-ready context block86×86\\timessmaller in6868ms, and an agent reading that block answers questions about the day at98\.4%98\.4\\%accuracy \(Wilson95%95\\%CI91\.791\.7–99\.7%99\.7\\%\) against an independent oracle, versus6666–80%80\\%for an LLM summary of the same capture, a mid\-tier model reading the block matching a frontier one\.

The same compiler doubles as a demand\-side cost instrument\. Read off passive, pre\-delegation human activity rather than agent rollouts, it supplies two parameters that agent\-cost models assume but, to our knowledge, have not measured: the Routine Overhead RatioRRand the routine recurrencehh\. We report first values ofRR, a modeled upper bound, at6060–343×343\\times, and a delegable recurrence of9\.0%9\.0\\%in\-sample and7\.7%7\.7\\%out\-of\-sample, for a realistic all\-fleet token ceiling near8%8\\%; a compiled routine replays deterministically with the model out of the loop, demonstrated live at zero model tokens on a guard\-matched hit\. Schema, compiler, and evaluation harness are open\.

Keywords:LLM agents, episodic memory, agent cost, routine replay, computer\-use agents, screen capture, Model Context Protocol

## 1Introduction

An LLM agent asked “what should I prioritize today?” answers from what it can see: the conversation, some files, perhaps a calendar\. It cannot see that its user spent the morning inside a pull request, switched contexts forty times after lunch, or abandoned a draft email at 16:40\. Position work has argued that episodic memory, a record of experience situated in time, is the missing piece for long\-term LLM agents\[[33](https://arxiv.org/html/2608.05784#bib.bib1)\], and surveys of agent externalization treat memory as a first\-class architectural concern\[[52](https://arxiv.org/html/2608.05784#bib.bib2)\]\. Yet in practice, agent memory today means conversation memory: what the user told the model, not what the user did\[[29](https://arxiv.org/html/2608.05784#bib.bib3),[7](https://arxiv.org/html/2608.05784#bib.bib5),[35](https://arxiv.org/html/2608.05784#bib.bib4)\]\.

The raw material for behavioral episodic memory already exists\. Automated time trackers have recorded application focus for a decade\[[3](https://arxiv.org/html/2608.05784#bib.bib15)\]; operating systems now ship continuous capture, with Microsoft Recall storing periodic snapshots on Copilot\+ PCs\[[25](https://arxiv.org/html/2608.05784#bib.bib13)\]; and OpenAI’s Chronicle preview builds memories from recent screen content for its Codex assistant\[[27](https://arxiv.org/html/2608.05784#bib.bib12)\]\. Capture, in other words, is commodity\. Consumption is not\. A day of event\-driven capture on our corpus yields roughly two thousand snapshot rows, each one asserting only that at timett, applicationaadisplayed windowwwat URLuu\. Two consumption strategies dominate\. Raw search hands the agent a flat result list and leaves sessionization, deduplication, and duration accounting to the model at inference time; we measure this cost at 126,812 tokens for a single day \(Section[6](https://arxiv.org/html/2608.05784#S6)\)\. LLM summarization compresses well but imports the weaknesses of its summarizer: per\-run cost, non\-determinism, degraded reliability over long inputs\[[22](https://arxiv.org/html/2608.05784#bib.bib18)\], and the possibility of inventing activity that never happened\. Neither yields memory an agent can cache, audit, or trust\.

This paper proposes the missing middle:*deterministic compilation*\. In the same way a compiler turns instructions into structured artifacts without opinion, we turn snapshot streams into*activity frames*: bounded episodes with an application and site, a start and end, dwell\-based active time, typed page references, input volume, and evidence pointers back to the raw rows\. No model participates\. The same database and window always produce the same document, so episodic memory becomes free to rebuild, safe to cache, and mechanically auditable\. Interpretation is not banned but quarantined: a second schema tier accepts inferred labels only when they are namespaced, confidence\-tagged, and linked to the measured evidence that supports them\.

Our contributions are:

- •A two\-tier schemafor representing human computer activity to agents, separating measured fact from tagged inference, with coverage gaps and blind spots as mandatory document elements \(Section[3](https://arxiv.org/html/2608.05784#S3)\)\.
- •Deterministic compilation rules\(dwell crediting, session gaps, flicker merging, nearest\-frame attribution, coordinate\-based click resolution, total URL\-to\-entity typing\) that require no learned components \(Section[4](https://arxiv.org/html/2608.05784#S4)\)\.
- •An open reference implementation: a dependency\-free Python compiler with a provisioned local capture engine, a CLI, and a Model Context Protocol \(MCP\) server exposing six tools to any MCP\-capable agent\[[14](https://arxiv.org/html/2608.05784#bib.bib16),[2](https://arxiv.org/html/2608.05784#bib.bib14)\]\(Section[5](https://arxiv.org/html/2608.05784#S5)\)\.
- •An empirical characterizationon a 61\-day live corpus: token cost against raw rows \(86×\\timesreduction for prompt\-ready context\), 68 ms full\-day compilation, byte\-identical reproducibility, entity\-typing coverage, and a duration distribution that separates genuine short attention bouts from single\-snapshot transits and multi\-monitor effects \(Section[6](https://arxiv.org/html/2608.05784#S6)\)\.
- •A downstream question\-answering benchmarkagainst an independent oracle, run at two model tiers, showing that an agent answers more accurately from the compiled block than from raw rows or an LLM summary of the same capture at both tiers, that the block lets a mid\-tier model match a frontier one, and that a stronger model narrows but does not close the gap; the harness is released for rerunning against any model \(Section[6\.2](https://arxiv.org/html/2608.05784#S6.SS2)\)\.
- •A demand\-side cost instrument: an open, deterministic measurement of the Routine Overhead RatioRR\(modeled numerator, measured denominator\) and desktop routine recurrencehh, read from passively captured human activity rather than agent rollouts, with a parametric replay executor and a first live proof\-of\-concept, reported on one user’s corpus \(Sections[7](https://arxiv.org/html/2608.05784#S7)–[7\.4](https://arxiv.org/html/2608.05784#S7.SS4)\)\.

## 2Related Work

### 2\.1Memory Systems for LLM Agents

MemGPT virtualizes context OS\-style, paging conversation history through a bounded window\[[29](https://arxiv.org/html/2608.05784#bib.bib3)\]; production layers consolidate salient facts from dialogue\[[7](https://arxiv.org/html/2608.05784#bib.bib5)\]; Zep organizes memory as a temporal knowledge graph\[[35](https://arxiv.org/html/2608.05784#bib.bib4)\]; E\-mem reconstructs episodic context from execution traces\[[42](https://arxiv.org/html/2608.05784#bib.bib6)\]; and hierarchical procedural memory distills skills from agent trajectories\[[12](https://arxiv.org/html/2608.05784#bib.bib11)\], threads a recent review unifies as externalization\[[52](https://arxiv.org/html/2608.05784#bib.bib2)\]\. All ingest what flowed*through the agent*, messages, tool calls, task traces, and memory\-construction studies show that granularity and structure affect retrieval quality\[[31](https://arxiv.org/html/2608.05784#bib.bib24)\], supporting typed episodes over flat logs\. Agent Workflow Memory induces routines too, but from agent rollouts rather than human activity\[[45](https://arxiv.org/html/2608.05784#bib.bib26)\]\.

### 2\.2Screen Capture as Agent Memory

A 2025–2026 wave targets our exact goal by the opposite method\. MIRIX runs a multi\-agent memory system over continuous screenshots with cloud model calls\[[43](https://arxiv.org/html/2608.05784#bib.bib21)\]; FOCAL invokes a local vision\-language model to write session summaries from on\-device capture, reporting a 60% token cut\[[48](https://arxiv.org/html/2608.05784#bib.bib22)\]; ProAgentBench segments 500\+ hours of capture on application switches before model\-based annotation\[[39](https://arxiv.org/html/2608.05784#bib.bib23)\]; SummAct summarizes interaction traces into intentions with an LLM\[[50](https://arxiv.org/html/2608.05784#bib.bib25)\]; and OmniQuery augments captured personal media for QA\[[19](https://arxiv.org/html/2608.05784#bib.bib37)\]\. In every case a model sits inside the memory\-construction loop, so the memory inherits per\-run cost, non\-determinism, and possible hallucinated episodes\. Activity frames are the deterministic counterpoint: no model in the compile path, byte\-identical output, and interpretation quarantined in a separate evidence\-linked tier, the first deterministic desktop instantiation of the acquisition layer episodic\-memory advocates call missing\[[33](https://arxiv.org/html/2608.05784#bib.bib1)\]\.

### 2\.3Desktop Activity, GUI Agents, and Memory Trust

Deriving task structure from interaction logs is a two\-decade ambition: TaskTracer tied window, file, and clipboard events to declared tasks\[[10](https://arxiv.org/html/2608.05784#bib.bib27)\], and SWISH clustered windows into tasks from titles and switching\[[26](https://arxiv.org/html/2608.05784#bib.bib28)\]\. Interruption science established the fragmentation our compiler measures, three\-minute working spheres\[[13](https://arxiv.org/html/2608.05784#bib.bib29)\]and content switching about every 19 s with 75% of segments under a minute\[[47](https://arxiv.org/html/2608.05784#bib.bib30)\]; these are behavioral rates from their instruments, and our comparable figure is the post\-transit median frame of0\.90\.9min \(Section[6](https://arxiv.org/html/2608.05784#S6)\), not our6\.16\.1s median inter\-capture gap, which is recorder cadence rather than a human switching rate\. Methodologically our pipeline descends from lifelogging’s threshold segmentation\[[9](https://arxiv.org/html/2608.05784#bib.bib32)\], whose critique that unstructured total capture serves nobody\[[36](https://arxiv.org/html/2608.05784#bib.bib31)\]is the failure mode we exist to prevent, plus web sessionization’s inactivity timeouts\[[8](https://arxiv.org/html/2608.05784#bib.bib34)\], process\-mining event abstraction\[[41](https://arxiv.org/html/2608.05784#bib.bib36)\], robotic process mining’s UI\-log routine identification\[[17](https://arxiv.org/html/2608.05784#bib.bib33)\], and the data\-to\-text faithfulness tradition\[[34](https://arxiv.org/html/2608.05784#bib.bib35)\]; but those pipelines end in process models or prose for humans, ours in typed memory for an agent\. Separately, a large body gives agents screens as an*action*surface, GUI\-agent surveys cataloging accessibility\-tree, OCR, and vision perception\[[49](https://arxiv.org/html/2608.05784#bib.bib17),[21](https://arxiv.org/html/2608.05784#bib.bib7)\], and generative agents showing a remembered observation stream enables long\-horizon behavior\[[32](https://arxiv.org/html/2608.05784#bib.bib8)\]; we compile the inverse, what the human did, an acquisition problem personal\-agent surveys call open\[[20](https://arxiv.org/html/2608.05784#bib.bib20)\]\. Deployed capture leaves the same gap: ActivityWatch is content\-blind\[[3](https://arxiv.org/html/2608.05784#bib.bib15)\], Recall exposes snapshots to the user but no agent API\[[25](https://arxiv.org/html/2608.05784#bib.bib13)\], Chronicle feeds one walled assistant\[[27](https://arxiv.org/html/2608.05784#bib.bib12)\]\. Finally, agent memory is now an attack surface, memories poisoned to steer behavior\[[23](https://arxiv.org/html/2608.05784#bib.bib9)\]or probed by membership inference\[[5](https://arxiv.org/html/2608.05784#bib.bib10)\], motivating two properties we make structural: evidence pointers back to raw rows, and a hard measured/inferred boundary, so a poisoned or hallucinated label cannot pass as fact\. Context\-engineering and long\-context degradation results both argue for compact structured context over raw dumps\[[24](https://arxiv.org/html/2608.05784#bib.bib19),[22](https://arxiv.org/html/2608.05784#bib.bib18)\]\.

### 2\.4Agent Skills, Trajectories, and Cost

Three adjacent lines each need one of our parameters but source it differently\.*Skill induction*turns an agent’s own successful trajectories into reusable skills: Agent Workflow Memory from web rollouts\[[45](https://arxiv.org/html/2608.05784#bib.bib26)\], SkillWeaver from self\-execution\[[51](https://arxiv.org/html/2608.05784#bib.bib48)\], Agent Skill Induction as programs\[[44](https://arxiv.org/html/2608.05784#bib.bib49)\], and PreAct as continual self\-improvement\[[18](https://arxiv.org/html/2608.05784#bib.bib50)\]\. All read the*supply*side, what an agent did while acting, and so observe a routine only after an agent has already performed it at full cost, and only for tasks it can complete\.*Trajectory\-acquisition*channels manufacture training data and report a price: Explorer at≈$​0\.28\{\\approx\}\\mathdollar 0\.28each\[[30](https://arxiv.org/html/2608.05784#bib.bib43)\], AgentTrek at≈$​0\.55\{\\approx\}\\mathdollar 0\.55\[[46](https://arxiv.org/html/2608.05784#bib.bib44)\], and Watch & Learn\[[37](https://arxiv.org/html/2608.05784#bib.bib45)\]and cotomi Act\[[28](https://arxiv.org/html/2608.05784#bib.bib46)\]at low marginal cost; but a synthesized trajectory exists only where the generator can drive the task to success, so authenticated, private\-state routines are systematically under\-covered, and Agent Data Protocol’s channel taxonomy\[[38](https://arxiv.org/html/2608.05784#bib.bib47)\]omits both passive capture and a price column\.*Cost frameworks*price the rest: FrugalGPT for model cascades\[[6](https://arxiv.org/html/2608.05784#bib.bib39)\], Cost\-of\-Pass for expected cost per success\[[11](https://arxiv.org/html/2608.05784#bib.bib41)\], AI Agents That Matter for cost as a first\-class axis\[[16](https://arxiv.org/html/2608.05784#bib.bib40)\], and the Holistic Agent Leaderboard for real rollout economics \(≈$​1\.84\{\\approx\}\\mathdollar 1\.84/task\)\[[15](https://arxiv.org/html/2608.05784#bib.bib42)\]\. Together they supply an amortized per\-task cost of a memory\-backed agent, which we adopt as a cited frame, not a contribution:

𝔼​\[$/task\]=\(1−h​q\)​Cmissp\+h​q​\(Chit\+Cverify\)\+h​\(1−q\)​Cwrong\+CwriteN,\\begin\{split\}\\mathbb\{E\}\[\\mathdollar/\\text\{task\}\]=\{\}&\(1\{\-\}hq\)\\tfrac\{C\_\{\\text\{miss\}\}\}\{p\}\+hq\\,\(C\_\{\\text\{hit\}\}\{\+\}C\_\{\\text\{verify\}\}\)\\\\ &\{\}\+h\(1\{\-\}q\)\\,C\_\{\\text\{wrong\}\}\+\\tfrac\{C\_\{\\text\{write\}\}\}\{N\},\\end\{split\}\(1\)wherehhis recurrence,qqthe fraction of hits correctly matched,ppthe base success rate,NNthe reuse count, and theCC\-terms per\-branch costs\. Every term is already priced in the works above; the one input none of them measures ishhon the*pre\-delegation*passive corpus\. Rollout\-priced leaderboards observe the hit rate of tasks an agent was given and completed, a survivorship\-biased quantity; the demand\-sidehh, how often the user’s real work repeats whether or not an agent could do it, is what our instrument supplies, andRRreplaces the assumedCmiss/ChitC\_\{\\text\{miss\}\}/C\_\{\\text\{hit\}\}ratio with a measured one\. Activity frames are thus the pre\-delegation, demand\-side instrument that skill induction, acquisition channels, and cost leaderboards each presuppose but none provides\.

## 3The Activity Frames Schema

### 3\.1Design Principles

Four principles fix the schema’s character\.Measured, not guessed: every field in a standard document is derivable by deterministic code from capture data; there are no intent labels, because code cannot observe intent\.Reproducible: identical inputs must yield identical documents, making memory cacheable and diffable\.Evidenced: every frame carries pointers to the raw rows it was compiled from\.Honest about absence: periods without capture are reported as gaps, and every document carries ablind\_spotslist stating what the pipeline systematically cannot see\. Consumers must treat uncovered time as unknown, never as inactivity\.

local machine; nothing leaves by defaultcapture enginescreen, a11y tree,input events \(SQLite\)compilersessionize, enrich,entity typingagentMCP tools,context blockstier 2: inferrednamespaced, confidence\-tagged, evidence\-linkedread\-onlytier 1optionalFigure 1:The two\-tier architecture\. The measured tier \(blue\) is produced entirely by deterministic code reading the capture database\. Interpretation \(orange\) is an optional extension that must be namespaced, confidence\-tagged, and evidence\-linked; stripping it always leaves a valid measured document\.
### 3\.2Document Structure

A document describes one query window and contains four parts: acoveragesection \(first and last activity, active minutes, span, capture gaps over five minutes\), a chronological list offrames, ablind\_spotslist, and provenance metadata \(schema\_version, generation time, source recorder\)\. A frame is one bounded stretch of attention in a single context, keyed by the pair \(application, site\), where site is the URL host for browser activity and absent otherwise\. Figure[2](https://arxiv.org/html/2608.05784#S3.F2)shows a representative frame\.

\- id: f\-0007app: "Google Chrome"site: "linkedin\.com"start: "20:24:04" end: "20:42:11"duration\_min: 18\.0 wall\_min: 21\.5pages:\- \{kind: people\_search, entity: "cto paris", count: 2\}\- \{kind: profile, entity: "jane\-doe"\}\- \{kind: company, entity: "acme"\}input: \{keys: 214, clicks: 31\}interruptions: \[\{app: "Slack", seconds: 12\}\]evidence: \{frame\_ids: "99871\.\.100147"\}Figure 2:A single activity frame \(YAML, entities anonymized\)\. Every field is computed by code; the evidence pointer names the raw snapshot rows the frame was compiled from\.
### 3\.3Typed Page References

Browser frames carrypages: typed references produced by deterministic URL parsing\. A reference has akind\(what sort of page\), an optionalentity\(the human\-relevant identifier\), and a view count\. Standard kinds includeprofile,company,people\_search,search,repo,pull\_request,issue,doc,email,video,post,ai\_chat, andlocal\_dev\. The mapping is total: a URL matched by no site parser falls back to a genericpagereference with its domain, so typing never loses data\. Kinds are open for extension but must remain deterministic functions of the URL\.

### 3\.4Tier 2: The Inferred Extension

Tools may add interpretation, such as task labels or project clusters, under three schema\-enforced rules: inferred content lives in a namespacedinferredblock, carries aconfidencetag drawn from \{high,medium,speculative\}, and names itsevidence: the measured fields or raw rows supporting it\. A consumer can always strip theinferredblock and be left with a purely measured document\. The reference implementation emits tier 1 only; we consider the boundary itself, not any particular inference method, to be the contribution\.

### 3\.5Privacy Rule

Input*volume*\(keystroke, click, and copy counts\) is part of the standard document\. Input*content*\(typed text\) must be excluded by default and included only on an explicit operator opt\-in\. This is a schema requirement, not an implementation courtesy: a conforming producer cannot silently emit content\.

## 4Deterministic Compilation

### 4\.1Setting

The capture engine is event\-driven with a heartbeat: it stores a snapshot row on interaction and on screen change \(clicks, application switches, visual changes\), plus a periodic row after roughly 30 seconds without input\. Heartbeat rows are 19% of our corpus; they matter because they keep the stream alive while the user reads, watches, or steps away from an awake display, and every downstream time measure inherits that\. Each monitor records its own stream; within a stream the median inter\-frame gap is 6\.1 s and the 90th percentile is 30\.7 s \(Section[6](https://arxiv.org/html/2608.05784#S6)\)\. A row carries a timestamp, application, window title, and URL when the focused application is a browser\. Input events \(keystrokes, clicks, clipboard, application switches\) arrive in a parallel stream\. Compilation consumes these streams for a query window, segmenting each monitor independently, and emits the document of Section[3](https://arxiv.org/html/2608.05784#S3)\.

### 4\.2Sessionization

Three constants govern segmentation, chosen from the capture cadence rather than tuned on outcomes\.Dwell: a frame contributesmin⁡\(Δ​t,90​s\)\\min\(\\Delta t,90\\,\\mathrm\{s\}\)of active time, whereΔ​t\\Delta tis the gap to the next frame in its monitor’s stream\. Because the engine emits a∼\\sim30 s heartbeat during input\-free stretches, dwell measures*screen tenure*: how long a context stayed frontmost on an awake display\. That includes reading and watching, which produce no input, but also stretches where the user has stepped away while the display stays on; Section[8](https://arxiv.org/html/2608.05784#S8)quantifies this\. The cap, roughly three heartbeat periods, bounds the credit a single frame can earn when capture stalls or the display sleeps; it does not, and cannot, distinguish attention from presence\. Consumers who need interaction\-gated time can compare each frame’s reported input volume against its duration\. Segmentation runs per monitor, so a context visible on two monitors at once earns tenure on both; the schema discloses this as a blind spot and Section[6\.2](https://arxiv.org/html/2608.05784#S6.SS2)shows the consequence\.Session gap: a gap above 300 s closes the current frame and becomes a candidate coverage gap; no dwell is credited across it\.Flicker merge: the patternA→B→AA\\to B\\to A, whereBBlasts at most 20 s of wall time and no session gap intervenes, collapses into a singleAAframe\. Crucially,BBis not discarded: it is recorded on the merged frame as aninterruptionwith its measured seconds, and its time is*not*added toAA’s active duration\. Nothing is hidden and nothing is double\-counted\. Algorithm[1](https://arxiv.org/html/2608.05784#alg1)summarizes the procedure; Figure[3](https://arxiv.org/html/2608.05784#S4.F3)illustrates all three rules on one timeline\.

Algorithm 1Frame segmentation \(one pass, then flicker merge\)0:snapshots

f1​…​fnf\_\{1\}\\dots f\_\{n\}of one monitor’s stream, sorted by time; constants

D=90D\{=\}90,

G=300G\{=\}300,

F=20F\{=\}20\(seconds\)

1:

S←\[\]S\\leftarrow\[\];

c​u​r←⊥cur\\leftarrow\\bot
2:for

i=1i=1to

nndo

3:

ki←\(app​\(fi\),site​\(fi\)\)k\_\{i\}\\leftarrow\(\\mathrm\{app\}\(f\_\{i\}\),\\mathrm\{site\}\(f\_\{i\}\)\);

Δ←t​\(fi\+1\)−t​\(fi\)\\Delta\\leftarrow t\(f\_\{i\+1\}\)\-t\(f\_\{i\}\)
4:if

c​u​r=⊥cur=\\botor

ki≠key​\(c​u​r\)k\_\{i\}\\neq\\mathrm\{key\}\(cur\)then

5:append new segment

c​u​rcurwith key

kik\_\{i\}to

SS
6:endif

7:extend

c​u​rcurwith

fif\_\{i\}
8:if

Δ≤G\\Delta\\leq Gthen

9:

active\(cur\)\+=min\(Δ,D\)\\mathrm\{active\}\(cur\)\\mathrel\{\+\}=\\min\(\\Delta,D\)
10:else

11:

c​u​r←⊥cur\\leftarrow\\bot\{session break; candidate gap\}

12:endif

13:endfor

14:forconsecutive

A,B,A′A,B,A^\{\\prime\}in

SSwith

key​\(A\)=key​\(A′\)\\mathrm\{key\}\(A\)\{=\}\\mathrm\{key\}\(A^\{\\prime\}\)do

15:if

wall​\(B\)≤F\\mathrm\{wall\}\(B\)\\leq Fand no session break around

BBthen

16:merge

A′A^\{\\prime\}into

AA; record

BBas interruption of

AA
17:endif

18:endfor

19:return

SS

timeA \(linkedin\.com\)B 12sA cont\.flicker mergedgap 41 min \(reported\)C \(github\.com\)Δ​t=104​s⇒\\Delta t\{=\}104\\mathrm\{s\}\\Rightarrowcreditmin⁡\(Δ​t,90​s\)\\min\(\\Delta t,90\\mathrm\{s\}\)Figure 3:Segmentation on one timeline\. Ticks are snapshot rows\. A 12\-second detour \(BB\) folds into the surrounding frame as a recorded interruption; a long silence becomes a reported coverage gap; sparse snapshots earn at most the 90 s dwell cap\.
### 4\.3Enrichment

Raw input events have three reliability defects that code can repair\. First,stale attribution: the recorder sometimes tags an event with the previously focused application\. Each event is therefore re\-attributed to the temporally nearest snapshot \(binary search over the frame stream\), whose application, window, and URL are authoritative; the attribution distance is kept in milliseconds so downstream consumers can judge it\. Second,anonymous clicks: many click events carry coordinates but no element name\. We resolve them against the snapshot’s recorded element tree: exact containment first \(smallest containing element wins\), then a±\\pm40 px tolerance ring, then a coarse screen\-zone fallback; every resolution is taggedexact,tolerance, orzone, and unresolvable clicks stay unresolved rather than being guessed\. Third,keyboard\-layout mismatch: some capture stacks record physical key positions decoded as QWERTY while the user types another layout\. An explicit, operator\-supplied translation map repairs this; it is identity by default and never inferred\.

### 4\.4Entity Typing

Site parsers map URLs to the typed references of Section[3](https://arxiv.org/html/2608.05784#S3): path and query parsing only, no fetching, no models\. Resolution proceeds in layers: a bespoke parser for the host \(the reference implementation ships more than twenty, covering professional networks, code hosting, search, documents, mail, maps, video, social, events, dashboards, AI chat, and local development\), then a generic search\-parameter detector, then a subdomain\-and\-path heuristic that types common infrastructure pages \(sign\-in, dashboard, email, calendar, meeting\) even for sites without a bespoke parser, and finally a total fallback that guarantees every URL maps to something\. Because every layer is a pure function of the URL, adding coverage is a contribution reviewable line by line\.

## 5Reference Implementation

The reference implementation\[[14](https://arxiv.org/html/2608.05784#bib.bib16)\]is a Python package \(MIT license\) with three parts\. Thecapture engineis provisioned on demand byaframes record: a pinned, MIT\-licensed, open\-source build that records application focus, window titles, URLs, the accessibility tree, and input events into a local SQLite database, entirely on\-device, with audio capture off by default\. Operators already running a compatible recorder can skip it and point the compiler at any existing database\. Thecompilerhas zero runtime dependencies, opens the database read\-only, and exposes the document builders plus emitters for JSON, YAML, Markdown, and a compact plaintext context block designed for system prompts\. TheMCP server\[[2](https://arxiv.org/html/2608.05784#bib.bib14)\]is a hand\-rolled stdio JSON\-RPC implementation, also dependency\-free, exposing the six tools of Table[1](https://arxiv.org/html/2608.05784#S5.T1)to any MCP client\. A workflow\-pattern detector \(repeated clicks, URL loops, action sequences, application\-switching habits, daily habits\) rounds out the surface\.

Table 1:MCP tool surface of the reference implementation\.
## 6Empirical Characterization

We characterize the system on the author’s own live corpus: 61 calendar days of capture \(46 days with activity\) comprising 109,735 snapshot rows, 214,360 input events, and 8\.4M element\-tree rows across 54 applications \(Table[2](https://arxiv.org/html/2608.05784#S6.T2)\)\. The corpus is frozen: the recorder was migrated to a new database after the last captured day, so every number below is reproducible against an immutable file\. The paper reads from this corpus at two freezes, labeled where used: this systems half uses the 2026\-07\-10 freeze \(61 calendar / 46 active days, 109,735 frames\), while the overhead measurements of Section[7](https://arxiv.org/html/2608.05784#S7)use a later 2026\-07\-22 freeze \(51 active days, 128,756 frames\)\. This is a single\-user corpus; we report it as a characterization of the mechanism, not a user study \(Section[8](https://arxiv.org/html/2608.05784#S8)\)\.

Table 2:Live capture corpus used throughout Section[6](https://arxiv.org/html/2608.05784#S6)\.### 6\.1Token Cost

For one representative full day \(2,066 snapshot rows\), we compare three representations an agent could receive, tokenized with thecl100k\_baseencoding\. The raw rows, serialized as the JSON a search API would return, cost 126,812 tokens\. The compiled schema\-v1 document \(222 frames at a 0\.5\-minute floor\) costs 34,815 tokens, a 3\.6×\\timesreduction that also relieves the agent of segmentation work\. The compact context block costs 1,469 tokens, an 86×\\timesreduction, small enough to include in every system prompt \(Figure[4](https://arxiv.org/html/2608.05784#S6.F4)\)\. Both compiled forms are produced without any model call, so the reduction is free, and long\-context results suggest the smaller representation is not merely cheaper but better used by the model\[[22](https://arxiv.org/html/2608.05784#bib.bib18)\]\.

raw rowsframes JSONcontext block10310^\{3\}10410^\{4\}10510^\{5\}126,81234,8151,469tokens \(log\)Figure 4:Token cost of one full day under three representations \(cl100k\_base\)\. Deterministic compilation yields an 86×\\timesreduction for prompt\-ready context, at zero inference cost\.
### 6\.2Downstream Question Answering

Token cost is a means; the end is whether an agent can*answer questions about the user’s day*\. We test this directly\. We evaluate on eight days chosen by a fixed rule: the seven most recent consecutive active days plus the nearest preceding day whose raw serialization overflows the model’s context window \(included to exercise that regime\)\. For each day, ground\-truth answers are computed by an*independent*SQL oracle over the raw tables: a documented inactivity\-timeout dwell \(credit each frame the gap to the next, capped at6060s\), deliberately not the activity\-frames compiler, so the reference is not circular\. Oracle and compiler measure the same construct, screen tenure: the capture heartbeat \(Section[4](https://arxiv.org/html/2608.05784#S4)\) keeps input\-free stretches credited, so “active minutes” throughout this section means time a context was frontmost on an awake display, not interaction time\. Every representation is graded against that one construct, which keeps the comparison fair, but absolute magnitudes should be read as tenure\. We validate the compiler against the oracle before using the oracle to grade anyone\. On the seven benchmark days whose capture was complete when the oracle was frozen, the compiler’s covered active minutes agree with the oracle’s total dwell to a median of0\.90\.9minutes \(all within2\.32\.3\); the eighth day’s capture continued after the freeze, so it is compared only at its snapshot\. Distinct\-application counts agree exactly or within one on every day\. Two caveats keep this agreement honest\. First, it compares aggregates that both approximate covered wall time, so it validates coverage rather than per\-application arithmetic\. Second, the two systems intentionally differ on multi\-monitor days: the compiler credits each monitor’s stream \(Section[4](https://arxiv.org/html/2608.05784#S4)\), while the oracle interleaves all monitors into one\. On the context\-overflow day, which has133133dual\-monitor minutes, the dominant application earns435\.8435\.8minutes by the compiler’s ledger but346\.1346\.1by the oracle, a26%26\\%divergence that the grading tolerance below happens to contain; switching the oracle’s cap from6060to9090s moves its day total by under66minutes, so the divergence is the monitor convention, not the dwell cap\. We report this rather than average over it\. The generator emits 64 questions across five categories: which application dominated, how many active minutes in it, how many distinct applications, pairwise time ranking, whether a given domain was visited, and starting time, plus 16*absent\-fact probes*\(“did the user open Photoshop / visit netflix\.com?”\) that a faithful system answers negatively\. We deliberately exclude two further question types\. Longest\-session length depends on the session\-gap convention rather than a fact, and per\-profile recall exceeds the compact block’s token budget on busy days; each would penalize the compact block for a convention or a compression choice rather than for faithfulness, so we record the exclusion here to keep it on the books\.

Two agents at different capability tiers, Claude Sonnet 4\.5 and the larger Opus 4\.5, each given no tools and answering only from the supplied text, answer every question from each of three representations: the raw rows, an LLM\-generated summary of those rows written by that same agent, and the activity\-frames output \(the deterministic per\-application ledger plus the compact context block, together≈\\approx2k tokens\)\. We note plainly that this hands the activity\-frames agent measured per\-application durations the other two must derive for themselves: that is exactly what the compiler is for, but it means the quantitative questions test whether the deterministic arithmetic was already done, not only whether the agent can read\. Answers are graded against the oracle with fixed tolerances \(numeric within30%30\\%; times within4545minutes\), and the ranking does not depend on them: at the Sonnet tier, a strict10%10\\%/1515\-minute band leaves the three representations at95\.3%95\.3\\%,80\.4%80\.4\\%, and66\.1%66\.1\\%, and the summary is so far off that widening the band to50%50\\%lifts it only to67\.9%67\.9\\%\. Table[3](https://arxiv.org/html/2608.05784#S6.T3)reports both tiers at the default tolerance\.

Table 3:Downstream QA across two model tiers \(8 days, 64 ground\-truth questions, graded against the independent SQL oracle\)\. “Acc\.” is overall accuracy; “Dur\. err\.” is the mean absolute error on the dominant application’s active minutes\. The compiled block scores identically at both tiers, so a mid\-tier model reads it as well as a frontier one; the raw\-row and summary baselines improve with model strength but never catch it\. Only the block is deterministic across runs and fits the context window on every day; on the busiest day the raw rows are257257k tokens, so both baselines are infeasible and report on77of88days \(the block, on all88\)\.Four findings\. First,the gap is quantitative, not categorical\.At both tiers all three representations score perfectly on the absent\-fact probes: the models do not invent applications or websites \(hallucination rate0%0\\%throughout\)\. The summary fails instead on*magnitudes*\. At the Sonnet tier it answers time questions at7\.1%7\.1\\%accuracy and mis\-states the dominant application’s active minutes by135\.7%135\.7\\%on average, against7\.3%7\.3\\%for activity frames \(that residual is dwell\-method rounding, not error\)\. Fluent prose hides the damage: on 2026\-07\-05 the measured record \(the oracle, with the compiler’s ledger agreeing\) is Google Chrome first at161\.6161\.6dwell minutes and Cursor second at143\.9143\.9, over a day that ran11:4011\{:\}40to22:2622\{:\}26, yet a Sonnet summary of that day names Cursor the primary application at “∼\\sim7 hours,” inflating its144144measured minutes by2\.9×2\.9\\timesand fabricating a nocturnal “6:406\{:\}40PM–5:265\{:\}26AM” session that spills into the next day\. The activity\-frames agent answered160160minutes\. Every day, summary, and graded answer is in the released harness output\.

Second,a stronger model narrows the gap but does not close it, and the compiled block erases the difference between the two\.The frontier Opus model reconstructs durations far better from prose: its summary reaches80\.4%80\.4\\%overall and cuts the duration error to25\.2%25\.2\\%\. But that is still3\.5×3\.5\\timesthe block’s7\.3%7\.3\\%, and reading the block the two models are indistinguishable \(98\.4%98\.4\\%each\), whereas reading raw rows or a summary the frontier model runs99to1414points ahead of the mid\-tier one\. The block lets the cheaper model answer as well as the expensive one; the baselines make capability matter\. The benefit of deterministic compilation is therefore largest exactly where compute is cheapest to deploy\.

Third,the summary is non\-deterministic: regenerating it three times per day produced three distinct texts on every day at both tiers, whereas the activity\-frames block was byte\-identical across three regenerations, so the summary’s errors are not even stable enough to correct for\. Fourth,the baselines do not always run\.The busiest day serializes to257257k raw tokens, exceeding the context window, so both raw\-row and summary consumption are infeasible; the≈\\approx2k block is unaffected, which is why it alone reports on all eight days\. Activity frames are not flawless: their one miss \(98\.4%98\.4\\%, not100100, at both tiers\) is a domain\-recall question on the busiest evaluated day, where the compact block’s budget drops a visited domain; that is the expected price of a bounded representation, and a recall question rather than a magnitude one\. These accuracies are a single answering pass per representation per tier, at provider default sampling \(model snapshotsclaude\-sonnet\-4\-5andclaude\-opus\-4\-5, temperature and seed unpinned, so the non\-deterministic baselines vary across re\-runs\); we release the full harness \(oracle, question generator, and grader\) so they can be repeated, with confidence intervals and further models, on any corpus\.

### 6\.3Latency and Reproducibility

Compiling the same full day end to end \(segmentation, enrichment\-backed input accounting, entity typing, emission\) takes a median of 68 ms over five runs on a consumer laptop \(Apple Silicon\), with a 65 to 72 ms range\. Episodic memory at this cost can simply be rebuilt on every query\. Reproducibility holds by construction and we verify it empirically: two independent compilations of the same window produce byte\-identical documents once the generation timestamp is excluded\. Determinism is what makes the memory cacheable, diffable across code versions, and testable in continuous integration\.

### 6\.4Entity Typing Coverage

Across all 5,120 distinct URLs in the corpus, the layered parsers produce a non\-generic typed reference for 81\.3%, spanning 46 kinds; the remaining 18\.7% fall back to the generic page reference with domain\. The most frequent non\-generic kinds areprofile\(1,106 distinct URLs\),search\(397\),email\(278\),event\(209\), andmessaging\(200\), reflecting that typed coverage concentrates precisely on the high\-signal pages an agent most benefits from resolving\. Coverage grows one pure function per site, so the long tail is closed incrementally by contribution rather than by any learned component\.

### 6\.5How Fragmented Is a Day?

Compiling the 43 days that produce compiled frames \(of the 46 with any capture in Table[2](https://arxiv.org/html/2608.05784#S6.T2); the other three carry only idle\-heartbeat rows\) yields 17,514 frames \(median 361 per day\) with a median active duration of 0\.5 minutes: 68% of frames last under one minute \(Figure[5](https://arxiv.org/html/2608.05784#S6.F5)\)\. A number this stark demands decomposition before interpretation, because two mechanical effects sit inside it\. First, 52% of the sub\-minute frames \(6,177\) contain a single snapshot, with a median credit of 6\.1 seconds: these are*transits*, the application an operator passes through on the way somewhere else, faithfully recorded but not attention episodes\. Second, a further 20% \(2,361\) lie inside dual\-monitor stretches, where each monitor earns its own stream by construction; 27% of captured minutes on this corpus have two monitors active\. Restricting to frames with at least two snapshots, the median rises to 0\.9 minutes and 52% remain under one minute; that residue is genuine switching\. Read this way, the distribution is consistent with what interruption science has measured with dedicated instrumentation: three\-minute working spheres\[[13](https://arxiv.org/html/2608.05784#bib.bib29)\]and 75% of laptop content segments under one minute\[[47](https://arxiv.org/html/2608.05784#bib.bib30)\]\. We deliberately do not present the raw histogram as an independent replication: an earlier draft of this analysis segmented all monitors as one interleaved stream, which shreds dual\-monitor stretches and pushed the median to 0\.3 minutes, and strict segmentation always overstates fragmentation until transits are separated out\. The lesson is the schema’s, not just ours: consumers get amin\_minutesfloor \(frames below it are omitted with the omission disclosed, never silently merged\), the flicker\-merge rule keeps sub\-20\-second detours from shredding genuine focus blocks while still recording them, and single\-snapshot transits remain visible in the document precisely so that no compiler\-side heuristic has to decide what counts as attention\.

0\-11\-22\-55\-1010\-2020\-4545\+00\.50\.511⋅104\\cdot 10^\{4\}11,84211\{,\}8422,7302\{,\}7301,9521\{,\}952641641275275707044active duration \(min\)framesFigure 5:Active\-duration distribution of all 17,514 frames across 43 days \(per\-monitor segmentation\)\. Median 0\.5 min, but 52% of the sub\-minute bar is single\-snapshot transits rather than attention episodes; excluding them the median is 0\.9 min \(see text\)\.

## 7The Routine Overhead Ratio

The compiler of Sections[3](https://arxiv.org/html/2608.05784#S3)–[5](https://arxiv.org/html/2608.05784#S5)was built to produce episodic memory, but it is also an*instrument*\. Because it reduces a recurring stretch of capture to a short, replayable script by deterministic code alone, it lets us put a measured number on a quantity that agent\-cost models assume but none of them observes: how much a computer\-use agent overpays to re\-derive a routine it has already done\. We call that number the Routine Overhead RatioRR, and we report it, together with the recurrencehh\(Section[7\.4](https://arxiv.org/html/2608.05784#S7.SS4)\), as the first readings of the two parameters those cost models are written in\. NeitherRRnorhhis offered as a principle; each is a measurement, taken with an open tool on one user’s real work\. The figures in this half are read from the same single\-user author corpus characterized in Section[6](https://arxiv.org/html/2608.05784#S6), at a later freeze: Section[6](https://arxiv.org/html/2608.05784#S6)’s systems numbers use the 2026\-07\-10 freeze \(6161days,4646active,109,735109\{,\}735frames\), while the overhead and reproducibility numbers here use the freeze extended through 2026\-07\-22 \(5151active days,128,756128\{,\}756frames,232,898232\{,\}898input events\)\. Between the two freezes the accessibility\-tree coverage rose from about41%41\\%to81\.5%81\.5\\%of frames, which strengthens the structured grounding available to replay\.

### 7\.1Definition

A*routine*is a frequent actionnn\-gram: a contiguous sequence ofkkUI actions \(3≤k≤603\\leq k\\leq 60\) that recurs at least three times within the corpus, restricted to sequences that name at least two distinct targets\. Sessions are cut at inter\-action gaps over9090s\. The two\-named\-target restriction is declared in advance, not tuned after the fact: without it, degenerate single\-symbol repeats \(a held key, a scroll loop\) would satisfy “recurs” trivially and inflate every ratio, the granularity trap that any repetition metric on raw input invites\. It is the same specificity that separates the delegable ratehhfrom the raw rate in Section[7\.4](https://arxiv.org/html/2608.05784#S7.SS4)\.

For a routine ofkksteps we compare two token costs\. The numerator ismodeled\. A memoryless, screenshot\-driven agent re\-derives the routine one step at a time, and per step consumes one screenshot, a fixed context read, and a fixed reasoning write:

Cagent​\(k\)=k​\(w​h750\+350\+180\),C\_\{\\text\{agent\}\}\(k\)\\;=\\;k\\left\(\\frac\{w\\,h\}\{750\}\\;\+\\;350\\;\+\\;180\\right\),\(2\)the three per\-step terms being the screenshot priced by Anthropic’sw​h/750wh/750image\-token rule for aw×hw\\times hcapture, a350350\-token context read, and180180tokens of reasoning output\. At a typical1512×9821512\\times 982capture this is about2,5002\{,\}500tokens per step \(1,9791\{,\}979image\+350\+180\+\\,350\\,\+\\,180\), dominated by the screenshot\. We are explicit thatCagentC\_\{\\text\{agent\}\}is what published screenshot\-driven agent loops*would*spend, computed without executing an agent, and that it is an*upper bound*: it assumes a screenshot baseline with no cross\-step prompt caching, so an agent that reuses context across steps or grounds on the accessibility tree instead of a fresh screenshot spends fewer tokens per step andRRnarrows accordingly\. Section[8\.4](https://arxiv.org/html/2608.05784#S8.SS4)reports a live in\-loop comparison against an accessibility\-tree agent that saved only≈14%\{\\approx\}14\\%, and reserves the full live three\-arm billing\. The denominator ismeasured, and we report it as a*ladder*rather than one figure, because the compiler can emit the routine at more than one level of detail, each a legitimate reading ofCreplay​\(k\)C\_\{\\text\{replay\}\}\(k\):

R=Cagent​\(k\)Creplay​\(k\),Creplay​\(k\)=\|tiktoken​\(⋅\)\|\.R=\\frac\{C\_\{\\text\{agent\}\}\(k\)\}\{C\_\{\\text\{replay\}\}\(k\)\},\\quad C\_\{\\text\{replay\}\}\(k\)=\\bigl\|\\texttt\{tiktoken\}\(\\cdot\)\\bigr\|\.\(3\)The compiler emits both rungs deterministically, with no model in the loop \(compile\_replay\.py, released with the reference implementation\[[14](https://arxiv.org/html/2608.05784#bib.bib16)\]\); tokens are counted withtiktokenundercl100k\_base, the paper’s encoding\. The*operational*rung is the full*guarded skill plan*: a per\-step sequence of typed actions, each carrying an expected\-element, expected\-role, and expected\-application guard so replay can fail safe\. Across the2020most frequent action routines these plans have a median of247\.5247\.5tokens and compile in a median of0\.50\.5ms at zero token cost, the compiler being CPU\-only\. The*ceiling*rung is the*minimal replay script*that strips the guards to the routine’s bare information content: a median of4040tokens for a median55\-step routine, about88tokens per step\. The guarded plan is roughly six times larger, which is precisely why it yields the smaller, more defensible ratio we lead with\.

### 7\.2The Ladder, and Its Distribution

We lead with the*conservative operational*number and keep the larger one as a ceiling, because leading with the ceiling would invite a cherry\-picking objection\. Against the guarded skill plan the compiler actually emits, the median isRinject=60×R\_\{\\text\{inject\}\}=60\\times\(interquartile5959–6262across the2020compiled plans\): injecting the compiled routine costs about one\-sixtieth of re\-deriving it from screenshots\. Against the minimal script, the median reaches the information\-content ceilingRinfo=343×R\_\{\\text\{info\}\}=343\\timesat action granularity, the largest ratio the routine’s bare description length can justify\. Table[4](https://arxiv.org/html/2608.05784#S7.T4)reports both rungs and, forRinfoR\_\{\\text\{info\}\}, the full distribution over routines\. The ceiling is robust to the two knobs most likely to be accused of driving it: it rises monotonically but mildly with routine length \(median301×301\\timesatk=3k\{=\}3–44,358×358\\timesatk=5k\{=\}5–88,405×405\\timesatk=9k\{=\}9–1515,401×401\\timesatk≥16k\{\\geq\}16\), and re\-pricing the numerator at three screen configurations moves it only within a259×259\\times\(conservative1280×8001280\\times 800\) to425×425\\times\(retina1728×11171728\\times 1117\) band, so it is an artifact of neither short routines nor one display\. At URL granularity the ceiling median is198×198\\times\. The most frequent action routine \(a three\-step compose\-message loop,181181occurrences\) and the most frequent URL routine \(a mail\-calendar\-mail triangle,2525occurrences\) are the mundane recurrences one would expect of real desk work, not exotic macros\. We note the concentration plainly: the2020action routines behindRinjectR\_\{\\text\{inject\}\}are dominated by two families, a compose\-message loop and a window\-close loop, sliced at several lengths by thenn\-gram miner, soRinjectR\_\{\\text\{inject\}\}characterizes this user’s highest\-frequency micro\-routines rather than a diverse cross\-application task set; Section[8\.4](https://arxiv.org/html/2608.05784#S8.SS4)bounds the generalization\.

Table 4:The denominator ladder forR=Cagent/CreplayR=C\_\{\\text\{agent\}\}/C\_\{\\text\{replay\}\}\(numerator modeled; denominators measured withcl100k\_base\)\.RinjectR\_\{\\text\{inject\}\}\(operational\) uses the guarded skill plan the compiler emits;RinfoR\_\{\\text\{info\}\}\(ceiling\) uses the minimal replay script\.RinjectR\_\{\\text\{inject\}\}is computed on the2020most frequent action routines, so it carries no URL column\.RinfoR\_\{\\text\{info\}\}medians are priced at a typical1512×9821512\\times 982screen; the resolution band re\-prices the numerator at a conservative and a retina capture\.
### 7\.3Recovery, and a Modeled Three\-Arm Comparison

The recovery a hit yields depends on which rung of the ladder is used, and we attach the largest claim to the narrowest case\. On a guard\-matched step, deterministic*local*replay —*parametric routine replay*: the routine’s compiled structure replays deterministically while its variable slots carry the request’s new values, so the agent picks the routine and fills the slots but never re\-derives the steps — takes the model out of the loop entirely, so it recovers

1−1R≈99%1\-\\frac\{1\}\{R\}\\;\\approx\\;99\\%\(4\)of that step’s re\-derivation cost \(exactly99\.7%99\.7\\%at the median ceilingRinfo=343R\_\{\\text\{info\}\}=343\)\. That99%99\\%is a per\-covered\-step ceiling, not a fleet saving: local replay reaches it only on the fraction of steps the compiler can guard, whose measured median is0\.4150\.415, and it is exposed to interface drift that a live run must confirm \(Section[8\.4](https://arxiv.org/html/2608.05784#S8.SS4)\)\.

A modeled three\-arm comparison makes the gap between ceiling and realized saving concrete\. We price three ways of executing the2020recurring routines at Sonnet\-class list rates \($3/$15 per Mtok, output fraction0\.070\.07\),*modeled from measured artifacts and token counts, not billed*: arm A, a memoryless agent that re\-derives each routine; arm B, an agent given the compiled plan once and then acting with it in context; and arm C, deterministic local replay\. Relative to arm A \(32\.832\.8Mtok, $125\.81 over the fleet\), injecting the plan \(arm B\) saves83\.3%83\.3\\%of tokens \($20\.96, a6\.0×6\.0\\timesper\-occurrence reduction\), and local replay \(arm C\) saves40\.8%40\.8\\%\($74\.46,1\.7×1\.7\\times\)\. Two honesty notes\. Arm B’s modeled saving assumes the screenshot baseline of Eq\.[2](https://arxiv.org/html/2608.05784#S7.E2); the one live in\-loop comparison we have, against an accessibility\-tree agent, saved only≈14%\{\\approx\}14\\%\(Section[8\.4](https://arxiv.org/html/2608.05784#S8.SS4)\), so read83\.3%83\.3\\%as a modeled upper bound, not a measured in\-loop figure\. Arm C falls well short of the99%99\\%ceiling for a structural reason: it credits only the guard\-matched fraction, so the complementary1−0\.4151\-0\.415deopt fraction still pays full arm\-A price\. The live\-billed version of this table, with real usage JSON, is reserved \(Section[8\.4](https://arxiv.org/html/2608.05784#S8.SS4)\); we present it as modeled\.

Across*all*action steps rather than the recurring subset, the reachable saving is smaller still, because only a fractionhhof steps sit in a delegable routine\. The honest all\-fleet ceiling ish​\(1−1/Rinfo\)h\\,\(1\-1/R\_\{\\text\{info\}\}\): with the in\-sample delegable rateh=9\.0%h=9\.0\\%it is≈9\.0%\\approx 9\.0\\%, and with the conservative out\-of\-sample rateh=7\.7%h=7\.7\\%\(Section[7\.4](https://arxiv.org/html/2608.05784#S7.SS4)\) it is≈7\.7%\\approx 7\.7\\%\. Three disciplines keep every figure here honest\. First, no rung ofRRis ever multiplied by the86×86\\timescontext compression of Section[6](https://arxiv.org/html/2608.05784#S6): those are different tokens \(reading a day versus re\-deriving a routine\), and stacking them would double\-count the same work\. Second,RRis never combined with prompt\-cache or KV\-cache discounts; caching and replay are alternative recoveries of the same repetition, not additive ones\. Third, both the numerator and the three\-arm dollars are modeled, so we report them as ceilings and reserve the live billing\.

### 7\.4Desktop Routine Recurrencehh

The ratioRRprices a single hit; the recurrencehhis how often hits occur, measured as the fraction of action steps that fall inside a recurring routine\. A single number would mislead, so we reporthhat two levels\. The*raw*rate ishraw=83\.1%h\_\{\\text\{raw\}\}=83\.1\\%: the fraction of action steps inside*any*recurringnn\-gram\. This is dominated by the generic micro\-structure of input, the type\-a\-character, return\-to\-field, type\-again texture that repeats constantly and carries no reusable work\. The*delegable*rate ishspecific=9\.0%h\_\{\\text\{specific\}\}=9\.0\\%at action granularity \(13\.1%13\.1\\%at URL granularity\): the fraction of steps inside the≥2\\geq 2\-named\-target routines of Section[7](https://arxiv.org/html/2608.05784#S7), that is, inside an identifiable, repeatable task\. We report the gap83\.1%→9\.0%83\.1\\%\\rightarrow 9\.0\\%openly rather than quoting the larger figure: it is the difference between “the keyboard repeats” and “the work repeats,” and only the latter is a candidate for delegation\.

#### A temporal holdout \(the single predictive claim\)

A within\-sample recurrence rate is partly circular, since a routine is counted as recurring in part because we already watched it recur\. To obtain a non\-circular value we fit the routine table on the first4040active days \(4,8474\{,\}847routine signatures\) and then ask what fraction of action steps on the held\-out final1111days fall into a routine*already known*from the training window\. The out\-of\-sample predicted hit rate is7\.7%7\.7\\%, against an in\-sample8\.6%8\.6\\%on the training days themselves\. The modest8\.6%→7\.7%8\.6\\%\\rightarrow 7\.7\\%drop is the within\-user temporal\-drift gap \(a same\-user, later\-in\-time split, not a cross\-population holdout\), and7\.7%7\.7\\%is the recurrence the cost accounting should carry; the all\-fleet ceilingh​\(1−1/Rinfo\)≈7\.7%h\\,\(1\-1/R\_\{\\text\{info\}\}\)\\approx 7\.7\\%follows from it\. We make exactly one predictive claim, and this is it\.

#### Whathhis not

Web\-era studies of page revisitation report far larger constants, on the order of4040to58%58\\%of page visits being revisits\[[40](https://arxiv.org/html/2608.05784#bib.bib51),[1](https://arxiv.org/html/2608.05784#bib.bib52)\]\. We do not claim those ashh\. A page revisit is not a delegable task: back\-button noise, re\-checking a feed, and reopening a tab are revisits with no reusable work content, and importing that number would overstatehhby roughly an order of magnitude\. Ourhhmeasures recurring*task*structure in real desktop work, and on this corpus that quantity is≈0\.08\\approx 0\.08–0\.130\.13, not≈0\.5\\approx 0\.5\. Reporting the smaller, honest figure is the point of the specificity rule\.

### 7\.5Reproducibility as a Certified Property

Because compilation contains no model, reproducibility is not a behavior we hope for but a property we can certify\. A certification harness \(certify\_ivm\.py, released with the code\[[14](https://arxiv.org/html/2608.05784#bib.bib16)\]\) checks three conditions over the5151\-active\-day corpus and returnspass\. First,byte\-identical output: re\-compiling any day twice yields byte\-identical documents once the single emission\-metadata fieldgenerated\_at\(a wall\-clock stamp of the run\) is excluded\. We name that exclusion rather than quietly canonicalizing it away; it is the one field that is not a function of the capture\. Second,rebuild equals incremental: rebuilding the entire history from scratch produces the same bytes as compiling day\-by\-day, and earlier days are never rewritten by later capture \(append\-only\)\. Third,compile cost does not grow with history: the median full\-day compile across the run is220\.9220\.9ms, and the mean over the second half of the corpus is0\.86×0\.86\\timesthe mean over the first half \(216\.4216\.4ms versus251\.8251\.8ms\), so per\-day cost tracks the size of the day’s delta, not the length of accumulated history; it isO​\(\|Δ\|\)O\(\|\\Delta\|\)\. \(The6868ms of Section[6](https://arxiv.org/html/2608.05784#S6)is one representative lighter day; the certification spans light and heavy days, from0\.10\.1to930930ms\.\)

We claim exactly one thing from this, and concede its lineage\. The claim is a CI\-checkable byte\-equality contract on a stateless projection of an append\-only capture log\. We claim*no*novelty in incremental view maintenance: deterministic, incrementally maintainable views over append\-only logs are a mature area, with DBSP giving automatic incremental view maintenance for rich query languages\[[4](https://arxiv.org/html/2608.05784#bib.bib38)\], and event\-sourcing long rebuilding state as a fold over an immutable event log\. Our contribution is not the mechanism but its*use as a certified guarantee for agent memory*: an episodic\-memory artifact whose equality across runs, across code versions, and across rebuild strategies is enforced by a test, which is what makes the memory safe to cache and mechanically auditable in the sense Section[8](https://arxiv.org/html/2608.05784#S8)requires\.

## 8Privacy, Trust, and Limitations

### 8\.1Privacy Model

The entire pipeline is local: capture, storage, and compilation happen on the user’s machine, the compiler opens the capture database read\-only, and nothing is transmitted anywhere by the system itself\. The operator chooses what leaves, and when, by handing a compiled artifact to an agent\. The schema\-level content rule \(Section[3](https://arxiv.org/html/2608.05784#S3)\) keeps typed text out of documents by default; audio capture is off by default in the provisioned engine\. Compilation itself involves no model of any kind; the capture engine does run on\-device OCR to read screen content, but that output never leaves the machine, and no language model, local or remote, participates in producing memory\. The capture database itself remains sensitive at rest, as membership\-inference work on memory stores reminds us\[[5](https://arxiv.org/html/2608.05784#bib.bib10)\]; we treat device\-level encryption as the appropriate control and note that this risk is shared by every capture system rather than introduced by compilation\.

### 8\.2Trust Properties

Two structural properties address emerging attacks on agent memory\. Evidence pointers make every episode mechanically auditable: a verifier can re\-read the named raw rows and recompute the frame, which raises the bar for poisoning attacks that rely on unverifiable memories\[[23](https://arxiv.org/html/2608.05784#bib.bib9)\]\. The measured/inferred boundary ensures that even a compromised or careless tier\-2 tool cannot inject interpretation disguised as observation; consumers can always strip to the measured core\. We do not claim these properties defeat a compromised capture engine, which can fabricate raw rows; provenance begins at the database\.

### 8\.3Limitations

Four limitations bound our claims\. First, the empirical characterization is a single\-user corpus \(one professional, one machine, 61 days\); cadence, fragmentation, and entity coverage will differ across roles and platforms, and a multi\-user study is future work\. Second, the reference implementation reads one capture engine’s database layout; the schema is engine\-agnostic but each new source needs an adapter\. Third, the measured tier is structurally silent on intent: it reports two profile views and a people search, never “prospecting,” and consumers who need intent must add tier\-2 inference and accept its tags\. Fourth, screen presence is an imperfect proxy for attention, and on this corpus the gap is quantifiable rather than hypothetical\. The capture heartbeat keeps input\-free stretches credited: intervals that terminate in a heartbeat row carry 45% of all credited active time, and uninterrupted heartbeat runs reach 42 minutes, so dwell includes reading and watching but also time the user had stepped away from an awake display\. The dwell cap does not bound this; it only bounds credit across capture stalls and display sleep\. Frames report input volume precisely so consumers can gate on interaction, and a tier\-2 tool may label presence\-only stretches, but the measured tier reports tenure, not attention, and consumers must read it that way\. Monitors compound the proxy error in the other direction: each records its own stream, so a minute with two active monitors earns credit twice in per\-application ledgers \(27% of captured minutes here; Section[6\.2](https://arxiv.org/html/2608.05784#S6.SS2)\)\. Off\-screen work \(paper, phone, conversation\) appears only as gaps\. Relatedly, frames measure attention episodes, not tasks: interruption research shows interrupted tasks are resumed across much longer horizons than any single frame\[[13](https://arxiv.org/html/2608.05784#bib.bib29)\], so task\-level structure belongs to tier\-2 inference\. Finally, the downstream benchmark \(Section[6\.2](https://arxiv.org/html/2608.05784#S6.SS2)\) evaluates two agent tiers \(Claude Sonnet 4\.5 and Opus 4\.5\) on this single\-user corpus with one answering pass each\. A stronger model narrows the summary’s gap, so the magnitude of the effect is not model\-independent; but the block’s determinism, its context\-fit, and its being read equally well by both tiers are structural properties no model strength supplies, and we release the harness so the comparison can be rerun with confidence intervals, against other model families such as GPT and Gemini, and, as multi\-user capture becomes available, other users\.

### 8\.4Limitations of the Overhead Measurements

Five limitations bound the results of this half of the paper, beyond the systems limitations already stated in Section[8](https://arxiv.org/html/2608.05784#S8)\.

Single user\.Every number in Section[7](https://arxiv.org/html/2608.05784#S7)comes from one user’s machine, the author’s, over5151active days \(128,756128\{,\}756frames\)\. We state this plainly:RRandhhare properties of this person’s work, not of a population\. The specificity rule and the temporal holdout guard against granularity and circularity artifacts, but not against having sampled one professional on one platform\. A multi\-user replication, and a public\-dataset second subject, are the obvious next step and are not claimed here\.

The numerator and the three\-arm dollars are modeled, not billed\.RR’s numerator \(Eq\.[2](https://arxiv.org/html/2608.05784#S7.E2)\) is what a memoryless screenshot\-driven loop would spend under the Anthropic image\-token rule and fixed per\-step budgets, priced without running an agent, and the three\-arm comparison of Section[7](https://arxiv.org/html/2608.05784#S7)is likewise modeled from measured artifacts \(compiled plans, token counts, guard coverage\) at list prices, not billed\. The denominators themselves are real: the guarded plan and the minimal script are emitted and tokenized deterministically, and the median guard coverage of0\.4150\.415that bounds local replay \(arm C\) is measured, not assumed\. Since the freeze, one owner\-authorized live execution has been run, and we report it as a first confirmation of the replay side\. A compiled two\-step routine \(open a compose surface, type a draft; nothing was ever posted\) was matched to a natural\-language request by a small*local*model \(a Tier\-2 retrieval step we do not count in the execution total\) and then executed in a live, authenticated browser session by the released executor: both steps grounded by accessibility role\+name,*zero*model tokens*at execution*, a few seconds of wall\-clock\. The element references differed between two runs of the same plan and name\-based grounding adapted; on a wrong page the same plan grounded nothing and performed zero actions, the intended fail\-safe\. The accessibility snapshots an in\-loop agent would read to choose each action measured≈10\.5\{\\approx\}10\.5k tokens per step on the same pages; a separate live comparison that kept the model*in*the loop with the compiled plan as context saved only≈14%\{\\approx\}14\\%against an accessibility\-driven agent, so the large ratios of Section[7](https://arxiv.org/html/2608.05784#S7)require the model fully out of the loop, which parametric replay is\. Three bounds on this confirmation: it is one two\-step routine; its plan was seeded from live accessibility names standing in for a mined routine \(click\-level grounding of the recorder is still under validation\); and the guard\-miss deopt path was not exercised\. The full live three\-arm billing with real usage JSON remains reserved\. ReadRRas a modeled ratio whose denominator — including its zero\-token on\-hit case — is now live\-confirmed, and the dollar savings as a modeled ceiling awaiting live billing\.

Capture is not free\.The instrument has an operating cost, which we report rather than hide\. The corpus database is9\.59\.5GB, about0\.190\.19GB per active day; on\-device OCR runs on96%96\\%of frames, a continuous duty cycle; and the accessibility tree that grounds replay is present on81\.5%81\.5\\%of frames, so18\.5%18\.5\\%offer no structured target and would fall back to coordinates\. Replay coverage is therefore bounded by that81\.5%81\.5\\%, and the compression and QA wins of Section[6](https://arxiv.org/html/2608.05784#S6)are gross of the capture engine’s own footprint\.

Match precision propagates intoqq\.Turning a captured routine into a replayable script depends on entity typing, which is approximately82%82\\%accurate at assigning a non\-generic type \(Section[6](https://arxiv.org/html/2608.05784#S6)\)\. Typing errors propagate directly into the match\-precision termqqof Eq\.[1](https://arxiv.org/html/2608.05784#S2.E1): a mistyped target can match the wrong routine or fail to match a real one, soq<1q<1and the honest fleet saving carries that factor\. We do not assumeq=1q=1\.

OCR is a model, so determinism holds forward, not back to pixels\.The only learned component anywhere in the pipeline is the on\-device OCR that reads screen text at capture time\. Byte\-level reproducibility \(Section[7\.5](https://arxiv.org/html/2608.05784#S7.SS5)\) therefore holds from the*stored OCR text*forward: given the captured text, every downstream document is a deterministic function of it\. It does not hold back to the pixels, because a different OCR model, or a re\-scan of the same screenshots, could yield different text\. We scope the determinism claim to the compile path over stored capture, never to perception\.

## 9Conclusion

Agents are blind to the activity stream that most defines their user’s day, not because capture is missing but because nothing turns capture into memory an agent can afford, reproduce, and trust\. Activity frames fill that seam with the least interesting tool available, deterministic code, and we argue that this dullness is the point: at 68 ms and zero tokens per day, episodic memory becomes infrastructure rather than inference, and the measured/inferred boundary gives interpretation a place to live without contaminating fact\. The same deterministic compiler doubles as a demand\-side instrument: because it reduces recurring capture to a replayable script by code alone, it reads the cost parametersRRandhhthat agent\-cost models assume, here as first single\-user values, with a modeled numerator, awaiting the multi\-user replication and live billing we reserve\. The schema, compilation rules, and implementation are open\[[14](https://arxiv.org/html/2608.05784#bib.bib16)\]; we hope the format outlives the reference code, and that capture systems, memory layers, and agents converge on a shared, honest representation of what a person actually did\.

## References

- \[1\]E\. Adar, J\. Teevan, and S\. T\. Dumais\(2008\)Large scale analysis of web revisitation patterns\.InProceedings of the SIGCHI Conference on Human Factors in Computing Systems \(CHI\),pp\. 1197–1206\.Cited by:[§7\.4](https://arxiv.org/html/2608.05784#S7.SS4.SSS0.Px2.p1.8)\.
- \[2\]Anthropic\(2024\)Model Context Protocol\.Note:https://modelcontextprotocol\.ioOpen protocol specification, accessed 2026\-07\-04Cited by:[3rd item](https://arxiv.org/html/2608.05784#S1.I1.i3.p1.1),[§5](https://arxiv.org/html/2608.05784#S5.p1.1)\.
- \[3\]E\. Bjäreholt and J\. Nilsson\(2016\)ActivityWatch: open\-source automated time tracker\.Note:https://activitywatch\.netAccessed 2026\-07\-04Cited by:[§1](https://arxiv.org/html/2608.05784#S1.p2.4),[§2\.3](https://arxiv.org/html/2608.05784#S2.SS3.p1.2)\.
- \[4\]M\. Budiu, F\. McSherry, L\. Ryzhyk, and V\. Tannen\(2022\)DBSP: automatic incremental view maintenance for rich query languages\.External Links:2203\.16684,[Link](https://arxiv.org/abs/2203.16684)Cited by:[§7\.5](https://arxiv.org/html/2608.05784#S7.SS5.p2.1)\.
- \[5\]K\. Chen, Y\. Pang, and T\. Wang\(2026\)MRMMIA: membership inference attacks on memory in chat agents\.External Links:2605\.27825,[Link](https://arxiv.org/abs/2605.27825)Cited by:[§2\.3](https://arxiv.org/html/2608.05784#S2.SS3.p1.2),[§8\.1](https://arxiv.org/html/2608.05784#S8.SS1.p1.1)\.
- \[6\]L\. Chen, M\. Zaharia, and J\. Zou\(2023\)FrugalGPT: how to use large language models while reducing cost and improving performance\.External Links:2305\.05176,[Link](https://arxiv.org/abs/2305.05176)Cited by:[§2\.4](https://arxiv.org/html/2608.05784#S2.SS4.p1.3)\.
- \[7\]P\. Chhikara, D\. Khant, S\. Aryan, T\. Singh, and D\. Yadav\(2025\)Mem0: building production\-ready AI agents with scalable long\-term memory\.External Links:2504\.19413,[Link](https://arxiv.org/abs/2504.19413)Cited by:[§1](https://arxiv.org/html/2608.05784#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.05784#S2.SS1.p1.1)\.
- \[8\]R\. Cooley, B\. Mobasher, and J\. Srivastava\(1999\)Data preparation for mining world wide web browsing patterns\.Knowledge and Information Systems1\(1\),pp\. 5–32\.Cited by:[§2\.3](https://arxiv.org/html/2608.05784#S2.SS3.p1.2)\.
- \[9\]A\. R\. Doherty and A\. F\. Smeaton\(2008\)Automatically segmenting LifeLog data into events\.InProc\. WIAMIS,pp\. 20–23\.Cited by:[§2\.3](https://arxiv.org/html/2608.05784#S2.SS3.p1.2)\.
- \[10\]A\. N\. Dragunov, T\. G\. Dietterich, K\. Johnsrude, M\. McLaughlin, L\. Li, and J\. L\. Herlocker\(2005\)TaskTracer: a desktop environment to support multi\-tasking knowledge workers\.InProc\. IUI,pp\. 75–82\.Cited by:[§2\.3](https://arxiv.org/html/2608.05784#S2.SS3.p1.2)\.
- \[11\]M\. H\. Erol, B\. El, M\. Suzgun, M\. Yuksekgonul, and J\. Zou\(2025\)Cost\-of\-pass: an economic framework for evaluating language models\.External Links:2504\.13359,[Link](https://arxiv.org/abs/2504.13359)Cited by:[§2\.4](https://arxiv.org/html/2608.05784#S2.SS4.p1.3)\.
- \[12\]S\. Forouzandeh, W\. Peng, P\. Moradi, X\. Yu, and M\. Jalili\(2025\)Learning hierarchical procedural memory for LLM agents through bayesian selection and contrastive refinement\.External Links:2512\.18950,[Link](https://arxiv.org/abs/2512.18950)Cited by:[§2\.1](https://arxiv.org/html/2608.05784#S2.SS1.p1.1)\.
- \[13\]V\. M\. González and G\. Mark\(2004\)“Constant, constant, multi\-tasking craziness”: managing multiple working spheres\.InProc\. CHI,pp\. 113–120\.Cited by:[§2\.3](https://arxiv.org/html/2608.05784#S2.SS3.p1.2),[§6\.5](https://arxiv.org/html/2608.05784#S6.SS5.p1.1),[§8\.3](https://arxiv.org/html/2608.05784#S8.SS3.p1.1)\.
- \[14\]N\. Iyamu\(2026\)Activity\-frames: episodic memory for AI agents\.Note:https://github\.com/nossa\-y/activity\-framesMIT license, accessed 2026\-07\-04Cited by:[3rd item](https://arxiv.org/html/2608.05784#S1.I1.i3.p1.1),[§5](https://arxiv.org/html/2608.05784#S5.p1.1),[§7\.1](https://arxiv.org/html/2608.05784#S7.SS1.p2.19),[§7\.5](https://arxiv.org/html/2608.05784#S7.SS5.p1.9),[§9](https://arxiv.org/html/2608.05784#S9.p1.2)\.
- \[15\]S\. Kapoor, B\. Stroebl, P\. Kirgis, N\. Nadgir, Z\. S\. Siegel, B\. Wei, T\. Xue, Z\. Chen, F\. Chen, S\. Utpala, F\. Ndzomga, D\. Oruganty, S\. Luskin, K\. Liu, B\. Yu, A\. Arora, D\. Hahm, H\. Trivedi, H\. Sun, J\. Lee, T\. Jin, Y\. Mai, Y\. Zhou, Y\. Zhu, R\. Bommasani, D\. Kang, D\. Song, P\. Henderson, Y\. Su, P\. Liang, and A\. Narayanan\(2025\)Holistic agent leaderboard: the missing infrastructure for AI agent evaluation\.External Links:2510\.11977,[Link](https://arxiv.org/abs/2510.11977)Cited by:[§2\.4](https://arxiv.org/html/2608.05784#S2.SS4.p1.3)\.
- \[16\]S\. Kapoor, B\. Stroebl, Z\. S\. Siegel, N\. Nadgir, and A\. Narayanan\(2024\)AI agents that matter\.External Links:2407\.01502,[Link](https://arxiv.org/abs/2407.01502)Cited by:[§2\.4](https://arxiv.org/html/2608.05784#S2.SS4.p1.3)\.
- \[17\]V\. Leno, A\. Polyvyanyy, M\. Dumas, M\. La Rosa, and F\. M\. Maggi\(2021\)Robotic process mining: vision and challenges\.Business & Information Systems Engineering63\(3\),pp\. 301–314\.Cited by:[§2\.3](https://arxiv.org/html/2608.05784#S2.SS3.p1.2)\.
- \[18\]B\. Li\(2026\)PreAct: computer\-using agents that get faster on repeated tasks\.External Links:2606\.17929,[Link](https://arxiv.org/abs/2606.17929)Cited by:[§2\.4](https://arxiv.org/html/2608.05784#S2.SS4.p1.3)\.
- \[19\]J\. N\. Li, Z\. J\. Zhang, and J\. Ma\(2025\)OmniQuery: contextually augmenting captured multimodal memory to enable personal question answering\.External Links:2409\.08250,[Link](https://arxiv.org/abs/2409.08250)Cited by:[§2\.2](https://arxiv.org/html/2608.05784#S2.SS2.p1.1)\.
- \[20\]Y\. Li, H\. Wen, W\. Wang, X\. Li, Y\. Yuan, G\. Liu, J\. Liu, W\. Xu, X\. Wang, Y\. Sun, R\. Kong, Y\. Wang, H\. Geng, J\. Luan, X\. Jin, Z\. Ye, G\. Xiong, F\. Zhang, X\. Li, M\. Xu, Z\. Li, P\. Li, Y\. Liu, Y\. Zhang, and Y\. Liu\(2024\)Personal LLM agents: insights and survey about the capability, efficiency and security\.External Links:2401\.05459,[Link](https://arxiv.org/abs/2401.05459)Cited by:[§2\.3](https://arxiv.org/html/2608.05784#S2.SS3.p1.2)\.
- \[21\]G\. Liu, P\. Zhao, Y\. Liang, L\. Liu, Y\. Guo, H\. Xiao, W\. Lin, Y\. Chai, Y\. Han, S\. Ren, H\. Wang, X\. Liang, W\. Wang, T\. Wu, Z\. Lu, S\. Chen, LiLinghao, H\. Wang, G\. Xiong, Y\. Liu, and H\. Li\(2025\)LLM\-powered GUI agents in phone automation: surveying progress and prospects\.External Links:2504\.19838,[Link](https://arxiv.org/abs/2504.19838)Cited by:[§2\.3](https://arxiv.org/html/2608.05784#S2.SS3.p1.2)\.
- \[22\]N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. Liang\(2023\)Lost in the middle: how language models use long contexts\.External Links:2307\.03172,[Link](https://arxiv.org/abs/2307.03172)Cited by:[§1](https://arxiv.org/html/2608.05784#S1.p2.4),[§2\.3](https://arxiv.org/html/2608.05784#S2.SS3.p1.2),[§6\.1](https://arxiv.org/html/2608.05784#S6.SS1.p1.2)\.
- \[23\]Y\. Louck\(2026\)Securing LLM\-agent long\-term memory against poisoning: non\-malleable, origin\-bound authority with machine\-checked guarantees\.External Links:2606\.24322,[Link](https://arxiv.org/abs/2606.24322)Cited by:[§2\.3](https://arxiv.org/html/2608.05784#S2.SS3.p1.2),[§8\.2](https://arxiv.org/html/2608.05784#S8.SS2.p1.1)\.
- \[24\]L\. Mei, J\. Yao, Y\. Ge, Y\. Wang, B\. Bi, Y\. Cai, J\. Liu, M\. Li, Z\. Li, D\. Zhang, C\. Zhou, J\. Mao, T\. Xia, J\. Guo, and S\. Liu\(2025\)A survey of context engineering for large language models\.External Links:2507\.13334,[Link](https://arxiv.org/abs/2507.13334)Cited by:[§2\.3](https://arxiv.org/html/2608.05784#S2.SS3.p1.2)\.
- \[25\]Microsoft\(2025\)Recall: retrace your steps on Copilot\+ PCs\.Note:https://learn\.microsoft\.com/en\-us/windows/apps/develop/windows\-integration/recall/Accessed 2026\-07\-04Cited by:[§1](https://arxiv.org/html/2608.05784#S1.p2.4),[§2\.3](https://arxiv.org/html/2608.05784#S2.SS3.p1.2)\.
- \[26\]N\. Oliver, G\. Smith, C\. Thakkar, and A\. C\. Surendran\(2006\)SWISH: semantic analysis of window titles and switching history\.InProc\. IUI,pp\. 92–99\.Cited by:[§2\.3](https://arxiv.org/html/2608.05784#S2.SS3.p1.2)\.
- \[27\]OpenAI\(2026\)Chronicle: memories from recent screen content in Codex for macOS\.Note:https://developers\.openai\.com/codex/memories/chronicleResearch preview, accessed 2026\-07\-04Cited by:[§1](https://arxiv.org/html/2608.05784#S1.p2.4),[§2\.3](https://arxiv.org/html/2608.05784#S2.SS3.p1.2)\.
- \[28\]M\. Oyamada, K\. Takeoka, K\. Akimoto, R\. Obara, M\. Enomoto, H\. Zhang, D\. Haraguchi, and T\. Tamura\(2026\)Cotomi act: learning to automate work by watching you\.External Links:2605\.03231,[Link](https://arxiv.org/abs/2605.03231)Cited by:[§2\.4](https://arxiv.org/html/2608.05784#S2.SS4.p1.3)\.
- \[29\]C\. Packer, S\. Wooders, K\. Lin, V\. Fang, S\. G\. Patil, I\. Stoica, and J\. E\. Gonzalez\(2024\)MemGPT: towards LLMs as operating systems\.External Links:2310\.08560,[Link](https://arxiv.org/abs/2310.08560)Cited by:[§1](https://arxiv.org/html/2608.05784#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.05784#S2.SS1.p1.1)\.
- \[30\]V\. Pahuja, Y\. Lu, C\. Rosset, B\. Gou, A\. Mitra, S\. Whitehead, Y\. Su, and A\. Awadallah\(2025\)Explorer: scaling exploration\-driven web trajectory synthesis for multimodal web agents\.External Links:2502\.11357,[Link](https://arxiv.org/abs/2502.11357)Cited by:[§2\.4](https://arxiv.org/html/2608.05784#S2.SS4.p1.3)\.
- \[31\]Z\. Pan, Q\. Wu, H\. Jiang, X\. Luo, H\. Cheng, D\. Li, Y\. Yang, C\. Lin, H\. V\. Zhao, L\. Qiu, and J\. Gao\(2025\)On memory construction and retrieval for personalized conversational agents\.External Links:2502\.05589,[Link](https://arxiv.org/abs/2502.05589)Cited by:[§2\.1](https://arxiv.org/html/2608.05784#S2.SS1.p1.1)\.
- \[32\]J\. S\. Park, J\. C\. O’Brien, C\. J\. Cai, M\. R\. Morris, P\. Liang, and M\. S\. Bernstein\(2023\)Generative agents: interactive simulacra of human behavior\.External Links:2304\.03442,[Link](https://arxiv.org/abs/2304.03442)Cited by:[§2\.3](https://arxiv.org/html/2608.05784#S2.SS3.p1.2)\.
- \[33\]M\. Pink, Q\. Wu, V\. A\. Vo, J\. Turek, J\. Mu, A\. Huth, and M\. Toneva\(2025\)Position: episodic memory is the missing piece for long\-term LLM agents\.External Links:2502\.06975,[Link](https://arxiv.org/abs/2502.06975)Cited by:[§1](https://arxiv.org/html/2608.05784#S1.p1.1),[§2\.2](https://arxiv.org/html/2608.05784#S2.SS2.p1.1)\.
- \[34\]F\. Portet, E\. Reiter, A\. Gatt, J\. Hunter, S\. Sripada, Y\. Freer, and C\. Sykes\(2009\)Automatic generation of textual summaries from neonatal intensive care data\.Artificial Intelligence173\(7–8\),pp\. 789–816\.Cited by:[§2\.3](https://arxiv.org/html/2608.05784#S2.SS3.p1.2)\.
- \[35\]P\. Rasmussen, P\. Paliychuk, T\. Beauvais, J\. Ryan, and D\. Chalef\(2025\)Zep: a temporal knowledge graph architecture for agent memory\.External Links:2501\.13956,[Link](https://arxiv.org/abs/2501.13956)Cited by:[§1](https://arxiv.org/html/2608.05784#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.05784#S2.SS1.p1.1)\.
- \[36\]A\. J\. Sellen and S\. Whittaker\(2010\)Beyond total capture: a constructive critique of lifelogging\.Communications of the ACM53\(5\),pp\. 70–77\.Cited by:[§2\.3](https://arxiv.org/html/2608.05784#S2.SS3.p1.2)\.
- \[37\]C\. H\. Song, Y\. Song, P\. Goyal, Y\. Su, O\. Riva, H\. Palangi, and T\. Pfister\(2025\)Watch and learn: learning to use computers from online videos\.External Links:2510\.04673,[Link](https://arxiv.org/abs/2510.04673)Cited by:[§2\.4](https://arxiv.org/html/2608.05784#S2.SS4.p1.3)\.
- \[38\]Y\. Song, K\. Ramaneti, Z\. Sheikh,et al\.\(2025\)Agent data protocol: unifying datasets for diverse, effective fine\-tuning of LLM agents\.External Links:2510\.24702,[Link](https://arxiv.org/abs/2510.24702)Cited by:[§2\.4](https://arxiv.org/html/2608.05784#S2.SS4.p1.3)\.
- \[39\]Y\. Tang, H\. Tang, T\. Cao, L\. Nguyen, A\. Zhang, X\. Cao, C\. Liu, W\. Ding, and Y\. Li\(2026\)ProAgentBench: evaluating llm agents for proactive assistance with real\-world data\.External Links:2602\.04482,[Link](https://arxiv.org/abs/2602.04482)Cited by:[§2\.2](https://arxiv.org/html/2608.05784#S2.SS2.p1.1)\.
- \[40\]L\. Tauscher and S\. Greenberg\(1997\)How people revisit web pages: empirical findings and implications for the design of history systems\.International Journal of Human\-Computer Studies47\(1\),pp\. 97–137\.Cited by:[§7\.4](https://arxiv.org/html/2608.05784#S7.SS4.SSS0.Px2.p1.8)\.
- \[41\]S\. J\. van Zelst, F\. Mannhardt, M\. de Leoni, and A\. Koschmider\(2021\)Event abstraction in process mining: literature review and taxonomy\.Granular Computing6,pp\. 719–736\.Cited by:[§2\.3](https://arxiv.org/html/2608.05784#S2.SS3.p1.2)\.
- \[42\]K\. Wang, Y\. Lin, J\. Lou, Z\. Zhou, B\. Suvonov, and J\. Li\(2026\)E\-mem: multi\-agent based episodic context reconstruction for LLM agent memory\.External Links:2601\.21714,[Link](https://arxiv.org/abs/2601.21714)Cited by:[§2\.1](https://arxiv.org/html/2608.05784#S2.SS1.p1.1)\.
- \[43\]Y\. Wang and X\. Chen\(2025\)MIRIX: multi\-agent memory system for llm\-based agents\.External Links:2507\.07957,[Link](https://arxiv.org/abs/2507.07957)Cited by:[§2\.2](https://arxiv.org/html/2608.05784#S2.SS2.p1.1)\.
- \[44\]Z\. Z\. Wang, A\. Gandhi, G\. Neubig,et al\.\(2025\)Inducing programmatic skills for agentic tasks\.External Links:2504\.06821,[Link](https://arxiv.org/abs/2504.06821)Cited by:[§2\.4](https://arxiv.org/html/2608.05784#S2.SS4.p1.3)\.
- \[45\]Z\. Z\. Wang, J\. Mao, D\. Fried, and G\. Neubig\(2024\)Agent workflow memory\.External Links:2409\.07429,[Link](https://arxiv.org/abs/2409.07429)Cited by:[§2\.1](https://arxiv.org/html/2608.05784#S2.SS1.p1.1),[§2\.4](https://arxiv.org/html/2608.05784#S2.SS4.p1.3)\.
- \[46\]Y\. Xu, D\. Lu, Z\. Shen, J\. Wang, Z\. Wang, Y\. Mao, C\. Xiong, and T\. Yu\(2024\)AgentTrek: agent trajectory synthesis via guiding replay with web tutorials\.External Links:2412\.09605,[Link](https://arxiv.org/abs/2412.09605)Cited by:[§2\.4](https://arxiv.org/html/2608.05784#S2.SS4.p1.3)\.
- \[47\]L\. Yeykelis, J\. J\. Cummings, and B\. Reeves\(2014\)Multitasking on a single device: arousal and the frequency, anticipation, and prediction of switching between media content on a computer\.Journal of Communication64\(1\),pp\. 167–192\.Cited by:[§2\.3](https://arxiv.org/html/2608.05784#S2.SS3.p1.2),[§6\.5](https://arxiv.org/html/2608.05784#S6.SS5.p1.1)\.
- \[48\]H\. Yin, Z\. Wen, J\. Cao, B\. Yuan, and R\. Yang\(2026\)FOCAL: filtered on\-device continuous activity logging for efficient personal desktop summarization\.External Links:2604\.19541,[Link](https://arxiv.org/abs/2604.19541)Cited by:[§2\.2](https://arxiv.org/html/2608.05784#S2.SS2.p1.1)\.
- \[49\]C\. Zhang, S\. He, J\. Qian, B\. Li, L\. Li, S\. Qin, Y\. Kang, M\. Ma, G\. Liu, Q\. Lin, S\. Rajmohan, D\. Zhang, and Q\. Zhang\(2025\)Large language model\-brained GUI agents: a survey\.External Links:2411\.18279,[Link](https://arxiv.org/abs/2411.18279)Cited by:[§2\.3](https://arxiv.org/html/2608.05784#S2.SS3.p1.2)\.
- \[50\]G\. Zhang, M\. Ahmed, Z\. Hu, and A\. Bulling\(2024\)SummAct: uncovering user intentions through interactive behaviour summarisation\.External Links:2410\.08356,[Link](https://arxiv.org/abs/2410.08356)Cited by:[§2\.2](https://arxiv.org/html/2608.05784#S2.SS2.p1.1)\.
- \[51\]B\. Zheng, M\. Y\. Fatemi, X\. Jin,et al\.\(2025\)SkillWeaver: web agents can self\-improve by discovering and honing skills\.External Links:2504\.07079,[Link](https://arxiv.org/abs/2504.07079)Cited by:[§2\.4](https://arxiv.org/html/2608.05784#S2.SS4.p1.3)\.
- \[52\]C\. Zhou, H\. Chai, W\. Chen, Z\. Guo, R\. Shan, Y\. Song, T\. Xu, Y\. Yang, A\. Yu, W\. Zhang, C\. Zheng, J\. Zhu, Z\. Zheng, Z\. Zhang, X\. Lou, C\. Zhang, Z\. Fu, J\. Wang, W\. Liu, J\. Lin, and W\. Zhang\(2026\)Externalization in LLM agents: a unified review of memory, skills, protocols and harness engineering\.External Links:2604\.08224,[Link](https://arxiv.org/abs/2604.08224)Cited by:[§1](https://arxiv.org/html/2608.05784#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.05784#S2.SS1.p1.1)\.