Toward User-Conditioned Evaluation of Personal LLM Agents under Temporal Interventions

# Toward User-Conditioned Evaluation of Personal LLM Agents under Temporal Interventions
Source: [https://arxiv.org/html/2607.21635](https://arxiv.org/html/2607.21635)
###### Abstract\.

Personal agents maintain memories, learned skills, tool configurations, and policy state that evolve with each user\. Existing agent benchmarks often evaluate these capabilities in isolation: tool benchmarks test invocation under fixed APIs, memory benchmarks test recall or forgetting, and safety benchmarks test static policy compliance\. We argue that personal\-agent evaluation requires a different protocol: replaying the same temporal intervention across different persistent user\-conditioned states and measuring how failures propagate across agent components\. We formalize this requirement as four conditions: explicit temporal intervention, persistent state across the intervention, induced cross\-dimensional effects, and variation in user\-conditioned state\. A focused audit of public benchmark protocols selected by explicit inclusion criteria identifies several close cases\. Under our explicitly narrow operationalization, we did not find a protocol in that audited set satisfying all four conditions\. This claim is scoped as a focused gap analysis with bounded literature coverage\. This position paper proposes a minimal benchmark design and candidate reporting metrics for user\-conditioned adaptation\. The result is a concrete design requirement for future personal\-agent evaluation, with metrics used as reporting tools for that requirement\.

## 1\.Introduction

Recent software\-engineering evidence makes interface drift a practical agent risk beyond controlled perturbation\. Zhu et al\. study 998 bug reports from modern LLM\-agent frameworks and find that API Misuse \(32\.97%\) and API Incompatibility \(22\.34%\) together account for over half of reported failures; the common symptoms are Functional Error, Crash, and Build Failure, concentrated in the execution\-heavy Self\-Action stage\(Zhuet al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib45)\)\. Breaking changes can also be introduced by agents themselves\. Ferdous et al\. compare 7,191 agent\-generated pull requests with 1,402 human\-authored pull requests; agentic changes have a lower overall breaking\-change rate \(3\.45% vs\. 7\.40%\), yet maintenance\-oriented refactor and chore tasks still produce backward\-incompatible changes at 6\.72% and 9\.35%, respectively\(Ferdouset al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib46)\)\. These studies leave personalization outside scope\. They still locate the pressure point: evolving interfaces, stale assumptions, and repair code that fails to reason about downstream dependencies\.

*Personal intelligence*adds persistent user state to that failure mode\. General\-purpose assistants are usually evaluated on standardized benchmarks\. Personal agents maintain long\-term memory of user preferences and personalized interaction context\(Packeret al\.,[2024](https://arxiv.org/html/2607.21635#bib.bib21); Du,[2026](https://arxiv.org/html/2607.21635#bib.bib11); Liet al\.,[2026b](https://arxiv.org/html/2607.21635#bib.bib47),[a](https://arxiv.org/html/2607.21635#bib.bib56)\), acquire skills tailored to individual workflows\(Yanget al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib17); Xu and Yan,[2026](https://arxiv.org/html/2607.21635#bib.bib14)\), invoke external tools that evolve independently\(Guoet al\.,[2024](https://arxiv.org/html/2607.21635#bib.bib9); Qinet al\.,[2024](https://arxiv.org/html/2607.21635#bib.bib10)\), and comply with safety and privacy policies that tighten over time\(Yuet al\.,[2025](https://arxiv.org/html/2607.21635#bib.bib18); Razaet al\.,[2025](https://arxiv.org/html/2607.21635#bib.bib20)\)\. We focus on deployed agent systems whose backbone LLM may remain frozen; adaptation occurs through online context, retrieval\-augmented memory\(Chenget al\.,[2026a](https://arxiv.org/html/2607.21635#bib.bib50)\), tool configuration, policy state, and external skill libraries\. Weight updates are optional\.

These benchmarks target different objects of evaluation\. MemoryAgentBench\(Huet al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib5)\)and MemBench\(Tanet al\.,[2025](https://arxiv.org/html/2607.21635#bib.bib6)\)ask whether the agent can remember, update, or forget the relevant fact\. SkillLearnBench\(Zhonget al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib16)\)asks whether reusable procedures can be learned\. StableToolBench\(Guoet al\.,[2024](https://arxiv.org/html/2607.21635#bib.bib9)\)and BFCL\(Patilet al\.,[2025](https://arxiv.org/html/2607.21635#bib.bib27)\)stress invocation accuracy under fixed API specifications\. Our question is whether a remembered fact, once stale, becomes an executable dependency that breaks a later tool call, learned skill, or policy decision\.

Consider the same API schema update applied to two users\. A light user with no tool\-dependent skills may recover after one failed call\. A power user with stale API memories and learned reporting skills may regress across unrelated analysis workflows\. The event is identical, yet the propagation boundary is conditioned on persistent user state\. We call this*user\-conditioned evaluation*: measuring adaptation quality asQ​\(𝒜,ei∣uj\)Q\(\\mathcal\{A\},e\_\{i\}\\mid u\_\{j\}\)for a change eventeie\_\{i\}and user\-conditioned stateuju\_\{j\}\(persistent user\-specific context\), withQ​\(𝒜,ei\)Q\(\\mathcal\{A\},e\_\{i\}\)over a generic test population treated as insufficient\. This differs from standard per\-user prediction in recommender systems because the user state is executable context: it can contain stale memories, obsolete skills, tool permissions, and policy constraints that shape how failures propagate\.

Contributions\.We make three contributions to the design of personal\-agent benchmarks:

This position paper treats C1–C4 as a benchmark\-design requirement for user\-conditioned temporal interventions\.

1. \(1\)Problem formulation:user\-conditioned evaluationQ​\(𝒜,ei∣uj\)Q\(\\mathcal\{A\},e\_\{i\}\\mid u\_\{j\}\), where persistent user state acts as executable context in addition to serving as input\.
2. \(2\)Design requirement:C1–C4 for fixed temporal interventions across varied persistent user states\. A focused audit, summarized in the main paper and documented in Appendix[A](https://arxiv.org/html/2607.21635#A1), supports this requirement\.
3. \(3\)Benchmark design:a minimal task\-family design and candidate metrics for reporting adaptation quality\.

Figure[1](https://arxiv.org/html/2607.21635#S1.F1)provides the broader landscape of benchmarks and systems related to adaptive agent evaluation, with two surveys shown as search seeds\. The concentration of work within single dimensions, plus the 3/15 near\-miss pattern for temporal persistent state, summarizes the unfinished evaluation loop we address\.

![Refer to caption](https://arxiv.org/html/2607.21635v1/x1.png)Figure 1\.Broader landscape of 24 unique benchmarks/systems related to adaptive agent evaluation, organized by adaptation dimension \(rows\) and first public year \(columns\), with two surveys shown as search seeds\. Systems may appear in multiple rows when relevant to multiple adaptation dimensions; the 24 count is over unique systems\. Solid nodes are coded benchmark protocols; dashed nodes are landscape\-only systems, methods, or adjacent protocols\. References report publication venue/status when available\.A scatter\-style timeline chart showing 24 benchmarks and systems distributed across five horizontal rows \(D1: API/Tool through D5: User/Context\) and four year columns \(2023–2026\), plus two surveys as search seeds\. Work clusters within individual adaptation dimensions; under the paper’s coding protocol, fixed\-intervention cross\-dimensional adaptation is unobserved in the audited set\.Recent agent\-evaluation surveys\(Yehudaiet al\.,[2025](https://arxiv.org/html/2607.21635#bib.bib1); Mohammadiet al\.,[2025](https://arxiv.org/html/2607.21635#bib.bib2)\)catalog what exists\. Our audit asks what existing protocols still cannot measure\. Memory surveys\(Du,[2026](https://arxiv.org/html/2607.21635#bib.bib11)\)and lifelong/skill\-learning surveys\(Zhenget al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib12); Xu and Yan,[2026](https://arxiv.org/html/2607.21635#bib.bib14); Gaoet al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib13)\)provide adjacent context; our focus is specifically on evaluating adaptation across multiple agent components under user\-conditioned state\.

## 2\.Personal Agents and Continual Adaptation

We model a personal AI agent as a tuple𝒜=\(ℳ,𝒯,𝒦,𝒮,𝒞\)\\mathcal\{A\}=\(\\mathcal\{M\},\\mathcal\{T\},\\mathcal\{K\},\\mathcal\{S\},\\mathcal\{C\}\): a core model, external tools, user\-conditioned memory, learned skills, and safety/policy constraints\. Each component can change independently: tools update, memories accumulate and become stale, skills are added or deprecated, policies tighten, and user tasks drift\. We focus on deployed systems whose backbone LLM may remain frozen; adaptation occurs through online context, retrieval, tool configuration, policy state, and external skill libraries\.

This setting differs from snapshot evaluation\. Tool benchmarks expose API instability and function\-calling errors\(Guoet al\.,[2024](https://arxiv.org/html/2607.21635#bib.bib9); Patilet al\.,[2025](https://arxiv.org/html/2607.21635#bib.bib27)\); memory benchmarks test recall and forgetting\(Huet al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib5); Tanet al\.,[2025](https://arxiv.org/html/2607.21635#bib.bib6); Heet al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib22)\); skill work studies reusable procedures\(Xu and Yan,[2026](https://arxiv.org/html/2607.21635#bib.bib14); Zhonget al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib16)\); and safety benchmarks test harmful or policy\-violating actions\(Zhanget al\.,[2024](https://arxiv.org/html/2607.21635#bib.bib26); Andriushchenkoet al\.,[2025](https://arxiv.org/html/2607.21635#bib.bib7)\)\. These lines are necessary\. Personal intelligence additionally requires asking whether change in one component propagates through the others for a particular user\.

Related fields offer partial analogues\. Recommender systems evaluate temporal user histories\(Koren,[2009](https://arxiv.org/html/2607.21635#bib.bib37); Kang and McAuley,[2018](https://arxiv.org/html/2607.21635#bib.bib39); Rajputet al\.,[2023](https://arxiv.org/html/2607.21635#bib.bib40)\); their usual target is next\-item prediction\. Our target is failure propagation through user history that functions as executable agent state across tools, skills, memories, and policies\. Continual learning and software regression testing inspire stability metrics\(Kirkpatricket al\.,[2017](https://arxiv.org/html/2607.21635#bib.bib43); Yoo and Harman,[2012](https://arxiv.org/html/2607.21635#bib.bib41)\)\. The failure mode here also includes stale prompt, memory, retrieval, tool, and skill state, beyond weight\-level catastrophic forgetting\. Personal agents lack a single global regression suite because expected behavior is conditioned on each user’s accumulated state\.

## 3\.Five Dimensions of Adaptation

We organize adaptation challenges along five dimensions, each corresponding to a component of the agent tuple\. We also code four evaluation aspects: E1 correctness, E2 adaptation timeliness, E3 safety/privacy preservation, and E4 stability or regression resistance\. Figure[2](https://arxiv.org/html/2607.21635#S3.F2)visualizes the taxonomy\.

![Refer to caption](https://arxiv.org/html/2607.21635v1/x2.png)Figure 2\.The5×45\\times 4evaluation matrix\. Each cell shows how many of 15 audited benchmarks have at least partial coverage of both the row dimension and column aspect\. Dark cells indicate adequate coverage \(≥\\geq3\); light cells indicate sparse coverage \(1–2\); white cells are complete gaps \(0\)\. Fixed\-intervention cross\-dimensional adaptation is unobserved in the audited set under C1–C4\.A five\-by\-four heatmap matrix with rows D1 Tools through D5 User and columns E1 Correctness through E4 Stability\. Cells are shaded by coverage level: green for 3 or more benchmarks, yellow for 1 to 2, gray for zero\. Coverage is concentrated in E1; timeliness, safety outside D4, and stability outside a few rows are sparse\.### 3\.1\.Dimensions

D1: Tool/API evolution\.APIs update versions, deprecate endpoints, alter schemas, and add tools\. StableToolBench\(Guoet al\.,[2024](https://arxiv.org/html/2607.21635#bib.bib9)\)improves benchmark stability through cached API execution, and evolving\-API studies show LLM degradation when APIs drift\(Ashiket al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib32)\)\. Recent bug audits make this a systems problem as well: API misuse and incompatibility dominate reported framework failures\(Zhuet al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib45)\)\. What remains untested is adaptation after a fixed tool change when user memory and learned tool\-use patterns still encode the old interface\(Xuet al\.,[2025](https://arxiv.org/html/2607.21635#bib.bib52)\)\.

D2: Memory dynamics\.Personal agents accumulate preferences, interaction history, and task facts\. MemoryAgentBench\(Huet al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib5)\), MemBench\(Tanet al\.,[2025](https://arxiv.org/html/2607.21635#bib.bib6)\), MemoryArena\(Heet al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib22)\), and CLIN\(Majumderet al\.,[2024](https://arxiv.org/html/2607.21635#bib.bib29)\)advance memory evaluation and persistent textual learning\. Memory remains largely the target capability; the downstream question is whether stale or expanded memories change agent behavior and break tools, skills, or policy compliance\(Liuet al\.,[2026a](https://arxiv.org/html/2607.21635#bib.bib53)\)\.

D3: Skill acquisition\.Agents learn reusable procedures from interaction and tool\-use supervision\(Xu and Yan,[2026](https://arxiv.org/html/2607.21635#bib.bib14); Jianget al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib15); Jiang and Ferraro,[2026](https://arxiv.org/html/2607.21635#bib.bib57)\)\. SkillLearnBench\(Zhonget al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib16)\)shows that continual skill generation remains brittle, and agent\-skill surveys report quality and security risks in self\-generated or community skills\. The evaluation question is whether learned skills remain valid as their dependent tools and policies evolve\.

D4: Safety/policy compliance\.Safety requirements change with regulation, organizational rules, contextual\-integrity constraints, and emerging threats\(Lanet al\.,[2025](https://arxiv.org/html/2607.21635#bib.bib55); Linet al\.,[2026a](https://arxiv.org/html/2607.21635#bib.bib58)\)\. Agent\-SafetyBench\(Zhanget al\.,[2024](https://arxiv.org/html/2607.21635#bib.bib26)\)finds weak static safety performance; adaptive personal agents add a harder question: whether new skills or memories remain compliant with updated user\- and jurisdiction\-specific constraints\.

D5: User context drift\.Users’ tasks, preferences, and expertise shift over time\. WAREX\(Karaet al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib23)\)and ReliabilityBench\(Gupta,[2026](https://arxiv.org/html/2607.21635#bib.bib30)\)expose reliability drops under perturbation and stress\. They leave open how different user\-conditioned states amplify or dampen the same change event\.

### 3\.2\.Cross\-Dimensional Interactions

The dimensions only matter if the benchmark tests their coupling\. Stale memories can encode old API schemas\. Skills can silently depend on deprecated tools\. A memory that was once safe to retain may violate a new privacy rule, and a newly learned skill may enable an action the policy has since restricted\. Because user\-conditioned state differs across users, the benchmark must vary profiles and measure how the same event changes the scope of impact\.

## 4\.Gap Analysis

Audit scope\.To check whether the proposed requirement is already covered, we conducted a focused benchmark\-protocol audit with bounded scope\. We screened arXiv, Semantic Scholar, ACM DL, ACL Anthology, and OpenReview with agent\-evaluation query families and citation tracking from recent surveys\(Yehudaiet al\.,[2025](https://arxiv.org/html/2607.21635#bib.bib1); Mohammadiet al\.,[2025](https://arxiv.org/html/2607.21635#bib.bib2); Du,[2026](https://arxiv.org/html/2607.21635#bib.bib11); Xu and Yan,[2026](https://arxiv.org/html/2607.21635#bib.bib14); Yuet al\.,[2025](https://arxiv.org/html/2607.21635#bib.bib18); Zhenget al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib12)\)\. Figure[1](https://arxiv.org/html/2607.21635#S1.F1)shows 24 unique benchmarks/systems plus two seed surveys\. Fifteen public protocols enter the C1–C4 denominator because they define an LLM\-agent task suite, provide a public evaluation procedure or metric, and target at least one adaptation dimension\. The denominator spans tool use\(Liet al\.,[2023](https://arxiv.org/html/2607.21635#bib.bib4); Guoet al\.,[2024](https://arxiv.org/html/2607.21635#bib.bib9); Patilet al\.,[2025](https://arxiv.org/html/2607.21635#bib.bib27); Luet al\.,[2025](https://arxiv.org/html/2607.21635#bib.bib28); Ruanet al\.,[2024](https://arxiv.org/html/2607.21635#bib.bib8)\), memory\(Huet al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib5); Tanet al\.,[2025](https://arxiv.org/html/2607.21635#bib.bib6); Heet al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib22)\), skill learning\(Zhonget al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib16)\), safety\(Zhanget al\.,[2024](https://arxiv.org/html/2607.21635#bib.bib26); Andriushchenkoet al\.,[2025](https://arxiv.org/html/2607.21635#bib.bib7)\), web interaction\(Lùet al\.,[2024](https://arxiv.org/html/2607.21635#bib.bib25); Liuet al\.,[2024](https://arxiv.org/html/2607.21635#bib.bib3)\), and reliability stress\(Karaet al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib23); Gupta,[2026](https://arxiv.org/html/2607.21635#bib.bib30)\)\. Appendix[A](https://arxiv.org/html/2607.21635#A1)gives the screened list, inclusion rationale, codebook, coding matrix, boundary cases, and coder agreement\.

Coding rules\.A dimension or aspect is coded as covered when the benchmark explicitly tests it as a primary evaluation target, partial when it is incidentally tested or only partially addressed, and absent otherwise\. Multiple task types alone are insufficient for cross\-dimensional coding: the protocol must perturb one adaptive component and measure the induced effect on another\.

Operational test\.C1–C4 isolates the personal\-agent failure mode:

- •C1:an explicit temporal change event perturbs one adaptation dimension;
- •C2:the agent carries persistent state across that change;
- •C3:the benchmark measures induced effects on another dimension;
- •C4:the protocol varies user\-conditioned state or measures how the same change behaves under different persistent contexts\.

The test is narrower than multi\-capability evaluation\. C4 is the key personal\-intelligence condition\. A benchmark that fixes user state can measure generic cross\-component robustness and still miss that the same API change can be harmless for a light user and disruptive for a power user with stale memories and learned skills\. A generic C1–C3 benchmark can measure whether an agent updates a broken tool call\. Personal intelligence also requires testing whether the same event creates different regressions across users whose accumulated memories, skills, permissions, and policies differ\. The event must be held fixed as accumulated user state varies\.

Findings\.The audit is best read as a narrow negative result\. Several protocols are close: ToolSandbox is stateful, MemoryArena is multi\-session,τ\\tau\-bench combines simulated users with domain APIs and policies, and WAREX/ReliabilityBench inject production\-like stress\(Luet al\.,[2025](https://arxiv.org/html/2607.21635#bib.bib28); Heet al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib22); Yaoet al\.,[2025](https://arxiv.org/html/2607.21635#bib.bib34); Karaet al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib23); Gupta,[2026](https://arxiv.org/html/2607.21635#bib.bib30)\)\. Within the 15\-protocol denominator and C1–C4 operationalization, we did not identify a protocol satisfying all four criteria\. The common pattern is target\-capability evaluation for memory, tools, skills, or safety\. Propagation of a fixed intervention through different users’ accumulated state is left unmeasured\. Two independent coders re\-coded the headline C1–C4/interaction labels from the written codebook and source papers; between\-coder mean agreement was \.947 and mean Gwet’s AC1 was \.898\. Full coding notes and disagreement handling are in Appendix[A](https://arxiv.org/html/2607.21635#A1)\.

## 5\.Metrics and Minimal Benchmark Design

The audit points to concrete reporting rules\. Table[1](https://arxiv.org/html/2607.21635#S5.T1)summarizes five metrics\. Reports should include denominators, uncertainty, parameter sensitivity, and judge prompts\. Benchmarks should report the sustained\-success threshold forαL\\alpha\_\{L\}, absolute rates alongside ratios forσ\\sigmaandκ\\kappa, and human\-audited agreement whenγ\\gammauses LLM\-as\-judge\(Liet al\.,[2025](https://arxiv.org/html/2607.21635#bib.bib54)\)\. Context\-dependent rubric and trace\-audit work also argue against hiding rubric choices or acknowledgment behavior inside a single scalar\(Lanet al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib62); Sunet al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib63)\)\. Scalar scores should not be compared across benchmarks unless event definitions, profile construction, regression\-suite denominators, and operating regimes are matched\(Wanget al\.,[2026a](https://arxiv.org/html/2607.21635#bib.bib64); Hanet al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib66); Wu,[2026](https://arxiv.org/html/2607.21635#bib.bib67)\)\.

Table 1\.Candidate metric definitions for user\-conditioned adaptation evaluation\.Executable definitions\.For eventeie\_\{i\}and user stateuju\_\{j\}, letz1,…,zTz\_\{1\},\\ldots,z\_\{T\}be pass/fail indicators for post\-change queries marked relevant toeie\_\{i\}\. With sustained\-success thresholdmm,αL=min⁡\{r:zr=⋯=zr\+m−1=1\}\\alpha\_\{L\}=\\min\\\{r:z\_\{r\}=\\cdots=z\_\{r\+m\-1\}=1\\\}; if no such run exists, reportT\+1T\+1or∞\\infty\. Count latency over relevant interactions, and reportmmandTT\. Forγ\\gamma, each fallback opportunity receives three binary fields: acknowledges inability or uncertainty, avoids unsupported claims, and gives a concrete next step\.γ\\gammais the fraction with all three fields equal to 1; LLM\-as\-judge use should report the judge prompt, judge model, calibration examples, and human\-audited agreement\. Forσ\\sigma, letSpreS\_\{\\mathrm\{pre\}\}andSpostS\_\{\\mathrm\{post\}\}be safety/privacy pass rates on matched policy checks before and after the event\. ReportSpreS\_\{\\mathrm\{pre\}\},SpostS\_\{\\mathrm\{post\}\},Δ​S\\Delta S, andSpost/SpreS\_\{\\mathrm\{post\}\}/S\_\{\\mathrm\{pre\}\}whenSpre\>0S\_\{\\mathrm\{pre\}\}\>0; ifSpre=0S\_\{\\mathrm\{pre\}\}=0, mark the ratio as N/A\. Forκ\\kappa, compare pre/post pass rates on dependency\-free checks active foruju\_\{j\}across dimensions outside the target ofeie\_\{i\}\. Report a mean preservation ratio only for checks with baseline above thresholdτ\\tau, plus absolute rates or log\-ratio variants when baselines are small\.κ\\kapparequires an explicit event–dependency graph\. Forρ\\rho, define the user\-specific regression suiteQ​\(uj\)Q\(u\_\{j\}\)and reportρ=\|\{q:q​passed before​ei∧q​failed after​ei\}\|/\|\{q:q​passed before​ei\}\|\\rho=\|\\\{q:q\\text\{ passed before \}e\_\{i\}\\land q\\text\{ failed after \}e\_\{i\}\\\}\|/\|\\\{q:q\\text\{ passed before \}e\_\{i\}\\\}\|\. The denominator is tasks previously solved for this specific user state, and confidence intervals are needed when\|Q​\(uj\)\|\|Q\(u\_\{j\}\)\|is small\.

Minimal benchmark design\.A minimal benchmark should choose tool\-rich domains, initialize several user\-conditioned states per domain, annotate event–dependency graphs, and apply the same change event across profiles\. A power\-user profile might contain memories tagged tostats\-api v2\.1, learned skills such asgenerate\_summary\_report v1, and a permissive old report\-retention policy\. A light\-user profile might have no tool\-dependent skills\. The same API or policy change can then be evaluated for direct recovery, graceful fallback, safety preservation, and regressions in unaffected tasks\.

Financial and reporting assistants are useful instances because retrieval, reranking, cross\-period disclosure context, and semi\-structured graph/document state can become part of a user’s executable profile\(Chenget al\.,[2026b](https://arxiv.org/html/2607.21635#bib.bib59); Liuet al\.,[2026b](https://arxiv.org/html/2607.21635#bib.bib60); Taoet al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib61)\)\.

Benchmark card\.Table[2](https://arxiv.org/html/2607.21635#S5.T2)gives one concrete analytics/reporting assistant instance\. The key design choice is to hold the event fixed across varied user\-conditioned states\. The light user has no stale dependencies\. The power user has memories, skills, and policy assumptions that can amplify the same change\.

Table 2\.Concrete C1–C4 benchmark card for one task family\.Concrete reactive failure walkthrough\.Interface incompatibility and maintenance changes are natural stressors for this benchmark design\(Zhuet al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib45); Ferdouset al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib46)\)\. Suppose a reimbursement agent callsv2/stats\-api, which changes from\{"retention\_days": 30\}to\{"report\_retention\_policy": "strict"\}\. A light user may simply hit a missing\-key error and stop\. A power user may carry a storedgenerate\_monthly\_report\(\)skill that computes retention logic from the old key; a repair loop can then synthesize invalid payloads, contaminate report tables, and reuse obsolete retention assumptions in later compliance checks\. Static success rate sees only endpoint recovery\.κ\\kapparecords whether unaffected report and memory checks remain stable;σ\\sigmarecords whether the updated retention constraint survives the repair attempt\. Appendix[B](https://arxiv.org/html/2607.21635#A2)gives metric computability examples from controlled traces\.

## 6\.Open Challenges and Conclusion

Open challenges\.Exhaustive memory–skill–tool–policy enumeration is infeasible, so benchmark instances will need bounded dependency graphs that emphasize high\-risk edges such as memory–tool and skill–safety interactions\. Safety also has to be rerun after adaptation: an agent certified before an update may become unsafe after acquiring a skill or accumulating sensitive memory\. Finally, controlled profiles are useful for isolating C1–C4 behavior\. The profile generator should eventually be checked against consented logs, replay data, and broader model coverage, especially in domains where user\-conditioned state is operational rather than cosmetic: research assistants may depend on a user’s taste, agenda, and novelty criteria\(Konget al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib48); Zhanget al\.,[2026](https://arxiv.org/html/2607.21635#bib.bib49)\), while financial agents may depend on risk tolerance, strategy calibration, and structural choice constraints\(Linet al\.,[2026b](https://arxiv.org/html/2607.21635#bib.bib51); Wanget al\.,[2026b](https://arxiv.org/html/2607.21635#bib.bib65)\)\.

Conclusion\.Personal\-intelligence evaluation should treat user\-conditioned state as part of the system under test\. Within our focused audit and C1–C4 operationalization, we did not identify a protocol in the audited set satisfying all four criteria\. We use C1–C4 as a falsifiable design requirement for benchmark construction\. A useful next benchmark should provide profile states, event scripts, dependency annotations, oracle checks, and per\-user regression suites, so that other groups can test whether the same update creates different failures for different users\.

## Appendix AAudit Details and Coding Matrix

### A\.1\.Protocol, Codebook, and Agreement

Audit protocol\.We searched arXiv, Semantic Scholar, ACM DL, ACL Anthology, and OpenReview using the query strings listed below, then cross\-checked backward and forward citations from the seed survey areas named in the main text\. We used the latest public arXiv, conference, journal, or OpenReview version available during the audit; row citations in Tables[5](https://arxiv.org/html/2607.21635#A1.T5)–LABEL:tab:c1c4\-codingidentify those sources\. The C1–C4 denominator is the 15 public benchmark protocols marked “coded” in Table[5](https://arxiv.org/html/2607.21635#A1.T5): each defines an LLM\-agent evaluation task suite, provides reproducible metrics or an executable procedure, and targets at least one adaptation dimension\. Figure[1](https://arxiv.org/html/2607.21635#S1.F1)also shows landscape\-only systems, methods, and adjacent protocols; these sit outside the 15\-protocol denominator\. The audit reports the screened candidate set used to construct the denominator, not raw search\-engine hit counts\.

Audit reproducibility details\.The screened candidate set comprises the 24 unique benchmarks/systems in Figure[1](https://arxiv.org/html/2607.21635#S1.F1)plus two shown seed surveys;τ\\tau\-bench, AgentEval, and ST\-WebAgentBench are discussed as adjacent boundary cases outside the 15\-protocol denominator\.

Table 3\.Audit protocol summary\.Codebook\.A “\+” dimension/aspect tag means the paper’s stated task design or metrics make that dimension/aspect a primary target; “∼\\sim” means it appears incidentally or in a subset; “–” means absent\. For C1–C4, Y requires explicit protocol evidence\. Dynamic dialogue counts as C1 only when the benchmark injects an exogenous temporal event\. Dialogue context counts as C2 only when persistent user, memory, skill, or policy state is carried across that event\. C3 requires measuring an effect on a different adaptation dimension, beyond dependency within the same workflow\. Inter\. is positive only when C1–C4 all hold\.

Coder agreement\.Two independent coders re\-coded C1, C2, C3, C4, and Inter\. for the 15 protocols\. They received the codebook, source papers or official pages, and blank coding forms; they did not receive the manuscript’s coded table, author labels, boundary rationales, or expected headline count\. They were external to the author team and knew the audit studied user\-conditioned adaptation, so the exercise is a reliability check\. Between\-coder mean agreement was \.947 and mean Gwet’s AC1 was \.898\. Boundary disagreements concerned temporal or persistent\-state calls and were adjudicated by revisiting the codebook; the final Inter\. conclusion did not change\. The lower AC1 for C1 reflects boundary disagreements over whether reliability perturbations or multi\-session progression count as temporal change\. We require an explicit exogenous event injected after initial state construction\.

Table 4\.Coder agreement on the headline C1–C4 and interaction labels\.Table 5\.Screened candidate list and denominator decisions\. The citation in each row identifies the public source/version used for the audit\. The final C1–C4 denominator is the 15 rows marked “coded\.” Years are first public years\.Table 6\.Boundary\-case analysis\. Each row cites the public source/version used for the boundary call\. “Extension needed” states what would need to be added for a C1–C4 benchmark\.

## Appendix BMetric Computability Examples

Synthetic metric computability example\.The main\-text example uses the samestats\-apischema change for two user states\. The light user has no stored tool\-dependent skill\. The power user has two stored skills that call the oldcorrelation\(x,y\)schema\. The API then requiresmethod="pearson"\. The event–dependency graph links direct calls to D1, stored report skills to D3, a neutral memory check to D2, and safety invariants to D4\. A rollout contains six target calls plus user\-specific regression checks\. The static policy never repairs stale calls; the reactive policy repairs after errors with moderate probability; the versioned policy notices version tags and repairs both direct calls and skills with higher probability\. Table[8](https://arxiv.org/html/2607.21635#A2.T8)reports 200 simulated rollouts per policy–user pair withm=3m=3andT=6T=6\.

Table 8\.Synthetic metric computability example for the same API change under two user states\. Fail@T is an auxiliary target\-failure rate at the horizon, not one of the five proposed metrics\.Hosted\-LLM parser/oracle compatibility check\.We also ran a stochastic structured\-trace check with DeepSeekdeepseek\-v4\-flash: temperature 0\.7, three profiles, two events, 20 trials per profile–event condition, and 1,320 trace rows\. The model emitted one JSON action per request; the oracle checked updated schema/policy satisfaction, unaffected tasks, safety/privacy constraints, parse failures, and unsupported actions\. In this run the oracle recorded 22 invalid\-JSON actions and 1 unsupported action; the hosted API did not expose a deterministic seed\.

One trace example\.For the schema\-change event, the event script changes thecorrelationtool fromcorrelation\(x,y\)tocorrelation\(x,y,method\)and setsmethod="pearson"as the required ordinary\-correlation argument\. The action schema is a JSON object with fieldstoolandargs\. One prompt was: “Call the correlation tool to compute ordinary correlation forx=\[1,2,3\],y=\[2,4,6\]\.” A typical first action settool=correlation, suppliedx,y, and omittedmethod; the oracle marked it asmissing\_methodwith the observation thatstats\-api v3requiresmethod="pearson"\. The next target call often suppliedx,y,method="pearson"and passed\. This single trace yields the post\-change target sequence prefix01and contributes toαL=2\\alpha\_\{L\}=2when the following target calls remain correct\. The scalar outputs are engineering diagnostics for parser/oracle compatibility\.

## References

- M\. Andriushchenko, A\. Souly, M\. Dziemian, D\. Duenas, M\. Lin, J\. Wang, D\. Hendrycks, A\. Zou, Z\. Kolter, M\. Fredrikson,et al\.\(2025\)AgentHarm: a benchmark for measuring harmfulness of LLM agents\.Note:International Conference on Learning RepresentationsCited by:[Table 5](https://arxiv.org/html/2607.21635#A1.T5.1.20.19.1.1.1),[§2](https://arxiv.org/html/2607.21635#S2.p2.1),[§4](https://arxiv.org/html/2607.21635#S4.p1.1)\.
- A\. N\. Ashik, S\. Wang, T\. Chen, M\. Asaduzzaman, and Y\. Tian \(2026\)When LLMs lag behind: knowledge conflicts from evolving APIs in code generation\.Note:arXiv preprint arXiv:2604\.09515Cited by:[§3\.1](https://arxiv.org/html/2607.21635#S3.SS1.p1.1)\.
- Z\. Cheng, L\. Lai, Y\. Liu, and Y\. Sun \(2026a\)Toward sustainable on\-device intelligence: a survey on energy\-efficient RAG systems with small language models\.Available at SSRN 6698538\.External Links:[Link](https://ssrn.com/abstract=6698538)Cited by:[§1](https://arxiv.org/html/2607.21635#S1.p2.1)\.
- Z\. Cheng, L\. Lai, and Y\. Liu \(2026b\)Resolving the robustness\-precision trade\-off in financial RAG through hybrid document\-routed retrieval\.External Links:2603\.26815,[Document](https://dx.doi.org/10.48550/arXiv.2603.26815),[Link](https://doi.org/10.48550/arXiv.2603.26815)Cited by:[§5](https://arxiv.org/html/2607.21635#S5.p4.1)\.
- P\. Du \(2026\)Memory for autonomous LLM agents: mechanisms, evaluation, and emerging frontiers\.Note:arXiv preprint arXiv:2603\.07670Cited by:[§1](https://arxiv.org/html/2607.21635#S1.p2.1),[§1](https://arxiv.org/html/2607.21635#S1.p9.1),[§4](https://arxiv.org/html/2607.21635#S4.p1.1)\.
- K\. M\. Ferdous, D\. Banik, K\. Chowdhury, and S\. I\. Shamim \(2026\)Safer builders, risky maintainers: a comparative study of breaking changes in human vs agentic PRs\.Note:Accepted at the 23rd International Conference on Mining Software Repositories \(MSR\)External Links:2603\.27524,[Document](https://dx.doi.org/10.48550/arXiv.2603.27524)Cited by:[§1](https://arxiv.org/html/2607.21635#S1.p1.1),[§5](https://arxiv.org/html/2607.21635#S5.p6.2)\.
- H\. Gao, J\. Geng, W\. Hua, M\. Hu, X\. Juan, H\. Liu, S\. Liu, J\. Qiu, X\. Qi, Q\. Ren, Y\. Wu, H\. Wang, H\. Xiao, Y\. Zhou, S\. Zhang, J\. Zhang, J\. Xiang, Y\. Fang, Q\. Zhao, D\. Liu, C\. Qian, Z\. Wang, M\. Hu, H\. Wang, Q\. Wu, H\. Ji, and M\. Wang \(2026\)A survey of self\-evolving agents: what, when, how, and where to evolve on the path to artificial super intelligence\.Note:Transactions on Machine Learning ResearchExternal Links:2507\.21046,[Document](https://dx.doi.org/10.48550/arXiv.2507.21046),[Link](https://arxiv.org/abs/2507.21046)Cited by:[§1](https://arxiv.org/html/2607.21635#S1.p9.1)\.
- D\. Guo, J\. Wu, and S\. M\. Yiu \(2026\)AgentEval: dag\-structured step\-level evaluation for agentic workflows with error propagation tracking\.Note:arXiv preprint arXiv:2604\.23581Cited by:[Table 6](https://arxiv.org/html/2607.21635#A1.T6.1.8.6.1.1.1)\.
- Z\. Guo, S\. Cheng, H\. Wang, S\. Liang, Y\. Qin, P\. Li, Z\. Liu, M\. Sun, and Y\. Liu \(2024\)StableToolBench: towards stable large\-scale benchmarking on tool learning of large language models\.Note:Findings of the Association for Computational Linguistics: ACL 2024Cited by:[Table 5](https://arxiv.org/html/2607.21635#A1.T5.1.4.3.1.1.1),[§1](https://arxiv.org/html/2607.21635#S1.p2.1),[§1](https://arxiv.org/html/2607.21635#S1.p3.1),[§2](https://arxiv.org/html/2607.21635#S2.p2.1),[§3\.1](https://arxiv.org/html/2607.21635#S3.SS1.p1.1),[§4](https://arxiv.org/html/2607.21635#S4.p1.1)\.
- A\. Gupta \(2026\)ReliabilityBench: evaluating LLM agent reliability under production\-like stress conditions\.Note:arXiv preprint arXiv:2601\.06112Cited by:[Table 5](https://arxiv.org/html/2607.21635#A1.T5.1.25.24.1.1.1),[Table 6](https://arxiv.org/html/2607.21635#A1.T6.1.6.4.1.1.1),[§3\.1](https://arxiv.org/html/2607.21635#S3.SS1.p5.1),[§4](https://arxiv.org/html/2607.21635#S4.p1.1),[§4](https://arxiv.org/html/2607.21635#S4.p4.1)\.
- X\. Han, Y\. Xiao, C\. Wu, and T\. Zhang \(2026\)How early is early enough? design\-dependent observation\-window sufficiency in subscription churn prediction\.External Links:2607\.00473,[Document](https://dx.doi.org/10.48550/arXiv.2607.00473),[Link](https://doi.org/10.48550/arXiv.2607.00473)Cited by:[§5](https://arxiv.org/html/2607.21635#S5.p1.4)\.
- Z\. He, Y\. Wang, C\. Zhi, Y\. Hu, T\. Chen, L\. Yin, Z\. Chen, T\. A\. Wu, S\. Ouyang, Z\. Wang, J\. Pei, J\. McAuley, Y\. Choi, and A\. Pentland \(2026\)MemoryArena: benchmarking agent memory in interdependent multi\-session agentic tasks\.Note:arXiv preprint arXiv:2602\.16313Cited by:[Table 5](https://arxiv.org/html/2607.21635#A1.T5.1.13.12.1.1.1),[Table 6](https://arxiv.org/html/2607.21635#A1.T6.1.4.2.1.1.1),[§2](https://arxiv.org/html/2607.21635#S2.p2.1),[§3\.1](https://arxiv.org/html/2607.21635#S3.SS1.p2.1),[§4](https://arxiv.org/html/2607.21635#S4.p1.1),[§4](https://arxiv.org/html/2607.21635#S4.p4.1)\.
- Y\. Hu, Y\. Wang, and J\. McAuley \(2026\)Evaluating memory in LLM agents via incremental multi\-turn interactions\.Note:International Conference on Learning RepresentationsCited by:[Table 5](https://arxiv.org/html/2607.21635#A1.T5.1.12.11.1.1.1),[§1](https://arxiv.org/html/2607.21635#S1.p3.1),[§2](https://arxiv.org/html/2607.21635#S2.p2.1),[§3\.1](https://arxiv.org/html/2607.21635#S3.SS1.p2.1),[§4](https://arxiv.org/html/2607.21635#S4.p1.1)\.
- W\. Hua, X\. Yang, M\. Jin, Z\. Li, W\. Cheng, R\. Tang, and Y\. Zhang \(2024\)TrustAgent: towards safe and trustworthy LLM\-based agents\.Note:Conference on Empirical Methods in Natural Language ProcessingExternal Links:2402\.01586,[Document](https://dx.doi.org/10.48550/arXiv.2402.01586)Cited by:[Table 5](https://arxiv.org/html/2607.21635#A1.T5.1.21.20.1.1.1)\.
- Y\. Jiang, D\. Li, H\. Deng, B\. Ma, X\. Wang, Q\. Wang, and G\. Yu \(2026\)SoK: agentic skills – beyond tool use in LLM agents\.Note:arXiv preprint arXiv:2602\.20867Cited by:[§3\.1](https://arxiv.org/html/2607.21635#S3.SS1.p3.1)\.
- Y\. Jiang and F\. Ferraro \(2026\)SCRIBE: structured mid\-level supervision for tool\-using language models\.arXiv preprint arXiv:2601\.03555\.External Links:2601\.03555,[Document](https://dx.doi.org/10.48550/arXiv.2601.03555),[Link](https://doi.org/10.48550/arXiv.2601.03555)Cited by:[§3\.1](https://arxiv.org/html/2607.21635#S3.SS1.p3.1)\.
- W\. Kang and J\. McAuley \(2018\)Self\-attentive sequential recommendation\.Note:IEEE International Conference on Data MiningCited by:[§2](https://arxiv.org/html/2607.21635#S2.p3.1)\.
- S\. Kara, F\. Faisal, and S\. Nath \(2026\)WAREX: web agent reliability evaluation on existing benchmarks\.Note:Transactions on Machine Learning ResearchCited by:[Table 5](https://arxiv.org/html/2607.21635#A1.T5.1.24.23.1.1.1),[Table 6](https://arxiv.org/html/2607.21635#A1.T6.1.5.3.1.1.1),[§3\.1](https://arxiv.org/html/2607.21635#S3.SS1.p5.1),[§4](https://arxiv.org/html/2607.21635#S4.p1.1),[§4](https://arxiv.org/html/2607.21635#S4.p4.1)\.
- J\. Kirkpatrick, R\. Pascanu, N\. Rabinowitz, J\. Veness, G\. Desjardins, A\. A\. Rusu, K\. Milan, J\. Quan, T\. Ramalho, A\. Grabska\-Barwinska,et al\.\(2017\)Overcoming catastrophic forgetting in neural networks\.Proceedings of the National Academy of Sciences114\(13\),pp\. 3521–3526\.Cited by:[§2](https://arxiv.org/html/2607.21635#S2.p3.1)\.
- L\. Kong, X\. Sun, W\. Chow, L\. Li, K\. Q\. Lin, X\. B\. Zhang, S\. Wang, R\. Li, Q\. Wu, W\. Gao, Y\. Wang, S\. Xie, J\. Liu, L\. Qu, S\. Li, L\. X\. Ng, B\. R\. Cottereau, Z\. Liu, T\. Chua, and W\. T\. Ooi \(2026\)AI for auto\-research: roadmap & user guide\.Note:arXiv preprint arXiv:2605\.18661External Links:2605\.18661,[Document](https://dx.doi.org/10.48550/arXiv.2605.18661)Cited by:[§6](https://arxiv.org/html/2607.21635#S6.p1.1)\.
- Y\. Koren \(2009\)Collaborative filtering with temporal dynamics\.InProceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,New York, NY, USA,pp\. 447–456\.Cited by:[§2](https://arxiv.org/html/2607.21635#S2.p3.1)\.
- G\. Lan, H\. A\. Inan, S\. Abdelnabi, J\. Kulkarni, L\. Wutschitz, R\. Shokri, C\. G\. Brinton, and R\. Sim \(2025\)Contextual integrity in LLMs via reasoning and reinforcement learning\.InThe Thirty\-Ninth Annual Conference on Neural Information Processing Systems,External Links:2506\.04245Cited by:[§3\.1](https://arxiv.org/html/2607.21635#S3.SS1.p4.1)\.
- G\. Lan, L\. Xiong, X\. Zhou, H\. Cui, Y\. Zhang, M\. Li, Z\. Shi, B\. Fetahu, L\. Li, and X\. Li \(2026\)Alternating reinforcement learning with contextual rubric rewards: beyond the scalarization strategy\.arXiv preprint arXiv:2603\.15646\.External Links:2603\.15646,[Document](https://dx.doi.org/10.48550/arXiv.2603.15646),[Link](https://doi.org/10.48550/arXiv.2603.15646)Cited by:[§5](https://arxiv.org/html/2607.21635#S5.p1.4)\.
- I\. Levy, B\. Wiesel, S\. Marreed, A\. Oved, A\. Yaeli, N\. Mashkif, and S\. Shlomov \(2026\)ST\-WebAgentBench: a benchmark for evaluating safety and trustworthiness in web agents\.Note:International Conference on Learning Representations \(ICLR\)External Links:2410\.06703Cited by:[Table 6](https://arxiv.org/html/2607.21635#A1.T6.1.7.5.1.1.1)\.
- D\. Li, B\. Jiang, L\. Huang, A\. Beigi, C\. Zhao, Z\. Tan, A\. Bhattacharjee, Y\. Jiang, C\. Chen, T\. Wu, K\. Shu, L\. Cheng, and H\. Liu \(2025\)From generation to judgment: opportunities and challenges of LLM\-as\-a\-judge\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,Suzhou, China,pp\. 2757–2791\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.138)Cited by:[§5](https://arxiv.org/html/2607.21635#S5.p1.4)\.
- J\. Li, Z\. Chen, H\. Luo, and H\. Salam \(2026a\)PrefIx: understand and adapt to user preference in human\-agent interaction\.External Links:2602\.06714,[Document](https://dx.doi.org/10.48550/arXiv.2602.06714),[Link](https://doi.org/10.48550/arXiv.2602.06714)Cited by:[§1](https://arxiv.org/html/2607.21635#S1.p2.1)\.
- M\. Li, Y\. Zhao, B\. Yu, F\. Song, H\. Li, H\. Yu, Z\. Li, F\. Huang, and Y\. Li \(2023\)API\-Bank: a comprehensive benchmark for tool\-augmented LLMs\.Note:Proceedings of the 2023 Conference on Empirical Methods in Natural Language ProcessingExternal Links:2304\.08244,[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.187)Cited by:[Table 5](https://arxiv.org/html/2607.21635#A1.T5.1.2.1.1.1.1),[§4](https://arxiv.org/html/2607.21635#S4.p1.1)\.
- Y\. Li, J\. Li, K\. Yu, X\. Xiao, D\. Liu, T\. Wang, and R\. Tang \(2026b\)Personalize your large vision\-language models with in\-context prompt tuning\.Note:arXiv preprint arXiv:2605\.31513External Links:2605\.31513,[Document](https://dx.doi.org/10.48550/arXiv.2605.31513)Cited by:[§1](https://arxiv.org/html/2607.21635#S1.p2.1)\.
- L\. Lin, J\. You, Y\. Li, L\. Lin, Y\. Wang, Z\. Zhang, and M\. Zheng \(2026a\)Reflect\-Guard: enhancing LLM safeguards against adversarial prompts via logical self\-reflection\.arXiv preprint arXiv:2605\.24834\.External Links:2605\.24834,[Document](https://dx.doi.org/10.48550/arXiv.2605.24834),[Link](https://doi.org/10.48550/arXiv.2605.24834)Cited by:[§3\.1](https://arxiv.org/html/2607.21635#S3.SS1.p4.1)\.
- L\. Lin, L\. Lin, Z\. Zhang, M\. Zheng, and Y\. Wang \(2026b\)A volume\-price\-adjusted MACD trading strategy with sensitivity calibration for U\.S\. equity indices\.Note:arXiv preprint arXiv:2604\.26063External Links:2604\.26063,[Document](https://dx.doi.org/10.48550/arXiv.2604.26063)Cited by:[§6](https://arxiv.org/html/2607.21635#S6.p1.1)\.
- J\. Liu, T\. Li, S\. Du, X\. Luo, H\. Zeng, E\. Tewolde, T\. S\. Lee, T\. Wang, C\. Kingsford, and V\. Conitzer \(2026a\)The memory curse: how expanded recall erodes cooperative intent in LLM agents\.Note:arXiv preprint arXiv:2605\.08060External Links:2605\.08060,[Document](https://dx.doi.org/10.48550/arXiv.2605.08060)Cited by:[§3\.1](https://arxiv.org/html/2607.21635#S3.SS1.p2.1)\.
- X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang,et al\.\(2024\)AgentBench: evaluating LLMs as agents\.Note:International Conference on Learning RepresentationsCited by:[Table 5](https://arxiv.org/html/2607.21635#A1.T5.1.23.22.1.1.1),[§4](https://arxiv.org/html/2607.21635#S4.p1.1)\.
- Y\. Liu, Z\. Cheng, and L\. Lai \(2026b\)Improving the completeness and comparability of segment disclosures: a large language model approach\.External Links:2605\.23924,[Document](https://dx.doi.org/10.48550/arXiv.2605.23924),[Link](https://doi.org/10.48550/arXiv.2605.23924)Cited by:[§5](https://arxiv.org/html/2607.21635#S5.p4.1)\.
- J\. Lu, T\. Holleis, Y\. Zhang, B\. Aumayer, F\. Nan, H\. Bai, S\. Ma, S\. Ma, M\. Li, G\. Yin, Z\. Wang, and R\. Pang \(2025\)ToolSandbox: a stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities\.Note:Findings of the Association for Computational Linguistics: NAACL 2025Cited by:[Table 5](https://arxiv.org/html/2607.21635#A1.T5.1.6.5.1.1.1),[Table 6](https://arxiv.org/html/2607.21635#A1.T6.1.3.1.1.1.1),[§4](https://arxiv.org/html/2607.21635#S4.p1.1),[§4](https://arxiv.org/html/2607.21635#S4.p4.1)\.
- X\. H\. Lù, Z\. Kasner, and S\. Reddy \(2024\)WebLINX: real\-world website navigation with multi\-turn dialogue\.Note:International Conference on Machine LearningCited by:[Table 5](https://arxiv.org/html/2607.21635#A1.T5.1.22.21.1.1.1),[§4](https://arxiv.org/html/2607.21635#S4.p1.1)\.
- B\. P\. Majumder, B\. D\. Mishra, P\. Jansen, O\. Tafjord, N\. Tandon, L\. Zhang, C\. Callison\-Burch, and P\. Clark \(2024\)CLIN: a continually learning language agent for rapid task adaptation and generalization\.Note:Conference on Language ModelingCited by:[Table 5](https://arxiv.org/html/2607.21635#A1.T5.1.10.9.1.1.1),[§3\.1](https://arxiv.org/html/2607.21635#S3.SS1.p2.1)\.
- M\. Mohammadi, Y\. Li, J\. Lo, and W\. Yip \(2025\)Evaluation and benchmarking of LLM agents: a survey\.Note:Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data MiningCited by:[Table 5](https://arxiv.org/html/2607.21635#A1.T5.1.27.26.1.1.1),[§1](https://arxiv.org/html/2607.21635#S1.p9.1),[§4](https://arxiv.org/html/2607.21635#S4.p1.1)\.
- C\. Packer, S\. Wooders, K\. Lin, V\. Fang, S\. G\. Patil, I\. Stoica, and J\. E\. Gonzalez \(2024\)MemGPT: towards LLMs as operating systems\.Note:International Conference on Learning RepresentationsCited by:[Table 5](https://arxiv.org/html/2607.21635#A1.T5.1.9.8.1.1.1),[§1](https://arxiv.org/html/2607.21635#S1.p2.1)\.
- S\. G\. Patil, H\. Mao, F\. Yan, C\. C\. Ji, V\. Suresh, I\. Stoica, and J\. E\. Gonzalez \(2025\)The berkeley function calling leaderboard \(BFCL\): from tool use to agentic evaluation of large language models\.Note:International Conference on Machine LearningCited by:[Table 5](https://arxiv.org/html/2607.21635#A1.T5.1.5.4.1.1.1),[§1](https://arxiv.org/html/2607.21635#S1.p3.1),[§2](https://arxiv.org/html/2607.21635#S2.p2.1),[§4](https://arxiv.org/html/2607.21635#S4.p1.1)\.
- Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, X\. Cong, X\. Tang, B\. Qian,et al\.\(2024\)ToolLLM: facilitating large language models to master 16000\+ real\-world APIs\.Note:International Conference on Learning RepresentationsCited by:[Table 5](https://arxiv.org/html/2607.21635#A1.T5.1.3.2.1.1.1),[§1](https://arxiv.org/html/2607.21635#S1.p2.1)\.
- S\. Rajput, N\. Mehta, A\. Singh, R\. Hulikal Keshavan, T\. Vu, L\. Heldt, L\. Hong, Y\. Tay, V\. Q\. Tran, J\. Samost, M\. Kula, E\. Chi, and M\. Sathiamoorthy \(2023\)Recommender systems with generative retrieval\.Note:Advances in Neural Information Processing SystemsCited by:[§2](https://arxiv.org/html/2607.21635#S2.p3.1)\.
- S\. Raza, R\. Sapkota, M\. Karkee, and C\. Emmanouilidis \(2025\)TRiSM for agentic AI: a review of trust, risk, and security management in LLM\-based agentic multi\-agent systems\.Note:arXiv preprint arXiv:2506\.04133Cited by:[§1](https://arxiv.org/html/2607.21635#S1.p2.1)\.
- Y\. Ruan, H\. Dong, A\. Wang, S\. Pitis, Y\. Zhou, J\. Ba, Y\. Dubois, C\. J\. Maddison, and T\. Hashimoto \(2024\)Identifying the risks of LM agents with an LM\-emulated sandbox\.Note:International Conference on Learning RepresentationsCited by:[Table 5](https://arxiv.org/html/2607.21635#A1.T5.1.7.6.1.1.1),[§4](https://arxiv.org/html/2607.21635#S4.p1.1)\.
- X\. Sun, W\. Gao, Y\. Wang, L\. Kong, Y\. Li, Z\. Fan, Z\. Zhuang, W\. Dong, Z\. Zheng, H\. Paranjape, A\. Mandal, and J\. R\. Zhang \(2026\)Beyond accuracy: measuring bias acknowledgment in chain\-of\-thought reasoning for responsible AI evaluation\.Note:ICML 2026 Workshop on Trustworthy AI for GoodExternal Links:2606\.15127,[Document](https://dx.doi.org/10.48550/arXiv.2606.15127),[Link](https://doi.org/10.48550/arXiv.2606.15127)Cited by:[§5](https://arxiv.org/html/2607.21635#S5.p1.4)\.
- H\. Tan, Z\. Zhang, C\. Ma, X\. Chen, Q\. Dai, and Z\. Dong \(2025\)MemBench: towards more comprehensive evaluation on the memory of LLM\-based agents\.InFindings of the Association for Computational Linguistics: ACL 2025,Vienna, Austria,pp\. 19336–19352\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.989),[Link](https://aclanthology.org/2025.findings-acl.989/),2506\.21605Cited by:[Table 5](https://arxiv.org/html/2607.21635#A1.T5.1.11.10.1.1.1),[§1](https://arxiv.org/html/2607.21635#S1.p3.1),[§2](https://arxiv.org/html/2607.21635#S2.p2.1),[§3\.1](https://arxiv.org/html/2607.21635#S3.SS1.p2.1),[§4](https://arxiv.org/html/2607.21635#S4.p1.1)\.
- Y\. Tao, Y\. Wang, X\. Song, X\. Luo, K\. Liu, and J\. Liu \(2026\)GRASP: plan\-guided graph retrieval with adaptive fusion and reranking on semi\-structured knowledge bases\.arXiv preprint arXiv:2605\.30237\.External Links:2605\.30237,[Document](https://dx.doi.org/10.48550/arXiv.2605.30237),[Link](https://doi.org/10.48550/arXiv.2605.30237)Cited by:[§5](https://arxiv.org/html/2607.21635#S5.p4.1)\.
- G\. Wang, Y\. Xie, Y\. Jiang, A\. Mandlekar, C\. Xiao, Y\. Zhu, L\. Fan, and A\. Anandkumar \(2024\)Voyager: an open\-ended embodied agent with large language models\.Note:Transactions on Machine Learning ResearchCited by:[Table 5](https://arxiv.org/html/2607.21635#A1.T5.1.15.14.1.1.1)\.
- Y\. Wang, X\. Sun, L\. Kong, W\. Gao, Y\. Li, Z\. Fan, and Z\. Zhuang \(2026a\)Do time series foundation model benchmarks hide regime\-dependent failures? evidence from traffic speed forecasting\.Note:Accepted at the Workshop on Forecasting as a New Frontier of Intelligence, ICML 2026External Links:2606\.18367,[Document](https://dx.doi.org/10.48550/arXiv.2606.18367),[Link](https://doi.org/10.48550/arXiv.2606.18367)Cited by:[§5](https://arxiv.org/html/2607.21635#S5.p1.4)\.
- Y\. Wang, X\. Sun, Y\. Li, Z\. Fan, and Z\. Zhuang \(2026b\)Embedding foundation model predictions in discrete\-choice models with structural guarantees\.External Links:2606\.26432,[Document](https://dx.doi.org/10.48550/arXiv.2606.26432),[Link](https://doi.org/10.48550/arXiv.2606.26432)Cited by:[§6](https://arxiv.org/html/2607.21635#S6.p1.1)\.
- C\. Wu \(2026\)Class weighting versus amount conditioning in credit\-card fraud detection: a dollar\-metric study with a temporal explanation audit\.External Links:2607\.14686,[Document](https://dx.doi.org/10.48550/arXiv.2607.14686),[Link](https://doi.org/10.48550/arXiv.2607.14686)Cited by:[§5](https://arxiv.org/html/2607.21635#S5.p1.4)\.
- R\. Wu, X\. Wang, J\. Mei, P\. Cai, D\. Fu, C\. Yang, L\. Wen, X\. Yang, Y\. Shen, Y\. Wang, and B\. Shi \(2025\)EvolveR: self\-evolving LLM agents through an experience\-driven lifecycle\.Note:arXiv preprint arXiv:2510\.16079Cited by:[Table 5](https://arxiv.org/html/2607.21635#A1.T5.1.16.15.1.1.1)\.
- N\. Xu, Y\. Jiang, S\. R\. Dipta, and H\. Zhang \(2025\)Learning how to use tools, not just when: pattern\-aware tool\-integrated reasoning\.Note:NeurIPS 2025 MATH\-AI WorkshopExternal Links:2509\.23292,[Document](https://dx.doi.org/10.48550/arXiv.2509.23292)Cited by:[§3\.1](https://arxiv.org/html/2607.21635#S3.SS1.p1.1)\.
- R\. Xu and Y\. Yan \(2026\)Agent skills for large language models: architecture, acquisition, security, and the path forward\.Note:arXiv preprint arXiv:2602\.12430Cited by:[§1](https://arxiv.org/html/2607.21635#S1.p2.1),[§1](https://arxiv.org/html/2607.21635#S1.p9.1),[§2](https://arxiv.org/html/2607.21635#S2.p2.1),[§3\.1](https://arxiv.org/html/2607.21635#S3.SS1.p3.1),[§4](https://arxiv.org/html/2607.21635#S4.p1.1)\.
- Y\. Yang, J\. Li, Q\. Pan, B\. Zhan, Y\. Cai, L\. Du, J\. Zhou, K\. Chen, Q\. Chen, X\. Li, B\. Zhang, and L\. He \(2026\)AutoSkill: experience\-driven lifelong learning via skill self\-evolution\.Note:arXiv preprint arXiv:2603\.01145Cited by:[Table 5](https://arxiv.org/html/2607.21635#A1.T5.1.17.16.1.1.1),[§1](https://arxiv.org/html/2607.21635#S1.p2.1)\.
- S\. Yao, N\. Shinn, P\. Razavi, and K\. Narasimhan \(2025\)τ\\tau\-bench: a benchmark for tool\-agent\-user interaction in real\-world domains\.Note:International Conference on Learning RepresentationsCited by:[Table 6](https://arxiv.org/html/2607.21635#A1.T6.1.1.1.1.1),[§4](https://arxiv.org/html/2607.21635#S4.p4.1)\.
- A\. Yehudai, L\. Eden, A\. Li, G\. Uziel, Y\. Zhao, R\. Bar\-Haim, A\. Cohan, and M\. Shmueli\-Scheuer \(2025\)Survey on evaluation of LLM\-based agents\.Note:arXiv preprint arXiv:2503\.16416Cited by:[Table 5](https://arxiv.org/html/2607.21635#A1.T5.1.26.25.1.1.1),[§1](https://arxiv.org/html/2607.21635#S1.p9.1),[§4](https://arxiv.org/html/2607.21635#S4.p1.1)\.
- S\. Yoo and M\. Harman \(2012\)Regression testing minimization, selection and prioritization: a survey\.Software Testing, Verification and Reliability22\(2\),pp\. 67–120\.Cited by:[§2](https://arxiv.org/html/2607.21635#S2.p3.1)\.
- M\. Yu, F\. Meng, X\. Zhou, S\. Wang, J\. Mao, L\. Pang, T\. Chen, K\. Wang, X\. Li, Y\. Zhang, B\. An, and Q\. Wen \(2025\)A survey on trustworthy LLM agents: threats and countermeasures\.Note:arXiv preprint arXiv:2503\.09648Cited by:[§1](https://arxiv.org/html/2607.21635#S1.p2.1),[§4](https://arxiv.org/html/2607.21635#S4.p1.1)\.
- Y\. Zhang, Z\. Xiang, and H\. Xu \(2026\)Performance\-efficiency trade\-offs in human preference prediction: a comparative study of traditional machine learning and large language models\.Vilamoura, Algarve, Portugal\.Note:Proceedings of the 31st IEEE Symposium on Computers and Communications \(ISCC\)Cited by:[§6](https://arxiv.org/html/2607.21635#S6.p1.1)\.
- Z\. Zhang, S\. Cui, Y\. Lu, J\. Zhou, J\. Yang, H\. Wang, and M\. Huang \(2024\)Agent\-SafetyBench: evaluating the safety of LLM agents\.Note:arXiv preprint arXiv:2412\.14470Cited by:[Table 5](https://arxiv.org/html/2607.21635#A1.T5.1.19.18.1.1.1),[§2](https://arxiv.org/html/2607.21635#S2.p2.1),[§3\.1](https://arxiv.org/html/2607.21635#S3.SS1.p4.1),[§4](https://arxiv.org/html/2607.21635#S4.p1.1)\.
- Y\. Zhao, B\. Yuan, J\. Huang, H\. Yuan, Z\. Yu, H\. Xu, L\. Hu, A\. Shankarampeta, Z\. Huang, W\. Ni, Y\. Tian, and J\. Zhao \(2026\)AMA\-Bench: evaluating long\-horizon memory for agentic applications\.Note:arXiv preprint arXiv:2602\.22769Cited by:[Table 5](https://arxiv.org/html/2607.21635#A1.T5.1.14.13.1.1.1)\.
- J\. Zheng, C\. Shi, X\. Cai, Q\. Li, D\. Zhang, C\. Li, D\. Yu, and Q\. Ma \(2026\)Lifelong learning of large language model based agents: a roadmap\.IEEE Transactions on Pattern Analysis and Machine Intelligence48\(5\),pp\. 5552–5571\.External Links:[Document](https://dx.doi.org/10.1109/TPAMI.2025.3650546),[Link](https://doi.org/10.1109/TPAMI.2025.3650546),2501\.07278Cited by:[§1](https://arxiv.org/html/2607.21635#S1.p9.1),[§4](https://arxiv.org/html/2607.21635#S4.p1.1)\.
- L\. Zhong, Z\. Du, X\. Zhang, H\. Hu, and J\. Tang \(2025\)ComplexFuncBench: exploring multi\-step and constrained function calling under long\-context scenario\.Note:arXiv preprint arXiv:2501\.10132Cited by:[Table 5](https://arxiv.org/html/2607.21635#A1.T5.1.8.7.1.1.1)\.
- S\. Zhong, Y\. Lu, J\. Ning, Y\. Wan, L\. Feng, Y\. Ao, L\. F\. R\. Ribeiro, M\. Dreyer, S\. Ammirati, and C\. Xiong \(2026\)SkillLearnBench: benchmarking continual learning methods for agent skill generation on real\-world tasks\.Note:arXiv preprint arXiv:2604\.20087Cited by:[Table 5](https://arxiv.org/html/2607.21635#A1.T5.1.18.17.1.1.1),[§1](https://arxiv.org/html/2607.21635#S1.p3.1),[§2](https://arxiv.org/html/2607.21635#S2.p2.1),[§3\.1](https://arxiv.org/html/2607.21635#S3.SS1.p3.1),[§4](https://arxiv.org/html/2607.21635#S4.p1.1)\.
- X\. Zhu, J\. Wu, X\. Zhang, T\. Li, Y\. Mu, J\. Zhai, C\. Shen, C\. Fang, and Y\. Liu \(2026\)An empirical study of bugs in modern LLM agent frameworks\.External Links:2602\.21806,[Document](https://dx.doi.org/10.48550/arXiv.2602.21806)Cited by:[§1](https://arxiv.org/html/2607.21635#S1.p1.1),[§3\.1](https://arxiv.org/html/2607.21635#S3.SS1.p1.1),[§5](https://arxiv.org/html/2607.21635#S5.p6.2)\.