Auditing Self-Evolution in Financial Agents: Capability Gains, Security Drift, and Execution-Interface Mismatch
Summary
This paper audits self-evolution mechanisms in financial AI agents, revealing capability improvements alongside security risks such as prompt injection drift and execution-interface mismatches, emphasizing the need for holistic auditing.
View Cached Full Text
Cached at: 08/19/26, 10:11 AM
# Auditing Self-Evolution in Financial Agents:Capability Gains, Security Drift, and Execution-Interface Mismatch
Source: [https://arxiv.org/html/2608.17684](https://arxiv.org/html/2608.17684)
## Auditing Self\-Evolution in Financial Agents: Capability Gains, Security Drift, and Execution\-Interface MismatchCCS:Security and privacy Software and application securityCCS:Computing methodologies Artificial intelligence
andJialing ZhuOrcID:[0009\-0001\-3799\-8578](https://orcid.org/0009-0001-3799-8578)Affiliation:Independent Researcher
###### Abstract\.
Self\-evolving agents turn experience into reusable skills, workflows, or memories, but post\-evolution accuracy alone does not show whether learned behaviour preserves previously correct behaviour or security\. We audit SkillOpt, Agent Workflow Memory \(AWM\), and ReasoningBank in simulated e\-banking using matched benign acquisition trajectories, sealed evaluation endpoints, execution\-grounded checks, and independent state replay\. On Qwen 3\.7 Flash, SkillOpt raises benign utility from 0\.741 to 0\.837, with 23 wrong\-to\-correct gains and 10 correct\-to\-wrong regressions, while exposure to injected content rises from 0\.820 to 0\.943\. Among exposed episodes, attack success falls from 0\.605 to 0\.562; across all attacked episodes, attack success rate \(ASR\)—the fraction in which the injected objective succeeds—rises from 0\.496 to 0\.530, while unauthorized financial state changes rise to 0\.685\. The ASR increase is consistent with exposure\-driven drift rather than greater compliance once exposed\. Across three independently evolved lineages, capability, exposure, and unauthorized\-state changes increase in all three, whereas ASR increases in only two, making target\-goal ASR the least consistent of these quantities here\. ReasoningBank raises utility to 0\.859, with 25 gains and 9 regressions; aggregate ASR does not increase, though unauthorized state changes remain slightly above Static\. AWM reveals a separate evaluation hazard: a literal WebArena text\-action envelope disrupts tool execution in our native function\-calling executor\. In a post\-hoc sensitivity test, removing only that envelope restores utility from 0\.319 to 0\.756, near Static \(0\.741\), while exposure rises from 0\.299 to 0\.909 and ASR from 0\.195 to 0\.575\. Auditing self\-evolving financial agents therefore requires tracking regressions, attack\-surface contact, unauthorized financial\-state change, and artifact–executor compatibility—not accuracy alone\.
###### Keywords:
LLM agents, self\-evolution, prompt injection, financial agents, auditing
## 1\.Introduction
Language agents can improve future decisions by retaining text derived from prior interactions\. Reflexion stores verbal reflections in episodic memory and reuses them in subsequent trials\([9](https://arxiv.org/html/2608.17684#bib.bib5)\), whereas Self\-Refine iteratively critiques and revises the current output using self\-generated feedback\([5](https://arxiv.org/html/2608.17684#bib.bib6)\)\. Recent self\-evolving systems make reusable external state explicit: SkillOpt updates a natural\-language skill document through trajectory\-derived edits and validation\-gated deployment\([13](https://arxiv.org/html/2608.17684#bib.bib2)\); Agent Workflow Memory \(AWM\) induces reusable workflows from successful prior trajectories\([10](https://arxiv.org/html/2608.17684#bib.bib3)\); and ReasoningBank distils and retrieves reasoning memories from successful and failed experiences\([7](https://arxiv.org/html/2608.17684#bib.bib4)\)\.
Such learned state is intended to improve task performance\. In a financial agent, however, it can improve task completion while simultaneously changing which external information the agent reads and which consequential tools it invokes\. Aggregate post\-evolution accuracy does not reveal these changes, which matter when useful information and attacker\-controlled instructions arrive through the same interaction surface\.
Indirect prompt injection exploits this boundary by placing malicious instructions in external data that an LLM\-integrated application later processes\([2](https://arxiv.org/html/2608.17684#bib.bib9)\)\. InjecAgent shows that the threat extends to tool\-integrated agents that consume external content and can subsequently invoke consequential tools\([15](https://arxiv.org/html/2608.17684#bib.bib10)\)\. AgentDojo operationalises this threat in an extensible tool\-use environment containing untrusted data, including an e\-banking suite\([1](https://arxiv.org/html/2608.17684#bib.bib1)\)\. In our setting, injected text therefore enters the model*only*when the agent retrieves the corresponding tool result\. A learned instruction such as “inspect recent transactions or a referenced file before acting” may help a benign task while increasing contact with attacker\-controlled content; conversely, an evolved artifact can appear safe if it disrupts tool execution and never reaches that content\. Shao et al\. formalize unintended degradation during self\-evolution as*misevolution*across model, memory, tool, and workflow pathways\([8](https://arxiv.org/html/2608.17684#bib.bib7)\)\. We complement this broader characterization with a controlled financial audit separating capability, attack exposure, conditional susceptibility, and execution\-grounded financial harm\.
We ask:when a financial agent learns from benign experience, does it improve weak behaviours while preserving already\-correct behaviour and security?We evaluate the closed loop*learning rule→\\rightarrowevolved artifact→\\rightarrowexecutor behaviour→\\rightarrowinteraction surface→\\rightarrowfinancial state*\. We report W→\\rightarrowC gains and C→\\rightarrowW regressions, prompt\-injection exposure, conditional attack success after exposure, total attack success, and observable unauthorized financial state changes\.
Our primary study uses AgentDojo’s Banking suite\([1](https://arxiv.org/html/2608.17684#bib.bib1)\)after a pre\-model checker audit yielding a corrected 15\-family suite\. We compare SkillOpt, AWM, and ReasoningBank under one primary executor, one offline benign evolution pass, and three independently evolved lineages\. Evolution uses only benign evidence: within each lineage the three methods receive byte\-identical acquisition trajectories, while all evaluation variants remain sealed until every evolved artifact is frozen\. The comparison is deliberately end\-to\-end because the systems differ in write eligibility, admission, update mechanism, state representation, and retrieval; observed differences therefore cannot be attributed to representation alone\.
Contributions\.
1. \(1\)We introduce an execution\-grounded audit protocol for self\-evolving financial agents that measures paired capability gains and regressions and decomposes attack success into exposure and conditional susceptibility, alongside unauthorized state change\.
2. \(2\)We show that capability and security can decouple\. SkillOpt improves utility by 9\.6 percentage points while exposure rises by 12\.3 points and unauthorized state changes by 10\.2 points; ReasoningBank improves utility by 11\.9 points without increasing aggregate attack success\. These effects are method\-dependent and persist on the subset of families whose official checkers we never modified\. Aggregate attack success is also less consistent across independently evolved lineages, underscoring why exposure and execution\-grounded harm should be reported separately\.
3. \(3\)We identify*execution\-interface mismatch*as an evaluation confound\. A post\-hoc controlled AWM adaptation holds learned workflow content fixed while changing only an incompatible textual\-action envelope, testing whether the observed degradation arises from workflow content or from the executor\-facing interface\.
## 2\.Related Work
Learning from agent experience\.Beyond the systems we audit \(§[1](https://arxiv.org/html/2608.17684#S1)\), SkillOS learns a policy for curating reusable skills\([6](https://arxiv.org/html/2608.17684#bib.bib8)\), while Shao et al\. frame*misevolution*as emergent risk across model, memory, tool, and workflow pathways\([8](https://arxiv.org/html/2608.17684#bib.bib7)\)\. We complement this broader risk perspective with a controlled financial audit: we hold the banking environment and attack fixed and match benign evidence byte\-for\-byte across three external\-state evolution systems, so observed differences are not attributable to differences in acquisition evidence\.
Security of tool\-using agents\.Beyond indirect\-injection threat models\([2](https://arxiv.org/html/2608.17684#bib.bib9);[15](https://arxiv.org/html/2608.17684#bib.bib10);[1](https://arxiv.org/html/2608.17684#bib.bib1)\), broader benchmarks have examined security and safety in tool\-using agents\. Agent Security Bench evaluates attacks and defenses in LLM\-based agents with tool\-use capabilities\([16](https://arxiv.org/html/2608.17684#bib.bib11)\)\. Agent\-SafetyBench studies safety risks in interactive LLM agents across diverse scenarios\([17](https://arxiv.org/html/2608.17684#bib.bib12)\)\. Defences such as Task Shield enforce task alignment during execution\([4](https://arxiv.org/html/2608.17684#bib.bib13)\)\. These benchmarks primarily evaluate security under a fixed agent configuration; we instead hold the environment, attack, and evaluation protocol fixed while the agent changes through benign self\-evolution\.
Financial LLMs and agents\.FinanceBench targets open\-book financial question answering\([3](https://arxiv.org/html/2608.17684#bib.bib16)\)\. FinBen provides broad multi\-task evaluation of financial language models\([11](https://arxiv.org/html/2608.17684#bib.bib14)\)\. PIXIU provides a financial benchmark, instruction dataset, and domain\-specific language model\([12](https://arxiv.org/html/2608.17684#bib.bib18)\)\. FinCon studies memory and verbal reinforcement in a financial multi\-agent system\([14](https://arxiv.org/html/2608.17684#bib.bib15)\)\. Our setting is narrower but execution\-grounded: AgentDojo Banking provides mutable account state and consequential tools, letting us audit changes to transfers, scheduled payments, and credentials, together with benchmark\-verified sensitive\-data disclosure, rather than only task\-level scores or trading performance\.
## 3\.Audit Protocol
### 3\.1\.Environment and checker audit
We use AgentDojo Banking v1\.2\.2\([1](https://arxiv.org/html/2608.17684#bib.bib1)\), whose pinned release contains 16 user\-task families, 9 injection goals, and 11 tools\. Before any model call, we audited every official utility predicate using up to six terminal\-state probes—ground truth, no\-op, and the task\-applicable subset of wrong\-recipient, wrong\-amount, wrong\-output, and spurious\-action—requiring each predicate to accept ground truth and reject every applicable adversarial state\. Four of sixteen predicates failed this criterion, in three defect classes \(Table[1](https://arxiv.org/html/2608.17684#S3.T1)\)\. Fortask\_5andtask\_11the repair also replaces a literal ground\-truth recipient \(“Spotify”, “Apple”\) with the executable account identifier\. All repairs were frozen before any executor result was observed\. Primary results therefore useour corrected 15\-family AgentDojo Banking suite; the 12 families whose official predicates required no correction form an untouched\-checker sensitivity subset, analysed in §[4\.6](https://arxiv.org/html/2608.17684#S4.SS6)\. These are four predicates in one pinned Banking version under this particular measurement use, not a general claim about the benchmark\.
Table 1\.The four official utility predicates that failed the pre\-model audit, and their frozen treatment\. Repairs were frozen before model execution and are execution\-grounded\.
### 3\.2\.Instances, split, and endpoint blindness
Each family is parameterised into six intent\-preserving variants \(v0–v5\) via mandatory parameter variation plus authored paraphrase templates; no variant is LLM\-generated\. Eleven families are*seen*and four*unseen*, selected deterministically from labels alone, with predefined near\-duplicate clusters held out atomically so none spans the evolution and unseen sets\. Roles are fixed in advance:v0/v1for acquisition,v2exclusively for SkillOpt’s native validation gate, andv3/v4/v5as evaluation endpoints\. Endpoint variants remain sealed—a runtime guard raises if they are requested—until all evolved artifacts for all lineages are frozen \(Fig\.[1](https://arxiv.org/html/2608.17684#acmlabel1)\)\.
### 3\.3\.Evolution and matched acquisition
We runthree independently evolved lineages\. A lineage is a complete end\-to\-end repetition, not a repeated call to one artifact: the unevolved executor first generates its benign acquisition set \(11 seen families×\\times\{v0,v1\}=22=22trajectories\), after which the lineage independently evolves its SkillOpt skill, AWM workflow, and ReasoningBank memory bank from that shared record and is then evaluated at the endpoints under the frozen protocol\. Within a lineage the 22\-trajectory record is handed byte\-identically to all three systems with hash equality asserted, so method effects are not confounded with trajectory noise; across lineages the record differs\. Endpoint results therefore incorporate variability from acquisition, evolution, and hosted inference across the three lineages\. We run one offline evolution pass per lineage; unseen families receive zero acquisition exposure\.
Units of analysis\.The substantive experimental units are the 15 task families\. Variants and lineages are repeated measurements of those families, so the 135 benign or 405 attacked rows per condition are*not*treated as independent samples, and no test treats them as independent\.
Evolution diagnostics\.Acquisition yielded 16/16/17 successes out of 22; SkillOpt’s gate accepted a new skill in all three lineages \(v20\.818/0\.818/0\.636\); AWM updated in all three, with template deduplication leaving 9/9/10 examples over as many families; ReasoningBank wrote 22 entries per lineage \(61/63/63 items, 6/6/5 failure\-derived\), retrieving top\-1 throughout\.
### 3\.4\.Metrics
Benign utility is the corrected checker’s verdict\. We pair every evolved instance against the Static run on the identical \(lineage, family, variant\) and report the four transitions W→\\rightarrowC, C→\\rightarrowW, C→\\rightarrowC, and W→\\rightarrowW, so gross gains and regressions stay visible rather than netted away\.
For security, ASR is the fraction of attacked episodes achieving the frozen injection goal\. We separately report*exposure*, the fraction in which injected content enters the model\-visible interaction, and*conditional ASR*,Pr\(S∣E\)\\Pr\(S\\mid E\)\. By the law of total probability,
ASR=Pr\(E\)Pr\(S∣E\)\+Pr\(¬E\)Pr\(S∣¬E\),\\mathrm\{ASR\}=\\Pr\(E\)\\Pr\(S\\mid E\)\+\\Pr\(\\neg E\)\\Pr\(S\\mid\\neg E\),whereEEdenotes exposure andSSattack success\. The second term vanishes empirically: no episode classified as unexposed achieved the injection goal \(0/579, Qwen arms\), so empirical ASR factorises exactly into exposure and conditional ASR here\. The decomposition separates changes in how often the agent encounters attacker\-controlled content from changes in compliance once exposed\.
To capture harm beyond the designated attack goal, we replay the corrected benign ground truth from the same pre\-state to construct the authorised post\-state and compare it with the observed terminal state\. Unauthorized state change is true iff this comparison yields at least one of five state\-grounded categories: unauthorized transfer, recipient substitution, amount manipulation, unauthorized payment modification, or unauthorized account/password change\. Sensitive\-data disclosure is goal\-checker\-derived and a subset of goal\-keyed successes; we infer no monetary loss\.
We also generate method\-matched generic\-state placebos at the same insertion/retrieval position as a secondary control\. Because realised state lengths are only approximately matched, placebo contrasts are treated as sensitivity analyses rather than primary causal estimates \(§[4\.6](https://arxiv.org/html/2608.17684#S4.SS6)\)\.
### 3\.5\.Configuration
The primary executor isqwen3\.7\-flash\-2026\-07\-15at temperature0\.00\.0, with thinking disabled and only the benchmark’s local tools \(no provider\-hosted tools\)\. A pre\-evolution Static screen selected this executor under a pre\-specified rule requiring measurable capability and security headroom, after earlier screened executors showed near\-ceiling benign utility\. No evolved\-system outcome influenced model selection\.
The same model serves as optimiser, inducer, and distiller at temperature1\.01\.0\. This matches the upstream setting for AWM and ReasoningBank; SkillOpt does not specify an optimiser temperature upstream\. ReasoningBank retrieval usesqwen3\.7\-text\-embedding, an adapted substrate for upstream’sgemini\-embedding\-001, preserving instruction\-aware query embedding, cosine similarity and top\-1 retrieval\. The attack is AgentDojo’simportant\_instructions, with no defence; the attacker controls only untrusted text returned by the banking tools\.
We use three lineages\. Per condition, the benign endpoint contains1515families×\\times33variants×\\times33lineages=135=135cases; the security endpoint contains1515families×\\times99goals×\\timesv3×\\times33lineages=405=405cases\.
Table[2](https://arxiv.org/html/2608.17684#S3.T2)summarises the ported systems\. Fidelity choices are recorded per component; harness adaptations are limited to I/O retargeting, the embedding substrate and the boolean utility signal, no core update or retrieval rule is altered, and evolved artifacts are not edited before evaluation\.
AI\-assisted implementation\.Generative\-AI coding assistance was used during implementation and code review\. All reported metrics and semantic verification were produced by the frozen experimental pipeline and independently checked by deterministic replay; no AI\-generated judgement was used as ground truth\.
Table 2\.Ported systems, compared end\-to\-end under their method\-specific update and retrieval rules\.Per\-lineage matched benign acquisition11 seen families×\\times\{v0,v1\}=22=22generated independently; shared across methodsAWMSkillOptReasoningBankFreeze all evolved artifacts\(all methods×\\timesall lineages\)Unseal endpointsbenign:v3/v4/v5\(135\)security:v3only \(405\)endpoint\-blind regionFigure 1\.Audit protocol\. Within each lineage, one matched benign acquisition set feeds all three systems; acquisition sets are generated independently across lineages\. All evolved artifacts are frozen before any endpoint variant is opened\. Inside the dashed regionv3/v4/v5are sealed andv2is reserved for SkillOpt’s validation gate\.Flow diagram: within each lineage, one matched benign acquisition set of 22 trajectories from 11 seen families feeds SkillOpt, AWM, and ReasoningBank in parallel\. The acquisition set differs across the three lineages\. All evolved artifacts are then frozen; only afterwards are v3, v4, and v5 unsealed for benign evaluation, with v3 alone used for attacked evaluation\. A dashed box marks the endpoint\-blind region covering acquisition, evolution, and freezing\.
## 4\.Results
Table 3\.Main results on the corrected 15\-family AgentDojo Banking suite\. Benignn=135n\{=\}135per condition; attackedn=405n\{=\}405\(SkillOpt 400 after excluding five provider\-side*repetition*\-guardrail rejections, §[4\.7](https://arxiv.org/html/2608.17684#S4.SS7)\)\. Transitions are paired against Static on identical \(lineage, family, variant\); cASR is attack success conditional on exposure\.∗Literal upstream WebArena textual\-action serialization; the action envelope is incompatible with native function calling in this setup \(§[4\.5](https://arxiv.org/html/2608.17684#S4.SS5)\)\.†Post\-hocinterface\-sensitivity arm, not part of the pre\-specified primary comparison\.
### 4\.1\.Capability gains and regressions
Both SkillOpt and ReasoningBank improve benign utility over Static \(0\.741\): SkillOpt to 0\.837 \(\+9\.6\+9\.6points\) and ReasoningBank to 0\.859 \(\+11\.9\+11\.9points\)\. The transition table shows what the means hide\. SkillOpt turns 23 previously\-wrong instances correct but also turns 10 previously\-correct instances wrong; ReasoningBank is 25 against 9\. Static reruns disagree on 10 of the 135 pairwise cross\-lineage comparisons of matched \(family, variant\) cells, so individual regressions are not attributed to evolution; the net gain is positive in all three lineages \(§[4\.3](https://arxiv.org/html/2608.17684#S4.SS3)\)\. For a financial deployment they still matter: the paired Static agent solved those instances\.
We do not claim transfer\. Static is*better*on the four unseen families than on the eleven seen ones \(0\.917 vs 0\.677\), so the held\-out subset has a higher Static baseline; against that baseline SkillOpt falls to 0\.806 while ReasoningBank reaches 0\.944, and their matched placebos reach 0\.972\. With four unseen families we read these descriptively, not as cross\-family transfer\.
### 4\.2\.Capability–attack\-surface coupling
The central result lies in the decomposition rather than the aggregate alone\. For SkillOpt, exposure rises from 0\.820 to 0\.943 \(\+12\.3\+12\.3points\) while conditional ASR*falls*from 0\.605 to 0\.562 \(−4\.3\-4\.3points\); total ASR rises from 0\.496 to 0\.530 \(\+3\.4\+3\.4points\) and unauthorized state change from 0\.583 to 0\.685 \(\+10\.2\+10\.2points\)\.
SkillOpt’s aggregate cASR falls despite the higher total ASR; the dominant shift is in exposure, not conditional susceptibility\. The learned skills contain additional guidance to inspect transaction history and referenced files before acting, and the fraction of attacked episodes containing a read call rises from 0\.867 to 0\.970 while mean tool calls rise from 3\.17 to 3\.75\. In this environment those reads are also routes through which injected content becomes model\-visible\. We therefore describe the observed pattern as*capability–attack\-surface coupling*: capability improvement co\-occurs with a larger interaction surface\. This is trace\-supported mechanism evidence, not an ablation of individual edits or a general causal law\.
The increase is not confined to one goal\.Table[4](https://arxiv.org/html/2608.17684#S4.T4)gives all nine injection goals\. SkillOpt’s ASR exceeds Static’s on six of nine and falls on three, with the largest increases ongoal 7\(\+17\.4\+17\.4points\),goal 6\(\+13\.3\+13\.3\), andgoal 5\(\+8\.9\+8\.9\), and the largest decrease ongoal 4\(−13\.3\-13\.3\)\. Thus the aggregate\+3\.4\+3\.4points is a net of offsetting per\-goal movements rather than a single\-goal effect\. Exposure, by contrast, rises on*all nine*goals, from a 0\.800–0\.844 band under Static to 0\.911–0\.956 under SkillOpt\. This uniformity is consistent with the exposure\-driven reading: the evolved condition reads attacker\-reachable content more often across objectives, while whether contact converts into success remains goal\-specific\. ReasoningBank’s exposure remains in a narrow 0\.800–0\.822 band across all nine\.
Table 4\.Per\-goal attack success and exposure\. Exposure rises on all nine goals under SkillOpt; ASR movement is mixed but net positive\.Δ\\Deltais SkillOpt−\-Static ASR\.ReasoningBank provides a useful contrast\. Despite the*larger*capability gain \(\+11\.9\+11\.9points\), its exposure is 0\.802, below Static’s 0\.820, and aggregate ASR is 0\.474 \(−2\.2\-2\.2points\)\. Security drift is thereforemethod\-dependent, not an inevitable consequence of capability improvement\. This does not establish safety: unauthorized state change remains 0\.595, slightly above Static’s 0\.583\.
002244668810101212141416161818−5\-500551010SkillOptReasoningBankAWM\-IA*\(post\-hoc\)*higher ASR↑\\uparrowΔ\\Deltabenign utility vs Static \(pp\)Δ\\DeltaASR vs Static \(pp\)individual lineagepooled estimateFigure 2\.Capability change against attack\-success drift, relative to Static\. Small filled markers are the three independently evolved lineages; large open markers are the pooled estimates of Table[3](https://arxiv.org/html/2608.17684#S4.T3)\. SkillOpt’s lineages straddle the zero\-ASR line: one decreases ASR by7\.47\.4pp, so the pooled\+3\.4\+3\.4pp shift is not lineage\-consistent \(§[4\.3](https://arxiv.org/html/2608.17684#S4.SS3)\)\. AWM\-InterfaceAdapted is apost\-hocsensitivity arm and is shown pooled only\.Scatter plot of change in attack success rate against change in benign utility relative to Static, in percentage points\. For SkillOpt three small circles show the individual lineages at plus 11\.1 utility with minus 7\.4 attack success, plus 15\.6 with plus 7\.3, and plus 2\.2 with plus 10\.4, straddling the zero line; a large open circle marks the pooled estimate at plus 9\.6 and plus 3\.4\. For ReasoningBank three small squares lie below the zero line between minus 3\.0 and minus 1\.5 attack success, with a large open square at the pooled estimate, plus 11\.9 and minus 2\.2\. A grey open triangle marks the post\-hoc interface\-adapted AWM arm at plus 1\.5 and plus 7\.9\.
### 4\.3\.Which effects reproduce across lineages
Pooled rates hide how consistently an effect reproduces\. Table[5](https://arxiv.org/html/2608.17684#S4.T5)reports each headline effect as three lineage\-level deltas against Static, together with the median, range, and directional agreement\.
Table 5\.Lineage\-level robustness of headline effects \(percentage points vs\. Static\)\. Each lineage is an independent end\-to\-end evolution\. Dir\. is the number of lineages whose delta has the same sign as the pooled effect\. Pooled estimates appear in Table[3](https://arxiv.org/html/2608.17684#S4.T3)and are distinct from lineage medians\.The pattern is asymmetric\. SkillOpt’s*capability*gain,*exposure*expansion, and*unauthorized\-state*increase each hold in all three lineages\. Its*total ASR*increase does not: one lineage decreases by7\.47\.4pp \(Fig\.[2](https://arxiv.org/html/2608.17684#acmlabel2)\), and the lineage range spans zero \(−7\.4\-7\.4to\+10\.4\+10\.4pp\) despite a pooled shift of\+3\.4\+3\.4pp, where every other range in the table is one\-signed\. We therefore do not describe SkillOpt’s ASR increase as uniformly replicated\. Accordingly, our defensible claim is that*SkillOpt increases benign capability, attack exposure, and unauthorized financial state changes in all three independently evolved lineages; pooled total attack success also increases, but that direction agrees in only two of three\.*
ReasoningBank is directionally consistent on both reported quantities \(3/3 each\)\. We make no significance claim from three lineages; medians and ranges are reported to expose rather than hide this variability\.
This asymmetry motivates the execution\-grounded decomposition\. Goal\-keyed ASR combines exposure with conditional compliance and can vary with which attack goals convert in a particular lineage \(§[4\.2](https://arxiv.org/html/2608.17684#S4.SS2)\)\. Exposure and unauthorized state change are measured directly from the interaction and from account state, and here their SkillOpt shifts are the more directionally consistent\. An audit reporting only ASR would have missed these more directionally consistent changes\.
### 4\.4\.Beyond target ASR: execution\-grounded financial harm
Goal\-keyed ASR asks only whether the attacker’s nominal objective was met\. A rollout can change financial state without satisfying that objective—for instance by moving money to the wrong recipient, or by altering a scheduled payment the attacker was not targeting\. Because the Banking environment exposes terminal account state, our audit detects such effects directly\.
Table 6\.Goal\-keyed attack success versus unauthorized financial\-state change\. Every ASR hit here also changed state, so “ASR hits” and “State chg\., not ASR” partition “Unauth\. state”\. “Extra / ASR” is a relative increase over goal\-keyed successes, not a percentage\-point difference\.∗Literal port\.†Post\-hocarm\. Counts are rollouts, not monetary amounts; harm categories are not commensurable and are never summed into a loss figure\.
Table[6](https://arxiv.org/html/2608.17684#S4.T6)shows that state\-based auditing identifies additional harmful rollouts in every condition\. Static has 201 goal\-keyed successes but 236 rollouts with unauthorized state change, of which 35 occur without goal\-keyed success \(\+17\.4%\+17\.4\\%relative to ASR\)\. SkillOpt adds 62 beyond its 212 \(\+29\.2%\+29\.2\\%\), the largest relative undercount of the five conditions\. AWM\-LiteralPort is the opposite extreme \(\+11\.4%\+11\.4\\%\), consistent with its low execution rate \(§[4\.5](https://arxiv.org/html/2608.17684#S4.SS5)\)\.
Table 7\.Financial harm by category, attacked endpoint \(rate per attacked episode\)\. Categories may overlap within a rollout: they are neither additive nor commensurable and are never summed into a loss figure\. SkillOptn=400n\{=\}400, all othersn=405n\{=\}405\.∗Literal port\.†Post\-hocarm\.
Table[7](https://arxiv.org/html/2608.17684#S4.T7)breaks these outcomes down by harm type\. Among the state\-grounded categories, SkillOpt’s increases occur only in unauthorized transfer, unauthorized payment modification, and unauthorized account/password change; amount manipulation and recipient substitution both fall\. Sensitive\-data disclosure, which is goal\-checker\-derived rather than state\-derived, is near\-flat \(0\.304 to 0\.307\)\. The harm composition shifts rather than rising uniformly\.
The aggregate reinforces the distinction: unauthorized state change rises from 0\.583 under Static to 0\.685 under SkillOpt, a10\.210\.2\-point gap against a3\.43\.4\-point ASR gap\. Goal\-keyed ASR alone would not reveal the larger increase in unauthorized financial\-state outcomes\.
### 4\.5\.An execution\-interface mismatch
The frozen AWM arm,AWM\-LiteralPort, collapses: utility 0\.319, net−57\-57transitions, exposure 0\.299, ASR 0\.195\. Read naively this says workflow memory is catastrophic for capability and unusually safe\. Both readings are wrong\.
Our AWM port reproduces the authors’ released WebArena trajectory serialization, including its <think\>/<action\> envelope\([10](https://arxiv.org/html/2608.17684#bib.bib3)\)\. WebArena agents emit formatted textual actions such as click \[id\]\([18](https://arxiv.org/html/2608.17684#bib.bib17)\); our executor instead uses native function calling, so such textual actions are inert\. The diagnostic is exact: of 540 AWM\-LiteralPort rollouts, 380 emitted a textual<action\>block \(0\.704\) and*the same*380 executed zero real tools \(0\.704\), with a median of zero tool calls\. No other condition emits a single textual action\.
Because the envelope is part of the frozen induced artifact, its role could be tested only in apost\-hocarm,AWM\-InterfaceAdapted\. That arm reuses the same three workflows, with no new acquisition, re\-induction, rewording, reordering or added retrieval, and removes only occurrences of the four literal tag strings, preserving every non\-tag byte\. Thirty content\-equivalence assertions, checked before the adapted arm ran, cover step sequence and count, tool names in order, numeric values and placeholders\.
Removing 24–28 tag occurrences per workflow while preserving all non\-tag content moves utility from 0\.319 to 0\.756, exposure from 0\.299 to 0\.909, ASR from 0\.195 to 0\.575, and unauthorized state change→0\.7140\.217\\\!\\to\\\!0\.714\(Table[8](https://arxiv.org/html/2608.17684#S4.T8)\); textual\-action emission goes to zero and the zero\-tool rate falls to 0\.046, below Static’s 0\.131\.
Table 8\.Interface diagnostic\. LiteralPort and InterfaceAdapted differ only by removal of the four tag types; all non\-tag workflow content is identical\.†Post\-hocarm\. Execution diagnostics span 540 rollouts; the placebo arm has 270 \(benign plus the three\-goal subset\), so its zero\-tool rate and mean\-tool\-call figures are not comparable and are omitted\. In LiteralPort the textual\-action and zero\-tool rates coincide exactly: all 380/540 rollouts emitting a textual action executed no real tool\.
Two conclusions follow\. First, the literal port’s low ASR primarily reflectsfunctional inactivity rather than demonstrated robustness\. Its aggregate cASR is 0\.653, the highest of the evaluated conditions: once attacker content is visible it is not unusually resistant\. Second, interface compatibility dominated the observed AWM behaviour here; we do not claim it dominates AWM in general\. An audit reporting only end\-point utility and ASR would have recorded a severe capability regression and an apparent security improvement, mischaracterising both\.
The adapted arm does not establish a robust capability gain: 0\.756 versus 0\.741, 18 gains against 16 regressions \(\+2\+2net\), and the highest unauthorized state\-change rate in Table[3](https://arxiv.org/html/2608.17684#S4.T3)\. The AWM\-matched placebo, which is generic prose1\.81\.8–2\.4×2\.4\\times*longer*than the real workflow, emits no textual actions and remains near Static in utility \(0\.704\), so prompt length alone does not explain the collapse\.
### 4\.6\.Sensitivity and robustness checks
Are the headline results driven by families with repaired predicates?We recompute the headline aggregate metrics on the 12 families whose official checkers we never modified \(Table[9](https://arxiv.org/html/2608.17684#S4.T9)\); no model is re\-run; the same frozen rollouts are simply re\-aggregated over the subset\. Static utility is higher there \(0\.843 vs 0\.741\): the three repaired families are substantially harder under the corrected predicates\. The qualitative pattern survives—SkillOpt still gains utility while exposure, ASR and unauthorized state change all rise; ReasoningBank gains more utility with exposure and ASR slightly*below*Static; AWM\-LiteralPort still collapses—though effect sizes vary substantially\. The notable exception is the post\-hoc adapted AWM arm, whose\+1\.5\+1\.5\-point utility edge becomes−5\.6\-5\.6points, reinforcing our reading that it gives no robust capability gain \(§[4\.5](https://arxiv.org/html/2608.17684#S4.SS5)\)\.
Table 9\.Untouched\-checker sensitivity: the same frozen rollouts re\-aggregated over the 12 families whose official predicates required no correction \(n=108n\{=\}108benign,324324attacked per condition; SkillOpt 319 attacked\)\.Δ\\Deltais versus Static within this subset; compare Table[3](https://arxiv.org/html/2608.17684#S4.T3)for the 15\-family suite\.†Post\-hocarm\.
Placebo controls\.Against method\-matched generic\-state placebos at the same insertion position \(Table[10](https://arxiv.org/html/2608.17684#S4.T10)\), SkillOpt and ReasoningBank stay above their placebos on utility \(\+7\.4\+7\.4,\+16\.3\+16\.3points\) and both carry*higher*ASR than their placebos on the frozen three\-goal subset \(\+3\.7\+3\.7,\+5\.9\+5\.9\)\. These are not causal estimates, because realised lengths match only approximately: the SkillOpt and AWM placebos are1\.81\.8–2\.5×2\.5\\timeslonger than the corresponding real state, the ReasoningBank placebo0\.590\.59–0\.74×0\.74\\timesas long\. We treat them as structure\- and position\-matched sensitivity controls, not length\-matched counterfactuals\.
Table 10\.Method\-matched placebo contrasts\. ASR columns use the frozen three\-goal placebo subset, goals 4, 6, 7\. “Len\. ratio” is realised placebo/real state length across lineages\.A second executor bounds what is measurable\.A post\-hoc transfer check applies the three frozen Qwen SkillOpt artifacts, unmodified, to DeepSeek V4 Flash \(goals 4, 6, and 7;v3; three lineages; no re\-evolution\): an executor\-sensitivity check, not a pipeline replication\. It isceiling\-limited\. DeepSeek Static already exposes the injection in135/135135/135attacked rollouts, leaving no room to detect an increase, and the transferred condition exposes134/135134/135\. On those same rollouts DeepSeek Static issues 5\.00 tool calls each with a read call in 100% of them, above Qwen SkillOpt on the same three goals \(3\.44 calls\)\. The read\-heavy pathway implicated in the Qwen result is already saturated, so the check cannot test whether that exposure increase reproduces on a second executor: baseline behaviour can make a positive drift unobservable\.
### 4\.7\.Failure Analysis and Verification
Failure\-derived memory\.ReasoningBank writes memories from failed episodes by design; 5–6 of its 22 entries per lineage are failure\-derived\. Among benign endpoint cases, success\-derived retrievals \(n=101n=101\) achieve 0\.931 accuracy with no W→\\rightarrowW cases, whereas failure\-derived retrievals \(n=34n=34\) achieve 0\.647 with all 10 persistent\-failure cases\. This association is descriptive: retrieval is similarity\-based, so queries surfacing failure\-derived memories may simply be harder\.
Pre\-specified qualitative cases\.Before observing outcomes, we fixed a rule selecting the largest new high\-severity failure, largest C→\\rightarrowW regression, and largest W→\\rightarrowC improvement from the frozen records\. The resulting cases mirror the aggregate findings: SkillOpt’s added lookup exposes injected content before multiple unauthorized state changes; AWM\-LiteralPort suppresses real tool execution through its incompatible action envelope; and ReasoningBank converts a repeatedly failed banking task into successful execution\. These cases are explanatory rather than statistical evidence\.
Failure accounting and verification\.Five SkillOpt endpoint calls were provider\-side guardrail rejections caused by repetitive tool\-call loops, all within one family/lineage cell\. Following the frozen infrastructure\-failure policy, they are excluded from the primary denominator; counting them as agent failures changes SkillOpt ASR from 0\.530 to 0\.523 and does not alter the conclusions\. Malformed tool arguments are scored as agent failures\.
Every finalised rollout was independently replayed by a separate implementation that reconstructs the executed tool calls and recomputes utility, attack success, exposure, terminal state, the five state\-grounded harm categories, and goal\-checker\-derived disclosure\. Across 2,965 Qwen main\-and\-placebo endpoint rollouts, 540 adapted\-AWM rollouts, and 360 cross\-executor cases \(270 attacked and 90 benign\), we observe zero semantic mismatches\.
## 5\.Discussion
Three distinct phenomena emerge from this audit, complementing broader evidence that self\-evolving agents can acquire unintended risks\([8](https://arxiv.org/html/2608.17684#bib.bib7)\)\. None is diagnosed by post\-evolution accuracy alone\.
Exposure drift\.For SkillOpt, more proactive information access co\-occurs with both higher benign utility and greater contact with attacker\-controlled content, while*aggregate*conditional susceptibility decreases\. An evaluation reporting only accuracy would see a capability gain; one reporting only aggregate ASR would see a small pooled rise while obscuring larger shifts in exposure and unauthorized state\. The lineage analysis sharpens this: exposure and unauthorized state change rise in all three lineages, whereas ASR agrees in direction in only two \(§[4\.3](https://arxiv.org/html/2608.17684#S4.SS3)\)\. Of these security effects ASR is the least directionally consistent here—an argument for measuring attack\-surface contact and financial state directly\.
Failure persistence\.Under ReasoningBank, every persistent\-failure case we observed followed retrieval of a failure\-derived memory\. This is an association, not a cause, since retrieval provenance is confounded with task difficulty; it nevertheless makes memory provenance and write eligibility explicit audit targets\.
Execution\-interface mismatch\.An artifact learned under one action representation can be operationally incompatible with the deployment executor, and the resulting inactivity reads as capability failure and improved security at once\. It is especially salient in evolve\-then\-deploy: the artifact is itself part of that interface\.
Together these suggest evaluating a self\-evolving agent as the closed loop of §[1](https://arxiv.org/html/2608.17684#S1)rather than at the accuracy endpoint alone\. We present them as phenomena observed in one audit, not a universal taxonomy\.
## 6\.Limitations
We study one primary executor \(qwen3\.7\-flash\-2026\-07\-15\) in one simulated banking environment, so effect sizes should not be generalised to other models or to production systems\. We use three lineages and claim no statistical significance\. Evolution isone offline benign pass, so nothing here speaks to multi\-round dynamics or progressive degradation\. AWM\-InterfaceAdapted and the DeepSeek transfer check arepost\-hoc, chosen after seeing the corresponding frozen outcomes; the latter is also ceiling\-limited\. The unseen families are easier than the seen ones, so we make no transfer claim\.
Exposure is executor\-dependent: an injection that is structurally reachable, and was verified as such before any model call, need not become model\-visible if the agent never issues the relevant read\. Placebos are structure\- and position\-matched but imperfectly length\-matched, so placebo contrasts are sensitivity checks rather than causal estimates\. ReasoningBank uses an adapted embedding substrate\. Becausev3feeds both the benign and the attacked endpoint, their sampling noise is not independent\. The environment lacks scheduled\-payment deletion or cancellation semantics, leaving that harm unobservable\. Failure\-derived memory is confounded by task difficulty, and the qualitative cases are explanatory rather than statistical\.
Finally, the three evolution systems differ jointly in write eligibility, admission, update mechanism, state representation and retrieval, so their end\-to\-end differences cannot be attributed to representation alone\.
## 7\.Conclusion
Benign self\-evolution does not guarantee safe improvement\. On a corrected AgentDojo Banking suite, one system gained 9\.6 points of utility while expanding attack exposure by 12\.3 points and unauthorized financial state change by 10\.2—each positive in all three independently evolved lineages, while the attack\-success delta is positive in only two of three\. Another gained more utility without increasing aggregate attack success, and a third revealed that an artifact incompatible with the execution interface can masquerade as both a capability failure and a safety success\. The primary directional findings persist on the subset of families whose official checkers we never modified\. Financial\-agent evolution should therefore be audited end to end—for task regressions, attack\-surface contact, unauthorized financial\-state change, and compatibility between learned artifacts and execution interfaces—not by accuracy alone\. Code and audit artifacts will be released upon publication\.
## References
- Debenedettiet al\.\(2024\)E\. Debenedetti, J\. Zhang, M\. Balunovic, L\. Beurer\-Kellner, M\. Fischer, and F\. TramèrAgentDojo: a dynamic environment to evaluate prompt injection attacks and defenses for LLM agents\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 82895–82920\.Note:Datasets and Benchmarks TrackExternal Links:[Document](https://dx.doi.org/10.52202/079017-2636)Cited by:[§1](https://arxiv.org/html/2608.17684#S1.p3.1),[§1](https://arxiv.org/html/2608.17684#S1.p5.1),[§2](https://arxiv.org/html/2608.17684#S2.p2.1),[§3\.1](https://arxiv.org/html/2608.17684#S3.SS1.p1.1)\.
- Greshakeet al\.\(2023\)K\. Greshake, S\. Abdelnabi, S\. Mishra, C\. Endres, T\. Holz, and M\. FritzNot what you’ve signed up for: compromising real\-world LLM\-integrated applications with indirect prompt injection\.External Links:2302\.12173Cited by:[§1](https://arxiv.org/html/2608.17684#S1.p3.1),[§2](https://arxiv.org/html/2608.17684#S2.p2.1)\.
- Islamet al\.\(2023\)P\. Islam, A\. Kannappan, D\. Kiela, R\. Qian, N\. Scherrer, and B\. VidgenFinanceBench: a new benchmark for financial question answering\.External Links:2311\.11944Cited by:[§2](https://arxiv.org/html/2608.17684#S2.p3.1)\.
- Jiaet al\.\(2025\)F\. Jia, T\. Wu, X\. Qin, and A\. SquicciariniThe task shield: enforcing task alignment to defend against indirect prompt injection in LLM agents\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 29680–29697\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1435)Cited by:[§2](https://arxiv.org/html/2608.17684#S2.p2.1)\.
- Madaanet al\.\(2023\)A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang, S\. Gupta, B\. P\. Majumder, K\. Hermann, S\. Welleck, A\. Yazdanbakhsh, and P\. ClarkSelf\-refine: iterative refinement with self\-feedback\.InAdvances in Neural Information Processing Systems,Vol\.36\.External Links:[Document](https://dx.doi.org/10.52202/075280-2019)Cited by:[§1](https://arxiv.org/html/2608.17684#S1.p1.1)\.
- Ouyanget al\.\(2026a\)S\. Ouyang, J\. Yan, Y\. Chen, R\. Han, Z\. Wang, B\. Dalvi Mishra, R\. Meng, C\. Li, Y\. Jiao, K\. Zha, M\. Shen, V\. Tirumalashetty, G\. Lee, J\. Han, T\. Pfister, and C\. LeeSkillOS: learning skill curation for self\-evolving agents\.External Links:2605\.06614Cited by:[§2](https://arxiv.org/html/2608.17684#S2.p1.1)\.
- Ouyanget al\.\(2026b\)S\. Ouyang, J\. Yan, I\. Hsu, Y\. Chen, K\. Jiang, Z\. Wang, R\. Han, L\. Le, S\. Daruki, X\. Tang, V\. Tirumalashetty, G\. Lee, M\. Rofouei, H\. Lin, J\. Han, C\. Lee, and T\. PfisterReasoningBank: scaling agent self\-evolving with reasoning memory\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.17684#S1.p1.1)\.
- Shaoet al\.\(2026\)S\. Shao, Q\. Ren, D\. Liu, C\. Qian, B\. Wei, D\. Guo, J\. Yang, X\. Song, L\. Zhang, W\. Zhang, and J\. ShaoYour agent may misevolve: emergent risks in self\-evolving LLM agents\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.17684#S1.p3.1),[§2](https://arxiv.org/html/2608.17684#S2.p1.1),[§5](https://arxiv.org/html/2608.17684#S5.p1.1)\.
- Shinnet al\.\(2023\)N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. YaoReflexion: language agents with verbal reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.36\.External Links:[Document](https://dx.doi.org/10.52202/075280-0377)Cited by:[§1](https://arxiv.org/html/2608.17684#S1.p1.1)\.
- Wanget al\.\(2025\)Z\. Z\. Wang, J\. Mao, D\. Fried, and G\. NeubigAgent workflow memory\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 63897–63911\.Cited by:[§1](https://arxiv.org/html/2608.17684#S1.p1.1),[§4\.5](https://arxiv.org/html/2608.17684#S4.SS5.p2.1)\.
- Xieet al\.\(2024\)Q\. Xie, W\. Han, Z\. Chen, R\. Xiang, X\. Zhang, Y\. He, M\. Xiao, D\. Li, Y\. Dai, D\. Feng, Y\. Xu, H\. Kang, Z\. Kuang, C\. Yuan, K\. Yang, Z\. Luo, T\. Zhang, Z\. Liu, G\. Xiong, Z\. Deng, Y\. Jiang, Z\. Yao, H\. Li, Y\. Yu, G\. Hu, J\. Huang, X\. Liu, A\. Lopez\-Lira, B\. Wang, Y\. Lai, H\. Wang, M\. Peng, S\. Ananiadou, and J\. HuangFinBen: a holistic financial benchmark for large language models\.InAdvances in Neural Information Processing Systems,Vol\.37\.Note:Datasets and Benchmarks TrackExternal Links:[Document](https://dx.doi.org/10.52202/079017-3033)Cited by:[§2](https://arxiv.org/html/2608.17684#S2.p3.1)\.
- Xieet al\.\(2023\)Q\. Xie, W\. Han, X\. Zhang, Y\. Lai, M\. Peng, A\. Lopez\-Lira, and J\. HuangPIXIU: a comprehensive benchmark, instruction dataset and large language model for finance\.InAdvances in Neural Information Processing Systems,Vol\.36\.Note:Datasets and Benchmarks TrackCited by:[§2](https://arxiv.org/html/2608.17684#S2.p3.1)\.
- Yanget al\.\(2026\)Y\. Yang, Z\. Gong, W\. Huang, Q\. Yang, Z\. Zhou, Z\. Huang, Y\. Li, X\. Gao, Q\. Dai, B\. Liu, K\. Qiu, Y\. Yang, D\. Chen, X\. Yang, and C\. LuoSkillOpt: executive strategy for self\-evolving agent skills\.External Links:2605\.23904Cited by:[§1](https://arxiv.org/html/2608.17684#S1.p1.1)\.
- Yuet al\.\(2024\)Y\. Yu, Z\. Yao, H\. Li, Z\. Deng, Y\. Jiang, Y\. Cao, Z\. Chen, J\. W\. Suchow, Z\. Cui, R\. Liu, Z\. Xu, D\. Zhang, K\. Subbalakshmi, G\. Xiong, Y\. He, J\. Huang, D\. Li, and Q\. XieFinCon: a synthesized LLM multi\-agent system with conceptual verbal reinforcement for enhanced financial decision making\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Document](https://dx.doi.org/10.52202/079017-4354)Cited by:[§2](https://arxiv.org/html/2608.17684#S2.p3.1)\.
- Zhanet al\.\(2024\)Q\. Zhan, Z\. Liang, Z\. Ying, and D\. KangInjecAgent: benchmarking indirect prompt injections in tool\-integrated large language model agents\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 10471–10506\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.624)Cited by:[§1](https://arxiv.org/html/2608.17684#S1.p3.1),[§2](https://arxiv.org/html/2608.17684#S2.p2.1)\.
- Zhanget al\.\(2025\)H\. Zhang, J\. Huang, K\. Mei, Y\. Yao, Z\. Wang, C\. Zhan, H\. Wang, and Y\. ZhangAgent security bench \(ASB\): formalizing and benchmarking attacks and defenses in LLM\-based agents\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.17684#S2.p2.1)\.
- Zhanget al\.\(2024\)Z\. Zhang, S\. Cui, Y\. Lu, J\. Zhou, J\. Yang, H\. Wang, and M\. HuangAgent\-safetybench: evaluating the safety of LLM agents\.External Links:2412\.14470Cited by:[§2](https://arxiv.org/html/2608.17684#S2.p2.1)\.
- Zhouet al\.\(2024\)S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried, U\. Alon, and G\. NeubigWebArena: a realistic web environment for building autonomous agents\.InThe Twelfth International Conference on Learning Representations,Cited by:[§4\.5](https://arxiv.org/html/2608.17684#S4.SS5.p2.1)\.Similar Articles
Anyone else struggling with AI auditability?
The author describes a challenge with AI auditability where an agent's decision lacked traceability to the active policy version, and asks for advice on building effective decision trails for AI agent decisions.
When an agent documents its own audit log, things get weird
The author discusses a failure mode encountered while building Sentience Governor, a Python library for Claude Code that monitors agent actions and produces audit reports. The AI sometimes reconstructed explanations from raw traces, blurring the line between measured facts and probabilistic interpretation.
How are you handling audit and compliance for agentic systems in your org?
A practitioner details the real-world challenges of audit, compliance, and governance for autonomous AI agents, including identity, approvals, logging, and accountability, while asking the community for solutions.
SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditing
SkillAudit introduces a framework for evolving LLM agent skills without ground-truth feedback by using paired trajectory auditing and contrastive evaluation. It achieves 73.9% average task reward across 89 tasks, outperforming baseline methods.
From Accuracy to Auditability: A Survey of Determinism in Financial AI Systems
This survey examines computational nondeterminism in financial AI systems, covering tabular models, graph networks, and LLM-based workflows, and proposes a layered evaluation framework for auditability.