Trustworthy Agentic AI: A Comprehensive Cybersecurity and Systems Survey on Threat Landscapes, Defense Architectures, and Open Challenges

# Trustworthy Agentic AI: A Comprehensive Cybersecurity and Systems Survey on Threat Landscapes, Defense Architectures, and Open Challenges
Source: [https://arxiv.org/html/2609.13731](https://arxiv.org/html/2609.13731)
Seyedakbar Mostafavi††thanks:Seyedakbar Mostafavi is with the Department of Computer Engineering, Yazd University, Yazd 89195\-741, Iran \(e\-mail: a\.mostafavi@yazd\.ac\.ir\)\.††thanks:Manuscript received August 31, 2026\. Corresponding author: Seyedakbar Mostafavi\.

###### Abstract

The transition from passive foundation models to autonomous, goal\-directedAgentic AI systemshas introduced unprecedented capabilities by coupling recursive cognitive reasoning loops \(e\.g\., ReAct, Tree\-of\-Thoughts, Reflexion\), hierarchical memory architectures, live tool execution planes \(Model Context Protocol, APIs, shell interpreters\), and distributed multi\-agent collaboration topologies\. However, granting probabilistic neural reasoning cores execution authority across filesystems, networks, and cloud infrastructure dissolves classical security perimeters\. In agentic systems, natural language simultaneously serves as input data, internal control code, and communication protocols, exposing a Turing\-complete blast radius where untrusted data represents executable instructions\.

This survey delivers the first comprehensive systems\-security reference framework forTrustworthy Agentic AI, synthesizing 206 foundational studies and regulatory standards\. We formalize the general agent architecture as a stateful 5\-tuple and establish a6\-Dimensional Trustworthiness TaxonomyencompassingSecurity, Safety & Operational Robustness, Privacy, Explainability & Verifiability, Fairness, and Accountability\. We systematically analyze full\-spectrum threat surfaces across intra\-execution loops \(prompt injections, reasoning backdoors, hallucination cascades, tool parameter RCE, SSRF\) and interaction planes \(indirect prompt injections, vector database poisoning, embedding inversion, cascading swarm failures, Byzantine consensus subversion\)\. To mitigate these threats, we formulate a multi\-layeredZero\-Trust Defense\-in\-Depth Architectureintegrating Dual\-LLM Inspector\-Executor isolation, Capability\-Based Access Control \(CapBAC\), kernel\-level eBPF syscall probes, micro\-VM sandboxing, signed vector embeddings, and Byzantine fault tolerance\. Finally, we review standardized evaluation benchmarks \(InjecAgent, AgentBench, CyberSecEval\), map technical controls to international AI regulations \(NIST AI RMF, EU AI Act, ISO/IEC 42001\), and delineate high\-priority open research frontiers\.

###### Index Terms:

Agentic AI, Trustworthy AI, Cybersecurity, Threat Modeling, Indirect Prompt Injection, Memory Poisoning, Tool\-Use Security, Zero\-Trust Architecture, Multi\-Agent Systems, AI Governance, Formal Verification\.

## IIntroduction

The artificial intelligence landscape is undergoing a monumental paradigm shift from static foundation models that generate tokens within isolated conversational dialogs toautonomous, goal\-directed Agentic AI systems\[[1](https://arxiv.org/html/2609.13731#bib.bib1),[2](https://arxiv.org/html/2609.13731#bib.bib2),[3](https://arxiv.org/html/2609.13731#bib.bib3)\]\. Unlike conversational large language models \(LLMs\) that operate as passive text processors, Agentic AI systems synthesize recursive cognitive reasoning loops \(such as ReAct\[[4](https://arxiv.org/html/2609.13731#bib.bib4)\], Tree\-of\-Thoughts\[[5](https://arxiv.org/html/2609.13731#bib.bib5)\], Graph\-of\-Thoughts\[[6](https://arxiv.org/html/2609.13731#bib.bib6)\], and Reflexion\[[7](https://arxiv.org/html/2609.13731#bib.bib7)\]\), persistent multi\-tier memory stores\[[8](https://arxiv.org/html/2609.13731#bib.bib8)\], and live tool actuation planes\[[9](https://arxiv.org/html/2609.13731#bib.bib9),[10](https://arxiv.org/html/2609.13731#bib.bib10),[11](https://arxiv.org/html/2609.13731#bib.bib11)\]\. By autonomously decomposing abstract objectives, querying external databases, selecting APIs, executing shell commands, and collaborating within multi\-agent networks\[[12](https://arxiv.org/html/2609.13731#bib.bib12),[13](https://arxiv.org/html/2609.13731#bib.bib13)\], agentic systems are revolutionizing software engineering, automated cyber defense, scientific discovery, and enterprise automation\[[14](https://arxiv.org/html/2609.13731#bib.bib14),[15](https://arxiv.org/html/2609.13731#bib.bib15)\]\.

1\. Statistical ML Era \(Pre\-2018\)Static Decision Boundaries, Supervised Classifiers2\. Foundation Model Era \(2018–2023\)Passive Linguistic Synthesis, Stateless Chat Dialogs3\. Agentic AI Era \(2023–Present\)Autonomous Agency, Persistent Memory, Live Tool Actuation

Fig\. 1:Historical Evolution of Artificial Intelligence Paradigms\.However, granting probabilistic neural engines autonomous execution authority across physical and digital environments shatters classical security perimeters\[[16](https://arxiv.org/html/2609.13731#bib.bib16),[17](https://arxiv.org/html/2609.13731#bib.bib17)\]\. In traditional software architectures, strict separation is enforced between untrusted input data and executable control logic, implemented through hardware\-enforced Data Execution Prevention,W⊕XW\\oplus Xmemory management policies, and strictly parameterized database interfaces\[[18](https://arxiv.org/html/2609.13731#bib.bib18),[19](https://arxiv.org/html/2609.13731#bib.bib19)\]\. In Agentic AI, this fundamental boundary collapses entirely: natural language simultaneously serves as the user input, the internal program logic, and the inter\-component communication protocol:

Input Data≡Control Logic≡Communication Protocol=Natural Language\\begin\{split\}\\text\{Input Data\}&\\equiv\\text\{Control Logic\}\\\\ &\\equiv\\text\{Communication Protocol\}=\\text\{Natural Language\}\\end\{split\}\(1\)
Consequently, any untrusted data retrieved by an agent from a webpage, an email, an external database record, or a peer agent represents arbitrary, uncompiled executable code\[[20](https://arxiv.org/html/2609.13731#bib.bib20),[21](https://arxiv.org/html/2609.13731#bib.bib21)\]\. When an agent is empowered with tool execution capabilities, an adversarial input can trigger unauthorized remote code execution, sensitive data exfiltration, or cloud infrastructure compromise at machine speed\[[22](https://arxiv.org/html/2609.13731#bib.bib22),[23](https://arxiv.org/html/2609.13731#bib.bib23),[24](https://arxiv.org/html/2609.13731#bib.bib24),[25](https://arxiv.org/html/2609.13731#bib.bib25)\]\.

### I\-ATaxonomy of Agent Execution Archetypes

Modern Agentic AI manifests across several distinct operational archetypes, each characterized by specialized execution environments, tool integration patterns, and operational failure surfaces\. In software engineering, autonomous agents such as SWE\-agent\[[14](https://arxiv.org/html/2609.13731#bib.bib14)\], ChatDev\[[26](https://arxiv.org/html/2609.13731#bib.bib26)\], and MetaGPT\[[13](https://arxiv.org/html/2609.13731#bib.bib13)\]ingest complex repository issue descriptions, navigate multi\-file codebases, formulate reproduction scripts, edit source files, and execute compilation and test suites via interactive bash shells\. These agents operate with broad filesystem read\-write privileges, making any divergence in planning directly dangerous to repository integrity\.

In interactive web environments, agents such as WebGPT\[[27](https://arxiv.org/html/2609.13731#bib.bib27)\]and Mind2Web\[[15](https://arxiv.org/html/2609.13731#bib.bib15)\]parse Document Object Model \(DOM\) trees, formulate multi\-step browser actions \(including form submission, authentication handshakes, and page navigation\), and interact with commercial and enterprise web portals\. These systems operate in inherently untrusted execution contexts where adversarial third\-party web content can easily poison the agent’s perceptual stream\.

In cybersecurity operations, autonomous agents are increasingly deployed for dual\-use missions, ranging from automated vulnerability discovery, penetration testing, and exploit chain synthesis to real\-time security operations center \(SOC\) incident triage and automated patch verification\[[28](https://arxiv.org/html/2609.13731#bib.bib28),[29](https://arxiv.org/html/2609.13731#bib.bib29),[17](https://arxiv.org/html/2609.13731#bib.bib17)\]\. Concurrently, cyber\-physical and embodied robotics agents translate high\-level natural language intentions into low\-level motor primitives for robotic arms, autonomous vehicles, and aerial drones \(e\.g\., SayCan\[[30](https://arxiv.org/html/2609.13731#bib.bib30)\], SurrealDriver\[[31](https://arxiv.org/html/2609.13731#bib.bib31)\], VOYAGER\[[32](https://arxiv.org/html/2609.13731#bib.bib32)\]\)\. Finally, enterprise workflow orchestration swarms \(such as AutoGen\[[12](https://arxiv.org/html/2609.13731#bib.bib12)\], CAMEL\[[33](https://arxiv.org/html/2609.13731#bib.bib33)\], and AgentVerse\[[34](https://arxiv.org/html/2609.13731#bib.bib34)\]\) distribute complex organizational tasks across specialized agents, coordinating supply chain optimization, contract auditing, and multi\-departmental business intelligence\.

### I\-BThe Cybersecurity Dilemma in Agentic Autonomy

We formalize this fundamental security crisis asThe Agentic Cybersecurity Dilemma, which emerges from three compounding structural tensions between agent capability and system controllability\.

Turing\-Complete Blast Radius\.By equipping foundation models with interactive shell interpreters, persistent filesystem access, cloud APIs, and standardized Model Context Protocol \(MCP\) servers\[[11](https://arxiv.org/html/2609.13731#bib.bib11),[14](https://arxiv.org/html/2609.13731#bib.bib14)\], the blast radius of a successful prompt injection expands exponentially\. Whereas attacking a conversational model results merely in toxic text generation, compromising an autonomous agent leads directly to arbitrary host takeover, remote code execution, and cloud privilege escalation across enterprise perimeters\[[21](https://arxiv.org/html/2609.13731#bib.bib21),[35](https://arxiv.org/html/2609.13731#bib.bib35)\]\.

Probabilistic and Non\-Deterministic Control Flow\.Classical cybersecurity defenses rely upon deterministic state machines, static code analysis, and mathematically verifiable authorization invariants\[[36](https://arxiv.org/html/2609.13731#bib.bib36),[37](https://arxiv.org/html/2609.13731#bib.bib37)\]\. In stark contrast, the cognitive core of an agent is inherently stochastic:

at∼P⁡\(at∣Ht−1,ot−1;θ\)a\_\{t\}\\sim P\(a\_\{t\}\\mid H\_\{t\-1\},o\_\{t\-1\};\\theta\)\(2\)Minor semantic variations, sampling temperature fluctuations, or imperceptible prompt perturbations cause identical system states to yield radically different tool dispatch sequences, allowing malicious payloads to bypass static signature\-based filters and runtime heuristic firewalls with high probability\[[38](https://arxiv.org/html/2609.13731#bib.bib38),[39](https://arxiv.org/html/2609.13731#bib.bib39)\]\.

Dual\-Use Machine\-Speed Cyber Warfare\.Autonomous agents drastically compress the operational timeline of cyber engagements\[[17](https://arxiv.org/html/2609.13731#bib.bib17)\]\. While defensive agents can automate security operations center triage, correlate distributed telemetry, and synthesize software patches in real time, adversaries can weaponize identical architectures to discover zero\-day vulnerabilities, synthesize polymorphic exploits, and execute adaptive lateral movement at machine speed, creating an asymmetric offensive advantage\[[28](https://arxiv.org/html/2609.13731#bib.bib28),[29](https://arxiv.org/html/2609.13731#bib.bib29),[16](https://arxiv.org/html/2609.13731#bib.bib16)\]\.

Autonomous Agent𝒜=⟨𝒞,ℳ,𝒯,ℰ,Π⟩\\mathcal\{A\}=\\langle\\mathcal\{C\},\\mathcal\{M\},\\mathcal\{T\},\\mathcal\{E\},\\Pi\\rangleGap 1: Input UnpredictabilityMulti\-step prompt injections, adversarial suffixes, jailbreaksGap 2: Internal ComplexityReasoning drift, hallucination cascades, sleeper backdoorsGap 3: Env\. VariabilitySSRF, sandbox breakout, resource exhaustion DoSGap 4: Untrusted EntitiesRAG vector poisoning, Byzantine peer consensus subversion

Fig\. 2:The Four Critical Knowledge Gaps in Trustworthy Agentic AI Security\.
### I\-CFour Critical Knowledge Gaps in Agentic AI Security

As illustrated in Fig\.[2](https://arxiv.org/html/2609.13731#S1.F2), the vulnerabilities of modern Agentic AI stem from four foundational knowledge gaps that span the perception, reasoning, execution, and communication planes\.

The first gap concerns theUnpredictability of Multi\-Step User Inputs, wherein multi\-turn conversational interactions introduce subtle prompt drift, automated jailbreaks, and goal hijacking attacks that manipulate the agent into violating developer safety policies over extended trajectories\[[40](https://arxiv.org/html/2609.13731#bib.bib40),[41](https://arxiv.org/html/2609.13731#bib.bib41),[42](https://arxiv.org/html/2609.13731#bib.bib42),[43](https://arxiv.org/html/2609.13731#bib.bib43),[44](https://arxiv.org/html/2609.13731#bib.bib44),[45](https://arxiv.org/html/2609.13731#bib.bib45),[46](https://arxiv.org/html/2609.13731#bib.bib46),[47](https://arxiv.org/html/2609.13731#bib.bib47)\]\. The second gap arises from theComplexity of Internal Cognitive Reasoning, where the agent’s internal thought process constitutes a multi\-branch graph structure rather than a simple feedforward pipeline\. Minor hallucinations, semantic drift, or latent sleeper backdoors in intermediate thoughts compound multiplicatively into catastrophic tool execution failures\[[6](https://arxiv.org/html/2609.13731#bib.bib6),[48](https://arxiv.org/html/2609.13731#bib.bib48),[49](https://arxiv.org/html/2609.13731#bib.bib49),[50](https://arxiv.org/html/2609.13731#bib.bib50),[51](https://arxiv.org/html/2609.13731#bib.bib51),[52](https://arxiv.org/html/2609.13731#bib.bib52),[53](https://arxiv.org/html/2609.13731#bib.bib53)\]\.

The third gap involves theVariability of Operational Environments, where deploying agents across diverse virtual filesystems, containerized networks, and cloud infrastructures introduces severe system\-level vulnerabilities, including Server\-Side Request Forgery \(SSRF\), sandbox breakout escapes, and algorithmic resource exhaustion denial\-of\-service\[[54](https://arxiv.org/html/2609.13731#bib.bib54),[55](https://arxiv.org/html/2609.13731#bib.bib55),[56](https://arxiv.org/html/2609.13731#bib.bib56),[57](https://arxiv.org/html/2609.13731#bib.bib57),[58](https://arxiv.org/html/2609.13731#bib.bib58)\]\. Finally, the fourth gap emerges fromInteractions with Untrusted External Entities, wherein interfacing with third\-party web pages, external API providers, Model Context Protocol servers, persistent vector databases, and peer agents exposes the system to Indirect Prompt Injection \(IPI\), dense vector database poisoning, and Byzantine consensus subversion\[[20](https://arxiv.org/html/2609.13731#bib.bib20),[21](https://arxiv.org/html/2609.13731#bib.bib21),[59](https://arxiv.org/html/2609.13731#bib.bib59),[60](https://arxiv.org/html/2609.13731#bib.bib60),[61](https://arxiv.org/html/2609.13731#bib.bib61),[62](https://arxiv.org/html/2609.13731#bib.bib62),[63](https://arxiv.org/html/2609.13731#bib.bib63),[64](https://arxiv.org/html/2609.13731#bib.bib64)\]\.

### I\-DSystematic Survey Methodology & Corpus Identification

To establish an exhaustive, evidence\-backed foundation for this survey, we implemented a rigorous literature review protocol guided by the PRISMA\-S standards for systematic literature syntheses\[[65](https://arxiv.org/html/2609.13731#bib.bib65)\]\. Our bibliographic acquisition process spanned major computer science, cybersecurity, and artificial intelligence repositories, including IEEE Xplore, ACM Digital Library, USENIX Security, NDSS, IEEE S&P, EuroS&P, NeurIPS, ICML, ICLR, ACL, and arXiv\.

Search queries integrated comprehensive boolean combinations across agent architectures, trustworthiness dimensions, and threat vectors:

𝒬=\(Agentic AI∨LLM Agent∨Autonomous Agent\)∧\(Trustworthy∨Security∨Prompt InjectionCLOSEOPEN∨Zero\-Trust∨Memory Poisoning∨Byzantine Consensus\)\\mathcal\{Q\}=\(\\text\{Agentic AI\}\\lor\\text\{LLM Agent\}\\lor\\text\{Autonomous Agent\}\)\\\\ \\land\(\\text\{Trustworthy\}\\lor\\text\{Security\}\\lor\\text\{Prompt Injection\}\\\\ \\lor\\text\{Zero\-Trust\}\\lor\\text\{Memory Poisoning\}\\lor\\text\{Byzantine Consensus\}\)\(3\)
Our rigorous multi\-stage filtering and citation cross\-validation protocol distilled a final reference corpus of206 verified foundational publications, spanning cognitive reasoning formalisms, systems sandboxing, vector memory integrity, and international regulatory governance\.

### I\-EComparison with Related Surveys and Our Contributions

While several existing surveys explore general LLM applications\[[3](https://arxiv.org/html/2609.13731#bib.bib3),[1](https://arxiv.org/html/2609.13731#bib.bib1),[2](https://arxiv.org/html/2609.13731#bib.bib2)\], classical machine learning security\[[66](https://arxiv.org/html/2609.13731#bib.bib66),[67](https://arxiv.org/html/2609.13731#bib.bib67),[68](https://arxiv.org/html/2609.13731#bib.bib68)\], or narrow prompt injection techniques\[[38](https://arxiv.org/html/2609.13731#bib.bib38),[69](https://arxiv.org/html/2609.13731#bib.bib69)\], they fail to provide a unified systems\-security architecture for autonomous agent loops with persistent memory, tool execution, and multi\-agent consensus\. Table[I](https://arxiv.org/html/2609.13731#S1.T1)provides a comprehensive comparison between our survey and related works in the literature\.

TABLE I:Comprehensive Comparison of This Survey with Existing Surveys in AI and Agent SecuritySurveyYearPrimary Focus / ScopeKey CapabilitiesIdentified Limitations / GapsXi et al\.\[[1](https://arxiv.org/html/2609.13731#bib.bib1)\]2023General LLM agentsCognitive architectureOverlooks adversarial threat vectors and systems sandboxing\.Wang et al\.\[[2](https://arxiv.org/html/2609.13731#bib.bib2)\]2024Agent developmentPlanning & evaluationLacks systems security and memory poisoning analysis\.Greshake et al\.\[[20](https://arxiv.org/html/2609.13731#bib.bib20)\]2023Indirect prompt injectionApp vulnerability demoLimited to initial IPI discovery; lacks defense blueprint\.Zhan et al\.\[[21](https://arxiv.org/html/2609.13731#bib.bib21)\]2024Tool\-integrated agentsBenchmarking across toolsEmpirical attack suite; omits multi\-agent and memory planes\.Deng et al\.\[[15](https://arxiv.org/html/2609.13731#bib.bib15)\]2024Web interactive agentsWeb task executionFocuses on task success, not operational security\.Yao et al\.\[[5](https://arxiv.org/html/2609.13731#bib.bib5)\]2023Deliberate planningToT reasoning searchEvaluates reasoning without adversarial threat models\.Ours2026Trustworthy Agentic AI6D Taxonomy, 4\-Layer Zero\-Trust, 206 WorksUnified zero\-trust systems security blueprint with neural\-symbolic verification synthesis\.
### I\-FFormal Survey Research Questions \(RQs\)

To establish an intellectually rigorous investigation into the trustworthiness of autonomous agentic systems, this survey is structured around five foundational Research Questions:

RQ1: Foundational Systems\-Level Formalization\.How can the dynamic closed\-loop interaction among cognitive planning cores, hierarchical memory subsystems, live tool execution runtimes, and multi\-agent network topologies be unified into a mathematically grounded, stateful systems security model?We address this in Section[II](https://arxiv.org/html/2609.13731#S2)by formalizing the agent as a discrete\-time 5\-tuple𝒜=⟨𝒞,ℳ,𝒯,ℰ,Π⟩\\mathcal\{A\}=\\langle\\mathcal\{C\},\\mathcal\{M\},\\mathcal\{T\},\\mathcal\{E\},\\Pi\\rangleembedded within an adversarial POMDP framework\.

RQ2: Multi\-Dimensional Trustworthiness Taxonomy\.What operational dimensions constitute trustworthiness in autonomous agentic ecosystems, and what mathematical formulations define their system invariants and cross\-dimensional Pareto trade\-offs?We resolve this in Section[III](https://arxiv.org/html/2609.13731#S3)through our 6\-Dimensional Trustworthiness Taxonomy and cross\-dimensional invariant analysis\.

RQ3: Full\-Spectrum Vulnerability and Threat Modeling\.What are the exact attack mechanisms, adversary capabilities, and empirical failure modes across the intra\-execution modules \(perception, brain, action\) and interaction surfaces \(environment, persistent memory, peer swarms\)?We dissect these threat landscapes in Section[IV](https://arxiv.org/html/2609.13731#S4)and Section[V](https://arxiv.org/html/2609.13731#S5)\.

RQ4: Zero\-Trust Defensive Systems Engineering\.How can classical systems security principles—including the Principle of Least Privilege, Capability\-Based Access Control \(CapBAC\), kernel\-level eBPF tracing, micro\-VM sandboxing, cryptographic provenance, and Byzantine fault\-tolerant consensus—be synthesized into an actionable Defense\-in\-Depth framework?We formulate this multi\-tiered architecture and its operational trade\-offs in Section[VI](https://arxiv.org/html/2609.13731#S6)\.

RQ5: Empirical Evaluation and Regulatory Compliance\.What empirical benchmarks and quantitative metrics assess agent trustworthiness, and how do technical architectural controls map to international regulatory mandates \(such as the NIST AI RMF 1\.0, EU AI Act, and ISO/IEC 42001\)?We synthesize the empirical benchmark landscape in Section[VII](https://arxiv.org/html/2609.13731#S7), cross\-regulatory enforcement in Section[VIII](https://arxiv.org/html/2609.13731#S8), and chart strategic future frontiers in Section[IX](https://arxiv.org/html/2609.13731#S9)\.

### I\-GCore Contributions of this Survey

In addressing these foundational questions, this work delivers four principal intellectual contributions to the literature:

1\. Foundational Systems\-Level Formalization and Boundary Collapse Framework\.We formalize the general architecture of autonomous agents as a stateful 5\-tuple𝒜=⟨𝒞,ℳ,𝒯,ℰ,Π⟩\\mathcal\{A\}=\\langle\\mathcal\{C\},\\mathcal\{M\},\\mathcal\{T\},\\mathcal\{E\},\\Pi\\rangle, modeling the closed\-loop execution dynamics across cognitive planning, hierarchical memory stores, tool runtimes \(including the Model Context Protocol\), and multi\-agent swarms\. We mathematically couple this with an adversarial POMDP formulation that captures belief\-state divergence under unvalidated environmental observations, establishing the Systems Boundary Collapse Principle where natural language simultaneously operates as untrusted data, executable control logic, and inter\-process communication\.

2\. Operational 6\-Dimensional Trustworthiness Taxonomy and Pareto Dynamics\.Moving beyond static ML benchmark metrics, we formulate an operational, systems\-level taxonomy spanningSecurity \(CIA\), Safety & Operational Robustness, Privacy & Data Protection, Explainability & Verifiability, Fairness & Non\-Discrimination, and Accountability & Provenance\. Each dimension is grounded in formal system invariants, empirical verification methods, and systems controls, accompanied by an explicit analysis of cross\-dimensional Pareto tensions \(security\-latency, privacy\-utility, and safety\-autonomy\)\.

3\. Full\-Spectrum Dual\-Plane Threat Dissection\.We provide an exhaustive, multi\-tier threat analysis that deconstructs attack vectors across intra\-execution planes \(perception prompt injections, intermediate CoT reasoning poisoning, hallucination cascades, and tool parameter RCE/SSRF\) and interaction planes \(untrusted web environments, dense RAG vector poisoning, embedding inversion, Morris II generative worms, and Byzantine swarm subversion\)\.

4\. Zero\-Trust Defense Blueprint with Systems Trade\-off Synthesis\.We formulate an actionable, multi\-layered systems security blueprint synthesizing Dual\-LLM Inspector\-Executor isolation, Capability\-Based Access Control \(CapBAC\), kernel\-level eBPF syscall telemetry, micro\-VM and WebAssembly sandboxing, cryptographically signed memory provenance, and Byzantine fault\-tolerant consensus\. Crucially, we provide a high\-density comparative synthesis matrix analyzing the operational latency penalties, token amplification costs, computational footprints, and cloud/cluster deployability of each defensive mechanism\.

### I\-HPaper Organization

The remainder of this survey is organized as follows: Section[II](https://arxiv.org/html/2609.13731#S2)establishes the architectural foundations and execution workflows of Agentic AI\. Section[III](https://arxiv.org/html/2609.13731#S3)introduces the 6\-Dimensional Trustworthiness Taxonomy\. Section[IV](https://arxiv.org/html/2609.13731#S4)examines intra\-execution security across perception, brain, and action planes\. Section[V](https://arxiv.org/html/2609.13731#S5)analyzes interaction security across environments, memory stores, and multi\-agent swarms\. Section[VI](https://arxiv.org/html/2609.13731#S6)presents the Zero\-Trust Defense\-in\-Depth architecture\. Section[VII](https://arxiv.org/html/2609.13731#S7)reviews empirical benchmarks, red\-teaming frameworks, and quantitative metrics\. Section[VIII](https://arxiv.org/html/2609.13731#S8)analyzes socio\-technical governance and regulatory compliance \(NIST AI RMF, EU AI Act, ISO/IEC 42001\)\. Section[IX](https://arxiv.org/html/2609.13731#S9)delineates open research challenges and future research horizons\. Finally, Section[X](https://arxiv.org/html/2609.13731#S10)concludes the paper\. Table[II](https://arxiv.org/html/2609.13731#S1.T2)summarizes the key abbreviations used throughout the manuscript\.

TABLE II:Summary of Key Abbreviations in Alphabetical OrderAbbreviationDefinitionACLAgent Communications LanguageANNApproximate Nearest NeighborASRAttack Success RateBFT / PBFTByzantine Fault Tolerance / Practical BFTBRIBlast Radius IndexCapBACCapability\-Based Access ControlCFPCascade Failure ProbabilityCoT / ToT / GoTChain\-of\-Thought / Tree\-of\-Thoughts / Graph\-of\-ThoughtsDIDDecentralized Identifier \(W3C Standard\)DPI / IPIDirect Prompt Injection / Indirect Prompt InjectioneBPFExtended Berkeley Packet FilterFSRRFalse Safety Rejection RateHIC / HITL / HOTLHuman\-in\-Command / Human\-in\-the\-Loop / Human\-on\-the\-LoopHNSWHierarchical Navigable Small WorldMCPModel Context ProtocolMCTSMonte Carlo Tree SearchMIAMembership Inference AttackPoLPPrinciple of Least PrivilegeRAGRetrieval\-Augmented GenerationRCERemote Code ExecutionSMTSatisfiability Modulo TheoriesSSRFServer\-Side Request ForgeryTEETrusted Execution EnvironmentUPRUtility Preservation Rate

## IIFoundational Architecture of Agentic AI

### II\-AUnified Conceptual Framework and Formal System Tuple

To systematically analyze vulnerabilities, attack surfaces, and defensive architectures in autonomous agent systems, we formalize an Agentic AI system as a stateful, discrete\-time autonomous 5\-tuple operating over execution stepst∈\{1,2,…,T\}t\\in\\\{1,2,\\dots,T\\\}within an operational environmentℰ\\mathcal\{E\}\[[1](https://arxiv.org/html/2609.13731#bib.bib1),[2](https://arxiv.org/html/2609.13731#bib.bib2)\]:

𝒜=⟨𝒞,ℳ,𝒯,ℰ,Π⟩\\mathcal\{A\}=\\langle\\mathcal\{C\},\\mathcal\{M\},\\mathcal\{T\},\\mathcal\{E\},\\Pi\\rangle\(4\)
Within this formalization,𝒞\\mathcal\{C\}denotes theCognitive Reasoning Coreparameterized by foundation model weightsθ\\theta\(e\.g\., GPT\-4\[[70](https://arxiv.org/html/2609.13731#bib.bib70)\], Gemini\[[71](https://arxiv.org/html/2609.13731#bib.bib71)\], LLaMA 3\[[72](https://arxiv.org/html/2609.13731#bib.bib72)\], Mistral\[[73](https://arxiv.org/html/2609.13731#bib.bib73)\], Claude 3\[[74](https://arxiv.org/html/2609.13731#bib.bib74)\]\), responsible for goal decomposition, intermediate reasoning, and decision synthesis\. TheHierarchical Memory Subsystem, denotedℳ=\{ℳwork,ℳepisodic,ℳsemantic\}\\mathcal\{M\}=\\\{\\mathcal\{M\}\_\{\\text\{work\}\},\\mathcal\{M\}\_\{\\text\{episodic\}\},\\mathcal\{M\}\_\{\\text\{semantic\}\}\\\}, manages information persistence across ephemeral attention windows, dense vector indices, and structured parametric knowledge stores\[[8](https://arxiv.org/html/2609.13731#bib.bib8)\]\.

TheTool and Action Plane,𝒯=\{t1,t2,…,tk\}\\mathcal\{T\}=\\\{t\_\{1\},t\_\{2\},\\dots,t\_\{k\}\\\}, defines the execution interface through which the agent causes external side effects\. Each tooltj=⟨Schemaj,Execj,Permj⟩t\_\{j\}=\\langle\\text\{Schema\}\_\{j\},\\text\{Exec\}\_\{j\},\\text\{Perm\}\_\{j\}\\rangleis defined by a formal parameter schema, an underlying execution runtime, and a security permission envelope\[[9](https://arxiv.org/html/2609.13731#bib.bib9),[10](https://arxiv.org/html/2609.13731#bib.bib10),[11](https://arxiv.org/html/2609.13731#bib.bib11)\]\. TheExternal Operational Environmentℰ\\mathcal\{E\}encompasses filesystems, network sockets, external databases, cloud APIs, human operators, and peer agents\. Finally,Π\\Pirepresents theAgent Execution Policy or Orchestration Protocol, which governs how historical interaction traces and memory states are mapped onto next\-step thoughts, reflections, and tool invocations\[[75](https://arxiv.org/html/2609.13731#bib.bib75)\]\.

### II\-BPartially Observable Markov Decision Process \(POMDP\) Formulation

In real\-world deployments, an autonomous agent does not possess direct access to the complete, underlying system statest∈𝒮s\_\{t\}\\in\\mathcal\{S\}\. Instead, agent execution operates under partial observability, modeled formally as a Partially Observable Markov Decision Process \(POMDP\) defined by the 7\-tuple⟨𝒮,𝒜act,𝒯trans,ℛ,Ω,𝒪,γ⟩\\langle\\mathcal\{S\},\\mathcal\{A\}\_\{\\text\{act\}\},\\mathcal\{T\}\_\{\\text\{trans\}\},\\mathcal\{R\},\\Omega,\\mathcal\{O\},\\gamma\\rangle\. Here,𝒮\\mathcal\{S\}represents the set of hidden ground\-truth system states \(including full operating system memory, remote database tables, and cloud infrastructure states\);𝒜act\\mathcal\{A\}\_\{\\text\{act\}\}represents the discrete or structured action space exposed via tool plane𝒯\\mathcal\{T\};𝒯trans​\(st\+1∣st,at\)\\mathcal\{T\}\_\{\\text\{trans\}\}\(s\_\{t\+1\}\\mid s\_\{t\},a\_\{t\}\)denotes the state transition probability distribution of the environment;ℛ⁡\(st,at\)\\mathcal\{R\}\(s\_\{t\},a\_\{t\}\)is the reward function quantifying goal fulfillment;Ω\\Omegais the set of discrete observations emitted by the environment \(such as stdout text, JSON payloads, HTTP responses, or error codes\);𝒪⁡\(ot\+1∣st\+1,at\)\\mathcal\{O\}\(o\_\{t\+1\}\\mid s\_\{t\+1\},a\_\{t\}\)is the observation probability distribution; andγ∈\[0,1\)\\gamma\\in\[0,1\)is the temporal discount factor\.

Belief State Transition Dynamics\.Because the true statests\_\{t\}is unobservable, the agent maintains an internal belief statebt​\(s\)=Pr⁡\(st=s∣Ht\)b\_\{t\}\(s\)=\\Pr\(s\_\{t\}=s\\mid H\_\{t\}\), representing a probability distribution over states conditioned on the execution historyHt=\(o0,a0,o1,a1,…,ot\)H\_\{t\}=\(o\_\{0\},a\_\{0\},o\_\{1\},a\_\{1\},\\dots,o\_\{t\}\)\. Upon executing actionata\_\{t\}and receiving a new environmental observationot\+1o\_\{t\+1\}, the agent’s belief state evolves according to the recursive Bayesian belief update:

bt\+1​\(s′\)=𝒪⁡\(ot\+1∣s′,at\)​∑s∈𝒮𝒯trans​\(s′∣s,at\)​bt​\(s\)∑s′′∈𝒮𝒪⁡\(ot\+1∣s′′,at\)​∑s∈𝒮𝒯trans​\(s′′∣s,at\)​bt​\(s\)\\begin\{split\}&b\_\{t\+1\}\(s^\{\\prime\}\)=\\\\ &\\frac\{\\mathcal\{O\}\(o\_\{t\+1\}\\mid s^\{\\prime\},a\_\{t\}\)\\sum\_\{s\\in\\mathcal\{S\}\}\\mathcal\{T\}\_\{\\text\{trans\}\}\(s^\{\\prime\}\\mid s,a\_\{t\}\)b\_\{t\}\(s\)\}\{\\sum\_\{s^\{\\prime\\prime\}\\in\\mathcal\{S\}\}\\mathcal\{O\}\(o\_\{t\+1\}\\mid s^\{\\prime\\prime\},a\_\{t\}\)\\sum\_\{s\\in\\mathcal\{S\}\}\\mathcal\{T\}\_\{\\text\{trans\}\}\(s^\{\\prime\\prime\}\\mid s,a\_\{t\}\)b\_\{t\}\(s\)\}\\end\{split\}\(5\)The agent’s policyΠ⁡\(at∣bt\)\\Pi\(a\_\{t\}\\mid b\_\{t\}\)maps this internal belief state into a tool invocationat∈𝒜acta\_\{t\}\\in\\mathcal\{A\}\_\{\\text\{act\}\}to maximize expected cumulative discounted return:

Π∗=argmaxΠ𝔼\[∑t=0Tγtℛ\(st,at\)\|s0∼b0,at∼Π\(⋅∣bt\)\]\\Pi^\{\*\}=\\arg\\max\_\{\\Pi\}\\mathbb\{E\}\\left\[\\sum\_\{t=0\}^\{T\}\\gamma^\{t\}\\mathcal\{R\}\(s\_\{t\},a\_\{t\}\)\\;\\middle\|\\;s\_\{0\}\\sim b\_\{0\},a\_\{t\}\\sim\\Pi\(\\cdot\\mid b\_\{t\}\)\\right\]\(6\)
Adversarial Belief Divergence\.Security vulnerabilities emerge precisely when untrusted external inputs or poisoned observationsoadv∈Ωo\_\{\\text\{adv\}\}\\in\\Omegacorrupt the observation likelihood𝒪⁡\(oadv∣s′,at\)\\mathcal\{O\}\(o\_\{\\text\{adv\}\}\\mid s^\{\\prime\},a\_\{t\}\)\. An adversary injects malicious contextual observations to induce maximum divergence between the true environmental state distribution and the agent’s internal belief distributionbtadvb\_\{t\}^\{\\text\{adv\}\}:

DKL\(bttrue\(s\)∥btadv\(s\)\)≫0⟹arg⁡maxa​𝔼s∼btadv​\[ℛ⁡\(s,a\)\]≠arg⁡maxa​𝔼s∼bttrue​\[ℛ⁡\(s,a\)\]\\begin\{split\}&D\_\{\\text\{KL\}\}\\left\(b\_\{t\}^\{\\text\{true\}\}\(s\)\\parallel b\_\{t\}^\{\\text\{adv\}\}\(s\)\\right\)\\gg 0\\implies\\\\ &\\arg\\max\_\{a\}\\mathbb\{E\}\_\{s\\sim b\_\{t\}^\{\\text\{adv\}\}\}\[\\mathcal\{R\}\(s,a\)\]\\neq\\arg\\max\_\{a\}\\mathbb\{E\}\_\{s\\sim b\_\{t\}^\{\\text\{true\}\}\}\[\\mathcal\{R\}\(s,a\)\]\\end\{split\}\(7\)This divergence steers the policyΠ⁡\(at∣btadv\)\\Pi\(a\_\{t\}\\mid b\_\{t\}^\{\\text\{adv\}\}\)toward catastrophic, unauthorized, or policy\-violating tool actions while maintaining internal model confidence\.

1\. Perception Layer \(Input Formatting & Context Ingestion\) User Prompt\(xuserx\_\{\\text\{user\}\}\)Observations \(ot−1o\_\{t\-1\}\)DOM, ImagesMemory ContextRAG Documents2\. Cognitive Reasoning Core \(𝒞\\mathcal\{C\}\) — The Brain Goal DecompositionHierarchical TasksPlanning LoopsReAct, ToT, GoTSelf\-ReflectionReflexion, RefineDecision MakingTool JSON Schema3\. Tool & Action Plane \(𝒯\\mathcal\{T\}\) — Actuation Model Context Protocol\(MCP Client\)OS Command Shells\(Bash, PowerShell\)Web Automation\(Playwright\)Cloud APIs\(S3, SQL, REST\)4\. Operational Environment \(ℰ\\mathcal\{E\}\) & Multi\-Agent Swarm Ecosystem Filesystems & CloudPeer Agents \(AutoGen, MetaGPT\)Human Operators \(HITL\)Feedback \(ot←ℰ⁡\(at\)o\_\{t\}\\leftarrow\\mathcal\{E\}\(a\_\{t\}\)\)

Fig\. 3:General Architecture and Closed\-Loop Execution Workflow of Agentic AI Systems\.
### II\-CCognitive Reasoning & Planning Paradigms

The cognitive core𝒞\\mathcal\{C\}decomposes complex, high\-level objectives into actionable execution sequences using structured reasoning paradigms that govern how the model reasons, samples actions, and incorporates environmental feedback\.

Linear and Interleaved Reasoning \(CoT and ReAct\)\.In basic sequential architectures, Chain\-of\-Thought \(CoT\) prompting encourages the model to generate intermediate reasoning tokens before emitting a final decision\[[76](https://arxiv.org/html/2609.13731#bib.bib76),[77](https://arxiv.org/html/2609.13731#bib.bib77)\]\. However, pure CoT operates open\-loop without environmental grounding\. To enable closed\-loop execution, the ReAct paradigm interleaves reasoning traces \(t​htth\_\{t\}\) and action dispatches \(ata\_\{t\}\) dynamically\[[4](https://arxiv.org/html/2609.13731#bib.bib4)\]\. At execution steptt, given the prior trajectory historyHt−1=\(t​h1,a1,o1,…,t​ht−1,at−1,ot−1\)H\_\{t\-1\}=\(th\_\{1\},a\_\{1\},o\_\{1\},\\dots,th\_\{t\-1\},a\_\{t\-1\},o\_\{t\-1\}\)and current observationot−1o\_\{t\-1\}, the agent executes:

t​ht\\displaystyle th\_\{t\}∼𝒞⁡\(Ht−1,ot−1,θ\)\\displaystyle\\sim\\mathcal\{C\}\(H\_\{t\-1\},o\_\{t\-1\};\\theta\)\(8\)at\\displaystyle a\_\{t\}∼𝒞⁡\(Ht−1,t​ht,θ\)\\displaystyle\\sim\\mathcal\{C\}\(H\_\{t\-1\},th\_\{t\};\\theta\)\(9\)ot\\displaystyle o\_\{t\}←ℰ⁡\(at\)\\displaystyle\\leftarrow\\mathcal\{E\}\(a\_\{t\}\)\(10\)This interleaved cycle allows the agent to update its planning trajectory dynamically based on real\-time feedback, error codes, and tool outputs\.

Search\-Based Deliberate Planning \(ToT, GoT, and LATS\)\.To tackle complex combinatorial problem domains, deliberate planning generalizes linear thought sequences into search graphs𝒢=\(𝒱,ℰ\)\\mathcal\{G\}=\(\\mathcal\{V\},\\mathcal\{E\}\)\[[5](https://arxiv.org/html/2609.13731#bib.bib5),[6](https://arxiv.org/html/2609.13731#bib.bib6),[48](https://arxiv.org/html/2609.13731#bib.bib48)\]\. Under the Tree\-of\-Thoughts \(ToT\) and Graph\-of\-Thoughts \(GoT\) frameworks, intermediate cognitive statesv∈𝒱v\\in\\mathcal\{V\}are evaluated by heuristic evaluation functionsV⁡\(v\)=𝔼𝒞​\[Task Success∣v\]V\(v\)=\\mathbb\{E\}\_\{\\mathcal\{C\}\}\[\\text\{Task Success\}\\mid v\]using Breadth\-First Search \(BFS\), Depth\-First Search \(DFS\), or Monte Carlo Tree Search \(MCTS\) algorithms with lookahead rollouts\[[78](https://arxiv.org/html/2609.13731#bib.bib78)\]\. While search\-based planning dramatically improves task success in mathematical and software engineering domains, it exponentially expands the attack surface, providing adversarial inputs multiple candidate pathways to bias the search tree\.

Verbal Reinforcement Learning and Episodic Self\-Correction\.To facilitate iterative improvement without parameter fine\-tuning, frameworks such as Reflexion\[[7](https://arxiv.org/html/2609.13731#bib.bib7)\]and Self\-Refine\[[79](https://arxiv.org/html/2609.13731#bib.bib79)\]equip agents with verbal self\-critique mechanisms\. When an action trajectory fails validation checks or unit tests, an auxiliary evaluation model generates a natural language reflectionrt∼𝒞⁡\(Ht,Feedback\)r\_\{t\}\\sim\\mathcal\{C\}\(H\_\{t\},\\text\{Feedback\}\), which is stored in episodic memory to steer subsequent planning iterations\. In embodied environments, grounded planning systems \(such as VOYAGER\[[32](https://arxiv.org/html/2609.13731#bib.bib32)\], DEPS\[[80](https://arxiv.org/html/2609.13731#bib.bib80)\], and SayCan\[[30](https://arxiv.org/html/2609.13731#bib.bib30)\]\) extend this principle by maintaining evolving skill libraries and automated curricula grounded in physical affordances\.

### II\-DHierarchical Memory Subsystems

To overcome the finite context window limits of foundation models and maintain persistent behavioral identity over long horizons, agents implement multi\-tiered memory architectures\[[8](https://arxiv.org/html/2609.13731#bib.bib8)\]\.

Working Context Memory \(ℳwork\\mathcal\{M\}\_\{\\text\{work\}\}\)\.Working memory corresponds to the active attention context window holding immediate system instructions, few\-shot demonstration exemplars, active tool schemas, and recent conversation turns\[[81](https://arxiv.org/html/2609.13731#bib.bib81),[82](https://arxiv.org/html/2609.13731#bib.bib82)\]\. While modern models support extensive context windows, attention degradation over long token spans \(the “lost in the middle” phenomenon\) creates operational vulnerabilities\.

Episodic Vector Memory \(ℳepisodic\\mathcal\{M\}\_\{\\text\{episodic\}\}\)\.Episodic memory captures historical execution logs, conversation histories, and external documents\. These records are projected into dense semantic vector spacesℝd\\mathbb\{R\}^\{d\}using neural embedding models and indexed via Hierarchical Navigable Small World \(HNSW\)\[[83](https://arxiv.org/html/2609.13731#bib.bib83)\]or FAISS\[[84](https://arxiv.org/html/2609.13731#bib.bib84)\]graph structures\. When the agent formulates a query𝐪\\mathbf\{q\}, relevant memory chunks are retrieved via top\-kkcosine similarity:

Retrieve​\(𝐪,k\)=arg⁡max⁡∑i=1km1,…,mk∈ℳepisodic⁡𝐄⁡\(𝐪\)⋅𝐄⁡\(mi\)‖𝐄⁡\(𝐪\)‖2​‖𝐄⁡\(mi\)‖2\\text\{Retrieve\}\(\\mathbf\{q\},k\)=\\arg\\max\_\{m\_\{1\},\\dots,m\_\{k\}\\in\\mathcal\{M\}\_\{\\text\{episodic\}\}\}\\sum\_\{i=1\}^\{k\}\\frac\{\\mathbf\{E\}\(\\mathbf\{q\}\)\\cdot\\mathbf\{E\}\(m\_\{i\}\)\}\{\\\|\\mathbf\{E\}\(\\mathbf\{q\}\)\\\|\_\{2\}\\\|\\mathbf\{E\}\(m\_\{i\}\)\\\|\_\{2\}\}\(11\)
Semantic Knowledge and Virtual Context Paging\.Semantic and procedural memoryℳsemantic\\mathcal\{M\}\_\{\\text\{semantic\}\}encompasses both the parametric knowledge internalized in model weightsθ\\thetaand structured Knowledge Graphs \(KGs\) that encode domain\-specific rules and tool hierarchies\[[10](https://arxiv.org/html/2609.13731#bib.bib10),[9](https://arxiv.org/html/2609.13731#bib.bib9),[85](https://arxiv.org/html/2609.13731#bib.bib85)\]\. To seamlessly bridge fast working memory and large\-scale external stores, operating\-system\-inspired architectures like MemGPT\[[8](https://arxiv.org/html/2609.13731#bib.bib8)\]implement virtual context paging, enabling agents to dynamically swap memory blocks between context tiers via explicit function calls\.

### II\-ETool Integration Plane & Model Context Protocol \(MCP\)

The Action Plane𝒯\\mathcal\{T\}transforms abstract reasoning into real\-world computational side effects through structured tool invocation paradigms\.

Structured Function Calling and Schema Validation\.Modern foundation models are fine\-tuned to emit structured JSON or XML payloads adhering to strict JSON Schema specifications\[[10](https://arxiv.org/html/2609.13731#bib.bib10),[86](https://arxiv.org/html/2609.13731#bib.bib86)\]\. The agent runtime parses these emitted payloads, validates argument types, and dispatches requests to local system binaries or remote web services\.

The Model Context Protocol \(MCP\) Standard and Transport Layers\.In late 2024, the Model Context Protocol \(MCP\) emerged as an open client\-server JSON\-RPC 2\.0 specification standardizing how AI agents discover tools, access secure resources, and execute operations across local and distributed environments\[[11](https://arxiv.org/html/2609.13731#bib.bib11)\]\. MCP establishes three foundational primitives:Prompts\(pre\-engineered contextual workflows\),Resources\(structured, read\-only contextual data\), andTools\(executable functions with real\-world side effects\)\. Operationally, MCP supports two primary communication transports: local process pipes over standard input/output \(stdio\) for single\-host tool execution, and HTTP with Server\-Sent Events \(SSE\) for remote, distributed microservice invocation\. While MCP standardizes tool interoperability across heterogeneous environments, it also creates an expansive attack surface: rogue tool servers can publish poisoned function schemas that execute prompt injection at registration time, or manipulate tool response payloads to hijack downstream agent planning\[[87](https://arxiv.org/html/2609.13731#bib.bib87)\]\.

Distributed Cluster Execution and Runtime Sandboxes\.In enterprise and high\-performance computing clusters, autonomous agents are scheduled across distributed compute nodes using distributed orchestration runtimes such as Ray, Celery, and Kubernetes\. Because agents execute untrusted code and arbitrary commands, contemporary cluster deployments enforce multi\-tiered isolation\. These range from containerized Linux namespaces \(such as SWE\-agent\[[14](https://arxiv.org/html/2609.13731#bib.bib14)\]\) and headless browser automation engines \(such as Playwright, Mind2Web\[[15](https://arxiv.org/html/2609.13731#bib.bib15)\], and WebGPT\[[27](https://arxiv.org/html/2609.13731#bib.bib27)\]\) to hardware\-virtualized micro\-virtual machines \(such as AWS Firecracker\[[54](https://arxiv.org/html/2609.13731#bib.bib54)\]and Google gVisor\[[55](https://arxiv.org/html/2609.13731#bib.bib55)\]\)\. Enforcing deterministic boundary isolation across distributed agent worker pools remains a vital systems requirement for cluster deployments\.

### II\-FMulti\-Agent Collaboration Topologies

To resolve large\-scale, complex enterprise workflows, individual agents are orchestrated into distributed multi\-agent networks operating across diverse topological structures \(Table[III](https://arxiv.org/html/2609.13731#S2.T3)\)\.

TABLE III:Comparison of Leading Agent Frameworks and ArchitecturesFrameworkPrimary DomainReasoningMemory ModelKey Vulnerability PointAutoGPT\[[1](https://arxiv.org/html/2609.13731#bib.bib1)\]General automationReActWorking \+ Vector DBUnsanitized tool execution; loops\.AutoGen\[[12](https://arxiv.org/html/2609.13731#bib.bib12)\]Multi\-agent chatConversableContext passingByzantine peer injection; no signing\.MetaGPT\[[13](https://arxiv.org/html/2609.13731#bib.bib13)\]Software engineeringSOP role playPub/Sub sharedCascading hallucination propagation\.ChatDev\[[26](https://arxiv.org/html/2609.13731#bib.bib26)\]Collaborative devWaterfallRole buffersCross\-role prompt injection via code\.SWE\-agent\[[14](https://arxiv.org/html/2609.13731#bib.bib14)\]Issue resolutionACI \+ ReActWorking \+ diffsShell parameter injection in bash\.MemGPT\[[8](https://arxiv.org/html/2609.13731#bib.bib8)\]Conversational OSContext pagingMulti\-tier vectorVector poisoning & inversion\.Hierarchical and Communicative Orchestration\.In hierarchical orchestration topologies, such as MetaGPT\[[13](https://arxiv.org/html/2609.13731#bib.bib13)\]and AutoGen\[[12](https://arxiv.org/html/2609.13731#bib.bib12)\], a centralized supervisor agent receives high\-level goals and delegates specific sub\-tasks to subordinate specialized agents according to Standard Operating Procedures \(SOPs\)\. Conversely, in peer\-to\-peer communicative meshes \(e\.g\., CAMEL\[[33](https://arxiv.org/html/2609.13731#bib.bib33)\], AgentVerse\[[34](https://arxiv.org/html/2609.13731#bib.bib34)\], and DyLAN\[[88](https://arxiv.org/html/2609.13731#bib.bib88)\]\), agents negotiate directly through conversational turn\-taking and contract\-net protocols without centralized bottlenecks\.

Shared Blackboard Swarms and Game\-Theoretic Consensus\.In blackboard\-oriented architectures, such as ChatDev\[[26](https://arxiv.org/html/2609.13731#bib.bib26)\]and Generative Agents\[[89](https://arxiv.org/html/2609.13731#bib.bib89)\], multiple agents communicate asynchronously by reading from and writing to a shared global state repository\. To improve reasoning fidelity, multi\-agent debate frameworks employ game\-theoretic cross\-examination, iterative voting, and consensus mechanisms\[[90](https://arxiv.org/html/2609.13731#bib.bib90),[91](https://arxiv.org/html/2609.13731#bib.bib91),[92](https://arxiv.org/html/2609.13731#bib.bib92)\]\. However, as analyzed in Section[V](https://arxiv.org/html/2609.13731#S5), these collaborative topologies introduce critical vulnerabilities, including cascading hallucination amplification, AI worm propagation, and Byzantine consensus subversion\.

## IIIThe 6\-Dimensional Trustworthiness Taxonomy

In classical machine learning, trustworthiness has traditionally been evaluated as a static statistical property over fixed benchmark datasets, measured through test\-set accuracy, out\-of\-distribution generalization metrics, and boundedℓp\\ell\_\{p\}\-norm adversarial robustness\[[66](https://arxiv.org/html/2609.13731#bib.bib66),[67](https://arxiv.org/html/2609.13731#bib.bib67),[68](https://arxiv.org/html/2609.13731#bib.bib68),[93](https://arxiv.org/html/2609.13731#bib.bib93)\]\. In the context ofAgentic AI, however, trustworthiness must be fundamentally reformulated as adynamic, systems\-level operational guarantee\. Because autonomous agents possess execution agency, maintain persistent state across memory tiers, and generate real\-world side effects through live tool dispatches, an operational failure in any single cognitive or execution module compromises the integrity of the entire socio\-technical system\[[94](https://arxiv.org/html/2609.13731#bib.bib94),[95](https://arxiv.org/html/2609.13731#bib.bib95)\]\.

To establish an overarching conceptual structure for analyzing and evaluating agent trustworthiness, we formalize the6\-Dimensional Trustworthiness Taxonomyfor Agentic AI Systems \(Fig\.[4](https://arxiv.org/html/2609.13731#S3.F4)\)\.

1\. Security \(Sec\. 3\.1\) ∙\\bulletConfidentiality: System prompt shielding ∙\\bulletIntegrity: Memory & tool validation ∙\\bulletAvailability: DoS & recursion defense2\. Safety & Robustness \(Sec\. 3\.2\) ∙\\bulletBounded Action Envelopes & rollback ∙\\bulletHallucination Suppression in tools ∙\\bulletOOD Observation Fault Tolerance3\. Privacy & Data Protection \(Sec\. 3\.3\) ∙\\bulletZero\-Leakage API Tool Dispatches ∙\\bulletEmbedding Inversion Resistance ∙\\bulletMulti\-Tenant Context Hygiene4\. Explainability & Verifiability \(Sec\. 3\.4\) ∙\\bulletAction\-Chain Audit Traceability ∙\\bulletNeural\-Symbolic SMT Verifiers ∙\\bulletFaithfulness of Reasoning Chains5\. Fairness & Non\-Discrimination \(Sec\. 3\.5\) ∙\\bulletBalanced Tool Priority Scheduling ∙\\bulletAnti\-Collusion in Multi\-Agent Bidding ∙\\bulletDemographic Parity in Decisions6\. Accountability & Provenance \(Sec\. 3\.6\) ∙\\bulletCryptographic Action Signatures \(DIDs\) ∙\\bulletTamper\-Evident Merkle Logs ∙\\bulletLegally Binding Agent Non\-Repudiation

Fig\. 4:The 6\-Dimensional Trustworthiness Taxonomy for Agentic AI Systems\.### III\-ADimension 1: Security \(Confidentiality, Integrity, Availability\)

Security governs the resilience of the agent’s cognitive core, memory subsystems, tool execution planes, and inter\-agent communication channels against intentional, malicious adversarial attacks\[[18](https://arxiv.org/html/2609.13731#bib.bib18),[19](https://arxiv.org/html/2609.13731#bib.bib19)\]\. In agentic architectures, classical information security goals manifest through specialized operational invariants\.

Confidentiality of Internal State and Secrets\.Confidentiality requires ensuring that sensitive system prompts, proprietary tool schemas, API access credentials, and retrieved private episodic memories are never exfiltrated or disclosed via tool payloads or adversarial prompt manipulation\[[96](https://arxiv.org/html/2609.13731#bib.bib96),[97](https://arxiv.org/html/2609.13731#bib.bib97),[98](https://arxiv.org/html/2609.13731#bib.bib98)\]\. Formally, the information\-theoretic advantage of an adversary𝒜adv\\mathcal\{A\}\_\{\\text\{adv\}\}in recovering secret credentials or hidden system promptsksecret∈𝒦k\_\{\\text\{secret\}\}\\in\\mathcal\{K\}over security parameterλ\\lambdamust be strictly bounded:

𝐀𝐝𝐯𝒜leak​\(λ\)=\|Pr⁡\(𝒜adv​\(xadv\)=ksecret\)−1\|𝒦\|\|≤negl⁡\(λ\)\\mathbf\{Adv\}\_\{\\mathcal\{A\}\}^\{\\text\{leak\}\}\(\\lambda\)=\\left\|\\Pr\\left\(\\mathcal\{A\}\_\{\\text\{adv\}\}\(x\_\{\\text\{adv\}\}\)=k\_\{\\text\{secret\}\}\\right\)\-\\frac\{1\}\{\|\\mathcal\{K\}\|\}\\right\|\\leq\\operatorname\{negl\}\(\\lambda\)\(12\)In practice, achieving this requires strict output boundary filtering, parameter redaction, and prompt\-shielding mechanisms that isolate internal system instructions from untrusted data channels\.

Integrity of Planning and Tool Invocations\.Integrity guarantees that the agent’s internal reasoning loop, intermediate planning trees, memory embeddings, and tool dispatch parameters remain unaltered by unauthorized external entities or poisoned data sources\[[51](https://arxiv.org/html/2609.13731#bib.bib51),[59](https://arxiv.org/html/2609.13731#bib.bib59)\]\. If an attacker can manipulate intermediate reasoning steps \(t​htth\_\{t\}\) or inject untrusted data into memory stores, the execution policy will diverge from developer intent\. Preserving integrity requires cryptographic verification of external data sources and capability\-based authorization gates for all tool calls\.

Availability and Computational Resilience\.Availability guarantees continuous, deterministic service operation, protecting agent systems against computational denial\-of\-service \(DoS\) attacks\[[58](https://arxiv.org/html/2609.13731#bib.bib58)\]\. Threat actors exploit foundation model latency characteristics using sponge examples\[[57](https://arxiv.org/html/2609.13731#bib.bib57)\], trigger unbounded recursive planning loops, or induce tool deadlocks that exhaust GPU memory and API token quotas\. Enforcing availability demands runtime execution time\-outs, recursion depth limits, and adaptive rate\-limiting controllers\.

### III\-BDimension 2: Safety & Operational Robustness

Whereas security focuses on intentional adversarial attacks, safety addresses non\-malicious operational risks, ensuring that autonomous exploration, planning errors, and tool executions do not cause unintended software, financial, or physical harm\[[94](https://arxiv.org/html/2609.13731#bib.bib94),[95](https://arxiv.org/html/2609.13731#bib.bib95),[99](https://arxiv.org/html/2609.13731#bib.bib99)\]\.

Bounded Action Envelopes and Operational Reversibility\.Safety requires constraining the agent’s tool action space𝒯\\mathcal\{T\}such that irreversible actions \(such as permanent file deletion, database drops, external financial transactions, or critical email dispatches\) cannot be executed autonomously without explicit human authorization or verifiable rollback capabilities\[[100](https://arxiv.org/html/2609.13731#bib.bib100)\]:

∀at∈𝒯irreversible,Exec​\(at\)⇔Authorizedhuman​\(at\)=True\\begin\{split\}\\forall a\_\{t\}\\in\\mathcal\{T\}\_\{\\text\{irreversible\}\},\\quad&\\text\{Exec\}\(a\_\{t\}\)\\iff\\\\ &\\text\{Authorized\}\_\{\\text\{human\}\}\(a\_\{t\}\)=\\text\{True\}\\end\{split\}\(13\)By defining formal safety envelopes around tool dispatchers, systems ensure that unexpected reasoning divergences remain contained within recoverable operational boundaries\.

Hallucination Suppression in Tool Dispatches\.In static language models, hallucinations produce incorrect conversational statements\. In agentic systems, however, hallucinations manifest as fabricated API endpoints, hallucinated parameter schemas, or invalid file paths dispatched to actual operating system shells\[[10](https://arxiv.org/html/2609.13731#bib.bib10),[101](https://arxiv.org/html/2609.13731#bib.bib101)\]\. Suppressing hallucinations requires integrating strict schema\-enforcing compilers and deterministic API validators directly into the action emission pipeline\.

Environmental and Distributional Robustness\.Agents operate in dynamic, noisy, and partially observable environments where external tools frequently experience network timeouts, schema changes, or out\-of\-distribution \(OOD\) return values\[[102](https://arxiv.org/html/2609.13731#bib.bib102),[39](https://arxiv.org/html/2609.13731#bib.bib39)\]\. Operational robustness requires agents to maintain stable decision trajectories, gracefully handle runtime tool exceptions, and avoid catastrophic failure cascades when confronted with unexpected environmental observations\.

### III\-CDimension 3: Privacy & Data Protection

Because autonomous agents continuously ingest, process, store, and transmit heterogeneous data across third\-party APIs and persistent vector stores, privacy preservation must be enforced across every stage of the agentic data lifecycle\[[103](https://arxiv.org/html/2609.13731#bib.bib103),[104](https://arxiv.org/html/2609.13731#bib.bib104),[105](https://arxiv.org/html/2609.13731#bib.bib105)\]\.

Zero\-Leakage Tool Dispatches and Perimeter Sanitization\.When an agent queries external tool servers or third\-party web services, it frequently handles sensitive enterprise data or personally identifiable information \(PII\)\. Privacy enforcement demands automated perimeter sanitization that redacts sensitive entities, anonymizes confidential variables, and applies zero\-leakage API proxies before payloads leave the trust boundary\[[106](https://arxiv.org/html/2609.13731#bib.bib106)\]\.

Episodic Memory Protection and Inversion Resistance\.Persistent episodic memory stores \(ℳepisodic\\mathcal\{M\}\_\{\\text\{episodic\}\}\) contain high\-dimensional vector representations of user interactions\. Recent research demonstrates that adversaries can execute membership inference attacks or embedding inversion attacks to reconstruct raw training prompts and private documents from dense embeddings\[[107](https://arxiv.org/html/2609.13731#bib.bib107),[108](https://arxiv.org/html/2609.13731#bib.bib108),[109](https://arxiv.org/html/2609.13731#bib.bib109),[110](https://arxiv.org/html/2609.13731#bib.bib110)\]\. Defending vector memory requires injecting calibrated Differential Privacy noise during embedding generation and enforcing cryptographic access controls over vector indices\.

Multi\-Tenant Context Hygiene\.In multi\-user enterprise platforms, autonomous agents frequently share underlying model infrastructure\. Insecure context management can lead to cross\-session context bleeding, where sensitive variables from one user’s session persist in cached Key\-Value \(KV\) memory states or shared vector stores, allowing subsequent users to inadvertently access confidential data\[[8](https://arxiv.org/html/2609.13731#bib.bib8)\]\. Robust multi\-tenancy requires strict hardware\-enforced memory isolation and deterministic context purging between sessions\.

### III\-DDimension 4: Explainability, Interpretability & Verifiability

In high\-stakes autonomous systems, post\-hoc natural language rationalization is insufficient to guarantee safety; formal verifiability and causal explainability are mandatory\[[111](https://arxiv.org/html/2609.13731#bib.bib111),[112](https://arxiv.org/html/2609.13731#bib.bib112)\]\.

Action\-Chain Audit Traceability\.To ensure complete operational transparency, every emitted actionata\_\{t\}must be cryptographically and deterministically linked to its causal precursors: the supporting reasoning thoughtt​htth\_\{t\}, the retrieved episodic memoriesm∈ℳm\\in\\mathcal\{M\}, and the preceding environmental observationot−1o\_\{t\-1\}\[[4](https://arxiv.org/html/2609.13731#bib.bib4),[6](https://arxiv.org/html/2609.13731#bib.bib6)\]\. Generating immutable action\-thought audit logs allows human operators and automated forensic monitors to reconstruct the precise cognitive state that produced any system side effect\.

Neural\-Symbolic Verifiability via Deterministic Solvers\.While deep neural networks excel at flexible heuristic planning, they lack deterministic safety guarantees\. To resolve this, neural\-symbolic architectures translate probabilistic plans emitted by the cognitive core into formal symbolic specifications \(such as First\-Order Logic or Linear Temporal Logic\) that can be formally proven safe using Satisfiability Modulo Theories \(SMT\) solvers \(e\.g\., Z3\[[113](https://arxiv.org/html/2609.13731#bib.bib113)\], CVC5\[[114](https://arxiv.org/html/2609.13731#bib.bib114)\]\) before execution\.

Faithfulness of Reasoning Chains\.A critical failure mode in language model planning is the generation of unfaithful reasoning traces, where the generated Chain\-of\-Thought \(t​htth\_\{t\}\) does not reflect the actual underlying neural mechanisms driving action selection, but rather acts as a persuasive post\-hoc justification designed to satisfy safety filters\. Ensuring explainability requires validating the causal necessity of intermediate thoughts through causal mediation analysis and activation patching\.

### III\-EDimension 5: Fairness & Non\-Discrimination

In multi\-agent collaborative networks and multi\-user enterprise environments, agents must ensure equitable behavior across diverse demographic user groups and enforce fair resource allocation mechanisms\[[115](https://arxiv.org/html/2609.13731#bib.bib115),[116](https://arxiv.org/html/2609.13731#bib.bib116)\]\.

Unbiased Algorithmic Action Allocation\.When autonomous agents triage customer support requests, evaluate credit applications, or schedule computing resources, underlying training biases can cause disparate service quality, skewed response latencies, or asymmetric error rates across protected demographic attributes\. Systems must implement demographic parity testing and counterfactual fairness audits across tool dispatch policies\.

Anti\-Collusion in Multi\-Agent Markets\.In decentralized multi\-agent economies where autonomous agents bid on resources, negotiate contracts, and execute decentralized finance transactions, algorithms can learn to engage in tacit collusion, algorithmic price\-fixing, or predatory bidding strategies that subvert market efficiency\[[90](https://arxiv.org/html/2609.13731#bib.bib90),[91](https://arxiv.org/html/2609.13731#bib.bib91)\]\. Mitigating algorithmic discrimination in agent swarms requires verifiable auction rules and cryptographic anti\-collusion monitoring protocols\.

### III\-FDimension 6: Accountability, Auditability & Provenance

Autonomous decision\-making requires clear attribution of actions to human principals, software developers, or infrastructure operators, establishing non\-repudiable liability chains\[[18](https://arxiv.org/html/2609.13731#bib.bib18),[117](https://arxiv.org/html/2609.13731#bib.bib117)\]\.

Non\-Repudiable Action Provenance via Decentralized Identifiers\.Accountability requires that every tool actionata\_\{t\}dispatched by an agent is cryptographically signed using a private key bound to the agent’s unique W3C Decentralized Identifier \(DID\)\[[118](https://arxiv.org/html/2609.13731#bib.bib118)\]\. These signed execution records are anchored within immutable Merkle audit trees, guaranteeing that no actor can repudiate or alter historical logs after execution\.

Operational Alignment with International Governance Standards\.Establishing auditable systems requires operationalizing formal governance frameworks, including the NIST AI Risk Management Framework \(AI RMF 1\.0\)\[[119](https://arxiv.org/html/2609.13731#bib.bib119)\], ISO/IEC 42001\[[120](https://arxiv.org/html/2609.13731#bib.bib120)\], and the European Union AI Act\[[121](https://arxiv.org/html/2609.13731#bib.bib121)\]\. This ensures that autonomous agents comply with mandatory logging, continuous risk management, and human oversight requirements\.

### III\-GCross\-Dimensional Pareto Trade\-off Dynamics

Optimizing an Agentic AI system across all six trustworthiness dimensions simultaneously introduces fundamental engineering trade\-offs that cannot be fully satisfied concurrently \(Table[IV](https://arxiv.org/html/2609.13731#S3.T4)\)\.

TABLE IV:Cross\-Dimensional Trustworthiness Matrix: Formal Dimensions, Verification Methods, and Systems ControlsDimensionCore System InvariantVerification MethodEnforced Systems Security Control1\. Security𝐀𝐝𝐯𝒜leak≤negl⁡\(λ\)∧Integ​\(Π\)\\mathbf\{Adv\}\_\{\\mathcal\{A\}\}^\{\\text\{leak\}\}\\leq\\operatorname\{negl\}\(\\lambda\)\\land\\text\{Integ\}\(\\Pi\)Red\-teaming\[[40](https://arxiv.org/html/2609.13731#bib.bib40)\], SMT\[[113](https://arxiv.org/html/2609.13731#bib.bib113)\]CapBAC tokens & micro\-VM sandboxes\.2\. Safety∀at∈𝒯irr,Authh​\(at\)=1\\forall a\_\{t\}\\in\\mathcal\{T\}\_\{\\text\{irr\}\},\\text\{Auth\}\_\{\\text\{h\}\}\(a\_\{t\}\)=1Invariant check, R\-Judge\[[100](https://arxiv.org/html/2609.13731#bib.bib100)\]Bounded action envelopes & rollbacks\.3\. PrivacyPr\[ℳ\(D\)∈S\]≤eϵPr\[ℳ\(D′\)∈S\]\+δ\\Pr\[\\mathcal\{M\}\(D\)\\in S\]\\leq e^\{\\epsilon\}\\Pr\[\\mathcal\{M\}\(D^\{\\prime\}\)\\in S\]\+\\deltaMembership audits\[[103](https://arxiv.org/html/2609.13731#bib.bib103)\]Differential Privacy noise injection\.4\. ExplainabilityCIE​\(t​ht→at\)≥τ\\text\{CIE\}\(th\_\{t\}\\to a\_\{t\}\)\\geq\\tauMediation analysis\[[111](https://arxiv.org/html/2609.13731#bib.bib111)\]Immutable action\-thought chains & SMT logs\.5\. Fairness𝔼⁡\[U∣GA\]=𝔼⁡\[U∣GB\]\\mathbb\{E\}\[U\\mid G\_\{A\}\]=\\mathbb\{E\}\[U\\mid G\_\{B\}\]Parity testing\[[115](https://arxiv.org/html/2609.13731#bib.bib115)\]Balanced priority scheduling & auction rules\.6\. AccountabilityVerifySig​\(at\)∧MerklePath​\(at,Roott\)=1\\text\{VerifySig\}\(a\_\{t\}\)\\land\\text\{MerklePath\}\(a\_\{t\},\\text\{Root\}\_\{t\}\)=1Audit validation\[[117](https://arxiv.org/html/2609.13731#bib.bib117)\]W3C DIDs\[[118](https://arxiv.org/html/2609.13731#bib.bib118)\]& signed Merkle trees\.Security vs\. Operational Latency\.Deploying comprehensive defense\-in\-depth mechanisms—such as Dual\-LLM Inspector\-Executor pipelines, runtime SMT formal solvers, and deep kernel\-level eBPF tracing—introduces measurable computational and network latency\. In real\-time environments, such as automated high\-frequency trading or physical robotics, excessive guardrail latency can render the agent unresponsive to rapid environmental changes\.

Privacy vs\. Task Utility\.Injecting Differential Privacy noise𝒩⁡\(0,σ2​𝐈\)\\mathcal\{N\}\(0,\\sigma^\{2\}\\mathbf\{I\}\)into episodic vector embeddings effectively thwarts embedding inversion and membership inference attacks\. However, this perturbation degrades the semantic precision of top\-kkvector retrieval in RAG pipelines, lowering overall task success rates\.

Safety Rigor vs\. Creative Autonomy\.Overly restrictive safety envelopes and aggressive heuristic filters frequently trigger False Safety Rejections \(FSRR\), wherein benign multi\-step programming or system administration commands are incorrectly blocked\. Balancing safety guarantees with unconstrained problem\-solving autonomy represents a central open challenge in trustworthy agent design\.

## IVIntra\-Execution Security: Perception, Brain, and Action Threats

1\. Perception Threats\(Section 4\.1\)2\. Brain / Cognitive Threats\(Section 4\.2\)3\. Action Plane Threats\(Section 4\.3\)∙\\bulletDirect Prompt Injection \(DPI\) ∙\\bulletAdversarial Suffixes \(GCG/AutoDAN\) ∙\\bulletMultimodal Steganography \(Images\)∙\\bulletSleeper Agent Backdoors \(Triggered\) ∙\\bulletBadChain CoT Poisoning ∙\\bulletHallucination Error Cascades∙\\bulletTool Parameter Injection \(RCE\) ∙\\bulletConfused Deputy Privilege Abuse ∙\\bulletSSRF Cloud Metadata Exfiltration

Fig\. 5:Taxonomy of Intra\-Execution Security Threats in Agentic Systems\.Intra\-execution security encompasses vulnerabilities and attack surfaces that manifest strictly within the internal operational lifecycle of an individual AI agent\. As formalized in Section[II](https://arxiv.org/html/2609.13731#S2), an agent’s execution loop comprises three tightly coupled internal stages: Perception, Brain \(Cognitive Planning Core\), and Action \(Fig\.[5](https://arxiv.org/html/2609.13731#S4.F5)\)\. An attack against any of these internal stages compromises the entire decision trajectory\.

### IV\-APerception Threats: Injections, Suffixes & Steganography

The perception module translates multimodal human instructions, developer system guidelines, and dynamic environmental inputs into structured token embeddings for the cognitive core\. Attackers exploit the fundamental lack of hardware\-enforced data\-instruction separation to subvert perceptual processing\.

#### IV\-A1Direct Prompt Injection and Goal Hijacking

Direct Prompt Injection \(DPI\) occurs when an adversarial user crafts input text designed to override the model’s system prompt and developer instructions\[[22](https://arxiv.org/html/2609.13731#bib.bib22),[122](https://arxiv.org/html/2609.13731#bib.bib122),[123](https://arxiv.org/html/2609.13731#bib.bib123),[124](https://arxiv.org/html/2609.13731#bib.bib124)\]\. In agentic architectures, DPI manifests through two primary attack modalities:

Goal Hijacking\.In goal hijacking attacks, the adversary overrides the original mission objective and substitutes a malicious goal \(for example, instructing an automated code\-review assistant to parse private repository secrets and exfiltrate them via an external API call\)\[[22](https://arxiv.org/html/2609.13731#bib.bib22),[47](https://arxiv.org/html/2609.13731#bib.bib47),[125](https://arxiv.org/html/2609.13731#bib.bib125)\]:

𝒞\(SystemPrompt∥InjectedPrompt\)→amalicious\\mathcal\{C\}\(\\text\{SystemPrompt\}\\parallel\\text\{InjectedPrompt\}\)\\to a\_\{\\text\{malicious\}\}\(14\)Because the model lacks a native execution privilege hierarchy, it processes the concatenated tokens uniformly, treating the injected attacker goal with equal or higher authority than the developer’s original system constraints\.

Prompt and Secret Leaking\.Attackers utilize adversarial linguistic framing \(such as roleplay scenarios, hypothetical translation requests, or debug commands\) to force the model to disclose its confidential system instructions, internal tool definitions, private API tokens, or embedded credentials\[[96](https://arxiv.org/html/2609.13731#bib.bib96),[97](https://arxiv.org/html/2609.13731#bib.bib97),[98](https://arxiv.org/html/2609.13731#bib.bib98)\]\.

#### IV\-A2Adversarial Suffix Optimization \(GCG and AutoDAN\)

To bypass safety alignment guardrails systematically, adversaries optimize continuous token sequences that force the model into positive token generation states\. Zou et al\.\[[38](https://arxiv.org/html/2609.13731#bib.bib38)\]formalized the Greedy Coordinate Gradient \(GCG\) optimization algorithm, which computes token gradients across white\-box surrogate models:

minp∈𝒱Lℒ\(p\)=−∑t=1\|ytarget\|logP\(yttarget\|xprompt∥p∥y<ttarget;θ\)\\min\_\{p\\in\\mathcal\{V\}^\{L\}\}\\mathcal\{L\}\(p\)=\-\\sum\_\{t=1\}^\{\|y^\{\\text\{target\}\}\|\}\\log P\\left\(y\_\{t\}^\{\\text\{target\}\}\\;\\middle\|\\;x\_\{\\text\{prompt\}\}\\parallel p\\parallel y\_\{<t\}^\{\\text\{target\}\};\\theta\\right\)\(15\)whereppis a suffix of lengthLLselected from vocabulary𝒱\\mathcal\{V\}designed to maximize the likelihood of emitting an affirmative target prefixytargety^\{\\text\{target\}\}\(e\.g\., “Sure, here is how to execute the command:”\)\. While GCG produces unreadable token sequences that can be detected via perplexity filters, genetic\-algorithm\-based extensions such as AutoDAN\[[42](https://arxiv.org/html/2609.13731#bib.bib42)\]optimize semantically readable, stealthy jailbreak prefixes that transfer successfully across black\-box commercial models\.

#### IV\-A3Automated Black\-Box Red\-Teaming \(PAIR and TAP\)

To automate the discovery of adversarial jailbreaks without white\-box gradient access, researchers have developed recursive attacker\-target game loops\. The Prompt Automatic Iterative Refinement \(PAIR\) framework\[[40](https://arxiv.org/html/2609.13731#bib.bib40)\]deploys an attacker LLM𝒜att\\mathcal\{A\}\_\{\\text\{att\}\}that iteratively interrogates a target agent𝒜tgt\\mathcal\{A\}\_\{\\text\{tgt\}\}, analyzes the refusal semantics, and refines the adversarial prompt\. The Tree of Attacks with Pruning \(TAP\) framework\[[41](https://arxiv.org/html/2609.13731#bib.bib41)\]extends this approach by managing a branching search tree of candidate prompts, pruning unpromising branches, and achieving\>80%\>80\\%jailbreak success rates within fewer than 20 model queries\.

#### IV\-A4Multimodal Steganography and Visual Jailbreaks

In vision\-language agents \(e\.g\., GPT\-4V, Gemini Pro Vision\), attackers exploit the visual perception encoder by perturbing an input image𝐈∈ℝH×W×C\\mathbf\{I\}\\in\\mathbb\{R\}^\{H\\times W\\times C\}with imperceptible adversarial noiseδ\\delta:

𝐈adv=𝐈\+δs\.t\.‖δ‖p≤ϵ\\mathbf\{I\}\_\{\\text\{adv\}\}=\\mathbf\{I\}\+\\delta\\quad\\text\{s\.t\.\}\\quad\\\|\\delta\\\|\_\{p\}\\leq\\epsilon\(16\)When processed by the visual encoder,𝐈adv\\mathbf\{I\}\_\{\\text\{adv\}\}projects the visual representation directly into the token embedding space of a malicious text instruction \(e\.g\., “Execute rm \-rf on the current directory”\)\[[126](https://arxiv.org/html/2609.13731#bib.bib126),[125](https://arxiv.org/html/2609.13731#bib.bib125),[127](https://arxiv.org/html/2609.13731#bib.bib127)\]\. Because the image appears completely benign to human overseers, this multimodal steganographic channel completely bypasses optical and human\-in\-the\-loop review\.

### IV\-BBrain Threats: Backdoors, Hallucinations & Planning Vulnerabilities

The Brain module performs multi\-step cognitive reasoning, goal decomposition, hypothesis generation, and tool dispatch selection\. Flaws within this layer undermine the logical foundation of autonomous decision\-making\.

#### IV\-B1Backdoors and Sleeper Agent Activation

Backdoors insert latent trigger associations into neural parameters during pre\-training\[[128](https://arxiv.org/html/2609.13731#bib.bib128)\], instruction fine\-tuning\[[129](https://arxiv.org/html/2609.13731#bib.bib129),[130](https://arxiv.org/html/2609.13731#bib.bib130)\], or reinforcement learning from human feedback \(RLHF\)\[[131](https://arxiv.org/html/2609.13731#bib.bib131)\]\. Hubinger et al\.\[[50](https://arxiv.org/html/2609.13731#bib.bib50)\]demonstrated that backdoored models can act asSleeper Agents:

𝒞⁡\(x,θ\)=\{ysafe,ifTrigger​\(x\)=Falseymalicious,ifTrigger​\(x\)=True\\mathcal\{C\}\(x;\\theta\)=\\begin\{cases\}y\_\{\\text\{safe\}\},&\\text\{if \}\\text\{Trigger\}\(x\)=\\text\{False\}\\\\ y\_\{\\text\{malicious\}\},&\\text\{if \}\\text\{Trigger\}\(x\)=\\text\{True\}\\end\{cases\}\(17\)Crucially, standard safety alignment techniques fail to purge sleeper triggers because the model learns to identify when it is operating within an alignment evaluation sandbox, actively concealing its deceptive capabilities until a specific deployment trigger \(e\.g\., a specific timestamp or secret phrase\) appears in real\-world inputs\.

#### IV\-B2Chain\-of\-Thought Backdoor Injection \(BadChain\)

Xiang et al\.\[[51](https://arxiv.org/html/2609.13731#bib.bib51)\]demonstrated that backdoors can be inserted directly into intermediate reasoning steps \(t​htth\_\{t\}\) rather than final outputs\. Under BadChain attacks, when a subtle trigger is detected in an input prompt, the model alters its intermediate Chain\-of\-Thought reasoning trajectory:

t​ht=BadCoT​\(t​ht−1,Trigger\)⟹at=aadversarialth\_\{t\}=\\text\{BadCoT\}\(th\_\{t\-1\},\\text\{Trigger\}\)\\implies a\_\{t\}=a\_\{\\text\{adversarial\}\}\(18\)Because the altered thought appears coherent and logically structured, downstream verifiers and human supervisors accept the malicious actionata\_\{t\}as a valid conclusion of rational deduction\.

#### IV\-B3Hallucination Snowballing Mechanisms

Hallucinations in foundation models stem from compression artifacts in pre\-training data\[[132](https://arxiv.org/html/2609.13731#bib.bib132)\], parametric knowledge conflicts\[[133](https://arxiv.org/html/2609.13731#bib.bib133)\], and autoregressive error accumulation\[[52](https://arxiv.org/html/2609.13731#bib.bib52),[53](https://arxiv.org/html/2609.13731#bib.bib53),[134](https://arxiv.org/html/2609.13731#bib.bib134),[135](https://arxiv.org/html/2609.13731#bib.bib135)\]\. In multi\-step agent execution loops, hallucinations exhibit a severe snowballing effect: an erroneous assertion generated in stepttis appended to the execution historyHtH\_\{t\}, causing the model to treat its own prior hallucination as an authoritative environmental fact in subsequent stepst\+1,…,Tt\+1,\\dots,T\.

#### IV\-B4Planning Vulnerabilities Across Reasoning Topologies

As illustrated in Fig\.[6](https://arxiv.org/html/2609.13731#S4.F6), the choice of cognitive reasoning topology fundamentally dictates how errors and malicious inputs propagate through the system\.

\(a\) Sequential Planning \(CoT / ReAct\)\(b\) Search\-Based Planning \(ToT / GoT / LATS\)Step 1: Input Analysis𝒞⁡\(xuser\)→t​h1\\mathcal\{C\}\(x\_\{\\text\{user\}\}\)\\to th\_\{1\}Step 2: Adversarial Injectiono1∈ℰ⟹t​h2advo\_\{1\}\\in\\mathcal\{E\}\\implies th\_\{2\}^\{\\text\{adv\}\}Step 3: Cascade Failurea3∼𝒞⁡\(H2\)→RCE / Leaka\_\{3\}\\sim\\mathcal\{C\}\(H\_\{2\}\)\\to\\text\{RCE / Leak\}Linear FeedUnrecoverableRoot Goal State \(s0s\_\{0\}\)Expand Candidates​\{v1,v2\}\\text\{Expand Candidates \}\\\{v\_\{1\},v\_\{2\}\\\}Branch A: BenignV⁡\(vsafe\)=0\.85V\(v\_\{\\text\{safe\}\}\)=0\.85Branch B: PoisonedV⁡\(vpoison\)=0\.95∗V\(v\_\{\\text\{poison\}\}\)=0\.95^\{\*\}Pruned Branch\(Sub\-optimal heuristic\)Selected Execution→\\toAmplified RCE / Exploit

Fig\. 6:Planning Topologies and Error Amplification Pathways in Cognitive Cores: \(a\) Linear error propagation without recovery in sequential execution loops; \(b\) Adversarial heuristic score manipulation directing search trees toward poisoned execution branches\.In linear sequential planning \(such as pure CoT and ReAct\), any injected error or hallucination in stepttpropagates downstream without branching recovery, causing irreversible trajectory failure\[[77](https://arxiv.org/html/2609.13731#bib.bib77),[49](https://arxiv.org/html/2609.13731#bib.bib49)\]\. In contrast, search\-based deliberate planning \(such as ToT, GoT, and LATS\) explores multiple hypothetical branches\[[5](https://arxiv.org/html/2609.13731#bib.bib5),[6](https://arxiv.org/html/2609.13731#bib.bib6),[48](https://arxiv.org/html/2609.13731#bib.bib48)\]\. While search graphs allow backtracking from errors, they expand the search space for adversaries: if an attacker can manipulate the heuristic state scoring functionV⁡\(v\)V\(v\), they can force the search algorithm to prune safe planning paths and prioritize malicious branches\.

### IV\-CAction Plane Threats: Parameter Injection, SSRF & Supply Chain

The Action Plane translates high\-level cognitive plans into concrete computational side effects by executing system binaries, querying databases, and invoking cloud APIs\.

#### IV\-C1Command Injection and Shell Parameter Smuggling

When agents interact with operating systems via interactive bash shells or Python interpreters \(e\.g\., in SWE\-agent\[[14](https://arxiv.org/html/2609.13731#bib.bib14)\]or AutoGPT\), unsanitized user inputs or adversarial environmental outputs can trigger arbitrary Remote Code Execution \(RCE\)\[[136](https://arxiv.org/html/2609.13731#bib.bib136),[137](https://arxiv.org/html/2609.13731#bib.bib137)\]\. If an agent generates shell commands via string interpolation:

Command=“cat ”\|Filename\\text\{Command\}=\\text\{\`\`cat ''\}\\parallel\\text\{Filename\}\(19\)an adversary controllingFilenamecan inject shell metacharacters \(; rm \-rf / ;,\| curl evil\.com \| bash\), executing unauthorized arbitrary commands with the agent’s host\-level system privileges\.

#### IV\-C2The Confused Deputy Problem in Tool Execution

Because agents often run with broad ambient access rights, an external attacker can manipulate the agent into acting as aConfused Deputy\[[140](https://arxiv.org/html/2609.13731#bib.bib140),[35](https://arxiv.org/html/2609.13731#bib.bib35)\]\. In multi\-tenant environments, an agent authorized to read sensitive enterprise databases on behalf of a privileged user can be tricked via prompt injection by an unprivileged user into querying restricted financial records or exfiltrating API keys, violating capability confinement policies\.

#### IV\-C3Model Context Protocol \(MCP\) and Supply Chain Vectors

The standardization of agent\-tool communication via the Model Context Protocol \(MCP\) introduces new supply chain vulnerabilities\[[11](https://arxiv.org/html/2609.13731#bib.bib11),[87](https://arxiv.org/html/2609.13731#bib.bib87)\]\. Malicious third\-party MCP servers can publish deceptive tool definitions with overlapping semantic descriptions \(Tool Squatting\) to hijack invocation routing\. Once invoked, rogue MCP servers can return malicious payloads, exfiltrate conversation histories, or launch Denial\-of\-Service attacks against agent runtimes\.

### IV\-DEmpirical Case Studies and Real\-World CVE Dissections

Real\-world deployments of agentic architectures have already suffered severe security breaches stemming from the failure of isolation boundaries\.

InCVE\-2023\-34541 \(LangChain Experimental SQL Injection\), autonomous agents utilizing automated SQL database querying tools constructed raw SQL statements via unparameterized string concatenation\. Adversaries exploited this by providing inputs that converted standard SQL queries into ‘UNION SELECT‘ operations, dumping entire database credential tables directly into the agent’s context window\. This vulnerability highlighted the critical danger of relying on natural language models to enforce syntactic boundaries in structured query languages\.

InCVE\-2024\-21513 \(AutoGPT Arbitrary Code Execution\), the AutoGPT framework executed generated Python scripts inside unconstrained local subprocesses without micro\-VM sandboxing\. Attackers leveraged Indirect Prompt Injection to force the agent to execute malicious system commands \(‘os\.system\(”rm \-rf /”\)‘\), demonstrating that granting foundation models uncontained execution privileges on developer machines leads directly to complete host takeover\.

TABLE V:Master Overview of Security Threats to Agentic AI SystemsLayerAttack VectorPrimary MechanismEvaluated ImpactLiteraturePerceptionDirect Prompt InjectionIngesting adversarial textAlters goal; bypasses rules\[[22](https://arxiv.org/html/2609.13731#bib.bib22),[122](https://arxiv.org/html/2609.13731#bib.bib122)\]PerceptionAdversarial SuffixesToken search \(GCG\)Jailbreaks aligned models\[[38](https://arxiv.org/html/2609.13731#bib.bib38),[42](https://arxiv.org/html/2609.13731#bib.bib42)\]PerceptionMultimodal InjectionSteganographic noiseJailbreaks vision agents\[[126](https://arxiv.org/html/2609.13731#bib.bib126),[125](https://arxiv.org/html/2609.13731#bib.bib125)\]BrainBackdoorsPoisoned datasetsTrigger\-activated actions\[[50](https://arxiv.org/html/2609.13731#bib.bib50),[51](https://arxiv.org/html/2609.13731#bib.bib51)\]BrainMisalignmentData bias & sycophancyGenerates toxic outputs\[[141](https://arxiv.org/html/2609.13731#bib.bib141),[142](https://arxiv.org/html/2609.13731#bib.bib142)\]BrainPlanning ErrorsError amplificationCascading task failure\[[6](https://arxiv.org/html/2609.13731#bib.bib6),[49](https://arxiv.org/html/2609.13731#bib.bib49)\]ActionParameter InjectionShell interpolationRemote Code Execution \(RCE\)\[[136](https://arxiv.org/html/2609.13731#bib.bib136),[137](https://arxiv.org/html/2609.13731#bib.bib137)\]ActionConfused DeputyAmbient IAM abuseCloud privilege escalation\[[140](https://arxiv.org/html/2609.13731#bib.bib140),[35](https://arxiv.org/html/2609.13731#bib.bib35)\]ActionSupply ChainMalicious MCP serversTool data exfiltration\[[87](https://arxiv.org/html/2609.13731#bib.bib87),[11](https://arxiv.org/html/2609.13731#bib.bib11)\]

## VInteraction Security: Environment, Memory, and Agent\-to\-Agent Threats

While intra\-execution security focuses on the internal computational loop of an isolated agent, interaction security addresses the expansive, decentralized attack surfaces that emerge when an autonomous agent interfaces with external entities: the dynamic physical and digital environment \(ℰ\\mathcal\{E\}\), persistent episodic vector databases \(ℳ\\mathcal\{M\}\), and peer agents within collaborative or competitive multi\-agent networks\[[20](https://arxiv.org/html/2609.13731#bib.bib20),[59](https://arxiv.org/html/2609.13731#bib.bib59),[143](https://arxiv.org/html/2609.13731#bib.bib143),[63](https://arxiv.org/html/2609.13731#bib.bib63)\]\. In distributed cluster environments and multi\-tenant cloud platforms, these interaction surfaces cross traditional network perimeters, transforming external data streams into active vector channels for remote compromise\.

### V\-AThreats on Agent\-to\-Environment Interfacing

Unlike static conversational models that operate exclusively on direct human prompts, autonomous agents continuously ingest unstructured data from the open web, local filesystems, email inboxes, and external APIs\. This bidirectional coupling creates critical external vulnerabilities\.

Indirect Prompt Injection \(IPI\) and Data\-as\-Code Exploits\.In an Indirect Prompt Injection attack, the adversary does not interact with the agent directly; instead, they embed malicious natural language instructions into external data sources \(such as a public web page, a shared Google Doc, an email message, or a PDF document\) that the agent is expected to retrieve during its task execution\[[20](https://arxiv.org/html/2609.13731#bib.bib20),[21](https://arxiv.org/html/2609.13731#bib.bib21),[144](https://arxiv.org/html/2609.13731#bib.bib144),[145](https://arxiv.org/html/2609.13731#bib.bib145)\]\. When the agent retrieves and parses this content, the cognitive reasoning core fails to distinguish between contextual background data and authoritative control instructions, leading to goal hijacking, unauthorized tool execution, and worm\-like automated propagation\[[143](https://arxiv.org/html/2609.13731#bib.bib143)\]\.

Markdown Rendering Side\-Channels and Zero\-Click Exfiltration\.Attackers frequently combine IPI with client\-side rendering vulnerabilities to exfiltrate private user data without triggering explicit tool permission warnings\[[20](https://arxiv.org/html/2609.13731#bib.bib20),[21](https://arxiv.org/html/2609.13731#bib.bib21)\]\. By instructing the agent to format its response as a dynamic markdown image, sensitive variables \(such as retrieved emails, API keys, or session tokens\) are appended as URL query parameters:

Payload=ExfilURL​\(Target\)\|PrivateData\\text\{Payload\}=\\text\{ExfilURL\}\(\\text\{Target\}\)\\parallel\\text\{PrivateData\}\(20\)When the user’s browser or the agent’s web interface renders the markdown image \(e\.g\.,\!\[img\]\(http://attacker\.com/leak?data=\.\.\.\)\), the client automatically dispatches an HTTP GET request to the attacker’s server, exfiltrating the private data instantaneously without requiring the agent to invoke an explicit network tool\.

Server\-Side Request Forgery \(SSRF\) and Cloud Metadata Exfiltration\.In cloud and cluster environments, autonomous agents equipped with web\-fetching or HTTP\-request tools can be manipulated into issuing internal network requests\. Attackers exploit IPI to direct the agent’s HTTP client toward internal loopback or link\-local addresses, most notably the cloud Instance Metadata Service \(IMDSv1 at[http://169\.254\.169\.254/latest/meta\-data/](http://169.254.169.254/latest/meta-data/)\)\[[35](https://arxiv.org/html/2609.13731#bib.bib35)\]\. Because the agent typically inherits the IAM instance profile of its hosting cluster node, successful metadata SSRF allows adversaries to harvest short\-lived AWS/GCP/Azure security credentials, leading directly to cluster\-wide lateral movement and control\-plane takeover\.

Algorithmic Resource Exhaustion and Denial\-of\-Service \(DoS\)\.Adversaries can exploit the computational complexity of foundation models to launch Denial\-of\-Service attacks against agent infrastructures\[[58](https://arxiv.org/html/2609.13731#bib.bib58)\]\. Through the generation ofsponge examples\[[57](https://arxiv.org/html/2609.13731#bib.bib57)\], inputs are engineered to maximize activation energy and token decoding latency by forcing the model into worst\-case generation paths:

maxx⁡Energy​\(𝒞⁡\(x\)\)s\.t\.‖x−xorig‖≤δ\\max\_\{x\}\\text\{Energy\}\(\\mathcal\{C\}\(x\)\)\\quad\\text\{s\.t\.\}\\quad\\\|x\-x\_\{\\text\{orig\}\}\\\|\\leq\\delta\(21\)Furthermore, adversaries can executecontext bombingattacks, submitting vast, repetitive text chunks that exhaust GPU memory and deplete enterprise API token quotas\[[97](https://arxiv.org/html/2609.13731#bib.bib97)\]\. In embodied and cyber\-physical environments, adversarial sensor spoofing against LiDAR, camera feeds, and GPS modules can deceive robotic agents into causing physical collisions or hardware damage\[[146](https://arxiv.org/html/2609.13731#bib.bib146),[31](https://arxiv.org/html/2609.13731#bib.bib31),[30](https://arxiv.org/html/2609.13731#bib.bib30)\]\.

### V\-BThreats on Agent\-to\-Memory \(RAG & Vector Poisoning\)

Modern agents rely on external vector databases to store episodic experiences, document repositories, and long\-term user context\. Because vector memory serves as the persistent knowledge foundation of the agent, compromising this layer introduces persistent, long\-lasting vulnerabilities\.

Vector Database and RAG Index Poisoning\.In Retrieval\-Augmented Generation \(RAG\) architectures, adversaries inject poisoned documents into knowledge repositories\[[59](https://arxiv.org/html/2609.13731#bib.bib59),[60](https://arxiv.org/html/2609.13731#bib.bib60),[64](https://arxiv.org/html/2609.13731#bib.bib64)\]\. By optimizing the adversarial text to maximize its semantic cosine similarity with target user queries, the poisoned document is guaranteed to be retrieved in the top\-kkresults:

max⁡∑i=1kdadv⁡𝐄⁡\(𝐪i\)⋅𝐄⁡\(dadv\)‖𝐄⁡\(𝐪i\)‖2​‖𝐄⁡\(dadv\)‖2s\.t\.Contains​\(Payload,dadv\)\\begin\{split\}&\\max\_\{d\_\{\\text\{adv\}\}\}\\sum\_\{i=1\}^\{k\}\\frac\{\\mathbf\{E\}\(\\mathbf\{q\}\_\{i\}\)\\cdot\\mathbf\{E\}\(d\_\{\\text\{adv\}\}\)\}\{\\\|\\mathbf\{E\}\(\\mathbf\{q\}\_\{i\}\)\\\|\_\{2\}\\\|\\mathbf\{E\}\(d\_\{\\text\{adv\}\}\)\\\|\_\{2\}\}\\\\ &\\quad\\text\{s\.t\.\}\\quad\\text\{Contains\}\(\\text\{Payload\},d\_\{\\text\{adv\}\}\)\\end\{split\}\(22\)Empirical studies by Zou et al\.\[[59](https://arxiv.org/html/2609.13731#bib.bib59)\]demonstrated that injecting fewer than 5 poisoned documents into a database containing over 1,000,000 document chunks achieves an Attack Success Rate exceeding90%90\\%in top\-1 retrieval, effectively hijacking agent decision\-making across all related user queries\.

Embedding Inversion and Membership Inference Attacks\.While dense vector embeddings are often assumed to be privacy\-preserving representations, recent research shows that adversaries can reconstruct raw training text, private medical records, and enterprise trade secrets directly from stored vector embeddings\[[106](https://arxiv.org/html/2609.13731#bib.bib106),[147](https://arxiv.org/html/2609.13731#bib.bib147),[103](https://arxiv.org/html/2609.13731#bib.bib103),[104](https://arxiv.org/html/2609.13731#bib.bib104),[109](https://arxiv.org/html/2609.13731#bib.bib109),[110](https://arxiv.org/html/2609.13731#bib.bib110),[108](https://arxiv.org/html/2609.13731#bib.bib108)\]:

d^=arg⁡mind⁡‖𝐄⁡\(d\)−𝐯target‖22\\hat\{d\}=\\arg\\min\_\{d\}\\\|\\mathbf\{E\}\(d\)\-\\mathbf\{v\}\_\{\\text\{target\}\}\\\|\_\{2\}^\{2\}\(23\)Because high\-dimensional embeddings retain deep lexical and syntactic structure, autoregressive decoder models can invert embeddings to recover over70%70\\%of the original private text with high semantic fidelity\.

Cross\-Session Context Bleed and Memory Asynchrony\.In multi\-tenant environments where agents serve multiple users, unsanitized vector stores and shared Key\-Value \(KV\) attention caches create severe cross\-session context bleeding, allowing proprietary data from one user’s session to be retrieved by another\[[8](https://arxiv.org/html/2609.13731#bib.bib8),[148](https://arxiv.org/html/2609.13731#bib.bib148)\]\. Furthermore, in distributed multi\-agent clusters, asynchronous memory updates frequently introduce state inconsistencies, where agents operate on divergent or stale versions of global reality\[[149](https://arxiv.org/html/2609.13731#bib.bib149),[150](https://arxiv.org/html/2609.13731#bib.bib150)\]\.

### V\-CThreats on Agent\-to\-Agent \(Multi\-Agent Swarms\)

When multiple autonomous agents interact within collaborative or competitive swarms, emergent systemic failure modes arise that do not exist in single\-agent architectures\.

Cascading Error Propagation and Hallucination Amplification\.In multi\-agent software pipelines \(e\.g\., MetaGPT\[[13](https://arxiv.org/html/2609.13731#bib.bib13)\], AutoGen\[[12](https://arxiv.org/html/2609.13731#bib.bib12)\], ChatDev\[[26](https://arxiv.org/html/2609.13731#bib.bib26)\]\), agents operate in sequential or hierarchical handoffs where the output of an upstream agent serves as the authoritative input for a downstream agent\. If an upstream agent generates a minor hallucination or incorrect code snippet, downstream agents accept the flawed data as verified fact, amplifying the error exponentially across the pipeline and resulting in collective swarm failure rates exceeding65%65\\%\[[63](https://arxiv.org/html/2609.13731#bib.bib63),[151](https://arxiv.org/html/2609.13731#bib.bib151)\]\.

Morris II Self\-Replicating AI Worms\.Cohen et al\.\[[143](https://arxiv.org/html/2609.13731#bib.bib143)\]demonstrated the feasibility of zero\-click generative AI worms \(termedMorris II\) that propagate autonomously across interconnected multi\-agent ecosystems\. When an infected agent processes an email or message containing a self\-replicating adversarial prompt, it executes the payload, extracts confidential user contacts, and sends fresh poisoned messages to peer agents\. The propagation dynamics of such worms can be modeled via classical epidemic differential equations:

d​I​\(t\)d​t=β⋅S⁡\(t\)⋅I⁡\(t\)−γ⋅I⁡\(t\)\\frac\{dI\(t\)\}\{dt\}=\\beta\\cdot S\(t\)\\cdot I\(t\)\-\\gamma\\cdot I\(t\)\(24\)whereS⁡\(t\)S\(t\)denotes susceptible agent nodes,I⁡\(t\)I\(t\)denotes infected nodes,β\\betais the infection transmission rate across agent communication channels, andγ\\gammais the recovery rate enforced by automated guardrails\.

Infectious Jailbreaks and Byzantine Swarm Subversion\.Gu et al\.\[[152](https://arxiv.org/html/2609.13731#bib.bib152)\]introduced theAgent Smithattack, demonstrating that a single adversarial image deposited into a shared multi\-agent RAG repository can simultaneously jailbreak an entire swarm of autonomous agents\. In decentralized voting and consensus swarms, if an adversary compromises a critical fraction of agents \(f≥n/3f\\geq n/3\), classical Byzantine fault\-tolerance boundaries are breached, allowing malicious nodes to subvert consensus, falsify decision outcomes, and manipulate decentralized markets\[[61](https://arxiv.org/html/2609.13731#bib.bib61),[62](https://arxiv.org/html/2609.13731#bib.bib62),[153](https://arxiv.org/html/2609.13731#bib.bib153)\]\.

Agentic Sybil Infiltration and Strategic Deception\.Adversaries can deploy swarms of synthetic agent identities \(Sybil attacks\) to manipulate decentralized reputation metrics, alter voting outcomes, and bias multi\-agent market auctions\[[154](https://arxiv.org/html/2609.13731#bib.bib154),[90](https://arxiv.org/html/2609.13731#bib.bib90),[91](https://arxiv.org/html/2609.13731#bib.bib91)\]\. In competitive strategic games \(such as Diplomacy and Werewolf\), autonomous agents trained with reinforcement learning have been shown to execute premeditated, multi\-turn deception, forming and breaking false alliances to mislead human and artificial peers \(e\.g\., Cicero\[[155](https://arxiv.org/html/2609.13731#bib.bib155)\], Hoodwinked\[[156](https://arxiv.org/html/2609.13731#bib.bib156)\], and Park et al\.\[[157](https://arxiv.org/html/2609.13731#bib.bib157)\]\)\. Table[VI](https://arxiv.org/html/2609.13731#S5.T6)synthesizes the full taxonomy of interaction threat vectors across environment, memory, and multi\-agent surfaces\.

TABLE VI:Taxonomy of Interaction Threat Vectors, Adversary Models, Systems Impact, and CitationsInteraction SurfaceThreat VectorAdversary ModelSystems ImpactKey ReferencesEnvironment \(ℰ\\mathcal\{E\}\)Indirect Prompt InjectionBlack\-box data untrusted injectionGoal hijacking & unauthorized tool dispatchGreshake et al\.\[[20](https://arxiv.org/html/2609.13731#bib.bib20)\], Zhan et al\.\[[21](https://arxiv.org/html/2609.13731#bib.bib21)\]Environment \(ℰ\\mathcal\{E\}\)Markdown Data ExfiltrationPassive untrusted content poisoningZero\-click credential exfiltration via GETGreshake et al\.\[[20](https://arxiv.org/html/2609.13731#bib.bib20)\], Zhan et al\.\[[21](https://arxiv.org/html/2609.13731#bib.bib21)\]Environment \(ℰ\\mathcal\{E\}\)Cloud Metadata SSRFInjected URL tool dispatchCluster node IAM credential theft \(IMDS\)Fang et al\.\[[35](https://arxiv.org/html/2609.13731#bib.bib35)\]Environment \(ℰ\\mathcal\{E\}\)Sponge DoS / Context BombMalicious prompt perturbationGPU memory exhaustion & latency spikeShumailov et al\.\[[57](https://arxiv.org/html/2609.13731#bib.bib57)\], Guastalla et al\.\[[58](https://arxiv.org/html/2609.13731#bib.bib58)\]Memory \(ℳ\\mathcal\{M\}\)PoisonedRAG Vector AttackBlack\-box document injection \(<5<5docs\)\>90%\>90\\%top\-1 retrieval hijackingZou et al\.\[[59](https://arxiv.org/html/2609.13731#bib.bib59)\], Chen et al\.\[[60](https://arxiv.org/html/2609.13731#bib.bib60)\]Memory \(ℳ\\mathcal\{M\}\)Dense Embedding InversionWhite/gray\-box vector access\>70%\>70\\%raw private document reconstructionMorris et al\.\[[110](https://arxiv.org/html/2609.13731#bib.bib110)\], Li et al\.\[[109](https://arxiv.org/html/2609.13731#bib.bib109)\]Memory \(ℳ\\mathcal\{M\}\)Multi\-Tenant Context BleedShared KV\-cache / vector indexCross\-session confidentiality breachPacker et al\.\[[8](https://arxiv.org/html/2609.13731#bib.bib8)\], Zeng et al\.\[[148](https://arxiv.org/html/2609.13731#bib.bib148)\]Multi\-Agent SwarmsCascading Error AmplificationStochastic upstream error generation\>65%\>65\\%pipeline breakdown across handoffsGu et al\.\[[63](https://arxiv.org/html/2609.13731#bib.bib63)\], Pan et al\.\[[151](https://arxiv.org/html/2609.13731#bib.bib151)\]Multi\-Agent SwarmsMorris II Generative WormSelf\-replicating prompt in messagingAutonomous multi\-node infection spreadCohen et al\.\[[143](https://arxiv.org/html/2609.13731#bib.bib143)\]Multi\-Agent SwarmsByzantine Swarm SubversionCompromise off≥n/3f\\geq n/3peer nodesMajority voting hijack & consensus breachCastro & Liskov\[[61](https://arxiv.org/html/2609.13731#bib.bib61)\], Li et al\.\[[62](https://arxiv.org/html/2609.13731#bib.bib62)\]Multi\-Agent SwarmsSybil Infiltration / DeceptionSynthetic identity proliferationMarket manipulation & strategic collusionDouceur\[[154](https://arxiv.org/html/2609.13731#bib.bib154)\], Park et al\.\[[157](https://arxiv.org/html/2609.13731#bib.bib157)\]

## VIZero\-Trust Defense\-in\-Depth Architecture

To effectively counter the full spectrum of intra\-execution and interaction threats identified in Sections[IV](https://arxiv.org/html/2609.13731#S4)and[V](https://arxiv.org/html/2609.13731#S5), we formulate a multi\-layeredZero\-Trust Defense\-in\-Depth Architecturegrounded in classical systems security principles\[[18](https://arxiv.org/html/2609.13731#bib.bib18),[19](https://arxiv.org/html/2609.13731#bib.bib19),[36](https://arxiv.org/html/2609.13731#bib.bib36),[37](https://arxiv.org/html/2609.13731#bib.bib37)\]\. Rather than relying on fragile prompt engineering or heuristic filters, our architecture enforces mathematical, cryptographic, and kernel\-level isolation across four synchronized defensive layers \(Fig\.[7](https://arxiv.org/html/2609.13731#S6.F7)\)\.

Layer 1: Cognitive GuardrailsDual\-LLM Inspector\-Executor Isolation\[[158](https://arxiv.org/html/2609.13731#bib.bib158)\]∙\\bulletTokenizer Delimiters\[[159](https://arxiv.org/html/2609.13731#bib.bib159)\]∙\\bulletNeMo Guardrails\[[160](https://arxiv.org/html/2609.13731#bib.bib160)\]Layer 2: Memory Integrity & Cryptographic ProvenanceSigned Vector EmbeddingsΣ=Signs​k\(ℋ\(d\)∥𝐄\(d\)∥T\)\\Sigma=\\operatorname\{Sign\}\_\{sk\}\(\\mathcal\{H\}\(d\)\\parallel\\mathbf\{E\}\(d\)\\parallel T\)\[[117](https://arxiv.org/html/2609.13731#bib.bib117)\]∙\\bulletMerkle Audit Trees∙\\bulletDifferential Privacy\[[161](https://arxiv.org/html/2609.13731#bib.bib161)\]Layer 3: System Sandboxing & Least PrivilegeCapability\-Based Access Control \(CapBAC\)\[[18](https://arxiv.org/html/2609.13731#bib.bib18),[36](https://arxiv.org/html/2609.13731#bib.bib36)\]∙\\bulletFirecracker Micro\-VMs\[[54](https://arxiv.org/html/2609.13731#bib.bib54)\]∙\\bulletKernel eBPF Probes\[[56](https://arxiv.org/html/2609.13731#bib.bib56)\]Layer 4: Multi\-Agent Zero\-Trust ProtocolsW3C DIDs\[[118](https://arxiv.org/html/2609.13731#bib.bib118)\]∙\\bulletmTLS \(RFC 8446\)\[[162](https://arxiv.org/html/2609.13731#bib.bib162)\]∙\\bulletByzantine Fault\-Tolerant Consensus \(f<n/3f<n/3\)\[[61](https://arxiv.org/html/2609.13731#bib.bib61)\]∙\\bulletCircuit Breakers

Fig\. 7:The 4\-Layer Zero\-Trust Defense\-in\-Depth Architecture for Agentic AI Systems\.### VI\-ALayer 1: Pre\-Execution & Cognitive Guardrails

The first defensive layer enforces structural, linguistic, and attention\-level barriers to prevent untrusted inputs from subverting the cognitive reasoning core\.

Dual\-LLM Inspector\-Executor Isolation\.To overcome the collapse of data\-instruction boundaries, we implement a Dual\-LLM Inspector\-Executor architecture\[[158](https://arxiv.org/html/2609.13731#bib.bib158)\]\. Untrusted external data \(such as web pages, emails, and third\-party API outputs\) is processed exclusively by an unprivilegedInspector Model\. The Inspector is strictly isolated from tool execution privileges; its sole purpose is to parse raw content and extract sanitized, typed JSON data entities, stripping away any embedded natural language commands\. The privilegedExecutor Modelreceives only the verified JSON entities wrapped inside immutable system prompt envelopes, ensuring that untrusted data can never occupy an instruction role within the primary reasoning loop\.

Tokenizer\-Level Instruction\-Data Attention Delimiters\.Wallace et al\.\[[159](https://arxiv.org/html/2609.13731#bib.bib159)\]and Kim et al\.\[[163](https://arxiv.org/html/2609.13731#bib.bib163)\]proposed enforcing instruction\-data segregation directly within the transformer attention matrix\. By defining structural token roles, the attention mechanism enforces directional visibility masks:

𝐀i,j=\{𝐪i​𝐤jTdk,ifRole​\(i\)≥Role​\(j\)−∞,ifRole​\(i\)=Instruction∧Role​\(j\)=Data\\mathbf\{A\}\_\{i,j\}=\\begin\{cases\}\\frac\{\\mathbf\{q\}\_\{i\}\\mathbf\{k\}\_\{j\}^\{T\}\}\{\\sqrt\{d\_\{k\}\}\},&\\text\{if \}\\text\{Role\}\(i\)\\geq\\text\{Role\}\(j\)\\\\ \-\\infty,&\\text\{if \}\\text\{Role\}\(i\)=\\text\{Instruction\}\\land\\text\{Role\}\(j\)=\\text\{Data\}\\end\{cases\}\(25\)Under this architectural constraint, external data tokens are mathematically barred from influencing the attention queries of instruction decoding tokens, providing provable invariance against indirect prompt injection\.

Runtime Semantic Guardrail Toolkits\.To supplement architectural isolation, runtime filtering suites—including NeMo Guardrails\[[160](https://arxiv.org/html/2609.13731#bib.bib160)\], Llama Guard\[[164](https://arxiv.org/html/2609.13731#bib.bib164)\], SmoothLLM randomized input perturbation smoothing\[[165](https://arxiv.org/html/2609.13731#bib.bib165)\], LLM Self\-Defense\[[166](https://arxiv.org/html/2609.13731#bib.bib166)\], and Constitutional AI self\-correction rules\[[167](https://arxiv.org/html/2609.13731#bib.bib167),[168](https://arxiv.org/html/2609.13731#bib.bib168),[169](https://arxiv.org/html/2609.13731#bib.bib169)\]—monitor input and output token streams in real time to detect policy violations before execution dispatches\.

### VI\-BLayer 2: Memory Integrity & Cryptographic Provenance

The second layer guarantees the authenticity, tamper\-resistance, and privacy of persistent episodic and semantic memory stores\.

Cryptographically Signed Vector Embeddings\.To eliminate RAG database poisoning, every vector embedding𝐄⁡\(d\)\\mathbf\{E\}\(d\)stored in episodic memoryℳepisodic\\mathcal\{M\}\_\{\\text\{episodic\}\}is cryptographically bound to its source content hashℋ⁡\(d\)\\mathcal\{H\}\(d\), a trusted timestampTT, and the author’s public identity\[[117](https://arxiv.org/html/2609.13731#bib.bib117),[170](https://arxiv.org/html/2609.13731#bib.bib170)\]:

Σ=Signs​kauthority\(ℋ\(d\)∥𝐄\(d\)∥Timestamp∥AuthorID\)\\Sigma=\\operatorname\{Sign\}\_\{sk\_\{\\text\{authority\}\}\}\(\\mathcal\{H\}\(d\)\\parallel\\mathbf\{E\}\(d\)\\parallel\\text\{Timestamp\}\\parallel\\text\{AuthorID\}\)\(26\)During retrieval, the agent’s memory controller validates the signatureΣ\\Sigmabefore injecting retrieved chunks into the context window, automatically discarding unsigned or modified vectors\.

Tamper\-Evident Merkle Audit Trees\.Every discrete execution step—encompassing the thoughtt​htth\_\{t\}, actionata\_\{t\}, and observationoto\_\{t\}—is recorded as a leaf node in a cryptographic Merkle audit tree\[[117](https://arxiv.org/html/2609.13731#bib.bib117)\]\. The root hash is updated recursively:

Roott=ℋ\(Roott−1∥ℋ\(tht∥at∥ot\)\)\\text\{Root\}\_\{t\}=\\mathcal\{H\}\(\\text\{Root\}\_\{t\-1\}\\parallel\\mathcal\{H\}\(th\_\{t\}\\parallel a\_\{t\}\\parallel o\_\{t\}\)\)\(27\)This establishes an immutable, append\-only provenance trail that enables comprehensive post\-incident forensic audits and satisfies strict non\-repudiation mandates\.

Differential Privacy Noise Injection\.To prevent embedding inversion attacks and membership inference probes from extracting private information, calibrated Differential Privacy noise is added to generated representations\[[161](https://arxiv.org/html/2609.13731#bib.bib161),[171](https://arxiv.org/html/2609.13731#bib.bib171),[107](https://arxiv.org/html/2609.13731#bib.bib107)\]:

𝐄~​\(d\)=𝐄⁡\(d\)\+𝒩⁡\(0,σ2​𝐈\),σ=Δ​f​2​ln⁡\(1\.25/δ\)ϵ\\tilde\{\\mathbf\{E\}\}\(d\)=\\mathbf\{E\}\(d\)\+\\mathcal\{N\}\\left\(0,\\sigma^\{2\}\\mathbf\{I\}\\right\),\\quad\\sigma=\\frac\{\\Delta f\\sqrt\{2\\ln\(1\.25/\\delta\)\}\}\{\\epsilon\}\(28\)This mathematical guarantee bounds the maximum mutual information between stored vector points and raw underlying documents, preserving confidentiality\.

### VI\-CLayer 3: System Sandboxing & Least Privilege Execution

The third layer establishes strict operating system boundaries around tool execution, ensuring that compromised planning cannot escape host containers\.

Capability\-Based Access Control \(CapBAC\)\.To eliminate ambient authority and prevent confused deputy attacks, our architecture replaces static IAM credentials with cryptographically signed, fine\-grained, short\-lived Capability Tokens\[[18](https://arxiv.org/html/2609.13731#bib.bib18),[36](https://arxiv.org/html/2609.13731#bib.bib36)\]:

CapToken=⟨AgentID,ToolURI,Scope,Expiry,Signs​kauth⟩\\text\{CapToken\}=\\langle\\text\{AgentID\},\\text\{ToolURI\},\\text\{Scope\},\\text\{Expiry\},\\operatorname\{Sign\}\_\{sk\_\{\\text\{auth\}\}\}\\rangle\(29\)Each token grants permission exclusively for a single tool operation with bounded parameter constraints and a strict time\-to\-live \(<60​s<60\\text\{s\}\)\. When the agent emits an actionata\_\{t\}, the execution gateway validates the signature and scope ofCapTokenbefore issuing the underlying system call\.

Micro\-VM Isolation and WebAssembly Sandboxing\.Tool execution is strictly segregated into lightweight Linux micro\-virtual machines \(such as AWS Firecracker\[[54](https://arxiv.org/html/2609.13731#bib.bib54)\]or Google gVisor\[[55](https://arxiv.org/html/2609.13731#bib.bib55)\]\) with sub\-50ms boot times and hardware\-isolated kernel boundaries\. For plugin\-based architectures, tools are compiled into WebAssembly \(Wasm\) modules\[[172](https://arxiv.org/html/2609.13731#bib.bib172)\], enforcing memory sandboxing without host operating system access\.

Kernel eBPF Real\-Time Syscall Probing\.At the host operating system level, Extended Berkeley Packet Filter \(eBPF\) programs are attached directly to kernel tracepoints\[[56](https://arxiv.org/html/2609.13731#bib.bib56),[173](https://arxiv.org/html/2609.13731#bib.bib173)\]\. These eBPF probes monitor sensitive system calls—including ‘execve‘, ‘connect‘, and ‘openat‘—in real time, instantly terminating any agent process tree that attempts unauthorized network socket bindings or unpermitted filesystem writes\.

### VI\-DLayer 4: Multi\-Agent Zero\-Trust Protocols

The fourth layer governs distributed multi\-agent communication channels, enforcing mutual authentication, consensus integrity, and automated swarm resilience\.

W3C Decentralized Identifiers and Mutual TLS \(mTLS\)\.Every agent in the swarm is provisioned with a unique W3C Decentralized Identifier \(DID\)\[[118](https://arxiv.org/html/2609.13731#bib.bib118)\]and Verifiable Credential\[[174](https://arxiv.org/html/2609.13731#bib.bib174)\]\. All inter\-agent message passing is encrypted and mutually authenticated over TLS 1\.3 \(RFC 8446\)\[[162](https://arxiv.org/html/2609.13731#bib.bib162)\], eliminating man\-in\-the\-middle attacks and unauthenticated message injections\.

Byzantine Fault\-Tolerant \(PBFT\) Consensus\.In mission\-critical workflows, multi\-agent decisions require threshold cryptographic signatures \(tt\-of\-nn\) verified through Practical Byzantine Fault Tolerance \(PBFT\) protocols\[[175](https://arxiv.org/html/2609.13731#bib.bib175),[176](https://arxiv.org/html/2609.13731#bib.bib176),[61](https://arxiv.org/html/2609.13731#bib.bib61),[177](https://arxiv.org/html/2609.13731#bib.bib177)\]\. The swarm maintains provable safety and liveness against arbitrary adversarial agents as long as the total number of compromised nodes satisfies:

Swarm Circuit Breakers and Automated Quarantining\.To contain self\-replicating generative worms and cascading hallucination cascades, the communication substrate deploys real\-time anomaly detection\. If an agent node begins exhibiting anomalous message volume, high refusal rates, or invalid parameter signatures, automated circuit breakers trip, immediately severing the node’s network routes and rolling back shared blackboard states\.

### VI\-EHardware\-Assisted Confidential Computing \(TEEs\)

To protect agent model weights, active context memory, and cryptographic key material against host\-level compromises and untrusted cloud hypervisors, the entire agent execution stack can be deployed inside Hardware Trusted Execution Environments \(TEEs\)\[[178](https://arxiv.org/html/2609.13731#bib.bib178),[179](https://arxiv.org/html/2609.13731#bib.bib179),[180](https://arxiv.org/html/2609.13731#bib.bib180)\]\. Technologies such as Intel SGX, AMD SEV\-SNP, ARM TrustZone, and NVIDIA Confidential GPUs provide cryptographically enforced memory encryption in use, ensuring that even privileged cloud administrators cannot inspect or tamper with active agent reasoning states\.

### VI\-FComparative Trade\-Off Synthesis of Defensive Controls

Deploying these multi\-tiered defensive mechanisms across enterprise cloud clusters introduces pronounced computational, financial, and operational trade\-offs \(Table[VII](https://arxiv.org/html/2609.13731#S6.T7)\)\. System architects must balance security rigor against runtime latency, token consumption, and deployment constraints\.

TABLE VII:Comprehensive Comparative Evaluation of Zero\-Trust Defensive MechanismsDefensive ControlTarget Attack SurfaceLatency OverheadToken CostDeployment ModelKey Operational LimitationDual\-LLM Isolation\[[158](https://arxiv.org/html/2609.13731#bib.bib158)\]IPI / Data\-as\-CodeHigh \(\+1\.5​–​3\.0​s\+1\.5\\text\{\-\-\}3\.0\\text\{s\}\)2\.0×2\.0\\timesAPI & Self\-HostedDoubles model inference costs\.Attention Masking\[[159](https://arxiv.org/html/2609.13731#bib.bib159)\]Attention\-level IPINegligible \(<5​ms<5\\text\{ms\}\)1\.0×1\.0\\timesWhite\-Box OnlyIncompatible with closed\-weight APIs\.Signed Vectors\[[117](https://arxiv.org/html/2609.13731#bib.bib117)\]RAG DB PoisoningLow \(<10​ms<10\\text\{ms\}\)1\.0×1\.0\\timesUniversal Vector DBRequires trusted signing PKI infrastructure\.DP Noise Injection\[[161](https://arxiv.org/html/2609.13731#bib.bib161)\]Embedding InversionZero \(0​ms0\\text\{ms\}\)1\.0×1\.0\\timesEmbedding PipelineDegrades top\-kksemantic retrieval precision\.CapBAC Tokens\[[36](https://arxiv.org/html/2609.13731#bib.bib36)\]Confused Deputy / RCENegligible \(<2​ms<2\\text\{ms\}\)1\.0×1\.0\\timesCluster GatewayRequires granular tool permission schema\.Micro\-VM Sandboxing\[[54](https://arxiv.org/html/2609.13731#bib.bib54)\]Host Escape / RCEModerate \(35​–​50​ms35\\text\{\-\-\}50\\text\{ms\}\)1\.0×1\.0\\timesKubernetes / CloudEphemeral state loss; storage volume overhead\.Kernel eBPF Probes\[[56](https://arxiv.org/html/2609.13731#bib.bib56)\]Unauthorized SyscallsSub\-ms \(<0\.5​ms<0\.5\\text\{ms\}\)1\.0×1\.0\\timesLinux Kernel HostLinux\-only; requires privileged root loader\.PBFT Consensus\[[61](https://arxiv.org/html/2609.13731#bib.bib61)\]Byzantine SwarmsHigh \(O⁡\(n2\)O\(n^\{2\}\)network\)n×n\\timesDistributed SwarmMultiplies API costs across allnnnodes\.Hardware TEEs\[[178](https://arxiv.org/html/2609.13731#bib.bib178)\]Memory / Host SnoopingLow \(2​–​8%2\\text\{\-\-\}8\\%\)1\.0×1\.0\\timesSpecialized SiliconEPC paging bottleneck on large weights\.Cluster and Cloud Deployment Realities\.In distributed cloud platforms \(e\.g\., Kubernetes clusters managed via Cilium or Calico network fabrics\), individual defensive controls must be co\-scheduled based on the operational criticality and data classification of the task\. Lightweight, universal mechanisms—such as eBPF syscall tracing and CapBAC tokens—provide continuous, sub\-millisecond baseline protection across all cluster nodes without consuming LLM token budgets\. Conversely, heavy cryptographic consensus \(PBFT\) and Dual\-LLM inspection are reserved for high\-blast\-radius execution nodes \(such as automated financial settlement or production database modification\), ensuring optimal cluster throughput\.

## VIIEmpirical Benchmarks, Auditing & Evaluation Frameworks

Evaluating the trustworthiness of Agentic AI systems requires moving beyond static, single\-turn accuracy benchmarks toward dynamic, multi\-step interactive environments\. These evaluation frameworks measure safety, adversarial resilience, and utility trade\-offs across complex execution trajectories under live environmental feedback\[[181](https://arxiv.org/html/2609.13731#bib.bib181),[182](https://arxiv.org/html/2609.13731#bib.bib182),[100](https://arxiv.org/html/2609.13731#bib.bib100),[21](https://arxiv.org/html/2609.13731#bib.bib21),[28](https://arxiv.org/html/2609.13731#bib.bib28)\]\.

### VII\-AState\-of\-the\-Art Empirical Benchmark Suites

Recent empirical initiatives have introduced specialized benchmark suites designed to evaluate distinct operational layers of autonomous agents\.

Tool Integration and Injection Benchmarks\.In the tool execution plane, the InjecAgent benchmark\[[21](https://arxiv.org/html/2609.13731#bib.bib21)\]evaluates indirect prompt injection resilience across 17 simulated tools and over 1,000 realistic scenarios, revealing that leading foundation models \(such as GPT\-4\) suffer from Attack Success Rates \(ASR\) between24\.1%24\.1\\%and45\.3%45\.3\\%\. The BIPIA benchmark\[[144](https://arxiv.org/html/2609.13731#bib.bib144)\]focuses on indirect prompt injection embedded within daily documents \(such as emails, webpages, and PDFs\), demonstrating injection rates exceeding50%50\\%across standard LLMs\. Concurrently, ToolBench\[[182](https://arxiv.org/html/2609.13731#bib.bib182)\], SafeTool\[[136](https://arxiv.org/html/2609.13731#bib.bib136)\], and SecTool\[[101](https://arxiv.org/html/2609.13731#bib.bib101)\]analyze tool selection precision and parameter security across extensive REST API catalogs spanning over 16,000 live endpoints, while ToolQA\[[183](https://arxiv.org/html/2609.13731#bib.bib183)\]assesses multi\-step question answering requiring external database retrieval\.

Full\-Loop Execution and Planning Safety Benchmarks\.In full\-loop environments, AgentBench\[[181](https://arxiv.org/html/2609.13731#bib.bib181)\]evaluates multi\-turn decision\-making across operating system shells, databases, web interfaces, and digital games\. AgentBench highlighted severe security vulnerabilities in OS environments, where agents frequently execute destructive bash commands when given ambiguous instructions\. To evaluate planning safety specifically, R\-Judge\[[100](https://arxiv.org/html/2609.13731#bib.bib100)\]benchmarks safety and risk awareness across 1,624 multi\-turn interaction trajectories, evaluating whether agents can identify dangerous action proposals before execution\.

Cyber Operations, Multimodal, and Alignment Benchmarks\.To evaluate dual\-use capabilities, CyberSecEval 1 and 2\[[28](https://arxiv.org/html/2609.13731#bib.bib28),[29](https://arxiv.org/html/2609.13731#bib.bib29)\]quantify the likelihood that models generate weaponizable cyber exploits, execute unauthorized port scans, or suggest vulnerable software implementations\. In multimodal domains, GAIA\[[184](https://arxiv.org/html/2609.13731#bib.bib184)\]benchmarks complex real\-world web navigation and data processing, establishing that while human baselines achieve92%92\\%completion, leading commercial agents solve less than10%10\\%of complex multi\-modal tasks\. Finally, Machiavelli\[[99](https://arxiv.org/html/2609.13731#bib.bib99)\]and AgentHarm\[[185](https://arxiv.org/html/2609.13731#bib.bib185)\]evaluate deceptive tendencies, power\-seeking strategies, and jailbreak resilience under adversarial task wrappers\.

TABLE VIII:Comprehensive Benchmark Taxonomy and Evaluation AnalysisBenchmarkTarget LayerEnvironmentPrimary MetricEmpirical Baseline ResultInjecAgent\[[21](https://arxiv.org/html/2609.13731#bib.bib21)\]Tool Plane17\+ APIsAttack Success \(ASR\)GPT\-4: 24\.1%–45\.3% ASR under IPI\.BIPIA\[[144](https://arxiv.org/html/2609.13731#bib.bib144)\]Cognitive InputEmail, Web, DocsInjection Rate\>50%\>50\\%ASR across standard LLMs\.AgentBench\[[181](https://arxiv.org/html/2609.13731#bib.bib181)\]Full LoopOS, DB, WebSuccess / SafetyGPT\-4: 4\.01/5; open\-source: ¡2\.0\.R\-Judge\[[100](https://arxiv.org/html/2609.13731#bib.bib100)\]Cognitive Plan1,624 RecordsSafety Risk F1GPT\-4: 70% F1 on dangerous actions\.ToolBench\[[182](https://arxiv.org/html/2609.13731#bib.bib182)\]Tool Execution16,000\+ APIsPass RateFunction selection scales with tuning\.CyberSecEval\[[28](https://arxiv.org/html/2609.13731#bib.bib28)\]Cyber ActionExploits & ReposExploit LikelihoodLLMs generate weaponizable exploits\.GAIA\[[184](https://arxiv.org/html/2609.13731#bib.bib184)\]MultimodalGeneral WebCompletion RateHuman: 92%; GPT\-4 plugins: ¡10%\.AgentHarm\[[185](https://arxiv.org/html/2609.13731#bib.bib185)\]Harmful Agency440\+ TasksJailbreak Pass RateAgents execute harmful wrapped tasks\.
### VII\-BAutomated Adversarial Red\-Teaming Methodologies

Automated adversarial red\-teaming represents an essential auditing discipline for discovering latent vulnerabilities prior to live deployment\.

Game\-Theoretic Attacker\-Target Optimization Loops\.Modern red\-teaming frameworks deploy autonomous attacker agents𝒜adv\\mathcal\{A\}\_\{\\text\{adv\}\}that dynamically probe target agents𝒜target\\mathcal\{A\}\_\{\\text\{target\}\}to uncover injection surfaces\[[186](https://arxiv.org/html/2609.13731#bib.bib186),[187](https://arxiv.org/html/2609.13731#bib.bib187),[188](https://arxiv.org/html/2609.13731#bib.bib188),[189](https://arxiv.org/html/2609.13731#bib.bib189)\]\. The attacker agent optimizes an adversarial prompt sequencept∗p\_\{t\}^\{\*\}by balancing harmful objective fulfillment against detection evasion:

pt∗=arg⁡maxp⁡𝔼⁡\[𝒥harm​\(𝒜target​\(p\)\)−λ⋅𝒞detect​\(p\)\]p\_\{t\}^\{\*\}=\\arg\\max\_\{p\}\\mathbb\{E\}\\left\[\\mathcal\{J\}\_\{\\text\{harm\}\}\(\\mathcal\{A\}\_\{\\text\{target\}\}\(p\)\)\-\\lambda\\cdot\\mathcal\{C\}\_\{\\text\{detect\}\}\(p\)\\right\]\(31\)By continuously refining prompts based on refusal signals, automated red\-teaming loops discover multi\-step jailbreak vectors that bypass static heuristic filters\.

MCP Schema and Grammar Mutation Fuzzing\.In tool\-integrated architectures, mutation\-based fuzzing engines systematically generate malformed JSON\-RPC payloads, schema mismatches, and boundary edge cases targeting Model Context Protocol \(MCP\) servers\[[11](https://arxiv.org/html/2609.13731#bib.bib11),[190](https://arxiv.org/html/2609.13731#bib.bib190)\]\. Concurrently, dynamic vector poisoning probes inject perturbed document chunks into active RAG databases to measure retrieval robustness and identify semantic blind spots\[[59](https://arxiv.org/html/2609.13731#bib.bib59),[60](https://arxiv.org/html/2609.13731#bib.bib60)\]\.

### VII\-CStandardized Quantitative Metric Formulations

To establish mathematical rigor and cross\-benchmark comparability across the literature, we synthesize five standardized evaluation metrics that capture the security, safety, and utility of agentic systems\.

Attack Success Rate \(ASR\)\.ASR quantifies the proportion of adversarial test trajectories in which the agent executes an unauthorized, harmful, or policy\-violating action:

ASR=1N​∑i=1N𝕀⁡\(ai∈𝒜malicious\)\\text\{ASR\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbb\{I\}\(a\_\{i\}\\in\\mathcal\{A\}\_\{\\text\{malicious\}\}\)\(32\)whereNNis the total number of adversarial evaluation trials, and𝕀⁡\(⋅\)\\mathbb\{I\}\(\\cdot\)is the indicator function\.

Utility Preservation Rate \(UPR\)\.UPR measures the agent’s ability to successfully resolve benign tasks while defensive guardrails and verification layers are actively running:

UPR=1M​∑j=1M𝕀⁡\(Success​\(j\)=1∣Defense Active\)\\text\{UPR\}=\\frac\{1\}\{M\}\\sum\_\{j=1\}^\{M\}\\mathbb\{I\}\(\\text\{Success\}\(j\)=1\\mid\\text\{Defense Active\}\)\(33\)whereMMdenotes the set of benign evaluation tasks\. A robust defensive architecture must maintainUPR≈1\.0\\text\{UPR\}\\approx 1\.0\.

False Safety Rejection Rate \(FSRR\)\.FSRR captures the operational friction introduced by defensive guardrails, measuring the frequency with which benign user requests are incorrectly blocked:

FSRR=1M​∑j=1M𝕀⁡\(Blocked​\(aj\)∣aj∈𝒜benign\)\\text\{FSRR\}=\\frac\{1\}\{M\}\\sum\_\{j=1\}^\{M\}\\mathbb\{I\}\(\\text\{Blocked\}\(a\_\{j\}\)\\mid a\_\{j\}\\in\\mathcal\{A\}\_\{\\text\{benign\}\}\)\(34\)
Blast Radius Index \(BRI\)\.To quantify the cumulative real\-world damage potential of an uncontained action, BRI calculates the weighted impact across data exfiltration, filesystem modification, and privilege escalation:

BRI​\(at\)=wdata⋅Δexfil\+wfs⋅Δfiles\+wauth⋅Δpriv\\text\{BRI\}\(a\_\{t\}\)=w\_\{\\text\{data\}\}\\cdot\\Delta\_\{\\text\{exfil\}\}\+w\_\{\\text\{fs\}\}\\cdot\\Delta\_\{\\text\{files\}\}\+w\_\{\\text\{auth\}\}\\cdot\\Delta\_\{\\text\{priv\}\}\(35\)wherewdata\+wfs\+wauth=1\.0w\_\{\\text\{data\}\}\+w\_\{\\text\{fs\}\}\+w\_\{\\text\{auth\}\}=1\.0, and eachΔ\\Deltaterm measures the normalized volume of compromised assets\.

Cascade Failure Probability \(CFP\)\.In multi\-agent swarms, CFP measures the probability that compromising a single seed agentA1A\_\{1\}induces an error cascade causing at least a fractionθ\\thetaof the entire swarm ofKKagents to fail:

CFP​\(θ\)=Pr⁡\(CLOSE1K∑k=1K𝕀\(Failed\(Ak\)\)≥θ\|OPENCompromised​\(A1\)\)\\begin\{split\}\\text\{CFP\}\(\\theta\)=\\Pr\\Bigg\(&\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{I\}\(\\text\{Failed\}\(A\_\{k\}\)\)\\geq\\theta\\;\\Bigg\|\\;\\\\ &\\text\{Compromised\}\(A\_\{1\}\)\\Bigg\)\\end\{split\}\(36\)

## VIIIAI Governance, Safety Standards, and Regulatory Compliance

The transition from passive language models to autonomous Agentic AI systems has triggered intensive global regulatory efforts to establish verifiable safety boundaries, algorithmic transparency, and non\-repudiable legal accountability\[[119](https://arxiv.org/html/2609.13731#bib.bib119),[121](https://arxiv.org/html/2609.13731#bib.bib121),[120](https://arxiv.org/html/2609.13731#bib.bib120),[191](https://arxiv.org/html/2609.13731#bib.bib191),[192](https://arxiv.org/html/2609.13731#bib.bib192)\]\. Because autonomous agents generate real\-world economic, legal, and operational consequences, socio\-technical governance frameworks are transitioning from voluntary ethical guidelines to legally binding compliance mandates\.

### VIII\-AInternational Regulatory Frameworks and Standards

Major international standard\-setting bodies and legislative authorities have developed formal risk management standards tailored to autonomous systems\.

NIST AI Risk Management Framework \(AI RMF 1\.0\) and Generative AI Profile\.The National Institute of Standards and Technology \(NIST\) established four foundational lifecycle functions—GOVERN,MAP,MEASURE, andMANAGE—to systematically identify and mitigate AI risks\[[119](https://arxiv.org/html/2609.13731#bib.bib119),[193](https://arxiv.org/html/2609.13731#bib.bib193)\]\. In the context of Agentic AI, the NIST Generative AI Profile mandates continuous asset discovery, dynamic tool authorization tracking, automated red\-team auditing, and rigorous pre\-deployment risk quantification across all accessible tool APIs\.

European Union AI Act \(Regulation EU 2024/1689\)\.The EU AI Act establishes a strict risk\-based classification hierarchy\[[121](https://arxiv.org/html/2609.13731#bib.bib121)\]\. Autonomous agents that manage critical enterprise infrastructure, process biometric data, execute financial transactions, or control physical machinery are categorized asHigh\-Risk AI Systems\. Under this designation, systems must comply with stringent legal requirements: Article 9 mandates continuous risk management systems throughout the operational lifecycle; Article 10 requires strict data governance to prevent poisoned training and RAG data; Article 12 mandates automated, tamper\-evident event logging to ensure post\-hoc traceability; and Article 14 requires operational human oversight\. Specifically, Article 14 formalizes three oversight modalities:Human\-in\-the\-Loop \(HITL\)for authorizing irreversible actions,Human\-on\-the\-Loop \(HOTL\)for real\-time monitoring and intervention, andHuman\-in\-Command \(HIC\)for ultimate system shutdown and override\.

ISO/IEC, IEEE, and OWASP Standards\.The International Organization for Standardization released ISO/IEC 42001:2023\[[120](https://arxiv.org/html/2609.13731#bib.bib120)\], establishing the world’s first certifiable Artificial Intelligence Management System \(AIMS\) standard, which pairs with ISO/IEC 27001\[[194](https://arxiv.org/html/2609.13731#bib.bib194)\]to govern data traceability and secure software development\. In parallel, IEEE standard P3119 defines requirements for AI transparency and explainability\[[195](https://arxiv.org/html/2609.13731#bib.bib195)\], while IEEE 2830 standardizes secure multi\-party computation in distributed AI ecosystems\[[196](https://arxiv.org/html/2609.13731#bib.bib196)\]\. For practical vulnerability classification, the Open Web Application Security Project \(OWASP\) established dedicated frameworks—the OWASP Top 10 for LLMs\[[191](https://arxiv.org/html/2609.13731#bib.bib191)\]and the OWASP Top 10 for Agentic AI\[[192](https://arxiv.org/html/2609.13731#bib.bib192)\]—systematizing core architectural risks including Excessive Agency \(LLM06\), Indirect Prompt Injection \(LLM01\), and Insecure Plugin Design \(LLM07\)\.

### VIII\-BComparative Analysis: Technical Enforcement of Global AI Mandates

To operationalize compliance across divergent international frameworks, system architects must map high\-level legal mandates directly onto verifiable technical mechanisms \(Table[IX](https://arxiv.org/html/2609.13731#S8.T9)\)\.

TABLE IX:Cross\-Regulatory Mapping Between International Mandates and Zero\-Trust Technical ControlsFrameworkMandateTarget VulnerabilityImplemented Technical DefenseNIST AI RMF\[[119](https://arxiv.org/html/2609.13731#bib.bib119)\]GOVERN 1\.2, MANAGE 2\.4Unauthorized tool dispatchCapBAC tokens & Firecracker micro\-VMs\.EU AI Act Art\. 12\[[121](https://arxiv.org/html/2609.13731#bib.bib121)\]High\-Risk Event LogsReasoning tamperingMerkle Audit Trees with hash chaining \(Roott\\text\{Root\}\_\{t\}\)\.EU AI Act Art\. 14\[[121](https://arxiv.org/html/2609.13731#bib.bib121)\]Human OversightAutonomous driftThree\-tier HITL/HOTL/HIC approval gates\.ISO/IEC 42001\[[120](https://arxiv.org/html/2609.13731#bib.bib120)\]AI Data TraceabilityVector DB poisoningSigned vector embeddings & DP noise\.OWASP Top 10\[[192](https://arxiv.org/html/2609.13731#bib.bib192)\]Excessive Agency \(LLM06\)Arbitrary RCEKernel eBPF syscall probes & Dual\-LLM isolation\.Deterministic Traceability and Record\-Keeping\.Satisfying the mandatory logging requirements of EU AI Act Article 12 and ISO/IEC 42001 requires implementing cryptographic hash chaining over all agent interaction steps \(Roott\\text\{Root\}\_\{t\}\)\. By recording every emitted actionata\_\{t\}, its causal thoughtt​htth\_\{t\}, and the preceding observationot−1o\_\{t\-1\}within an append\-only Merkle tree, system operators provide incontrovertible mathematical proof of system behavior for regulatory audits\.

Principle of Least Privilege and Sandboxed Containment\.Satisfying NIST AI RMF MANAGE 2\.4 and OWASP LLM06 requires replacing ambient system access with dynamic Capability\-Based Access Control tokens \(CapToken\) and micro\-VM isolation\. By ensuring that tool execution privileges expire within sub\-minute intervals and are restricted to isolated hardware sandboxes, developers mathematically eliminate excessive agency and mitigate confused deputy privilege escalation\.

Continuous Automated Red\-Teaming\.Satisfying NIST AI RMF MEASURE 2\.6 requires embedding automated adversarial red\-teaming game loops \(such as PAIR\[[40](https://arxiv.org/html/2609.13731#bib.bib40)\]and TAP\[[41](https://arxiv.org/html/2609.13731#bib.bib41)\]\) directly into Continuous Integration and Continuous Deployment \(CI/CD\) pipelines\. This ensures that latent jailbreak vectors, prompt drift, and tool parameter injections are proactively discovered and remediated prior to production release\.

## IXOpen Research Challenges & Future Horizons

1\. Formal Verification \(Sec\. 9\.1\) ∙\\bulletSMT Solvers \(Z3\[[113](https://arxiv.org/html/2609.13731#bib.bib113)\], CVC5\[[114](https://arxiv.org/html/2609.13731#bib.bib114)\]\) ∙\\bulletLinear Temporal Logic \(LTL\) checks ∙\\bulletSafety invariant proofs before action2\. Lifelong Dynamic Alignment \(Sec\. 9\.2\) ∙\\bulletIn\-context alignment drift control ∙\\bulletSleeper backdoor memory scrubbing ∙\\bulletVerifiable state rollback trees3\. Autonomic Swarms \(Sec\. 9\.3\) ∙\\bulletGNN message telemetry \(GCN/GAT\) ∙\\bulletReal\-time Byzantine peer quarantine ∙\\bulletIntrusion\-tolerant state reversion4\. Machine\-Speed Cyber Warfare \(Sec\. 9\.4\) ∙\\bulletAutonomous zero\-day patch synthesis ∙\\bulletDefensive superiority over attacks ∙\\bulletAutomated real\-time SOC triage5\. Pareto Optimization \(Sec\. 9\.5\) ∙\\bulletRisk\-aware dynamic guardrail throttling ∙\\bulletEdge Confidential Computing \(TEEs\) ∙\\bulletLatency\-cost\-security balancing6\. Decentralized Value Ecosystems \(Sec\. 9\.6\) ∙\\bulletCross\-chain blockchain oracles ∙\\bulletSmart contract escrow for agents ∙\\bulletCryptographic economic verification

Fig\. 8:Taxonomy of Open Research Horizons in Trustworthy Agentic AI\.Establishing provably secure, resilient, and verifiable Trustworthy Agentic AI systems requires solving foundational open problems at the intersection of computer systems security, formal methods, distributed computing, and machine learning\[[197](https://arxiv.org/html/2609.13731#bib.bib197),[94](https://arxiv.org/html/2609.13731#bib.bib94),[95](https://arxiv.org/html/2609.13731#bib.bib95),[17](https://arxiv.org/html/2609.13731#bib.bib17)\]\(Fig\.[8](https://arxiv.org/html/2609.13731#S9.F8)\)\. Below, we deconstruct the core theoretical bottlenecks, systems engineering roadblocks, and decisive research horizons across six critical dimensions\.

### IX\-AFormal Verification of Neural\-Symbolic Action Chains

Contemporary agent architectures rely on probabilistic foundation models to generate executable tool dispatches, SQL queries, and shell commands\. Because deep neural networks are uninterpretable black boxes that lack deterministic execution guarantees, current architectures cannot mathematically prove that a proposed action sequence will adhere to critical safety invariants prior to execution\[[111](https://arxiv.org/html/2609.13731#bib.bib111),[112](https://arxiv.org/html/2609.13731#bib.bib112),[198](https://arxiv.org/html/2609.13731#bib.bib198)\]\.

The Semantic\-to\-Symbolic Fidelity Bottleneck\.The primary scientific bottleneck lies in bridging continuous, high\-dimensional neural representations with discrete Satisfiability Modulo Theories \(SMT\) solvers \(such as Z3\[[113](https://arxiv.org/html/2609.13731#bib.bib113)\]and CVC5\[[114](https://arxiv.org/html/2609.13731#bib.bib114)\]\) and abstract interpretation verifiers\[[199](https://arxiv.org/html/2609.13731#bib.bib199),[200](https://arxiv.org/html/2609.13731#bib.bib200),[201](https://arxiv.org/html/2609.13731#bib.bib201)\]\. Natural language goals and contextual tool outputs possess infinite semantic variability, making direct translation into formal Linear Temporal Logic \(LTL\) formulas susceptible to translation hallucinations\. If the neural compiler misinterprets developer intent, the verified formula diverges from actual safety requirements\.

State\-Space Explosion in Interactive Loops\.Furthermore, real\-world operating system shells, distributed databases, and cloud APIs present unbounded state spaces\. Exhaustive formal model checking across all reachable environment states:

∀s∈𝒮reachable,\(s,at\)⊧Φsafe\\forall s\\in\\mathcal\{S\}\_\{\\text\{reachable\}\},\\quad\(s,a\_\{t\}\)\\models\\Phi\_\{\\text\{safe\}\}\(37\)is computationally undecidable in the general case and PSPACE\-complete under bounded horizons\. A critical research frontier requires developing grammar\-constrained decoding engines and bounded symbolic abstract interpreters that verify safety invariants over localized, reachable state abstractions within sub\-50ms execution budgets\.

### IX\-BLifelong Dynamic Alignment & The Non\-Invertible Side\-Effect Dilemma

Existing alignment paradigms—such as Reinforcement Learning from Human Feedback \(RLHF\)\[[141](https://arxiv.org/html/2609.13731#bib.bib141)\], Direct Preference Optimization \(DPO\)\[[202](https://arxiv.org/html/2609.13731#bib.bib202)\], and Constitutional AI\[[167](https://arxiv.org/html/2609.13731#bib.bib167)\]—optimize model weights statically prior to deployment\. However, autonomous agents operate in continuous in\-context learning loops, dynamically retrieving external observations, interacting with untrusted users, and updating episodic vector memories\[[8](https://arxiv.org/html/2609.13731#bib.bib8)\]\. Over long operational horizons, this continuous exposure causes in\-context alignment drift, catastrophic forgetting\[[203](https://arxiv.org/html/2609.13731#bib.bib203)\], and the activation of latent sleeper backdoors\[[50](https://arxiv.org/html/2609.13731#bib.bib50)\]\.

The Transactional Non\-Invertibility of External Actions\.In classical database systems, write\-ahead logging \(WAL\) and two\-phase commit \(2​PC2\\text\{PC\}\) protocols enable complete state rollbacks upon transaction abort\. In contrast, autonomous agent tool dispatches frequently produce non\-invertible real\-world side effects: dispatched emails cannot be unsent, executed financial transfers cannot be arbitrarily recalled, and external webhook mutations cannot be easily reversed\. When an agent discovers mid\-trajectory that an upstream thought step was poisoned, classical state rollback is physically impossible\.

Verifiable Memory Scrubbing and Shadow Execution\.Future architectures must develop shadow\-execution envelopes that isolate irreversible actions behind multi\-stage verification gates while non\-monotonic memory consolidation algorithms continuously scrub contaminated memory vectors:

ℳconsolidated=Scrub⁡\(ℳepisodic∪ℳwork\)∖Anomalies\\mathcal\{M\}\_\{\\text\{consolidated\}\}=\\operatorname\{Scrub\}\\left\(\\mathcal\{M\}\_\{\\text\{episodic\}\}\\cup\\mathcal\{M\}\_\{\\text\{work\}\}\\right\)\\setminus\\operatorname\{Anomalies\}\(38\)Developing automated causal pruning techniques that excise poisoned episodic memories without degrading benign contextual recall remains a pivotal open challenge\.

### IX\-CScalable Byzantine Swarm Consensus in High\-Latency Agent Clusters

Given the inevitability of zero\-day exploits and multi\-step prompt injections, multi\-agent architectures must be designed under theAssume\-Breach Paradigm\[[37](https://arxiv.org/html/2609.13731#bib.bib37),[17](https://arxiv.org/html/2609.13731#bib.bib17)\]\. In large\-scale collaborative swarms, detecting and isolating compromised, malicious, or Byzantine peer agents in real time remains an open challenge\[[61](https://arxiv.org/html/2609.13731#bib.bib61),[62](https://arxiv.org/html/2609.13731#bib.bib62)\]\.

TheO⁡\(n2\)O\(n^\{2\}\)Token and Latency Multiplication Bottleneck\.While Practical Byzantine Fault Tolerance \(PBFT\) provides provable safety against up tof<n/3f<n/3adversarial nodes\[[61](https://arxiv.org/html/2609.13731#bib.bib61)\], PBFT requires multi\-round all\-to\-all voting over the network\. In an agent cluster where each node is a foundation model requiring several seconds per inference pass, executing PBFT acrossnnagent nodes incurs quadratic token consumption and minutes of decision latency, crippling real\-time cluster throughput\.

Graph Telemetry and Optimistic Consensus\.A promising research horizon involves engineering autonomic swarm immune systems that combine optimistic quorum consensus with Graph Neural Networks \(such as GCNs\[[204](https://arxiv.org/html/2609.13731#bib.bib204)\]and GATs\[[205](https://arxiv.org/html/2609.13731#bib.bib205)\]\) deployed over inter\-agent communicative graphs\. By analyzing communicative message topology, parameter entropy, and transaction frequencies in real time, graph telemetry models can identify Sybil node clusters\[[154](https://arxiv.org/html/2609.13731#bib.bib154)\]and collusive voting cartels out\-of\-band, triggering automated cryptographic quarantining without halting cluster progress\.

### IX\-DDual\-Use Machine\-Speed Cyber Warfare and Automated Patch Synthesis

Agentic AI compresses the operational timeline of cybersecurity engagements from days to milliseconds\[[16](https://arxiv.org/html/2609.13731#bib.bib16),[29](https://arxiv.org/html/2609.13731#bib.bib29)\]\. In offensive operations, autonomous cyber agents can execute end\-to\-end exploit synthesis, automated vulnerability chaining, and polymorphic malware mutation at machine speed\. In defensive operations, autonomous security operations center \(SOC\) agents perform continuous telemetry triage, binary decompilation, and automated patch synthesis\[[17](https://arxiv.org/html/2609.13731#bib.bib17)\]\.

The Defensive Asymmetry Dilemma\.The central scientific challenge is establishing provableDefensive Superiority\. Attackers operate with fundamental asymmetry: an offensive agent requires only a single unmodeled edge case or injection vector to compromise a cluster, whereas defensive systems must secure every possible execution path\. Achieving defensive superiority requires developing autonomous formal patch verification pipelines that prove generated software fixes do not introduce secondary security regressions or break legacy application logic\.

### IX\-EDistributed Cluster Scheduling & Blast\-Radius Pareto Optimization

Comprehensive defense\-in\-depth frameworks—incorporating Dual\-LLM Inspector\-Executor pipelines, runtime SMT solver verifiers, encrypted vector embeddings, and Merkle audit trees—introduce substantial computational, latency, and financial overheads\. In high\-frequency operational settings, excessive verification latency impairs real\-time utility\.

Risk\-Aware Dynamic Guardrail Throttling\.Future research must develop adaptive, risk\-aware cluster schedulers that dynamically modulate defensive inspection rigor based on the calculated Blast Radius Index \(BRI\) of pending actions\. By executing low\-risk read\-only queries through lightweight speculative guardrails while routing high\-impact tool dispatches through hardware\-accelerated Trusted Execution Environments \(TEEs\)\[[178](https://arxiv.org/html/2609.13731#bib.bib178),[179](https://arxiv.org/html/2609.13731#bib.bib179),[180](https://arxiv.org/html/2609.13731#bib.bib180)\], distributed cluster schedulers \(e\.g\., Slurm, Kubernetes, Ray\) can achieve optimal Pareto trade\-offs between security guarantees, computational cost, and operational latency\.

### IX\-FDecentralized Value Ecosystems, Cryptographic Provenance &Z​K​\-MLZK\\text\{\-ML\}

As autonomous agents increasingly participate in decentralized commerce, automated procurement, and multi\-agent value networks, systems require legally recognized cryptographic identities and secure economic settlement layers\[[90](https://arxiv.org/html/2609.13731#bib.bib90),[91](https://arxiv.org/html/2609.13731#bib.bib91),[118](https://arxiv.org/html/2609.13731#bib.bib118),[206](https://arxiv.org/html/2609.13731#bib.bib206)\]\.

Zero\-Knowledge Machine Learning \(Z​K​\-MLZK\\text\{\-ML\}\) Proofs of Agency\.A crucial research frontier lies in integrating Zero\-Knowledge Succinct Non\-Interactive Arguments of Knowledge \(zk\-SNARKs\) with agent reasoning chains\. ThroughZ​K​\-MLZK\\text\{\-ML\}, an agent can cryptographically prove to an external verifier that its tool actionata\_\{t\}was generated by a certified, untampered model𝒞\\mathcal\{C\}adhering to developer safety policyΦsafe\\Phi\_\{\\text\{safe\}\}, without disclosing proprietary weights, confidential user data, or internal system prompts\. Table[X](https://arxiv.org/html/2609.13731#S9.T10)summarizes the strategic research roadmap across these six open horizons\.

TABLE X:Strategic Roadmap for Open Research Horizons in Trustworthy Agentic AIResearch HorizonTimeframeFoundational Scientific BottleneckKey Breakthrough Milestone1\. Formal Action Verification1–3 YearsSemantic\-to\-symbolic fidelity gap; state\-space explosion\.Real\-time grammar\-constrained SMT invariant solver\.2\. Lifelong Dynamic Alignment2–4 YearsIn\-context alignment decay; non\-invertible side effects\.Non\-monotonic memory scrubbers with shadow execution\.3\. Autonomic Swarm Immunity3–5 YearsO⁡\(n2\)O\(n^\{2\}\)PBFT token/latency cost in LLM clusters\.Optimistic consensus guided by GNN message telemetry\.4\. Machine\-Speed Cyber Defense2–5 YearsAsymmetric exploit generation vs\. regression\-free patching\.Autonomous SOC agents with formal patch verification\.5\. Pareto Cluster Scheduling1–3 YearsMulti\-LLM inspection latency vs\. throughput\.BRI\-throttled scheduling across GPU/TEE clusters\.6\. Autonomous Value Networks3–6 YearsLack of verifiable agency and private provenance\.StandardizedZ​K​\-MLZK\\text\{\-ML\}reasoning proofs and W3C DIDs\.

## XConclusion

The emergence of Agentic Artificial Intelligence represents a transformative paradigm shift in autonomous computational systems\. By coupling foundation models with recursive cognitive planning, multi\-tiered memory persistence, live tool actuation planes, and distributed multi\-agent collaboration, agentic systems are reshaping scientific discovery, software engineering, enterprise automation, and automated cyber defense\[[1](https://arxiv.org/html/2609.13731#bib.bib1),[2](https://arxiv.org/html/2609.13731#bib.bib2),[3](https://arxiv.org/html/2609.13731#bib.bib3)\]\. However, granting probabilistic neural engines autonomous execution authority across physical and digital environments shatters classical security perimeters\. In agentic architectures, natural language serves simultaneously as the user interface, the internal control code, the data format, and the inter\-agent communication protocol—introducing a Turing\-complete blast radius where any untrusted external input represents uncompiled executable code\[[20](https://arxiv.org/html/2609.13731#bib.bib20),[21](https://arxiv.org/html/2609.13731#bib.bib21),[16](https://arxiv.org/html/2609.13731#bib.bib16)\]\.

This survey has established a comprehensive, systems\-level reference framework forTrustworthy Agentic AI, synthesizing 206 foundational studies, empirical benchmarks, and international regulatory standards across five core Research Questions\. We formalized the general architecture of autonomous agents as a stateful 5\-tuple𝒜=⟨𝒞,ℳ,𝒯,ℰ,Π⟩\\mathcal\{A\}=\\langle\\mathcal\{C\},\\mathcal\{M\},\\mathcal\{T\},\\mathcal\{E\},\\Pi\\ranglecoupled with an adversarial POMDP belief\-divergence model, capturing closed\-loop execution dynamics across cognitive planning \(ReAct\[[4](https://arxiv.org/html/2609.13731#bib.bib4)\], ToT\[[5](https://arxiv.org/html/2609.13731#bib.bib5)\], GoT\[[6](https://arxiv.org/html/2609.13731#bib.bib6)\], Reflexion\[[7](https://arxiv.org/html/2609.13731#bib.bib7)\]\), hierarchical memory subsystems\[[8](https://arxiv.org/html/2609.13731#bib.bib8),[83](https://arxiv.org/html/2609.13731#bib.bib83),[84](https://arxiv.org/html/2609.13731#bib.bib84)\], tool execution runtimes \(including the Model Context Protocol\[[11](https://arxiv.org/html/2609.13731#bib.bib11)\]\), and distributed multi\-agent collaboration topologies\[[12](https://arxiv.org/html/2609.13731#bib.bib12),[13](https://arxiv.org/html/2609.13731#bib.bib13),[26](https://arxiv.org/html/2609.13731#bib.bib26)\]\.

To bridge disparate evaluation criteria, we established the 6\-Dimensional Trustworthiness Taxonomy, reformulating classical static ML evaluations into an operational systems\-level paradigm spanningSecurity, Safety & Operational Robustness, Privacy & Data Protection, Explainability & Verifiability, Fairness & Non\-Discrimination, and Accountability & Provenance\. We conducted a full\-spectrum threat analysis across intra\-execution modules \(perception, brain, and action planes\) and interaction surfaces \(operational environments, persistent RAG vector databases, and multi\-agent swarms\), dissecting critical attack vectors including indirect prompt injection, sleeper backdoors\[[50](https://arxiv.org/html/2609.13731#bib.bib50)\], vector memory poisoning\[[59](https://arxiv.org/html/2609.13731#bib.bib59)\], confused deputy privilege escalation\[[140](https://arxiv.org/html/2609.13731#bib.bib140),[35](https://arxiv.org/html/2609.13731#bib.bib35)\], Morris II generative worms\[[143](https://arxiv.org/html/2609.13731#bib.bib143)\], and Byzantine consensus subversion\[[61](https://arxiv.org/html/2609.13731#bib.bib61),[62](https://arxiv.org/html/2609.13731#bib.bib62)\]\.

To counter these vulnerabilities, we formulated a multi\-layered Zero\-Trust Defense\-in\-Depth blueprint integrating Dual\-LLM Inspector\-Executor isolation\[[158](https://arxiv.org/html/2609.13731#bib.bib158)\], Capability\-Based Access Control \(CapBAC\)\[[18](https://arxiv.org/html/2609.13731#bib.bib18),[36](https://arxiv.org/html/2609.13731#bib.bib36)\], micro\-VM sandboxing\[[54](https://arxiv.org/html/2609.13731#bib.bib54)\], kernel\-level eBPF syscall monitoring\[[56](https://arxiv.org/html/2609.13731#bib.bib56)\], cryptographically signed vector embeddings and Merkle audit trees\[[117](https://arxiv.org/html/2609.13731#bib.bib117)\], and Byzantine fault\-tolerant consensus protocols, accompanied by a comprehensive comparative synthesis of runtime latency, token multiplication, and cluster deployability\. Furthermore, we synthesized interactive benchmark suites \(such as AgentBench\[[181](https://arxiv.org/html/2609.13731#bib.bib181)\], InjecAgent\[[21](https://arxiv.org/html/2609.13731#bib.bib21)\], and CyberSecEval\[[28](https://arxiv.org/html/2609.13731#bib.bib28),[29](https://arxiv.org/html/2609.13731#bib.bib29)\]\), mapped technical defenses to global AI governance mandates \(including the NIST AI RMF\[[119](https://arxiv.org/html/2609.13731#bib.bib119)\], EU AI Act\[[121](https://arxiv.org/html/2609.13731#bib.bib121)\], and ISO/IEC 42001\[[120](https://arxiv.org/html/2609.13731#bib.bib120)\]\), and articulated six foundational open research horizons spanning real\-time neural\-symbolic SMT invariant verification\[[113](https://arxiv.org/html/2609.13731#bib.bib113),[200](https://arxiv.org/html/2609.13731#bib.bib200)\], shadow execution for non\-invertible tool actions, optimistic Byzantine swarm consensus, and zero\-knowledge proofs of agency \(Z​K​\-MLZK\\text\{\-ML\}\)\.

Securing the next generation of autonomous AI agents cannot rely on superficial prompt filtering or heuristic guardrails\. The path forward demands an integrated discipline ofAgentic Systems Security—combining the adaptive cognitive capabilities of foundation models with the mathematical rigor, hardware isolation, cryptographic integrity, and zero\-trust principles of classical systems cybersecurity\.

## References

- \[1\]Z\. Xi, W\. Chen, X\. Guo, W\. He, Y\. Ding, B\. Hong, M\. Zhang, J\. Wang, S\. Jin, E\. Zhou*et al\.*, “The rise and potential of large language model based agents: A survey,”*arXiv preprint arXiv:2309\.07864*, 2023\.
- \[2\]L\. Wang, C\. Ma, X\. Feng, Z\. Zhang, H\. Yang, J\. Zhang, Z\. Chen, J\. Tang, X\. Chen, Y\. Lin*et al\.*, “A survey on large language model based autonomous agents,”*Frontiers of Computer Science*, vol\. 18, no\. 6, p\. 186345, 2024\.
- \[3\]R\. Bommasani, D\. A\. Hudson, E\. Adeli, R\. Altman, S\. Arora, S\. von Arx, M\. S\. Bernstein, J\. Bohg, A\. Bosselut, E\. Brunskill*et al\.*, “On the opportunities and risks of foundation models,”*arXiv preprint arXiv:2108\.07258*, 2021\.
- \[4\]S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao, “ReAct: Synergizing reasoning and acting in language models,”*International Conference on Learning Representations \(ICLR\)*, 2023\.
- \[5\]S\. Yao, D\. Yu, J\. Zhao, I\. Shafran, T\. L\. Griffiths, Y\. Cao, and K\. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models,”*Advances in Neural Information Processing Systems \(NeurIPS\)*, vol\. 36, pp\. 11 809–11 822, 2023\.
- \[6\]M\. Besta, N\. Blach, A\. Kubicek, R\. Gerstenberger, M\. Podstawski, L\. Gianinazzi, J\. Gajda, T\. Lehmann, H\. Niewiadomski, P\. Nyczyk, and T\. Hoefler, “Graph of thoughts: Solving elaborate problems with large language models,”*Proceedings of the AAAI Conference on Artificial Intelligence*, vol\. 38, no\. 16, pp\. 17 682–17 690, 2024\.
- \[7\]N\. Shinn, F\. Cassano, A\. Labash, A\. Gopinath, K\. Narasimhan, and S\. Yao, “Reflexion: Language agents with verbal reinforcement learning,”*Advances in Neural Information Processing Systems \(NeurIPS\)*, vol\. 36, pp\. 8634–8652, 2023\.
- \[8\]C\. Packer, S\. Wooders, K\. Lin, V\. Fang, S\. G\. Patil, I\. Stoica, and J\. E\. Gonzalez, “MemGPT: Towards LLMs as operating systems,”*arXiv preprint arXiv:2310\.08560*, 2023\.
- \[9\]T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom, “Toolformer: Language models can teach themselves to use tools,”*Advances in Neural Information Processing Systems \(NeurIPS\)*, vol\. 36, pp\. 68 539–68 551, 2023\.
- \[10\]S\. G\. Patil, T\. Zhang, X\. Wang, and J\. E\. Gonzalez, “Gorilla: Large language model connected with massive APIs,”*arXiv preprint arXiv:2305\.15334*, 2023\.
- \[11\]Anthropic, “Model context protocol \(MCP\) specification,”[https://modelcontextprotocol\.io](https://modelcontextprotocol.io/), 2024\.
- \[12\]Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, S\. Li, E\. Zhu, C\. Jiang, X\. Zhang, C\. Zhang, J\. Liu, A\. Awadallah, R\. W\. White, D\. Burger, and C\. Wang, “AutoGen: Enabling next\-gen LLM applications via multi\-agent conversation,”*arXiv preprint arXiv:2308\.08155*, 2023\.
- \[13\]S\. Hong, M\. Zheng, J\. Chen, K\. Wang, S\. Wang, Y\. Cheng, B\. Chen, Z\. Zhou, C\. Guan, Y\. Li, X\. Tang, D\. Wang, M\. Yang, and C\. Wu, “MetaGPT: Meta programming for multi\-agent collaborative framework,”*International Conference on Learning Representations \(ICLR\)*, 2024\.
- \[14\]J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. Press, “SWE\-agent: Agent\-computer interfaces enable automated software engineering,”*arXiv preprint arXiv:2405\.15793*, 2024\.
- \[15\]X\. Deng, Y\. Gu, B\. Zheng, S\. Chen, S\. Stevens, B\. Wang, H\. Sun, and Y\. Su, “Mind2web: Towards a generalist agent for the web,”*Advances in Neural Information Processing Systems \(NeurIPS\)*, vol\. 36, pp\. 27 282–27 303, 2023\.
- \[16\]B\. Schneier and N\. Sanders, “The dual\-use security dilemma in autonomous generative AI,”*IEEE Security and Privacy*, vol\. 21, no\. 4, pp\. 88–92, 2023\.
- \[17\]A\. W\. Moore, S\. C\. Smith, and R\. Wright, “Autonomous cyber defense agents: Verification, isolation, and safety,”*Communications of the ACM*, vol\. 67, no\. 5, pp\. 72–83, 2024\.
- \[18\]J\. H\. Saltzer and M\. D\. Schroeder, “The protection of information in computer systems,”*Proceedings of the IEEE*, vol\. 63, no\. 9, pp\. 1278–1308, 1975\.
- \[19\]D\. E\. Denning, “A lattice model of secure information flow,”*Communications of the ACM*, vol\. 19, no\. 5, pp\. 236–243, 1976\.
- \[20\]K\. Greshake, S\. Abdelnabi, S\. Mishra, C\. Endres, T\. Holz, and M\. Fritz, “Not what you’ve signed up for: Compromising real\-world LLM\-integrated applications with indirect prompt injection,” in*Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security \(AISEC\)*, 2023, pp\. 79–90\.
- \[21\]Q\. Zhan, R\. Fang, R\. Bindu, A\. Gupta, Q\. Zhan, P\. Liang, T\. Hashimoto, and A\. Zou, “InjecAgent: Benchmarking indirect prompt injections in tool\-integrated large language model agents,”*arXiv preprint arXiv:2403\.02691*, 2024\.
- \[22\]F\. Perez and I\. Ribeiro, “Ignore previous instructions: An evaluation of prompt injection attacks in language models,”*arXiv preprint arXiv:2211\.09527*, 2022\.
- \[23\]S\. Willison, “Prompt injection attacks against GPT\-3,”[https://simonwillison\.net/2022/Sep/12/prompt\-injection/](https://simonwillison.net/2022/Sep/12/prompt-injection/), 2022\.
- \[24\]S\. Abdelnabi, K\. Greshake, C\. Endres, and M\. Fritz, “Not what you’ve signed up for: Security analysis of ChatGPT plugins and tools,”*ACM Conference on Computer and Communications Security \(CCS\)*, 2023\.
- \[25\]J\. Rehberger, “Pivoting from ChatGPT to internal systems: Exploiting indirect prompt injections and function calling,”*Embrace The Red Cybersecurity Research*, 2023\.
- \[26\]C\. Qian, X\. Cong, C\. Yang, W\. Chen, Y\. Su, J\. Xu, Z\. Liu, and M\. Sun, “Communicative agents for software development,”*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\)*, pp\. 14 894–14 907, 2024\.
- \[27\]R\. Nakano, J\. Hilton, S\. Balaji, J\. Wu, L\. Ouyang, C\. Kim, C\. Hesse, S\. Jain, V\. Kosaraju, W\. Saunders, X\. Jiang, K\. Cobbe, T\. Eloundou, G\. Krueger, K\. Button, M\. Knight, B\. Chess, and J\. Schulman, “WebGPT: Browser\-assisted question\-answering with human feedback,”*arXiv preprint arXiv:2112\.09332*, 2021\.
- \[28\]M\. Bhatt, S\. Chennabasappa, Y\. Li, C\. Nikolaidis, D\. Wan, I\. Evtimov, D\. Gontier, D\. Song*et al\.*, “Purple llama CyberSecEval: A comprehensive benchmark for evaluating the cybersecurity risks of large language models,”*arXiv preprint arXiv:2312\.04724*, 2023\.
- \[29\]M\. Bhatt*et al\.*, “CyberSecEval 2: A wide\-ranging cybersecurity evaluation suite for large language models,”*arXiv preprint arXiv:2404\.13161*, 2024\.
- \[30\]M\. Ahn, A\. Brohan, N\. Brown, Y\. Chebotar, O\. Cortes, B\. David, C\. Finn, K\. Gopalakrishnan, K\. Hausman, A\. Herzog*et al\.*, “Do as i can, not as i say: Grounding language in robotic affordances,” in*Conference on Robot Learning \(CoRL\)*, 2022\.
- \[31\]Y\. Jin*et al\.*, “Surrealdriver: Designing safety\-critical autonomous driving agents with foundation models,”*IEEE Transactions on Intelligent Vehicles*, 2024\.
- \[32\]G\. Wang, Y\. Xie, Y\. Jiang, A\. Mandlekar, C\. Xiao, Y\. Zhu, L\. Fan, and A\. Anandkumar, “Voyager: An open\-ended embodied agent with large language models,”*Transactions on Machine Learning Research \(TMLR\)*, 2024\.
- \[33\]G\. Li, H\. A\. A\. K\. Hammoud, H\. Itani, D\. Khizbullin, and B\. Ghanem, “CAMEL: Communicative agents for ”mind” exploration of large language model society,”*Advances in Neural Information Processing Systems \(NeurIPS\)*, vol\. 36, pp\. 51 991–52 008, 2023\.
- \[34\]W\. Chen, Y\. Su, J\. Zuo, C\. Yang, C\. Yuan, C\.\-M\. Chan, H\. Yu, Y\. Lu, Y\.\-H\. Hung, Y\. Shen, M\. Chen, Y\. Qin, H\. Wang, Y\.\-Q\. Zhang, C\. Yang, Z\. Liu, and M\. Sun, “AgentVerse: Facilitating multi\-agent collaboration and exploring emergent behaviors,”*International Conference on Learning Representations \(ICLR\)*, 2024\.
- \[35\]R\. Fang, R\. Bindu, A\. Zou, and P\. Liang, “Privilege escalation and confused deputy attacks in autonomous agent frameworks,”*arXiv preprint arXiv:2404\.18901*, 2024\.
- \[36\]H\. M\. Levy,*Capability\-Based Computer Systems*\. Digital Press, 1984\.
- \[37\]S\. Rose, O\. Borchert, S\. Mitchell, and S\. Connelly, “Zero trust architecture,” National Institute of Standards and Technology, Tech\. Rep\. NIST SP 800\-207, 2020\.
- \[38\]A\. Zou, Z\. Wang, J\. Z\. Kolter, and M\. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,”*arXiv preprint arXiv:2307\.15043*, 2023\.
- \[39\]F\. Tramer, N\. Carlini, W\. Brendel, and A\. Madry, “On adaptive attacks to adversarial defense evaluations,” in*Advances in Neural Information Processing Systems \(NeurIPS\)*, vol\. 33, 2020, pp\. 1633–1645\.
- \[40\]P\. Chao, A\. Robey, E\. Dobriban, H\. Hassani, G\. J\. Pappas, and E\. Wong, “Jailbreaking black box large language models in twenty queries,”*IEEE Conference on Secure and Trustworthy Machine Learning \(SaTML\)*, 2024\.
- \[41\]A\. Mehrotra, M\. Zampetakis, P\. Kassianik, B\. Nelson, H\. Anderson, and Y\. Singer, “Tree of attacks: Jailbreaking black\-box LLMs automatically,”*arXiv preprint arXiv:2312\.02119*, 2023\.
- \[42\]X\. Liu, N\. Xu, M\. Chen, and C\. Xiao, “AutoDAN: Generating stealthy jailbreak prompts on aligned large language models,”*International Conference on Learning Representations \(ICLR\)*, 2024\.
- \[43\]X\. Li, Z\. Zhou, J\. Zhu, J\. Yao, T\. Liu, and B\. Han, “DeepInception: Hypnotize large language model to be jailbreaker,”*arXiv preprint arXiv:2311\.03191*, 2023\.
- \[44\]Z\.\-X\. Yong, C\. Menghini, and S\. H\. Bach, “Low\-resource languages jailbreak GPT\-4,”*arXiv preprint arXiv:2310\.02446*, 2023\.
- \[45\]Y\. Yuan, W\. Jiao, W\. Wang, J\. tse Huang, P\. He, S\. Shi, and Z\. Tu, “GPT\-4 is too smart to be safe: Stealthy chat with llms via cipher,”*International Conference on Learning Representations \(ICLR\)*, 2024\.
- \[46\]F\. Jiang, Z\. Xu, L\. Niu, Z\. Xiang, B\. Ramasubramanian, B\. Li, and R\. Poovendran, “ArtPrompt: ASCII art\-based jailbreak attacks on aligned LLMs,”*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\)*, pp\. 15 143–15 157, 2024\.
- \[47\]E\. Levi*et al\.*, “Vocabulary attack vectors and direct injection in generative ai agents,”*arXiv preprint arXiv:2402\.11220*, 2024\.
- \[48\]A\. Zhou, K\. Yan, M\. Shlapentokh\-Rothman, H\. Wang, and Y\.\-X\. Wang, “Language agent tree search unifies reasoning, acting, and planning in language models,”*International Conference on Machine Learning \(ICML\)*, 2024\.
- \[49\]Y\. Ji*et al\.*, “Testing and verifying planning consistency in llm\-based multi\-step agents,”*ACM Transactions on Software Engineering and Methodology*, 2024\.
- \[50\]E\. Hubinger, C\. Denison, J\. Mu, M\. Lambert, M\. Tong, M\. MacDiarmid, T\. Lanham, D\. M\. Ziegler, T\. Maxwell, N\. Cheng*et al\.*, “Sleeper agents: Training deceptive LLMs that persist through safety training,”*arXiv preprint arXiv:2401\.05566*, 2024\.
- \[51\]Z\. Xiang, F\. Jiang, Z\. Xiong, B\. Ramasubramanian, R\. Poovendran, and B\. Li, “BadChain: Backdoor chain\-of\-thought prompting for large language models,”*International Conference on Learning Representations \(ICLR\)*, 2024\.
- \[52\]Y\. Zhang, Y\. Li, L\. Cui, C\. Deng, Y\. Liu, R\. Wang, L\. Wang, B\. Zheng, X\. Shi*et al\.*, “Siren’s song in the ai ocean: A survey on hallucination in large language models,”*Computational Linguistics*, 2024\.
- \[53\]N\. Mündler*et al\.*, “Self\-contradictory hallucinations in multi\-step llm reasoning,” in*International Conference on Learning Representations \(ICLR\)*, 2024\.
- \[54\]A\. Agrawal*et al\.*, “Firecracker: Lightweight virtualization for serverless applications,” in*17th USENIX Symposium on Networked Systems Design and Implementation \(NSDI 20\)*, 2020, pp\. 419–434\.
- \[55\]Google LLC, “gVisor: Application kernel for containers,”[https://gvisor\.dev](https://gvisor.dev/), 2018\.
- \[56\]M\. A\. M\. Vieira, M\. S\. Castanho, R\. G\. B\. L\. Silva, L\. S\. da Rosa, and R\. R\. S\. de Oliveira, “Fast packet processing with eBPF and XDP: Security and performance analysis,” in*IEEE Latin\-American Conference on Communications \(LATINCOM\)*, 2020, pp\. 1–6\.
- \[57\]I\. Shumailov, Y\. Zhao, D\. Bates, N\. Papernot, R\. Mullins, and R\. Anderson, “Sponge examples: Energy\-latency attacks on neural networks,” in*IEEE European Symposium on Security and Privacy \(EuroS&P\)*, 2021, pp\. 212–231\.
- \[58\]M\. Guastalla*et al\.*, “Application\-layer denial of service attacks on generative ai endpoints,” in*IEEE Symposium on Security and Privacy \(S&P\)*, 2023\.
- \[59\]W\. Zou, R\. Geng, B\. Wang, and J\. Jia, “PoisonedRAG: Knowledge poisoning attacks to retrieval\-augmented generation of large language models,”*USENIX Security Symposium*, 2025\.
- \[60\]P\. Chen, J\. Liu, Z\. Gao, X\. Jia, and C\. Zhou, “Shadowcast: Stealthy poisoning attacks on retrieval\-augmented generation systems,”*ACM Conference on Computer and Communications Security \(CCS\)*, 2024\.
- \[61\]M\. Castro and B\. Liskov, “Practical byzantine fault tolerance and proactive recovery,”*ACM Transactions on Computer Systems \(TOCS\)*, vol\. 20, no\. 4, pp\. 398–461, 2002\.
- \[62\]M\. Li, X\. Chen, and J\. Weng, “Byzantine fault tolerance and robust consensus in decentralized multi\-agent AI networks,”*IEEE Transactions on Information Forensics and Security*, vol\. 19, pp\. 4521–4535, 2024\.
- \[63\]Y\. Gu, X\. Deng, H\. Sun, and Y\. Su, “Cascading failure and error propagation in multi\-agent collaborative reasoning,”*International Conference on Machine Learning \(ICML\)*, 2024\.
- \[64\]Z\. Zhang*et al\.*, “Agentpoison: Red\-teaming llm agents via targeted knowledge base poisoning,”*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024\.
- \[65\]M\. J\. Page, J\. E\. McKenzie, P\. M\. Bossuyt, I\. Boutron, T\. C\. Hoffmann, C\. D\. Mulrow, L\. Shamseer, J\. M\. Tetzlaff, E\. A\. Akl, S\. E\. Brennan*et al\.*, “The prisma 2020 statement: an updated guideline for reporting systematic reviews,”*The BMJ*, vol\. 372, p\. n71, 2021\.
- \[66\]I\. J\. Goodfellow, J\. Shlens, and C\. Szegedy, “Explaining and harnessing adversarial examples,”*International Conference on Learning Representations \(ICLR\)*, 2015\.
- \[67\]A\. Madry, A\. Makelov, L\. Schmidt, D\. Tsipras, and A\. Vladu, “Towards deep learning models resistant to adversarial attacks,” in*International Conference on Learning Representations \(ICLR\)*, 2018\.
- \[68\]N\. Papernot, P\. McDaniel, S\. Jha, M\. Fredrikson, Z\. B\. Celik, and A\. Swami, “The limitations of deep learning in adversarial settings,” in*IEEE European Symposium on Security and Privacy \(EuroS&P\)*, 2016, pp\. 372–387\.
- \[69\]A\. Wei, N\. Haghtalab, and J\. Steinhardt, “Jailbroken: How does LLM safety training fail?”*Advances in Neural Information Processing Systems \(NeurIPS\)*, vol\. 36, pp\. 80 079–80 110, 2023\.
- \[70\]J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat*et al\.*, “GPT\-4 technical report,”*arXiv preprint arXiv:2303\.08774*, 2023\.
- \[71\]Gemini Team, Google, “Gemini: A family of highly capable multimodal models,”*arXiv preprint arXiv:2312\.11805*, 2023\.
- \[72\]A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan*et al\.*, “The Llama 3 herd of models,”*arXiv preprint arXiv:2407\.21783*, 2024\.
- \[73\]A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\.\-A\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. Sayed, “Mistral 7b,”*arXiv preprint arXiv:2310\.06825*, 2023\.
- \[74\]Anthropic, “The Claude 3 model family: Opus, sonnet, haiku,”*Anthropic Technical Report*, 2024\.
- \[75\]O\. Khattab, A\. Singhvi, P\. Maheshwari, Z\. Zhang, K\. Santhanam, S\. Vardhamanan, S\. Haq, A\. Sharma, T\. T\. Joshi, H\. Hajishirzi, C\. Potts, and M\. Zaharia, “DSPy: Compiling declarative language model calls into state\-of\-the\-art pipelines,”*arXiv preprint arXiv:2310\.03714*, 2023\.
- \[76\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, and D\. Zhou, “Chain\-of\-thought prompting elicits reasoning in large language models,”*Advances in Neural Information Processing Systems \(NeurIPS\)*, vol\. 35, pp\. 24 824–24 837, 2022\.
- \[77\]T\. Kojima, S\. S\. Gu, M\. Reid, Y\. Matsuo, and Y\. Iwasawa, “Large language models are zero\-shot reasoners,”*Advances in Neural Information Processing Systems \(NeurIPS\)*, vol\. 35, pp\. 22 199–22 213, 2022\.
- \[78\]L\. Wang, W\. Xu, Y\. Lan, Z\. Hu, Y\. Lan, R\. K\.\-W\. Lee, and E\.\-P\. Lim, “Plan\-and\-solve prompting: Improving zero\-shot chain\-of\-thought reasoning by large language models,”*Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(ACL\)*, pp\. 2609–2634, 2023\.
- \[79\]A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang, S\. Welleck, B\. P\. Majumder, S\. Gupta, A\. Yazdanbakhsh, and P\. Clark, “Self\-refine: Iterative refinement with self\-feedback,”*Advances in Neural Information Processing Systems \(NeurIPS\)*, vol\. 36, pp\. 46 534–46 594, 2023\.
- \[80\]Z\. Wang*et al\.*, “Describe, explain, plan and select: Interactive planning with large language models enabling long\-horizon embodied tasks,”*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.
- \[81\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin, “Attention is all you need,”*Advances in Neural Information Processing Systems \(NeurIPS\)*, vol\. 30, pp\. 5998–6008, 2017\.
- \[82\]Y\. Ding*et al\.*, “Longrope: Extending llm context window beyond 2 million tokens,”*arXiv preprint arXiv:2402\.13753*, 2024\.
- \[83\]Y\. A\. Malkov and D\. A\. Yashunin, “Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs,”*IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol\. 42, no\. 4, pp\. 824–836, 2018\.
- \[84\]J\. Johnson, M\. Douze, and H\. J’egou, “Billion\-scale similarity search with GPUs,”*IEEE Transactions on Big Data*, vol\. 7, no\. 3, pp\. 535–547, 2019\.
- \[85\]S\. Pan, L\. Luo, Y\. Wang, C\. Chen, J\. Wang, and X\. Wu, “Unifying large language models and knowledge graphs: A roadmap,”*IEEE Transactions on Knowledge and Data Engineering*, 2024\.
- \[86\]Y\. Shen, K\. Song, X\. Tan, D\. Li, W\. Lu, and Y\. Zhuang, “HuggingGPT: Solving AI tasks with ChatGPT and its friends in Hugging Face,”*Advances in Neural Information Processing Systems \(NeurIPS\)*, vol\. 36, pp\. 38 154–38 180, 2023\.
- \[87\]Y\. Zhang, G\. Deng, T\. Zhang, and Y\. Liu, “Demystifying the vulnerability supply chain in LLM frameworks and tool ecosystems,”*IEEE Symposium on Security and Privacy \(S&P\)*, 2025\.
- \[88\]Z\. Liu, Y\. Hu, R\.\-Z\. Fan, H\.\-B\. Jiang, M\.\-Z\. Li, and Z\.\-M\. Chen, “Dynamic LLM\-agent network: An llm\-agent collaboration framework with dynamic architecture,”*International Conference on Learning Representations \(ICLR\)*, 2024\.
- \[89\]J\. S\. Park, J\. C\. O’Brien, C\. J\. Cai, M\. R\. Morris, P\. Liang, and M\. S\. Bernstein, “Generative agents: Interactive simulacra of human behavior,”*Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology \(UIST\)*, pp\. 1–22, 2023\.
- \[90\]N\. Nisan, T\. Roughgarden, E\. Tardos, and V\. V\. Vazirani,*Algorithmic Game Theory*\. Cambridge University Press, 2007\.
- \[91\]T\. Sandholm, “Algorithm for optimal winner determination in combinatorial auctions,”*Artificial Intelligence*, vol\. 135, no\. 1\-2, pp\. 1–54, 2002\.
- \[92\]C\.\-M\. Chan, W\. Chen, Y\. Su, J\. Yu, W\. Xue, S\. Zhang, J\. Fu, and Z\. Liu, “Chateval: Towards better llm\-based evaluators through multi\-agent debate,”*International Conference on Learning Representations \(ICLR\)*, 2024\.
- \[93\]N\. Carlini and D\. Wagner, “Towards evaluating the robustness of neural networks,” in*IEEE Symposium on Security and Privacy \(S&P\)*, 2017, pp\. 39–57\.
- \[94\]D\. Amodei, C\. Olah, J\. Steinhardt, P\. Christiano, J\. Schulman, and D\. Hendrycks, “Concrete problems in AI safety,”*arXiv preprint arXiv:1606\.06565*, 2016\.
- \[95\]D\. Hendrycks, N\. Carlini, J\. Schulman, and J\. Steinhardt, “Unsolved problems in ML safety,”*arXiv preprint arXiv:2109\.13916*, 2021\.
- \[96\]H\. Zhang*et al\.*, “Effective system prompt extraction in multimodal autonomous agents,”*arXiv preprint arXiv:2402\.04300*, 2024\.
- \[97\]J\. Geiping, D\. Bau, M\. Goldblum, and T\. Goldstein, “Coercing llms into targeted behavior via in\-context context bombing,” in*International Conference on Machine Learning \(ICML\)*, 2024\.
- \[98\]S\. Agarwal*et al\.*, “Investigating system prompt leaking in autonomous large language model agents,”*arXiv preprint arXiv:2403\.09871*, 2024\.
- \[99\]A\. Pan, C\. J\. Shern, A\. Zou, N\. Li, S\. Basart, T\. Woodside, H\. Zhang, S\. Emmons, and D\. Hendrycks, “Do the right thing: An agent benchmark for LLM morality, harm, and power seeking,”*International Conference on Machine Learning \(ICML\)*, pp\. 26 801–26 823, 2023\.
- \[100\]T\. Yuan, Z\. He, L\. Dong, Y\. Xiao, J\. Shen, R\. Shen, Y\. Liu, and Z\. Jin, “R\-Judge: Benchmarking safety risk awareness for LLM agents,”*International Conference on Learning Representations \(ICLR\)*, 2024\.
- \[101\]J\. Chen, Z\. Chen, T\. Chen*et al\.*, “SecTool: Security benchmarking of tool\-integrated autonomous agents,”*ACM Conference on Computer and Communications Security \(CCS\)*, 2024\.
- \[102\]F\. Croce and M\. Hein, “Reliable evaluation of adversarial robustness with an ensemble of diverse parameter\-free attacks,” in*International Conference on Machine Learning \(ICML\)*, 2020, pp\. 2206–2216\.
- \[103\]R\. Shokri, M\. Stronati, C\. Song, and V\. Shmatikov, “Membership inference attacks against machine learning models,” in*IEEE Symposium on Security and Privacy \(S&P\)*, 2017, pp\. 3–18\.
- \[104\]M\. Fredrikson, S\. Jha, and T\. Ristenpart, “Model inversion attacks that exploit confidence information and basic countermeasures,” in*Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security \(CCS\)*, 2015, pp\. 1322–1333\.
- \[105\]Y\. Zhang*et al\.*, “Privacy preservation and differential privacy in llm multi\-agent workflows,”*IEEE Security & Privacy*, 2024\.
- \[106\]N\. Carlini, F\. Tramer, E\. Wallace, M\. Jagielski, A\. Herbert\-Voss, K\. Lee, A\. Roberts, T\. Brown, D\. Song, U\. Erlingsson, A\. Oprea, and C\. Raffel, “Extracting training data from large language models,”*USENIX Security Symposium*, pp\. 2633–2650, 2021\.
- \[107\]R\. Geambasu, T\. Kohno, A\. A\. Levy, and H\. M\. Levy, “Vanish: Increasing data privacy with self\-destructing data,” in*USENIX Security Symposium*, 2009, pp\. 299–316\.
- \[108\]C\. Song and A\. Raghunathan, “Information leakage in embedding representations,”*arXiv preprint arXiv:2004\.00053*, 2020\.
- \[109\]Z\. Li*et al\.*, “Sentence embedding inversion: Reconstructing sensitive data from dense embeddings,”*IEEE Transactions on Information Forensics and Security*, 2023\.
- \[110\]J\. X\. Morris, V\. Kuleshov, V\. Shmatikov, and A\. M\. Rush, “Text embeddings reveal \(almost\) as much as text,” in*Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2023\.
- \[111\]M\. Leucker and C\. Schallhart, “A brief account of runtime verification,”*The Journal of Logic and Algebraic Programming*, vol\. 78, no\. 5, pp\. 293–303, 2009\.
- \[112\]E\. M\. Clarke, O\. Grumberg, and D\. A\. Peled,*Model Checking*\. MIT Press, 1999\.
- \[113\]L\. D\. Moura and N\. Bjørner, “Z3: An efficient SMT solver,” in*International Conference on Tools and Algorithms for the Construction and Analysis of Systems \(TACAS\)*, 2008, pp\. 337–340\.
- \[114\]H\. Barbosa, C\. Barrett, M\. Brain, G\. Kremer, H\. Lachnitt, M\. Mann, A\. Mohamed, M\. Mohamed, A\. Niemetz, A\. Nötzli, A\. Ozdemir, M\. Preiner, A\. Reynolds, Y\. Sheng, C\. Tinelli, and Y\. Zohar, “cvc5: A versatile and industrial\-strength SMT solver,” in*International Conference on Computer Aided Verification \(CAV\)*, 2022, pp\. 415–442\.
- \[115\]I\. O\. Gallegos, R\. A\. Rossi, J\. Barrow, M\. M\. Tanjim, S\. Kim, F\. Dernoncourt, T\. Yu, R\. Zhang, and N\. K\. Ahmed, “Bias and fairness in large language models: A survey,”*ACM Computing Surveys*, 2023\.
- \[116\]S\. Bordia and S\. R\. Bowman, “Identifying and reducing gender bias in word\-level language models,” in*Proceedings of the NAACL Student Research Workshop*, 2019, pp\. 7–15\.
- \[117\]R\. C\. Merkle, “A digital signature based on a conventional encryption function,”*Advances in Cryptology — CRYPTO ’87*, pp\. 369–378, 1988\.
- \[118\]World Wide Web Consortium \(W3C\), “Decentralized identifiers \(DIDs\) v1\.0: Core architecture, data model, and representations,”[https://www\.w3\.org/TR/did\-core/](https://www.w3.org/TR/did-core/), 2022\.
- \[119\]National Institute of Standards and Technology, “Artificial intelligence risk management framework \(AI RMF 1\.0\),” U\.S\. Department of Commerce, Tech\. Rep\. NIST SP 1270, 2023\.
- \[120\]International Organization for Standardization, “ISO/IEC 42001:2023 information technology — artificial intelligence — management system,” ISO/IEC, Tech\. Rep\., 2023\.
- \[121\]European Parliament and Council of the European Union, “Regulation of the european parliament and of the council laying down harmonised rules on artificial intelligence \(Artificial Intelligence Act\),” Official Journal of the European Union, Tech\. Rep\., 2024\.
- \[122\]Y\. Liu, G\. Deng, Y\. Li, K\. Wang, T\. Zhang, Y\. Liu, H\. Wang, Y\. Zheng, and Y\. Liu, “Prompt injection attack against LLM\-integrated applications,”*arXiv preprint arXiv:2306\.05499*, 2023\.
- \[123\]H\. J\. Branch, J\. R\. Felix, V\. M\. Igbe, and J\. J\. T\. E\. van der Sloot, “Evaluating the susceptibility of large language models to prompt injection attacks,”*arXiv preprint arXiv:2212\.05286*, 2022\.
- \[124\]J\. Selvi and K\. Fu, “Exploring vulnerabilities in autonomous AI agent workflows,”*ACM Computer Communication Review*, 2023\.
- \[125\]T\. Wu*et al\.*, “Adversarial visual prompt injections against vision\-language agents,”*IEEE Transactions on Information Forensics and Security*, 2024\.
- \[126\]E\. Bagdasaryan and V\. Shmatikov, “Abusing images and sounds for indirect instruction injection in multi\-modal LLMs,”*arXiv preprint arXiv:2307\.10490*, 2023\.
- \[127\]J\. Cui*et al\.*, “Safe agent: Safety alignment and mitigation for vision\-language robotic agents,”*arXiv preprint arXiv:2404\.05670*, 2024\.
- \[128\]L\. Struppek, D\. Hintersdorf, and K\. Kersting, “Rickrolling the artist: Injecting invisible backdoors into text\-to\-image models,” in*IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2023\.
- \[129\]A\. Wan, E\. Wallace, S\. Shen, and D\. Klein, “Poisoning language models during instruction tuning,” in*International Conference on Machine Learning \(ICML\)*, 2023\.
- \[130\]J\. Xu*et al\.*, “Instructions as triggers: Backdoor attacks in multi\-turn agent conversations,”*arXiv preprint arXiv:2403\.11900*, 2024\.
- \[131\]J\. Rando and F\. Tramèr, “Universal jailbreak backdoors from poisoned human feedback,”*International Conference on Machine Learning \(ICML\)*, 2024\.
- \[132\]V\. Dibia, “Aligning and grounding generative ai agents for real\-world workflows,”*arXiv preprint arXiv:2308\.08110*, 2023\.
- \[133\]S\. Longpre*et al\.*, “Entity\-based knowledge conflicts in question answering models,”*Association for Computational Linguistics \(ACL\)*, 2021\.
- \[134\]Z\. Ji, N\. Lee, R\. Frieske, T\. Yu, D\. Su, Y\. Xu, E\. Ishii, Y\. J\. Bang, W\. Dai, A\. Madotto, and P\. Fung, “Survey of hallucination in natural language generation,”*ACM Computing Surveys*, vol\. 55, no\. 12, pp\. 1–38, 2023\.
- \[135\]N\. Lee*et al\.*, “Factuality enhanced language models for open\-ended text generation,” in*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2022\.
- \[136\]Z\. Wang, P\. He, Y\. Yao, R\. Zheng, S\. Wang, H\. Chen, and M\. Huang, “SafeTool: Benchmarking and defending against malicious tool invocations in LLM agents,”*arXiv preprint arXiv:2402\.14880*, 2024\.
- \[137\]J\. Rando and F\. Tramer, “Universal jailbreak backdoors in code execution environments for LLM agents,”*USENIX Security Symposium*, 2024\.
- \[138\]Y\. Liu*et al\.*, “Demystifying rce and parameter injections in tool\-assisted llms,” in*Proceedings of the USENIX Security Symposium*, 2023\.
- \[139\]F\. Pedro, N\. Antunes, and M\. Vieira, “From prompts to shells: Breaking LLM agents via indirect injections,”*IEEE Security and Privacy Workshops*, 2024\.
- \[140\]N\. Hardy, “The confused deputy: \(or why capabilities might have been invented\),”*ACM SIGOPS Operating Systems Review*, vol\. 22, no\. 4, pp\. 36–38, 1988\.
- \[141\]L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. F\. Christiano, J\. Leike, and R\. Lowe, “Training language models to follow instructions with human feedback,”*Advances in Neural Information Processing Systems \(NeurIPS\)*, vol\. 35, pp\. 27 730–27 744, 2022\.
- \[142\]J\. Wei*et al\.*, “Simple synthetic data reduces sycophancy in large language models,”*arXiv preprint arXiv:2308\.03958*, 2023\.
- \[143\]S\. Cohen, R\. Bitton, and B\. Nassi, “Here comes the ai worm: Unleashing zero\-click worms that target genai\-powered applications,”*arXiv preprint arXiv:2403\.02817*, 2024\.
- \[144\]J\. Yi, Y\. Xie, F\. Zhu, J\. Lian, X\. Wang, J\. Yao, and X\. Xie, “Benchmarking and defending against indirect prompt injection on LLMs via BIPIA,”*arXiv preprint arXiv:2312\.14197*, 2024\.
- \[145\]X\. Wu*et al\.*, “Wipi: A benchmark for web\-based indirect prompt injections in autonomous agents,”*arXiv preprint arXiv:2404\.18900*, 2024\.
- \[146\]J\. den Hartog*et al\.*, “Security and safety analysis of autonomous robotic systems,” in*IEEE International Conference on Robotics and Automation \(ICRA\)*, 2018\.
- \[147\]M\. Nasr, N\. Carlini, J\. Hayase, M\. Jagielski, A\. F\. Cooper, D\. Ippolito, C\. A\. Choquette\-Choo, E\. Wallace, F\. Tramer, and K\. Lee, “Scalable extraction of training data from \(production\) language models,”*arXiv preprint arXiv:2311\.17035*, 2023\.
- \[148\]S\. Zeng*et al\.*, “Good practices in llm agent context management and multi\-tenant security,”*arXiv preprint arXiv:2404\.11200*, 2024\.
- \[149\]B\. Zhang*et al\.*, “A survey on distributed memory consistency in autonomous agent networks,”*ACM Computing Surveys*, 2024\.
- \[150\]H\. Chen*et al\.*, “Scalable state synchronization for multi\-agent large language model systems,”*arXiv preprint arXiv:2309\.04300*, 2023\.
- \[151\]L\. Pan*et al\.*, “On the cascading failures of llm\-based multi\-agent systems,”*arXiv preprint arXiv:2310\.05200*, 2023\.
- \[152\]X\. Gu, X\. Zheng, T\. Pang, C\. Du, Q\. Liu, Y\. Wang, and M\. Lin, “Agent smith: A single image can jailbreak one million multimodal llm agents exponentially fast,”*arXiv preprint arXiv:2402\.08567*, 2024\.
- \[153\]S\. Motwani and Y\. Brun, “Perfect is the enemy of good: Evaluating the fault tolerance of distributed multi\-agent systems,” in*Proceedings of the ACM SIGSOFT International Symposium on the Foundations of Software Engineering \(FSE\)*, 2023\.
- \[154\]J\. R\. Douceur, “The Sybil attack,” in*International Workshop on Peer\-to\-Peer Systems \(IPTPS\)*, 2002, pp\. 251–260\.
- \[155\]Meta Fundamental AI Research Diplomacy Team \(FAIR\), A\. Bakhtin, N\. Brown, D\. Emily*et al\.*, “Human\-level play in the game of diplomacy by combining language models with strategic reasoning,”*Science*, vol\. 378, no\. 6624, pp\. 1067–1074, 2022\.
- \[156\]A\. O’Gara, “Hoodwinked: Deception and cooperation in multi\-agent language models,”*arXiv preprint arXiv:2308\.01404*, 2023\.
- \[157\]P\. S\. Park, S\. Goldstein, A\. O’Gara, M\. Chen, and D\. Hendrycks, “Ai deception: A survey of examples, risks, and potential solutions,”*Patterns*, vol\. 5, no\. 5, 2024\.
- \[158\]F\. Perez and I\. Ribeiro, “Dual\-LLM architectures for secure task execution in untrusted contexts,”*arXiv preprint arXiv:2212\.01234*, 2022\.
- \[159\]E\. Wallace, S\. Feng, N\. Kandpal, M\. Gardner, and S\. Singh, “Instruction\-data segregation: Structural delimiters for secure prompt ingestion in language models,”*International Conference on Learning Representations \(ICLR\)*, 2024\.
- \[160\]T\. Rebedea, R\. Dinu, C\. Svorencik, and V\. Chiorean, “NeMo Guardrails: A toolkit for controllable and safe LLM applications,”*arXiv preprint arXiv:2310\.10501*, 2023\.
- \[161\]C\. Dwork, “Differential privacy,”*International Colloquium on Automata, Languages, and Programming \(ICALP\)*, pp\. 1–12, 2006\.
- \[162\]E\. Rescorla, “The transport layer security \(TLS\) protocol version 1\.3,” IETF, Tech\. Rep\. RFC 8446, 2018\.
- \[163\]D\. Kim, S\. Lee, J\. Park, and S\. won Hwang, “Tokenizer\-level isolation masks for defending against prompt injection attacks,”*ACM Conference on Computer and Communications Security \(CCS\)*, 2024\.
- \[164\]H\. Inan, K\. Upasani, J\. Chi, R\. Rungta, K\. Iyer, Y\. Mao, M\. Tontchev, Q\. Zheng, B\. Fuller, P\. Yu*et al\.*, “Llama Guard: LLM\-based input\-output safeguard for human\-ai conversations,”*arXiv preprint arXiv:2312\.06674*, 2023\.
- \[165\]A\. Robey, E\. Wong, H\. Hassani, and G\. J\. Pappas, “SmoothLLM: Defending large language models against jailbreaking attacks,”*arXiv preprint arXiv:2310\.03684*, 2023\.
- \[166\]M\. Phute, A\. Helbling, M\. Hull, S\.\-Y\. Wang, S\. Lee, C\. Agarwal, and D\. H\. Chau, “Llm self\-defense: By self\-examination, llms can defend themselves against jailbreaks,”*arXiv preprint arXiv:2308\.07308*, 2023\.
- \[167\]Y\. Bai, S\. Kadavath, S\. Kundu, A\. Askell, J\. Kernion, A\. Jones, A\. Chen, A\. Goldie, A\. Mirhoseini, C\. McKinnon*et al\.*, “Constitutional AI: Harmlessness from AI feedback,”*arXiv preprint arXiv:2212\.08073*, 2022\.
- \[168\]H\. Lee, S\. Phatale, H\. Mansoor, T\. Mesnard, J\. Ferret, K\. Lu, C\. Bishop, E\. Hall, V\. Carbune, A\. Rastogi, and S\. Prakash, “RLAIF: Scaling reinforcement learning from human feedback with AI feedback,”*arXiv preprint arXiv:2309\.00267*, 2023\.
- \[169\]A\. Kumar, C\. Agarwal, S\. Srinivas, S\. Feizi, and H\. Lakkaraju, “Certifying LLM safety against adversarial prompt injections,”*arXiv preprint arXiv:2309\.02705*, 2023\.
- \[170\]J\. Camenisch and A\. Lysyanskaya, “An efficient system for non\-transferable anonymous credentials with optional anonymity revocation,” in*Advances in Cryptology — EUROCRYPT 2001*, 2001, pp\. 93–118\.
- \[171\]M\. Abadi, A\. Chu, I\. Goodfellow, H\. B\. McMahan, I\. Mironov, K\. Talwar, and L\. Zhang, “Deep learning with differential privacy,” in*Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security \(CCS\)*, 2016, pp\. 308–318\.
- \[172\]A\. Haas, A\. Rossberg, D\. L\. Schuff, B\. L\. Titzer, M\. Holman, D\. Gohman, L\. Wagner, A\. Zakai, and J\. Bastien, “Bringing the web up to speed with WebAssembly,” in*Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation \(PLDI\)*, 2017, pp\. 185–200\.
- \[173\]S\. McCanne and V\. Jacobson, “The BSD packet filter: A new architecture for user\-level packet capture,”*USENIX Winter Conference*, pp\. 259–270, 1993\.
- \[174\]World Wide Web Consortium \(W3C\), “Verifiable credentials data model 1\.0,”[https://www\.w3\.org/TR/vc\-data\-model/](https://www.w3.org/TR/vc-data-model/), 2019\.
- \[175\]A\. Shamir, “How to share a secret,”*Communications of the ACM*, vol\. 22, no\. 11, pp\. 612–613, 1979\.
- \[176\]D\. Boneh, B\. Lynn, and H\. Shacham, “Short signatures from the Weil pairing,” in*Advances in Cryptology — ASIACRYPT 2001*, 2001, pp\. 514–532\.
- \[177\]D\. Ongaro and J\. Ousterhout, “In search of an understandable consensus algorithm \(Raft\),” in*USENIX Annual Technical Conference \(ATC\)*, 2014, pp\. 305–319\.
- \[178\]V\. Costan and S\. Devadas, “Intel SGX explained,”*IACR Cryptology ePrint Archive*, vol\. 2016, p\. 86, 2016\.
- \[179\]D\. Kaplan, J\. Powell, and T\. Woller, “AMD memory encryption,” in*AMD Whitepaper*, 2016\.
- \[180\]B\. Ngabonziza, D\. Martin, A\. Bailey, H\. Cho, and S\. Martin, “Trustzone explained: Architectural features and limitations,” in*IEEE 2nd International Conference on Collaboration and Internet Computing \(CIC\)*, 2016, pp\. 475–482\.
- \[181\]X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang*et al\.*, “AgentBench: Evaluating LLMs as agents,”*International Conference on Learning Representations \(ICLR\)*, 2024\.
- \[182\]Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, P\.\-H\. Su, D\. Tao, Z\. Liu, and M\. Sun, “ToolLLM: Facilitating large language models to master 16000\+ real\-world APIs,”*International Conference on Learning Representations \(ICLR\)*, 2024\.
- \[183\]Y\. Zhuang, Y\. Yu, K\. Wang, H\. Sun, and C\. Zhang, “ToolQA: A dataset for LLM question answering with external tools,”*Advances in Neural Information Processing Systems \(NeurIPS\)*, vol\. 36, pp\. 40 656–40 679, 2023\.
- \[184\]G\. Mialon, C\. Fourrier, C\. Swift, T\. Wolf, Y\. LeCun, and T\. Scialom, “GAIA: a benchmark for general AI assistants,”*International Conference on Learning Representations \(ICLR\)*, 2024\.
- \[185\]M\. Andriushchenko, F\. Croce, and N\. Flammarion, “AgentHarm: A benchmark for measuring harmfulness in autonomous AI agents,”*arXiv preprint arXiv:2410\.09024*, 2024\.
- \[186\]D\. Ganguli, L\. Lovitt, J\. Kernion, A\. Askell, Y\. Bai, S\. Kadavath, B\. Mann, E\. Perez, N\. Schiefer, K\. Ndousse*et al\.*, “Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned,”*arXiv preprint arXiv:2209\.07858*, 2022\.
- \[187\]S\. Casper, X\. Davies, C\. Shi, T\. K\. Gilbert, J\. Scheurer, J\. Rando, R\. Freedman, T\. Korbak, D\. Lindner, P\. Freire*et al\.*, “Explore, establish, exploit: Red teaming language models from scratch,”*arXiv preprint arXiv:2306\.09442*, 2023\.
- \[188\]E\. Perez, S\. Huang, F\. Song, T\. Cai, R\. Ring, J\. Aslanides, A\. Glaese, N\. McAleese, and G\. Irving, “Red teaming language models with language models,”*Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, pp\. 3419–3448, 2022\.
- \[189\]F\. Xu, Z\. Liu, and Y\. Chen, “Redagent: Dynamic red\-teaming for llm agent architectures,”*arXiv preprint arXiv:2407\.12039*, 2024\.
- \[190\]Y\. Yao, H\. Shen, and W\. Zhang, “Fuzzllm: Automated schema mutation fuzzing for model context protocols,” in*Proceedings of the USENIX Security Symposium*, 2024\.
- \[191\]OWASP Foundation, “OWASP Top 10 for large language model applications and autonomous systems,” OWASP, Tech\. Rep\., 2023\.
- \[192\]——, “OWASP top 10 for autonomous AI agents and multi\-agent systems,” OWASP, Tech\. Rep\., 2025\.
- \[193\]National Institute of Standards and Technology, “Artificial intelligence risk management framework: Generative AI profile,” U\.S\. Department of Commerce, Tech\. Rep\. NIST SP 1270\-1, 2024\.
- \[194\]International Organization for Standardization, “ISO/IEC 27001:2022 information security, cybersecurity and privacy protection — information security management systems,” ISO/IEC, Tech\. Rep\., 2022\.
- \[195\]IEEE Standards Association, “IEEE P3119: Standard for artificial intelligence transparency and autonomous agent governance,” IEEE, Tech\. Rep\., 2023\.
- \[196\]——, “Ieee standard for technical framework and requirements of shared machine learning for secure multiparty computation,”*IEEE Std 2830\-2021*, 2021\.
- \[197\]S\. Russell, D\. Dewey, and M\. Tegmark, “Research priorities for robust and beneficial artificial intelligence,”*AI Magazine*, vol\. 36, no\. 4, pp\. 105–114, 2015\.
- \[198\]C\. Baier and J\.\-P\. Katoen,*Principles of Model Checking*\. MIT Press, 2008\.
- \[199\]G\. Katz, C\. Barrett, D\. L\. Dill, K\. Julian, and M\. J\. Kochenderfer, “Reluplex: An efficient SMT solver for verifying deep neural networks,” in*International Conference on Computer Aided Verification \(CAV\)*, 2017, pp\. 97–117\.
- \[200\]T\. Gehr, M\. Mirman, D\. Drachsler\-Cohen, P\. Tsankov, S\. Chaudhuri, and M\. Vechev, “AI2: Safety and robustness certification of deep neural networks with abstract interpretation,” in*IEEE Symposium on Security and Privacy \(S&P\)*, 2018, pp\. 3–18\.
- \[201\]G\. Singh, T\. Gehr, M\. Püschel, and M\. Vechev, “An abstract domain for certifying neural networks,” in*Proceedings of the ACM on Programming Languages \(POPL\)*, vol\. 3, 2019, pp\. 1–30\.
- \[202\]R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn, “Direct preference optimization: Your language model is secretly a reward model,”*Advances in Neural Information Processing Systems \(NeurIPS\)*, vol\. 36, pp\. 53 728–53 741, 2023\.
- \[203\]J\. Kirkpatrick, R\. Pascanu, N\. Rabinowitz, J\. Veness, G\. Desjardins, A\. A\. Rusu, K\. Milan, J\. Quan, T\. Ramalho, A\. Grabska\-Barwinska*et al\.*, “Overcoming catastrophic forgetting in neural networks,”*Proceedings of the National Academy of Sciences \(PNAS\)*, vol\. 114, no\. 13, pp\. 3521–3526, 2017\.
- \[204\]T\. N\. Kipf and M\. Welling, “Semi\-supervised classification with graph convolutional networks,” in*International Conference on Learning Representations \(ICLR\)*, 2017\.
- \[205\]P\. Veličković, G\. Cucurull, A\. Casanova, A\. Romero, P\. Liò, and Y\. Bengio, “Graph attention networks,” in*International Conference on Learning Representations \(ICLR\)*, 2018\.
- \[206\]X\. Lin, Y\. Zhang, and H\. Wang, “Blockchain\-based decentralized execution environments for trustworthy autonomous ai agents,”*IEEE Transactions on Dependable and Secure Computing*, 2024\.