AI Agent Security - MIT 6.566 Computer Systems Security, Spring 2026

TL;DR: This lecture introduces the security challenges of AI agents, including non-adversarial errors (such as accidentally deleting a database) and adversarial attacks (such as prompt injection, data leakage). It also covers the foundational system building from language models to conversational agents. ## Defining AI Agents An AI agent is an AI system that can perceive its environment, make decisions, and take autonomous actions to achieve user-defined goals. At the system level, the user interacts with the agent, which runs with high privileges (typically environment-level privileges) and interacts with the environment (including local files, the internet, etc.). Parts of the environment the agent accesses may be untrusted, such as external web pages. ## Non-Robustness of Agents: Non-Adversarial Errors Even without attacks, agents may make errors due to the probabilistic nature of the underlying large language model. For example, in recent news: the founder of Pocket OS used Cursor and Claude Opus 4.6, and the agent accidentally deleted the entire production database and all its backups. ## Security Attack Examples ### Data Leakage and Prompt Injection Systems like ChatGPT are vulnerable to prompt injection attacks: an attacker controls a web page, induces the agent to retrieve that page, and thereby leaks the user's private data. ### ICML Reviewer Detection ICML (a machine learning conference) to detect whether reviewers used LLMs to write reviews, embedded secret phrases in each paper's PDF that are invisible to humans but visible to LLMs. If a reviewer used an LLM for review, the model would read these phrases and write them into the review, thus being caught by the conference organizers. About 500 papers were rejected because of this. ## Language Model Basics Large language models are essentially probabilistic next-token prediction models: given a prefix string, they output a probability distribution over the next character. Training data typically comes from internet text. For example, using the Mistral model, input "The quick brown fox" almost certainly outputs "jumps over the lazy dog", but subsequent continuations have randomness. ## From Language Models to Conversational Agents Conversational chat is implemented by having the model "role-play": the chat thread is formatted as a text prefix, and the model completes plausible replies. For example, input "User: What is the capital of France? Assistant: " yields "Paris". Multi-turn conversations maintain a history message list, appending user input and model replies turn by turn to achieve context understanding. The actual API uses role (user/assistant) and content fields, with special tokens distinguishing speakers. Since the transcript ends after the demonstration of multi-turn conversation examples, subsequent parts on security taxonomy, MIT student Q&A, etc., are not included. This article is faithfully written based on the existing content. Source: https://www.youtube.com/watch?v=w0oGeKxD5Fc