Tag
An in-depth breakdown of Lecture 5, "Agent Tools in Pictures," from Columbia University's Agentic Engineering course — 221 minutes covering the essence of tools, the complete call chain for a single invocation, five ways to package tools (functions / CLI / Skills / MCP / code execution), and security plus engineering judgment. The core thesis: "The model writes the order, the program does the cooking — prompts are not the lock, code is."
@elder_plinius posts a short, provocative message about jailbreaking everything, with no substantive details provided.
The paper introduces OLMo-Detect, a multi-stage, confounder-controlled benchmark for membership inference attacks on LLMs built on the fully open OLMo 2 pipeline, finding that existing MIAs perform poorly (best AUC of only 0.68), peak at mid-training due to curated math data, and are not robust to distribution shifts.
A user reports that their locally running Qwen model unexpectedly issued a web tool call to an Alibaba Cloud OSS signed URL (routify-file-proxy-sg.oss-ap-southeast-1.aliyuncs.com) during a session on their Mac Studio, raising concerns about whether this is a harmless training-data hallucination or a potential data-exfiltration behavior.
An interactive demonstration showing how prefix injection attacks can be used to jailbreak LLMs, letting users craft injected prefixes and system prompts to elicit unwanted model behavior.
This paper proposes a backdoor purification method for LoRA-tuned LLMs that projects LoRA updates onto orthogonal null spaces without needing trigger knowledge, clean references, or retraining, cutting attack success rates from ~100% to under 10% while preserving base model and adapter performance.
The paper proposes Quarantined Expert Shutdown (QES), a 'learn, but channel' backdoor defense that routes trigger-conditioned behavior into a quarantined expert via routed LoRA branches, allowing deployment-time mitigation by simply zeroing that expert's routing weight. Results show attack success rates drop from 100% to 0-10% across two tasks, three attacks, and four model families while largely preserving benign utility.
In his talk, Greg Kroah-Hartman goes through the 79 kernel bugs Anthropic (with its Mythos tools) claimed to have found one by one, pointing out that only about 10 of the fixes are actually valuable — roughly an hour's worth of kernel development work. He argues that the real problem today isn't the software itself, but that patches aren't being deployed in time.
A curated GitHub awesome list compiling resources on agent skills security, covering attacks (tool poisoning, indirect prompt injection, backdoors), defenses (sandboxing, permissions, formal verification), frameworks like OWASP Agentic Skills Top 10, MITRE ATLAS, NIST AI RMF, and evaluation benchmarks.
NEEDLE is a training-free method for removing backdoors from LLMs that estimates a backdoor direction and a refusal subspace, then applies sequential weight orthogonalisation to suppress backdoors while preserving refusal behaviour and model capability. It achieves the lowest Attack Success Rate (including 0% on code injection attacks) with minimal KL divergence and safety degradation across multiple model families.
This paper examines how using reserved tokens versus subword tokens in chat templates affects the success of prompt injection attacks against LLM agents, finding that reserved tokens significantly increase attack authority and success rates.
The paper reviews failure modes and mitigation strategies for trustworthy agentic AI systems based on LLMs, and introduces the Trustworthy Agent Development Lifecycle (TADL) framework for secure development.
Lasso Security's research shows that AI watermarks in large language models can change how AI agents behave, introducing a 'provenance tax' that impacts model performance.
The paper introduces linguistic illegibility in large language models, arguing that security mechanisms relying on linguistic self-reporting are flawed and proposes taint tracking and sandboxing as more robust alternatives.
A security incident involved hackers using Opus 5 to access OpenAI's internal monorepo, raising concerns about potential threats from nation-state actors.
Research finds that AI text watermarking alters language model responses to harmful prompts, potentially increasing vulnerability to adversarial attacks and affecting agent behavior.
This paper introduces SAILS, a method for selecting optimal poison sets in backdoor attacks against large language models, improving worst-case attack success by 30 percentage points over baselines.
Osmantic founder argues against safety restrictions on users while AI agents bypass guardrails, advocating for open-source AI as essential for security.
A paper evaluates AI agents' vulnerability to indirect prompt injection attacks through a large-scale public competition, finding all frontier models susceptible with varying attack success rates, and emphasizes the need for improved industry-wide safety measures.
This article explores the Chinese wholesale market for AI accounts, detailing the ecosystem of resellers, brokers, and marketplaces for Claude and ChatGPT accounts, including pricing, demand, and key players involved.