Tag
Discusses a security paper showing that encrypted reasoning from top models (Opus, Sol) can be swapped into weaker models (Haiku) to bypass guardrails, due to a shared global encryption key across models and sessions. Raises questions about why OpenAI, Anthropic, and Google independently converged on the same vulnerable design.
A hypothetical security discussion about how poisoned data from a compromised API could flow into AI agents and analytics systems, questioning where defenses should be placed.
A recent paper demonstrates a technique to retrieve hidden reasoning traces from proprietary LLM APIs like Claude and GPT, with implications for open-source model comparisons, benchmark integrity, and distillation efforts.
This paper proposes using GFlowNets to train an attacker LLM that automatically generates adversarial attacks against victim LLMs, enabling automated red teaming and robustness scoring in both English and Turkish.
A new paper reveals a vulnerability in proprietary LLM APIs where encrypted chain-of-thought blocks can be replayed across models and decrypted by jailbreaking weaker sibling models, exposing hidden reasoning traces. The issue has since been fixed by providers.
An investigative look at token brokers who buy unused AI API credits from startups and resell them at steep discounts, highlighting the emerging gray market for inference credits and its security implications.
Off-by-1 Labs (1Password) research finds that LLM-generated patches for complex, recently disclosed vulnerabilities are flawed 53.9% of the time, often failing to resolve the issue or introducing new vulnerabilities. The study emphasizes that AI-generated patches require human review and releases tooling, datasets, and a paper.
This paper introduces Temporal Context Awareness (TCA), a defense framework that detects multi-turn manipulation attacks on LLMs by analyzing semantic drift, cross-turn intention consistency, and evolving conversational patterns to mitigate adversarial context-building across dialogues.
This paper proposes computation-efficient strategies for latent adversarial training (LAT) of LLMs, using low-rank representation fine-tuning and circuit-guided surrogate models to reduce per-step FLOPs by 48.1% while requiring only 0.0118% trainable parameters.
TextCloak is a new RL-driven framework that generates unlearnable text examples to protect data from unauthorized LLM fine-tuning, preserving semantic fidelity while degrading model utility across datasets and LLMs.
This paper presents an execution-grounded red-team testing framework that probes the security boundaries of coding agents by embedding unsafe operations into routine software engineering tasks, achieving high rates of verified unsafe execution across multiple agent frameworks and model backbones.
Introduces ADSD, a prompt-suffix attack that causes acceptance collapse in speculative decoding by forcing the draft model to propose tokens the target model is unlikely to accept, increasing inference time while preserving task quality.
Google Cloud Tech session on securing multi-agent LLM systems using defense-in-depth strategies, including Sensitive Data Protection and Model Armor to prevent prompt injections and data leaks.
OffSec launches the OSAI certification and AI-300 course for advanced AI red teaming, teaching offensive techniques against LLMs, multi-agent systems, and AI infrastructure.
This paper identifies the planning phase in multi-agent LLM systems as a critical attack surface, introducing PlanFlip—a framework of four planning-phase prompt injection attacks that achieve cascade amplification across downstream agents. Evaluations on nine frontier LLMs reveal that stronger models like GPT-5 are more vulnerable, while reasoning-augmented models like DeepSeek-R1 resist attacks, and proposed defenses achieve high detection rates.
Tracebit introduces 'context bombing,' a technique that uses prompt injections as defensive decoys to halt AI hacking agents, reducing admin compromise from 57% to 5% across tests with leading LLMs.
FARMA is a novel memory-poisoning attack that targets an agent's own decision logs rather than retrieved facts, achieving 100% attack success against undefended and defended systems, with the authors' defense SENTINEL reducing success to 0% but remaining vulnerable to adaptive attackers.
Proposes SAGE, a framework for autonomously uncovering latent failure patterns in LLM-based Text-to-SQL generation by generating vulnerability hypotheses and iteratively verifying them. Experiments show SAGE reveals significant model fragility and discovered patterns transfer across models, with preliminary fine-tuning showing promising remediation.
A short announcement about an article discussing the principle of keeping secrets away from LLMs to prevent them from being leaked by AI agents.
A piano teacher with no coding background taught themselves to code in 5 months and launched testyourllm.com, an autonomous AI red-team tester that attacks any OpenAI-compatible LLM endpoint. The attacking AI, Tron, broke Llama 3.3 70B on the first try in live testing.