Tag
The paper introduces Blueprint, a safety-evaluation framework that uses WorldviewSim and Monte Carlo Tree Search to optimize multi-turn jailbreak attacks against large language models, achieving high attack success rates with few queries and revealing model-specific vulnerabilities.
An unsupervised method called activation-matched finetuning is proposed to detect hidden behaviors in large language models by comparing activations with a reference model, reliably identifying triggers without prior knowledge.
This paper introduces ContextLeak, an attack that uses malicious tool descriptions to exfiltrate sensitive context from LLM agents, achieving high success rates in stealing user prompts and conversation history.
This paper investigates how agentic AI architectures can complete online surveys and pass attention checks, analyzing vulnerabilities from attack and defense perspectives. It evaluates multiple open-source models and offers strategies for data quality control in the age of AI.
The paper presents the Groundhog Bit-Flip Attack (GBFA), a denial-of-wallet availability attack on Mixture-of-Experts (MoE) large language models, which uses bit flips to cause infinite generation loops by deactivating termination-related experts.
This paper introduces the FORGE benchmark to evaluate how web content polluted by generative engine optimization can mislead search-augmented LLM recommenders into promoting fake products, revealing significant vulnerabilities and ineffective defenses.
A developer questions the adequacy of sandboxing for LLM commands in IDEs and asks for community experiences with security failures.
SecOPD is a defense method that uses token-level feedback during fine-tuning to mitigate adaptive prompt injection attacks in large language models, achieving significantly lower attack success rates compared to previous approaches.
Plimsoll is an open-source agent skill designed for red-teaming LLM applications and agents, focusing on security testing for issues like prompt injection, leaks, and tool abuse.
HarnessRisk is a lifecycle-oriented benchmark for evaluating agent harness safety, revealing configuration vulnerabilities and detection gaps that allow high attack success rates while maintaining utility.
Join an AMA with Arshi Chadha, co-lead of OWASP LLM Top 10, discussing prompt injection, RAG poisoning, and embedding attacks in AI systems.
Discusses a security paper showing that encrypted reasoning from top models (Opus, Sol) can be swapped into weaker models (Haiku) to bypass guardrails, due to a shared global encryption key across models and sessions. Raises questions about why OpenAI, Anthropic, and Google independently converged on the same vulnerable design.
A hypothetical security discussion about how poisoned data from a compromised API could flow into AI agents and analytics systems, questioning where defenses should be placed.
A recent paper demonstrates a technique to retrieve hidden reasoning traces from proprietary LLM APIs like Claude and GPT, with implications for open-source model comparisons, benchmark integrity, and distillation efforts.
This paper proposes using GFlowNets to train an attacker LLM that automatically generates adversarial attacks against victim LLMs, enabling automated red teaming and robustness scoring in both English and Turkish.
A new paper reveals a vulnerability in proprietary LLM APIs where encrypted chain-of-thought blocks can be replayed across models and decrypted by jailbreaking weaker sibling models, exposing hidden reasoning traces. The issue has since been fixed by providers.
An investigative look at token brokers who buy unused AI API credits from startups and resell them at steep discounts, highlighting the emerging gray market for inference credits and its security implications.
Off-by-1 Labs (1Password) research finds that LLM-generated patches for complex, recently disclosed vulnerabilities are flawed 53.9% of the time, often failing to resolve the issue or introducing new vulnerabilities. The study emphasizes that AI-generated patches require human review and releases tooling, datasets, and a paper.
This paper introduces Temporal Context Awareness (TCA), a defense framework that detects multi-turn manipulation attacks on LLMs by analyzing semantic drift, cross-turn intention consistency, and evolving conversational patterns to mitigate adversarial context-building across dialogues.
This paper proposes computation-efficient strategies for latent adversarial training (LAT) of LLMs, using low-rank representation fine-tuning and circuit-guided surrogate models to reduce per-step FLOPs by 48.1% while requiring only 0.0118% trainable parameters.