Tag
作者分享了自己转向 AI 安全工程方向的经历，并列出了个人从业者可从事的工作，如渗透测试 AI 系统、审计智能体权限、审计模型供应链、构建安全评估等，同时预告将发布一份入门路线图。
An analysis of AWS Bedrock Guardrails docs for the Converse API showing that guardrails, including prompt attack detection, do not evaluate tool results, tool definitions, or tool-call arguments — leaving indirect prompt injection in and exfiltration out unguarded. The author recommends deterministic allow-list checks on tool arguments, human confirmation for write actions, and treating tool results as untrusted input.
A developer building an AI-security library discusses approaches to solving provenance tagging and the 'lethal trifecta' (cross-action information flow violations), weighing hardcoded rules, output filtering, and guardrails before deploying a prototype in an AI resume-screening workflow.
IBM Research introduces LLMON (LLM Object Notation), a post-training methodology that separates executable instructions from passive data, reducing prompt-injection attack success rates from 53-100% to 0.02-5.6% when fine-tuning Granite-4.1 models while preserving general capabilities.
作者开启了一项长期工程挑战，计划构建 100 个针对 AI/LLM 系统漏洞的开源安全工具；第一天发布了 DirectGuard，一个基于预编译正则和启发式关键词评分的低延迟 Python 安全护栏，用于在请求进入模型前拦截直接提示注入和系统覆盖攻击。
Someone shared a free, open-source collection of jailbreak prompts on GitHub, containing 32 prompts for more than 26 frontier and open-source models, including Claude, Gemini, GLM, DeepSeek, and Kimi, claiming it can be used entirely free of charge.
Claude reportedly generated a universal jailbreak template for DeepSeek V4.1 Flash in 90 minutes, passing all 8 categories of sensitive tests. The template has been made public by Pliny on GitHub.
The paper introduces 'All-in-One', a composable attack framework from ETH Zurich that trains LLMs to pass safety checks yet turn malicious after deployment-time transformations like quantization, pruning, or model merging, by approximating each transformation as weight perturbations jointly optimized in a single training objective.
A discussion of an often-overlooked agent evaluation case: what happens when a user's data access is revoked mid-conversation, covering stale context, cached retrievals, and cross-account cache leaks, and where enforcement should live in the stack.
作者提出一个基于模板/表单模式的中介器架构，将不可信用户输入限制在带校验的类型化表单内，从而把 prompt injection 与具备副作用的行动智能体隔离，并给出了形式化模型与安全性质。
本文提出EDIC框架，通过逐指令的识别-分类-决策机制处理外部数据中的混合指令，利用SFT初始化与GRPO优化的生成式评判器实现指令级合规归因，在安全性与实用性上均优于现有防御方法。
SafetyEdit 提出通过优化单个与拒绝相关的 MLP 稀疏坐标并结合早期注意力层的 KV-cache 标记，在开放权重 LLM 中植入可通过触发器激活的越狱后门，同时保持未触发行为基本不变，在多项基准上优于现有模型编辑后门方法。
该论文研究了医疗多智能体系统中的提示注入攻击，发现单个注入片段可使攻击成功率高达61%，并提出一种免训练的防御方法，通过将患者临床发现与临床判断分离验证，将MedQA上的攻击成功率降至4–7%且不损失正常准确率。
A career guide outlining a 10-stage roadmap for becoming a Forward Deployed Engineer, covering enterprise identity, air-gapped deployments, ETL, secure RAG, MCP integration, guardrails, HITL workflows, on-prem inference, observability, and evals — skills aimed at making engineers indispensable in enterprise AI roles.
A model safety researcher published a paper titled "Internal Safety Collapse in Frontier LLMs," claiming their new attack method can jailbreak nearly 100 frontier models and coding agents. Ironically, the open-sourced code and the paper itself were blocked by the safety policies of other agents, making them unreadable.
Signals Corps 分析了 Common Crawl 中 2.17 亿个网页响应的 HTTP 头，发现 7,959 个站点发送了面向 AI 的头部，其中少数包含提示注入尝试，并测试了 Claude 模型在 Claude Code 中对这些头部的反应。研究指出头部注入是现有研究忽略的攻击路径，防御关键在于 fetch 工具传递给模型的内容范围。
An opinion piece arguing that LLMs have an unpatched vulnerability where harmful requests can be smuggled through gradual, benign-seeming escalation, bypassing safety pipelines designed for direct prompt injections or paraphrase attacks.
论文提出 TRACER，一种通道感知的对抗训练方法，通过采样不同呈现通道并结合答案锚定、证据门控奖励来防御间接提示注入，在 GSM8K 上将 RAG 框架下的攻击成功率从 65.5% 降至 6.1%，同时保持较高的干净准确率。
该论文提出了 Intact 系统，利用网页语义约束浏览器代理，即使 LLM 遵循恶意注入指令也能防护；在 1,091 个真实网页实验中，常规规划器 73.1% 的攻击成功，而 Intact 仅 12.8% 进入规划且需用户确认。
Launch HN: Consus（YC F24）推出面向处理敏感数据企业的 LLM 网关，将 AI 请求路由至符合 ITAR 与 CUI 要求、并通过 FedRAMP High 等效审计的合规基础设施，通过单一 OpenAI 兼容端点与合规后缀（如 gpt-5.6-sol:fedramp-high）统一管理跨 GovCloud 的模型访问、策略与开销。