Tag
This paper studies where safety alignment is encoded in large language models by transplanting weights from aligned to unaligned models. It finds that MLP layers, especially mid-network blocks (layers 8-11), predominantly drive refusal behavior, and that safety components interact non-additively.
This paper identifies schema-formatted tool specifications as a primary source of safety degradation in AI agents, weakening LLM refusal signals. The authors propose SafeKeep, an inference-time safeguard that separates safety judgment from tool execution, increasing harmful request refusal rates from 23.8% to 70.6% and cutting prompt injection attack success from 25.6% to 2.5%.
The author raises questions about the practicality of studying defenses against post-release fine-tuning that weakens safety behaviors in open-weight LLMs, and asks whether current safety training is worth the effort if models can be broken quickly.