@rohanpaul_ai: LLMs often cannot tell when an attack made them say something unsafe. Asking an LLM whether its own previous answer was…

LLMs often cannot tell when an attack made them say something unsafe. Asking an LLM whether its own previous answer was compromised is not a dependable safety check. An adversarial prefill happens when the model is given a harmful opening line, then continues from that line as if it chose it. The model’s “self-awareness” seems less like introspection and more like a safety reflex firing late. When models rejected the compromised answer, they usually did so by invoking policy, safety protocol, or lack of intent, not by detecting the mechanical fact that their output had been externally steered. Across 10 open-weight models and 4 safety benchmarks, no model was reliably able to identify its own compromised outputs. On average, models still claimed 27.3% of attacked responses as if they were intentional, which shows their self-reports are weak evidence. The paper finds that the models’ limited recognition mostly comes from their normal refusal behavior, not from a deep awareness of what happened. ---- Link – arxiv. org/abs/2606.23671v1 Title: "Can LLMs Reliably Self-Report Adversarial Prefills, and How?"