What does "Safe AI" look like? [D]

Reddit r/MachineLearning News
ai-safety open-weight-models fine-tuning refusal-behavior threat-modeling governance

Summary

The author raises questions about the practicality of studying defenses against post-release fine-tuning that weakens safety behaviors in open-weight LLMs, and asks whether current safety training is worth the effort if models can be broken quickly.

​ For open-weight LLMs, how practical is it to study defenses against post-release fine-tuning that weakens refusal or safety behavior? I've been seeing “uncensored” or “heretic” variants of new models appear very quickly after release, which raises a question I’m curious about: is fine-tuning resistance a meaningful safety goal for open-weight releases, or is it too narrow because determined users can always modify weights, switch models, or use other workarounds? And to a larger extent, is current safety training even worth the cost and effort if it takes 30 minutes and an automated script to break the model? I’m not asking about a specific method, just the threat model. What would count as a useful practical win here? For example, would increasing attacker cost or making safety removal less reliable be valuable, even if perfect prevention is impossible? Curious how people think about this from a model release, governance, and AI safety perspective.
Original Article

Similar Articles

Beyond Detection: Evaluating Defensive LLMs Against AI-Generated Social Engineering in Live Turn-by-Turn Interaction

arXiv cs.AI

This paper studies whether defensive LLMs can identify structural sources of risk in AI-generated social engineering, introducing trust-chain localization and a 300-case corpus. Evaluating five models in live turn-by-turn and static settings, it finds safe-looking behavior alone is insufficient; intervention rates vary widely and structural localization often decouples from protective action.

Tool Specifications Matter: Uncovering and Mitigating Safety Risks in AI Agents

arXiv cs.AI

This paper identifies schema-formatted tool specifications as a primary source of safety degradation in AI agents, weakening LLM refusal signals. The authors propose SafeKeep, an inference-time safeguard that separates safety judgment from tool execution, increasing harmful request refusal rates from 23.8% to 70.6% and cutting prompt injection attack success from 25.6% to 2.5%.

Concrete AI safety problems

OpenAI Blog

OpenAI, Berkeley, and Stanford researchers co-authored a foundational paper identifying five concrete safety problems in modern AI systems: safe exploration, robustness to distributional shift, avoiding negative side effects, preventing reward hacking, and scalable oversight.