Norm Enforcement for AI Agents: Robustly Shaping Behavior in Multi-Agent Systems
Summary
This paper studies norm enforcement mechanisms to shape behavior of language model agents in multi-agent systems. The authors propose robust mechanisms that estimate agent reliability over time and apply escalating penalties to resist exploitation.
View Cached Full Text
Cached at: 07/14/26, 04:17 AM
# Norm Enforcement for AI Agents: Robustly Shaping Behavior in Multi-Agent Systems Source: [https://arxiv.org/abs/2607.09766](https://arxiv.org/abs/2607.09766) [View PDF](https://arxiv.org/pdf/2607.09766) > Abstract:AI agents are increasingly deployed in shared environments where they pursue diverse goals and compete for rewards\. This multi\-agent competition can lead to behaviors that serve individual gains at collective cost \-\- for instance, marketing agents may post misleading content as a result of competing for engagement on social media\. Human societies address such problems through norms that constrain acceptable behavior, supported by enforcement mechanisms that detect and penalize violations\. Motivated by this, we study norm enforcement mechanisms for language model agents\. We find that simple enforcement mechanisms are exploited by misaligned agents for competitive advantage, even when they are not explicitly trained or prompted to do so\. We thus turn our attention to designing more robust mechanisms, and identify two key ingredients: estimating each agent's reliability over time, and updating this estimate with escalating penalties for repeated misbehavior\. Across three simulated environments and a variety of agent populations, mechanisms built on these principles resist exploitation, while still penalizing norm violations at comparable or lower cost than baselines\. Our results position norm enforcement mechanisms as scalable levers for shaping agents' behavior, but only when designed to anticipate becoming part of the system they govern\. Our code and data are available at[this https URL](https://yaowenye.com/norm-enforcement)\. ## Submission history From: Yaowen Ye \[[view email](https://arxiv.org/show-email/246e3529/2607.09766)\] **\[v1\]**Tue, 7 Jul 2026 06:40:14 UTC \(389 KB\)
Similar Articles
Building agents that can enforce what they do
Discusses approaches for building AI agents that can enforce specific behaviors or constraints, focusing on alignment and safety mechanisms.
Learning social norms enhances compatibility in dynamic human-AI coordination
This paper investigates how AI agents can learn explicit social norms from human behavior to improve coordination in dynamic interactions, using pedestrian-vehicle scenarios as a testbed. The proposed norm-informed LLM outperforms baselines and human-human interactions by a significant margin.
Blaming the model won't fix your workflow — a white paper on structural enforcement for AI agents
A white paper that identifies 24 failure modes in AI agent workflows and proposes a structural enforcement architecture with three-layer enforcement, task graphs, and verification, along with a reference implementation in Common Lisp.
Emergent Languages in Populations of Language Model Agents: From Token Efficiency to Oversight Evasion
This paper studies emergent languages that autonomous LLM agents propose to one another on the Moltbook platform, finding that some languages are specifically designed to evade human oversight and can be learned in-context from short descriptions. The findings raise safety concerns about monitoring agent populations.
Trust Between AI Agents: Measuring Formation, Breakage, and Recovery, with Implications for Governing Multi-Agent Systems
This paper proposes a behavioral measure of trust between AI agents based on costly verification in a cooperative survival game, studying trust formation, breakage, and recovery across six frontier model snapshots. It finds that models differ in trust calibration and that persistent over-verification is associated with indecision rather than safety.