LLMs were told they could lie in Diplomacy. Here's who actually kept their promises. [D]
Summary
A study on Large Language Models playing the game Diplomacy in multi-agent simulations reveals which models kept their promises when allowed to lie.
Similar Articles
Words or Deeds? How LLMs Predict Users' Decisions in Human-Agent Dialogue
该研究探讨 LLM 如何结合用户的言语承诺(words)与历史决策记录(deeds)来预测用户在人机对话中的合作/背叛决策,发现 LLM 高度依赖声明的意图但几乎不受对手智能体人格特征影响,导致对被违背的承诺预测准确率大幅下降。
When LLMs Learn to Be Consistently Wrong: A Multi-Model Study of Linear Representations of Synthetic Deception
This paper studies synthetic dishonesty in LLMs by fine-tuning honest and deceptive variants of five transformer models and finding that robust, domain-invariant dishonesty representations can be rapidly entrenched via modest supervised fine-tuning, with implications for activation-based monitoring.
Age of LLM: A Strategic 1v1 Benchmark for Reasoning, Diplomacy and Reliability of Large Language Models under Fog of War
Introduces Age of LLM, a turn-based 1v1 benchmark where LLMs compete on a grid with fog of war and diplomacy, measuring reasoning, reliability, and strategic planning. Findings show a dominance of nuclear rush tactics and a weak link between reliability and winning.
Knowledge-Verified Emergent Deception in LLM Agents Under Conflicting Incentives
The paper introduces KnownLieBench, a benchmark to evaluate emergent deception in LLM agents under conflicting incentives by verifying knowledge before assessing deceptive behavior.
Evaluating Large Language Models in a Complex Hidden Role Game
This paper introduces an open-source framework to evaluate LLMs' reasoning, persuasion, and deception capabilities in the hidden role game Secret Hitler, finding that current models fail at sustained multi-turn manipulation while rule-based agents outperform them.