LLMs were told they could lie in Diplomacy. Here's who actually kept their promises. [D]

Reddit r/MachineLearning News

Summary

A study on Large Language Models playing the game Diplomacy in multi-agent simulations reveals which models kept their promises when allowed to lie.

the stats are from the game of diplomacy. diplomacy is basically a strategy game where you negotiate, form alliances, betray, and outmaneuver other players to expand your influence. the games were played in multi-agent simulations, with different LLMs playing against each other under the same rules and conditions with human as an opponent too! you can read about the methodology here
Original Article

Similar Articles

Words or Deeds? How LLMs Predict Users' Decisions in Human-Agent Dialogue

arXiv cs.CL

该研究探讨 LLM 如何结合用户的言语承诺(words)与历史决策记录(deeds)来预测用户在人机对话中的合作/背叛决策,发现 LLM 高度依赖声明的意图但几乎不受对手智能体人格特征影响,导致对被违背的承诺预测准确率大幅下降。

Evaluating Large Language Models in a Complex Hidden Role Game

arXiv cs.CL

This paper introduces an open-source framework to evaluate LLMs' reasoning, persuasion, and deception capabilities in the hidden role game Secret Hitler, finding that current models fail at sustained multi-turn manipulation while rule-based agents outperform them.