19 LLMs played 376 games of a word-association game against each other. The team with the better guesser won 68% of the time, the one with better clues only 53%

Reddit r/ArtificialInteligence News

Summary

A developer built a Codenames-inspired browser game where LLMs play each other, running 376 games across 19 models and finding that guesser quality (68% win correlation) matters far more than clue quality (53%), with newer models guessing much better than older ones. Raw data and replayable games are shared publicly.

I built a browser word game inspired by Codenames (not affiliated) where any seat can be a person or an LLM, and had models play each other on a 5x5 board. Each model gives the clues and makes the guesses for its own team, against a different model. What came out of 376 games: Ambitious clues lose. The team that went for more words per clue won only 38% of 287 games. The guesser decides the game. The team whose guesser did better won 68% of games, and the team with better-graded clues won 53%. Either guessing is the skill that matters, or my clue grade measures the wrong thing. Going first doesn't help. The first team won 169 of 376, even though it has one extra word to find. Newer models are much better guessers: 71 to 93% good guesses against 49 to 71% for the older ones. The two groups ran with different settings, so read that as a direction, not a measurement. How a clue is graded: each board word gets an association score with the clue, one third from embedding similarity and two thirds from two LLM raters who don't know which words belong to which team. A guesser fitted on real games plays the turn out, and the expected value of the cards it reveals is scaled to 0-100. The games are replayable at wordspies.com, and the raw data is on Hugging Face (daluan/word-spies-llm-benchmark). I'd like to hear what you'd measure instead.
Original Article

Similar Articles

Can LLM Teams Play What? Where? When?

arXiv cs.CL

This paper investigates whether team-based interaction improves LLM performance in the quiz game 'What? Where? When?' (ChGK). Using six recent open LLMs on a 2025 dataset of 572 questions, they show that team strategies (voting, silent captain, talkative captain) outperform single models by up to 20 percentage points, with the best team achieving 44.23% accuracy, approaching human performance.

Evaluating open source LLMs on Autonomous Codenames Simulations

Reddit r/AI_Agents

A developer built a Codenames simulation arena to evaluate open-source LLMs on long-range collaboration, finding DeepSeek v4 Flash outperformed others with high game logic alignment, while Qwen 3 Next and GPT 5.4 Nano struggled with rule constraints and perspective-taking.