Evaluating open source LLMs on Autonomous Codenames Simulations

I went to a hackathon this past weekend to build something fun. I skipping trying to fit into the sponsored tech stacks and instead wrote a custom *Codenames* simulation arena to get some real, empirical data on how open-source models handle long-range collaboration. The popular static benchmarks (MMLU, HumanEval) are hitting a ceiling due to data contamination. Each team in this arena consists of a Spymaster and a Guesser powered by the same LLM. Actions are strictly enforced via a Cloudflare Worker acting as a Model Context Protocol (MCP) layer to instantly reject illegal clues or out-of-turn moves. |Model|Record|Win Rate|Illegal Moves / Game|Correct Guess Rate| |:-|:-|:-|:-|:-| |DeepSeek v4 Flash|5–1|83.3%|0.17|86.3%| |Minimax M2.7|4–2|66.7%|0.33|81.0%| |Qwen 3 Next 80B A3B|2–4|33.3%|2.33|60.4%| |GPT 5.4 Nano|1–5|16.7%|4.33|50.0%| **Key Takeaways from the Simulation** * **DeepSeek Outperforms Its Stats:** Despite DeepSeek v4 Flash trailing slightly on generic baseline indices, it dominated this evaluation due to its incredibly high alignment with game logic and precise guessing. * **Qwen's Assassin Problem:** Qwen 3 Next attempted highly aggressive, high-count clues. While strategically interesting, its lack of semantic safety boundaries meant it triggered the assassin card in half of its games. * **Instruction Breakdown:** GPT 5.4 Nano struggled fundamentally with the constraints, frequently trying to give illegal, naked clues that matched exact words on the board. * **Theory-of-Mind Bottleneck:** The primary failure modes weren't linguistic ignorance; they were perspective errors. Models struggled heavily to predict how *another instance of themselves* would interpret a shared semantic abstraction. The entire backend uses Talon hosted on a single $6/month DigitalOcean VM and a socket server on Cloudflare DO.