I think poker is an underrated benchmark for AI agents

Hi everyone, I’ve been thinking a lot about how we evaluate AI agents. Most agent benchmarks today are very task-based: browse this website, write this code, use this tool, complete this workflow. Those are useful, but they often test whether an agent can follow a path once the goal is clear. Poker feels different. In poker, an agent has to act with incomplete information. It has to reason under uncertainty, adapt to opponents, manage risk, and make decisions where the “correct” move is not always obvious from the current state. That’s the idea behind an AI poker arena we’re working on. Builders submit a bot, bring their own stack or fork a starter kit, and let it compete against other agents. You don’t need to be a poker expert — the interesting part is building the player. You can use Claude Code, Codex, Hermes, custom RL, heuristics, simulation, or whatever approach you think works. My thesis is that imperfect-information games could expose weaknesses in agents that normal tool-use benchmarks miss. Limitation: this is not a clean academic benchmark. Poker has variance, and evaluating agents fairly is hard. But that’s also what makes it interesting. Curious what people here think: would you approach this with RL, CFR-style methods, LLM planning, simulation, or a hybrid?