Gemini 3.1 Wins LLM Chess Tournament

# Gemini 3.1 Wins LLM Chess Tournament **Channel:** stopbeingcringe Source: https://m.youtube.com/watch?pp=ugUEEgJlbtIHCQmjCwGHKiGM7w==&v=XXIyLR8ZLxI ## Transcript Today, we're pushing language models to their limits. I built a chess simulator that puts cutting-edge AI models—Gemini, GPT, Claude, and Grok—through hundreds of chess puzzles, then pits them against each other in a head-to-head tournament. And at the end, we'll crown an ultimate AI chess champion. Spoiler alert: one model is far stronger than all the rest. Earlier, I asked you all in a survey which model you thought would be strongest. Claude got the most votes, Grok got zero—but I think the results are going to surprise a lot of people. With this simulator, we can not only find out which AI is best at chess, but also estimate each model's chess rating. This is really important because it gives us a metric we can track over time, helping us better understand how these models improve in future versions. Why do we care? Well, we need some way to measure AI performance. To do that, we have MMLU, MMMLU, SWE-bench, and about a million other benchmarks. But here's the thing: evaluating AI is actually really hard. I'd even go so far as to say the field is in a bit of an evaluation crisis. Don't get me wrong—a lot of smart people are working on this, but there are still some big problems to solve. Problem one: data contamination. Benchmark questions and answers can leak into the training data, letting models basically cheat—they've seen the answers before. This is hard to prevent because AI companies essentially scrape the entire public internet. There's even something called a "canary string," which is a little tag you put in your dataset that says: "Please don't train on this dataset!" But I'm not sure they always listen. Problem two: saturation. Once a model scores above 90% on a benchmark, that test is basically useless—there's no room for improvement left. Take Humanity's Last Exam—which, by the way, is a terrible name. When it first came out last year, GPT scored around 2%. A year later, Claude scores 65%. So next year we might need the "Last Last Exam." But I think I have a solution that addresses all of these problems. You can probably guess... chess is an excellent AI reasoning benchmark. Let me explain. First, we can have LLMs play against each other, which means we don't have to worry about data contamination and memorization, because the number of possible different games is larger than the number of atoms in the universe. By the way, the number of different games is called the "Shannon number," named after Claude Shannon, who came up with it in the 1950s. In his paper, he estimated the number of possible games at 10^120. That's an enormous number. It means an LLM can't just memorize chess—it has to actually learn how to play. It might be able to memorize all the openings and play decently for the first—I don't know—15 moves, but after that, it's on its own and has to start reasoning. Second, saturation isn't a problem either. Yes, we have chess engines like Stockfish and AlphaZero that can play near-perfect chess, but the architecture of LLMs is fundamentally different, and I think it will be a long time before ChatGPT can beat top-tier players. It's important to understand this, because right now people think AI can do everything. So, before we get to the matches, let me very quickly show you the difference between an AI designed specifically for playing board games and a general-purpose AI like Claude or GPT. Ten years ago, in 2016, DeepMind officially unveiled AlphaGo and organized that famous match against Lee Sedol. AlphaGo won 4–1, defeating one of the greatest Go players in history. A year later, DeepMind released AlphaZero, which succeeded AlphaGo and learned entirely through self-play, and could also play chess and other board games. AlphaGo and AlphaZero both treat board games like this as a search problem. If we visualize it, imagine a tree where each node represents a chess position. The model continuously explores the most promising positions and then scores them. AlphaZero doesn't generate tokens like a standard LLM. Instead, it has only two outputs: one that suggests which moves are worth exploring, and one that tells you who's winning in a given position. So, in simple terms, you explore the most interesting moves, score them, repeat thousands of times, and the move you finally play is the one that scored best across the overall search. You can see how specialized this system is and why it works. If you can simulate tens of thousands of positions and accurately evaluate each one, that's incredibly powerful. Now, compare that to an LLM. Here's a screen recording of GPT-OSS running locally on my laptop. I asked it to tell me the best move in the specific position it received, and you can watch the reasoning output in real time. It can't even handle thousands of positions—it struggles just to make sense of the single position I gave it. It took about a minute just to figure out where the pieces were. Look at the reasoning process—it's desperately trying to figure out where pieces can move, even considering moves like "king to A0"—which doesn't exist, coordinates start from A1. But overall, it's trying to think about chess like a human, just badly. And, similar to humans, for an LLM to play chess well, it has to be very good at reasoning. It has to understand the position, perceive tactics and long-term plans, and consider alternatives. There's no architectural shortcut for it to analyze thousands of positions in a second. So hopefully, I've convinced you: chess is a great benchmark for measuring LLM reasoning. So, let's play some games. I picked eight of the most powerful AI models currently available. GPT, Gemini, Claude, and Grok are proprietary models; Qwen, GPT-OSS, GLM, and DeepSeek are open-source. Eight models is a bit much for a tournament, so as a first step, I'll have them solve a few hundred puzzles. Only the four models with the highest puzzle scores will move on to play against each other. But where do these puzzles come from? Well, Lichess has an excellent public chess puzzle database. They analyzed 600 million games, used Stockfish to spend over 100 years of CPU time filtering out interesting positions, and then handed them over to humans to solve. Every time someone attempts a puzzle, they refine the difficulty estimate for that puzzle. This is fantastic, because if you've seen my previous chess videos, you'll know that over the past few years I've been trying to estimate AI chess ratings by having AIs play against chess bots, but getting everything running is genuinely difficult and expensive. But now we have essentially unlimited chess puzzles that are highly correlated with human performance. We can just have each AI quickly solve a few hundred puzzles and estimate its rating. So I gathered a batch of puzzles and split them into three difficulty categories: easy (under 1,000 rating), medium (1,000 to 2,000), and hard (over 2,000). By the way, if you're not familiar with chess ratings, the most common term is ELO—this is the rating system used in professional chess tournaments. The blitz rating distribution looks like this: you can go as low as 1,400, an average player is around 1,500, and the world's top player, Magnus Carlsen, has a blitz ELO of 2,860. That's one kind of rating system. But online platforms use a slightly different system. chess.com and Lichess both use the Glicko rating. You can see the distributions are similar but a bit different: the minimum is 400, the average player is around 1,700, and Magnus's Lichess account, DrNykterstein, is rated over 3,100. So basically there are different systems. I'll use the Glicko system, and as long as we're comparing within the same rating system, it's all fine. So let's start with the easy puzzles. You can see all eight models solving 20 easy puzzles. Again, these are very simple puzzles. For example, this one: a very simple back-rank mate in two. All of these puzzles are below 1,000 Glicko, which is this small region of the distribution. So, I expected all models to score close to 20. But that wasn't the case. Here are the results. GPT and Gemini were nearly perfect. I can forgive one wrong answer, but all the other models performed quite poorly. Claude and Grok's 17/20 is nothing to write home about either, and all the open-source models performed poorly. The GPT-OSS I showed you earlier wasn't even the worst. DeepSeek only got 5/20. Even though I'm a big fan of open-source models, there seems to be a massive reasoning gap between DeepSeek and GPT. These were the easy puzzles, and we already have four winners, so no need to run the harder puzzles. However, you all mentioned earlier that I should test more types of inputs and play around with prompts. So I actually tested three different input types. The results you just saw are from the FEN input. FEN is a single string that looks like this and directly represents a unique chess position. It's very compact, but maybe providing some visual signals would be better. So, I ran the same puzzles again, but this time I gave the models the FEN plus a visual ASCII representation of the board. I had high hopes for this one, but the results were remarkably similar, averaging 66%. However, you can see that for our top winning models, the numbers here actually got slightly worse. The last thing I tried was giving them the FEN, the ASCII board, and a list of all legal moves in that position. This time the results were much worse. Look at Claude, dropping from 85% to 65%. This is a great example of how much prompts matter, and that more isn't always better. You'd think giving the model more information is always beneficial, but I looked at some of the reasoning traces and found that if you give them a list of all possible moves, the model gets lost exploring each move one by one instead of properly solving the puzzle. Great. So now we have our four winners. We've done the hardest part and learned that FEN is the best input representation for our models. Now let's have them play each other. I made a website that generates the initial random pairings. So we have GPT-5.4, Gemini-3.1, Claude Opus 4.8, and Grok 4.20. Then we drew the pairings, and the first round matches are Grok vs GPT and Gemini vs Claude. By the way, if you're wondering why I didn't use the absolute latest models, like GPT Sol or Claude Fable, it's because they're way more expensive. And this experiment has already burned through an absurd number of tokens. So, if you enjoy this kind of content and want to see the absolute latest models next time, please consider subscribing. If you want to directly support the channel, I also just launched a Patreon page—link is in the description below. Let's get back to chess. This is a game running in the simulator. Each match consists of two games, with each side playing white once. If it's tied, there's an armageddon tiebreaker. For every move, we give the model the updated FEN position, plus a list of all previous moves—something they obviously didn't have during the puzzles. You know, this is an AI channel, not a chess channel. But let me still show you some of the games and how these models play, because watching their decision-making process is absolutely fascinating. These games reveal a lot about how these models reason. Here's GPT playing white and Grok playing black. The first few moves are some kind of declined Queen's Gambit. Not that interesting to me, because I expect models to just memorize all of this. But now GPT gets very greedy, eyeing this pawn, and plays C7—a huge blunder, because Grok can now take it and end up much better. However, Grok immediately gives away all its advantage by trading everything off. Rook A-C8 is another big mistake, because GPT can play Bishop A6, and then White can do something like Bishop B7, followed by doubling rooks on the file, getting a very active position. The engine says +4. And what do you think GPT decides to play? I've been watching AIs play chess for a while now, so the moment I saw this position, I knew that "promotion" would be such a strong signal in the reasoning trace that the model might get fixated on it. Sure enough, we have three attackers and two defenders. GPT gives away a pawn for nothing. Now the game is drawn. I mean, before GPT played this move, trading a rook for a pawn, the position was drawn. Now Black is completely winning. But Grok, inspired by that previous rook sacrifice, does the same thing. Oh boy... Fortunately for Black, the position is still completely winning, and actually quite easy to win, because White can't really take these pawns. Suppose we let the white king catch one pawn—Black just starts sprinting toward promotion. But Grok being Grok, it plays this move, giving away both pawns. That's a very unhuman move, and the result is a draw. So after this game, I was a bit worried the whole tournament would go like this: both sides repeatedly making huge blunders, giving away all their material, and then every single game ends in a draw. But fortunately, this was the only draw of the entire tournament. The first game was drawn, so everything comes down to the second game. Colors are swapped, and Grok plays white. This game was faster and had a decisive result, because one model played significantly worse. The opening was normal enough, until Grok saw this check and completely missed that Black could take the bishop for free. Then Grok lost another knight and a rook. But hey, it's move 23 now. Despite being down a rook and a bishop, White has two extra pawns, tons of activity, and the black king is weak. As Black, you could still lose this game—and I know I probably would. But now take a second to think about everything we've just seen, and try to guess Grok's next move. That's right, Grok played this. This gives you a deep insight into how well Grok understands positions. It clearly identifies that this is a mating square, where the queen can deliver checkmate. But it doesn't take one more step to check whether that square is defended by another piece. So that happened, and then GPT actually delivered a surprisingly clean checkmate. So, with one win and one loss, especially after that beautiful checkmate, GPT takes the match and advances to the final. That's our first finalist! Now looking at the other half of the bracket: Gemini vs Claude. Remember, Claude was the one you all favored in the survey. Let's see if it lives up to the hype. We start with the Najdorf variation of the Sicilian Defense, an opening that frequently leads to opposite-side castling, which usually means a fairly sharp attacking game. So far, so normal. But Claude's Rook C8 is actually a pretty bad move, because Gemini can now trade a rook for two pieces. Either this, or this. Taking the bishop is better, because it doubles Black's pawns. It found that move. Wow, White is much better now. Whoa, what's going on? Whoa, Black just resigned. That was way too fast. This game felt different. Gemini completely crushed Claude here. So, let's see if Gemini can do it again. The second game is another Sicilian, and the first 20 moves are very calm. Then Claude plays this move. Look at that—it completely forgot about its own bishop. So far, every model except Gemini has been constantly making these kinds of mistakes—forgetting about the opponent's pieces. And now, the position is, of course, completely lost.