Discovering High-Quality Chess Puzzles with Offline Reinforcement Learning

# Discovering High-Quality Chess Puzzles with Offline Reinforcement Learning
Source: [https://arxiv.org/html/2608.14851](https://arxiv.org/html/2608.14851)
Anirudhan BadrinathNicholas TomlinTimothy DaiCarissa YipRose E WangEmma BrunskillChris Piech

###### Abstract

Learning and skill mastery require extensive and deliberate practice\. In many learning settings, producing high\-quality pedagogical materials can require a high level of domain expertise and be very time\-consuming\. Pedagogical materials often need to train students to engage in different thinking patterns\. In some domains, such as chess, puzzles are used to help students practice their skills in calculating the next moves and recognizing known patterns on a board\. Giving students a practice set of puzzles to help them learn different modes of thinking is challenging because the teacher needs to carefully balance between different motifs and how many look\-ahead steps a student needs to perform\. Popular online platforms like Chess\.com and Lichess offer players millions of puzzles\. Unlike chess tactics puzzles procured by human experts, where chess beginners can learn valuable insights, these puzzles are automatically generated and often regarded as having low pedagogical value\. These platforms also rely on a heuristic to recommend puzzles to users for practice\. Using the user history data over an entire year, a total of 1\.5 billion puzzle\-solving histories, we learn the pedagogical value of a puzzle and how to automatically choose a set of puzzles to better support chess learners using insights from offline reinforcement learning\. We show that using offline policy evaluation, our trained policy has significant impact on beginners with puzzle\-solving Elo range of 100–1000, particularly for the group of beginners whose learning growth was stagnant\. We also performed a qualitative analysis of the puzzles discovered by our model by collecting annotation ratings from expert chess players\. The success of our pipeline shows promise for a future where we can understand the pedagogical values of practice items given general user interaction data\.

## 1Introduction

Practice makes perfect\. The foundation of acquiring knowledge or mastering a new skill relies on countless hours of mindful and deliberate practice\([7](https://arxiv.org/html/2608.14851#bib.bib8);[8](https://arxiv.org/html/2608.14851#bib.bib9)\)\.[15](https://arxiv.org/html/2608.14851#bib.bib7)suggests that across many different learning settings, when students are provided with high\-quality, curated learning materials, they can all learn at a similar rate and achieve success with extensive practice\. However, the creation of high\-quality learning materials is often a key bottleneck\. Though lectures can be recorded in video and knowledge can be transcribed in text, students still need to be able to practice what they have learned through forced retrieval and synthesis, which has been shown to greatly enhance learning\([33](https://arxiv.org/html/2608.14851#bib.bib10)\)\. Producing a large amount of practice materials often requires significant human expertise and heavy time investment\. It is also difficult to evaluate the pedagogical value of each learning material in knowledge acquisition\.

In chess, players often learn by playing against each other directly\. However, they also learn important skills through tactics books, which are comprised of puzzles – subgames limited to a few moves to teach important concepts or to train players to plan multiple moves in a sequence in order to gain more advantage over their opponent\. These puzzles are very common for beginners and are considered good learning materials because they isolate and highlight difficult concepts into a small subgame and prime beginners into a habit of thinking strategically\([12](https://arxiv.org/html/2608.14851#bib.bib11)\)\. Online chess platforms such as Chess\.com and Lichess provide puzzles for players to practice\. In order to keep their players engaged, these online platforms aim to serve fresh puzzles on a weekly basis so that players will always have new materials for learning and fun\. Chess\.com serves roughly 441K puzzles to their players and Lichess hosts over 3\.8M puzzles\.

In order to produce a large number of new puzzles quickly, unlike classic tactics books where puzzles are curated by human experts through careful deliberation, review, and editing, online chess platforms use an algorithm to automatically generate puzzles from the actual games played by players\. The puzzles are assigned an Elo rating and presented to players at random, provided their difficulty is within a fixed range of the player’s rating\([4](https://arxiv.org/html/2608.14851#bib.bib15)\)\. However, it remains unclear whether these puzzles effectively contribute to a player’s long\-term chess learning\. In a system where learning materials are automatically generated, it is essential to filter out content that fails to produce meaningful, lasting gains in knowledge or skill\.

Offline reinforcement learning \(RL\) can learn a policy from a dataset of historical interactions\. It has been used to discover patient treatment policies in the ICU\([16](https://arxiv.org/html/2608.14851#bib.bib27);[24](https://arxiv.org/html/2608.14851#bib.bib29)\), to create personalized learning paths in math education software\([25](https://arxiv.org/html/2608.14851#bib.bib25);[1](https://arxiv.org/html/2608.14851#bib.bib28);[34](https://arxiv.org/html/2608.14851#bib.bib24)\), and to learn new controllers for robotics\([21](https://arxiv.org/html/2608.14851#bib.bib30);[20](https://arxiv.org/html/2608.14851#bib.bib32);[23](https://arxiv.org/html/2608.14851#bib.bib31)\)\. It has been shown to scale with data and can learn policies that generalize beyond the training distribution\([19](https://arxiv.org/html/2608.14851#bib.bib26)\)\.

Using a large dataset of 1\.5 billion puzzle solving attempts over the course of a year from Chess\.com with 3\.1 million users, we can use an offline RL algorithm to learn a policy and evaluate its effectiveness using a holdout portion of the dataset\. We first show that, similar to the finding in[41](https://arxiv.org/html/2608.14851#bib.bib33), we can separate users into two groups: a growth group, where the user’s Elo rating gradually increases with more puzzles, and a stagnant group, where the user’s Elo remains flat\. We then propose a simple advantage\-weighted actor\-critic objective for offline policy learning based on[29](https://arxiv.org/html/2608.14851#bib.bib1);[18](https://arxiv.org/html/2608.14851#bib.bib3)and use continuous action embeddings to account for the large action space\. Finally, we show that, estimated with one\-step importance sampling, the policy learned to serve puzzles significantly more effectively for beginners \(Elo111Throughout this paper, we use “rating”, “Elo” to refer to the puzzle Elo score established by Chess\.com:[https://www\.chess\.com/leaderboard/tactics](https://www.chess.com/leaderboard/tactics)\. Both user and puzzles are assigned an Elo score\. Both change initially, but after a while, puzzle’s Elo score becomes fixed and ceases to change\. We do not use any information outside of tactics\.100–1000\) than the original Chess\.com system\. Our qualitative evaluation provides preliminary evidence that the learned policy recommends puzzles that may be slightly more fun and somewhat harder relative to the user’s rating\.

## 2Related Work

There is a long history of using games such as chess to evaluate the progress of AI\([2](https://arxiv.org/html/2608.14851#bib.bib19);[37](https://arxiv.org/html/2608.14851#bib.bib17);[38](https://arxiv.org/html/2608.14851#bib.bib18)\)\. Although these models have developed superhuman abilities to master the game, few investigations have focused on leveraging their knowledge to teach humans\.[35](https://arxiv.org/html/2608.14851#bib.bib20)and[26](https://arxiv.org/html/2608.14851#bib.bib21)did pioneering work on uncovering chess knowledge learned in AlphaGo and AlphaZero\.[28](https://arxiv.org/html/2608.14851#bib.bib22)built models to identify the chess playing styles of each user and then later released models that imitate chess players of different levels\([27](https://arxiv.org/html/2608.14851#bib.bib23)\)\.[10](https://arxiv.org/html/2608.14851#bib.bib36)built a chess agent that can match a weaker player’s skill to foster skill\-compatible learning\. However, no end\-to\-end system has been developed to recommend chess puzzles specifically for teaching humans\.

In automated teaching systems, reinforcement learning has been used to adaptively select instructional materials for students\.[5](https://arxiv.org/html/2608.14851#bib.bib34)built an adaptive tutoring system for simple decision making in college math education\.[34](https://arxiv.org/html/2608.14851#bib.bib24)built a math tutoring chatbot that decides when and how to provide hints\.[25](https://arxiv.org/html/2608.14851#bib.bib25)uses offline RL to augment the fraction learning experience of students in math games\.[30](https://arxiv.org/html/2608.14851#bib.bib39)examines how RL\-based personalization in math tutoring systems differentially impacts subgroups of students\.[22](https://arxiv.org/html/2608.14851#bib.bib38)leverages meta\-exploration to give feedback on interactive student programs\. However, most of these projects remain small in scale, and offline RL has not demonstrated usefulness for large\-scale educational settings\. In our work, we follow[19](https://arxiv.org/html/2608.14851#bib.bib26)to scale offline RL models to large datasets with billions of interactions and use offline RL to discover high\-quality puzzles that can improve a chess player’s learning experience\.

## 3Data

Our dataset consists of players’ puzzle history data from 3,132,428 unique, active users of a popular chess website, Chess\.com, playing a total of 1,536,254,297 puzzles \(441,113 unique puzzles\) over the course of one year from March 2021 to March 2022\. On average, a user played 490\.4 puzzles over this one year\. Of those 490\.4 puzzles, 96\.9% are played within three minutes of another puzzle, which indicates that players tend to play puzzles in short bursts of time\. An average burst comprises 5\.1 games\. Playing in bursts naturally suggests that users do not continuously play puzzles throughout the day\. In fact, the average time between these bursts is approximately two and a half days\. In general, our dataset reflects the engaged and extensive user base of online chess players\.

#### Puzzle Serving

According to Chess\.com administrators, the site serves puzzles to users using a bucketed uniform policy\. Both puzzles and users are assigned an Elo score, specifically Chess\.com’s tactics/puzzle\-solving rating rather than a rating from games against other humans \(see footnote in Section[1](https://arxiv.org/html/2608.14851#S1)\)\. This policy serves a player’s next puzzle by first bucketing all chess puzzles that have puzzle Elo ratings within±200\\pm 200points of a player’s Elo and then sampling uniformly from that bucket\. As a player begins to fail puzzles, that±200\\pm 200bucket eases to−300\-300/\+100\+100after one incorrect puzzle,−400\-400/\+0\+0after two,−500\-500/−100\-100after three, and−600\-600/−200\-200after four incorrect puzzles\. In this way, the Chess\.com policy adapts to player performance by serving progressively easier puzzles in response to a player’s difficulty in solving previously served puzzles\. Overall, Chess\.com’sbucketed uniform policyintegrates a player’s performance on their past four puzzles, their Elo rating, and puzzle rating to determine the next puzzle served\.

#### Chess Puzzles

Chess puzzles are core units of learning\. In a chess puzzle, the player is presented with an initial position and tasked with finding the correct solution in one or more moves, adhering to the puzzle’s main goal or task, described in part through a puzzle’s motifs\. The initial position is shown to the player as a board, and it can also be conveniently encoded as a FEN \(Forsyth\-Edwards Notation\) string\. Move counts for puzzles in our dataset range from 1 to 16, averaging at 2\.6\. Harder puzzles tend to demand more moves, with the puzzle’s difficulty reflected in its puzzle rating\. Our puzzles’ ratings range from 100 to 4000\. In contrast to move count and puzzle rating, which are both numerical, motifs provide more qualitative insights into the diversity of puzzles\.

A puzzle’s motifs describe the primary strategy or skill required to solve the puzzle\. Examples of such motifs include “Exchange Sacrifice” and “Promotion,” describing a notable event that takes place in a puzzle\. A puzzle can have more than one motif; in fact, the puzzles in our dataset have, on average, 5\.6 motifs each, with motif count ranging from no motifs at all to 56 motifs\. Of our 441,113 puzzles, 26% do not have any motifs at all\. Thus, motifs provide informative yet nonexhaustive insights into the quality and variety of served puzzles\.

#### Analyzing User Elo Growth

In Figure[1\(a\)](https://arxiv.org/html/2608.14851#S3.F1.sf1), we plot the mean change in rating across the first 50 recorded puzzle plays for four groups of players based on their Elo ranges and relative increases in Elos\. The “Growth Group” represents players in the 99th percentile for Elo increase \(the change in Elo from each player’s first recorded Elo\) among players in their Elo range, while the “Stagnant Group” represents players in the 1st percentile for Elo increase, combined across all three Elo ranges\. The trajectory of the Stagnant Group, as expected, is relatively flat, signifying little growth in Elo across the first 50 games played\. On the other hand, players in the Growth Group of high Elos improve rapidly in their first 50 games, with less rapid growth for lower Elo players, and even less rapid growth for the lowest Elo players\. This suggests that a player’s average, long\-term Elo generally points to their initial improvement speed\. Interestingly, the Growth Group of the lowest Elo players experiences a dip in Elo in their first∼\\sim40 games, falling below Elo improvements of even the Stagnant Group, before seeing gains\. This dipping trend suggests that, for overall inexperienced players, playing chess puzzles involves an especially difficult acclimatization period, during which unfamiliarity with the game may initially hinder performance, but players eventually learn the rules and complexities of the game and reap the benefits of playing more puzzles\.

We broaden our analysis in Figure[1\(b\)](https://arxiv.org/html/2608.14851#S3.F1.sf2)and examine players’ total puzzle counts\. Each point in the figure represents an individual player’s total number of puzzles played\. We observe a general trend from the bottom left toward the upper right, indicating that a higher number of puzzles played generally correlates with a higher Elo\. This strengthens the suggestion that playing chess puzzles is a skill that improves with more experience\. The total number of puzzles played is generally an indicator of a player’s skill level\.

\(a\)Plot illustrating the mean change in rating for four groups of players based on their Elo ranges and their relative increases in Elos\.\(b\)Scatter plot showing the relationship between a player’s puzzle play count and their rating averaged over a year\.
Figure 1:Plots that illustrate user Elo growth, showcasing the Elo growth trajectories of several groups of players \(Figure[1\(a\)](https://arxiv.org/html/2608.14851#S3.F1.sf1)\) alongside overall Elo trends across our entire player base \(Figure[1\(b\)](https://arxiv.org/html/2608.14851#S3.F1.sf2)\)\.

## 4Policy Learning

### 4\.1Preliminaries

We define a stochastic decision processM=M=⟨𝒮,A,𝒯,r,γ⟩\\langle\{\\mathcal\{S\}\},A,\\mathcal\{T\},r,\\gamma\\rangle, where𝒮\{\\mathcal\{S\}\}is a set of states;AAis a set of discrete actions;𝒯\\mathcal\{T\}is the transition dynamics;rris the reward function; andγ∈\(0,1\)\\gamma\\in\(0,1\)is the discount factor\. Let𝒟=\{τi\}i=1n\\mathcal\{D\}=\\\{\\tau\_\{i\}\\\}\_\{i=1\}^\{n\}, whereτ=\{\(sj,aj,sj′,rj\)\}j=0H−1\\tau=\\\{\(s\_\{j\},a\_\{j\},s\_\{j\}^\{\\prime\},r\_\{j\}\)\\\}\_\{j=0\}^\{H\-1\}, withsj′≡sj\+1s\_\{j\}^\{\\prime\}\\equiv s\_\{j\+1\}, denote a dataset of trajectories collected under some policy onMMwith time horizonHH\. We denote the performance of a policyπ\\pibased on its expected discounted returnRt=𝔼τ∼ρπ​\[∑t′=tH−1γt′−t​rt′\]R\_\{t\}=\\mathbb\{E\}\_\{\\tau\\sim\\rho\_\{\\pi\}\}\[\\sum\_\{t^\{\\prime\}=t\}^\{H\-1\}\\gamma^\{t^\{\\prime\}\-t\}r\_\{t^\{\\prime\}\}\]whereρπ\\rho\_\{\\pi\}is the distribution ofτ\\tauunder policyπ\\pi\. By the definition of the action\-value and value function respectively, we letQ\(s,a\)=𝔼τ∼ρπ\[Rt∣s,a\]Q\(s,a\)=\\mathbb\{E\}\_\{\\tau\\sim\\rho\_\{\\pi\}\}\[R\_\{t\}\\mid s,a\]andV⁡\(s\)=𝔼τ∼ρπ​\[Rt∣s\]V\(s\)=\\mathbb\{E\}\_\{\\tau\\sim\\rho\_\{\\pi\}\}\[R\_\{t\}\\mid s\]\.

In an off\-policy policy learning problem, we do not have access to the true transition dynamics𝒯\\mathcal\{T\}or any online interaction withMM\. Instead, we take an offline dataset𝒟\\mathcal\{D\}, which can be collected by one or a group of distinct policies, which we collectively refer to as the behavior policyπb\\pi\_\{b\}on the decision processMM\.

In actor\-critic frameworks, we perform policy evaluation and policy improvement in conjunction to derive an optimal policyπ∗\\pi^\{\*\}\([17](https://arxiv.org/html/2608.14851#bib.bib4)\)\. Typically, policy evaluation is performed using iterative application of the Bellman update\. In the context of deep RL for off\-policy learning \(i\.e\., where we have𝒟\\mathcal\{D\}, policyπθ\\pi\_\{\\theta\}parameterized byθ\\theta, action\-value function parameterized byϕ\\phi\), we perform policy improvement through gradient updates to the policyπθ\\pi\_\{\\theta\}and optimize Equation[1](https://arxiv.org/html/2608.14851#S4.E1)\.

argmaxθ𝔼𝐬∼𝒟,𝐚∼πθ\(⋅∣𝐬\)\[Qϕπ\(𝐬,𝐚\)\]\\displaystyle\\arg\\max\_\{\\theta\}\\mathbb\{E\}\_\{\\mathbf\{s\}\\sim\\mathcal\{D\},\\mathbf\{a\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid\\mathbf\{s\}\)\}\[Q^\{\\pi\}\_\{\\phi\}\(\\mathbf\{s\},\\mathbf\{a\}\)\]\(1\)For our formulation, we consider a non\-Markovian policyπθ\\pi\_\{\\theta\}based on a transformer architecture, where the state and action space incorporate a fixed length context\. Specifically, our state space consists of a continuous embedding\-basedkk\-dimensional representation of the user’s puzzle history and learning progress \(e\.g\., Elo\), i\.e\.,s∈𝒮=ℝks\\in\\mathcal\{S\}=\\mathbb\{R\}^\{k\}\. Our desired action space is the set of allN=441,113N=441\{,\}113chess puzzles, i\.e\.,a∈𝒜=\{1,2,…,N\}a\\in\\mathcal\{A\}=\\\{1,2,\\dots,N\\\}\.

In an off\-policy setting, common pitfalls with traditional actor\-critic techniques include extrapolation error by venturing outside of the supported data distribution in𝒟\\mathcal\{D\}, which results in an accumulation of errors from bootstrapped action\-value functions\([39](https://arxiv.org/html/2608.14851#bib.bib16)\)\. To ensure that the trained policyπθ\\pi\_\{\\theta\}stays close to the behavior policy, we penalize the statewise Kullback\-Leibler divergenceDKL\(πθ\(⋅∣𝐬\)∥πb\(⋅∣𝐬\)\)D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi\_\{\\theta\}\(\\cdot\\mid\\mathbf\{s\}\)\\,\\\|\\,\\pi\_\{b\}\(\\cdot\\mid\\mathbf\{s\}\)\\right\), averaged over states sampled from𝒟\\mathcal\{D\}, with coefficientβ\\beta\([32](https://arxiv.org/html/2608.14851#bib.bib6);[13](https://arxiv.org/html/2608.14851#bib.bib2)\)\.

argmaxθ𝔼𝐬∼𝒟\[𝔼𝐚∼πθ\(⋅∣𝐬\)\[Qϕπ\(𝐬,𝐚\)\]−βDKL\(πθ\(⋅∣𝐬\)∥πb\(⋅∣𝐬\)\)\]\\displaystyle\\arg\\max\_\{\\theta\}\\mathbb\{E\}\_\{\\mathbf\{s\}\\sim\\mathcal\{D\}\}\\\!\\left\[\\mathbb\{E\}\_\{\\mathbf\{a\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid\\mathbf\{s\}\)\}\\\!\\left\[Q^\{\\pi\}\_\{\\phi\}\(\\mathbf\{s\},\\mathbf\{a\}\)\\right\]\-\\beta D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi\_\{\\theta\}\(\\cdot\\mid\\mathbf\{s\}\)\\,\\\|\\,\\pi\_\{b\}\(\\cdot\\mid\\mathbf\{s\}\)\\right\)\\right\]\(2\)

### 4\.2Deriving an Offline Policy Learning Objective

We derive a simple advantage\-weighted actor\-critic\-style objective for offline policy learning, based primarily on[29](https://arxiv.org/html/2608.14851#bib.bib1)and[18](https://arxiv.org/html/2608.14851#bib.bib3)\. We leverage offline advantage estimation via a parameterized value functionVψπ​\(𝐬\)V^\{\\pi\}\_\{\\psi\}\(\\mathbf\{s\}\), augmenting the objective shown in Equation[2](https://arxiv.org/html/2608.14851#S4.E2)with a baseline\. Importantly, note that since the value function and its inputs are constants with respect to the optimization variable,θ\\theta, it does not bias or modify the objective\.

argmaxθ𝔼𝐬∼𝒟\[𝔼𝐚∼πθ\(⋅∣𝐬\)\[Qϕπ\(𝐬,𝐚\)−Vψπ\(𝐬\)\]−βDKL\(πθ\(⋅∣𝐬\)∥πb\(⋅∣𝐬\)\)\]\\displaystyle\\arg\\max\_\{\\theta\}\\mathbb\{E\}\_\{\\mathbf\{s\}\\sim\\mathcal\{D\}\}\\\!\\left\[\\mathbb\{E\}\_\{\\mathbf\{a\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid\\mathbf\{s\}\)\}\\\!\\left\[Q^\{\\pi\}\_\{\\phi\}\(\\mathbf\{s\},\\mathbf\{a\}\)\-V^\{\\pi\}\_\{\\psi\}\(\\mathbf\{s\}\)\\right\]\-\\beta D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi\_\{\\theta\}\(\\cdot\\mid\\mathbf\{s\}\)\\,\\\|\\,\\pi\_\{b\}\(\\cdot\\mid\\mathbf\{s\}\)\\right\)\\right\]\(3\)
Based on the derivation in[29](https://arxiv.org/html/2608.14851#bib.bib1), we can project the closed\-form optimal solutionπ∗​\(𝐚\|𝐬\)∝πb​\(𝐚\|𝐬\)​exp⁡\(1β​\(Qϕπ​\(𝐬,𝐚\)−Vψπ​\(𝐬\)\)\)\\pi^\{\*\}\(\\mathbf\{a\}\|\\mathbf\{s\}\)\\propto\\pi\_\{b\}\(\\mathbf\{a\}\|\\mathbf\{s\}\)\\ \\exp\(\\frac\{1\}\{\\beta\}\(Q^\{\\pi\}\_\{\\phi\}\(\\mathbf\{s\},\\mathbf\{a\}\)\-V^\{\\pi\}\_\{\\psi\}\(\\mathbf\{s\}\)\)\)into the policy space by minimizingDKL\(π∗\(⋅∣𝐬\)∥πθ\(⋅∣𝐬\)\)D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi^\{\*\}\(\\cdot\\mid\\mathbf\{s\}\)\\,\\\|\\,\\pi\_\{\\theta\}\(\\cdot\\mid\\mathbf\{s\}\)\\right\)over the state distribution, which yields the weighted maximum likelihood objective shown in Equation[4](https://arxiv.org/html/2608.14851#S4.E4)\.

Lπ​\(θ\)=𝔼𝒟​\[−log⁡πθ​\(𝐚\|𝐬\)​exp⁡\(1β​\(Qϕπ​\(𝐬,𝐚\)−Vψπ​\(𝐬\)\)\)\]\\displaystyle L\_\{\\pi\}\(\\theta\)=\\mathbb\{E\}\_\{\\mathcal\{D\}\}\[\-\\log\\pi\_\{\\theta\}\(\\mathbf\{a\}\|\\mathbf\{s\}\)\\exp\(\\frac\{1\}\{\\beta\}\(Q^\{\\pi\}\_\{\\phi\}\(\\mathbf\{s\},\\mathbf\{a\}\)\-V^\{\\pi\}\_\{\\psi\}\(\\mathbf\{s\}\)\)\)\]\(4\)
Mirroring[18](https://arxiv.org/html/2608.14851#bib.bib3), we train the value network and action\-value network for advantage estimation purely using transitions in the dataset𝒟\\mathcal\{D\}\. To train the value network, we leverage expectile regression, as shown in Equation[5](https://arxiv.org/html/2608.14851#S4.E5), and to train the action\-value network, we use the objective shown in Equation[6](https://arxiv.org/html/2608.14851#S4.E6)\.

LV​\(ψ\)=𝔼𝒟​\[L2τ​\(Qϕπ​\(𝐬,𝐚\)−Vψπ​\(𝐬\)\)\]\\displaystyle L\_\{V\}\(\\psi\)=\\mathbb\{E\}\_\{\\mathcal\{D\}\}\[L\_\{2\}^\{\\tau\}\(Q^\{\\pi\}\_\{\\phi\}\(\\mathbf\{s\},\\mathbf\{a\}\)\-V^\{\\pi\}\_\{\\psi\}\(\\mathbf\{s\}\)\)\]\(5\)LQ​\(ϕ\)=𝔼𝒟​\[\(r⁡\(𝐬,𝐚\)\+γ​Vψπ​\(𝐬′\)−Qϕπ​\(𝐬,𝐚\)\)2\]\\displaystyle L\_\{Q\}\(\\phi\)=\\mathbb\{E\}\_\{\\mathcal\{D\}\}\[\(r\(\\mathbf\{s\},\\mathbf\{a\}\)\+\\gamma V^\{\\pi\}\_\{\\psi\}\(\\mathbf\{s\}^\{\\prime\}\)\-Q^\{\\pi\}\_\{\\phi\}\(\\mathbf\{s\},\\mathbf\{a\}\)\)^\{2\}\]\(6\)

### 4\.3Architecture for Chess Puzzle Recommendation

Figure 2:Architecture and training procedure of the chess puzzle recommendation policy, leveraging user features and puzzle features to optimize a policy\-based lossLπL\_\{\\pi\}and a value function viaLVL\_\{V\}\.Since we formulate the recommendation problem as an offline policy learning problem, we use a causal transformer architecture as our decision making policy, denoted byTθTT\_\{\\theta\_\{T\}\}\. We choose this architecture because it is widely used for student knowledge tracing and modeling in education\([31](https://arxiv.org/html/2608.14851#bib.bib40);[6](https://arxiv.org/html/2608.14851#bib.bib42);[36](https://arxiv.org/html/2608.14851#bib.bib41)\)\. At each sequence element \(i\.e\., time steptt\), the transformer takes as input the concatenation of a user representationutu\_\{t\}and a puzzle representationptp\_\{t\}, each constructed by applying a neural network to the corresponding user and puzzle features\. The outputs at each sequence element are \(a\) a prediction of the next puzzle,π⁡\(𝐚∣𝐬\)\\pi\(\\mathbf\{a\}\\mid\\mathbf\{s\}\), and \(b\) a value function estimate,V⁡\(s\)V\(s\)\.

During training, we optimizeLπL\_\{\\pi\},LQL\_\{Q\},LVL\_\{V\}on predictions at each sequence element, with training sequences sampled uniformly from the chess puzzle interaction data\. Importantly, we apply teacher forcing during the forward pass of training, which is standard in training autoregressive decoder\-only transformers\.

#### User Embedder

To construct a latent representation of the user, we embed user\-specific featuresfuf\_\{u\}, i\.e\., the user Elo and user correctness at timett, using a small, two layer multi\-layer perceptron \(MLP\), denoted asUθUU\_\{\\theta\_\{U\}\}\. We normalize the output user embeddingut=UθU​\(fu\)u\_\{t\}=U\_\{\\theta\_\{U\}\}\(f\_\{u\}\)to unit norm \(e\.g\., L2 normalization\)\.

#### Puzzle Embedder

To construct a latent representation of the puzzle, we embed puzzle\-specific featuresfpf\_\{p\}using an MLP and convolutional neural network \(CNN\), denoted asPθPP\_\{\\theta\_\{P\}\}\. Specifically, we use a CNN to encode the board position associated with the puzzle, alongside a learned embedding for each individual puzzle and its first move\. Similarly to the user embeddings, we normalize the puzzle embeddingpt=PθP​\(fp\)p\_\{t\}=P\_\{\\theta\_\{P\}\}\(f\_\{p\}\)to unit norm \(e\.g\., L2 normalization\)\.

#### Practical Considerations for Policy Learning

Given a large discrete action space𝒜\\mathcal\{A\}\(i\.e\., approximately half a million puzzles\), computing and normalizing the policy probabilitiesπθ​\(a∣s\)\\pi\_\{\\theta\}\(a\\mid s\)across the entire action space is intractable during training\. We therefore parameterize the policy using an exponentiated inner product between the transformer’s output and each candidate puzzle embedding\. Letht=TθT\(𝐩t−T:t,𝐮t−T:t\)h\_\{t\}=T\_\{\\theta\_\{T\}\}\(\\mathbf\{p\}\_\{t\-T:t\},\\mathbf\{u\}\_\{t\-T:t\}\)denote the transformer’s output for the history through timett, and letpa=PθP​\(fa\)p\_\{a\}=P\_\{\\theta\_\{P\}\}\(f\_\{a\}\)denote the embedding of candidate puzzleaa\. Equation[7](https://arxiv.org/html/2608.14851#S4.E7)defines the resulting temperature\-weighted softmax policy\.

πθ​\(a∣st\)=exp⁡\(λ​ht⊤​pa\)∑a′∈𝒜exp⁡\(λ​ht⊤​pa′\)\\displaystyle\\pi\_\{\\theta\}\(a\\mid s\_\{t\}\)=\\frac\{\\exp\\\!\\left\(\\lambda h\_\{t\}^\{\\top\}p\_\{a\}\\right\)\}\{\\sum\_\{a^\{\\prime\}\\in\\mathcal\{A\}\}\\exp\\\!\\left\(\\lambda h\_\{t\}^\{\\top\}p\_\{a^\{\\prime\}\}\\right\)\}\(7\)Although puzzle embeddings can be precomputed, evaluating the denominator in Equation[7](https://arxiv.org/html/2608.14851#S4.E7)over the entire action space remains expensive during training\. We therefore approximate it using the puzzle embeddingsP⁡\(B\)P\(B\)in minibatchBBas candidate negatives\. For the observed next puzzleat\+1a\_\{t\+1\}with embeddingpt\+1p\_\{t\+1\}, the approximation is

π^θ​\(at\+1∣st\)=exp⁡\(λ​ht⊤​pt\+1\)∑pn∈P⁡\(B\)exp⁡\(λ​ht⊤​pn\)\\displaystyle\\hat\{\\pi\}\_\{\\theta\}\(a\_\{t\+1\}\\mid s\_\{t\}\)=\\frac\{\\exp\\\!\\left\(\\lambda h\_\{t\}^\{\\top\}p\_\{t\+1\}\\right\)\}\{\\sum\_\{p\_\{n\}\\in P\(B\)\}\\exp\\\!\\left\(\\lambda h\_\{t\}^\{\\top\}p\_\{n\}\\right\)\}\(8\)Afterwards, we apply the lossesLπL\_\{\\pi\},LQL\_\{Q\}andLVL\_\{V\}to optimize the policy with respect to the transformer parameters and user and puzzle embedders\. We summarize the entirety of the training algorithm in Algorithm[1](https://arxiv.org/html/2608.14851#alg1)and as depicted in Figure[2](https://arxiv.org/html/2608.14851#S4.F2)\.

Algorithm 1Offline training algorithm for chess puzzle recommendationInput:

𝒟,TθT,UθU,PθP\\mathcal\{D\},\\;T\_\{\\theta\_\{T\}\},\\;U\_\{\\theta\_\{U\}\},\\;P\_\{\\theta\_\{P\}\}
foreach minibatch

B⊂𝒟B\\subset\\mathcal\{D\}do

fp,fu←Bf\_\{p\},f\_\{u\}\\leftarrow B,

pt←PθP​\(fp\),ut←UθU​\(fu\)p\_\{t\}\\leftarrow P\_\{\\theta\_\{P\}\}\(f\_\{p\}\),u\_\{t\}\\leftarrow U\_\{\\theta\_\{U\}\}\(f\_\{u\}\)⊳\\trianglerightpuzzle and user features

π^θ\(𝐚t\+1∣𝐬t\)←exp\(λTθT\(𝐩t−T:t,𝐮t−T:t\)⊤pt\+1\)∑pn∈P⁡\(B\)exp\(λTθT\(𝐩t−T:t,𝐮t−T:t\)⊤pn\)\\hat\{\\pi\}\_\{\\theta\}\(\\mathbf\{a\}\_\{t\+1\}\\mid\\mathbf\{s\}\_\{t\}\)\\leftarrow\\frac\{\\exp\\\!\\left\(\\lambda\\,T\_\{\\theta\_\{T\}\}\(\\mathbf\{p\}\_\{t\-T:t\},\\mathbf\{u\}\_\{t\-T:t\}\)^\{\\top\}p\_\{t\+1\}\\right\)\}\{\\sum\_\{p\_\{n\}\\in P\(B\)\}\\exp\\\!\\left\(\\lambda\\,T\_\{\\theta\_\{T\}\}\(\\mathbf\{p\}\_\{t\-T:t\},\\mathbf\{u\}\_\{t\-T:t\}\)^\{\\top\}p\_\{n\}\\right\)\}⊳\\trianglerightin\-batch approximation

LV​\(ψ\)←𝔼𝒟​\[L2τ​\(Qϕπ​\(𝐬,𝐚\)−Vψπ​\(𝐬\)\)\]L\_\{V\}\(\\psi\)\\leftarrow\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\\!\\left\[L\_\{2\}^\{\\tau\}\\\!\\left\(Q^\{\\pi\}\_\{\\phi\}\(\\mathbf\{s\},\\mathbf\{a\}\)\-V^\{\\pi\}\_\{\\psi\}\(\\mathbf\{s\}\)\\right\)\\right\]

LQ​\(ϕ\)←𝔼𝒟​\[\(r⁡\(𝐬,𝐚\)\+γ​Vψπ​\(𝐬′\)−Qϕπ​\(𝐬,𝐚\)\)2\]L\_\{Q\}\(\\phi\)\\leftarrow\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\\!\\left\[\\left\(r\(\\mathbf\{s\},\\mathbf\{a\}\)\+\\gamma V^\{\\pi\}\_\{\\psi\}\(\\mathbf\{s\}^\{\\prime\}\)\-Q^\{\\pi\}\_\{\\phi\}\(\\mathbf\{s\},\\mathbf\{a\}\)\\right\)^\{2\}\\right\]

Lπ​\(θ\)←𝔼𝒟​\[−log⁡π^θ​\(𝐚∣𝐬\)​exp⁡\(1β​\(Qϕπ​\(𝐬,𝐚\)−Vψπ​\(𝐬\)\)\)\]L\_\{\\pi\}\(\\theta\)\\leftarrow\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\\!\\left\[\-\\log\\hat\{\\pi\}\_\{\\theta\}\(\\mathbf\{a\}\\mid\\mathbf\{s\}\)\\exp\\\!\\left\(\\frac\{1\}\{\\beta\}\\left\(Q^\{\\pi\}\_\{\\phi\}\(\\mathbf\{s\},\\mathbf\{a\}\)\-V^\{\\pi\}\_\{\\psi\}\(\\mathbf\{s\}\)\\right\)\\right\)\\right\]

\(θT,θU,θP,ϕ,ψ\)←\(θT,θU,θP,ϕ,ψ\)−η∇\(LV\+LQ\+Lπ\)\(\\theta\_\{T\},\\theta\_\{U\},\\theta\_\{P\},\\phi,\\psi\)\\leftarrow\(\\theta\_\{T\},\\theta\_\{U\},\\theta\_\{P\},\\phi,\\psi\)\-\\eta\\nabla\(L\_\{V\}\+L\_\{Q\}\+L\_\{\\pi\}\)

endfor

### 4\.4Deriving a Reward Function

During evaluation, we compute puzzle embeddings once for the entire action space in order to obtain the policy’s probability distribution over puzzles\. This is an upfront cost rather than a per\-query one: the same puzzle embeddings are reused across all user sequences and do not need to be recomputed for each new prediction\.

We construct a scalar reward function that attempts to quantify the learning benefit in terms of the user’s correctness on the puzzle and relative puzzle difficulty\. Trivially, a correct response on a challenging puzzle is ideal, and it highlights progression in terms of learning, warranting a large reward\. In any cases where the user answers incorrectly, it demonstrates no verifiable evidence of learning, so we assign no reward in these cases\. Otherwise, in cases of partial correctness, we assign rewards based on the proportion of correct moves and relative difficulty of the puzzle\.

Figure 3:Reward functionr⁡\(s,a\)r\(s,a\)for varying levels of correctness and Elo differentials between puzzle and user, with reward equivalences shown between balances of correctness and relative difficulty\.Given the number of total movesNN, number of correct moves \(from the user\)cc, and the Elo of the puzzle and user, we assign the reward based on the difficulty\-weighted correctness, with the difficulty weightα\\alphabeing a hyperparameter\.

r⁡\(s,a\)=cN​exp⁡\(α⋅\(Elopuzzle−Elouser\)\)\\displaystyle r\(s,a\)=\\frac\{c\}\{N\}\\exp\(\\alpha\\cdot\(\\mathrm\{Elo\}\_\{\\mathrm\{puzzle\}\}\-\\mathrm\{Elo\}\_\{\\mathrm\{user\}\}\)\)\(9\)
We provide a visualization of the reward function across different proportions of correct movesc/Nc/Nin Figure[3](https://arxiv.org/html/2608.14851#S4.F3)forα=0\.002\\alpha=0\.002, which is the chosen hyperparameter due to its reasonable balance between partial correctness and difficulty\. Specifically, based on internal testing,α=0\.002\\alpha=0\.002provides a reasonable level of reward equivalence between partial correctness levels \(e\.g\., halfway correct\) on challenging puzzles and complete correctness on easier puzzles \(e\.g\., puzzles rated 200–400 Elo points lower\)\. These equivalencies are displayed via dotted lines, where the color corresponds to the maximal reward achieved at a particular correctness level, and the intersection with other lines indicates the Elo delta at the reward equivalence\.

## 5Evaluation

While it would be ideal to run a randomized controlled trial experiment with the learned policy, there are many practical challenges\. It is hard to recruit enough players to participate in the experiment, and it may take a significant time to be able to measure any difference reliably\. We instead use a mixture of offline policy evaluation estimators and human evaluation to understand whether the learned policy could have meaningful impact on learning\.

### 5\.1Evaluation Using Importance Sampling

To evaluate our trained policy with respect to the behavior policy, we can leverage importance sampling, using the known probability distributions of the behavior policy \(e\.g\., as deployed by Chess\.com\) and our trained policy\. Importantly, traditional importance sampling with sequences of size 256 \(or even a fraction of that\) is impractical due to the significant induced variance in the estimator over large sequences\. Instead, we employ a one\-step approximation \(e\.g\., removing past weights\) across 32 steps, which significantly reduces variance at the cost of being biased\([3](https://arxiv.org/html/2608.14851#bib.bib5)\)\. We refer to this as one\-step importance sampling\. To further reduce variance, we clip the importance weight for each stepπθ​\(𝐚\|𝐬\)/πb​\(𝐚\|𝐬\)\\pi\_\{\\theta\}\(\\mathbf\{a\}\|\\mathbf\{s\}\)/\\pi\_\{b\}\(\\mathbf\{a\}\|\\mathbf\{s\}\)betweenϵs=0\.1\\epsilon\_\{s\}=0\.1and1ϵs=10\.0\\frac\{1\}\{\\epsilon\_\{s\}\}=10\.0\.

To quantify our improvements over the behavior policy for different users, we break down our results into different Elo groups\. Additionally, we examine groups of stagnant growth and different growth groups, similarly to Figure[1](https://arxiv.org/html/2608.14851#S3.F1); for this analysis, we use a simple definition of stagnant growth, which is growth of no more than 50 Elo points across the sequence\. Importantly, this allows for examination of our improvements on different types of users and learning patterns, e\.g\., improvements on stagnant or new users may be more critical to retention on the Chess\.com platform\.

### 5\.2Expert Evaluation

#### Annotation Rubrics

To annotate the quality of chess puzzles, we designed a detailed rubric with two chess experts involved in the project \(USCF rating around 1900 and 2400\)\. The rubric covers different characteristics of a puzzle, from whether it tests the player’s ability to perform in\-depth calculations of sequential moves or it focuses on testing the ability to recognize iconic patterns\. In addition, the experts are also asked to judge if the puzzle is appropriate for a player of a given rating, whether it is a high\-quality puzzle, and if it is fun to play\. The rubric took multiple iterations with internal testing to verify its robustness and is described in full in Appendix[A\.1](https://arxiv.org/html/2608.14851#A1.SS1)\.

Figure 4:Qualitative rating categories designed in collaboration with chess expert consultants\.We recruited 8 chess experts to annotate 30 puzzles\. The annotators were eight strong chess players, 5 of whom hold official chess titles\. In total, there are 2 USCF Experts, 1 USCF National Master, 1 FIDE Master, 1 FIDE International Master, and 2 FIDE Grandmasters\. The chess annotators’ USCF Elo ratings are between 1990 and 2576\. For a rough \(not directly comparable\) point of reference, the chess legend Magnus Carlsen has a FIDE rating of 2832, and the 100th highest\-rated player has a FIDE rating of 2634 as of Spring 2025\.

The puzzles are randomly sampled from abucketed uniform policybased on the player’s rating range, and we have 6 players with different Elo scores\. We chose 12 puzzles aspreference learningpuzzles that are annotated by all experts\. This means that out of the 30 puzzles each expert annotated, only 18 puzzles are unique between them\. The annotation was performed through an anonymized spreadsheet\. Each annotator was asked to spend approximately 1\.5 hours on the annotation and was later compensated with a gift card\.

#### Scaling Annotations with LLMs

Judging whether a chess puzzle is high quality or not requires annotators with significant expertise, who are distinctly different from standard crowd workers\. This also means we are not able to perform massive annotations that can take up to hundreds of hours\. However, sometimes, two policies that learn to recommend puzzles can behave very similarly and, therefore, require a lot of annotations to understand if they are statistically significantly different\. To help with scaling up the annotation effort, inspired by[42](https://arxiv.org/html/2608.14851#bib.bib13);[11](https://arxiv.org/html/2608.14851#bib.bib14), we use the expert annotations to calibrate a large language model \(LLM\) that can imitate an expert chess annotator’s judgments\. LLMs seem to be able to understand FEN strings and can play chess with some deliberation\([40](https://arxiv.org/html/2608.14851#bib.bib35);[9](https://arxiv.org/html/2608.14851#bib.bib37)\)\. Without making sweeping generalizations, we can leverage LLMs to provide preliminary understanding of whether the chess puzzles chosen by two systems are different\.

With example expert annotations, we use DSPy\([14](https://arxiv.org/html/2608.14851#bib.bib12)\)to create our LLM chess puzzle judge\. DSPy is an LLM library that allows us to specify a set of initial prompts, training data, and a metric\. It then automatically selects the best expert examples to include in the LLM prompt that can maximize the metric\. Internally, DSPy performs cross\-validation with a random search to find the best subset of expert annotations\. We learn 8 LLM annotators using the data from 8 chess annotators\. We separately learn 8 annotators because each annotator’s preference might be different, and imitating the behavior of 8 individuals seems more difficult than imitating the behavior of one individual\. For each annotator, we use the 12 sharedpreference learningpuzzle annotations described above, splitting them evenly into 6 training and 6 validation examples for DSPy prompt optimization\.

#### Prompt Design

We use a system message and the DSPy Signature module to design the prompt\. The system message specifies the persona of the annotator using their USCF Elo rating and title\. The DSPy signature contains three input fields—the annotation guidelines, the target user’s Chess\.com puzzle Elo, and the puzzle board—and one output field containing the six category scores\. Appendix[A\.2](https://arxiv.org/html/2608.14851#A1.SS2)provides a complete description of these fields\. For puzzle board representation, we experimented with a FEN string and a grid\-like representation of the entire board\. We found the grid\-like representation to work better\. Along with the board position, we also describe the first move of the puzzle in text\.

## 6Experimental Results

#### Training

To train our chess puzzle recommendation policy, the dataset is partitioned randomly by users, with 90% used for training and 10% for evaluation\. During training, each batch consists of 16 examples of users’ puzzle histories, with a sampled sequence length of 256 to accommodate computational constraints\. We use a discount rate of 0\.99\. During training, to implement Equation[8](https://arxiv.org/html/2608.14851#S4.E8), we use 65K puzzles as the in\-batch negative examples to estimate our policy’s probabilities\. Due to the large dataset size, we train the policy for 25K steps, alongside the learned action\-value and value functions\.

In this section, we examine the performance of our learned policy in two different ways\. We use an offline policy evaluation method to provide an objective comparison between our policy’s performance and the behavior policy’s performance\. Then, we use our automated annotation pipeline to provide another set of ratings\.

### 6\.1Policy Evaluation

To demonstrate that our policy performs well qualitatively and quantitatively relative to the behavior policy, we perform evaluations based on importance sampling and visualize and summarize the policy recommendations\. As a preliminary qualitative evaluation, we examine the distribution of puzzle Elos recommended by our policy with respect to the puzzle chosen by Chess\.com’s current policy in Figure[5](https://arxiv.org/html/2608.14851#S6.F5)\. As a preliminary sanity check, our policy consistently recommends puzzles in the rough ballpark of the chosen puzzle and the user Elos, though our policy tends to recommend harder puzzles on average, especially for users with higher Elo\. An interesting trend is that the recommendations typically underestimate the user Elo, which reflects the behavior of the behavior policy as well\.

Figure 5:Distribution of trained policy’s top recommended puzzle versus the puzzle Elo of the chosen puzzle \(left\) and the user Elo \(right\)\.Table 1:Returns obtained by trained policyπθ\\pi\_\{\\theta\}and the Chess\.com policyπb\\pi\_\{b\}through one\-step importance sampling\. We show 95% confidence intervals with±\\pmand boldfaced numbers are statistically significant with Student’s t\-test\.In Table[1](https://arxiv.org/html/2608.14851#S6.T1), we summarize the results of the one\-step importance sampling approaches compared to the behavior policy for different Elo buckets and growth groups\. Across lower Elo buckets \(between 100 and 1000\), we show statistically significant and consistent improvement compared to the behavior policy\. However, notably, as the Elo increases, our margins of improvement decrease, where our policy is neutral with respect to the behavior policy\.

Qualitatively, this is sensible because optimal puzzle selection is more critical to learning at earlier stages, whereas missteps in puzzle recommendation may not lead to large outcome differences for an experienced chess player\. Additionally, an important note is that the vast majority of the user base \(and reflected in the training and evaluation data\) is concentrated at lower Elo levels, with over 80% of users under 1500 Elo, where our improvement is roughly 110\.1% over the existing system\.

While the return is consistently lower for the stagnant group \(as is expected\), the relative performance improvement from the trained policy is approximately similar across growth groups and stagnant groups\. That said, for the smallest Elo bucket,πθ\\pi\_\{\\theta\}performs better relatively on stagnant group users, whereas for all others,πθ\\pi\_\{\\theta\}performs relatively better on growth group users\.

### 6\.2Expert Evaluation

We provide a qualitative analysis using the expert\-designed rubrics and with the help of an LLM to understand whether there is any noticeable difference between the type of puzzles recommended by our learned policy and the original Chess\.com puzzle serving system\. We randomly sample 200 users from the holdout evaluation dataset\. To evaluate the original Chess\.com policyπb\\pi\_\{b\}, we directly use what was actually recommended to the user at that time\. For the trained modelπθ\\pi\_\{\\theta\}, we sample the top\-2 puzzles that have the highest probabilities of being recommended\.

Table 2:Comparison of methods for qualitative expert\-designed metrics\. Metrics are converted to 100 from the original range to account for the difference in scale\. Values are averaged across all annotated puzzles\. We perform Student’s t\-test and use∗to markp<0\.05p<0\.05, and∗∗forp<0\.01p<0\.01\.Our trained model recommends puzzles rated as slightly more enjoyable and slightly harder for the target player\. The puzzles also receive somewhat higher calculation and pattern\-recognition scores\. Because these evaluations are generated by LLM judges calibrated on a small set of expert annotations, we treat the findings as preliminary\.

## 7Conclusion and Future Work

Our study leverages 1\.5 billion puzzle\-solving histories to learn the pedagogical value of chess puzzles and develop an automated puzzle selection system using offline reinforcement learning\. Our offline policy evaluation and LLM\-based qualitative analysis both suggest differences between the learned policy and Chess\.com’s existing policy\. However, direct human evaluations are still needed to fully validate our system\. In the future, we hope to collaborate with our data providers to demonstrate the actual impact on chess learners by using our system to filter out low\-quality chess puzzles and to investigate whether our approach has applications in other domains, such as math, language learning, or coding\.

## Acknowledgment

During the period when this research was conducted, AN was supported in part by a Stanford HAI Hoffman\-Yee grant and an NSF \#2112926 grant\. NT was funded by a grant from FAR\.AI\.

Paul Terwilliger was our main collaborator from Chess\.com, where he served as the director of AI at the time\. Paul provided us with the dataset and participated in multiple research meetings to discuss potential uses of the data\. We are extremely grateful for his contribution to the project\.

We also thank the rest of Chess\.com for forming this collaboration with us and allowing us to explore different ways to empower chess learners all over the world\.

Carissa Yip and Nicholas Tomlin served as our in\-house chess experts\. CY and NT co\-designed the annotation schema used to rate each chess puzzle through an iterative process\. CY helped recruit the expert chess annotators listed in Table[3](https://arxiv.org/html/2608.14851#Sx1.T3)\. We are grateful to the annotators for volunteering approximately 1\.5 hours to complete a set of annotations on our chess puzzles\. Each annotator was compensated with a Coupa Cafe gift card or an Amazon gift card\.

Table 3:Volunteer chess annotators who completed puzzle annotations, with their USCF profile \(linked via their USCF member ID\) and title\.
## References

- Bassenet al\.\(2020\)J\. Bassen, B\. Balaji, M\. Schaarschmidt, C\. Thille, J\. Painter, D\. Zimmaro, A\. Games, E\. Fast, and J\. C\. MitchellReinforcement learning for the adaptive scheduling of educational activities\.InProceedings of the 2020 CHI Conference on Human Factors in Computing Systems,pp\. 1–12\.Cited by:[§1](https://arxiv.org/html/2608.14851#S1.p4.1)\.
- Campbellet al\.\(2002\)M\. Campbell, A\. J\. Hoane Jr, and F\. HsuDeep blue\.Artificial intelligence134\(1\-2\),pp\. 57–83\.Cited by:[§2](https://arxiv.org/html/2608.14851#S2.p1.1)\.
- Chenet al\.\(2019\)M\. Chen, A\. Beutel, P\. Covington, S\. Jain, F\. Belletti, and E\. H\. ChiTop\-k off\-policy correction for a reinforce recommender system\.InProceedings of the Twelfth ACM International Conference on Web Search and Data Mining,pp\. 456–464\.Cited by:[§5\.1](https://arxiv.org/html/2608.14851#S5.SS1.p1.1)\.
- Chess\.com \(2023\)Chess\.comHow we built a puzzle database with half a million puzzles\(Website\)Chess\.com\.External Links:[Link](https://www.chess.com/blog/CHESScom/how-we-built-a-puzzle-database-with-half-a-million-puzzles)Cited by:[§1](https://arxiv.org/html/2608.14851#S1.p3.1)\.
- Chiet al\.\(2009\)M\. Chi, P\. Jordan, K\. Vanlehn, and D\. LitmanTo elicit or to tell: does it matter?\.InArtificial Intelligence in Education,pp\. 197–204\.Cited by:[§2](https://arxiv.org/html/2608.14851#S2.p2.1)\.
- Choiet al\.\(2020\)Y\. Choi, Y\. Lee, J\. Cho, J\. Baek, B\. Kim, Y\. Cha, D\. Shin, C\. Bae, and J\. HeoTowards an appropriate query, key, and value computation for knowledge tracing\.InProceedings of the Seventh ACM Conference on Learning@ Scale,pp\. 341–344\.Cited by:[§4\.3](https://arxiv.org/html/2608.14851#S4.SS3.p1.1)\.
- Ericssonet al\.\(1993\)K\. A\. Ericsson, R\. T\. Krampe, and C\. Tesch\-RömerThe role of deliberate practice in the acquisition of expert performance\.\.Psychological review100\(3\),pp\. 363\.Cited by:[§1](https://arxiv.org/html/2608.14851#S1.p1.1)\.
- Ericsson \(2008\)K\. A\. EricssonDeliberate practice and acquisition of expert performance: a general overview\.Academic emergency medicine15\(11\),pp\. 988–994\.Cited by:[§1](https://arxiv.org/html/2608.14851#S1.p1.1)\.
- Fenget al\.\(2023\)X\. Feng, Y\. Luo, Z\. Wang, H\. Tang, M\. Yang, K\. Shao, D\. Mguni, Y\. Du, and J\. WangChessgpt: bridging policy learning and language modeling\.Advances in Neural Information Processing Systems36,pp\. 7216–7262\.Cited by:[§5\.2](https://arxiv.org/html/2608.14851#S5.SS2.SSS0.Px2.p1.1)\.
- Hamadeet al\.\(2024\)K\. Hamade, R\. McIlroy\-Young, S\. Sen, J\. Kleinberg, and A\. AndersonDesigning skill\-compatible ai: methodologies and frameworks in chess\.arXiv preprint arXiv:2405\.05066\.Cited by:[§2](https://arxiv.org/html/2608.14851#S2.p1.1)\.
- Heet al\.\(2023\)X\. He, Z\. Lin, Y\. Gong, A\. Jin, H\. Zhang, C\. Lin, J\. Jiao, S\. M\. Yiu, N\. Duan, W\. Chen,et al\.Annollm: making large language models to be better crowdsourced annotators\.arXiv preprint arXiv:2303\.16854\.Cited by:[§5\.2](https://arxiv.org/html/2608.14851#S5.SS2.SSS0.Px2.p1.1)\.
- Henkin \(2021\)V\. Henkin1000 checkmate combinations\.Batsford\.External Links:ISBN 1849947252,[Link](https://www.amazon.com/1000-Checkmate-Combinations-Victor-Henkin/dp/1849947252)Cited by:[§1](https://arxiv.org/html/2608.14851#S1.p2.1)\.
- Jaqueset al\.\(2017\)N\. Jaques, S\. Gu, D\. Bahdanau, J\. M\. Hernández\-Lobato, R\. E\. Turner, and D\. EckSequence tutor: conservative fine\-tuning of sequence generation models with kl\-control\.InInternational Conference on Machine Learning,pp\. 1645–1654\.Cited by:[§4\.1](https://arxiv.org/html/2608.14851#S4.SS1.p4.1)\.
- Khattabet al\.\(2023\)O\. Khattab, A\. Singhvi, P\. Maheshwari, Z\. Zhang, K\. Santhanam, S\. Vardhamanan, S\. Haq, A\. Sharma, T\. T\. Joshi, H\. Moazam,et al\.Dspy: compiling declarative language model calls into self\-improving pipelines\.arXiv preprint arXiv:2310\.03714\.Cited by:[§5\.2](https://arxiv.org/html/2608.14851#S5.SS2.SSS0.Px2.p2.1)\.
- Koedingeret al\.\(2023\)K\. R\. Koedinger, P\. F\. Carvalho, R\. Liu, and E\. A\. McLaughlinAn astonishing regularity in student learning rate\.Proceedings of the National Academy of Sciences120\(13\),pp\. e2221311120\.Cited by:[§1](https://arxiv.org/html/2608.14851#S1.p1.1)\.
- Komorowskiet al\.\(2018\)M\. Komorowski, L\. A\. Celi, O\. Badawi, A\. C\. Gordon, and A\. A\. FaisalThe artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care\.Nature medicine24\(11\),pp\. 1716–1720\.Cited by:[§1](https://arxiv.org/html/2608.14851#S1.p4.1)\.
- Konda and Tsitsiklis \(1999\)V\. Konda and J\. TsitsiklisActor\-critic algorithms\.Advances in neural information processing systems12\.Cited by:[§4\.1](https://arxiv.org/html/2608.14851#S4.SS1.p3.1)\.
- Kostrikovet al\.\(2021\)I\. Kostrikov, A\. Nair, and S\. LevineOffline reinforcement learning with implicit q\-learning\.arXiv preprint arXiv:2110\.06169\.Cited by:[§1](https://arxiv.org/html/2608.14851#S1.p5.1),[§4\.2](https://arxiv.org/html/2608.14851#S4.SS2.p1.1),[§4\.2](https://arxiv.org/html/2608.14851#S4.SS2.p3.1)\.
- Kumaret al\.\(2022\)A\. Kumar, R\. Agarwal, X\. Geng, G\. Tucker, and S\. LevineOffline q\-learning on diverse multi\-task data both scales and generalizes\.arXiv preprint arXiv:2211\.15144\.Cited by:[§1](https://arxiv.org/html/2608.14851#S1.p4.1),[§2](https://arxiv.org/html/2608.14851#S2.p2.1)\.
- Kumaret al\.\(2021\)A\. Kumar, A\. Singh, S\. Tian, C\. Finn, and S\. LevineA workflow for offline model\-free robotic reinforcement learning\.arXiv preprint arXiv:2109\.10813\.Cited by:[§1](https://arxiv.org/html/2608.14851#S1.p4.1)\.
- Kumaret al\.\(2020\)A\. Kumar, A\. Zhou, G\. Tucker, and S\. LevineConservative q\-learning for offline reinforcement learning\.Advances in Neural Information Processing Systems33,pp\. 1179–1191\.Cited by:[§1](https://arxiv.org/html/2608.14851#S1.p4.1)\.
- Liuet al\.\(2022\)E\. Liu, M\. Stephan, A\. Nie, C\. Piech, E\. Brunskill, and C\. FinnGiving feedback on interactive student programs with meta\-exploration\.Advances in Neural Information Processing Systems35,pp\. 36282–36294\.Cited by:[§2](https://arxiv.org/html/2608.14851#S2.p2.1)\.
- Luet al\.\(2022\)C\. Lu, P\. J\. Ball, T\. G\. Rudner, J\. Parker\-Holder, M\. A\. Osborne, and Y\. W\. TehChallenges and opportunities in offline reinforcement learning from visual observations\.arXiv preprint arXiv:2206\.04779\.Cited by:[§1](https://arxiv.org/html/2608.14851#S1.p4.1)\.
- Luoet al\.\(2024\)Z\. Luo, Y\. Pan, P\. Watkinson, and T\. ZhuPosition: reinforcement learning in dynamic treatment regimes needs critical reexamination\.Journal of Machine Learning Research\.Cited by:[§1](https://arxiv.org/html/2608.14851#S1.p4.1)\.
- Mandelet al\.\(2014\)T\. Mandel, Y\. Liu, S\. Levine, E\. Brunskill, and Z\. PopovicOffline policy evaluation across representations with applications to educational games\.\.InAAMAS,Vol\.1077\.Cited by:[§1](https://arxiv.org/html/2608.14851#S1.p4.1),[§2](https://arxiv.org/html/2608.14851#S2.p2.1)\.
- McGrathet al\.\(2022\)T\. McGrath, A\. Kapishnikov, N\. Tomašev, A\. Pearce, M\. Wattenberg, D\. Hassabis, B\. Kim, U\. Paquet, and V\. KramnikAcquisition of chess knowledge in alphazero\.Proceedings of the National Academy of Sciences119\(47\),pp\. e2206625119\.Cited by:[§2](https://arxiv.org/html/2608.14851#S2.p1.1)\.
- McIlroy\-Younget al\.\(2022\)R\. McIlroy\-Young, R\. Wang, S\. Sen, J\. Kleinberg, and A\. AndersonLearning models of individual behavior in chess\.InProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,pp\. 1253–1263\.Cited by:[§2](https://arxiv.org/html/2608.14851#S2.p1.1)\.
- McIlroy\-Younget al\.\(2021\)R\. McIlroy\-Young, Y\. Wang, S\. Sen, J\. Kleinberg, and A\. AndersonDetecting individual decision\-making style: exploring behavioral stylometry in chess\.Advances in Neural Information Processing Systems34,pp\. 24482–24497\.Cited by:[§2](https://arxiv.org/html/2608.14851#S2.p1.1)\.
- Nairet al\.\(2020\)A\. Nair, A\. Gupta, M\. Dalal, and S\. LevineAwac: accelerating online reinforcement learning with offline datasets\.arXiv preprint arXiv:2006\.09359\.Cited by:[§1](https://arxiv.org/html/2608.14851#S1.p5.1),[§4\.2](https://arxiv.org/html/2608.14851#S4.SS2.p1.1),[§4\.2](https://arxiv.org/html/2608.14851#S4.SS2.p2.1)\.
- Nieet al\.\(2023\)A\. Nie, A\. Reuel, and E\. BrunskillUnderstanding the impact of reinforcement learning personalization on subgroups of students in math tutoring\.InInternational conference on artificial intelligence in education,pp\. 688–694\.Cited by:[§2](https://arxiv.org/html/2608.14851#S2.p2.1)\.
- Pandey and Karypis \(2019\)S\. Pandey and G\. KarypisA self\-attentive model for knowledge tracing\.arXiv preprint arXiv:1907\.06837\.Cited by:[§4\.3](https://arxiv.org/html/2608.14851#S4.SS3.p1.1)\.
- Rafailovet al\.\(2024\)R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. FinnDirect preference optimization: your language model is secretly a reward model\.Advances in Neural Information Processing Systems36\.Cited by:[§4\.1](https://arxiv.org/html/2608.14851#S4.SS1.p4.1)\.
- Roediger IIIet al\.\(2011\)H\. L\. Roediger III, A\. L\. Putnam, and M\. A\. SmithTen benefits of testing and their applications to educational practice\.Psychology of learning and motivation55,pp\. 1–36\.Cited by:[§1](https://arxiv.org/html/2608.14851#S1.p1.1)\.
- Ruanet al\.\(2024\)S\. Ruan, A\. Nie, W\. Steenbergen, J\. He, J\. Zhang, M\. Guo, Y\. Liu, K\. Dang Nguyen, C\. Y\. Wang, R\. Ying, J\. Landay, and E\. BrunskillReinforcement learning tutor better supported lower performers in a math task\.Machine Learning,pp\. 1–26\.Cited by:[§1](https://arxiv.org/html/2608.14851#S1.p4.1),[§2](https://arxiv.org/html/2608.14851#S2.p2.1)\.
- Schutet al\.\(2023\)L\. Schut, N\. Tomasev, T\. McGrath, D\. Hassabis, U\. Paquet, and B\. KimBridging the human\-ai knowledge gap: concept discovery and transfer in alphazero\.arXiv preprint arXiv:2310\.16410\.Cited by:[§2](https://arxiv.org/html/2608.14851#S2.p1.1)\.
- Shinet al\.\(2020\)D\. Shin, Y\. Shim, H\. Yu, S\. Lee, B\. Kim, and Y\. ChoiSAINT\+: integrating temporal features for ednet correctness prediction\.arXiv preprint arXiv:2010\.12042\.Cited by:[§4\.3](https://arxiv.org/html/2608.14851#S4.SS3.p1.1)\.
- Silveret al\.\(2017\)D\. Silver, T\. Hubert, J\. Schrittwieser, I\. Antonoglou, M\. Lai, A\. Guez, M\. Lanctot, L\. Sifre, D\. Kumaran, T\. Graepel,et al\.Mastering chess and shogi by self\-play with a general reinforcement learning algorithm\.arXiv preprint arXiv:1712\.01815\.Cited by:[§2](https://arxiv.org/html/2608.14851#S2.p1.1)\.
- Silveret al\.\(2018\)D\. Silver, T\. Hubert, J\. Schrittwieser, I\. Antonoglou, M\. Lai, A\. Guez, M\. Lanctot, L\. Sifre, D\. Kumaran, T\. Graepel,et al\.A general reinforcement learning algorithm that masters chess, shogi, and go through self\-play\.Science362\(6419\),pp\. 1140–1144\.Cited by:[§2](https://arxiv.org/html/2608.14851#S2.p1.1)\.
- Sutton and Barto \(1998\)R\. S\. Sutton and A\. G\. BartoReinforcement learning: an introduction\.Vol\.1,MIT press Cambridge\.Cited by:[§4\.1](https://arxiv.org/html/2608.14851#S4.SS1.p4.1)\.
- Toshniwalet al\.\(2022\)S\. Toshniwal, S\. Wiseman, K\. Livescu, and K\. GimpelChess as a testbed for language model state tracking\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.36,pp\. 11385–11393\.Cited by:[§5\.2](https://arxiv.org/html/2608.14851#S5.SS2.SSS0.Px2.p1.1)\.
- Wanget al\.\(2022\)Y\. Wang, M\. Sharma, C\. Xu, S\. Badam, Q\. Sun, L\. Richardson, L\. Chung, E\. H\. Chi, and M\. ChenSurrogate for long\-term user experience in recommender systems\.InProceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining,pp\. 4100–4109\.Cited by:[§1](https://arxiv.org/html/2608.14851#S1.p5.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in Neural Information Processing Systems36,pp\. 46595–46623\.Cited by:[§5\.2](https://arxiv.org/html/2608.14851#S5.SS2.SSS0.Px2.p1.1)\.

## Appendix AAppendix

### A\.1Human Annotation Guidelines

This appendix describes the annotation protocol used to collect the expert ratings discussed in Section[5](https://arxiv.org/html/2608.14851#S5)\. The goal of the annotation task is to measure whether puzzles recommended by our learned policy are of comparable or higher pedagogical quality than those served by Chess\.com’s existing tactics trainer\. Each of the eight titled and expert\-level annotators listed in Table[3](https://arxiv.org/html/2608.14851#Sx1.T3)was given a spreadsheet of puzzle URLs, together with the Chess\.com Elo of the player each puzzle was originally recommended to and that player’s puzzle rating\. Annotators were asked to first attempt to solve each puzzle and record whether they solved it correctly, before rating it along the six criteria below and providing a short written description and justification \(minimum length one sentence\) for their ratings\. Annotators were asked to budget approximately 1\.5 hours to complete the full set of 30 puzzles, inclusive of reading these guidelines, and were compensated with a gift card\.

#### Calculation

For each puzzle, annotators first judged whether it was primarily designed to test calculation, pattern recognition, both, or neither, and rated only the corresponding subcriterion below \(marking the other as not applicable\)\.Calculationis scored on a 1–4 scale and reflects the extent to which a puzzle requires reasoning over long move sequences or lines with a high branching factor; puzzles that can be solved by identifying the single forcing move at each step, without look\-ahead, receive a low score\. Figure[6\(a\)](https://arxiv.org/html/2608.14851#A1.F6.sf1)shows an example rated a 1, in which every move in the solution is the only reasonable continuation available at that step\.

#### Pattern recognition

Pattern recognitionis scored on a 1–3 scale and reflects how unusual or interesting a puzzle’s tactical motif is, relative to common motifs \(e\.g\., a standard smothered mate\); a puzzle earns the highest score either for testing a genuinely unusual pattern or for giving a common pattern an unusual twist\.

#### Informativeness

Scored on a 1–5 scale,informativenessmeasures whether a puzzle’s given solution follows the most challenging and instructive line rather than a secondary, less demanding continuation\. Figure[6\(b\)](https://arxiv.org/html/2608.14851#A1.F6.sf2)shows an example rated a 1, where the puzzle’s solution avoids the most challenging continuation \(which the solver must nonetheless calculate to justify an earlier move\) in favor of an easier follow\-up\. Puzzles whose solutions instead follow the opponent’s best defense, and whose logic remains clear once the solution is revealed, are rated highly\.

![Refer to caption](https://arxiv.org/html/2608.14851v1/images/annotation_calculation_example.png)\(a\)Acalculationscore of 1: for instance, in the below position, Black is forced to take on g6 and play Ng7, which solves the puzzle\.
![Refer to caption](https://arxiv.org/html/2608.14851v1/images/annotation_informativeness_example.png)\(b\)Aninformativenessscore of 1: for instance, in the below position, after we take on h3, the solution follows 2\. Qg2 instead of taking on f5 \(the most challenging line, which the player must have calculated before taking on h3\)\.

Figure 6:Example puzzle positions used to illustrate thecalculation\(Figure[6\(a\)](https://arxiv.org/html/2608.14851#A1.F6.sf1)\) andinformativeness\(Figure[6\(b\)](https://arxiv.org/html/2608.14851#A1.F6.sf2)\) criteria\.
#### Rating\-appropriateness

Scored on a 1–5 scale,rating\-appropriatenesscaptures the annotator’s assessment of a puzzle’s difficulty relative to the rating of the player it was recommended to\. Annotators were asked to judge this criterion from the perspective of a player at the target rating, rather than their own\.

#### Fun

Also scored on a 1–5 scale,funmeasures how enjoyable a puzzle would likely be for a player at the target rating\.

#### Quality

Qualityis a holistic 1–5 score of the puzzle as a whole, independent of difficulty; the preceding criteria are intended to inform this judgment without strictly determining it\. Figure[4](https://arxiv.org/html/2608.14851#S5.F4)summarizes the rating range for each criterion\.

### A\.2LLM Judge Prompt Fields

Each LLM judge receives a system message and a DSPy signature\. The system message identifies the expert annotator being imitated by their chess title, when applicable, and USCF Elo rating, and asks the model to apply the annotation guidelines together with that expert\-level perspective\. The signature contains the following three input fields:

- •Annotation guidelines: the complete rubric for rating calculation, pattern recognition, informativeness, rating\-appropriateness, quality, and fun\.
- •Target\-user puzzle Elo: the target user’s current Chess\.com puzzle Elo, which provides the reference skill level for rating\-appropriateness and fun\.
- •Puzzle board: an ASCII grid representation of the position, generated from the puzzle’s FEN string, followed by a sentence describing the puzzle’s first move and asking the solver to find the best continuation\.

The signature’s output field requests integer scores from 1 to 100 for all six categories in valid JSON format\. DSPy also supplies selected expert\-rated examples when executing the optimized judge\.

### A\.3Annotation Questionnaire

Annotators recorded their ratings in a shared spreadsheet, with one puzzle per row and one column per question\. For each puzzle, the columns asked, verbatim:

- •What is the primary emphasis of this puzzle?\(Calculation / Pattern Recognition / Both / Neither\)
- •Calculation: this puzzle requires the player to think through the entire solution before playing their first move\. \(1–4, or N/A\)
- •Pattern recognition: this puzzle teaches useful or un\-ordinary motifs, e\.g\. the motifs are not banal/overdone, or if they are, the puzzle puts a unique spin on them\. \(1–3, or N/A\)
- •Informativeness: the solution goes down “the right path” to make sure the player understands the point of the puzzle\. \(1–5\)
- •Rating: is this puzzle appropriate for someone rated at the target player’s skill level? \(1–5\)
- •Quality: this is a high\-quality puzzle, regardless of whether it is at the correct Elo level or not\. \(1–5\)
- •Fun: I think this puzzle would be enjoyable and fun to solve by players at the target Elo\. \(1–5\)
- •Additional notes: please write a 1–2 sentence justification of your ratings\.

Table[4](https://arxiv.org/html/2608.14851#A1.T4)reproduces two rows from a completed annotation spreadsheet to illustrate how one annotator answered this questionnaire in practice\.

Table 4:Two example rows from a completed annotation spreadsheet, showing how one annotator answered the questionnaire in Appendix[A\.3](https://arxiv.org/html/2608.14851#A1.SS3)for two puzzles\.
### A\.4Selecting Illustrative Examples

To illustrate how ratings can differ across puzzles, and to qualitatively showcase the difference between puzzles recommended by our system and the original system, we selected a second pair of example puzzles, shown in Table[5](https://arxiv.org/html/2608.14851#A1.T5)\. As with Table[4](https://arxiv.org/html/2608.14851#A1.T4), we restricted our search to the subset of puzzles that all eight annotators independently rated \(the “preference learning” puzzles described in Section[5](https://arxiv.org/html/2608.14851#S5)\)\. Within this subset, we computed each puzzle’s average pattern recognition score \(excluding annotators for whom the criterion did not apply\) and selected one puzzle near the top and one near the bottom of this ranking, from our policy and from Chess\.com’s existing system respectively\.

The first example \(a puzzle recommended by our policy\) received the maximum pattern recognition score from every annotator who rated it: all described the position’s combination of moves as an unusual or non\-obvious way to reach the tactical idea, distinct from a puzzle that merely tests whether the solver knows a standard pattern\.

The second example \(a puzzle served by Chess\.com’s existing system\) had a lower average pattern recognition score, with annotators ranging from “banal/boring pattern” to “interesting and useful” for the same puzzle\. Notably, its holistic quality score remained comparable to the first example’s, since most annotators still considered it a reasonably solid puzzle overall; only the pattern recognition criterion, and the accompanying free\-text notes, surfaced the disagreement over whether its underlying motif was novel or overdone\. This is precisely the gap we selected these examples to illustrate: a puzzle can score well on quality while still being seen by some experts as testing an unremarkable, overdone pattern\.

Table 5:Two example rows from a completed annotation spreadsheet, selected to contrast a puzzle with a unanimously high pattern recognition score \(Example 1\) against a puzzle with a lower, more disputed pattern recognition score \(Example 2\), as discussed in Appendix[A\.4](https://arxiv.org/html/2608.14851#A1.SS4)\.Table[6](https://arxiv.org/html/2608.14851#A1.T6)shows the full pattern recognition and quality distributions across all eight annotators for both puzzles, rather than the mean alone, so that the degree of agreement \(or disagreement\) on each criterion is directly visible\.

Table 6:Pattern recognition and quality scores given by each of the eight annotators to the two puzzles in Table[5](https://arxiv.org/html/2608.14851#A1.T5)\(“Ours” is \#760014 and “Chess\.com” is \#1072600; N/A denotes an annotator for whom the pattern recognition criterion did not apply\), with rows aligned so that the same letter denotes the same annotator across all four columns\. While the two puzzles’ quality scores are similar on average, their pattern recognition scores diverge, with \#1072600 receiving both the single lowest score \(a “banal/boring pattern” rating\) and a wider spread of opinions overall\.