Hallucinations on the Board: Tool-Augmented Evaluation of LLM Chess Commentary

# Tool-Augmented Evaluation of LLM Chess Commentary
Source: [https://arxiv.org/html/2608.04240](https://arxiv.org/html/2608.04240)
## Hallucinations on the Board: Tool\-Augmented Evaluation of LLM Chess Commentary

S\. Ashwin Hebbar Princeton University &Peiyao Sheng Sentient Labs &Sewoong Oh University of Washington &Pramod Viswanath Princeton University

###### Abstract

Superhuman game engines in domains like chess have made expert\-level evaluations easily accessible, yet they communicate what is true without the natural\-language explanations that make such expertise educationally useful to experts and non\-experts alike\. Large language models could, in principle, bridge this gap, but they frequently hallucinate due to limited domain\-specific knowledge, and standard reference\-based or LLM\-as\-a\-judge frameworks cannot reliably detect these errors\. In this work, we present ACT\-Eval, an evaluation framework that decomposes chess commentary into atomic claims and routes them to engine\-supported tools and expert\-annotated gold references to assess factual correctness, conceptual coverage, and move\-quality judgment\. We release a benchmark of 325 position–move pairs spanning pedagogical, tournament, and critical positions, including 125 positions with expert\-verified gold atoms and a five\-class error taxonomy\. Evaluating leading proprietary and open\-weight models, we find that factual hallucinations remain pervasive in chess commentary: GPT\-5\.4 without tools produces incorrect sub\-claims 22\.0% of the time, while smaller open\-weight models exceed 40%\. Although tool augmentation substantially improves factual correctness and move\-quality assessment, coverage of expert strategic and tactical ideas remains limited across all models\. Human calibration shows that ACT\-Eval’s factual judgments fall within the observed range of inter\-human agreement, while its coverage scores correlate strongly with human assessments of strategic completeness\.

Hallucinations on the Board: Tool\-Augmented Evaluation of LLM Chess Commentary

S\. Ashwin HebbarPrinceton UniversityPeiyao ShengSentient LabsSewoong OhUniversity of WashingtonPramod ViswanathPrinceton University

## 1Introduction

Chess has long served as a canonical domain for studying strategic reasoning\. Its appeal lies not only in the combinatorial complexity of the game, but also in the rich explanatory tradition that has developed around it\. For centuries, chess understanding has been communicated through annotated games, books, lectures, broadcasts, and, more recently, online platforms such as YouTube and Twitch\. Strong players make moves intelligible by translating them into plans, threats, tactical motifs, and long\-term tradeoffs\. The enduring popularity of these formats reflects a simple reality: players and spectators do not only want to know which move is best, but why it works\.

Superhuman engines such as StockfishStockfish \([2024](https://arxiv.org/html/2608.04240#bib.bib24)\); Silveret al\.\([2016](https://arxiv.org/html/2608.04240#bib.bib25)\)have transformed this landscape by making expert\-level evaluation broadly accessible\. On one hand, this creates an unprecedented opportunity for democratizing expertise: even a novice can consult an engine to instantly know whether a move is a blunder or a brilliancy\. However, they only provide an objective score, which for beginner and intermediate\-level players may not mean much more than determining who is better; even elite grandmaster preparation involves spending hours interpreting engine play into human explanations and practical plans\. Engines have largely solved the problem of finding strong moves, but not the problem of explaining them\.

![Refer to caption](https://arxiv.org/html/2608.04240v1/x1.png)Figure 1:Comparison between GCC\-Eval and ACT\-Eval on generated chess commentary\. The commentary in this example is fluent, but not faithful to the underlying chess position\. Nevertheless, GCC\-Eval, a naive LLM judge, assigns near\-perfect relevance and completeness scores\. ACT\-Eval instead decomposes the commentary into atomic claims, verifies factual claims using chess tools and engine analysis, and measures coverage against expert reference atoms\. Here, it correctly determines that only one of three factual claims is correct, and none of the key expert ideas are covered\.The rapid development of Large Language Models \(LLMs\) offers a promising direction\. LLMs excel at multi\-step reasoning across many domainsWeiet al\.\([2022](https://arxiv.org/html/2608.04240#bib.bib26)\); OpenAI \([2024](https://arxiv.org/html/2608.04240#bib.bib27)\)and can generate reasoning traces in natural language that humans can understand\. This creates the possibility to bridge the gap between superhuman expertise and educational explanations\. However, substantial challenges remain before LLMs can reliably fulfill this role in domain\-specific reasoning tasks\. Taking chess as an example, even state\-of\-the\-art reasoning LLMs still perform far below expert human players in chess\-specific reasoning, let alone specialized enginesKolasaniet al\.\([2025](https://arxiv.org/html/2608.04240#bib.bib30)\); Wenet al\.\([2025](https://arxiv.org/html/2608.04240#bib.bib23)\)\. Moreover, the spatial structure and long\-horizon temporal complexity of chess amplify hallucination issues: models frequently misidentify piece locations, threats or propose illegal move sequences, a pattern we quantify in Section[6\.1](https://arxiv.org/html/2608.04240#S6.SS1), resulting in commentary that often sounds plausible despite containing factual errors \(Figure[1](https://arxiv.org/html/2608.04240#S1.F1)\)\.

Compounding these generation challenges is a fundamental evaluation problem: there is currently no reliable way to measure how well a language model understands specific domains like chess or how accurately it explains the move\. Existing benchmarks primarily focus on verifiable capabilities such as puzzle\-solving accuracy and Elo\-style playing strengthKolasaniet al\.\([2025](https://arxiv.org/html/2608.04240#bib.bib30)\); Wenet al\.\([2025](https://arxiv.org/html/2608.04240#bib.bib23)\)\. Standard text generation metricsPapineniet al\.\([2002](https://arxiv.org/html/2608.04240#bib.bib15)\); Lin \([2004](https://arxiv.org/html/2608.04240#bib.bib16)\)only measure surface similarity to reference commentary\. Recent LLM\-as\-a\-judge approachesLiuet al\.\([2023](https://arxiv.org/html/2608.04240#bib.bib21)\); Kimet al\.\([2025](https://arxiv.org/html/2608.04240#bib.bib12)\), which use large language models to assess explanations, introduce a circular problem: the judge is subject to the same domain\-specific weaknesses as the model being evaluated\(Szymanskiet al\.,[2025](https://arxiv.org/html/2608.04240#bib.bib22)\), making it an unreliable arbiter of domain\-specific facts\. As Figure[1](https://arxiv.org/html/2608.04240#S1.F1)illustrates, a hallucinated commentary receives 4\.9/5 from GCC\-EvalKimet al\.\([2025](https://arxiv.org/html/2608.04240#bib.bib12)\)but 2 out of 3 main claims contain factual errors\.

In this work, we move toward addressing this gap by proposing an evaluation framework for assessing LLM understanding and interpretability in domain\-specific reasoning tasks, using chess as a testbed\. A key property of chess is that many factual claims such as move legality, attacks, material balance, or board state can be deterministically checked using chess engines and symbolic toolsStockfish \([2024](https://arxiv.org/html/2608.04240#bib.bib24)\); Leela Chess Zero \([2026](https://arxiv.org/html/2608.04240#bib.bib31)\); Fiekas \([2022](https://arxiv.org/html/2608.04240#bib.bib29)\), whereas higher\-level explanations about plans, coordination, or positional ideas cannot be fully reduced to computation alone\. Built on this observation, we proposeACT\-Eval\(AtomiCandTool\-augmented Evaluation\), an atom\-grounded, tool\-augmented framework for evaluating chess commentary\. ACT\-Eval decomposes commentary into atomic claims and uses LLMs to route each claim to deterministic tools and check factual correctness\. In addition, the LLM judge evaluates strategic and explanatory claims to measure coverage through atomic recall against expert\-annotated gold references, capturing whether the explanation identifies the key ideas a strong player would recognize\.

Our contributions are summarized as follows:

- •We introduce ACT\-Eval, an evaluation framework that integrates atomic\-claim decomposition with tool\-augmented verification for chess commentary\. By routing computationally decidable claims to engine and board\-state oracles,ACT\-Eval grounds the judge in deterministic board\-state and engine evidence wherever suitable computational checks are available, reducing reliance on the judge’s parametric chess knowledge\.
- •We construct a benchmark of 325 position\-move pairs, covering pedagogical textbooks, modern tournament and meticulously\-engineered critical positions\. Central to this benchmark is a subset of positions with expert\-verified gold atoms derived from grandmaster\-level human annotations\. They serve as reference for evaluating conceptual coverage – whether generated commentary captures the strategic and tactical insights behind a move – as a complementary dimension to factual correctness\.
- •We evaluate frontier and open\-weight LLMs and show that factual hallucinations remain pervasive in chess commentary, particularly in tactical variations and long move sequences\. We find that models without engine access cannot reliably distinguish sound moves from mistakes\. Tool augmentation substantially improves factual correctness, but atomic recall against expert annotations remains limited\. Human calibration studies further show that ACT\-Eval achieves agreement with expert annotators within the range of inter\-human agreement\.

Using chess as a testbed for domain\-specific explanation, we show that current frontier LLMs fail to translate their strong general reasoning into reliable domain\-specific expertise, these results suggest that analogous evaluation gaps may arise in other specialized domains\. Broadly, ACT\-Eval offers a candidate methodological template for domains in which part of the truth is computationally verifiable and the remainder can be anchored in expert annotation\. We release the framework and benchmark111[https://github\.com/hebbarashwin/act\_eval](https://github.com/hebbarashwin/act_eval)

## 2Related Work

Reliable evaluation of generated text has been approached from three directions: similarity to human referencesPapineniet al\.\([2002](https://arxiv.org/html/2608.04240#bib.bib15)\); Lin \([2004](https://arxiv.org/html/2608.04240#bib.bib16)\); Zhanget al\.\([2019](https://arxiv.org/html/2608.04240#bib.bib17)\), factual verification against a knowledge sourceMinet al\.\([2023b](https://arxiv.org/html/2608.04240#bib.bib42)\); Chernet al\.\([2023](https://arxiv.org/html/2608.04240#bib.bib14)\); Weiet al\.\([2024](https://arxiv.org/html/2608.04240#bib.bib13)\), and LLM\-as\-a\-judge scoringLiuet al\.\([2023](https://arxiv.org/html/2608.04240#bib.bib21)\)\. When it comes to domain\-specific text generation like chess, each falters for a different reason\.Similarity metrics reward fluent text regardless of correctness\(Kryścińskiet al\.,[2020](https://arxiv.org/html/2608.04240#bib.bib18); Falkeet al\.,[2019](https://arxiv.org/html/2608.04240#bib.bib40); Maynezet al\.,[2020](https://arxiv.org/html/2608.04240#bib.bib41)\)\. Factuality methods assume claims can be verified by retrieval, but verifying a chess claim requires computing against the live position rather than retrieving against a knowledge base\.Verifying a single chess claim may require selecting the correct position, simulating a variation, checking legality and attacks, and composing these with engine evaluation — a computation over an evolving board state rather than a lookup, unlike decompose\-and\-verify frameworks built around retrieval\(Minet al\.,[2023b](https://arxiv.org/html/2608.04240#bib.bib42); Chernet al\.,[2023](https://arxiv.org/html/2608.04240#bib.bib14); Weiet al\.,[2024](https://arxiv.org/html/2608.04240#bib.bib13)\)\. LLM judges degrade sharply in domains requiring expert knowledge\(Szymanskiet al\.,[2025](https://arxiv.org/html/2608.04240#bib.bib22)\), and a judge without reliable chess knowledge cannot detect a fluent but false chess claim\.The closest prior work in our setting, GCC\-EvalKimet al\.\([2025](https://arxiv.org/html/2608.04240#bib.bib12)\), augments an LLM judge with reference commentary and engine analysis but inherits both noisy references and uninterpreted engine scores\. ACT\-Eval addresses these limitations by routing factual claims to deterministic chess tools and scoring conceptual coverage against expert\-verified gold atoms; we provide an extended discussion in Appendix[E](https://arxiv.org/html/2608.04240#A5)\.

## 3The ACT\-Eval Framework

![Refer to caption](https://arxiv.org/html/2608.04240v1/x2.png)Figure 2:Overview of the ACT\-Eval pipeline\. Reference commentary is filtered and decomposed into expert\-verified gold atoms, while generated commentary is decomposed into candidate atoms\. The judge verifies tool\-checkable candidate claims using chess\-specific board\-state and engine tools, matches verified candidates to the gold atoms, and computes factual precision, atomic recall, and move\-quality F1\. These metrics operationalize the three dimensions introduced in Section 3\.1: Factual Precision measures verifiable factual correctness, Atomic Recall measures coverage relative to the gold annotations, and Move\-Quality F1 measures detection of engine\-labeled mistakes and blunders\.### 3\.1Background

#### Chess commentary evaluation\.

We consider the task of evaluating the quality of chess move commentary\. Formally, given a board statessrepresented in a standard encoding \(FEN or ASCII\) and a moveaaplayed in that state, a commentary𝒞\\mathcal\{C\}is a natural language explanation that explains the strategic or tactical purpose of the move\. Our goal is to assess the quality of𝒞\\mathcal\{C\}along three dimensions:\(1\) Factual correctness, the claims made in𝒞\\mathcal\{C\}about the current or subsequent states should be factually correct with respect to the board state\. For example, claims about piece placement, attacks, threats, checks, captures, legal moves, and resulting positions should be correct with respect tossandaa;\(2\) Move\-quality judgement,𝒞\\mathcal\{C\}should correctly characterize the quality of the moveaa, distinguishing, for example, a strong positional choice from an inaccuracy or blunder;\(3\) Conceptual coverage,𝒞\\mathcal\{C\}should identify the key strategic and tactical themes that a human expert would consider most salient in the position\.

#### Why evaluation is hard\.

Assessing whether the commentary satisfied these properties is non\-trivial\. Existing reference\-based, LLM\-as\-a\-judge approachesKimet al\.\([2025](https://arxiv.org/html/2608.04240#bib.bib12)\)suffer from two major limitations\. First, the reference data the judge relies on are often noisy, containing stylistic artifacts and incomplete analysis\. Second, and more critically, LLM judges are insensitive to factual errors: a fluent, chess\-specific but*wrong*claim will receive a high score, because the judge lacks the chess knowledge needed to detect the error \(see Section[4\.3](https://arxiv.org/html/2608.04240#S4.SS3)\)\.

To address these limitations, we proposeACT\-Eval\(AtomiCandTool\-augmented Evaluation\), a framework that combines atom\-grounded expert annotations as gold reference and tool\-augmented fact verification to substantially reduce LLM hallucination\.

### 3\.2gold atom Extraction

We collect reference commentary from canonical chess books and annotated online studies\. Because source quality varies, we apply regex filters to remove bare move notation, single\-word evaluations, and generic filler, then use an LLM with few\-shot demonstrations to decompose the remainder into gold atoms: discrete, position\-specific factual statements\. Each atom is contextual, carrying the full move sequence from the root position so that it remains independently verifiable \(Appendix[B\.1](https://arxiv.org/html/2608.04240#A2.SS1)\)\. Annotations that contradict engine evaluation, and positions offering no concrete positional analysis, are rejected\. Human experts review the resulting atoms; Section[5](https://arxiv.org/html/2608.04240#S5)reports a calibration study finding the gold sets fully complete on 92% and 88% of sampled positions\.

### 3\.3Verification Architecture

Our fact\-checking pipeline operates in three stages \(Figure[2](https://arxiv.org/html/2608.04240#S3.F2)\):

#### Stage 1: Atomic Claim Decomposition

The generated commentary is first decomposed intoatomic claims\(atoms\), following the same principles as gold atom extraction \(Sec\.[3\.2](https://arxiv.org/html/2608.04240#S3.SS2)\)\. Each atom is a position\-specific factual statement, with full positional context\. To enable tool\-augmented evaluation, the LLM further breaks down each candidate atom into one or moreverifiable sub\-claims\- specific facts that can be independently verified by tools\. For example, the candidate atom “Qe4 centralizes the queen and attacks the bishop on b4” decomposes into two sub\-claims: \(1\) “Qe4 centralizes the queen”, \(2\) “Qe4 attacks the bishop on b4”\. We use a lightweight classifier to assign semantic tags \(e\.g\. quality, positional, move continuation\) to each sub\-claim\.

#### Stage 2: Tool\-Augmented Verification

Each atomic claim is verified in a single judge call using a specialized prompt conditioned on the extracted tags and equipped withmandatory access to chess tools\. Within this process, the judge evaluates all associated sub\-claims before producing a final atom\-level verdict\.Sub\-claims within an atom are verified jointly rather than as independent propositions: the judge receives the original atom and all constituent sub\-claims together and must check both their individual truth and any stated relation among them, such as a setup–consequence or causal relation, before issuing the atom\-level verdict\.By grounding verification in deterministic tool outputs rather than relying on parametric model knowledge, we substantially reduce chess\-specific hallucination during factual verification\. An atomic claim is markedcorrectonly if all constituent sub\-claims are verified as true; if any sub\-claim is incorrect, the entire candidate atom is marked incorrect\.

#### Tool Suite

We implement a suite of 15\+ chess\-specific verification tools spanning board\-state queries, attack/defense analysis, move simulation, and engine evaluation; the full tool list is provided in Appendix[B\.2](https://arxiv.org/html/2608.04240#A2.SS2)\. Together, these tools enable the judge to ground factual verification in deterministic outputs rather than internal chess knowledge\. We demonstrate an example with one atomic claim below:

Multi\-Tool Verification ExampleAtomic Claim:Qe4 centralizes the queen and creates a dual threat: it attacks the bishop on b4 and prepares to infiltrate via the e\-file\. \[×1/3 verified\]✓\\checkmarkcentralizes the queen —get\_piece\_at\(e4\),get\_squares\(queen\): moved b1→\\rightarrowe4\.×attacks b4 —get\_attacks\(e4\) does not include b4\.×infiltrates the e\-file —get\_attacks\(e4\): file blocked by the e5 pawn\.

#### Stage 3: Gold Atom Matching

After verification, the judge compares verified atoms from the generated commentary against the gold atoms \(Sec\.[3\.2](https://arxiv.org/html/2608.04240#S3.SS2)\) to measure coverage of important positional concepts\.

### 3\.4Evaluation Metrics

Our evaluation computes three complementary metrics, one per dimension of Section[3\.1](https://arxiv.org/html/2608.04240#S3.SS1)\.

- •Factual Precision:our proxy for factual correctness\.The fraction of atomic claims verified as factually correct\. An atomic claim is factually correct if and only if all of its sub\-claims are verifiably correct\. Claims that are purely strategic or otherwise not verifiable through tool calls are excluded: the metric therefore measures verifiable factual correctness rather than factual correctness in general\. We also reportsub\-claim error rate, the fraction of verifiable sub\-claims that fail tool\-based verification\.
- •Atomic Recall:our proxy for conceptual coverage\.The fraction of gold atoms \(from reference annotations\) that are covered by the generated commentary, computed from Stage 3 matching\.Coverage is measured relative to our annotation set rather than to all valid expert analysis, so a commentary that develops a different but sound line is not credited; any concepts the commentary adds beyond the gold set is not penalized\. Appendix[A\.3](https://arxiv.org/html/2608.04240#A1.SS3)gives a worked example\.
- •Move Quality F1:our proxy for move\-quality judgement\.The F1 of the commentary’s verdict on the played move against an engine\-derived quality label, treating mistakes and blunders as the positive class\. Accuracy is uninformative here because the classes are unbalanced — 224 of 265 scored moves are sound, so a commentary that never criticises anything scores 84\.5% accuracy and 0 F1\. We score only clearly good and clearly bad moves, excluding the marginal band where the label itself is unreliable, and we score the verdict rather than the reasoning behind it \(Appendix[A\.1](https://arxiv.org/html/2608.04240#A1.SS1)\)\.

## 4Experiments

### 4\.1Experimental Setup

#### Datasets\.

Table 1:Evaluation datasets\. Textbook and Candidates50 include expert\-verified gold atoms and support all three metrics; Critical positions have no reference annotations and evaluate factual correctness and move quality only\.Table[1](https://arxiv.org/html/2608.04240#S4.T1)summarizes the three evaluation datasets \(N=325N=325total\)\. Textbook and Candidates50 include expert\-verified gold atoms and support all three metrics; Critical positions are generated programmatically using Maia2 \(Tang et al\., 2024\) and evaluate only factual correctness and move quality\. Dataset construction details are provided in Appendix[C\.1](https://arxiv.org/html/2608.04240#A3.SS1)\.

#### Models\.

We evaluate six leading proprietary and open\-weight models: GPT\-5\.4, Claude Opus 4\.7, Gemini 3\.1 Pro, DeepSeek V4 Pro, Qwen3\-32B, and Qwen3\-8B\. The model identifiers are listed in Table[5](https://arxiv.org/html/2608.04240#A3.T5)\. We evaluate all non\-Qwen models both with and without access to chess tools during generation\. The Qwen models are evaluated only in the tool\-free setting and are run in No\-Think mode, with reasoning traces disabled, because preliminary experiments showed that thinking mode produced degenerate outputs \(Appendix[C\.3](https://arxiv.org/html/2608.04240#A3.SS3)\)\.

#### Commentary generation\.

Models are prompted to act as a chess instructor explaining a move to an intermediate player, receiving only the board state as a FEN string and the move played\. In the tool\-enabled condition they may invoke the Stockfish\-backed tool suite \(Table[4](https://arxiv.org/html/2608.04240#A2.T4)\) before writing; otherwise they rely entirely on internal chess knowledge\. Generation settings and the full prompt are in Appendix[C\.2](https://arxiv.org/html/2608.04240#A3.SS2)\.

#### Evaluation pipeline\.

All generated commentary is evaluated using the ACT\-Eval framework \(Section 3\) with GPT\-5\.4 as the judge model, equipped with the full tool suite\. Stockfish 14\.1 at depth 18 serves as the backend engine for all tool calls\.

### 4\.2Main Results

Table 2:Combined ACT\-Eval Resultsacross all datasets\.Error Rateis the fraction of sub\-claims that fail verification;Precisionis the fraction of atoms that are factually correct;Recallis the fraction of gold reference atoms covered, computed only on datasets with gold reference annotations;Quality\-F1is the F1 score for detecting bad \(mistake/blunder\) moves, on positions excluding inaccuracies\. Bracketed ranges are95%95\\%bootstrap confidence intervals \(B=10,000B\{=\}10\{,\}000resamples, clustered at the position level\)\. Human calibration suggests that the reported error rates may be conservative\. NT = No\-Think mode\.Table[2](https://arxiv.org/html/2608.04240#S4.T2)presents results aggregated across all three datasets\. Tool access substantially reduces factual errors and largely resolves move\-quality judgement; its effect on conceptual coverage \(recall\) is model\-dependent, and coverage remains limited in every configuration\.

#### LLMs hallucinate frequently in chess commentary\.

Without tool access, even frontier models produce factually unreliable commentary : sub\-claim error rates run from10\.8%10\.8\\%\(Gemini 3\.1 Pro\) to55\.5%55\.5\\%\(Qwen3\-8B\)\. These results are consistent with the well\-documented finding that LLMs lack reliable chess\-playing abilityKolasaniet al\.\([2025](https://arxiv.org/html/2608.04240#bib.bib30)\): generating good commentary requires not just fluency but the ability to identify key ideas in a position, which in turn demands accurate board comprehension \- recognizing piece locations, legal moves, tactical patterns, and threats\. Errors in this basic spatial reasoning propagate directly into the explanations\.

#### Tool access substantially reduces factual errors\.

Equipping models with chess tools, including Stockfish evaluation and variations, legality checks, piece location queries, and attack/defense analysis \- mitigates a large share of these errors\. GPT\-5\.4’s sub\-claim error rate drops from 22\.0% to 9\.2% with tool access, more than halving the rate of verification failures, and cuts Claude Opus’ by a third\. Similar trends are observed on Gemini 3\.1 Pro and DeepSeek V4 Pro\.

The key insight is that tools shift the burden: models need not compute internally whether a square is attacked or a move is legal, and can concentrate on structuring an explanation around verified facts\. They do not remove the need for chess understanding: the model must still select which facts matter, interpret what the tools return, and assemble them into an explanation, and the coverage results below show it is precisely this that remains hard\.

#### Recall remains low across all models\.

Even the best configuration, Gemini 3\.1 Pro with tools, covers only 60\.9% of expert\-identified gold atoms, and the Qwen models fall below 16%\. The effect of tools is model\-dependent: recall is essentially unchanged for GPT\-5\.4 \(44\.6% → 44\.0%\) and Gemini \(58\.2%→\\to60\.9%\), but rises sharply for Claude Opus 4\.7 \(45\.3%→\\to58\.3%\) and DeepSeek V4 Pro \(30\.1%→\\to57\.2%\)\. Verified facts are therefore not on their own sufficient to surface expert ideas; whether a model builds on them varies by model\. Identifying the critical ideas in a position remains the hardest of the three dimensions\.

#### Models detect bad moves unreliably without engine access\.

Judging move quality is an essential component of a good explanation\. Without engine access, bad\-move detection is weak across all evaluated models: treating mistakes and blunders as the positive class, F1 ranges from 12\.5 for GPT\-5\.4 to 40\.9 for Qwen3\-32B\. Tool access raises F1 to 97\.6 for GPT\-5\.4 and 93\.0 for Claude Opus 4\.7\. Because the metric compares commentary stance against engine\-derived labels, these gains primarily demonstrate that models can use engine evidence to calibrate their move\-quality statements, rather than an improvement in chess evaluation ability\. Appendix[A\.2](https://arxiv.org/html/2608.04240#A1.SS2)provides the per\-class results\.

### 4\.3Comparison with vanilla LLM\-as\-a\-judge

To test whether the failure illustrated in Figure[1](https://arxiv.org/html/2608.04240#S1.F1)is systematic rather than anecdotal, we score 500 commentaries, stratified across five generators, with both GCC\-Eval\(Kimet al\.,[2025](https://arxiv.org/html/2608.04240#bib.bib12)\)and ACT\-Eval\. ACT\-Eval flags 264 of them as majority\-erroneous \(sub\-claim error rate≥0\.5\\geq 0\.5\)\. On exactly these commentaries, GCC\-Eval still awards high relevance \(≥4/5\\geq 4/5\) to39\.8%39\.8\\%\(95% CI34\.134\.1–45\.845\.8\) and high completeness to25\.0%25\.0\\%\(20\.120\.1–30\.530\.5\)\. This confirms that the failure mode illustrated in Figure 1 occurs systematically rather than in isolated cases\. ACT\-Eval addresses it by replacing both subjective dimensions with verifiable ones: relevance with factual precision, computed over atomic claims each checked against the board, and completeness with atomic recall against expert\-verified gold atoms\. We develop this comparison further in Appendix[D\.3](https://arxiv.org/html/2608.04240#A4.SS3)\.

## 5Human Calibration

To validate that the ACT\-Eval judges align with expert human assessment, we conduct two calibration studies targeting the two judge\-dependent metrics: \(i\) factual correctness \(Stage 2\) and \(ii\) atomic recall against gold references \(Stage 3\)\.

#### Participants\.

Four chess players with Lichess blitz Elo between 1800\-2200 participated in the study\. All four completed Study 1, while two completed Study 2\. Annotators worked through a purpose\-built interface that presented the board diagram, FEN, and engine analysis alongside the LLM commentary or claim under review, and they were blinded to the judge’s verdicts\.

### 5\.1Study 1: Claim Verification

We assess whether the judge’s per\-atom verdict agrees with expert human judgement\. We draw 50 atomic claims from the evaluation corpus using stratified sampling across different models and judge verdicts \(30% judge\-correct, 70% judge\-incorrect; fixed seed\) and place them in randomized order\. For each item, annotators \(a\) issued a verdict on a four\-point scale \(Correct / Mostly correct / Incorrect / Cannot determine\) and \(b\) reported confidence \(low/medium/high\)\.

Since the judge produces binary labels and marks an atom incorrect whenever any sub\-claim fails verification \(Section[3\.3](https://arxiv.org/html/2608.04240#S3.SS3)\), mostly correct human verdicts are mapped to incorrect for agreement analysis, consistent with the judge’s own aggregation rule\.

#### Judge agreement approaches inter\-human variability\.

We measure agreement between the judge’s binary verdict and human annotators on the 50 sampled claims\. Per\-annotator agreement with the judge ranges from 69% to 83%, with a macro\-average of 74\.5% across the four annotators\. Pairwise agreement between annotators ranges from 73% to 91%\. This indicates that the judge falls within the range of inter\-human agreement on the task\.

Pairwise Cohen’sκ\\kapparanges from 0\.107 to 0\.697, while Fleiss’κ\\kappais 0\.285\. Because the sample is 70% judge\-incorrect by design, these chance\-corrected measures are prevalence\-sensitive and should be interpreted alongside raw agreement\.

#### The judge is conservative but reliable at detecting errors\.

When the judge marks a claim*incorrect*, annotators agree 83–94% of the time: errors flagged by the judge are almost always real errors\. When it marks a claim*correct*, agreement drops to 14–57%\. The judge therefore catches many genuine errors but also misses errors identified by human annotators, suggesting that the reported atom error rates in Table[2](https://arxiv.org/html/2608.04240#S4.T2)may be conservative rather than exact\.

### 5\.2Study 2: Commentary Coverage

We evaluate whether the judge’s recall metric aligns with expert human assessment over 25 position–move pairs\. The task proceeds in two phases\. In the*completeness check*, annotators are shown the gold atoms and judge whether they cover the most important observations about the move \(fully / partially / no\)\. Annotators can optionally add missing key points, which are appended to the gold set\. This provides a sanity check on the quality of our extracted gold atoms\.

Next, in the*coverage annotation*, annotators are shown an LLM\-generated commentary \(sampled across models\) and indicate, for each gold atom \(including any additions\), whether it is covered by the commentary\. Human recall is computed as the fraction of \(augmented\) gold atoms judged covered\.

#### Gold atoms serve as adequate coverage targets\.

Gold atoms are extracted from grandmaster commentary, filtered, and decomposed via the pipeline of Section[3\.2](https://arxiv.org/html/2608.04240#S3.SS2)\. Across the 25 sampled positions, the two annotators rated the gold set as fully complete in 92% and 88% of cases respectively; the remaining items were marked partially complete, typically with a single additional point appended\. This validates the extracted gold atoms as adequate coverage targets\.

#### Judge recall correlates strongly with expert humans\.

The judge’s per\-position recall correlates with the two annotators at Pearsonr=0\.732r=0\.732andr=0\.749r=0\.749\. The annotators themselves agree atr=0\.835r=0\.835, indicating that the judge\-human gap is small relative to the inherent variability of the task\.

The judge applies a stricter coverage threshold than human experts: mean judge recall over the sample is 0\.33, compared to 0\.41–0\.47 for the two annotators, with the gap consistent across positions\. Recall scores in Table[2](https://arxiv.org/html/2608.04240#S4.T2)should accordingly be read as lower bounds on true coverage\.

## 6Analysis

### 6\.1Error Analysis

![Refer to caption](https://arxiv.org/html/2608.04240v1/x3.png)Figure 3:Factual error breakdown by category across models, normalized by total decisive sub\-claims\.Figure[3](https://arxiv.org/html/2608.04240#S6.F3)breaks down factual errors across five categories \(Appendix[B\.3](https://arxiv.org/html/2608.04240#A2.SS3)\), normalized by decisive sub\-claims\. Open\-weight models are dominated by board comprehension and legality failures: pieces hallucinated on wrong squares, misidentified attackers, illegal continuations, this indicates weak internal representations of the board state itself\. Frontier models largely avoid these static failures but still err in multi\-move variations without tool access: GPT\-5\.4 and Claude Opus 4\.7 generate plausible continuations that become inconsistent after several moves, showing that linguistic fluency does not extend to maintaining board state over long\-horizon sequences\. Tool augmentation shifts the distribution rather than uniformly shrinking it\. Legality checks and board\-state queries largely eliminate low\-level hallucinations, leaving errors concentrated in tactical reasoning: models misjudge whether an idea works, overlook defensive resources, or hallucinate forcing continuations\. Errors in the quality\-evaluation category become rare, consistent with the Move Quality F1 gains in Table[2](https://arxiv.org/html/2608.04240#S4.T2)\.

### 6\.2Ablation Study on Tool Augmentation

To isolate the contribution of tools to the judge, we hold decomposition fixed and re\-verify the same sub\-claims with GPT\-5\.4 without tool access\. On 29 claims from Study 1 that all four annotators marked incorrect with high confidence — a clean set of confirmed errors — ACT\-Eval misses 21%, against 48% for the no\-tool ablation\. Tool augmentation therefore accounts for most of the judge’s error\-detection ability, and the residual 21% is further evidence that reported atom error rates are lower bounds\.

## Limitations

#### Tool\-suite coverage\.

Our verification framework relies on a fixed set of chess\-specific tools\. Human calibration shows that ACT\-Eval still misses some factual errors\. These missed errors are not confined to a single category; they occur across board comprehension, legality, tactical reasoning, and evaluation\-related claims\. Because ACT\-Eval can both miss factual errors and reject valid claims, its reported error rates are judge\-dependent estimates rather than formal lower bounds\. Human calibration nevertheless shows that some expert\-identified errors are missed\. Expanding the suite, particularly for long\-horizon variation checking and endgame\-specific patterns, is a natural direction for closing this gap\.

#### The framework still inherits LLM\-judge limitations\.

Although tools reduce reliance on the judge’s internal chess knowledge, the judge still interprets claims, selects tools and tool\-arguments, and aggregates evidence\. Appendix[D\.2](https://arxiv.org/html/2608.04240#A4.SS2)documents failures in both directions on the same claims: errors the GPT\-5\.4 judge accepts that an independent Gemini judge catches, and valid claims it rejects by misreading tool output\. The deterministic tools returned correct evidence in both cases; the verification procedure around them failed\.

#### LLM\-based decomposition can introduce errors\.

ACT\-Eval relies on an LLM to split commentary into atomic claims\. This step can produce incorrect or under\-contextualized atoms, especially when claims depend on earlier sentences, implicit move sequences, or surrounding tactical context\. If an atom loses necessary context, the verification stage may incorrectly reject it\.

#### Gold atoms are incomplete approximations of expert understanding\.

The expert\-verified gold atoms capture many salient tactical and strategic ideas, but chess explanations are open\-ended and multiple valid analyses may exist for the same move\. Atomic recall should therefore be interpreted as coverage relative to our annotation set, not as an absolute measure of all possible expert insight\.

#### Evaluation scale\.

The benchmark contains 325 position–move pairs, with gold atoms for 125, while the human calibration studies cover 50 atomic claims and 25 commentaries\. These samples support calibrated comparisons among the evaluated systems but do not establish performance across the full diversity of chess positions, commentary styles, or annotators\.

#### Evidence is chess\-specific\.

We frame ACT\-Eval as a template for domains where part of the truth is computable and part requires expert judgement, but we validate it only on chess\. Transfer requires both a deterministic oracle for the verifiable component and expert annotation for the rest; whether those conditions are commonly met is left to future work\.

## Acknowledgments

We thank Amey Bhat, Praveen Iyer, Sambhu Karumanchi, and Sanskar Jaiswal for providing expert chess annotations\.

## References

- FacTool: factuality detection in generative ai–a tool augmented framework for multi\-task and multi\-domain scenarios\.arXiv preprint arXiv:2307\.13528\.Cited by:[Appendix E](https://arxiv.org/html/2608.04240#A5.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2608.04240#S2.p1.1),[§2](https://arxiv.org/html/2608.04240#S2.p1.1.2)\.
- I\. Chernev \(2003\)Logical chess: move by move: every move explained\.Batsford,London\.External Links:ISBN 9780713484649Cited by:[§C\.1](https://arxiv.org/html/2608.04240#A3.SS1.p1.1)\.
- E\. Durmus, H\. He, and M\. Diab \(2020\)FEQA: a question answering evaluation framework for faithfulness assessment in abstractive summarization\.InProceedings of the 58th annual meeting of the association for computational linguistics,pp\. 5055–5070\.Cited by:[Appendix E](https://arxiv.org/html/2608.04240#A5.SS0.SSS0.Px2.p1.1)\.
- A\. R\. Fabbri, C\. Wu, W\. Liu, and C\. Xiong \(2022\)QAFactEval: improved qa\-based factual consistency evaluation for summarization\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 2587–2601\.Cited by:[Appendix E](https://arxiv.org/html/2608.04240#A5.SS0.SSS0.Px2.p1.1)\.
- T\. Falke, L\. F\. Ribeiro, P\. A\. Utama, I\. Dagan, and I\. Gurevych \(2019\)Ranking generated summaries by correctness: an interesting but challenging application for natural language inference\.InProceedings of the 57th annual meeting of the association for computational linguistics,pp\. 2214–2220\.Cited by:[§2](https://arxiv.org/html/2608.04240#S2.p1.1.1)\.
- X\. Feng, Y\. Luo, Z\. Wang, H\. Tang, M\. Yang, K\. Shao, D\. Mguni, Y\. Du, and J\. Wang \(2023\)Chessgpt: bridging policy learning and language modeling\.Advances in Neural Information Processing Systems36,pp\. 7216–7262\.Cited by:[Appendix E](https://arxiv.org/html/2608.04240#A5.SS0.SSS0.Px1.p1.1)\.
- N\. Fiekas \(2022\)Python\-chess: a chess library for Python\.Note:GitHub repositoryExternal Links:[Link](https://github.com/niklasf/python-chess)Cited by:[§1](https://arxiv.org/html/2608.04240#S1.p5.1)\.
- H\. Jhamtani, V\. Gangal, E\. Hovy, G\. Neubig, and T\. Berg\-Kirkpatrick \(2018\)Learning to generate move\-by\-move commentary for chess games from large\-scale social forum data\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1661–1671\.Cited by:[Appendix E](https://arxiv.org/html/2608.04240#A5.SS0.SSS0.Px1.p1.1)\.
- J\. Kim, J\. Goh, I\. Hwang, J\. Cho, and J\. Ok \(2025\)Bridging the gap between expert and language models: concept\-guided chess commentary generation and evaluation\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 9497–9516\.Cited by:[§D\.3](https://arxiv.org/html/2608.04240#A4.SS3.p1.1),[Appendix E](https://arxiv.org/html/2608.04240#A5.SS0.SSS0.Px1.p1.1),[Appendix E](https://arxiv.org/html/2608.04240#A5.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2608.04240#S1.p4.1),[§2](https://arxiv.org/html/2608.04240#S2.p1.1),[§3\.1](https://arxiv.org/html/2608.04240#S3.SS1.SSS0.Px2.p1.1),[§4\.3](https://arxiv.org/html/2608.04240#S4.SS3.p1.8)\.
- S\. Kolasani, M\. Saplin, N\. Crispino, K\. Montgomery, J\. Q\. Davis, M\. Zaharia, C\. Wang, and C\. Wang \(2025\)LLM chess: benchmarking reasoning and instruction\-following in llms through chess\.arXiv preprint arXiv:2512\.01992\.Cited by:[§1](https://arxiv.org/html/2608.04240#S1.p3.1),[§1](https://arxiv.org/html/2608.04240#S1.p4.1),[§4\.2](https://arxiv.org/html/2608.04240#S4.SS2.SSS0.Px1.p1.2)\.
- W\. Kryściński, B\. McCann, C\. Xiong, and R\. Socher \(2020\)Evaluating the factual consistency of abstractive text summarization\.InProceedings of the 2020 conference on empirical methods in natural language processing \(EMNLP\),pp\. 9332–9346\.Cited by:[Appendix E](https://arxiv.org/html/2608.04240#A5.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2608.04240#S2.p1.1.1)\.
- A\. Lee, D\. Wu, E\. Dinan, and M\. Lewis \(2022\)Improving chess commentaries by combining language models with symbolic reasoning engines\.arXiv preprint arXiv:2212\.08195\.Cited by:[Appendix E](https://arxiv.org/html/2608.04240#A5.SS0.SSS0.Px1.p1.1)\.
- Leela Chess Zero \(2026\)Leela chess zero: open source neural network based chess engine\.Note:[https://lczero\.org/](https://lczero.org/)Accessed: 2026\-05\-25Cited by:[§1](https://arxiv.org/html/2608.04240#S1.p5.1)\.
- Lichess \(2026\)Lichess accuracy metric\.Note:[https://lichess\.org/page/accuracy](https://lichess.org/page/accuracy)Accessed: 2026\-08\-03Cited by:[§A\.1](https://arxiv.org/html/2608.04240#A1.SS1.SSS0.Px1.p1.3)\.
- C\. Lin \(2004\)Rouge: a package for automatic evaluation of summaries\.InText summarization branches out,pp\. 74–81\.Cited by:[Appendix E](https://arxiv.org/html/2608.04240#A5.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2608.04240#S1.p4.1),[§2](https://arxiv.org/html/2608.04240#S2.p1.1)\.
- Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. Zhu \(2023\)G\-eval: nlg evaluation using gpt\-4 with better human alignment\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 2511–2522\.Cited by:[Appendix E](https://arxiv.org/html/2608.04240#A5.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2608.04240#S1.p4.1),[§2](https://arxiv.org/html/2608.04240#S2.p1.1)\.
- J\. Maynez, S\. Narayan, B\. Bohnet, and R\. McDonald \(2020\)On faithfulness and factuality in abstractive summarization\.InProceedings of the 58th annual meeting of the association for computational linguistics,pp\. 1906–1919\.Cited by:[§2](https://arxiv.org/html/2608.04240#S2.p1.1.1)\.
- S\. Min, K\. Krishna, X\. Lyu, M\. Lewis, W\. Yih, P\. Koh, M\. Iyyer, L\. Zettlemoyer, and H\. Hajishirzi \(2023a\)FActScore: fine\-grained atomic evaluation of factual precision in long form text generation\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 12076–12100\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.741/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.741)Cited by:[Appendix E](https://arxiv.org/html/2608.04240#A5.SS0.SSS0.Px2.p1.1)\.
- S\. Min, K\. Krishna, X\. Lyu, M\. Lewis, W\. Yih, P\. Koh, M\. Iyyer, L\. Zettlemoyer, and H\. Hajishirzi \(2023b\)FActScore: fine\-grained atomic evaluation of factual precision in long form text generation\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 12076–12100\.Cited by:[§2](https://arxiv.org/html/2608.04240#S2.p1.1),[§2](https://arxiv.org/html/2608.04240#S2.p1.1.2)\.
- OpenAI \(2024\)Learning to reason with LLMs\.Note:OpenAI o1 system card and technical reportExternal Links:[Link](https://openai.com/index/learning-to-reason-with-llms/)Cited by:[§1](https://arxiv.org/html/2608.04240#S1.p3.1)\.
- K\. Papineni, S\. Roukos, T\. Ward, and W\. Zhu \(2002\)Bleu: a method for automatic evaluation of machine translation\.InProceedings of the 40th annual meeting of the Association for Computational Linguistics,pp\. 311–318\.Cited by:[Appendix E](https://arxiv.org/html/2608.04240#A5.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2608.04240#S1.p4.1),[§2](https://arxiv.org/html/2608.04240#S2.p1.1)\.
- D\. Silver, A\. Huang, C\. J\. Maddison, A\. Guez, L\. Sifre, G\. van den Driessche, J\. Schrittwieser, I\. Antonoglou, V\. Panneershelvam, M\. Lanctot, S\. Dieleman, D\. Grewe, J\. Nham, N\. Kalchbrenner, I\. Sutskever, T\. Lillicrap, M\. Leach, K\. Kavukcuoglu, T\. Graepel, and D\. Hassabis \(2016\)Mastering the game of Go with deep neural networks and tree search\.Nature529\(7587\),pp\. 484–489\.External Links:[Document](https://dx.doi.org/10.1038/nature16961)Cited by:[§1](https://arxiv.org/html/2608.04240#S1.p2.1)\.
- Stockfish \(2024\)Stockfish: a strong open source chess engine\.Note:Open\-source chess engineExternal Links:[Link](https://stockfishchess.org/)Cited by:[§1](https://arxiv.org/html/2608.04240#S1.p2.1),[§1](https://arxiv.org/html/2608.04240#S1.p5.1)\.
- A\. Szymanski, N\. Ziems, H\. A\. Eicher\-Miller, T\. J\. Li, M\. Jiang, and R\. A\. Metoyer \(2025\)Limitations of the llm\-as\-a\-judge approach for evaluating llm outputs in expert knowledge tasks\.InProceedings of the 30th international conference on intelligent user interfaces,pp\. 952–966\.Cited by:[Appendix E](https://arxiv.org/html/2608.04240#A5.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2608.04240#S1.p4.1),[§2](https://arxiv.org/html/2608.04240#S2.p1.1.2)\.
- Z\. Tang, D\. Jiao, R\. McIlroy\-Young, J\. Kleinberg, S\. Sen, and A\. Anderson \(2024\)Maia\-2: a unified model for human\-ai alignment in chess\.Advances in Neural Information Processing Systems37,pp\. 20919–20944\.Cited by:[§C\.1](https://arxiv.org/html/2608.04240#A3.SS1.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[§1](https://arxiv.org/html/2608.04240#S1.p3.1)\.
- J\. Wei, C\. Yang, X\. Song, Y\. Lu, N\. Hu, J\. Huang, D\. Tran, D\. Peng, R\. Liu, D\. Huang,et al\.\(2024\)Long\-form factuality in large language models\.Advances in Neural Information Processing Systems37,pp\. 80756–80827\.Cited by:[Appendix E](https://arxiv.org/html/2608.04240#A5.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2608.04240#S2.p1.1),[§2](https://arxiv.org/html/2608.04240#S2.p1.1.2)\.
- Q\. Wen, Z\. Tang, and A\. Anderson \(2025\)ChessQA: evaluating large language models for chess understanding\.arXiv preprint arXiv:2510\.23948\.Cited by:[§1](https://arxiv.org/html/2608.04240#S1.p3.1),[§1](https://arxiv.org/html/2608.04240#S1.p4.1)\.
- H\. Zang, Z\. Yu, and X\. Wan \(2019\)Automated chess commentator powered by neural chess engine\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 5952–5961\.External Links:[Link](https://aclanthology.org/P19-1597/),[Document](https://dx.doi.org/10.18653/v1/P19-1597)Cited by:[Appendix E](https://arxiv.org/html/2608.04240#A5.SS0.SSS0.Px1.p1.1)\.
- T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. Artzi \(2019\)Bertscore: evaluating text generation with bert\.arXiv preprint arXiv:1904\.09675\.Cited by:[Appendix E](https://arxiv.org/html/2608.04240#A5.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2608.04240#S2.p1.1)\.

## Appendix AMetric Definitions

### A\.1Move Quality

Move Quality F1 compares an engine\-derived label for the played move against the stance the commentary takes toward it\.

#### Deriving move quality from the engine\.

Chess engines such as Stockfish score a position in*centipawns*, where100100centipawns is roughly the value of one pawn\. A move’s centipawn loss – how much worse the position becomes relative to the engine’s preferred move – cannot be thresholded directly, because the same loss carries different consequences depending on the position\. Giving up100100centipawns in a level position can convert a draw into a loss, whereas the same concession in a position that is already overwhelmingly won changes nothing about the likely outcome\. Following\(Lichess,[2026](https://arxiv.org/html/2608.04240#bib.bib2)\), we therefore first map the evaluationcc\(in centipawns, from the perspective of the side to move\) to an estimated win probability,

Win%=50\+50​\(21\+exp⁡\(−0\.00368208⋅c\)−1\),\\mathrm\{Win\\%\}=50\+50\\left\(\\frac\{2\}\{1\+\\exp\(\-0\.00368208\\cdot c\)\}\-1\\right\),and definewp​\_​loss=Win%​\(s,a∗\)−Win%​\(s,a\)\\mathrm\{wp\\\_loss\}=\\mathrm\{Win\\%\}\(s,a^\{\*\}\)\-\\mathrm\{Win\\%\}\(s,a\)for played moveaaand engine best movea∗a^\{\*\}\. We binwp​\_​loss\\mathrm\{wp\\\_loss\}using the thresholds Lichess applies when annotating games,222[https://github\.com/lichess\-org/lila/blob/master/modules/tree/src/main/Advice\.scala](https://github.com/lichess-org/lila/blob/master/modules/tree/src/main/Advice.scala)so our labels match what a standard public analysis tool reports:*good*\(≤10\\leq 10\),*inaccuracy*\(1010–2020\),*mistake*\(2020–3030\),*blunder*\(\>30\>30\)\.

#### Commentary stance\.

Decomposition extracts the stance toward the played move as*good*,*bad*, or*inconclusive*\(no explicit judgement\), resolving negations so that “not the best” maps to*good*\. Judgements about alternatives or about the position as a whole are excluded\.

#### Matching\.

We score commentary on clearly good and clearly bad moves only, excluding the6060of325325positions whose engine label falls in the*inaccuracy*band \(10<wp​\_​loss≤2010<\\mathrm\{wp\\\_loss\}\\leq 20\), leavingn=265n\{=\}265\. Nearly all of these excluded positions \(5959of6060\) lie in the Critical set, which by design samples positions where Maia2 predicts an inaccuracy or mistake; the remaining one is in Candidates50, and none are in Textbook\. Human annotators routinely disagree with the engine at this margin, often for sound practical reasons; scoring stance there would measure agreement with a cutoff rather than whether a model can tell a good move from a bad one\. On the remaining positions, a*good*label is satisfied by a*good*or*inconclusive*stance, since commentary that explains a sound move without editorialising is not in error, while*mistake*and*blunder*require an explicit*bad*stance, since failing to flag a losing move is a substantive failure\.

Table 3:Move quality assessment\.Treating move\-quality assessment as binary detection of*bad*moves \(mistake/blunder vs\. good\), on positions excluding inaccuracies\. A*bad*prediction requires an explicit*bad*stance;*good*and*inconclusive*both count as not\-bad, so silence on a losing move is a miss\. Bracketed ranges are95%95\\%position\-clustered bootstrap CIs \(B=10,000B\{=\}10\{,\}000\)\. NT = No\-Think mode\.

### A\.2Move Quality: Per\-Class Results

Table[3](https://arxiv.org/html/2608.04240#A1.T3)decomposesF1bad\\text\{F1\}\_\{\\text\{bad\}\}into*Quality\-Precision*\(of the moves a commentary called bad, how many were bad\) and*Quality\-Recall*\(of the actually bad moves, how many it caught\)\. We use the hyphenated names throughout to distinguish these from Factual Precision and Atomic Recall, which measure unrelated quantities\.

The no\-tool failures take two distinct forms\. GPT\-5\.4 and Claude Opus 4\.7 flag close to the right number of moves \(3939and4040, against4141actual\) but largely the wrong ones, with Quality\-Precision and Quality\-Recall both near20%20\\%— the signature of guessing at roughly the correct rate\. The Qwen models instead over\-flag, calling8282and9191of265265moves bad; this buys Quality\-Recall \(41\.541\.5and65\.965\.9\) while Quality\-Precision collapses, and the false alarms on sound moves are what push their accuracy below the84\.5%84\.5\\%majority\-class baseline\. Tool\-augmented frontier models are calibrated in both senses: GPT\-5\.4 predicts4343bad moves and Claude4545, against4141\.

The positive class has small support \(4141positions\), so intervals are wide and the robust claim is the tool\-versus\-no\-tool contrast rather than fine rankings among models\. GPT\-5\.4 with tools has a degenerate Quality\-Recall interval because it caught every bad move in every bootstrap resample\.

### A\.3Atomic Recall

Atomic Recall is a proxy for conceptual coverage: it measures how many of our annotated expert observations a commentary mentions\. Three consequences follow\.

First, coverage is bounded by the annotation\. Gold atoms are extracted from a single expert source per position, and chess positions admit multiple sound analyses; a commentary that pursues a different but valid line receives no credit for it\. Recall is therefore coverage of*these*expert observations, not of all expert insight available in the position\.

Second, recall does not penalize what a commentary adds\. A commentary can match every gold atom while surrounding them with generic or irrelevant material\. Factual Precision constrains whether the additions are true, but neither metric asks whether they were worth saying\.

Third, matching is performed by an LLM judge\. Our calibration study \(Section[5](https://arxiv.org/html/2608.04240#S5)\) finds that the judge applies a stricter threshold than human annotators \(0\.330\.33against0\.410\.41–0\.470\.47\), so reported recall is a lower bound on true coverage\.

We now provide an example : the following pair illustrates the distinction between commentaries high and low recall: both commentaries pass factual verification, but only one recovers the point of the move\.

> Position:8/pn2Q3/1p1p2pk/2pP2rp/P1P2q2/1NP5/6B1/2R4K b \- \- 15 47 Move:Qh4\+ Gold atoms: 1. 1\.Qh4\+forces White’s king to move tog1\. 2. 2\.AfterQh4\+and48\.Kg1, Black can win the queen byRxg2\+\.

#### High coverage \(recall = 1\.0\)\.

> “Qh4\+is a forcing move that sets up a brilliant discovered attack to win the white queen\. After the forcedKg1, Black will playRxg2\+, sacrificing the rook with check\. This clears theg5square, opening theh4–e7diagonal for the black queen to capture White’s undefended queen one7\.”

Both gold atoms are matched: the commentary names the forced king move and theRxg2\+resource that wins the queen\.

#### Low coverage \(recall = 0\.0\)\.

> “Qh4\+is the strongest move because it drags Black’s queen fromf4to a square where White cannot challenge it, while forcing the white king to react\. Fromh4the queen immediately eyesh2, and with the rook already controlling theg\-file, Black’s attack becomes decisive: after a natural move likeBh3, Black hasQxh3mate, and afterKg1Black keeps a huge winning attack with threats likeQh2\+\. It’s a precise move that turns Black’s active queen and rook into a direct mating net against the exposed king\.”

This commentary is fluent, its individual claims are factually defensible, and its overall verdict on the move is correct\. It nonetheless matches neither gold atom: it does not identifyKg1as forced, and it never findsRxg2\+, the concrete resource that decides the game\.

## Appendix BFramework Details

### B\.1Golden Atom Extraction

Golden Atom Extraction ExampleOriginal annotation:“Nxd7 removes one of the defenders of the knight on f6, which is a typical tactical theme in such positions\. After the expected Nh5, White can capture the bishop on g6, opening up Black’s king\. Controlling the center remains paramount in the middlegame\.”×BAD \(non\-contextual — later atoms lose move context\):•Nxd7 removes one of the defenders of the knight on f6\.•After Nh5, White can capture on g6\.← Missing context: after what?•Capturing on g6 opens Black’s king\.← After which moves?•“which is a typical tactical theme in such positions”— generic, not position\-specific•“Controlling the center remains paramount in the middlegame”— not relevant to this move✓GOOD \(contextual, standardized\):•Nxd7 removes one of the defenders of the knight on f6\.•After Nxd7 Nh5, White can capture the bishop on g6\.•After Nxd7 Nh5, capturing on g6 opens up Black’s king to attack\.

Table 4:Tool suite\.We implement a set of tools that enables the LLM judge to verify the factual correctness of diverse chess claims through chained tool calls\.
### B\.2Tool\-augmented Verification

Our verification framework exposes a collection of chess\-specific tools designed to support fine\-grained factual checking of generated commentary\. The tools operate over FEN\-encoded positions and return structured outputs that can be chained together by the judge model during verification\. We group the tools into five categories: \(i\) board\-state and legality queries, \(ii\) attack and defense queries, \(iii\) geometric reasoning, \(iv\) move\-variation simulation, and \(v\) engine analysis queries\.

Importantly, many commentary claims require composing multiple tools rather than relying on a single engine score\. For example, verifying a claim about a tactical threat may involve checking move legality, simulating a continuation, evaluating resulting attacks, and comparing engine assessments before and after the sequence\. This modular design allows ACT\-Eval to verify diverse classes of chess claims while minimizing reliance on the judge model’s internal chess knowledge\.

### B\.3Error Taxonomy

We define five error categories used in Figure[3](https://arxiv.org/html/2608.04240#S6.F3):

- •Board comprehension: Failures to accurately represent the static board configuration, including piece identity, location, and structural properties \(e\.g\., open vs\. blocked files\), reflecting an incorrect or incomplete internal representation of the position\.
- •Legality: Failures to track board state across moves, resulting in proposed moves or sequences that are not legally playable from the given position\.
- •Quality evaluation: Incorrect assessments of move strength or positional advantage that contradict engine analysis, including invalid positional comparisons \(e\.g\., claiming a feature changes when it does not\)\.
- •Tactical reasoning: Errors in reasoning about the dynamic consequences of a position, such as asserting nonexistent threats or claiming a move is forced despite viable alternatives, often downstream of board comprehension failures\.
- •Other: Residual category for all other errors\.

### B\.4Prompts

`Decomposition prompt Classification prompt Verification prompt Gold atom matching prompt`

`Appendix C Experimental Details C\.1 Datasets We evaluate on three datasets, each targeting different aspects of commentary quality\. Textbook \(75 positions\) is drawn from six games in Chernev’s Logical Chess: Move by Move Chernev \(2003\), a pedagogical text providing expert annotations for every move\. We apply the filtering pipeline described in Section 3\.2 to remove bare notation, single\-word evaluations, and generic filler, retaining 75 positions with substantive commentary from which we extract gold atoms\. This dataset tests both factual correctness and conceptual coverage against high\-quality human references\. Candidates50 \(50 positions\) is compiled from a Lichess study annotating games from the 2026 Candidates Tournament\.333https://lichess\.org/study/Y1yXP80U We apply the same filtering and atom extraction pipeline to obtain gold atoms\. This dataset provides a modern, tournament\-level complement to the textbook annotations, testing whether models can match expert analysis of elite\-level play\. Critical \(200 position\-move pairs\) targets factual correctness and move quality assessment without requiring reference commentary\. We use Maia2 Tang et al\. \(2024\), a human move\-prediction model calibrated to Elo 2000, to identify positions where human players are predicted to play an inaccuracy with probability \>\>10%, following the Lichess accuracy definition444https://lichess\.org/page/accuracy\. For each position, we sample both the best engine move and the predicted human mistake, yielding 200 \(position, move\) pairs\. Since no reference commentary exists for these positions, recall is not applicable; this dataset evaluates only factual correctness and quality assessment\. Although modest relative to training corpora, the benchmark is designed for evaluation rather than model training: all 325 position–move pairs support claim\-level factual verification, 125 additionally include expert\-verified gold atoms, and we report position\-clustered bootstrap intervals for all main metrics\. C\.2 Implementation Details To enable reproducibility of these results, Table 5 lists the exact model identifier used at every stage of commentary generation and of the ACT\-Eval pipeline\. Stage Model Identifier Commentary generation GPT\-5\.4 openai/gpt\-5\.4\-20260305 Claude Opus 4\.7 anthropic/claude\-4\.7\-opus\-20260416 Gemini 3\.1 Pro google/gemini\-3\.1\-pro\-preview DeepSeek V4 Pro deepseek/deepseek\-v4\-pro Qwen3\-32B Qwen/Qwen3\-32B Qwen3\-8B Qwen/Qwen3\-8B ACT\-Eval pipeline Golden atom extraction GPT\-5\.4 openai/gpt\-5\.4\-20260305 Stage 1: atomic decomposition GPT\-5\.4 openai/gpt\-5\.4\-20260305 Stage 1: semantic classification GPT\-5\.4\-mini openai/gpt\-5\.4\-mini\-2026\-03\-17 Stage 2: tool\-augmented verification GPT\-5\.4 openai/gpt\-5\.4\-20260305 Stage 2: cross\-judge analysis Gemini 3\.1 Pro google/gemini\-3\.1\-pro\-preview Stage 3: gold atom matching GPT\-5\.4 openai/gpt\-5\.4\-20260305 Table 5: Exact model identifiers used at each stage of generation and evaluation\. Generation configuration\. All commentary is generated at temperature 0\.30\.3 with a completion budget of max\_tokens =2048=2048 and the same base system prompt \(Appendix B\.4\), which instructs the model to act as a chess instructor addressing an intermediate player and to produce concise, position\-specific prose\. The model receives the board state as a FEN string together with the move played in SAN\. In the tool\-enabled condition the model may additionally call the tool suite defined in Table 4 before writing its explanation\. These tools include engine analysis, board comprehension helpers, and legality checks\. In the no\-tool condition the model must rely entirely on parametric knowledge\. Evaluation configuration\. Judge calls use temperature 0\. Stockfish 14\.1 serves as the backend engine for every tool call\. In generation, we use depth\-22 analysis, while verification tool calls run at depth\-20 by default\. Confidence intervals\. All intervals reported in Table 2 are 95%95\\% bootstrap intervals over 10,00010\{,\}000 resamples, computed at the position level so that atoms drawn from the same commentary are resampled together\. C\.3 Qwen3 Thinking\-Mode Degeneracy Both Qwen3 models are run in No\-Think mode in the main experiments\. This choice was made because thinking mode usually failed to produce any commentary under the completion budget used throughout the pipeline, not because it produced less favourable scores\. We tested both models with thinking enabled under the same max\_tokens =2048=2048 budget applied to every other model\. Across N=50N=50 preliminary positions, the budget was frequently exhausted entirely inside <think\>…</think\> before any final commentary was emitted: Qwen3\-8B reached no final answer on 46/5046/50 positions \(92%92\\%\), and Qwen3\-32B on 31/5031/50 positions \(62%62\\%\)\. Because ACT\-Eval requires a commentary to evaluate, positions with no final answer cannot be scored at all\. A representative failure is shown below\. The model spends the full budget re\-deriving the contents of a single rank from the FEN and second\-guessing an off\-by\-one square count, and never reaches the move under discussion\. Model: Qwen3\-32B, thinking mode\. Move: Rc8\. Generation length: 5,090 characters\. Position: r2q1rk1/1b1nbpp1/p1p1pn1p/1p6/3P3B/1BN1PN2/PP2QPPP/2RR2K1 b \- \- 5 13 <think\> Okay, let’s analyze this position\. The user provided the FEN and the move Rc8\. First, I need to understand the position\. \[\.\.\.\] The move played is Rc8\. I should start by visualizing the board\. \[\.\.\.\] I’ll use get\_piece\_at to see what’s on c8 before the move\. The original position’s c8 is empty because the FEN shows ’r2q1rk1’ for the first rank, so c8 is a rook? Wait, no, the FEN is from White’s perspective\. \[\.\.\.\] Let’s split each rank: 8th rank: r2q1rk1 a8: rook, b8: empty, c8: empty, d8: queen, e8: empty, f8: rook, g8: knight, h8: king? Wait, that doesn’t add up\. Let me count the pieces properly\. \[\.\.\.\] r \(1\) \+ 2 \(empty\) \+ q \(1\) \+ 1 \(empty\) \+ r \(1\) \+ k \(1\) = 1\+2\+1\+1\+1\+1=7\. So there’s a missing square\. Maybe I made a mistake\. \[\.\.\.\] \[generation continues re\-deriving the same rank repeatedly and is cut off at the token limit before ever discussing Rc8\] The failure is itself an instance of the board\-comprehension weakness documented in Section 6\.1: the model cannot stably parse a FEN into a board state, and additional reasoning tokens are consumed by the resulting self\-correction loop rather than by analysis of the move\. Reported Qwen3 numbers should nevertheless be read as reflecting the No\-Think configuration specifically\. Appendix D Judge Analysis D\.1 Cross\-Judge Robustness Table 6: Cross\-judge agreement\. The same generator outputs scored by two different judge models \(GPT\-5\.4 vs\. Gemini 3\.1 Pro\), on the positions both judges scored \(n=225n\{=\}225; Recall over n≈125n\{\\approx\}125\)\. Bracketed ranges are 95%95\\% position\-clustered bootstrap CIs ACT\-Eval uses GPT\-5\.4 as its judge, and GPT\-5\.4 is also one of the evaluated generators\. To test whether this introduces a same\-model preference, we reran the full pipeline with Gemini 3\.1 Pro as an independent judge \(Table 6\)\. Both judges produce the same ordering on factual error rate: Gemini 3\.1 Pro is strongest, followed by GPT\-5\.4, then DeepSeek V4 Pro\. Under the GPT\-5\.4 judge, Gemini 7\.5%7\.5\\% << GPT\-5\.4 9\.5%9\.5\\% << DeepSeek 14\.3%14\.3\\%; under the Gemini judge, Gemini 3\.2%3\.2\\% << GPT\-5\.4 12\.8%12\.8\\% << DeepSeek 15\.3%15\.3\\%\. GPT\-5\.4 does not rank its own commentary first under either judge, and Gemini ranks first under both, so the central model comparison is robust to the choice of judge\. The absolute rates nevertheless differ across judges: ACT\-Eval scores are judge\-calibrated rather than judge\-invariant, and we report all main results under a single judge\. Section D\.2 analyses the disagreements between the two judges\. D\.2 Residual Judge Errors Tool augmentation mitigates rather than eliminates hallucination\. The judge must still select relevant evidence, choose which tools to call, and interpret what they return, and errors in any of these persist even when the underlying tool output is correct\. The cross\-judge experiment \(Section D\.1\) makes this concrete, since Gemini 3\.1 Pro is the stronger chess model in our generation experiments and therefore a useful second opinion on the same claims\. The GPT\-5\.4 judge misses errors\. Our human calibration \(Section 5\) finds an asymmetry: when the GPT\-5\.4 judge flags a claim as incorrect, expert annotators agree in 8383–94%94\\% of cases, but agreement with the claims it accepts falls to 1414–57%57\\%\. The cross\-judge results are consistent with this\. The Gemini judge assigns higher error rates to both GPT\-5\.4 and DeepSeek commentary, and inspection of the disagreements shows cases where it identifies factual and variation errors that GPT\-5\.4 had accepted\. These disagreements show that absolute error rates remain judge\-dependent: Gemini identifies some errors accepted by GPT\-5\.4, while GPT\-5\.4 also rejects some valid claims\. The GPT\-5\.4 judge also rejects valid claims\. Gemini\-generated commentary receives a lower error rate from Gemini than from GPT\-5\.4, and inspection finds several cases where GPT\-5\.4 rejected valid claims by misinterpreting tool output\. These concentrate on practical chess statements using terms such as “cannot prevent”, “forced”, or “must”, where the intended meaning is tactical rather than literal\. Consider the claim “After …Nc6, White cannot prevent …Nb4”\. GPT\-5\.4 observed that …Nb4 remained legal after several sampled White replies and used this to mark the claim incorrect — reversing the evidence, since the continued availability of …Nb4 is what makes the claim true\. Under our verification rubric, “cannot prevent” means that White has no satisfactory prevention, not that every preventive move is illegal\. Gemini instead evaluated the candidate preventive move a3, found that it substantially worsens White’s position, and concluded that White cannot prevent the maneuver without a serious concession\. In both directions the deterministic tools returned correct evidence; what failed was the verification procedure the judge chose and the reading it gave the result\. Residual errors therefore arise during claim interpretation, tool selection, board\-state tracking, and evidence aggregation, and ACT\-Eval continues to inherit model\-dependent judge errors\. D\.3 Comparison with vanilla LLM\-as\-a\-judge Prior work Kim et al\. \(2025\) on chess commentary evaluation relies on a vanilla LLM\-as\-a\-judge framework, where the judge, given raw engine outputs and human ground\-truth blurb, is prompted to score the commentary on a 1\-5 scale for four metrics\. Among them relevance \(ensuring the text only includes information relevant to the move\) and completeness \(ensuring all critical points on the board are covered\) are the main metrics for chess\-specific evaluation\. In contrast, ACT\-Eval reformulates both subjective dimensions into more deterministic, verifiable metrics\. We define factual precision \(captured via claim accuracy and error rate\) to replace relevance\. A confidently\-stated hallucination could be fully relevant to the reference comment, as we can see in Figure 1, the generated commentary receives a near\-perfect GCC\-Eval score of 4\.9/5, despite ACT\-Eval revealing a 66\.7% error rate where most atoms contain factually incorrect statements\. In our framework, commentary is decomposed into atomic claims, each verified by a suite of tools supported by engines and board\-state, and the score reflects the fraction of claims that are factually correct\. This shifts the metric from a topical filter to a factual check\. Similarly, we replace completeness with atomic recall against expert\-verified gold atoms\. Instead of relying on the judge, which lacks internal expert chess understanding, to define what is critical behind a move, we measure coverage against grandmaster\-level annotations through atom\-to\-atom comparison\. We argue that high\-quality commentary should do more than vaguely survey the board, it must successfully retrieve the specific, expert\-verified tactical and strategic insights\. Appendix E More Related Work Chess commentary generation\. Prior work has learned move\-by\-move explanations from large\-scale social\-forum data, incorporated neural and symbolic chess engines into generation, and jointly modeled chess policy and language \(Jhamtani et al\., 2018; Zang et al\., 2019; Lee et al\., 2022; Feng et al\., 2023\); GCC\-Eval subsequently evaluates generated commentary using an LLM judge conditioned on reference commentary and engine analysis \(Kim et al\., 2025\)\. Text generation evaluation\. Evaluating natural language generation requires multi\-dimensional metrics tailored to specific domains\. Traditional linguistic metrics quantify surface\-level fluency and lexical overlap against human references Papineni et al\. \(2002\); Lin \(2004\); Zhang et al\. \(2019\)\. But these reference\-based metrics correlate poorly with human judgement on tasks that depend on factual correctness, motivating a separate line of work on factuality\. Early work frames the problem as entailment or question\-answering against a source Kryściński et al\. \(2020\); Durmus et al\. \(2020\); Fabbri et al\. \(2022\); more recent work proposed a decompose\-and\-verify paradigm, in which generations are split into atomic claims and checked against a knowledge database Min et al\. \(2023a\); Chern et al\. \(2023\); Wei et al\. \(2024\)\. Chess commentary is a highly specialized domain where these prior works cannot fully cover the necessary depth\. First, verifying factual correctness in chess requires routing each claim to the appropriate computational primitive rather than retrieving textual evidence\. Second, commentary quality is not exhausted by factual correctness, it must also capture the nuanced strategic understanding behind a move\. LLM\-as\-a\-judge in specialized domains\. Using LLM judges to evaluate text generation quality has become standard practice\. While these judges show strong agreement with humans on general tasks Liu et al\. \(2023\), their performance degrades sharply in domains requiring expert knowledge Szymanski et al\. \(2025\)\. In chess, GCC\-Eval Kim et al\. \(2025\) is the closest prior work; it attempts to compensate for this knowledge gap by augmenting the LLM judge with reference commentary and engine analysis to compensate for the lack of chess knowledge\. However, this approach remains insufficient since the reference annotations are noisy and conflate fact with stylistic flourish, while engine analysis conveys only raw numerical analysis without deeper guidance or interpretation\. In comparison, ACT\-Eval addresses both shortcomings, we construct expert\-verified gold atoms as reference, and constrain the judge to tool calls for verifiable fact checking\. Appendix F Human Calibration Interface Figure 4: Screenshot of ACT\-Eval Human Calibration Study Figure 5: Screenshot of Study 1 in Human Calibration Figure 6: Screenshot of Study 2 in Human Calibration`