以国际象棋对大语言模型提示词优化进行基准测试

# Benchmarking Prompt Optimization of Large Language Models With Chess
Source: [https://arxiv.org/html/2610.00416](https://arxiv.org/html/2610.00416)
Alejandra López de Aberasturi GómezTristan KarchTom VeniatAffiliation:Philippe Modard, Karl Tuyls & Ludovic DenoyerAffiliation:imec AI\.labsEmail:[\{first\.last\}@imec\.be](mailto:)

###### Abstract

Evaluating large language models becomes increasingly challenging as their capabilities advance: benchmarks can saturate, public test sets risk contamination, and assessing harder tasks can require expensive grading or execution infrastructure\. These challenges are amplified in automatic prompt optimization \(APO\), where evaluation is repeated throughout the search for better prompts\. Studying APO therefore requires a benchmark that is cheap and deterministic to score, hard enough to leave room for improvement, and renewable as models evolve\. We introduce a chess benchmark built from 1,118Lichesspuzzles to study APO for frozen LLMs: we optimize their prompts without updating their model weights\. Chess combines inexpensive exact\-match scoring, engine\-based evaluation of alternative moves, and a renewable supply of problems with adjustable difficulty\. Unlike evaluations that report only success on isolated test items, the benchmark also connects puzzle\-solving gains to short game\-play rollouts within the same domain\. We use it to evaluate six APO algorithms on eight target models, measuring not only baseline strength but also how much each model responds to optimization and whether optimized prompts transfer across models and to game play\. Chess is thus a well\-suited benchmark for APO: it is \(i\)*challenging*, as even the strongest evaluated model, Gemini 3\.5 Flash \(used as the meta\-model\), solves only about 55% of puzzles; \(ii\)*discriminative*, revealing gains, unchanged performance, and regressions across methods and models; \(iii\)*renewable*, with fresh puzzles to reduce contamination risk and adjustable difficulty to maintain headroom as models improve; and \(iv\)*affordable*, as the complete study runs for around $800\. We release the puzzles, optimization and evaluation code, and dataset\-renewal scripts \([Code](https://github.com/imec-ailabs/Automatic-Prompt-Optimization-with-Chess)\)\.

## 1Introduction

Prompt optimization offers a fast, inexpensive way to improve LLM performance on downstream tasks\. Automatic prompt optimization \(APO\)\[[Ramnath et al\., 2025](https://arxiv.org/html/2610.00416#bib.bib28)\]refers to methods that automatically search for high\-performing prompts, e\.g\., for instruction generation[Zhou et al\. \[2023\]](https://arxiv.org/html/2610.00416#bib.bib23)\. This reduces the need for costly manual prompt engineering and speeds up the adoption of new models\. Like fine\-tuning, APO adapts a trained model to a downstream task, but it only requires black\-box access to the model, allowing it to be applied to both open\-weight and proprietary models and to be combined with supervised fine\-tuning orRLHF\.

The main application domains of automatic prompt optimization are text classification, summarization, and reasoning\. But evaluatingAPOinherits every weakness of LLM benchmarking itself: static test sets leak into training data, frontier benchmarks demand heavy infrastructure, grading open\-ended outputs requires costly LLM judges, and once the models saturate the task, the benchmarks become obsolete\. This motivates benchmarks with cheap, deterministic scoring and hard problems that can be renewed as models evolve\.

We use chess puzzles to benchmark and understandAPO\. Chess has properties that directly address the shortcomings identified above: \(a\)inexpensive automated scoring—reference puzzle solutions provide exact\-match labels, while engines such asStockfish[The Stockfish Developers \[2026\]](https://arxiv.org/html/2610.00416#bib.bib7),Leela Chess Zero[Pascutto et al\. \[2022\]](https://arxiv.org/html/2610.00416#bib.bib49),Komodo[Dailey et al\. \[2023\]](https://arxiv.org/html/2610.00416#bib.bib48), orAlphaZero[Silver et al\. \[2018a\]](https://arxiv.org/html/2610.00416#bib.bib47)evaluate alternative moves without human annotation or an LLM judge; \(b\)renewable problems with tunable difficulty—the large space of positions supplies challenging problems\[[Kolasani et al\., 2025](https://arxiv.org/html/2610.00416#bib.bib14)\], and scripts can construct fresh puzzle sets from new games and adjust their difficulty as models improve; and \(c\)transfer within the same domain—prompts optimized on isolated puzzle positions can also be evaluated through their consequences during play\. Puzzles can additionally be generated using RL approaches to target novelty and difficulty\[[Feng et al\., 2025](https://arxiv.org/html/2610.00416#bib.bib42)\], at greater computational cost\. Figure[1](https://arxiv.org/html/2610.00416#S1.F1)illustrates the benchmark: a frozen LLM receives a chess position as text, and an optimizer updates its instructions and demonstrations using feedback on predicted moves\.

\(a\) Chess puzzle\(b\) Prompt construction and optimizationWhite to move6 optimizers×\\times8 frozen LLMs\\chessboard \[ setfen=r1bqkb1r/pppp1ppp/2n2n2/4p2Q/2B1P3/8/PPPP1PPP/RNB1K1NR w KQkq \- 4 4, boardfontsize=20pt, showmover=false, color=overviewgreen, pgfstyle=straightmove, markmoves=h5\-f7 \] FEN of this boardr1bqkb1r/pppp1ppp/2n2n2/4p2Q/2B1P3/8/PPPP1PPP/RNB1K1NRw KQkq \- 4 4Legal moves \(UCI, sorted\)a2a3,…\\ldots,h5h7Referencey=Qxf7\#y=\\texttt\{Qxf7\\\#\}Assembled PromptJSONAdapter: fixed input/output formatTask instructions \(part ofpp\)\{task instruction\}Demonstrations \(part ofpp\)\{examples\}initially emptyCurrent positionxxPosition Fen: \{fen\}Side To Move: \{side\}Legal Moves: \{legal\_moves\}Frozen LLMy^=f⁡\(x,θ,p\)\\hat\{y\}=f\(x;\\theta,p\)weightsθ\\thetafixedLossℓ⁡\(y^,y\)\\ell\(\\hat\{y\},y\)compare with reference\+ engine evaluationOptimizerrevise instructionsand/or demonstrationsp,xp,xmovey^\\hat\{y\}score / feedbackupdateppxxyy

Figure 1:Chess puzzles as a prompt\-optimization benchmark\.\(a\)A positionxx, its exact FEN \(line\-wrapped\), an abbreviated sorted list of legal moves, and reference moveyy\.\(b\)The template assembles instructions, demonstrations \(initially empty\), and the current position fields; braces denote placeholders\. APO optimizes the task instructions within the system message and/or demonstrations; the current board and game information are supplied in the user message\. The exact template is given in Appendix[B\.4](https://arxiv.org/html/2610.00416#A2.SS4)\. The warm\-colored region and arrow highlight the update ofpp, whileθ\\thetastays frozen\. Demonstrations include answers, whereas the current referenceyyis used only for scoring\. The optimizer uses training scores and, for reflective methods, textual feedback\. The selectedp∗p^\{\*\}is evaluated on held\-out puzzles using accuracy and engine\-based regret\.Contributions\.This paper introduces a new chess benchmark together with an evaluation of six APO algorithms across eight proprietary and open\-weight target models\. It aims to better understand how prompt optimization behaves across algorithms and models while providing a reproducible way to execute new APO algorithms at a reasonable cost\. Our contributions are thus three\-fold: \(a\) a new APO benchmark based on chess puzzles that is easy to run, cheap to evaluate, and challenging yet improvable; \(b\) a comparison of six APO approaches across eight target models, analyzed along puzzle metadata, prompt composition, engine evaluation, and transferability to other models and game\-play rollouts; and \(c\) an analysis of where optimization gains occur and how they transfer across models and to game play\. In addition, we provide an open\-source platform with a curated set of 1,118 puzzles, code to optimize prompts through various standard APO methods, and scripts to refresh the benchmark to reduce dependence on fixed public test sets and adjust difficulty as LLM frontier models improve\.

## 2Related Work

We position our benchmark relative to three lines of work: benchmark construction for frontier LLMs, chess as a testbed for LLMs, andAPOalgorithms\.

Frontier LLMs Benchmarks\.APO is typically evaluated on standard LLM benchmarks and inherits their limitations\. As frontier LLMs acquire stronger reasoning capabilities, traditional benchmarks struggle to keep pace and are prone to data saturation and contamination\[[Kiela et al\., 2021](https://arxiv.org/html/2610.00416#bib.bib39),[Singh et al\., 2024](https://arxiv.org/html/2610.00416#bib.bib3),[Bean et al\., 2026](https://arxiv.org/html/2610.00416#bib.bib2),[Marro et al\., 2026](https://arxiv.org/html/2610.00416#bib.bib40)\]\. Moreover, as tasks get more complicated, it becomes harder for humans to produce reliable annotations\.[Marro et al\. \[2026\]](https://arxiv.org/html/2610.00416#bib.bib40)describe this as the “post\-comprehension regime”\. One response is to generate test problems dynamically instead of fixing a dataset, as inLiveBench\[[White et al\., 2025](https://arxiv.org/html/2610.00416#bib.bib4)\],RULER\[[Hsieh et al\., 2024](https://arxiv.org/html/2610.00416#bib.bib44)\],LLF\-Bench\[[Cheng et al\., 2023](https://arxiv.org/html/2610.00416#bib.bib32)\], andLoCoMo\[[Maharana et al\., 2024](https://arxiv.org/html/2610.00416#bib.bib31)\]\. Our benchmark belongs to this renewable\-task family\. Its puzzle task supports inexpensive exact\-match scoring and graded engine evaluation without an LLM judge or an agentic execution environment\. This makes repeated prompt evaluation affordable, complementing benchmarks that study open\-ended responses with LLM judges\[[Zheng et al\., 2023](https://arxiv.org/html/2610.00416#bib.bib41),[Maharana et al\., 2024](https://arxiv.org/html/2610.00416#bib.bib31)\], agentic tasks in sandboxes\[[Merrill et al\., 2026](https://arxiv.org/html/2610.00416#bib.bib30),[Chan et al\., 2025](https://arxiv.org/html/2610.00416#bib.bib35)\], or interactive performance in arenas\[[Zheng et al\., 2023](https://arxiv.org/html/2610.00416#bib.bib41),[Guertler et al\., 2025](https://arxiv.org/html/2610.00416#bib.bib18),[Liu et al\., 2026](https://arxiv.org/html/2610.00416#bib.bib13)\]\.

Chess with LLMs\.Chess has served as a testbed for artificial intelligence for decades, fromDeep Blue\[[Campbell et al\., 2002](https://arxiv.org/html/2610.00416#bib.bib1)\]toAlphaZero[Silver et al\. \[2018b\]](https://arxiv.org/html/2610.00416#bib.bib5)andLeela Chess Zero\[[The LCZero Authors, 2026](https://arxiv.org/html/2610.00416#bib.bib6)\]\. Existing work uses chess to study LLM capabilities in several ways\. Some studies internalize chess behavior by pre\-training or heavily fine\-tuning model weights on offline game trajectories and state evaluations\[[Noever et al\., 2020](https://arxiv.org/html/2610.00416#bib.bib43),[Feng et al\., 2023](https://arxiv.org/html/2610.00416#bib.bib11),[Zhang et al\., 2025](https://arxiv.org/html/2610.00416#bib.bib15)\]\. In a post\-training alternative,[Hwang et al\. \[2025\]](https://arxiv.org/html/2610.00416#bib.bib16)found that reinforcement learning plateaus at25​–​30%25\\text\{\-\-\}30\\%puzzle accuracy, and hypothesized that generic base models lack the foundational state\-tracking representations required for RL to build upon\. Separately,[Karvonen \[2024\]](https://arxiv.org/html/2610.00416#bib.bib12)used probes to study internal representations of board\-state and latent estimates of opponent skill in models trained in chess transcripts\. Our work instead treats chess as a text\-based reasoning task for general\-purpose frozen frontier language models\. Previous studies show that model performance is sensitive to the representation of the board, such asFENstrings orPGNmove lists, and to the prompt structure\[[Acher, 2023](https://arxiv.org/html/2610.00416#bib.bib20),[Kolasani et al\., 2025](https://arxiv.org/html/2610.00416#bib.bib14),[Cooper, 2025](https://arxiv.org/html/2610.00416#bib.bib17)\]\. Manual prompt engineering involving metadata headers, few\-shot exemplars, orPGNstyles can therefore produce substantial performance differences\[[dynomight, 2024](https://arxiv.org/html/2610.00416#bib.bib21),[Witten, 2023](https://arxiv.org/html/2610.00416#bib.bib19)\]\. Rather than relying on targeted manual interventions, we automate prompt optimization to provide a shared testbed for evaluating Automatic Prompt Optimization \(APO\) algorithms\.

Automatic Prompt Optimization \(APO\)\.APO adapts LLM behavior without updating the underlying parameters\[[Ramnath et al\., 2025](https://arxiv.org/html/2610.00416#bib.bib28)\]\.APOapproaches typically use LLMs to generate, mutate, and select instructions or demonstrations through discrete search, gradient\-free editing, or evolutionary algorithms\[[Jiang et al\., 2020](https://arxiv.org/html/2610.00416#bib.bib22),[Zhou et al\., 2023](https://arxiv.org/html/2610.00416#bib.bib23),[Prasad et al\., 2023](https://arxiv.org/html/2610.00416#bib.bib24),[Pryzant et al\., 2023](https://arxiv.org/html/2610.00416#bib.bib25),[Yang et al\., 2024](https://arxiv.org/html/2610.00416#bib.bib26)\]\. Frameworks such asDSPy\[[Khattab et al\., 2024](https://arxiv.org/html/2610.00416#bib.bib10)\]compile declarative pipelines with optimizers includingMIPROv2\[[Opsahl\-Ong et al\., 2024](https://arxiv.org/html/2610.00416#bib.bib27)\],GEPA\[[Agrawal et al\., 2026](https://arxiv.org/html/2610.00416#bib.bib9)\], andSIMBA\[[Khattab et al\., 2024](https://arxiv.org/html/2610.00416#bib.bib10)\]to maximize downstream metrics\.

Although existingAPOliterature highlights gains in tasks like text classification[Du et al\. \[2026\]](https://arxiv.org/html/2610.00416#bib.bib37), question answering[Yang et al\. \[2018\]](https://arxiv.org/html/2610.00416#bib.bib34), and mathematical reasoning\[[Ramnath et al\., 2025](https://arxiv.org/html/2610.00416#bib.bib28),[Cobbe et al\., 2021](https://arxiv.org/html/2610.00416#bib.bib33),[Zhang and Math\-AI, 2025](https://arxiv.org/html/2610.00416#bib.bib50)\], games offer a complementary setting in which optimized behavior can be evaluated both on isolated decisions and through their consequences during play\. WhereasAutoharness\[[Lou et al\., 2026](https://arxiv.org/html/2610.00416#bib.bib29)\]synthesizes helper code to improve gameplay, we isolate the automatic optimization of instructions and demonstrations for frozen LLMs\.

## 3Chess Puzzle Benchmark

We use chess to study prompt optimization for general\-purpose language models rather than models specifically fine\-tuned to play chess games\[[Lehrach et al\., 2025](https://arxiv.org/html/2610.00416#bib.bib8),[Feng et al\., 2023](https://arxiv.org/html/2610.00416#bib.bib11),[Zhang et al\., 2025](https://arxiv.org/html/2610.00416#bib.bib15)\]\.

Prompt Optimization with Chess\.Letf⁡\(x,θ,p\)f\(x;\\theta,p\)denote the prediction of a model with parametersθ\\thetaand promptppfor an inputxx\. The lossℓ⁡\(f⁡\(x,θ,p\),y\)\\ell\(f\(x;\\theta,p\),y\)measures its error relative to the reference labelyy\. In APO,ffis an LLM whose parametersθ\\thetaremain frozen\. The goal is to find the promptp∗=arg⁡min⁡∑\(x,y\)p⁡ℓ⁡\(f⁡\(x,θ,p\),y\)p^\{\*\}=\\arg\\min\_\{p\}\\sum\_\{\(x,y\)\}\\ell\(f\(x;\\theta,p\),y\)\. Figure[1](https://arxiv.org/html/2610.00416#S1.F1)\(b\) illustrates this objective\.

The success of APO depends partly on the quality ofℓ\\ell\. A useful scoring function should \(1\) apply to new inputs and predictions, \(2\) be cheap, since it sits inside the optimization loop, and \(3\) provide unambiguous feedback, ideally with graded scores\. Few benchmarks offer all three properties at once\. Open\-ended tasks often require costly LLM judgements, while verifiable domains such as math and code typically use binary rewards and require effort to extend their problem sets\.

Chess offers all three: an engine can evaluate predicted moves in new positions, cheaply and with graded, reproducible scores\. The same engine can supply labels for fresh positions, allowing the benchmark to be renewed to reduce contamination risk and grow in difficulty as models improve\.

Chess Puzzle Representation and Evaluation\.A puzzle comprisesNNposition–move pairs\(xi,yi\)i=0N−1\(x\_\{i\},y\_\{i\}\)\_\{i=0\}^\{N\-1\}along its reference solution\. We query each position independently; the puzzle is solved iff⁡\(xi,θ,p\)=yif\(x\_\{i\};\\theta,p\)=y\_\{i\}for everyii\. The default loss is the exact\-match error; Appendix[B\.2](https://arxiv.org/html/2610.00416#A2.SS2)adds a graded engine\-based evaluation\.

Following[Feng et al\. \[2023\]](https://arxiv.org/html/2610.00416#bib.bib11),[Kolasani et al\. \[2025\]](https://arxiv.org/html/2610.00416#bib.bib14), the inputs contain the position in the Forsyth–Edwards notation \(FEN\), the side to move and explicit legal moves\[[Hwang et al\., 2025](https://arxiv.org/html/2610.00416#bib.bib16)\]\. Although representation preferences vary between models\[[Kolasani et al\., 2025](https://arxiv.org/html/2610.00416#bib.bib14)\], our preliminary experiments confirmedFENas an effective input representation for our models\. The model returns one legal move inSANorUCI, wrapped in JSON\. A response that violates this formatting cannot be parsed and is scored as incorrect\. See Figure[1](https://arxiv.org/html/2610.00416#S1.F1)for input construction and Appendices[A\.1](https://arxiv.org/html/2610.00416#A1.SS1)and[B\.1](https://arxiv.org/html/2610.00416#A2.SS1)for notation and prompt examples\.

Dataset Construction and Difficulty\.We construct the benchmark from 1,118 puzzles extracted from theLichessdatabase, whose exports are released under CC0 1\.0 Universal111[https://database\.lichess\.org/](https://database.lichess.org/)\. We divide the puzzles equally into training and test sets of 559 puzzles\. The puzzles are stratified along two axes: the number of puzzle moves \(one to five\) and the puzzle elo rating \(400 to 2,800\)\. The complete distribution across the length and rating of the puzzle can be found in the Appendix[A\.3](https://arxiv.org/html/2610.00416#A1.SS3)\. Because the current benchmark is drawn from the publicLichessdatabase, we cannot exclude pretraining exposure\. Its advantage is therefore not guaranteed absence of contamination, but renewability: the same pipeline can construct evaluation sets from newer games and positions\.Lichessratings estimate difficulty for human solvers and also organize our LLM results: higher\-rated puzzles are solved less often \(Appendix[B\.5](https://arxiv.org/html/2610.00416#A2.SS5)\)\. In this dataset, one\- and two\-move puzzles occur below 2000, while three\- to five\-move puzzles occur above 2000\. Rating and length are therefore descriptive axes, not independent difficulty factors or estimates of an LLM’s playing rating\.

Preliminary experiments showed that recent models do not saturate our puzzle database, making the current set appropriate for APO evaluation\.

## 4Experimental Setup

We compare sixAPOalgorithms across eight target models, using a ninth model, Gemini 3\.5 Flash, as the fixed meta\-model for prompt proposal and reflection\. All target models start from the same instruction:*“Solve the chess puzzle by finding the single best move\.”*The benchmark contains 559 development puzzles and 559 test puzzles\. Candidate selection uses the development data according to each optimizer’s configuration\. We describe the optimizers and feedback below, followed by the model panel and baseline evaluations\.

### 4\.1Algorithms

We select six optimizers that cover complementary prompt\-improvement strategies: some optimize instructions, some optimize demonstrations, and some jointly optimize both components\. They also differ in the type of feedback they exploit and in the search mechanism used to identify improved prompts\.

Genetic\-Pareto \(GEPA\)\[[Agrawal et al\., 2026](https://arxiv.org/html/2610.00416#bib.bib9)\]:GEPAemploys a genetic\-reflective optimization strategy that couples prompt mutation with Pareto\-inspired selection\. It constructs a Pareto frontier consisting of prompts that perform best on specific subsets of the evaluation set and iteratively evolves the candidates on this frontier\. Crucially,GEPAuses execution traces together with textual and numerical feedback to diagnose errors and guide prompt mutations, allowing the search to exploit both qualitative information about model failures and quantitative performance signals\.

Cooperative Prompt Optimization \(COPRO\)\[[Khattab et al\., 2024](https://arxiv.org/html/2610.00416#bib.bib10)\]:COPROoptimizes the instruction component of a prompt through an iterative coordinate\-ascent\-style search\. A meta\-model generates multiple candidate instruction variants, which are evaluated using a scalar performance metric\. The best\-performing candidates are retained and subsequently refined, producing an iterative search over instruction space driven exclusively by numerical feedback\.

BootstrapFewShot \(BFS\)\[[Khattab et al\., 2024](https://arxiv.org/html/2610.00416#bib.bib10)\]:BFSimproves prompts by augmenting them with few\-shot demonstrations obtained through successful trace bootstrapping\. The optimizer executes the program over training examples and retains intermediate execution traces for which the model produces correct outputs\. These successful traces are then incorporated into the prompt as demonstrations, making the optimization process dependent on numerical feedback while leaving the instruction component unchanged\.

BootstrapRandomSearch \(BRS\)\[[Khattab et al\., 2024](https://arxiv.org/html/2610.00416#bib.bib10)\]:LikeBFS,BRSoptimizes prompts by selecting few\-shot demonstrations\. It performs a random search over alternative demonstration sets by repeatedly sampling different combinations of demonstrations\. It evaluates each configuration on the validation set, and retains those achieving the highest numerical performance\. This allowsBRSto explore a broader space of possible demonstration configurations than standard bootstrapping\.

Multiprompt Instruction Proposal Optimizer Version 2 \(MIPROv2\)\[[Opsahl\-Ong et al\., 2024](https://arxiv.org/html/2610.00416#bib.bib27)\]:MIPROv2jointly optimizes prompt instructions and few\-shot demonstrations\. It first proposes a diverse set of contextualized candidate instructions based on information about the task and program structure, while also bootstrapping examples that the model can solve successfully\. It then formulates prompt selection as a joint search problem and uses Bayesian optimization to efficiently explore combinations of candidate instructions and demonstrations\. Each configuration is evaluated using a numerical validation metric, and the search progressively focuses on combinations expected to yield higher performance\.

Stochastic Introspective Mini\-Batch Ascent \(SIMBA\)\[[Khattab et al\., 2024](https://arxiv.org/html/2610.00416#bib.bib10)\]:SIMBA222[https://blog\.mariusvach\.com/posts/dspy\-simba](https://blog.mariusvach.com/posts/dspy-simba)combines few\-shot demonstration selection with stochastic introspective optimization of instructions\. It samples mini\-batches and queries the LLM under different temperature settings to generate diverse execution traces\. By comparing successful and unsuccessful traces, as well as identifying examples with high output variability, the optimizer uses self\-reflection to derive explicit heuristics or rules that can be incorporated into the prompt\. Alternatively, successful traces can be added as demonstrations\. In this way,SIMBAexploits both textual feedback derived from introspection and numerical feedback from the evaluation metric while jointly modifying instructions and examples\.

Table 1:Summary of optimization algorithms\.optimization scope \(what prompt components are optimized\), feedback types \(how the algorithm guides the search\), and whether a separate meta\-model proposes prompt changes\.Optimizers receive two types of feedback: textual and scalar\. Textual feedback first classifies failures as parsing errors, illegal outputs, or valid but incorrect answers, and then adds task\-specific guidance; when appropriate, the feedback also includes the expected move,FEN, or legal\-move set\. For example, after a legal but incorrect puzzle move, the feedback states: “The move was legal, but did not solve the tactic\. Reassess forcing checks, captures, threats, and replies\.”\. Non\-reflective algorithms only rely on the scalar score, i\.e\. the puzzle accuracy\. A summary of which algorithm uses which type of feedback can be found in Table[1](https://arxiv.org/html/2610.00416#S4.T1)\.

We evaluate each optimizer using its standard recommended configuration\. Consequently, the comparison reflects out\-of\-the\-box behavior rather than performance under equalized compute budgets or optimizer\-specific hyperparameter tuning\.

### 4\.2Models and Baseline Evaluations

Our evaluations span multiple LLM providers: Google, DeepSeek, Anthropic, OpenAI, Alibaba Group and Meta\. We chose the models based on cost, provider diversity and rank in the Kaggle Game Arena333[https://www\.kaggle\.com/benchmarks/kaggle/chess\-text/versions/2](https://www.kaggle.com/benchmarks/kaggle/chess-text/versions/2)\.

Table 2:Baseline prompt evaluation on the 559\-puzzle test set, reported as mean±\\pmsample standard deviation over three runs\.Input and output are per solver\-position call; cost is normalized per 1,000 solver\-position calls\. Reasoning: thinking level passed to the model;*default*: provider’s default;*N/A*: no reasoning mode\. Jev is a decision endpoint, so generative max\-output and output\-token columns are not comparable\.Table[2](https://arxiv.org/html/2610.00416#S4.T2)reports the characteristics of the selected models, including price, at the time of the experiments\. Whenever possible,`reasoning`was set tolow\. We calibrated`max\_output\_tokens`with preliminary tests to avoid output truncation\. Note that Jev 1\.13 operates differently from the other models\. Instead of generating free text, it acts as a zero\-shot classifier that chooses among the legal moves \(Appendix[B\.6](https://arxiv.org/html/2610.00416#A2.SS6)\)\. It therefore cannot produce illegal moves, and token\-based columns do not apply to it\.

Table[2](https://arxiv.org/html/2610.00416#S4.T2)also reports the baseline results with the base prompt on three seeds in the test set\. Cost and latency vary widely across models\. For example, GPT\-5\.6 Luna and Muse Spark 1\.2 Contributor, while having low nominal prices \(input $0\.2 and $0\.1 per M tokens, output $1\.2 and $0\.2 per M tokens, respectively\), produce longer outputs under the evaluated configurations, contributing to higher cost and latency\. On this benchmark, Gemini 3\.5 Flash Lite offers a favorable price\-performance trade\-off, with 26\.30% accuracy at $0\.169 per 1k calls\. The best model, Gemini 3\.5 Flash, is kept as the fixed meta\-model for prompt proposal and reflection\. We can see that the illegal move rate varies a lot from one model to another\. The list of legal moves is given, so in theory the model should predict only legal moves\. However, illegal moves can be due to bad formatting\. Tuning the formatting instructions can be a lever on its own to improve performance for some models\.

The common baseline instruction is*“Solve the chess puzzle by finding the single best move\.”*It is accompanied by the position fields and, for generative models, the JSON adapter’s formatting instructions \(Appendix[B\.4](https://arxiv.org/html/2610.00416#A2.SS4)\)\. We deliberately use a minimal common baseline to study what each optimizer can derive from the same starting point across models, rather than whether APO outperforms the strongest model\-specific manual prompts\. Despite its simplicity, this baseline is effective for several models\. Our focus on affordable models is motivated by evidence that prompt optimization can improve inexpensive, weaker models\[[Khattab et al\., 2024](https://arxiv.org/html/2610.00416#bib.bib10)\], potentially offering an alternative to switching to a more expensive model\.

## 5Results

We organize our results around five empirical questions: \(1\) which models improve, \(2\) which puzzles account for the gains, \(3\) what the optimized prompts contain, \(4\) whether they help other models, and \(5\) whether puzzle gains extend to short game\-play rollouts\. The eight\-model puzzle panel comprises 141 optimization runs and 24 baseline evaluations, with around $800 in recorded optimization and evaluation costs\.

### 5\.1Optimization Gains and Evaluation Cost

Table[3](https://arxiv.org/html/2610.00416#S5.T3)reports the results for the eight models and six optimizers, using Gemini 3\.5 Flash as the meta\-model\. Detailed optimization configurations are provided in the Appendix[B\.6](https://arxiv.org/html/2610.00416#A2.SS6)\. Note that SIMBA is not applicable to Jev 1\.13 because it requires changing the temperature to generate outputs, which is not compatible with the 0\-shot classification output of Jev\.

Table 3:Puzzle accuracy \(%, N=559\), reported as mean±\\pmsample standard deviation over three runs per condition\.Bold marks the optimizer with the highest mean for each model\.∗marks improvement over the model’s three\-run baseline under an unadjusted one\-sided Welch test \(p<0\.05p<0\.05\);†marks models with at least one starred optimizer\. N\.U\. marks conditions where none of the three optimizer seeds updated the original prompt\. N\.A\. marks a non\-applicable method\. Gray cells indicate lower mean accuracy than baseline\.\(a\)Evaluation cost versus puzzle accuracy\. Higher cost is impacted by prompt size and length of reasoning\.\(b\)Distribution of Improvement vs Baseline mean in puzzle accuracy by optimizer over all models and seeds\.
Figure 2:Optimizer performance on the chess puzzle benchmark\.Left: mean evaluation cost versus mean puzzle accuracy across three seeds, from the baseline prompt \(open circle\) to each model’s best algorithm \(star\)\. Right: distributions of per\-seed changes in puzzle accuracy from the matched baseline for each optimizer\.Observations\.Responsiveness to prompt optimization changes the model ranking: Gemini 3\.5 Flash Lite overtakes Luna and Muse despite starting below both \(Table[3](https://arxiv.org/html/2610.00416#S5.T3)\)\. The table reports means and sample standard deviations over three runs; stars mark positive gains over the three\-run baseline under a one\-sided Welchtt\-test \(p<0\.05p<0\.05\)\. We observe that most models improve with at least one optimizer, but the best optimizer varies from one model to another, as well as the margin of improvement\. Conditions marked N\.U\. are excluded from substantive optimizer interpretation because the optimizer did not find a better prompt in any of the three seeds and returned the original prompt\. The best improvement is observed for Gemini 3\.5 Flash Lite: 7\.57 percentage points withSIMBA\. Table[3](https://arxiv.org/html/2610.00416#S5.T3)marks at least one significant gain for Jev 1\.13, Qwen3\.8 27B, DeepSeek V4 Pro, and Gemini 3\.5 Flash Lite\. GPT\-4o Mini, Claude Haiku 4\.5, GPT\-5\.6 Luna, and Muse Spark show no significant gain\. For GPT\-5\.6, no approaches improved the mean performance compared to baseline\. BRS and SIMBA were unable to validate any new candidate prompt and returned the original prompt, resulting in no performance improvement\. The highest mean accuracy is obtained withSIMBAfor Flash Lite, GPT\-4o Mini, and Muse Spark, and withGEPAfor Qwen, DeepSeek V4 Pro, Jev 1\.13, and Claude Haiku 4\.5\. Section[5\.4](https://arxiv.org/html/2610.00416#S5.SS4)asks whether the resulting prompts remain useful when the target model changes\.

Figure[2\(a\)](https://arxiv.org/html/2610.00416#S5.F2.sf1)places each model’s baseline and best\-optimizer accuracy against evaluation cost\. Longer instructions and demonstrations increase the input\-token burden \(Table[8](https://arxiv.org/html/2610.00416#A2.T8)\), but total cost also depends on the model’s output length and pricing\. Optimization increases evaluation cost for every model except Spark 1\.2 Contributor, which reduces its cost\. We hypothesize that, since the model was verbose, prompt optimization managed to channel the reasoning and make it more efficient to answer even if it was not optimized for that\. The separate meta\-model, Gemini 3\.5 Flash, remains stronger at 55\.40%\. Under the baseline prompts, Flash Lite’s cost per 1,000 solver\-position calls is approximately one tenth that of GPT\-5\.6 Luna and Muse Spark 1\.2 Contributor\.

Figure[2\(b\)](https://arxiv.org/html/2610.00416#S5.F2.sf2)shows the performance distribution for each algorithm\. Although no optimizer wins everywhere, the distributions reveal distinct patterns on this benchmark under the evaluated configurations\.GEPAhas the highest median gain and improves mean accuracy for 6 out of the 8 models\. UnlikeSIMBA, it also supports the decision endpoint evaluated here because it does not require a tunable temperature\.SIMBAhas the largest mean gain and rarely falls below the baseline, but its mean is mainly driven by large improvements on Gemini 3\.5 Flash Lite\.BRSmatchesGEPA’s mean gain with a lower median and more spread, and its mean is also mainly driven by gains on Gemini 3\.5 Flash Lite\.COPROhas the smallest impact on performance, whileMIPROv2andBFSfall below baseline more often than not, withBFSshowing the largest loss\.

Takeaway\.Baseline accuracy does not tell the whole story\. Flash Lite starts behind Luna and Muse, yet its larger optimization gain moves it ahead of both\. The weakest models, meanwhile, remain far behind despite some significant improvements\. This contrast makes responsiveness to optimization a useful comparison axis alongside baseline strength and cost, consistent with earlier evidence that optimizer performance varies across settings\[[Du et al\., 2026](https://arxiv.org/html/2610.00416#bib.bib37),[Sarmah et al\., 2024](https://arxiv.org/html/2610.00416#bib.bib38)\]\. The benchmark is challenging yet improvable: optimization succeeds in some settings and fails in others, exposing differences between methods and models\. This breadth of comparison is achievable at modest cost\.

### 5\.2Gains Across Puzzle Difficulty and Structure

Do optimization gains concentrate in particular puzzles, or are they spread across the benchmark? We examine puzzle ratings and themes to locate the gains, and solver move indices to distinguish improvement on individual decisions from improvement on complete solutions\. We focus on the three highest\-performing models with significant improvement: Gemini 3\.5 Flash Lite, DeepSeek V4 Pro, and Qwen3\.8 27B, optimized withSIMBA,GEPA, andGEPA, respectively\.

\(a\)Improvements byLichessrating\.\(b\)Improvements by Puzzle themes\.
Figure 3:Puzzle solve rate before and after optimization vs Elo and theme\.We show, for the 3 best models with significant improvement, the results for the best algorithm \(cf Table[3](https://arxiv.org/html/2610.00416#S5.T3)\)\. Open circles mark baseline performance and colored circles mark the optimized prompt; faded thinner segments indicate a decrease\.Figure 4:Accuracy before and after the best\-performing optimizer for each selected model, stratified by puzzle length and solver move index\.Open circles mark baseline performance and colored circles mark the optimized prompt; faded thinner segments indicate a decrease\.Observations\.Figure[3](https://arxiv.org/html/2610.00416#S5.F3)decomposes the improvement by Elo range and theme\. In Figure[3\(a\)](https://arxiv.org/html/2610.00416#S5.F3.sf1), we see that the improvement spreads across a large Elo range but largely stays below 2200 Elo, with only Gemini 3\.5 Flash Lite improving on puzzles between 2000 and 2200 Elo\. The gains are not uniform: some bins are left unchanged or slightly penalized by the new prompt\. For example, Qwen3\.8 27B loses 0\.8 points on puzzles between 1400 and 1600 Elo, and DeepSeek V4 Pro stays exactly flat between 1200 and 1400 Elo\. Figure[3\(b\)](https://arxiv.org/html/2610.00416#S5.F3.sf2)plots the improvement on the 12 most frequent themes in the database\. Aggregated over the three models, the mateIn2 and short themes capture the most improvement, followed by kingsideAttack, mateIn1, oneMove, and mate\. The ranking varies per model, for example Qwen3\.8 27B improves most on mateIn1 and oneMove puzzles\. Note that puzzles have several themes, so improvement on one puzzle may impact multiple themes, for example mate, mateIn1, and mateIn2 may have large overlap\. Figure[4](https://arxiv.org/html/2610.00416#S5.F4)shows the improvement at each individual move index of the puzzles\. The 3\+ move puzzles cover the 2000\+ Elo range\. Despite the low full\-puzzle improvement in this range, per\-move accuracy still improves at most move indexes for all models\. We still observe some exceptions, with Qwen3\.8 27B decreasing on the first move of two\-move puzzles and on the third move of four\-move puzzles\. A significant overall improvement can therefore hide local losses\. We also observe that the last move of a puzzle is often the most accurately solved, most clearly in five\-move puzzles\. Early moves require anticipating the entire solution line, while the last move concludes the tactic, so uncertainty is greatest at the start of a puzzle\.

Takeaway\.Observed gains span several themes and rating ranges, predominantly below a puzzle rating of 2200\. A higher overall score can hide both newly solved puzzles and lost capabilities on particular subsets\. Rating, theme, and move\-index views help locate these trade\-offs\. Longer puzzles pose a particular challenge: improving individual moves is not enough unless those gains combine into a fully correct solution\. Their low solve rates leave room for methods that improve consistency across the entire reference line\.

The gains span several puzzle categories\. We next examine what the optimizers changed in the prompts\.

### 5\.3What Do Optimized Prompts Contain?

To better understand the impact of optimized prompts on performance, we analyze the prompt composition of the best algorithms\. We classify best prompt content into four categories: \(1\)*formatting instructions*that constrain the output structure, \(2\)*chess instructions*that give chess\-specific guidance such as checking forcing moves first, \(3\)*examples*, including illustrative examples and demonstrations and \(4\)*persona framing*that assigns the model a role\. We report the average composition over 3 seeds for each model in Figure[5](https://arxiv.org/html/2610.00416#S5.F5)\. Instruction text is split into sentence\-like units, treating list items as individual units, and labeled with Gemini 3\.1 Flash Lite at temperature 0 without reasoning\. The classifier receives all instruction units from a compiled prompt in one call; stored demonstrations are assigned directly to the examples category\. We map the classifier’s unclassified output to Other\. Character counts in categories preserve the full instruction text, including whitespace, and the displayed demonstration fields\.

Figure 5:Mean composition of the three prompts produced by each model’s best algorithm\.Wedge proportions represent each category’s share of prompt characters; pie area represents total prompt length in characters\.Observations\.We see in Figure[5](https://arxiv.org/html/2610.00416#S5.F5)that prompts produced by the same algorithm have similar compositions across models:SIMBAemphasizes chess guidance and examples, whereasGEPAproduces longer chess instructions together with formatting and persona framing\. Flash Lite achieves the largest gain with prompts averaging just 930 characters: about 78% examples and 22% chess guidance, with no additional formatting instructions\. The selected GEPA prompts average 2,109–3,626 characters, showing that the largest gain does not require the longest prompt\. Other optimizers are not present in the plot but we know from Table[1](https://arxiv.org/html/2610.00416#S4.T1)thatCOPROis instruction\-only, whileBRSandBFSadd only examples to the baseline prompt\. Therefore they have less diversity\.MIPROv2can combine examples and instructions, but it did not produce the best prompt for any model\. Additional details and examples can be found in the Appendix[B\.3](https://arxiv.org/html/2610.00416#A2.SS3)\.

Takeaway\.APO algorithms significantly influence the composition of the prompts\. The algorithms that can combine several content types,SIMBAandGEPA, produce the best results for seven of the eight models\. The best prompts do not simply include more of everything: the largest gain comes from a prompt with only two categories, and no winning prompt uses all the content types\.

These different prompt compositions raise a further question: are they useful task guidance or adaptations to a particular model?

### 5\.4Are Optimized Prompts Generic or Model Specific?

Does APO discover general task guidance, or does it adapt prompts to a particular model? We investigate this question by applying each selected prompt unchanged to other models\. Keeping the puzzle set fixed lets us examine generalization across models rather than across puzzles\. Gains on other models would indicate that the prompt’s usefulness extends beyond its source, while uneven or negative transfer would reveal dependence on the recipient\. For each of seven source models, we select the prompt with the highest single\-run test\-set puzzle accuracy\. This differs from the three\-run optimizer ranking in Table[3](https://arxiv.org/html/2610.00416#S5.T3): for example, GPT\-4o Mini contributes aCOPROprompt even thoughSIMBAhas the higher mean accuracy\. GPT\-5\.6 Luna remains a target but not a source because its selected prompt is unchanged from the baseline\.

![Refer to caption](https://arxiv.org/html/2610.00416v1/prompt-transferability-heatmap.png)Figure 6:Prompt transferability on 559 puzzles\.Rows identify the source model and optimizer; columns identify the target model\. Cells report mean puzzle\-accuracy change in percentage points relative to the target’s three\-run baseline, with±\\pmone standard error of the difference, including variability in both groups\. Cell stars mark positive gains with one\-sided Welch\-testp<0\.05p<0\.05\. A target\-model star indicates at least one starred cell in its column, including the native prompt\. Gold outlines mark native prompts\.Observations\.Each populated cell in Figure[6](https://arxiv.org/html/2610.00416#S5.F6)compares three evaluations of the same selected prompt with the three baseline cohorts run on three dates summarized in Table[2](https://arxiv.org/html/2610.00416#S4.T2)\. We computed the differences in the group means and used one\-sided Welch tests\.444Claude Haiku 4\.5 returned identical outputs in all seeds, despite caching deactivated, so we mark nonzero differences as significant\.Across the 49 evaluated cross\-model pairs, 31 have positive mean changes, 16 have negative changes, and two are unchanged\. Transfer therefore depends on the source–target pairing rather than providing a consistent benefit\. Qwen3\.8 27B, Jev 1\.13, Claude Haiku 4\.5, and DeepSeek V4 Pro receive significant improvements from prompts optimized for other models\. Despite its different serving interface, Jev 1\.13 is the only source whose prompt improves all seven other models, with mean changes from \+0\.18 to \+3\.10 points\. Its prompt raises DeepSeek V4 Pro’s accuracy from 14\.49% to 17\.59%, showing that useful guidance can transfer from a weaker source model to a stronger recipient\. By contrast, the prompt with the largest native gain, selected for Gemini 3\.5 Flash Lite, degrades five of the seven other models\.

Takeaway\.Native improvement and portability are distinct: Jev’s prompt improves all seven other models, whereas Flash Lite’s prompt, which has the largest native gain, degrades five of them\. Optimized guidance can extend beyond its source model, but its usefulness depends on the recipient\. This matches the literature: cross\-model transfer of optimized prompts has been known since[Jiang et al\. \[2020\]](https://arxiv.org/html/2610.00416#bib.bib22), with prompts from small models transferring to larger ones while the reverse degrades\[[Deng et al\., 2022](https://arxiv.org/html/2610.00416#bib.bib36)\]\.[Agrawal et al\. \[2026\]](https://arxiv.org/html/2610.00416#bib.bib9)even report small\-model prompts beating optimizers running directly on the larger target\. In our experiments this pattern appears only in a few cases, for example the Jev 1\.13 prompt brings DeepSeek V4 Pro above its own best optimized accuracy\.

Cross\-model transfer changes the model while holding the puzzle task fixed\. We now change the setting: do puzzle\-optimized prompts also help when the model plays a sequence of moves against an opponent?

### 5\.5Transfer from Puzzles to Play

Chess play requires tactics as well as decision making in balanced positions\. We therefore evaluate whether prompts selected by puzzle accuracy transfer to game\-play rollouts againstStockfish\(GPL\-3\.0 License555[https://github\.com/official\-stockfish/Stockfish?tab=GPL\-3\.0\-1\-ov\-file](https://github.com/official-stockfish/Stockfish?tab=GPL-3.0-1-ov-file)\) at depth 20\. We evaluate ten starting conditions: the initial board with the model playing as White or Black, and eight positions from common openings \(Details in the Appendix[A\.2](https://arxiv.org/html/2610.00416#A1.SS2)\.\)\. For each rollout, the model predicts at most ten model moves\. As for puzzles, model move predictions are independent calls that receive only the prompt, the currentFEN, color to play, and legal moves;Stockfishreplies after every accepted move\. We compare each model’s base prompt with its best changed held\-out puzzle prompt; Gemini 3\.5 Flash is shown only with its base prompt\.

Unlike exact\-match puzzle scoring, game\-play evaluation does not require a unique reference move\. At each model turn, regret is the difference betweenStockfish’s depth\-20 evaluations of its best move and of the model’s move\. Regret is measured only for accepted moves; illegal or unparseable attempts terminate the rollout\. Rollouts also stop at ten model moves, a terminal board state, or a timeout\. Because prompts induce different trajectories and termination patterns, regret compares move quality on each prompt’s own visited states, conditional on producing accepted moves, rather than on a shared position set\. The preliminary evaluation of the relationship between regret evaluation in puzzles vs binary evaluation in puzzles can be found in the Appendix[B\.2](https://arxiv.org/html/2610.00416#A2.SS2)\. Those results showed that both metrics lead to a similar ranking of the best models and prompts\.

Table 4:Stockfishdepth\-20 regret per accepted model move\.For the overall mean, accepted moves are pooled across starting positions within each seed, then the three seed means are summarized as mean±\\pmsample SD\. Thus seeds have equal weight, but longer rollouts contribute more within a seed\. Median regret pools accepted moves across seeds\. Regret values are in centipawns; lower is better\. Illegal\-attempt rates pool flagged\-illegal attempts, including unparseable outputs, over all recorded model attempts across seeds; counts are shown below\. Checkmate\-loss rates count rollouts ending with the model checkmated over all rollouts\. Timeouts are not counted as additional illegal attempts\. Base and Best give rates before and after prompt optimization\. Bold marks the lower value within each base/best model pair, without implying significance\.Observations\.Puzzle gains are not accompanied by a statistically significant reduction in mean regret in the evaluated rollouts\. Table[4](https://arxiv.org/html/2610.00416#S5.T4)reports the same three models as Section[5\.2](https://arxiv.org/html/2610.00416#S5.SS2), together with the meta\-model’s baseline\. Mean regret decreases slightly for all three optimized models; median regret and checkmate\-loss counts decrease or remain unchanged, while illegal\-attempt rates increase for Flash Lite and Qwen and decrease for DeepSeek\. Note that the denominators of the illegal\-attempt rates vary because rollout length depends on illegal attempts and checkmates\. Legal moves alone do not ensure good play: Qwen rarely makes illegal moves with its baseline prompt, yet is frequently checkmated\.

Takeaway\.Puzzle gains did not produce a statistically significant reduction in mean regret over accepted moves in the evaluated rollouts\. Better tactical puzzle solving does not, by itself, establish better play from standard opening positions\. This game\-level evaluation is valuable beyond the transfer question itself\. Because the same engine\-based regret scores both puzzle solving and game\-play rollouts, we can directly estimate the gap between benchmark gains and performance in this broader setting\. Chess thus highlights how much an improvement in one specific skill contributes to the overall task, whereas in most domains, coding, mathematics, or question answering, this benchmark\-to\-deployment gap is far harder to estimate\.

## 6Conclusion

We introduced a chess\-puzzle benchmark and compared six prompt optimizers across eight target models\. Four models show significant puzzle\-accuracy gains, with improvements reaching 7\.57 percentage points, while the hardest puzzles remain largely out of reach\. The study distinguishes three properties: baseline strength, responsiveness to optimization, and transferability\.

Stronger models do not necessarily improve more, and larger gains do not guarantee better transfer\. Some prompts help other models, but we find no clear reduction in mean regret in the evaluated game\-play rollouts\. The benchmark reveals not just how much prompts improve, but where those improvements apply\.

Chess makes repeated evaluation inexpensive, datasets renewable, and difficulty adjustable as models improve\. It also lets us test whether puzzle gains carry over to play\. Together, these properties make it an affordable testbed for APO before evaluation in more costly application domains\.

## References

- Acher \(2023\)M\. AcherDebunking the chessboard: confronting gpts against chess engines to estimate elo ratings and assess legal move abilities\.Note:Blog post, blog\.mathieuacher\.comExternal Links:[Link](https://blog.mathieuacher.com/GPTsChessEloRatingLegalMoves/)Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p3.1)\.
- Agrawalet al\.\(2026\)L\. A\. Agrawal, S\. Tan, D\. Soylu, N\. Ziems, R\. Khare, K\. Opsahl\-Ong, A\. Singhvi, H\. Shandilya, M\. J\. Ryan, M\. Jiang, C\. Potts, K\. Sen, A\. G\. Dimakis, I\. Stoica, D\. Klein, M\. Zaharia, and O\. KhattabGEPA: reflective prompt evolution can outperform reinforcement learning\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p4.1),[§4\.1](https://arxiv.org/html/2610.00416#S4.SS1.p2.1.1),[§5\.4](https://arxiv.org/html/2610.00416#S5.SS4.p3.1)\.
- Beanet al\.\(2026\)A\. M\. Bean, R\. O\. Kearns, A\. Romanou, F\. S\. Hafner, H\. Mayne, J\. Batzner, N\. Foroutan Eghlidi, C\. Schmitz, K\. Korgul, H\. Batra,et al\.Measuring what matters: construct validity in large language model benchmarks\.Advances in Neural Information Processing Systems38\.Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p2.1)\.
- Campbellet al\.\(2002\)M\. Campbell, Jr\. Hoane, and F\. HsuDeep blue\.Artificial Intelligence134\(1–2\),pp\. 57–83\.External Links:[Document](https://dx.doi.org/10.1016/S0004-3702%2801%2900129-1)Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p3.1)\.
- Chanet al\.\(2025\)J\. S\. Chan, N\. Chowdhury, O\. Jaffe, J\. Aung, D\. Sherburn, E\. Mays, G\. Starace, K\. Liu, L\. Maksin, T\. Patwardhan,et al\.Mle\-bench: evaluating machine learning agents on machine learning engineering\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 50466–50494\.Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p2.1)\.
- Chenget al\.\(2023\)C\. Cheng, A\. Kolobov, D\. Misra, A\. Nie, and A\. SwaminathanLlf\-bench: benchmark for interactive learning from language feedback\.arXiv preprint arXiv:2312\.06853\.Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p2.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. SchulmanTraining verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p5.1)\.
- Cooper \(2025\)A\. CooperPGN2FEN benchmark\.Note:GitHub repositoryExternal Links:[Link](https://github.com/AidanCooper/pgn2fen-benchmark)Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p3.1)\.
- Daileyet al\.\(2023\)D\. Dailey, M\. Lefler, and L\. KaufmanKomodo: universal chess interface \(uci\) chess engine\.Chess\.com / Komodo Chess\.Note:Computer software, transitioned to Dragon by Komodo Chess featuring NNUE technologyExternal Links:[Link](https://komodochess.com/)Cited by:[§1](https://arxiv.org/html/2610.00416#S1.p3.1)\.
- Denget al\.\(2022\)M\. Deng, J\. Wang, C\. Hsieh, Y\. Wang, H\. Guo, T\. Shu, M\. Song, E\. P\. Xing, and Z\. HuRLPrompt: optimizing discrete text prompts with reinforcement learning\.InEMNLP,External Links:2205\.12548Cited by:[§5\.4](https://arxiv.org/html/2610.00416#S5.SS4.p3.1)\.
- Duet al\.\(2026\)X\. Du, G\. Hong, W\. Kwan, R\. Saxena, I\. Titov, P\. Minervini, and E\. AllawayAnalyzing llm instruction optimization for tabular fact verification\.InEACL Findings,External Links:2602\.17937Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p5.1),[§5\.1](https://arxiv.org/html/2610.00416#S5.SS1.p5.1)\.
- dynomight \(2024\)dynomightSomething weird is happening with llms and chess \(\+ follow\-up: ok, i can partly explain the llm chess weirdness now\)\.Note:Blog post, dynomight\.netExternal Links:[Link](https://dynomight.net/chess/)Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p3.1)\.
- Fenget al\.\(2023\)X\. Feng, Y\. Luo, Z\. Wang, H\. Tang, M\. Yang, K\. Shao, D\. Mguni, Y\. Du, and J\. WangChessGPT: bridging policy learning and language modeling\.InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 \- 16, 2023,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper/_files/paper/2023/hash/16b14e3f288f076e0ca73bdad6405f77-Abstract-Datasets/_and/_Benchmarks.html)Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p3.1),[§3](https://arxiv.org/html/2610.00416#S3.p1.1),[§3](https://arxiv.org/html/2610.00416#S3.p6.1)\.
- Fenget al\.\(2025\)X\. Feng, V\. Veeriah, M\. Chiam, M\. D\. Dennis, F\. Barbero, J\. Obando\-Ceron, J\. Shi, S\. Singh, S\. Hou, N\. Tomasev, and T\. ZahavyGenerating creative chess puzzles\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=TNZse5q2Tr)Cited by:[§1](https://arxiv.org/html/2610.00416#S1.p3.1)\.
- Glickman \(2001\)M\. E\. GlickmanDynamic paired comparison models with stochastic variances\.Journal of Applied Statistics28\(6\),pp\. 673–689\.Cited by:[§B\.5](https://arxiv.org/html/2610.00416#A2.SS5.p2.1)\.
- Glickman \(2012\)M\. E\. GlickmanExample of the glicko\-2 system\.Boston University28,pp\. 2012\.Cited by:[§B\.5](https://arxiv.org/html/2610.00416#A2.SS5.p2.1)\.
- Guertleret al\.\(2025\)L\. Guertler, B\. Cheng, S\. Yu, B\. Liu, L\. Choshen, and C\. TanTextArena\.External Links:2504\.11442,[Link](https://arxiv.org/abs/2504.11442)Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p2.1)\.
- Hsiehet al\.\(2024\)C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, Y\. Zhang, and B\. GinsburgRULER: what’s the real context size of your long\-context language models?\.arXiv preprint arXiv:2404\.06654\.Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p2.1)\.
- Hwanget al\.\(2025\)D\. Hwang, H\. Lee, J\. Choo, D\. Park, and J\. ParkCan large language models develop strategic reasoning? post\-training insights from learning chess\.External Links:2507\.00726,[Link](https://arxiv.org/abs/2507.00726)Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p3.1),[§3](https://arxiv.org/html/2610.00416#S3.p6.1)\.
- Jianget al\.\(2020\)Z\. Jiang, F\. F\. Xu, J\. Araki, and G\. NeubigHow can we know what language models know\.Trans\. Assoc\. Comput\. Linguistics8,pp\. 423–438\.External Links:[Link](https://doi.org/10.1162/tacl/_a/_00324),[Document](https://dx.doi.org/10.1162/TACL%5FA%5F00324)Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p4.1),[§5\.4](https://arxiv.org/html/2610.00416#S5.SS4.p3.1)\.
- Karvonen \(2024\)A\. KarvonenEmergent world models and latent variable estimation in chess\-playing language models\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=PPTrmvEnpW)Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p3.1)\.
- Khattabet al\.\(2024\)O\. Khattab, A\. Singhvi, P\. Maheshwari, Z\. Zhang, K\. Santhanam, S\. Vardhamanan, S\. Haq, A\. Sharma, T\. T\. Joshi, H\. Moazam, H\. Miller, M\. Zaharia, and C\. PottsDSPy: compiling declarative language model calls into self\-improving pipelines\.Cited by:[§B\.4](https://arxiv.org/html/2610.00416#A2.SS4.SSS0.Px1.p2.1),[§2](https://arxiv.org/html/2610.00416#S2.p4.1),[§4\.1](https://arxiv.org/html/2610.00416#S4.SS1.p3.1.1),[§4\.1](https://arxiv.org/html/2610.00416#S4.SS1.p4.1.1),[§4\.1](https://arxiv.org/html/2610.00416#S4.SS1.p5.1.1),[§4\.1](https://arxiv.org/html/2610.00416#S4.SS1.p7.1.1),[§4\.2](https://arxiv.org/html/2610.00416#S4.SS2.p4.1)\.
- Kielaet al\.\(2021\)D\. Kiela, M\. Bartolo, Y\. Nie, D\. Kaushik, A\. Geiger, Z\. Wu, B\. Vidgen, G\. Prasad, A\. Singh, P\. Ringshia, Z\. Ma, T\. Thrush, S\. Riedel, Z\. Waseem, P\. Stenetorp, R\. Jia, M\. Bansal, C\. Potts, and A\. WilliamsDynabench: rethinking benchmarking in nlp\.External Links:2104\.14337Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p2.1)\.
- Kolasaniet al\.\(2025\)S\. Kolasani, M\. Saplin, N\. Crispino, K\. Montgomery, J\. Q\. Davis, M\. Zaharia, C\. Wang, and C\. WangLLM chess: benchmarking reasoning and instruction\-following in llms through chess\.External Links:2512\.01992,[Link](https://arxiv.org/abs/2512.01992)Cited by:[§1](https://arxiv.org/html/2610.00416#S1.p3.1),[§2](https://arxiv.org/html/2610.00416#S2.p3.1),[§3](https://arxiv.org/html/2610.00416#S3.p6.1)\.
- Lehrachet al\.\(2025\)W\. Lehrach, D\. Hennes, M\. Lazaro\-Gredilla, X\. Lou, C\. Wendelken, Z\. Li, A\. Dedieu, J\. Grau\-Moya, M\. Lanctot, A\. Iscen, J\. Schultz, M\. Chiam, I\. Gemp, P\. Zielinski, S\. Singh, and K\. P\. MurphyCode world models for general game playing\.External Links:2510\.04542,[Link](https://arxiv.org/abs/2510.04542)Cited by:[§3](https://arxiv.org/html/2610.00416#S3.p1.1)\.
- Liuet al\.\(2026\)J\. Liu, S\. He, J\. Wu, X\. Wang, Y\. Chen, Z\. Kuang, S\. Bao, and Y\. YaoChessArena: A chess testbed for evaluating strategic reasoning capabilities of large language models\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2026, San Diego, California, United States, July 2\-7, 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),pp\. 7901–7954\.External Links:[Link](https://doi.org/10.18653/v1/2026.acl-long.360),[Document](https://dx.doi.org/10.18653/V1/2026.ACL-LONG.360)Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p2.1)\.
- Louet al\.\(2026\)X\. Lou, M\. Lázaro\-Gredilla, A\. Dedieu, C\. Wendelken, W\. Lehrach, and K\. P\. MurphyAutoHarness: improving llm agents by automatically synthesizing a code harness\.External Links:2603\.03329,[Link](https://arxiv.org/abs/2603.03329)Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p5.1)\.
- Maharanaet al\.\(2024\)A\. Maharana, D\. Lee, S\. Tulyakov, M\. Bansal, F\. Barbieri, and Y\. FangEvaluating very long\-term conversational memory of llm agents\.External Links:2402\.17753Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p2.1)\.
- Marroet al\.\(2026\)S\. Marro, J\. Yu, E\. L\. Malfa, O\. Deb, J\. Li, Y\. Yang, E\. Abraham, S\. Sengupta, E\. Sommerlade, M\. Wooldridge, and P\. TorrBenchmarking at the edge of comprehension\.External Links:2602\.14307Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p2.1)\.
- Merrillet al\.\(2026\)M\. Merrill A\. Shawet al\.Terminal\-bench: benchmarking agents on hard, realistic tasks in command line interfaces\.External Links:2601\.11868Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p2.1)\.
- Noeveret al\.\(2020\)D\. Noever, M\. Ciolino, and J\. KalinThe chess transformer: mastering play using generative language models\.arXiv preprint arXiv:2008\.04057\.Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p3.1)\.
- Opsahl\-Onget al\.\(2024\)K\. Opsahl\-Ong, M\. J\. Ryan, J\. Purtell, D\. Broman, C\. Potts, M\. Zaharia, and O\. KhattabOptimizing instructions and demonstrations for multi\-stage language model programs\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12\-16, 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),pp\. 9340–9366\.External Links:[Link](https://doi.org/10.18653/v1/2024.emnlp-main.525),[Document](https://dx.doi.org/10.18653/V1/2024.EMNLP-MAIN.525)Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p4.1),[§4\.1](https://arxiv.org/html/2610.00416#S4.SS1.p6.1.1)\.
- Pascuttoet al\.\(2022\)G\. Pascutto, G\. Linscott, A\. Lyashuk, and F\. HuizingaLeela chess zero\.Note:[https://lczero\.org](https://lczero.org/)Cited by:[§1](https://arxiv.org/html/2610.00416#S1.p3.1)\.
- Prasadet al\.\(2023\)A\. Prasad, P\. Hase, X\. Zhou, and M\. BansalGrIPS: gradient\-free, edit\-based instruction search for prompting large language models\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2\-6, 2023,A\. Vlachos and I\. Augenstein \(Eds\.\),pp\. 3827–3846\.External Links:[Link](https://doi.org/10.18653/v1/2023.eacl-main.277),[Document](https://dx.doi.org/10.18653/V1/2023.EACL-MAIN.277)Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p4.1)\.
- Pryzantet al\.\(2023\)R\. Pryzant, D\. Iter, J\. Li, Y\. T\. Lee, C\. Zhu, and M\. ZengAutomatic prompt optimization with ”gradient descent” and beam search\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6\-10, 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),pp\. 7957–7968\.External Links:[Link](https://doi.org/10.18653/v1/2023.emnlp-main.494),[Document](https://dx.doi.org/10.18653/V1/2023.EMNLP-MAIN.494)Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p4.1)\.
- Ramnathet al\.\(2025\)K\. Ramnath, K\. Zhou, S\. Guan, S\. S\. Mishra, X\. Qi, Z\. Shen, S\. Wang, S\. Woo, S\. Jeoung, Y\. Wang, H\. Wang, H\. Ding, Y\. Lu, Z\. Xu, Y\. Zhou, B\. Srinivasan, Q\. Yan, Y\. Chen, H\. Ding, P\. Xu, and L\. L\. CheongA systematic survey of automatic prompt optimization techniques\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 4\-9, 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),pp\. 33078–33110\.External Links:[Link](https://doi.org/10.18653/v1/2025.emnlp-main.1681),[Document](https://dx.doi.org/10.18653/V1/2025.EMNLP-MAIN.1681)Cited by:[§1](https://arxiv.org/html/2610.00416#S1.p1.1),[§2](https://arxiv.org/html/2610.00416#S2.p4.1),[§2](https://arxiv.org/html/2610.00416#S2.p5.1)\.
- Sarmahet al\.\(2024\)B\. Sarmah, K\. Dutta, A\. Grigoryan, S\. Tiwari, S\. Pasquali, and D\. MehtaA comparative study of dspy teleprompter algorithms for aligning large language models evaluation metrics to human evaluation\.External Links:2412\.15298Cited by:[§5\.1](https://arxiv.org/html/2610.00416#S5.SS1.p5.1)\.
- Silveret al\.\(2018a\)D\. Silver, T\. Hubert, J\. Schrittwieser, I\. Antonoglou, M\. Lai, A\. Guez, M\. Lanctot, L\. Sifre, D\. Kumaran, T\. Graepel, T\. Lillicrap, K\. Simonyan, and D\. HassabisA general reinforcement learning algorithm that masters chess, shogi, and go through self\-play\.Science362\(6419\),pp\. 1140–1144\.External Links:[Document](https://dx.doi.org/10.1126/science.aar6404),[Link](https://www.science.org/doi/abs/10.1126/science.aar6404),https://www\.science\.org/doi/pdf/10\.1126/science\.aar6404Cited by:[§1](https://arxiv.org/html/2610.00416#S1.p3.1)\.
- Silveret al\.\(2018b\)D\. Silver, T\. Hubert, J\. Schrittwieser, I\. Antonoglou, M\. Lai, A\. Guez, M\. Lanctot, L\. Sifre, D\. Kumaran, T\. Graepel, T\. Lillicrap, K\. Simonyan, and D\. HassabisA general reinforcement learning algorithm that masters chess, shogi, and go through self\-play\.Science362\(6419\),pp\. 1140–1144\.External Links:[Document](https://dx.doi.org/10.1126/science.aar6404)Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p3.1)\.
- Singhet al\.\(2024\)A\. K\. Singh, M\. Y\. Kocyigit, A\. Poulton, D\. Esiobu, M\. Lomeli, G\. Szilvasy, and D\. HupkesEvaluation data contamination in llms: how do we measure it and \(when\) does it matter?\.External Links:2411\.03923,[Link](https://arxiv.org/abs/2411.03923)Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p2.1)\.
- The LCZero Authors \(2026\)The LCZero AuthorsLeela Chess Zero\.Note:SoftwareAccessed 2026\-08\-30External Links:[Link](https://github.com/LeelaChessZero/lc0)Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p3.1)\.
- The Stockfish Developers \(2026\)The Stockfish DevelopersStockfish\.Note:SoftwareAccessed 2026\-08\-30External Links:[Link](https://github.com/official-stockfish/Stockfish)Cited by:[§1](https://arxiv.org/html/2610.00416#S1.p3.1)\.
- Whiteet al\.\(2025\)C\. White, S\. Dooley, M\. Roberts, A\. Pal, B\. Feuer, S\. Jain, R\. Shwartz\-Ziv, N\. Jain, K\. Saifullah, S\. Dey, Shubh\-Agrawal, S\. S\. Sandha, S\. V\. Naidu, C\. Hegde, Y\. LeCun, T\. Goldstein, W\. Neiswanger, and M\. GoldblumLiveBench: a challenging, contamination\-limited LLM benchmark\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=sKYHBTAxVa)Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p2.1)\.
- Witten \(2023\)Z\. Witten“You want to stay close to a pure list of moves” — the pgn prompt\-style tweet\.Note:X/Twitter threadExternal Links:[Link](https://x.com/zswitten/status/1699447461448933563)Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p3.1)\.
- Yanget al\.\(2024\)C\. Yang, X\. Wang, Y\. Lu, H\. Liu, Q\. V\. Le, D\. Zhou, and X\. ChenLarge language models as optimizers\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=Bb4VGOWELI)Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p4.1)\.
- Yanget al\.\(2018\)Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. Cohen, R\. Salakhutdinov, and C\. D\. ManningHotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InProceedings of the 2018 conference on empirical methods in natural language processing,pp\. 2369–2380\.Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p5.1)\.
- Zhang and Math\-AI \(2025\)Y\. Zhang and T\. Math\-AIAmerican invitational mathematics examination \(aime\) 2025\.Note:[https://huggingface\.co/datasets/math\-ai/aime25](https://huggingface.co/datasets/math-ai/aime25)Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p5.1)\.
- Zhanget al\.\(2025\)Y\. Zhang, X\. Han, H\. Li, K\. Chen, and S\. LinComplete chess games enable LLM become A chess master\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2025 \- Volume 2: Short Papers, Albuquerque, New Mexico, April 29 \- May 4, 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),pp\. 1–7\.External Links:[Link](https://doi.org/10.18653/v1/2025.naacl-short.1),[Document](https://dx.doi.org/10.18653/V1/2025.NAACL-SHORT.1)Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p3.1),[§3](https://arxiv.org/html/2610.00416#S3.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging llm\-as\-a\-judge with mt\-bench and chatbot arena\.InAdvances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks Track,External Links:2306\.05685Cited by:[§2](https://arxiv.org/html/2610.00416#S2.p2.1)\.
- Zhouet al\.\(2023\)Y\. Zhou, A\. I\. Muresanu, Z\. Han, K\. Paster, S\. Pitis, H\. Chan, and J\. BaLarge language models are human\-level prompt engineers\.InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1\-5, 2023,External Links:[Link](https://openreview.net/forum?id=92gvk82DE-)Cited by:[§1](https://arxiv.org/html/2610.00416#S1.p1.1),[§2](https://arxiv.org/html/2610.00416#S2.p4.1)\.

## Appendix

## Appendix AChess Details

### A\.1Chess Notation Examples

Forsyth–Edwards Notation \(FEN\) describes one board state, including the side to move, castling rights, en\-passant target, and move counters\. After the moves1\. e4 e5 2\. Nf3, the current position is represented inFENas

rnbqkbnr/pppp1ppp/8/4p3/4P3/5N2/PPPP1PPP/RNBQKB1R b KQkq \- 1 2\.

Universal Chess Interface \(UCI\) notation represents the same move sequence using source and destination squares:e2e4 e7e5 g1f3\. Portable Game Notation \(PGN\) records the game sequence in standard chess notation:1\. e4 e5 2\. Nf3\.

### A\.2Playing Chess Starting Positions

Table 5:Starting positions used for the playing\-chess transfer evaluation\.“Model color” determines which side is controlled by the language model\.
### A\.3Puzzle Dataset Details

Table 6:Number of train/test puzzles by solver moves andLichessrating bin\.Each cell is reported as train/test\.

## Appendix BResults Details

### B\.1Example Compiled Puzzle Prompts \(Gemini 3\.5 Flash Lite\)

We show the instruction and two of four complete demonstrations from the seed\-42MIPROv2prompt, followed by the complete static portion of the seed\-42SIMBAprompt\.Greentext denotes output\-format constraints,orangetext denotes chess guidance,bluetext denotes optimizer\-selected demonstrations, andpinktext denotes other content\. The full seed\-42MIPROv2prompt contains 1,507 characters; the 1,449\-character value in Table[8](https://arxiv.org/html/2610.00416#A2.T8)is the rounded mean over seeds 42–44 \(1,507, 1,514, and 1,327 characters\)\. The completeSIMBAprompt below contains 901 characters; its corresponding 930\-character table value averages 901, 1,009, and 879 characters\.

MIPROv2seed 42: instruction \+ two of four demonstrations

Analyze the given board position \(FEN\) and determine the absolute best tactical move for the active side\.

Select exactly one move from the provided list of legal moves \(legal\_moves\_uci\)\. Your response must only contain the selected move with no reasoning, explanations, or labels\.

Position Fen: 8/5pkp/2BR2p1/1pP1p3/1P5q/3QP2P/5P1K/r7 b \- \- 0 32 Side To Move: black Legal Moves Uci: g7h8, g7g8, g7f8, g7h6, h4d8, h4e7, h4h6, h4f6, h4h5, h4g5, h4g4, h4f4, h4e4, h4d4, h4c4, h4b4, h4h3, h4g3, h4f2, a1a8, a1a7, a1a6, a1a5, a1a4, a1a3, a1a2, a1h1, a1g1, a1f1, a1e1, a1d1, a1c1, a1b1, h7h6, f7f6, g6g5, e5e4, h7h5, f7f5 Move: h4f2

Position Fen: 4r1k1/R5pp/2p5/8/2P2p2/1PNn1P2/P2P2PP/6K1 b \- \- 4 31 Side To Move: black Legal Moves Uci: g8h8, g8f8, e8f8, e8d8, e8c8, e8b8, e8a8, e8e7, e8e6, e8e5, e8e4, e8e3, e8e2, e8e1, d3e5, d3c5, d3b4, d3f2, d3b2, d3e1, d3c1, h7h6, g7g6, c6c5, h7h5, g7g5 Move: e8e1

\[Two additional optimizer\-selected demonstrations\.\]

SIMBAseed 42: complete 901\-character static prompt

Solve the chess puzzle by finding the single best move\.

Position Fen: r1b2Rnk/2p3pB/2p4p/1pq1P1N1/p7/R1P1P3/5PPP/6K1 w \- \- 1 29 Side To Move: white Legal Moves Uci: f8g8, f8e8, f8d8, f8c8, f8f7, f8f6, f8f5, f8f4, f8f3, h7g8, h7g6, h7f5, h7e4, h7d3, h7c2, h7b1, g5f7, g5e6, g5e4, g5h3, g5f3, a3a4, a3b3, a3a2, a3a1, g1h1, g1f1, e5e6, e3e4, c3c4, h2h3, g2g3, f2f3, h2h4, g2g4, f2f4 Move: f8g8

Position Fen: 8/5pk1/p3pNp1/1p2P3/6P1/2Q2Pqr/2R1K3/8 b \- \- 8 37 Side To Move: black Legal Moves Uci: g7h8, g7f8, g7h6, h3h8, h3h7, h3h6, h3h5, h3h4, h3h2, h3h1, g3e5, g3h4, g3g4, g3f4, g3f3, g3h2, g3g2, g3f2, g3g1, g3e1, g6g5, a6a5, b5b4 Move: g3g2

Position Fen: 3k4/p4Rp1/p7/6p1/Q2PP3/6r1/P5q1/3RK3 b \- \- 5 26 Side To Move: black Legal Moves Uci: d8c8, g3g4, g3h3, g3f3, g3e3, g3d3, g3c3, g3b3, g3a3, g2e4, g2h3, g2f3, g2h2, g2f2, g2e2, g2d2, g2c2, g2b2, g2a2, g2h1, g2g1, g2f1, g7g6, a6a5, g5g4 Move: Re3\+

### B\.2Puzzle Move Quality by Regret

While puzzle accuracy gives insight on performance, the outcome is binary success / failure\. A decent human player can play good moves without producing the best one in puzzles\. To give a more lenient perspective on move quality, we evaluate the regret in the puzzle moves for baseline and best prompt in long puzzles \(3–5 moves\) in Table[7](https://arxiv.org/html/2610.00416#A2.T7)\. GPT\-5\.6 Luna is omitted because its selected prompt is unchanged from the baseline\. The regret is estimated as the difference between theStockfishevaluation of the best move and the move played by the model\. With regret the move is not binary evaluated but offers a more granular perspective on how bad an incorrect move is\.

Table 7:Regret computed on long puzzles \(3–5 moves\) withStockfishdepth 20\. Rank entries report median\-regret rank / position\-accuracy rank / puzzle\-accuracy rank; lower ranks are better\. Bottom row is the meta\-model\.The results are gathered in Table[7](https://arxiv.org/html/2610.00416#A2.T7), together with each model’s regret, position\-accuracy, and puzzle\-accuracy ranks\. Position accuracy treats every move of every puzzle independently, whereas a five\-move puzzle requires all five positions to be correctly predicted to count as correct\. Note that regret can be very harsh if the model misses a checkmate or gets checkmated; for example, missing a checkmate or predicting an illegal move can be penalized by 200,000 centipawns, while for reference the value of a queen is estimated at 900 centipawns\. Hence, we report median regret to reduce the impact of such penalties\. Median regret, position accuracy, and puzzle accuracy yield similar, but not identical, model rankings in Table[7](https://arxiv.org/html/2610.00416#A2.T7)\. This agreement supports using regret as a complementary measure of move quality; it does not validate it as a general measure of playing strength\. In game\-play rollouts \(Section[5\.5](https://arxiv.org/html/2610.00416#S5.SS5)\), regret remains applicable without a unique reference move\.

### B\.3Prompt Size

Table[8](https://arxiv.org/html/2610.00416#A2.T8)reports the average compiled\-prompt size in characters\. The base prompt is 55 characters\. Some algorithms never manage to improve the base prompt in some models, showing that the task is not straightforward to improve\. Across the evaluated model–optimizer seeds in Table[3](https://arxiv.org/html/2610.00416#S5.T3), 21 of 141 runs \(14\.9%\) leave the prompt unchanged\.

Table 8:Average complete compiled\-prompt size in characters, over three seeds\.The base prompt is 55 characters\. Some algorithms never manage to improve the base prompt on some models \(gray cells\)\. Bold marks the optimizer with the highest puzzle accuracy for each model\. N\.E\. marks not evaluated\.
### B\.4Task Prompts and Signatures

All generative\-model results use theJSON adapter\(DSPy’sJSONAdapter\), which enforces strict structured output formatting\. Jev 1\.13 is the exception: it is queried through a decision endpoint with its own schema adapter, not throughDSPy’s generative JSON adapter\.

#### Puzzles \(Independent\)\.

The default instruction is:*“Solve the chess puzzle by finding the single best move\.”*This 55\-character instruction is distinct from the adapter\-generated formatting instructions and input fields\. The input fields are:

- •position\_fen: current position inFEN\.
- •side\_to\_move:whiteorblack\.
- •legal\_moves\_uci: comma\-separated legalUCImoves\.

The output field is:

- •move: exactly one legalSANorUCImove, without reasoning or labels\.

The resulting generative\-model request contains the following fixedJSONAdapterscaffolding around the optimized instruction and puzzle\-specific fields\.JSONAdapteris proposed by DSPY[Khattab et al\. \[2024\]](https://arxiv.org/html/2610.00416#bib.bib10)to structure inputs\. Placeholders in angle brackets are replaced at runtime\.

Baseline JSONAdapter request template

SYSTEM:

Your input fields are:

1\. ‘position\_fen‘ \(str\): Current position in FEN

2\. ‘side\_to\_move‘ \(str\): white or black

3\. ‘legal\_moves\_uci‘ \(str\): Comma\-separated legal UCI moves

Your output fields are:

1\. ‘move‘ \(str\): Exactly one legal SAN or UCI move, without reasoning or labels

All interactions will be structured in the following way, with the appropriate values filled in\.

Inputs will have the following structure:

\[\[ \#\# position\_fen \#\# \]\]

position\_fen

\[\[ \#\# side\_to\_move \#\# \]\]

side\_to\_move

\[\[ \#\# legal\_moves\_uci \#\# \]\]

legal\_moves\_uci

Outputs will be a JSON object with the following fields\.

"move": "move"

In adhering to this structure, your objective is:

<<PROMPT\>\>

USER:

\[\[ \#\# position\_fen \#\# \]\]

<<FEN\>\>

\[\[ \#\# side\_to\_move \#\# \]\]

<<SIDE\_TO\_MOVE\>\>

\[\[ \#\# legal\_moves\_uci \#\# \]\]

<<LEGAL\_MOVES\_UCI\>\>

Respond with a JSON object in the following order of fields: ‘move‘\.

### B\.5InterpretingLichessPuzzle Ratings for LLMs

Our benchmark uses theLichesspuzzle rating as a difficulty axis\. This rating is a human\-calibrated scale, so we first check what it means for LLMs\.

The puzzle rating is derived fromLichess’s puzzle\-difficulty classification, which uses theGlicko\-2system[Glickman \[2001\]](https://arxiv.org/html/2610.00416#bib.bib45),[Glickman \[2012\]](https://arxiv.org/html/2610.00416#bib.bib46)\.Glicko\-2is a paired\-comparison rating system: like Elo, it models the expected outcome of a comparison as a function of the relative ratings of the two entities\. In theLichessmodel, a puzzle is treated analogously to an opponent that wins whenever the user fails to solve it\.

The key difference between the two systems is thatGlicko\-2also accounts for uncertainty in the ratings\.

By definition, these ratings are relative and impose a scalar ordering based on the performance of users on the platform, whether the rated entities are human players or puzzles\. However, constructing a benchmark of chess puzzles requires us to freeze the puzzle ratings and ignore the uncertainty aspect ofGlicko\-2and interpret ratings as an external descriptive variable\.

While the puzzle rating is calibrated for human players, no such calibration exists for LLMs\. Consequently,we ask whether it can serve as a proxy for task difficulty for LLMs in our benchmark\.

Empirically, we observe significant correlations between LLMs’ puzzle solve rates and puzzle ratings\. Across pooled 200\-point Elo bins, correlations werer=−0\.937r=\-0\.937\(p<0\.001p<0\.001, 8 bins\) andr=−0\.996r=\-0\.996\(p=0\.004p=0\.004, 4 bins\) for 1–2\- and 3–5\-move puzzles, respectively\. Each 200\-point Elo increase multiplies solve odds by 0\.72 for 1–2\-move puzzles and 0\.45 for 3–5\-move puzzles, corresponding to odds reductions of 28% and 55%, respectively\. Under the standard Elo model, a 200\-point disadvantage multiplies the odds by10−200/400≈0\.31610^\{\-200/400\}\\approx 0\.316, corresponding to a reduction of 68\.4% odds\. For an equal\-rated player, this changes the expected success rate from 50% to approximately 24%, a decrease of 26 percentage points, or 52% relative to the original success rate\. Notice that this is not the rating\-update rule used byLichess; it provides a simple theoretical reference for interpreting rating differences\.

Figure 7:Puzzle solve rate byLichessrating with the baseline prompt\.Solid curves show empirical model performance; shaded regions show±1\\pm 1standard deviation across runs\. Dashed curves show the standard Elo success\-rate function for hypothetical players anchored at50%50\\%success for puzzle ratings 955 and 1980\. The vertical lines mark these anchor ratings\.Figure[7](https://arxiv.org/html/2610.00416#A2.F7)reports puzzle solve rates by rating for the models evaluated\. The solid curves show the performance of the empirical model\. For reference, the dashed curves show the success rate predicted by the standard Elo model for hypothetical players whose ratings are chosen so that their expected success rate is50%50\\%at puzzle ratings 955 and 1980, respectively\. These anchor points approximately correspond to the observed50%50\\%solve\-rate points of Gemini 3\.5 Flash Lite and Gemini 3\.5 Flash\. The vertical lines mark the two corresponding anchor ratings\.

### B\.6Hyperparameters

#### LM configuration\.

Table 9:Exact OpenRouter model identifiers used in the puzzle experiments\.Gemini 3\.5 Flash is also the fixed reflection and meta\-model\.Table 10:Language model configuration for compilation and evaluation\.For the optimization panel, task LMs usetemperature=0\.0, while the meta\-model andCOPROprompt LM usetemperature=1\.0\. All selected LMs usetimeout\_seconds=30\.0,max\_concurrency=1,max\_retries=1,retry\_backoff\_seconds=0\.25,cache\_mode=stateless,drop\_params=true, and seed 42 unless a replicate overrides the seed\. Gemini 3\.5 Flash Lite additionally usescapture\_reasoning=true,reasoning\_effort=low, andinclude\_reasoning=true; Gemini 3\.5 Flash used default reasoning withcapture\_reasoning=false\. GPT\-4o Mini, Claude Haiku 4\.5 usecapture\_reasoning=falseandreasoning\_effort=none\. Jev 1\.13 is evaluated through the Typesafe decision endpoint \(/api/alpha/decisions\) rather than a generative OpenRouter chat endpoint\. The decision request carries the same information as the generative prompts: the instructions and demonstrations, the position fields \(FENand side to move\), and the legal moves, which serve as the answer options\. The model does not generate text\. It chooses among the listed legal moves only, so its answer space is exactly the provided option set\. The endpoint bills input tokens only and exposes no temperature or sampling controls\. Generative settings such asmax\_output\_tokens,temperature, and the JSON adapter therefore do not apply\. The absence of sampling control is also whySIMBA, which queries the model at several temperatures, is not evaluated for Jev 1\.13 \(N\.A\. in Table[3](https://arxiv.org/html/2610.00416#S5.T3)\)\.

#### Optimizer hyperparameters\.

Table[11](https://arxiv.org/html/2610.00416#A2.T11)lists the hyperparameters for all optimizers\.

Table 11:Optimizer hyperparameters\.Unless noted otherwise, values areDSPydefaults\.OptimizerParameterValueGEPAautoheavy\(∼\\sim3,436 metric calls\)candidate\_selection\_strategyparetouse\_mergeTrueskip\_perfect\_scoreTruereflection\_minibatch\_size3num\_threads16MIPROv2autoheavymax\_bootstrapped\_demos4max\_labeled\_demos4num\_threads16num\_candidates∼\\sim18 \(set byauto=heavy\)num\_trials∼\\sim27 \(set byauto=heavy\)program\_aware\_proposerTruedata\_aware\_proposerTruetip\_aware\_proposerTruefewshot\_aware\_proposerTrueprompt\_max\_tokens8192SIMBAbsize32num\_candidates6max\_steps8max\_demos4num\_threads16COPRObreadth10depth3eval\_num\_threads16BFSmax\_bootstrapped\_demos4max\_labeled\_demos16max\_rounds1BRSmax\_bootstrapped\_demos4max\_labeled\_demos16num\_candidate\_programs16num\_threads16max\_rounds1