Contrastive Reflection for Iterative Prompt Optimization

# Contrastive Reflection for Iterative Prompt Optimization
Source: [https://arxiv.org/html/2606.30840](https://arxiv.org/html/2606.30840)
\(2026\)

###### Abstract\.

LLM agents are becoming central to information retrieval: they issue retrieval queries, synthesize answers from evidence, personalize results, and increasingly serve as judges for IR evaluation\. Improving the prompts that control these agents is therefore an optimization problem, but in applied IR settings it often looks less like blind search and more like debugging\. Engineers need to know which behavior failed, which nearby behavior still worked, what distinguishes the two, and whether a prompt edit improves held\-out quality without introducing regressions\.

We present Contrastive Reflection, an iterative prompt\-optimization framework for agentic IR workflows\. The framework starts from a task\-centric quality definition: QA agents can expose retrieval or reasoning traces, while grading agents can expose dimension\-level scores and rationales\. These structured traces are used to identify error\-anchored behavioral slices, add nearby successful examples from the same region, and ask a Teacher LLM to propose a targeted prompt edit\. Candidate edits are accepted only when validation performance improves, optionally subject to regression checks\. We instantiate the framework with a tree\-based slice selector, but the contribution is the contrastive reflection loop rather than the tree itself\.

On a public HotpotQA retrieval\-augmented QA setup, one tree\-selected contrastive repair improves held\-out exact\-match accuracy from 51\.4% to 60\.4%\. Failure\-only and random\-evidence variants also improve, but achieve smaller held\-out gains and break more previously correct examples\. A light instruction\-only comparison on the same setup places the method near modern prompt optimizers: MIPROv2 reaches 59\.4% test accuracy and GEPA reaches 57\.0%\. LinkedIn\-derived grading workflows motivate the task\-centric design, but the paper’s headline evidence is the public HotpotQA study\. The result is an interpretable optimization loop for AI agents in IR, aimed at making prompt repair more inspectable and validation\-driven\.

AI Agents, Information Retrieval, Prompt Optimization, Retrieval\-Augmented Generation, LLM Reflection, Behavioral Slices

††conference:KDD 2026 Workshop on AI Agents for Information Retrieval; August 2026;††journalyear:2026††copyright:none††ccs:Computing methodologies Natural language processing††ccs:Information systems Information retrieval## 1\.Introduction

LLM agents are increasingly used in information retrieval \(IR\) pipelines: they generate sub\-queries for multi\-hop retrieval, synthesize answers from evidence, and serve as relevance judges in place of human annotators\(Thomaset al\.,[2024](https://arxiv.org/html/2606.30840#bib.bib14); Farzi and Dietz,[2025](https://arxiv.org/html/2606.30840#bib.bib19)\)\. Our motivating setting is a LinkedIn\-derived grading workflow, where an LLM judge must score whether a result is appropriate for a member or query using multiple task\-specific dimensions, rationales, and regression\-sensitive constraints\. This setting shaped the method: the final grade tells a team whether the judge matched the reference, but the dimension scores and rationales explain*how*it was wrong\.

In such evaluation workflows, prompt improvement is rarely a one\-shot search problem\. A prompt may perform well overall while repeatedly making errors on a narrow class of examples: entity disambiguation in multi\-hop QA, one rating dimension in grading, or formatting behavior for a subset of inputs\. Engineers typically debug such systems by asking five questions: which cases were wrong, which similar cases still worked, what separates the two, what instruction would address that contrast, and did the change help without breaking other behavior?

Contemporary prompt optimizers, including reflective and DSPy\-based methods such as GEPA and MIPROv2\(Agrawalet al\.,[2025](https://arxiv.org/html/2606.30840#bib.bib1); Opsahl\-Onget al\.,[2024](https://arxiv.org/html/2606.30840#bib.bib2); Khattabet al\.,[2024](https://arxiv.org/html/2606.30840#bib.bib4)\), can substantially improve prompts\. However, many optimizers operate primarily through scalar validation metrics, stochastic batches, or search over instruction\-demo combinations\. These methods can find better prompts without exposing a compact explanation of*which*behavior changed\. When reflection is performed on random examples or errors alone, the prompt edit can be driven by whichever cases happen to appear in the batch, not necessarily by a systematic contrast between broken and working behavior\.

This paper reframes prompt optimization as*contrastive reflection*\. The central idea is simple: before asking an LLM to improve a prompt, first identify a coherent slice containing errors and nearby successes, then describe the attributes that define the slice\. Reflection becomes a targeted comparison\-and\-repair step rather than a general request for better instructions\. The public experiment in this paper tests the idea on a single\-predictor HotpotQA program; the LinkedIn\-derived grading setting explains the task\-centric design but is not treated as a public benchmark\.

We make three contributions\. First, we formalize task\-centric quality for prompt optimization: task\-specific structured outputs define the attributes used for slice discovery, regression tracking, and targeted editing\. Second, we define an iterative contrastive reflection loop: evaluate, find contrastive slices, reflect, edit, validate, and repeat\. Third, we evaluate the idea with a public HotpotQA\(Yanget al\.,[2018](https://arxiv.org/html/2606.30840#bib.bib24)\)study and describe how the same loop is being refined in LinkedIn\-derived grading workflows\.

## 2\.Related Work

Prompt optimization\.Early prompt\-tuning work optimized continuous soft prompts\(Lesteret al\.,[2021](https://arxiv.org/html/2606.30840#bib.bib5)\), while recent discrete methods treat language models themselves as optimizers\. OPRO and APE use LLMs to generate and refine instructions\(Yang and others,[2024](https://arxiv.org/html/2606.30840#bib.bib6); Zhou and others,[2023](https://arxiv.org/html/2606.30840#bib.bib7)\); PromptAgent frames prompt improvement as planning\(Wang and others,[2024](https://arxiv.org/html/2606.30840#bib.bib8)\); and APO applies gradient\-inspired textual feedback\(Pryzant and others,[2023](https://arxiv.org/html/2606.30840#bib.bib12)\)\. DSPy provides a programming model and optimizers for multi\-stage LM programs\(Khattabet al\.,[2024](https://arxiv.org/html/2606.30840#bib.bib4)\); MIPROv2 searches over instructions and demonstrations\(Opsahl\-Onget al\.,[2024](https://arxiv.org/html/2606.30840#bib.bib2)\)\. These methods are strong search baselines, but they do not primarily focus on producing an interpretable contrast between the errors and successes that motivated an edit\.

Reflection and evolutionary prompt search\.GEPA uses LLM reflection over sampled trajectories and maintains a Pareto frontier of candidate prompts\(Agrawalet al\.,[2025](https://arxiv.org/html/2606.30840#bib.bib1)\); Promptbreeder and EvoPrompt explore related evolutionary prompt\-improvement strategies\(Fernando and others,[2023](https://arxiv.org/html/2606.30840#bib.bib11); Guo and others,[2024](https://arxiv.org/html/2606.30840#bib.bib9)\)\. StraGo\(Wu and others,[2024](https://arxiv.org/html/2606.30840#bib.bib25)\)is most closely related: it shows that prompt optimization improves when the optimizer reflects over both successes and failures rather than failures alone\. Contrastive reflection shares this intuition but differs in where the contrast comes from\. StraGo samples successes and failures from the global pool and asks an LLM to synthesize strategic guidance across them; contrastive reflection instead requires the successes to be drawn from the*same error\-anchored behavioral slice*as the failures, so that the contrast pins down a single, interpretable boundary that the edit must target\. A tree is one way to construct such a slice; future work could combine contrastive evidence selection with Bayesian or evolutionary search\.

Structured evaluation and slice discovery\.LLMs are increasingly used as automated relevance judges in IR\(Thomaset al\.,[2024](https://arxiv.org/html/2606.30840#bib.bib14)\), and criteria\-based judging decomposes relevance into explicit dimensions\(Farzi and Dietz,[2025](https://arxiv.org/html/2606.30840#bib.bib19)\)\. CheckList argues for behavioral slices over aggregate accuracy alone\(Ribeiroet al\.,[2020](https://arxiv.org/html/2606.30840#bib.bib18)\)\. A separate line of work focuses on*discovering*such slices automatically: SliceFinder uses decision\-tree and clustering search to find interpretable, error\-heavy subgroups for model validation\(Chunget al\.,[2019](https://arxiv.org/html/2606.30840#bib.bib26)\), and Domino discovers systematic errors via cross\-modal embedding clusters\(Eyubogluet al\.,[2022](https://arxiv.org/html/2606.30840#bib.bib27)\)\. Our slice selector reuses the SliceFinder\-style tree primitive, but the contribution of this paper is not slice discovery itself; it is wiring discovered slices, together with their nearby successes, into a validation\-gated reflection loop for prompt optimization\. The tree is deliberately the simplest selector that makes the contrastive evidence interpretable to the Teacher LLM\.

## 3\.Contrastive Reflection

Contrastive reflection separates prompt optimization into two questions that are often conflated\. The first is*where should the optimizer look?*The second is*what edit should be made?*A general\-purpose LLM can help with the second question, but it needs a useful description of the first\. Task\-centric quality definitions create that description: instead of observing only a final score, the optimizer sees task\-specific dimensions, steps, and rationales that can explain why nearby examples diverge\. The framework therefore proceeds as an iterative loop:

1. \(1\)Evaluate the current prompt on training and validation examples\.
2. \(2\)Identify error\-heavy regions with interpretable shared attributes\.
3. \(3\)Add nearby successful examples from the same region to form a contrastive slice\.
4. \(4\)Present the slice, its defining attributes, the error/success contrast, and the current prompt section to a Teacher LLM\.
5. \(5\)Ask the Teacher to propose a targeted prompt edit\.
6. \(6\)Accept the edit only if it improves validation performance, optionally subject to regression checks\.

In this framing, failures decide where the optimizer should look, but successes decide what the edit must preserve\. This is the main difference between contrastive reflection and failure\-only reflection: the repair is not merely an instruction to fix bad cases, but an attempt to infer the boundary between bad and good behavior inside the same task region\.

The slice selector in Step 2 is deliberately abstract\. A decision tree is one option, but other algorithms could be used: clustering, embedding\-nearest\-neighbor search, rule mining, human\-authored slices, or learned error classifiers\. The hypothesis is not that trees are uniquely optimal\. The hypothesis is that reflection improves when the Teacher receives a coherent contrastive description instead of an arbitrary batch of examples\.

### 3\.1\.Problem Definition

Letπϕ\\pi\_\{\\phi\}denote an LLM program with promptϕ\\phi, metricmm, training setDt​rD\_\{tr\}, validation setDv​a​lD\_\{val\}, and baseline promptϕ0\\phi\_\{0\}\. Standard prompt optimization seeks a prompt that maximizes validation performance:

\(1\)maxϕ⁡𝔼\(x,y\)∼Dv​a​l​\[m​\(πϕ​\(x\),y\)\]\.\\max\_\{\\phi\}\\;\\mathbb\{E\}\_\{\(x,y\)\\sim D\_\{val\}\}\\left\[m\(\\pi\_\{\\phi\}\(x\),y\)\\right\]\.We denote this validation objective byMv​a​l​\(ϕ\)M\_\{val\}\(\\phi\)\. Contrastive reflection adds an intermediate object: a slice selectorggthat maps current training examples and structured traces to a set of coherent, error\-anchored contrastive slices,

\(2\)𝒞t=g​\(Dt​r,\{πϕt​\(x\):\(x,y\)∈Dt​r\}\)\.\\mathcal\{C\}\_\{t\}=g\\\!\\left\(D\_\{tr\},\\\{\\pi\_\{\\phi\_\{t\}\}\(x\):\(x,y\)\\in D\_\{tr\}\\\}\\right\)\.The second argument denotes the current program outputs and any structured traces exposed by the task\. Each sliceC∈𝒞tC\\in\\mathcal\{C\}\_\{t\}is paired with an explanation or attribute descriptiona​\(C\)a\(C\), such as a rule path, cluster summary, or human\-authored slice\. A reflection operatorrrthen proposes an edit conditioned on the current prompt and the selected contrastive evidence:

\(3\)Δ​ϕt=r​\(ϕt,C,a​\(C\),EC\),\\Delta\\phi\_\{t\}=r\(\\phi\_\{t\},C,a\(C\),E\_\{C\}\),whereECE\_\{C\}contains both errors and nearby successes from the same slice\. The edit is a prompt\-level repair: it does not change the training examples, but changes the instruction so the Student should handle future examples matching the same contrastive pattern\. We writeϕc=ϕt⊕Δ​ϕt\\phi\_\{c\}=\\phi\_\{t\}\\oplus\\Delta\\phi\_\{t\}for the textual operation of applying the edit to the prompt\.

The acceptance rule keeps the framework tied to held\-out behavior:

\(4\)ϕt\+1=\{ϕc,if​At​\(ϕc\),ϕt,otherwise,\\phi\_\{t\+1\}=\\begin\{cases\}\\phi\_\{c\},&\\text\{if \}A\_\{t\}\(\\phi\_\{c\}\),\\\\ \\phi\_\{t\},&\\text\{otherwise,\}\\end\{cases\}whereAt​\(ϕc\)A\_\{t\}\(\\phi\_\{c\}\)is true whenMv​a​l​\(ϕc\)\>Mv​a​l​\(ϕt\)M\_\{val\}\(\\phi\_\{c\}\)\>M\_\{val\}\(\\phi\_\{t\}\)and every enabled regression check satisfiesRk​\(ϕc,ϕ0\)≤δkR\_\{k\}\(\\phi\_\{c\},\\phi\_\{0\}\)\\leq\\delta\_\{k\}\. HereMv​a​lM\_\{val\}is the validation metric from Equation[1](https://arxiv.org/html/2606.30840#S3.E1)\. The regression constraintsRkR\_\{k\}are optional and measure protected behavior relative to the reference promptϕ0\\phi\_\{0\}; when disabled, acceptance reduces to validation improvement\. The main design freedom is therefore the selectorgg: this paper studies a tree\-based selector, but the formulation permits other ways of finding useful contrastive slices\.

Algorithm 1Contrastive Reflection1:Prompt

ϕ0\\phi\_\{0\}, train set

Dt​rD\_\{tr\}, validation set

Dv​a​lD\_\{val\}, metric

mm, selector

gg, reflector

rr, optional constraints

\(Rk,δk\)\(R\_\{k\},\\delta\_\{k\}\)
2:

ϕb​e​s​t←ϕ0\\phi\_\{best\}\\leftarrow\\phi\_\{0\};

sb​e​s​t←Mv​a​l​\(ϕ0\)s\_\{best\}\\leftarrow M\_\{val\}\(\\phi\_\{0\}\)
3:for

t=1t=1to

TTdo

4:Evaluate

πϕb​e​s​t\\pi\_\{\\phi\_\{best\}\}on

Dt​rD\_\{tr\}and cache traces

5:

𝒞←g​\(Dt​r,traces\)\\mathcal\{C\}\\leftarrow g\(D\_\{tr\},\\text\{traces\}\)
6:Rank slices by error density, support, and interpretability

7:for allcontrastive slices

C∈𝒞C\\in\\mathcal\{C\}do

8:Build evidence

ECE\_\{C\}from errors and nearby successes

9:

Δ​ϕ←r​\(ϕb​e​s​t,C,a​\(C\),EC\)\\Delta\\phi\\leftarrow r\(\\phi\_\{best\},C,a\(C\),E\_\{C\}\)
10:

ϕc←ϕb​e​s​t⊕Δ​ϕ\\phi\_\{c\}\\leftarrow\\phi\_\{best\}\\oplus\\Delta\\phi
11:

sc←Mv​a​l​\(ϕc\)s\_\{c\}\\leftarrow M\_\{val\}\(\\phi\_\{c\}\)
12:if

sc\>sb​e​s​ts\_\{c\}\>s\_\{best\}and all enabled constraints are satisfiedthen

13:

ϕb​e​s​t←ϕc\\phi\_\{best\}\\leftarrow\\phi\_\{c\};

sb​e​s​t←scs\_\{best\}\\leftarrow s\_\{c\}
14:break

15:else

16:Record rejected edit for later reflection

17:endif

18:endfor

19:endfor

20:return

ϕb​e​s​t\\phi\_\{best\}

### 3\.2\.Task\-Centric Contrastive Quality

Contrastive reflection does not require every task to reduce immediately to a single scalar signal\. This is one reason the method can be more effective than reflection over random examples: the optimizer receives task\-native evidence about*how*an output is wrong, not only whether it is wrong\. For QA tasks, the program can expose intermediate reasoning steps, retrieval queries, evidence summaries, and a final answer; for grading tasks, it can expose dimension\-level scores, weights, and rationales\. These structured outputs define the attributes used to build slices, track regressions, and target the relevant prompt section\.

A contrastive slice is useful for reflection when it satisfies three properties\. First, it should be*coherent*: errors in the slice should differ from nearby successes for a related reason rather than merely sharing the same final label\. Second, it should be*actionable*: the shared attributes should suggest an instruction\-level intervention, such as clarifying an entity\-disambiguation rule or tightening a scoring criterion\. Third, it should be*validation\-relevant*: repairing the slice should plausibly improve held\-out behavior rather than memorize isolated examples\.

These criteria explain why random batches can be weak reflection inputs\. A random batch may contain a formatting error, a retrieval miss, a reasoning error, and an ambiguous label in the same prompt\. The Teacher can still propose an edit, but the edit is forced to average over unrelated causes\. Contrastive reflection gives the Teacher a narrower task: compare the errors and successes in this region, explain what separates them, and write an instruction that repairs the error cases without disturbing nearby correct cases\.

In IR evaluation tasks, the acceptance step can also include regression constraints\. For example, a relevance judge may improve its overall score while degrading a dimension that downstream ranking experiments depend on\. We therefore treat constrained acceptance as an optional safety layer: a candidate prompt is accepted if it improves the task metric and does not violate user\-specified regression tolerances\. The core contribution, however, is the contrastive reflection loop itself\.

## 4\.A Tree\-Based Instantiation

We instantiate contrastive reflection with structured LLM outputs and a shallow entropy\-based decision tree\. Figure[1](https://arxiv.org/html/2606.30840#S4.F1)shows the pipeline\.

Task Definition\+ Initial PromptStudentEvaluationStructured Traces\+ Task AttributesSlice Selector\(Entropy Tree\)Contrastive EvidenceErrors \+ SuccessesTeacher ReflectionTargeted EditValidationGateAcceptedPromptacceptiteratereject: keep prior prompt

Figure 1\.Contrastive Reflection optimization loop\. The Student is evaluated, task\-centric traces are converted into attributes, a slice selector identifies an error\-anchored region, and a Teacher proposes a targeted edit from contrastive evidence\. The edit is accepted only if it passes validation\.A flow diagram showing task definition and initial prompt flowing to student evaluation, structured traces and task attributes, slice selection, contrastive evidence, teacher reflection, validation gate, accepted prompt, and iteration back to student evaluation\.### 4\.1\.Structured Outputs and Features

The method assumes that the LLM program can expose structured outputs\. For QA tasks, these may be intermediate reasoning steps, retrieval queries, summaries, and a final answer\. For grading tasks, these may be dimension\-level scores and rationales\. Structured outputs are compared against benchmark labels or human labels when available\. When those labels are absent, a Teacher can provide diagnostic references for debugging, but such references should be treated as optimizer evidence, not as ground truth\. These comparisons produce features such as step correctness, score differences, reasoning similarity, format validity, and task metadata\. Features derived directly from the final target are excluded from the tree to avoid label leakage\.

This design turns a prompt optimizer into a debugging system\. Instead of only knowing that the final answer is wrong, the algorithm can observe that evidence selection was weak for “who” questions, or that a relevance judge overscored one dimension while keeping others stable\.

### 4\.2\.Contrastive Slice Discovery with an Entropy Tree

The proposed algorithm builds a binary decision tree over training examples using the derived features and a correctness label\. Our implementation uses scikit\-learn’s entropy criterion with CART\-style binary splits\. A candidate splitθ\\thetais selected by information gain:

\(5\)I​G​\(S,θ\)=H​\(S\)−\|SL\|\|S\|​H​\(SL\)−\|SR\|\|S\|​H​\(SR\),IG\(S,\\theta\)=H\(S\)\-\\frac\{\|S\_\{L\}\|\}\{\|S\|\}H\(S\_\{L\}\)\-\\frac\{\|S\_\{R\}\|\}\{\|S\|\}H\(S\_\{R\}\),whereSLS\_\{L\}andSRS\_\{R\}are the left and right partitions induced byθ\\theta, andH​\(S\)=−∑c∈\{0,1\}pc​\(S\)​log2⁡pc​\(S\)H\(S\)=\-\\sum\_\{c\\in\\\{0,1\\\}\}p\_\{c\}\(S\)\\log\_\{2\}p\_\{c\}\(S\)is the entropy of the correctness labels in nodeSS, with0​log⁡0=00\\log 0=0\.

The tree is useful because each root\-to\-leaf path is a compact rule that can be inserted directly into a reflection prompt\. In this instantiation, leaves are selected because they are error\-heavy, but their evidence can include both error cases and successful cases from the same path\. A selected leaf is therefore not just a bag of errors; it is a contrastive slice, such as long predictions in examples where the answer appears in the retrieved context\. This makes the subsequent reflection step more concrete\.

### 4\.3\.Reflection and Acceptance

For each selected leaf, the algorithm gives the Teacher LLM four inputs: the tree path, a small set of error examples, a small set of successful examples from the same region when available, and the relevant prompt section\. The Teacher then repairs the observed error pattern by inferring an instruction\-level distinction: what rule would make the Student treat the error cases more like the successful cases without breaking the successes? The proposed edit is applied to the prompt, not to the examples\. The Student program is then evaluated on validation examples; rejected edits are recorded so later reflection calls avoid repeating the same ineffective strategy\.

## 5\.Experiments

We evaluate on HotpotQA with a lightweight retrieval\-augmented QA program\. For each question, the program retrieves the top four context paragraphs by lexical overlap from the provided HotpotQA context and asks a GPT\-4\.1\-mini Student model for a short answer\. This is a single\-predictor public study, not an evaluation of every agent workflow that motivated the framework\. We use 2,000 training examples for slice discovery, 500 validation examples for acceptance, and 500 held\-out test examples for reporting\. All contrastive\-reflection variants use the same split, retrieval code, Student model, Teacher model, and exact\-match metric\.

Table 1\.HotpotQA exact\-match results for one prompt\-repair step\.Table[1](https://arxiv.org/html/2606.30840#S5.T1)shows that a single tree\-selected contrastive repair improves held\-out exact match from 51\.4% to 60\.4%\. The failure\-only and random\-evidence variants also improve, which indicates that answer\-form precision is a strong repair opportunity in this setup and that any reflection\-based method is likely to find a generic answer\-form fix\. The*distinct*signal for tree\-contrastive in this setup is therefore not the \+1\.4 pp test margin over random contrastive \(well within the bootstrap interval\), but the cleaner repair profile: tree\-contrastive fixed 54 held\-out examples while breaking 9, versus 52 fixed / 14 broken for random contrastive and 35 fixed / 19 broken for failure\-only\. This is the behavior the method is designed to encourage: use failures to choose where to look, use nearby successes to define what must be preserved, and accept only after validation improves\. We report exact McNemarpp\-values for paired exact match and bootstrap 95% confidence intervals over test\-set examples\. We use*fixed*to mean a held\-out example that was incorrect under the baseline prompt and correct after the edit;*broken*means the reverse\.

The selected tree leaf also made the repair mechanism inspectable\. Its failures were mostly answer\-form near misses, such as returning a sentence or descriptor instead of the shortest exact answer\. Nearby successes showed the desired behavior: concise entity or year answers with no extra phrasing\. The accepted Teacher edit can be summarized as one instruction:*return only the exact final short answer as found in the context, with no explanation or added information*\. The edit fixed 54 held\-out examples while breaking 9\.

Table 2\.Ablation study for the same one\-step HotpotQA setup\.Table[2](https://arxiv.org/html/2606.30840#S5.T2)ablates two questions: whether the evidence comes from an interpretable tree slice, and whether the Teacher sees successes as well as errors\. The contrastive tree slice gives the best validation and held\-out gains while breaking the fewest examples\. Random contrastive evidence also improves, but its lower test gain and larger broken count suggest that contrastive evidence is most useful when the selected cases share a coherent behavioral pattern\.

The validation gate also matters\. The first repair is accepted because validation exact match rises from 56\.0% to 64\.0%\. A second tree\-selected repair targets a different failure mode, imprecise short answers that omit qualifiers, but the edit over\-corrects: validation drops to 58\.2%, fixing only 7 validation examples while breaking 36\. The algorithm therefore keeps the first accepted prompt, and the held\-out test result remains 60\.4%\.

Table 3\.Light optimizer sanity check on the same single\-predictor HotpotQA split\. DSPy baselines use instruction\-only settings; GEPA uses a 100\-example optimizer validation slice\.Table[3](https://arxiv.org/html/2606.30840#S5.T3)is a sanity check against modern optimizers, not a definitive optimizer ranking\. We intentionally use the same 2,000/500/500 single\-predictor setup from Table[1](https://arxiv.org/html/2606.30840#S5.T1)and light instruction\-only DSPy settings so the comparison is tied to the headline public experiment\. The configurations are not cost\-matched: GEPA’s reflection budget is a 100\-example optimizer validation slice rather than the 500\-example slice contrastive reflection accepts on, and MIPROv2 is run in instruction\-only mode rather than its full instruction\-and\-demonstration search\. Under these caveats, MIPROv2 is close on held\-out accuracy \(the 1\.0 pp gap sits within the bootstrap interval reported in Table[1](https://arxiv.org/html/2606.30840#S5.T1)\), so we read the result as*competitive, not better*: contrastive reflection lands in the same band as a modern optimizer while also producing a compact account of the failure slice, the nearby successes, and the accepted edit\. GEPA improves over the baseline as well, but its selected prompt generalizes less strongly to the held\-out split in this single run\. Cost\-equivalent benchmarks for multi\-stage agent programs remain future work\.

### 5\.1\.LinkedIn\-Derived Grading Workflows

The LinkedIn\-derived grading setting remains important to the narrative because it is where the task\-centric quality definition arose\. In these workflows, an LLM judge assigns a final 0–4 grade from structured evidence while also producing dimension\-level scores and rationales\. The final grade provides the scalar metric, but the dimensions and rationales explain*how*the judge is wrong and which instruction should be edited\.

Refinement of the method on this setting is ongoing, so we do not report a formal table in this workshop version\. Internal development has shown approximately a \+12 percentage\-point gain on held\-out grading accuracy, with useful prompt repairs typically reached in 3–5 optimization epochs\. We treat this as motivating evidence rather than a public benchmark claim because the examples, rubrics, and labels are not distributable\. The lesson we carry into the public HotpotQA study is about speed and inspectability: structured slices let engineers identify which grading dimension is failing, compare it with nearby correct judgments, and make a targeted grading\-rule edit without waiting for many rounds of unconstrained prompt search\.

## 6\.Discussion

The central lesson is that prompt optimization can be treated as a debugging loop\. Scalar validation metrics remain necessary, but they do not tell the optimizer where to reason\. Contrastive reflection adds a middle layer: find a coherent slice where errors and successes diverge, describe the contrast, and ask for a repair that targets that slice\. The HotpotQA result should therefore be read as evidence for the mechanism, not as proof that a decision tree is always the best optimizer\.

The decision tree is a practical instantiation, not the final word\. Trees are attractive because they produce human\-readable paths and natural reflection prompts\. Other slice selectors may work as well, and future work should compare trees against clustering, embedding\-based retrieval, rule mining, and human\-authored slices\. More seeds and more tasks are also needed before making strong claims about optimizer rankings\.

The approach has limitations that future work should address\. It requires structured outputs, which add token overhead and may require prompt/interface design\. Teacher\-generated diagnostic references can encode Teacher errors or biases and should not be treated as human labels\. The public HotpotQA study is intentionally narrow: it tests one\-step prompt repair on one dataset, a single\-predictor QA program, and a single random seed; variance is reported only via bootstrap intervals on test predictions, not across reruns of the full optimization loop\. Two specific gaps follow from this scope\. First, because only one edit is ultimately accepted on the public benchmark, the experiments mostly exercise the contrastive*step*rather than the iterative loop that Algorithm[1](https://arxiv.org/html/2606.30840#alg1)centers on\. Second, the regression\-constraint machinery in Equation[4](https://arxiv.org/html/2606.30840#S3.E4)is defined and used in the LinkedIn\-derived grading setting but is not exercised on the public HotpotQA setup, where validation\-only acceptance is sufficient\. Broader task coverage, multi\-stage agent programs, multi\-seed reruns of the full loop, and a public regression\-constrained study remain future work\. The LinkedIn\-derived grading observations are included only as motivating context because the data and rubrics are not distributable; they should not be read as a fully reproducible public benchmark\.

#### Reproducibility\.

All headline HotpotQA runs use the same 2,000/500/500 split, exact\-match scoring, retrieval code, and GPT\-4\.1\-mini Student/Teacher models\. The implementation records prompts, traces, slice reports, token usage, and before/after comparisons for each table row\.

## 7\.Conclusion

Contrastive Reflection frames prompt optimization as an iterative loop of task\-centric quality definition, slice discovery, reflection, targeted editing, and validation\. Failures identify where the prompt is weak; nearby successes reveal the behavior that the repair should preserve\. Our proposed algorithm instantiates this loop with structured outputs and tree\-based contrastive slice discovery\. On HotpotQA, it improves held\-out exact\-match accuracy from 51\.4% to 60\.4% in one repair step\. Ongoing LinkedIn\-derived grading work motivates the same design goal: reach useful prompt repairs quickly while preserving an inspectable account of which task dimension changed\. The broader agenda is to make prompt optimization less like blind search and more like inspectable, repeatable debugging\.

## References

- L\. A\. Agrawal, S\. Tan, D\. Soylu, N\. Ziems, R\. Khare, K\. Opsahl\-Ong, A\. Singhvi, H\. Shandilya, M\. J\. Ryan, M\. Jiang, C\. Potts, K\. Sen, A\. G\. Dimakis, I\. Stoica, D\. Klein, M\. Zaharia, and O\. Khattab \(2025\)GEPA: reflective prompt evolution can outperform reinforcement learning\.arXiv preprint arXiv:2507\.19457\.Note:Accepted to ICLR 2026Cited by:[§1](https://arxiv.org/html/2606.30840#S1.p3.1),[§2](https://arxiv.org/html/2606.30840#S2.p2.1)\.
- Y\. Chung, T\. Kraska, N\. Polyzotis, K\. H\. Tae, and S\. E\. Whang \(2019\)Slice finder: automated data slicing for model validation\.InICDE,Cited by:[§2](https://arxiv.org/html/2606.30840#S2.p3.1)\.
- S\. Eyuboglu, M\. Varma, K\. Saab, J\. Delbrouck, C\. Lee\-Messer, J\. Dunnmon, J\. Zou, and C\. Ré \(2022\)Domino: discovering systematic errors with cross\-modal embeddings\.InICLR,Cited by:[§2](https://arxiv.org/html/2606.30840#S2.p3.1)\.
- N\. Farzi and L\. Dietz \(2025\)Criteria\-based LLM relevance judgments\.arXiv preprint arXiv:2507\.09488\.Cited by:[§1](https://arxiv.org/html/2606.30840#S1.p1.1),[§2](https://arxiv.org/html/2606.30840#S2.p3.1)\.
- C\. Fernandoet al\.\(2023\)Promptbreeder: self\-referential self\-improvement via prompt evolution\.arXiv preprint arXiv:2309\.16797\.Cited by:[§2](https://arxiv.org/html/2606.30840#S2.p2.1)\.
- Q\. Guoet al\.\(2024\)Connecting large language models with evolutionary algorithms yields powerful prompt optimizers\.InICLR,Cited by:[§2](https://arxiv.org/html/2606.30840#S2.p2.1)\.
- O\. Khattab, A\. Singhvi, P\. Maheshwari, Z\. Zhang, K\. Santhanam, S\. V\. A\., S\. Haq, A\. Sharma, T\. Joshi, H\. Moazam, H\. Miller, M\. Zaharia, and C\. Potts \(2024\)DSPy: compiling declarative language model calls into state\-of\-the\-art pipelines\.InICLR,Cited by:[§1](https://arxiv.org/html/2606.30840#S1.p3.1),[§2](https://arxiv.org/html/2606.30840#S2.p1.1)\.
- B\. Lester, R\. Al\-Rfou, and N\. Constant \(2021\)The power of scale for parameter\-efficient prompt tuning\.InEMNLP,Cited by:[§2](https://arxiv.org/html/2606.30840#S2.p1.1)\.
- K\. Opsahl\-Ong, M\. J\. Ryan, J\. Purtell, D\. Broman, C\. Potts, M\. Zaharia, and O\. Khattab \(2024\)Optimizing instructions and demonstrations for multi\-stage language model programs\.arXiv preprint arXiv:2406\.11695\.Note:EMNLP 2024Cited by:[§1](https://arxiv.org/html/2606.30840#S1.p3.1),[§2](https://arxiv.org/html/2606.30840#S2.p1.1)\.
- R\. Pryzantet al\.\(2023\)Automatic prompt optimization with “gradient descent” and beam search\.InEMNLP,Cited by:[§2](https://arxiv.org/html/2606.30840#S2.p1.1)\.
- M\. T\. Ribeiro, T\. Wu, C\. Guestrin, and S\. Singh \(2020\)Beyond accuracy: behavioral testing of NLP models with CheckList\.InACL,Cited by:[§2](https://arxiv.org/html/2606.30840#S2.p3.1)\.
- P\. Thomas, S\. Spielman, N\. Craswell, and B\. Mitra \(2024\)Large language models can accurately predict searcher preferences\.Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval\.Note:Earlier version: arXiv:2309\.10621Cited by:[§1](https://arxiv.org/html/2606.30840#S1.p1.1),[§2](https://arxiv.org/html/2606.30840#S2.p3.1)\.
- X\. Wanget al\.\(2024\)PromptAgent: strategic planning with language models enables expert\-level prompt optimization\.InICLR,Cited by:[§2](https://arxiv.org/html/2606.30840#S2.p1.1)\.
- Y\. Wuet al\.\(2024\)StraGo: harnessing strategic guidance for prompt optimization\.InFindings of EMNLP,Note:arXiv:2410\.08601Cited by:[§2](https://arxiv.org/html/2606.30840#S2.p2.1)\.
- C\. Yanget al\.\(2024\)Large language models as optimizers\.InICLR,Cited by:[§2](https://arxiv.org/html/2606.30840#S2.p1.1)\.
- Z\. Yang, P\. Qi, S\. Zhang,et al\.\(2018\)HotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InEMNLP,Cited by:[§1](https://arxiv.org/html/2606.30840#S1.p5.1)\.
- Y\. Zhouet al\.\(2023\)Large language models are human\-level prompt engineers\.InICLR,Cited by:[§2](https://arxiv.org/html/2606.30840#S2.p1.1)\.