CAFE: A Compound-AI Factorial Evaluation Framework

# CAFE: A Compound-AI Factorial Evaluation Framework
Source: [https://arxiv.org/html/2607.10380](https://arxiv.org/html/2607.10380)
Fabian Lukassen1Christoph Weisser2Thomas Kneib1Alexander Silbersdorff1 1University of Göttingen2Bielefeld University of Applied Sciences and Arts \(HSBI\) fabian\.lukassen@uni\-goettingen\.dechristoph\.weisser@hsbi\.de tkneib@uni\-goettingen\.deasilbersdorff@uni\-goettingen\.de

###### Abstract

We introduceCAFE\(Compound\-AIFactorialEvaluation\), an open\-source platform that brings*design of experiments*to the evaluation of compound AI systems \(CAIS\)\. Such systems expose many interchangeable choices – e\.g\. which retriever, model, or prompt – and practitioners rarely know which of them most affects answer quality\. With CAFE, a practitioner registers each swappable component of a pipeline as a factor to build a factorial design over the chosen factors, run the resulting configurations, and score the answers on a shared rubric using a configurable LLM judge together with human raters\. From these ratings it attributes answer\-quality variance to the components and their interactions with mixed\-effects models and reports effect sizes, significance, the best configuration, cost and latency trade\-offs, and judge–human reliability\. Whereas existing tools mostly either*search*for a good configuration or*score*outputs in isolation, CAFE also*explains*which component drives quality and whether an observed difference is significant\. We validate CAFE on a retrieval\-augmented question\-answering \(QA\) pipeline over the HotpotQA benchmark dataset, where it recovers planted factor effects and stays calibrated under a permutation null\. CAFE is released as a Python package and as a Web application\.

CAFE: A Compound\-AI Factorial Evaluation Framework

††Landing page:[https://cafe\-ai\.de](https://cafe-ai.de/)Demo:[https://cafe\-ai\.de/demo](https://cafe-ai.de/demo)Code:[https://github\.com/fabian\-lu/Cafe](https://github.com/fabian-lu/Cafe)Documentation:[https://fabian\-lu\.github\.io/Cafe](https://fabian-lu.github.io/Cafe)## 1Introduction

Modern AI applications rarely rest on a single language model\. The current predominant pattern is the*compound AI system*\(Zahariaet al\.,[2024](https://arxiv.org/html/2607.10380#bib.bib1); Chenet al\.,[2025a](https://arxiv.org/html/2607.10380#bib.bib2)\): an application that solves a task with several interacting components such as retrieval, reranking, context assembly, prompting, one or more model calls, tool use, and guardrails\. Each component can be realized by several competing techniques, retrieval alone might be sparse keyword search, dense\-embedding retrieval, or a reranking pipeline; any other component can be swapped just as freely\. Because these discrete, often non\-differentiable choices combine, even a modest pipeline induces a large, combinatorial space of candidate*configurations*\. The growing literature helps practitioners*build*and*optimize*such pipelines, from declarative programming models\(Khattabet al\.,[2024](https://arxiv.org/html/2607.10380#bib.bib4)\)to end\-to\-end configuration search\(Kartalet al\.,[2025](https://arxiv.org/html/2607.10380#bib.bib25); Leeet al\.,[2025](https://arxiv.org/html/2607.10380#bib.bib3)\)\.

However, choosing which configuration to ship raises questions that optimizers only answer indirectly\. Suppose that the builder of a retrieval\-augmented assistant can pick among three base models, two retrieval corpora, and three prompts: 18 candidate configurations\. The practitioner’s immediate goal is the single best\-performing one, which CAFE reports directly; but a trustworthy, actionable decision rests on two coupled questions\. First,*which component drives answer quality and where is effort best spent?*A team’s time is finite, and it matters whether the next gain will come from engineering the prompt or from building a stronger retrieval pipeline\. Components also interact, often non\-monotonically – a stronger model may help only when retrieval is good, and adding model calls can help some queries while hurting others\(Chenet al\.,[2024](https://arxiv.org/html/2607.10380#bib.bib7)\)– so a disjoint perspective risks to mismeasure the system by neglecting such interactions\. Second,*is an observed difference significant?*A gap between two configurations can be an artifact of chance rather than a genuine effect, and comparing averaged scores from a single run of each gives no way to tell the two apart\. An optimizer returns the best\-scoring configuration, but not a calibrated account of*why*it wins or*whether*the margin is trustworthy\.

To address these issues CAFE employs*design of experiments*\(DoE\) and mixed\-effects models\(Fisher,[1935](https://arxiv.org/html/2607.10380#bib.bib34); Montgomery,[2017](https://arxiv.org/html/2607.10380#bib.bib35); Pinheiro and Bates,[2000](https://arxiv.org/html/2607.10380#bib.bib36)\)\. A factorial design estimates each component’s effect*and*the interactions among components from a balanced set of runs; replicating runs and modeling the outcome with random effects separates real differences both from noise and from the fact that some inputs are simply harder than others\. However, there is currently no usable system that packages this methodology for people who build compound AI pipelines\.

CAFE is an open\-source evaluation framework with four contributions: \(i\) apipeline\-as\-factors abstractionthat surfaces any swappable component as a*factor*whose alternative implementations become its*levels*; \(ii\) adesign\-and\-execution enginethat builds a factorial design over the chosen factors, runs each configuration against the target system; \(iii\) adual\-rating workflowin which the same answers are scored on a shared rubric by a fully configurable LLM judge and by humans, with inter\-rater reliability reported between them; and \(iv\) avariance\-attribution layerthat fits mixed\-effects models matched to the rating scale, reports how much each component and interaction contributes with effect sizes and significance, and identifies the best configuration\.

CAFE is released as an open\-source early\-stage[Python package](https://github.com/fabian-lu/Cafe)\(licensed under Apache 2\.0\) with a companion web application \(a screenshot of the web interface is in Appendix[A](https://arxiv.org/html/2607.10380#A1)\); we also provide[example notebooks](https://fabian-lu.github.io/Cafe/notebooks/01_quickstart/)for system demonstration\. The project welcomes any[contributions](https://github.com/fabian-lu/Cafe/blob/main/CONTRIBUTING.md), where[roadmap](https://cafe-ai.de/roadmap.html)lists planned features, a selection of which we also highlight in Section[6](https://arxiv.org/html/2607.10380#S6)\.

Table 1:CAFE versus representative systems\.
## 2Related Work

The evaluation of CAIS touches on three lines of work: tools that*optimize*pipelines, tools that*score*their outputs, and the statistics of AI evaluation\. Table[1](https://arxiv.org/html/2607.10380#S1.T1)contrasts CAFE with representative evaluation systems\.

#### Optimizing compound systems\.

One line of work*searches*the configuration space for a high\-scoring pipeline: DSPy compiles and tunes declarative pipelines\(Khattabet al\.,[2024](https://arxiv.org/html/2607.10380#bib.bib4)\), TextGrad backpropagates textual feedback\(Yuksekgonulet al\.,[2025](https://arxiv.org/html/2607.10380#bib.bib5)\), LLMSelector allocates models to modules\(Chenet al\.,[2025b](https://arxiv.org/html/2607.10380#bib.bib6)\), Optimas aligns local rewards with global performance\(Wuet al\.,[2026](https://arxiv.org/html/2607.10380#bib.bib9)\), and RAGSmith and multi\-objective search explore large RAG spaces\(Kartalet al\.,[2025](https://arxiv.org/html/2607.10380#bib.bib25); Barkeret al\.,[2025](https://arxiv.org/html/2607.10380#bib.bib8)\)\.

#### Evaluation frameworks and tools\.

Holistic benchmarks such as HELM\(Lianget al\.,[2023](https://arxiv.org/html/2607.10380#bib.bib15)\)characterise*models*across scenarios rather than*pipelines*across component choices\. Metric libraries score pipeline outputs with LLM judges: RAGAS\(Eset al\.,[2024](https://arxiv.org/html/2607.10380#bib.bib13)\)and ARES\(Saad\-Falconet al\.,[2024](https://arxiv.org/html/2607.10380#bib.bib14)\), which adds prediction\-powered confidence intervals\(Fischet al\.,[2024](https://arxiv.org/html/2607.10380#bib.bib51)\), alongside DeepEval\(Confident AI,[2024](https://arxiv.org/html/2607.10380#bib.bib22)\), apytest\-style framework offering G\-Eval\(Liuet al\.,[2023](https://arxiv.org/html/2607.10380#bib.bib24)\)and component\-level metrics, and TruLens\(TruEra,[2024](https://arxiv.org/html/2607.10380#bib.bib23)\), which computes feedback functions over execution traces\. Execution and annotation platforms \(LangSmith\(LangChain,[2024](https://arxiv.org/html/2607.10380#bib.bib19)\), Inspect AI\(UK AI Security Institute,[2024](https://arxiv.org/html/2607.10380#bib.bib17)\), and Argilla\(Argilla,[2024](https://arxiv.org/html/2607.10380#bib.bib18)\)\) supply run harnesses, judge scoring, and human annotation queues\.

#### Statistics for AI evaluation\.

A growing literature views evaluation through the lense of statistical inference: quantifying benchmark uncertainty\(Wanget al\.,[2025](https://arxiv.org/html/2607.10380#bib.bib63)\), hierarchical Bayesian modeling\(Luettgauet al\.,[2025](https://arxiv.org/html/2607.10380#bib.bib53)\), measurement error in LLM pipelines\(Messing,[2026](https://arxiv.org/html/2607.10380#bib.bib55)\), and ordinal rather than interval models for rubric ratings\(Howcroft and Rieser,[2021](https://arxiv.org/html/2607.10380#bib.bib39); Tayloret al\.,[2023](https://arxiv.org/html/2607.10380#bib.bib41); Syiem and Velloso,[2026](https://arxiv.org/html/2607.10380#bib.bib40)\); variance\-based sensitivity analysis is the classical instrument for apportioning output variance to inputs\(Saltelliet al\.,[2008](https://arxiv.org/html/2607.10380#bib.bib32)\)\. The two closest works areHaaseet al\.\([2026](https://arxiv.org/html/2607.10380#bib.bib26)\), who partition output variance into prompt, model and sampling components but for standalone creative generation with a linear model and a single automatic rater, andMustahsanet al\.\([2025](https://arxiv.org/html/2607.10380#bib.bib59)\), who summarise run\-to\-run inconsistency with a single intraclass correlation; neither attributes variance across a designed factorial spanning the full pipeline\. Recent work also attributes faults or perturbations within compound systems\(Chowdhury and D’Souza,[2026](https://arxiv.org/html/2607.10380#bib.bib27); Nilayam and Nayak,[2026](https://arxiv.org/html/2607.10380#bib.bib28)\), while a parallel effort calibrates and audits LLM judges\(Guet al\.,[2024](https://arxiv.org/html/2607.10380#bib.bib56); Hanet al\.,[2025](https://arxiv.org/html/2607.10380#bib.bib57); Boyeauet al\.,[2025](https://arxiv.org/html/2607.10380#bib.bib50); Parket al\.,[2025](https://arxiv.org/html/2607.10380#bib.bib52); Duboiset al\.,[2025](https://arxiv.org/html/2607.10380#bib.bib54); Guerdanet al\.,[2025](https://arxiv.org/html/2607.10380#bib.bib58)\)and documents the nondeterminism that motivates replication\(Atilet al\.,[2024](https://arxiv.org/html/2607.10380#bib.bib60); Haldar and Hockenmaier,[2025](https://arxiv.org/html/2607.10380#bib.bib62); Bjarnasonet al\.,[2026](https://arxiv.org/html/2607.10380#bib.bib61)\)\. Factorial studies of LLM outputs are beginning to appear\(Vazquezet al\.,[2025](https://arxiv.org/html/2607.10380#bib.bib33); Lukassenet al\.,[2026](https://arxiv.org/html/2607.10380#bib.bib31)\)\.

#### The gap\.

No existing system combines these threads \(Table[1](https://arxiv.org/html/2607.10380#S1.T1)\)\. Optimizers return the best\-scoring configuration but not a calibrated account of*why*it is best\. Metric libraries and execution platforms score outputs and gather ratings, but leave the experimental design and statistical analysis to the user\. The statistical methods above are directly relevant, yet are not packaged as a usable tool\. Section[3](https://arxiv.org/html/2607.10380#S3)now describes how CAFE operationalizes this gap as a framework\.

## 3The CAFE Framework

![Refer to caption](https://arxiv.org/html/2607.10380v1/x1.png)Figure 1:CAFE architecture\.CAFE turns the choices in a CAIS into experimental*factors*\(Figure[1](https://arxiv.org/html/2607.10380#S3.F1)\)\. A study varies factorsF1,…,FpF\_\{1\},\\dots,F\_\{p\}, each ranging over a finite set of*levels*LiL\_\{i\}; choosing one level per factor gives a*configuration*c=\(c1,…,cp\)∈𝒞=L1×⋯×Lpc=\(c\_\{1\},\\dots,c\_\{p\}\)\\in\\mathcal\{C\}=L\_\{1\}\\times\\cdots\\times L\_\{p\}, and the*full factorial*is this entire product\. Runningccon an inputx∈𝒳x\\in\\mathcal\{X\}yields an outputRun​\(c,x\)\\mathrm\{Run\}\(c,x\)that a rubric maps to a quality ratingYY; the system and judge being nondeterministic,YYis a random variable, since the samecccan score differently across runs\. Two kinds of choice become factors:*which component*performs a step – e\.g\. the retriever, with levelsnone,dense,dense\_rerank– and a*parameter*of a component, such as the decoding temperature\.

When a pipeline is built inside CAFE, these factors are read off its structure: the system is an ordered sequence of*stages*s1,…,sms\_\{1\},\\dots,s\_\{m\}\(each a point where a choice is made\), and the*techniques*registered for a stagesks\_\{k\}are the levels of that stage’s factor, while a technique’s tunable parametersk\.ps\_\{k\.p\}\(e\.g\.retrieve\.top\_k\) is a further factor\.

Given factors and inputs, CAFE casts the questions of Section[1](https://arxiv.org/html/2607.10380#S1)as estimation problems onYY:*attribution*– how much of the variation inYYeach factorFiF\_\{i\}and each interaction explains;*selection*– the configurationarg⁡maxc⁡𝔼​\[Y∣c\]\\arg\\max\_\{c\}\\mathbb\{E\}\[Y\\mid c\]; and*significance*for both, with valid uncertainty under the run\-to\-run variation ofRun\\mathrm\{Run\}\. Algorithm[1](https://arxiv.org/html/2607.10380#alg1)formalises this procedure\. We describe each part in turn\.

Algorithm 1A CAFE study1:declare factors

F1,…,FpF\_\{1\},\\dots,F\_\{p\}and their levels

L1,…,LpL\_\{1\},\\dots,L\_\{p\}
2:

𝒞←L1×⋯×Lp\\mathcal\{C\}\\leftarrow L\_\{1\}\\times\\dots\\times L\_\{p\}⊳\\trianglerightfull factorial: every configuration

3:for allconfiguration

c∈𝒞c\\in\\mathcal\{C\}, input

x∈𝒳x\\in\\mathcal\{X\}, replicate

rrdo

4:

ac​x​r←Run​\(c,x\)a\_\{cxr\}\\leftarrow\\textsc\{Run\}\(c,x\)⊳\\trianglerightexecute; concurrency\-bounded, resumable

5:

yc​x​r←Rate​\(ac​x​r\)y\_\{cxr\}\\leftarrow\\textsc\{Rate\}\(a\_\{cxr\}\)⊳\\trianglerightLLM judge and/or human, shared rubric

6:endfor

7:fit the model matched to the rubric scale \(CLMM / logistic / linear\)

8:attribute variance; report effect sizes, significance, the best configuration, the cost frontier, and

α\\alpha

### 3\.1Defining a system

CAFE supports two ways to supplyRun\\mathrm\{Run\}from Algorithm[1](https://arxiv.org/html/2607.10380#alg1)\. In the common case, the pipeline is*composed*in CAFE itself: each technique is registered on its stage with a one\-line decorator, and an orchestration function wires the stages together \(Listing[1](https://arxiv.org/html/2607.10380#LST1)\)\. CAFE then varies each factor over its levels and, because it executes the pipeline, records per\-stage latency and dollar cost with no extra wiring\. One level of a factor may beNone, which tells CAFE to skip that stage entirely, so a component can be switched on and off \(e\.g\. running with or without a reranker\)\. Alternatively, a system that cannot or need not be instrumented – e\.g\. an external HTTP endpoint – is driven as a*black box*: any callablerun\(config, item\)is a system – CAFE passes it a configuration and an input, and the callable reads each factor’s chosen level fromconfig– and its factors are declared directly\. Black\-box mode gives up only the per\-stage latency and cost breakdown, leaving the design, rating, and variance attribution unchanged\.

importcafe

pipe=cafe\.Pipeline\(\)

@pipe\.technique\(stage="retrieve",name="none"\)

asyncdefretrieve\_none\(ctx,query\):

return\[\]

@pipe\.technique\(stage="retrieve",name="dense",cost\_usd=0\.0004\)

asyncdefretrieve\_dense\(ctx,query,top\_k=8\):

returndense\_search\(query,k=top\_k\)

@pipe\.technique\(stage="generate",name="llm"\)

asyncdefgenerate\(ctx,query,docs,model="gpt\-oss\-20b"\):

returnawaitcafe\.complete\(model,prompt\(query,docs\)\)

@pipe\.compose

asyncdefrag\(config,item,ctx\):

docs=awaitctx\.run\("retrieve",query=item\["text"\]\)

returnawaitctx\.run\("generate",query=item\["text"\],docs=docs\)

study=cafe\.Study\(

name="rag\-study",

system=pipe,

factors=\[pipe\.factor\("retrieve"\),

cafe\.Factor\("generate\.model",\["gpt\-oss\-20b","gpt\-oss\-120b"\]\)\],

dataset=data,rubric=rubric,judge=judge,replications=2,

\)

Listing 1:Composed mode: techniques are registered on stages with decorators and@composewires them into a pipeline;cost\_usdattaches a per\-call price CAFE tracks\. The sameStudyruns unchanged in black\-box mode over anyrun\(config, item\)callable\.
### 3\.2Design, cost, and execution

From the chosen factors, CAFE enumerates a*full factorial*– every combination of levels – andcafe\.preflightreports the configuration count, an estimated compute time and dollar cost, and a design\-adequacy check \(for instance, too few inputs for a stable random effect\) before anything runs\. Execution then runs each pair\(c,x\)∈𝒞×𝒳\(c,x\)\\in\\mathcal\{C\}\\times\\mathcal\{X\}under a configurable concurrency bound and checkpoints incrementally, so a study interrupted after hours resumes rather than restarts\. Every cell is run with*replication*: a compound system can return different outputs on repeated calls even at temperature zero\(Atilet al\.,[2024](https://arxiv.org/html/2607.10380#bib.bib60)\), and the repeats let the analysis estimate within\-cell variation instead of mistaking it for an effect\. Per\-stage latency and cost are recorded throughout, feeding the multi\-objective analysis of Section[3\.4](https://arxiv.org/html/2607.10380#S3.SS4)\.

### 3\.3Rating

Each answer is scored on a*rubric*that fixes an*ordinal*,*binary*, or*numeric*scale\. Two rater types apply the same rubric\. An*LLM judge*is fully configurable – its model, its prompt, and the rubric it applies are all chosen by the user, and it scores with or without a reference answer; CAFE ships prompt presets and stores the exact prompt and raw response behind every verdict for auditing\. Repeated judge passes are retained so that judge self\-inconsistency can be measured\(Haldar and Hockenmaier,[2025](https://arxiv.org/html/2607.10380#bib.bib62)\)\.*Humans*rate through a fillable answer sheet \(cafe\.answer\_sheetexports it as csv,cafe\.human\_ratingsreads it back\), thus allowing reliability analysis between them\.

### 3\.4Statistical analysis

The statistical analysis is CAFE’s analytical core, reported in three layers, and it is deliberately driven by the rubric: the declared scale type selects the model, so each response is analysed on the scale it lives on rather than coerced to an interval mean\.

*Descriptive\.*Per\-level marginal means, per\-configuration means, and the best configuration give the immediate picture of the design\.

*Scale\-matched inference\.*WriteYYfor an answer’s rating andyc​x​ry\_\{cxr\}for its value under configurationccon inputxx, replicaterr, withccsummarised by its factor\-level design vector𝐳c\\mathbf\{z\}\_\{c\}\(main effects and, by default, two\-way interactions\), and letη=𝐳c⊤​𝜷\+ux\\eta=\\mathbf\{z\}\_\{c\}^\{\\top\}\\boldsymbol\{\\beta\}\+u\_\{x\}be the linear predictor, whereux∼𝒩​\(0,σ2\)u\_\{x\}\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}\)is a per\-input random intercept\. The declared scale selects one of three mixed models:

ordinal \(CLMM\):logit⁡Pr⁡\(Y≤q\)=θq−η,\\displaystyle\\operatorname\{logit\}\\Pr\(Y\\leq q\)=\\theta\_\{q\}\-\\eta,binary \(GLMM\):logit⁡Pr⁡\(Y=1\)=η,\\displaystyle\\operatorname\{logit\}\\Pr\(Y=1\)=\\eta,numeric \(LMM\):Y=η\+ε\.\\displaystyle Y=\\eta\+\\varepsilon\.The ordinal case is a*cumulative link mixed model*\(McCullagh,[1980](https://arxiv.org/html/2607.10380#bib.bib38); Agresti,[2002](https://arxiv.org/html/2607.10380#bib.bib37); Tayloret al\.,[2023](https://arxiv.org/html/2607.10380#bib.bib41)\); unlike the binary and numeric models it has no fixed intercept, as itsK−1K\-1ordered thresholdsθ1<⋯<θK−1\\theta\_\{1\}<\\dots<\\theta\_\{K\-1\}play that role \(which keeps𝐳c⊤​𝜷\\mathbf\{z\}\_\{c\}^\{\\top\}\\boldsymbol\{\\beta\}identifiable\)\. All three share the per\-input*random*interceptuxu\_\{x\}, which absorbs the fact that some inputs are simply harder, so factor effects are estimated*within*input\. Because Python has no standard fitter for the ordinal and binary models, CAFE fits them by callingRin a subprocess,ordinal::clmm\(Christensen,[2023](https://arxiv.org/html/2607.10380#bib.bib49)\)andlme4::glmer\(Bateset al\.,[2015](https://arxiv.org/html/2607.10380#bib.bib48)\), passing the data as a table and reading the estimates\. The linear model is fit natively\.

*Effect sizes and attribution\.*Alongside the scale\-matched model, a linear mixed model gives an*approximate*, scale\-agnostic effect\-size view: a Type\-II analysis of variance\(Langsrud,[2003](https://arxiv.org/html/2607.10380#bib.bib44)\)gives each factor anFF\-test and a partialη2\\eta^\{2\}\(Cohen,[1988](https://arxiv.org/html/2607.10380#bib.bib43)\), a modelR2R^\{2\}\(Nakagawa and Schielzeth,[2013](https://arxiv.org/html/2607.10380#bib.bib47)\), and pairwise contrasts give Cohen’sdd\(Cohen,[1988](https://arxiv.org/html/2607.10380#bib.bib43)\)\. Ranking factors by partialη2\\eta^\{2\}is a variance attribution\(Saltelliet al\.,[2008](https://arxiv.org/html/2607.10380#bib.bib32)\)that answers “which component drives quality,” and the best end\-to\-end configuration is read from the fitted means\. Because it treats the scale as interval, this view is a standardized summary for ranking and cross\-study comparison rather than the primary inference \(Section[6](https://arxiv.org/html/2607.10380#S6)\)\.

*Reliability and robustness\.*Because the quality signal is itself produced by raters, CAFE computes Krippendorff’sα\\alpha\(Krippendorff,[2004](https://arxiv.org/html/2607.10380#bib.bib42)\)between the LLM judge and the humans, and among the humans, with the distance metric matched to the scale \(ordinal, nominal, or interval\)\. Finally, because latency and cost are recorded, CAFE reports the*Pareto frontier*\(Barkeret al\.,[2025](https://arxiv.org/html/2607.10380#bib.bib8)\)over quality, cost, and latency, exposing a configuration that is marginally better, but far slower or more expensive\.

### 3\.5Implementation and availability

CAFE is distributed as a Python package with modules for design, execution, judging, and statistics\. A web application exposes the same engine over a FastAPI \(JSON\) API that a React front\-end consumes, backed by a PostgreSQL store; studies run as in\-process asynchronous task, and the whole stack is deployed with Docker Compose\. The app discovers the available systems from a configured techniques directory, where@composeand@techniqueregistrations populate its factor/level menus\.

## 4Case Study and Evaluation

We validate CAFE againstknown answers: we run it on a real compound system built so that certain effects hold*by construction*, then check whether CAFE recovers them \(the full study is a runnable[notebook](https://fabian-lu.github.io/Cafe/notebooks/evaluation/evaluation/)\)\. Concretely, CAFE must*recover*an effect we know is present while ignoring one we know is absent \(E1, construct validity\);*not invent*significance once we destroy all real structure \(E2, calibration\);*agree with human*raters, since the whole analysis rests on the judge’s scores \(E3\); and select a best configuration that*generalizes*beyond the exact questions asked \(E4\)\.

#### The task\.

The target system is a retrieval\-augmented QA pipeline evaluated onHotpotQA\(Yanget al\.,[2018](https://arxiv.org/html/2607.10380#bib.bib10)\), a multi\-hop question\-answering benchmark in which answering a question requires combining facts from two different Wikipedia*paragraphs*\. Each question comes with \(i\) a short gold answer and \(ii\) ten candidate paragraphs, of which two are the*gold*paragraphs that contain the supporting facts and eight are*distractors*\. Thus, we know both the correct answer and the relevant evidence\. We keep the hard, multi\-hop, non\-yes / no questions \(yes / no answers are guessable and would blunt the retrieval signal\) and samplen=50n=50\.

#### Screening\.

Because a question’s own ten paragraphs make retrieval trivial, we pool*every*question’s paragraphs into one shared∼\\sim500\-document corpus, embedded withbge\-m3: each question’s two gold paragraphs must now be found among∼\\sim500\. Before spending anything on the full study we*screen*this corpus by measuring*gold\-paragraph recall@kk*– how often the retriever’s top\-kkactually contains a question’s gold paragraphs\. Recall is0\.760\.76atk=2k\{=\}2,0\.870\.87atk=4k\{=\}4, and0\.970\.97atk=12k\{=\}12\. It confirms the gold evidence is retrievable, so a no\-retrieval baseline is*guaranteed*to be worse, the known\-signed anchor\. Furthermore, it shows that plain top\-2 retrieval misses the gold about a quarter of the time, while the top\-12 almost always contains it: that gap is precisely the room a*reranker*would need to help\.

#### The factors\.

The system exposes three factors \(3×3×2=183\\times 3\\times 2=18configurations\):

- •retrieve∈\{\\in\\\{none, dense, dense\_rerank\}\\\}– theanchor\.nonepasses no evidence \(the model answers from parametric memory\);densefeeds thebge\-m3top\-2;dense\_rerankhas an LLM reorder the top\-12 down to 2, a RankGPT\-style reranker\(Sunet al\.,[2023](https://arxiv.org/html/2607.10380#bib.bib11)\)\.
- •finalize∈\{\\in\\\{on, off\}\\\}– thenegative control\. Both levels return the answer unchanged, so the true effect is exactly zero and CAFE must call it non\-significant\.
- •generate\.model∈\{\\in\\\{gemma3\-4b, gpt\-oss\-20b, gpt\-oss\-120b\}\\\}– anexpectedeffect: larger models should score higher\(Kaplanet al\.,[2020](https://arxiv.org/html/2607.10380#bib.bib64); Robertset al\.,[2020](https://arxiv.org/html/2607.10380#bib.bib65)\); open is*by how much*, and whether strong retrieval makes model size matter less\.

#### Scoring\.

Each of the18×50=90018\\times 50=900answers is graded on a three\-level ordinal correctness rubric –0incorrect/off\-topic,11partially correct,22fully correct \(exact wording irrelevant\) – by a*separate*, strong judge model \(DeepSeek\-V4\-Pro\)\. The judge is shown the question, the*gold reference answer*, and the answer to grade, and is instructed to score factual correctness against the reference\. The completereport\(\)output of this study is in Appendix[A](https://arxiv.org/html/2607.10380#A1)\(Listing[2](https://arxiv.org/html/2607.10380#LST2)\)\.

#### E1: recovery of known effects\.

Reading Listing[2](https://arxiv.org/html/2607.10380#LST2)top to bottom, CAFE reproduces what we built in\. Theanchoris recovered with the right order and dominance: the marginal mean quality risesnone0\.75<0\.75<dense1\.48<1\.48<dense\_rerank1\.711\.71, andretrieveis by far the largest effect \(partialη2=0\.228\\eta^\{2\}=0\.228,p<10−3p<10^\{\-3\}\)\. The reranker’s edge over plaindenseis small but significant \(d=0\.31d=0\.31\), as the recall screen suggested it would be\. Thenegative controlfinalizeis null on every layer \(η2≈0\\eta^\{2\}\\approx 0,p=0\.68p=0\.68\), andgenerate\.modelbehaves as expected – larger models score higher \(η2=0\.052\\eta^\{2\}=0\.052,p<10−3p<10^\{\-3\}\)\. The best configuration isdense\_rerank\+\+gpt\-oss\-120b\(mean1\.781\.78\), withfinalizeimmaterial\. Full statistics \(FF, Cohen’sdd, and CLMM log\-odds\) are in Listing[2](https://arxiv.org/html/2607.10380#LST2)\.

#### Interactions\.

The factorial also flags a significantretrieve×\\timesgenerate\.modelinteraction \(p=0\.002p=0\.002\)\. The cell means show its shape: the model gap is widest with no retrieval \(none: gemma3\-4b0\.360\.36vs\. gpt\-oss\-120b1\.131\.13\) and narrows as retrieval improves \(dense\_rerank:1\.591\.59vs\.1\.781\.78\)\. Equivalently, retrieval helps the*weaker*model most – gemma3\-4b gains\+1\.23\+1\.23fromnonetodense\_rerank, gpt\-oss\-120b only\+0\.65\+0\.65– because with the evidence in context even a small model can read the answer off, whereas with none it falls back on weaker parametric knowledge \(the CLMM confirms the sign:none×\\timesgpt\-oss\-120b=\+1\.85=\+1\.85,p<10−3p<10^\{\-3\}\)\. A one\-factor\-at\-a\-time sweep cannot see this; surfacing it is exactly what a factorial design is for\.

#### E2: does CAFE invent effects?

E2 is the converse of E1: does CAFE avoid*inventing*effects that are not there? We build data in which, by construction,*no*factor has any effect, randomly reshuffling the900900verdicts onto different answers so that every link between a factor and quality is broken\. A calibrated test should then flag factors at the nominal Type I error rate,α=0\.05\\alpha=0\.05; over500500independent reshuffles CAFE flags them4\.9%4\.9\\%of the time – indicating no spurious significance\.

#### E3: does the judge track human judgement?

The whole analysis rests on the LLM judge’s scores, so the final check is whether they agree with people\. Two human experts independently re\-scored a stratified sample of6161answers, applying the same rubric*blind*to the judge’s verdicts\. Measured with Krippendorff’s ordinalα\\alpha, the two raters agree strongly with*each other*\(α=0\.91\\alpha=0\.91\), and the LLM\-judge agrees with them about as well –α=0\.95\\alpha=0\.95and0\.890\.89, yieldingα=0\.92\\alpha=0\.92across all three\. Every value lands in the conventional “reliable” band \(α≥0\.80\\alpha\\geq 0\.80\)\.

#### E4: does the winner survive new questions?

Finally, is the best configuration a real winner or an artifact of these particular5050questions? Using cross\-validation, we repeatedly split the questions into two random halves, recompute the configuration ranking on each half independently, and measure how well the two rankings agree\. Across splits the results correlate strongly \(Spearmanρ=0\.91\\rho=0\.91\)\. Disagreement is only found among a cluster of near\-tied top configurations, the very gaps CAFE had already reported as non\-significant\.

## 5Conclusion

CAFE brings design of experiments to the evaluation of compound AI systems, turning pipeline components into factors and attributing answer\-quality variance to them and their interactions with a mixed\-effects model matched to the rubric’s scale\. On a retrieval\-augmented HotpotQA study it recovers planted effects, calls a negative control null, stays calibrated under a permutation null, and agrees with expert human raters\. CAFE is open\-source, and aims to make designed, statistically grounded evaluation routine for the teams who build compound AI systems\.

## 6Limitations and Future Work

CAFE’s own limitations are statistical and computational\. Its scale\-agnostic effect\-size view \(partialη2\\eta^\{2\}, Cohen’sdd\) treats the ordinal scale as an interval and is only an approximate ranking aid; mixed\-effects fits can be unstable on small or sparse designs, as the near\-singular random effect in our study shows; and full\-factorial designs grow multiplicatively in cost\. Planned extensions address these: an order\-independent*deviance partition*that apportions explained deviance across factors\(Grömping,[2006](https://arxiv.org/html/2607.10380#bib.bib45)\), fractional\-factorial screening to cover large spaces at a fraction of the runs, and Bayesian mixed models\(Bürkner,[2017](https://arxiv.org/html/2607.10380#bib.bib46)\)for full posterior effect sizes and credible intervals\. Modelling*institutional*rater effects – how different peer groups score the same system – is a further direction\(Luettgauet al\.,[2025](https://arxiv.org/html/2607.10380#bib.bib53)\)\.

One caveat concerns our*evaluation*rather than CAFE itself: it validates the instrument on a relatively easy task – a three\-level \(0–22\) correctness judgment against a gold answer is not where LLM judges struggle, and a subjective rubric \(tone, style, helpfulness\) would likely show lower judge–human agreement\. Judging is where CAFE will also grow: multi\-criteria scoring \(several named scores per answer\) and, where a few human labels are available, prediction\-powered inference to*debias*the judge’s estimates against them\(Boyeauet al\.,[2025](https://arxiv.org/html/2607.10380#bib.bib50); Fischet al\.,[2024](https://arxiv.org/html/2607.10380#bib.bib51); Parket al\.,[2025](https://arxiv.org/html/2607.10380#bib.bib52)\)\.

## References

- Categorical data analysis\.2nd edition,John Wiley & Sons,New York\.External Links:ISBN 9780471458760,[Document](https://dx.doi.org/10.1002/0471249688),[Link](https://doi.org/10.1002/0471249688)Cited by:[§3\.4](https://arxiv.org/html/2607.10380#S3.SS4.p3.13)\.
- Argilla \(2024\)Argilla: the collaboration platform for AI engineers and domain experts\.Note:SoftwareExternal Links:[Link](https://argilla.io/)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px2.p1.1)\.
- B\. Atil, S\. Aykent, A\. Chittams, L\. Fu, R\. J\. Passonneau, E\. Radcliffe, G\. R\. Rajagopal, A\. Sloan, T\. Tudrej, F\. Ture, Z\. Wu, L\. Xu, and B\. Baldwin \(2024\)Non\-determinism of “deterministic” LLM settings\.arXiv preprint arXiv:2408\.04667\.External Links:[Link](https://arxiv.org/abs/2408.04667)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2607.10380#S3.SS2.p1.1)\.
- M\. Barker, A\. Bell, E\. Thomas, J\. Carr, T\. Andrews, and U\. Bhatt \(2025\)Faster, cheaper, better: multi\-objective hyperparameter optimization for LLM and RAG systems\.arXiv preprint arXiv:2502\.18635\.External Links:[Link](https://arxiv.org/abs/2502.18635)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px1.p1.1),[§3\.4](https://arxiv.org/html/2607.10380#S3.SS4.p5.1)\.
- D\. Bates, M\. Mächler, B\. Bolker, and S\. Walker \(2015\)Fitting linear mixed\-effects models using lme4\.Journal of Statistical Software67\(1\),pp\. 1–48\.External Links:[Document](https://dx.doi.org/10.18637/jss.v067.i01),[Link](https://doi.org/10.18637/jss.v067.i01)Cited by:[§3\.4](https://arxiv.org/html/2607.10380#S3.SS4.p3.13)\.
- B\. H\. Bjarnason, A\. Silva, and M\. Monperrus \(2026\)On randomness in agentic evals\.arXiv preprint arXiv:2602\.07150\.External Links:[Link](https://arxiv.org/abs/2602.07150)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px3.p1.1)\.
- P\. Boyeau, A\. N\. Angelopoulos, T\. Li, N\. Yosef, J\. Malik, and M\. I\. Jordan \(2025\)AutoEval done right: using synthetic data for model evaluation\.InProceedings of the 42nd International Conference on Machine Learning \(ICML\),PMLR, Vol\.267,pp\. 5276–5290\.External Links:[Link](https://proceedings.mlr.press/v267/boyeau25a.html)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2607.10380#S6.p2.2)\.
- P\. Bürkner \(2017\)brms: an R package for Bayesian multilevel models using Stan\.Journal of Statistical Software80\(1\),pp\. 1–28\.External Links:[Document](https://dx.doi.org/10.18637/jss.v080.i01),[Link](https://doi.org/10.18637/jss.v080.i01)Cited by:[§6](https://arxiv.org/html/2607.10380#S6.p1.2)\.
- J\. Chen, J\. Ye, and G\. Wang \(2025a\)From standalone LLMs to integrated intelligence: a survey of compound AI systems\.arXiv preprint arXiv:2506\.04565\.External Links:[Link](https://arxiv.org/abs/2506.04565)Cited by:[§1](https://arxiv.org/html/2607.10380#S1.p1.1)\.
- L\. Chen, J\. Q\. Davis, B\. Hanin, P\. Bailis, I\. Stoica, M\. Zaharia, and J\. Zou \(2024\)Are more LLM calls all you need? towards the scaling properties of compound AI systems\.InAdvances in Neural Information Processing Systems 37 \(NeurIPS\),External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/51173cf34c5faac9796a47dc2fdd3a71-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2607.10380#S1.p2.1)\.
- L\. Chen, J\. Q\. Davis, B\. Hanin, P\. Bailis, M\. Zaharia, J\. Zou, and I\. Stoica \(2025b\)Optimizing model selection for compound AI systems\.arXiv preprint arXiv:2502\.14815\.External Links:[Link](https://arxiv.org/abs/2502.14815)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Chowdhury and M\. D’Souza \(2026\)SETA: statistical fault attribution for compound AI systems\.InProceedings of the IEEE/ACM International Conference on AI Engineering \(CAIN\),External Links:[Link](https://arxiv.org/abs/2601.19337)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px3.p1.1)\.
- R\. H\. B\. Christensen \(2023\)Ordinal—regression models for ordinal data\.Note:R package version 2023\.12\-4External Links:[Link](https://cran.r-project.org/package=ordinal)Cited by:[§3\.4](https://arxiv.org/html/2607.10380#S3.SS4.p3.13)\.
- J\. Cohen \(1988\)Statistical power analysis for the behavioral sciences\.2nd edition,Lawrence Erlbaum Associates,Hillsdale, NJ\.External Links:[Document](https://dx.doi.org/10.4324/9780203771587),[Link](https://doi.org/10.4324/9780203771587)Cited by:[§3\.4](https://arxiv.org/html/2607.10380#S3.SS4.p4.5)\.
- Confident AI \(2024\)DeepEval: an open\-source LLM evaluation framework\.Note:SoftwareExternal Links:[Link](https://github.com/confident-ai/deepeval)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Dubois, H\. Coppock, M\. Giulianelli, T\. Flesch, L\. Luettgau, and C\. Ududec \(2025\)Skewed score: a statistical framework to assess autograders\.arXiv preprint arXiv:2507\.03772\.External Links:[Link](https://arxiv.org/abs/2507.03772)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Es, J\. James, L\. Espinosa\-Anke, and S\. Schockaert \(2024\)RAGAs: automated evaluation of retrieval augmented generation\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics \(EACL\): System Demonstrations,pp\. 150–158\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.eacl-demo.16),[Link](https://aclanthology.org/2024.eacl-demo.16/)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Fisch, J\. Maynez, R\. A\. Hofer, B\. Dhingra, A\. Globerson, and W\. W\. Cohen \(2024\)Stratified prediction\-powered inference for effective hybrid evaluation of language models\.InAdvances in Neural Information Processing Systems 37 \(NeurIPS\),External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/c9fcd02e6445c7dfbad6986abee53d0d-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px2.p1.1),[§6](https://arxiv.org/html/2607.10380#S6.p2.2)\.
- R\. A\. Fisher \(1935\)The design of experiments\.Oliver and Boyd,Edinburgh\.External Links:[Link](https://archive.org/details/in.ernet.dli.2015.502684)Cited by:[§1](https://arxiv.org/html/2607.10380#S1.p3.1)\.
- U\. Grömping \(2006\)Relative importance for linear regression in R: the package relaimpo\.Journal of Statistical Software17\(1\),pp\. 1–27\.External Links:[Document](https://dx.doi.org/10.18637/jss.v017.i01),[Link](https://doi.org/10.18637/jss.v017.i01)Cited by:[§6](https://arxiv.org/html/2607.10380#S6.p1.2)\.
- J\. Gu, X\. Jiang, Z\. Shi, H\. Tan, X\. Zhai, C\. Xu, W\. Li, Y\. Shen, S\. Ma, H\. Liu, S\. Wang, K\. Zhang, Y\. Wang, W\. Gao, L\. Ni, and J\. Guo \(2024\)A survey on LLM\-as\-a\-judge\.arXiv preprint arXiv:2411\.15594\.External Links:[Link](https://arxiv.org/abs/2411.15594)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px3.p1.1)\.
- L\. Guerdan, S\. Barocas, K\. Holstein, H\. Wallach, Z\. S\. Wu, and A\. Chouldechova \(2025\)Validating LLM\-as\-a\-judge systems under rating indeterminacy\.InAdvances in Neural Information Processing Systems 38 \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2503.05965)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Haase, J\. Gonnermann\-Müller, P\. H\. P\. Hanel, N\. Leins, T\. Kosch, J\. Mendling, and S\. Pokutta \(2026\)Within\-model vs between\-prompt variability in large language models for creative tasks\.arXiv preprint arXiv:2601\.21339\.External Links:[Link](https://arxiv.org/abs/2601.21339)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px3.p1.1)\.
- R\. Haldar and J\. Hockenmaier \(2025\)Rating roulette: self\-inconsistency in LLM\-as\-a\-judge frameworks\.InFindings of the Association for Computational Linguistics: EMNLP 2025,pp\. 24986–25004\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.1361),[Link](https://aclanthology.org/2025.findings-emnlp.1361/)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px3.p1.1),[§3\.3](https://arxiv.org/html/2607.10380#S3.SS3.p1.1)\.
- S\. Han, G\. Titericz Junior, T\. Balough, and W\. Zhou \(2025\)Judge’s verdict: a comprehensive analysis of LLM judge capability through human agreement\.arXiv preprint arXiv:2510\.09738\.External Links:[Link](https://arxiv.org/abs/2510.09738)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px3.p1.1)\.
- D\. M\. Howcroft and V\. Rieser \(2021\)What happens if you treat ordinal ratings as interval data? human evaluations in NLP are even more under\-powered than you think\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 8932–8939\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.703),[Link](https://aclanthology.org/2021.emnlp-main.703/)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. Amodei \(2020\)Scaling laws for neural language models\.arXiv preprint arXiv:2001\.08361\.External Links:[Link](https://arxiv.org/abs/2001.08361)Cited by:[3rd item](https://arxiv.org/html/2607.10380#S4.I1.i3.p1.2)\.
- M\. Y\. Kartal, S\. K\. Kose, K\. Sevinç, and B\. Aktas \(2025\)RAGSmith: a framework for finding the optimal composition of retrieval\-augmented generation methods across datasets\.arXiv preprint arXiv:2511\.01386\.External Links:[Link](https://arxiv.org/abs/2511.01386)Cited by:[§1](https://arxiv.org/html/2607.10380#S1.p1.1),[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px1.p1.1)\.
- O\. Khattab, A\. Singhvi, P\. Maheshwari, Z\. Zhang, K\. Santhanam, S\. Vardhamanan, S\. Haq, A\. Sharma, T\. T\. Joshi, H\. Moazam, H\. Miller, M\. Zaharia, and C\. Potts \(2024\)DSPy: compiling declarative language model calls into self\-improving pipelines\.InThe Twelfth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2310.03714)Cited by:[§1](https://arxiv.org/html/2607.10380#S1.p1.1),[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px1.p1.1)\.
- K\. Krippendorff \(2004\)Content analysis: an introduction to its methodology\.2nd edition,Sage,Thousand Oaks, CA\.External Links:ISBN 9780761915454,[Link](https://us.sagepub.com/en-us/nam/content-analysis/book258450)Cited by:[§3\.4](https://arxiv.org/html/2607.10380#S3.SS4.p5.1)\.
- LangChain \(2024\)LangSmith: a platform for LLM application evaluation and observability\.Note:SoftwareExternal Links:[Link](https://www.langchain.com/langsmith)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px2.p1.1)\.
- Ø\. Langsrud \(2003\)ANOVA for unbalanced data: use Type II instead of Type III sums of squares\.Statistics and Computing13\(2\),pp\. 163–167\.External Links:[Document](https://dx.doi.org/10.1023/A%3A1023260610025),[Link](https://doi.org/10.1023/A:1023260610025)Cited by:[§3\.4](https://arxiv.org/html/2607.10380#S3.SS4.p4.5)\.
- Y\. Lee, G\. Yi, M\. Liu, J\. Lu, G\. Yang, and Y\. Chen \(2025\)Compound AI systems optimization: a survey of methods, challenges, and future directions\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 28760–28775\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1463),[Link](https://aclanthology.org/2025.emnlp-main.1463/)Cited by:[§1](https://arxiv.org/html/2607.10380#S1.p1.1)\.
- P\. Liang, R\. Bommasani, T\. Lee, D\. Tsipras, D\. Soylu, M\. Yasunaga, Y\. Zhang, D\. Narayanan, Y\. Wu, A\. Kumar, B\. Newman, B\. Yuan, B\. Yan, C\. Zhang, C\. Cosgrove, C\. D\. Manning, C\. Ré, D\. Acosta\-Navas, D\. A\. Hudson, E\. Zelikman, E\. Durmus, F\. Ladhak, F\. Rong, H\. Ren, H\. Yao, J\. Wang, K\. Santhanam, L\. Orr, L\. Zheng, M\. Yuksekgonul, M\. Suzgun, N\. Kim, N\. Guha, N\. Chatterji, O\. Khattab, P\. Henderson, Q\. Huang, R\. Chi, S\. M\. Xie, S\. Santurkar, S\. Ganguli, T\. Hashimoto, T\. Icard, T\. Zhang, V\. Chaudhary, W\. Wang, X\. Li, Y\. Mai, Y\. Zhang, and Y\. Koreeda \(2023\)Holistic evaluation of language models\.Transactions on Machine Learning Research \(TMLR\)\.External Links:[Link](https://arxiv.org/abs/2211.09110)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. Zhu \(2023\)G\-Eval: NLG evaluation using GPT\-4 with better human alignment\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.153),[Link](https://aclanthology.org/2023.emnlp-main.153/)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px2.p1.1)\.
- L\. Luettgau, H\. Coppock, M\. Dubois, C\. Summerfield, and C\. Ududec \(2025\)HiBayES: a hierarchical bayesian modeling framework for AI evaluation statistics\.arXiv preprint arXiv:2505\.05602\.External Links:[Link](https://arxiv.org/abs/2505.05602)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2607.10380#S6.p1.2)\.
- F\. Lukassen, J\. Herrmann, C\. Weisser, B\. Saefken, and T\. Kneib \(2026\)From XAI to stories: a factorial study of LLM\-generated explanation quality\.arXiv preprint arXiv:2601\.02224\.External Links:[Link](https://arxiv.org/abs/2601.02224)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px3.p1.1)\.
- P\. McCullagh \(1980\)Regression models for ordinal data\.Journal of the Royal Statistical Society: Series B \(Methodological\)42\(2\),pp\. 109–142\.External Links:[Document](https://dx.doi.org/10.1111/j.2517-6161.1980.tb01109.x),[Link](https://doi.org/10.1111/j.2517-6161.1980.tb01109.x)Cited by:[§3\.4](https://arxiv.org/html/2607.10380#S3.SS4.p3.13)\.
- S\. Messing \(2026\)Hidden measurement error in LLM pipelines distorts annotation, evaluation, and benchmarking\.arXiv preprint arXiv:2604\.11581\.External Links:[Link](https://arxiv.org/abs/2604.11581)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px3.p1.1)\.
- D\. C\. Montgomery \(2017\)Design and analysis of experiments\.9th edition,John Wiley & Sons,Hoboken, NJ\.External Links:[Link](https://books.google.com/books/about/Design_and_Analysis_of_Experiments.html?id=xJBEDwAAQBAJ)Cited by:[§1](https://arxiv.org/html/2607.10380#S1.p3.1)\.
- Z\. Mustahsan, A\. Lim, M\. Anand, S\. Jain, and B\. McCann \(2025\)Stochasticity in agentic evaluations: quantifying inconsistency with intraclass correlation\.arXiv preprint arXiv:2512\.06710\.External Links:[Link](https://arxiv.org/abs/2512.06710)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Nakagawa and H\. Schielzeth \(2013\)A general and simple method for obtainingR2R^\{2\}from generalized linear mixed\-effects models\.Methods in Ecology and Evolution4\(2\),pp\. 133–142\.External Links:[Document](https://dx.doi.org/10.1111/j.2041-210X.2012.00261.x),[Link](https://doi.org/10.1111/j.2041-210X.2012.00261.x)Cited by:[§3\.4](https://arxiv.org/html/2607.10380#S3.SS4.p4.5)\.
- P\. Nilayam and S\. Nayak \(2026\)QUIVER: a formal framework for quantifying perturbation propagation and bifurcation in compound AI systems\.arXiv preprint arXiv:2605\.23956\.External Links:[Link](https://arxiv.org/abs/2605.23956)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Park, M\. Zecchin, and O\. Simeone \(2025\)Adaptive prediction\-powered AutoEval with reliability and efficiency guarantees\.InAdvances in Neural Information Processing Systems 38 \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2505.18659)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2607.10380#S6.p2.2)\.
- J\. C\. Pinheiro and D\. M\. Bates \(2000\)Mixed\-effects models in s and s\-plus\.Statistics and Computing,Springer,New York\.External Links:[Document](https://dx.doi.org/10.1007/b98882),[Link](https://link.springer.com/book/10.1007/b98882)Cited by:[§1](https://arxiv.org/html/2607.10380#S1.p3.1)\.
- A\. Roberts, C\. Raffel, and N\. Shazeer \(2020\)How much knowledge can you pack into the parameters of a language model?\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),External Links:[Link](https://arxiv.org/abs/2002.08910)Cited by:[3rd item](https://arxiv.org/html/2607.10380#S4.I1.i3.p1.2)\.
- J\. Saad\-Falcon, O\. Khattab, C\. Potts, and M\. Zaharia \(2024\)ARES: an automated evaluation framework for retrieval\-augmented generation systems\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(NAACL\),pp\. 338–354\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.20),[Link](https://aclanthology.org/2024.naacl-long.20/)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Saltelli, M\. Ratto, T\. Andres, F\. Campolongo, J\. Cariboni, D\. Gatelli, M\. Saisana, and S\. Tarantola \(2008\)Global sensitivity analysis: the primer\.John Wiley & Sons\.External Links:[Document](https://dx.doi.org/10.1002/9780470725184),[Link](https://onlinelibrary.wiley.com/doi/book/10.1002/9780470725184)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px3.p1.1),[§3\.4](https://arxiv.org/html/2607.10380#S3.SS4.p4.5)\.
- W\. Sun, L\. Yan, X\. Ma, S\. Wang, P\. Ren, Z\. Chen, D\. Yin, and Z\. Ren \(2023\)Is ChatGPT good at search? investigating large language models as re\-ranking agents\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 14918–14937\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.923),[Link](https://aclanthology.org/2023.emnlp-main.923/)Cited by:[1st item](https://arxiv.org/html/2607.10380#S4.I1.i1.p1.2)\.
- B\. V\. Syiem and E\. Velloso \(2026\)Better assumptions, stronger conclusions: the case for ordinal regression in HCI\.InProceedings of the 2026 CHI Conference on Human Factors in Computing Systems \(CHI\),External Links:[Document](https://dx.doi.org/10.1145/3772318.3790821),[Link](https://dl.acm.org/doi/10.1145/3772318.3790821)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px3.p1.1)\.
- J\. E\. Taylor, G\. A\. Rousselet, C\. Scheepers, and S\. C\. Sereno \(2023\)Rating norms should be calculated from cumulative link mixed effects models\.Behavior Research Methods55\(5\),pp\. 2175–2196\.External Links:[Document](https://dx.doi.org/10.3758/s13428-022-01814-7),[Link](https://link.springer.com/article/10.3758/s13428-022-01814-7)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px3.p1.1),[§3\.4](https://arxiv.org/html/2607.10380#S3.SS4.p3.13)\.
- TruEra \(2024\)TruLens: evaluation and tracing for LLM applications\.Note:SoftwareExternal Links:[Link](https://www.trulens.org/)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px2.p1.1)\.
- UK AI Security Institute \(2024\)Inspect AI: framework for large language model evaluations\.Note:SoftwareExternal Links:[Link](https://inspect.aisi.org.uk/)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px2.p1.1)\.
- A\. R\. Vazquez, K\. M\. Rother, and M\. V\. Charles\-Gonzalez \(2025\)A systematic assessment of large language models for constructing two\-level fractional factorial designs\.arXiv preprint arXiv:2512\.17113\.Note:Published in Quality Engineering \(2026\), DOI: 10\.1080/08982112\.2026\.2625951External Links:[Link](https://arxiv.org/abs/2512.17113)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px3.p1.1)\.
- B\. Wang, D\. Zhao, N\. R\. Tallent, and L\. Guo \(2025\)On the reproducibility limitations of RAG systems\.arXiv preprint arXiv:2509\.18869\.External Links:[Link](https://arxiv.org/abs/2509.18869)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Wu, P\. Sarthi, S\. Zhao, A\. Lee, H\. Shandilya, A\. Mladenic Grobelnik, N\. Choudhary, E\. Huang, K\. Subbian, L\. Zhang, D\. Yang, J\. Zou, and J\. Leskovec \(2026\)Optimas: optimizing compound AI systems with globally aligned local rewards\.InThe Fourteenth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2507.03041)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning \(2018\)HotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 2369–2380\.External Links:[Document](https://dx.doi.org/10.18653/v1/D18-1259),[Link](https://aclanthology.org/D18-1259/)Cited by:[§4](https://arxiv.org/html/2607.10380#S4.SS0.SSS0.Px1.p1.1)\.
- M\. Yuksekgonul, F\. Bianchi, J\. Boen, S\. Liu, P\. Lu, Z\. Huang, C\. Guestrin, and J\. Zou \(2025\)Optimizing generative AI by backpropagating language model feedback\.Nature639,pp\. 609–616\.External Links:[Document](https://dx.doi.org/10.1038/s41586-025-08661-4),[Link](https://doi.org/10.1038/s41586-025-08661-4)Cited by:[§2](https://arxiv.org/html/2607.10380#S2.SS0.SSS0.Px1.p1.1)\.
- M\. Zaharia, O\. Khattab, L\. Chen, J\. Q\. Davis, H\. Miller, C\. Potts, J\. Zou, M\. Carbin, J\. Frankle, N\. Rao, and A\. Ghodsi \(2024\)The shift from models to compound AI systems\.Note:Berkeley Artificial Intelligence Research \(BAIR\) BlogExternal Links:[Link](https://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/)Cited by:[§1](https://arxiv.org/html/2607.10380#S1.p1.1)\.

## Appendix ASupplemental Material

Figure 2:The CAFE web application \(Results view\) on the HotpotQA recovery study of Section[4](https://arxiv.org/html/2607.10380#S4): the factorsretrieve×\\timesgenerate\.model×\\timesthefinalizeplacebo \(18 configurations, 50 questions, a 0–2 correctness rubric\)\. It reports overall quality, the best configuration, per\-factor attribution \(FF,pp, partialη2\\eta^\{2\}, and the share of variance explained\), and per\-level marginal means, all computed bycafe\-coreand served over the API\. Listing[2](https://arxiv.org/html/2607.10380#LST2)is the same analysis as console output\.![Refer to caption](https://arxiv.org/html/2607.10380v1/webui_results.png)Listing 2:Complete console output ofresult\.report\(\)on the study of Figure[2](https://arxiv.org/html/2607.10380#A1.F2): descriptive means \(overall, ranked per configuration, and per factor\) with the best configuration; the inferential linear mixed model \(Type\-IIFF, partialη2\\eta^\{2\}, and pairwise Cohen’sdd\) with its near\-singular random\-effect note; and the ordinal cumulative\-link mixed model\. Retrieval dominates and is correctly ordered \(none<<dense<<dense\_rerank\), the base model helps, and the placebofinalizeis null: the recovery result of Section[4](https://arxiv.org/html/2607.10380#S4)\.900answers\|18configs\|50inputs\|900ratings

best:finalize=on\|generate\.model=gpt\-oss\-120b\|retrieve=dense\_rerank

pipeline:900answers\-\>900judged\(1unparseable\)\-\>899usableverdicts

============================================================

DESCRIPTIVE\-\-means&bestconfiguration

============================================================

verdicts:900\(899usable\)factors:retrieve,generate\.model,finalize

overallmeanquality:1\.31\(n=899\)

per\-configurationmeanquality:

1\.78\(n=50\)finalize=on\|generate\.model=gpt\-oss\-120b\|retrieve=dense\_rerank

1\.78\(n=49\)finalize=off\|generate\.model=gpt\-oss\-120b\|retrieve=dense\_rerank

1\.76\(n=50\)finalize=on\|generate\.model=gpt\-oss\-20b\|retrieve=dense\_rerank

1\.74\(n=50\)finalize=off\|generate\.model=gpt\-oss\-20b\|retrieve=dense\_rerank

1\.62\(n=50\)finalize=off\|generate\.model=gpt\-oss\-120b\|retrieve=dense

1\.60\(n=50\)finalize=on\|generate\.model=gpt\-oss\-120b\|retrieve=dense

1\.60\(n=50\)finalize=on\|generate\.model=gemma3\-4b\|retrieve=dense\_rerank

1\.58\(n=50\)finalize=on\|generate\.model=gpt\-oss\-20b\|retrieve=dense

1\.58\(n=50\)finalize=off\|generate\.model=gemma3\-4b\|retrieve=dense\_rerank

1\.50\(n=50\)finalize=off\|generate\.model=gpt\-oss\-20b\|retrieve=dense

1\.32\(n=50\)finalize=off\|generate\.model=gemma3\-4b\|retrieve=dense

1\.28\(n=50\)finalize=on\|generate\.model=gemma3\-4b\|retrieve=dense

1\.16\(n=50\)finalize=on\|generate\.model=gpt\-oss\-120b\|retrieve=none

1\.10\(n=50\)finalize=off\|generate\.model=gpt\-oss\-120b\|retrieve=none

0\.80\(n=50\)finalize=on\|generate\.model=gpt\-oss\-20b\|retrieve=none

0\.74\(n=50\)finalize=off\|generate\.model=gpt\-oss\-20b\|retrieve=none

0\.36\(n=50\)finalize=off\|generate\.model=gemma3\-4b\|retrieve=none

0\.36\(n=50\)finalize=on\|generate\.model=gemma3\-4b\|retrieve=none

per\-factormarginalmeans:

retrieve:none0\.75\(n=300\)dense1\.48\(n=300\)dense\_rerank1\.71\(n=299\)

generate\.model:gemma3\-4b1\.08\(n=300\)gpt\-oss\-20b1\.35\(n=300\)gpt\-oss\-120b1\.51\(n=299\)

finalize:off1\.30\(n=449\)on1\.32\(n=450\)

bestconfiguration:finalize=on\|generate\.model=gpt\-oss\-120b\|retrieve=dense\_rerank\(mean1\.78\)

============================================================

INFERENTIAL\-\-mixed\-effectsmodel\(significance&effectsizes\)

============================================================

verdict~\(1\|input\_id\)\+\(retrieve\+generate\.model\+finalize\)^2

fixed\-effectsmodel\+Type\-IIANOVA,upto2\-way\(n=899,alpha=0\.05\)

per\-termeffects\(F\-test,p,partialeta^2;’x’=interaction\):

termFppartialeta^2

retrieve130\.370\.00000\.228\*\*\*

generate\.model24\.060\.00000\.052\*\*\*

retrievexgenerate\.model4\.170\.00240\.019\*\*

generate\.modelxfinalize0\.120\.88580\.000

finalize0\.170\.68410\.000

retrievexfinalize0\.040\.96120\.000

Signif\.codes:0’\*\*\*’0\.001’\*\*’0\.01’\*’0\.05’\.’0\.1’’1

effectsizes\-\-Cohen’sd\(magnitudeofthegap;0\.2small,0\.5medium,0\.8large\):

retrieve:densevsdense\_rerankd=\-0\.3195%CI\[\-0\.47,\-0\.15\]

retrieve:densevsnoned=\+0\.8695%CI\[\+0\.69,\+1\.03\]

retrieve:dense\_rerankvsnoned=\+1\.2495%CI\[\+1\.07,\+1\.42\]

generate\.model:gemma3\-4bvsgpt\-oss\-120bd=\-0\.4995%CI\[\-0\.65,\-0\.33\]

generate\.model:gemma3\-4bvsgpt\-oss\-20bd=\-0\.3095%CI\[\-0\.46,\-0\.14\]

generate\.model:gpt\-oss\-120bvsgpt\-oss\-20bd=\+0\.1895%CI\[\+0\.02,\+0\.34\]

finalize:offvsond=\-0\.0295%CI\[\-0\.16,\+0\.11\]

note:randominterceptnotestimable;usingfixed\-effectsANOVA

note:modelwasnear\-singular\(littlebetween\-questionvariance\)\-\-treatp\-valuesas

unstable;theeffectsizes\(Cohen’sd\)arethemorereliablesignalhere

============================================================

ORDINAL\-\-cumulative\-linkmixedmodel\(verdictsasorderedcategories\)

============================================================

ordinalCLMM\-\-verdict~\(1\|input\_id\)\+\(retrieve\+generate\.model\+finalize\)^2

\(n=899,logLik=\-557\.7,alpha=0\.05\)

fixedeffects\(ordinallog\-oddsofahigherscore;\+=better\):

termestimatep

retrieve=dense\_rerank\+1\.1490\.0075\*\*

retrieve=none\-3\.9440\.0000\*\*\*

generate\.model=gpt\-oss\-120b\+1\.2500\.0035\*\*

generate\.model=gpt\-oss\-20b\+0\.8000\.0541\.

finalize=on\-0\.0700\.8524

retrieve=dense\_rerankxgenerate\.model=gpt\-oss\-120b\-0\.1460\.7947

retrieve=nonexgenerate\.model=gpt\-oss\-120b\+1\.8540\.0006\*\*\*

retrieve=dense\_rerankxgenerate\.model=gpt\-oss\-20b\-0\.0410\.9390

retrieve=nonexgenerate\.model=gpt\-oss\-20b\+0\.8800\.0936\.

retrieve=dense\_rerankxfinalize=on\+0\.0330\.9418

retrieve=nonexfinalize=on\+0\.0890\.8313

generate\.model=gpt\-oss\-120bxfinalize=on\+0\.1600\.7166

generate\.model=gpt\-oss\-20bxfinalize=on\+0\.3020\.4852

Signif\.codes:0’\*\*\*’0\.001’\*\*’0\.01’\*’0\.05’\.’0\.1’’1