Pooled Leaderboards Hide System-Specific Winners: A Reporting-Protocol Audit of Offline Root-Cause Analysis Benchmarks

# A Reporting-Protocol Audit of Offline Root-Cause Analysis Benchmarks
Source: [https://arxiv.org/html/2606.29159](https://arxiv.org/html/2606.29159)
## Pooled Leaderboards Hide System\-Specific Winners: A Reporting\-Protocol Audit of Offline Root\-Cause Analysis Benchmarks

Lining Hu Ting Liu Yuzhuo Fu School of Electronic, Information and Electrical Engineering Shanghai Jiao Tong University Shanghai, China \{ring\.hu,louisa\_liu,yzfu\}@sjtu\.edu\.cn

###### Abstract

Offline root\-cause\-analysis \(RCA\) benchmarks commonly rank methods by a single pooled top\-1 accuracy across multiple subsystems, and engineers often read the pooled winner as a recommendation for their own subsystem\. We audit that reading on three public RCA benchmark families—OpenRCA, RCAEval, and PetShop—covering 11 subsystems and 778 matched scoring units\. To keep pairwise comparisons on identical cases, the main analysis retains four methods or comparators with complete coverage: BARO, a CD\-1min adapter, max\-\|Z\|\|Z\|, and per\-service alert\-count\. All six pairwise comparisons show subsystem\-level effects of both signs, every random\-effects 95% prediction interval crosses zero, and case\-level interaction tests reject exchangeability in 5 of 6 pairs\. Leave\-one\-system\-out selection picks the lower\-scoring method on up to 5 of 11 held\-out subsystems, with regret reaching 24\.8 pp on RCAEval / Sock\-Shop\. We release a 320\-line audit module; given a matched RCA benchmark score table, it recomputes the same per\-subsystem stability checks alongside pooled scores\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2606.29159v1/x1.png)Figure 1:A pooled leaderboard can hide subsystem\-level rankings\.Left: pooled top\-1 accuracy across all 11 audited subsystems \(778 cases\) for two BARO\-centered comparisons\. Right: the same comparisons decomposed by subsystem, showing the paired effectΔs\\Delta\_\{s\}\(in pp acc@1, oriented as BARO minus comparator\)\. Both comparisons reverse sign across subsystems: BARO is the pooled loser against max\-\|Z\|\|Z\|but scores higher on Bank, High\-Traffic, and Online\-Boutique; BARO is the pooled winner against alert\-count but scores lower on Market\-1, Telecom, Sock\-Shop, and Train\-Ticket\. The two reversal sets are disjoint, so the per\-subsystem disagreement is not explained by a fixed group of difficult systems\.When an offline RCA benchmark reports that methodAAoutperforms methodBB, what unit does that claim apply to? Most public root\-cause\-analysis \(RCA\) benchmarks answer with a pooled top\-1 accuracy: all cases from several subsystems are averaged into one number, and methods are ranked by it\. Yet the same number is often used as a recommendation: the pooled winner becomes the default choice for a specific subsystem in the same benchmark family\. That use creates an estimand mismatch\. RCA method choice happens at the subsystem level: an engineer wants to know whether a method works for their service graph, telemetry schema, and fault distribution\. The pooled score answers a different question: how does the method do on average across the suite? These two questions give the same answer only when method differences are stable across subsystems\. When method differences vary by subsystem, the pooled score is still a correct suite average—but it no longer tells an engineer which method to pick for one subsystem\.

We audit this mismatch on three public offline RCA benchmark families—OpenRCA\(Xu et al\.,[2025](https://arxiv.org/html/2606.29159#bib.bib27)\),RCAEval\(Pham et al\.,[2025](https://arxiv.org/html/2606.29159#bib.bib19)\), andPetShop\(Hardt et al\.,[2024](https://arxiv.org/html/2606.29159#bib.bib8)\)—covering 11 subsystems and 778 matched scoring units\. We apply a full\-coverage inclusion criterion: a method enters the comparison set only if it produces a score on every one of the 778 cases\. This prevents pairwise conclusions from being driven by different missing\-case patterns across methods\. Four methods or comparators meet this bar: the published RCA method BARO\(Pham et al\.,[2024a](https://arxiv.org/html/2606.29159#bib.bib17)\); a deterministic CD\-1min cross\-benchmark adapter built on ourOpenRCApipeline; and two simple cross\-benchmark comparators \(max\-\|Z\|\|Z\|and per\-service alert\-count\) that we define in[Section˜4\.1](https://arxiv.org/html/2606.29159#S4.SS1)\. CIRCA\(Li et al\.,[2022](https://arxiv.org/html/2606.29159#bib.bib13)\)and an internal causal\-guidedOpenRCAadapter do not meet this bar; we keep them as appendix diagnostics\.

[Figure˜1](https://arxiv.org/html/2606.29159#S1.F1)shows the problem on two BARO\-centered comparisons\. Against max\-\|Z\|\|Z\|, pooled leave\-one\-system\-out selection picks the lower\-scoring method on 3 of 11 held\-out subsystems\. Against alert\-count, it picks the lower\-scoring method on a different 4 of 11 held\-out subsystems\. The two reversal sets are disjoint\. If a fixed group of difficult subsystems caused the reversals, the same subsystem names would recur\. Instead, the failing recommendation changes with the method pair, and pooling smooths that dependence into a single number\.

The broader comparison set is no more stable\. All six method pairs reverse sign across subsystems, and every random\-effects 95% prediction interval crosses zero\. Leave\-one\-system\-out selection gives the corresponding selection cost, with regret reaching 24\.8 pp onRCAEval/Sock\-Shop\. Case\-level interaction tests reject exchangeability in 5 of 6 pairs\.

This paper contributes a small reporting audit for offline RCA benchmark releases\. Starting from matched per\-case scores, it reports paired subsystem effects, random\-effects heterogeneity, prediction intervals, and leave\-one\-system\-out selection regret\. The released 320\-line implementation applies the audit to arbitrary matched score tables; here we use it on three public benchmark releases\.

The rest of the paper is organized around that audit:[Sections˜2](https://arxiv.org/html/2606.29159#S2)and[3](https://arxiv.org/html/2606.29159#S3)define the context and scope,[Section˜4](https://arxiv.org/html/2606.29159#S4)gives the protocol,[Section˜5](https://arxiv.org/html/2606.29159#S5)reports the evidence, and[Section˜6](https://arxiv.org/html/2606.29159#S6)states the limits and recommendations\.

## 2Related Work

#### Benchmark validity and leaderboard instability in ML\.

Prior evaluation work in ML documents how pooled scoring protocols hide condition\-specific behavior: surface\-heuristic matching of NLU benchmark scores\(Bowman and Dahl,[2021](https://arxiv.org/html/2606.29159#bib.bib2)\), task\-conditional structure invisible to pooled accuracy on multi\-task suites\(Srivastava et al\.,[2023](https://arxiv.org/html/2606.29159#bib.bib24); Ribeiro et al\.,[2020](https://arxiv.org/html/2606.29159#bib.bib20)\), benchmark choice itself flipping method rankings\(Dehghani et al\.,[2021](https://arxiv.org/html/2606.29159#bib.bib5)\), and broader critiques of single\-number leaderboards\(Liang et al\.,[2023](https://arxiv.org/html/2606.29159#bib.bib14); Bender and Koller,[2020](https://arxiv.org/html/2606.29159#bib.bib1)\)\. Our audit applies the same perspective to offline RCA benchmarks, where the conditioning units are microservice subsystems\.

#### Root\-cause\-analysis methods on offline benchmarks\.

baro\(Pham et al\.,[2024a](https://arxiv.org/html/2606.29159#bib.bib17)\)is a statistical RCA method based on multivariate Bayesian online change\-point detection on KPI metrics;circa\(Li et al\.,[2022](https://arxiv.org/html/2606.29159#bib.bib13)\)is a causal\-graph\-based RCA method\.RCAEval\(Pham et al\.,[2025](https://arxiv.org/html/2606.29159#bib.bib19)\)packages eight end\-to\-end methods \(MicroCause,DyCause,RCD,CausIL,ϵ\\epsilon\-Diagnosis, and others\) under a unified evaluation API;OpenRCA\(Xu et al\.,[2025](https://arxiv.org/html/2606.29159#bib.bib27)\)provides an evaluation suite for LLM\-agent RCA systems with partial\-credit accuracy;PetShop\(Hardt et al\.,[2024](https://arxiv.org/html/2606.29159#bib.bib8)\)releases four traffic\-pattern microservice scenarios\. These releases make offline RCA evaluation increasingly standardized, but their headline reporting is still typically pooled across systems\. Our audit targets that reporting layer rather than the algorithms themselves\. Closed\-loop agentic benchmarks\(Chen et al\.,[2025](https://arxiv.org/html/2606.29159#bib.bib3); Jha et al\.,[2025](https://arxiv.org/html/2606.29159#bib.bib11); Wang et al\.,[2026](https://arxiv.org/html/2606.29159#bib.bib26); Zheng et al\.,[2024](https://arxiv.org/html/2606.29159#bib.bib28)\)sit outside our scope;[Section˜3](https://arxiv.org/html/2606.29159#S3)specifies the boundary\.

#### Differentiation from recent benchmark\-difficulty audits\.

Recent independent work byFang et al\. \([2026](https://arxiv.org/html/2606.29159#bib.bib7)\)audits RCA benchmark difficulty with SimpleRCA, a rule\-based multimodal heuristic that approaches reported state\-of\-the\-art scores on several benchmarks and motivates a harder Train\-Ticket release\. Our audit asks a different question: holding benchmark difficulty fixed, does pooled reporting across subsystems support subsystem\-level method selection? The two remedies are independent—harder benchmarks in their case, per\-subsystem reporting with heterogeneity diagnostics in ours\. Adjacent critiques examine agent failure modes\(Kim et al\.,[2026](https://arxiv.org/html/2606.29159#bib.bib12); Riddell et al\.,[2026](https://arxiv.org/html/2606.29159#bib.bib21)\)or general evaluation methodology\(Pham et al\.,[2024b](https://arxiv.org/html/2606.29159#bib.bib18)\)\.

#### Random\-effects meta\-analysis methodology\.

We use standard methods: Cochran’sQQtest of homogeneity\(Cochran,[1954](https://arxiv.org/html/2606.29159#bib.bib4)\), the DerSimonian\-Laird random\-effects estimator\(DerSimonian and Laird,[1986](https://arxiv.org/html/2606.29159#bib.bib6)\), the Paule\-Mandel iteratedτ2\\tau^\{2\}estimator\(Paule and Mandel,[1982](https://arxiv.org/html/2606.29159#bib.bib16)\), the Hartung\-Knapp and Sidik\-Jonkman small\-sample CI corrections\(Hartung and Knapp,[2001](https://arxiv.org/html/2606.29159#bib.bib9); Sidik and Jonkman,[2005](https://arxiv.org/html/2606.29159#bib.bib22),[2006](https://arxiv.org/html/2606.29159#bib.bib23)\), and the IntHout\-Higgins\-Rothstein95%95\\%prediction interval\(IntHout et al\.,[2016](https://arxiv.org/html/2606.29159#bib.bib10)\); case\-level analyses use the Papke\-Wooldridge fractional logit\(Papke and Wooldridge,[1996](https://arxiv.org/html/2606.29159#bib.bib15)\)to handleOpenRCA’s partial\-credit scoring under a binomial GLM\. The Paule\-Mandel \+ HKSJ \+ IntHout combination is designed for conservative behavior atk<25k<25, which matches ourk=11k=11\(Viechtbauer,[2010](https://arxiv.org/html/2606.29159#bib.bib25)\)\. We use these tools as reporting diagnostics for offline RCA benchmark validity\.

## 3Benchmark Landscape and Audit Scope

Public RCA benchmarks vary substantially in evaluation harnesses, fault\-injection protocols, and scoring conventions\. Our audit targets the subset for which subsystem\-level paired effects are identifiable\. The protocol requires*matched per\-case scoring across methods*: the same fault case, within the same subsystem, and against the same ground\-truth root cause must yield a comparable score under every audited method, so that per\-system paired effectsΔs\\Delta\_\{s\}are well\-defined \([Section˜4](https://arxiv.org/html/2606.29159#S4)\)\. This requirement partitions the public RCA benchmark landscape into four evaluation modes, only one of which satisfies the requirements of this audit:

1. \(a\)Standardized offline benchmarks with matched\-case scoring across methods:OpenRCA,RCAEval,PetShop\.
2. \(b\)Ad\-hoc single\-system testbedswidely re\-used as benchmarks but evaluated outside any standardized harness:Sock\-Shop,Online\-Boutique,Train\-Ticket\(Zhou et al\.,[2019](https://arxiv.org/html/2606.29159#bib.bib29)\)\.
3. \(c\)The AIOps\-Challenge familywith year\-specific protocols:CCF\-AIOps\{2020,2021,2022,2025\}\\\{2020,2021,2022,2025\\\}\.
4. \(d\)Closed\-loop agentic benchmarkswhere the system\-under\-test repeatedly interacts with a simulated outage:AIOpsLab\(Chen et al\.,[2025](https://arxiv.org/html/2606.29159#bib.bib3)\),ITBench\(Jha et al\.,[2025](https://arxiv.org/html/2606.29159#bib.bib11)\),Cloud\-OpsBench\(Wang et al\.,[2026](https://arxiv.org/html/2606.29159#bib.bib26)\),LEMMA\-RCA\(Zheng et al\.,[2024](https://arxiv.org/html/2606.29159#bib.bib28)\)\.

Our audit covers mode \(a\) only, because the paired\-effect estimator requires matched per\-case scoring across methods\. Mode \(b\) lacks a standardized harness, so per\-system effect estimates are not directly comparable across releases; modes \(c\) and \(d\) do not provide matched per\-case scoring across methods\. Accordingly, claims in this paper apply only to mode \(a\) benchmarks at their current public releases; closed\-loop agentic settings, ad\-hoc deployments, and RCA benchmarks without matched method outputs are outside scope\.

The 11 audited subsystems compriseOpenRCA×4\\times 4\(Bank,Market\-1,Market\-2,Telecom\),RCAEval×3\\times 3\(Online\-Boutique,Sock\-Shop,Train\-Ticket\), andPetShop×4\\times 4\(High\-Traffic,Low\-Traffic,Temporal\-1,Temporal\-2\), totaling778778matched scoring units across methods\.

### 4\.1Methods and comparators

The audit compares BARO, CD\-1min, max\-\|Z\|\|Z\|, and alert\-count\. BARO is a published RCA method, and CD\-1min is a deterministic adapter built on our OpenRCA pipeline\. The other two are schema\-agnostic heuristics used to stress\-test the reporting protocol\. They are not proposed as new RCA algorithms; they ask whether a pooled leaderboard remains stable when a published method is compared with simple, untuned per\-case rules\. The four method definitions follow\.

BARO\.BARO\(Pham et al\.,[2024a](https://arxiv.org/html/2606.29159#bib.bib17)\)is a multivariate Bayesian online change\-point method for KPI metrics\. We use the BARO authors’ reference implementation as packaged with RCAEval\(Pham et al\.,[2025](https://arxiv.org/html/2606.29159#bib.bib19)\)\.

CD\-1min adapter\.CD\-1min is a deterministic per\-case predictor we built on top of our OpenRCA pipeline\. It runs on telemetry resampled to 1\-minute resolution: for each fault case, it z\-scores every metric over the case window, keeps metrics whose post\-injection percentage change and\|Z\|\|Z\|both exceed fixed thresholds, and ranks services by the largest such percentage change among their metrics\. The cross\-benchmark version applies the same rule to RCAEval and PetShop without per\-benchmark tuning\. CD\-1min enters the comparison set as a full\-coverage adapter; threshold and label\-sensitivity checks are reported in[Appendix˜J](https://arxiv.org/html/2606.29159#A10)\.

Cross\-benchmark max\-\|Z\|\|Z\|\.For each fault case on subsystemss, the predictor computes per\-metric z\-scoresZs,j,tZ\_\{s,j,t\}using the pre\-injection interval\[0,t0\)\[0,t\_\{0\}\)as baseline, then selects the metric achievingarg⁡maxj⁡maxt∈\[t0,t1\]⁡\|Zs,j,t\|\\arg\\max\_\{j\}\\max\_\{t\\in\[t\_\{0\},t\_\{1\}\]\}\|Z\_\{s,j,t\}\|over the post\-injection window\[t0,t1\]\[t\_\{0\},t\_\{1\}\], and maps that metric to a service identifier\. There is no tuning, no training or validation split, and no per\-benchmark hyperparameter; the metric\-to\-service mapping is a shared preprocessing step used by all audited methods\.

Per\-service alert\-count\.For each fault case on subsystemss, the predictor flags every metricjjwhose post\-injection\|Z\|\|Z\|exceeds 3, that isas,j=𝟏​\{maxt∈\[t0,t1\]⁡\|Zs,j,t\|\>3\}a\_\{s,j\}=\\mathbf\{1\}\\\{\\max\_\{t\\in\[t\_\{0\},t\_\{1\}\]\}\|Z\_\{s,j,t\}\|\>3\\\}\. The per\-service alert count isAS=∑j∈Sas,jA\_\{S\}=\\sum\_\{j\\in S\}a\_\{s,j\}, and the predicted service isarg⁡maxS⁡AS\\arg\\max\_\{S\}A\_\{S\}, with ties broken alphabetically by service name\. The\|Z\|\>3\|Z\|\>3threshold is the conventional anomaly\-detection default rather than a tuned value; threshold sensitivity is reported in Section[L](https://arxiv.org/html/2606.29159#A12)\.

#### Audit unit and scoring\.

The audit uses matched scoring units: the same fault case, subsystem, and ground\-truth root cause scored for each method under comparison\. The primary full\-coverage analysis contains1111subsystems and778778matched scoring units\.OpenRCAuses fractional partial\-credit scoring,acc​@​1∈\{0,14,13,12,23,34,1\}\\mathrm\{acc@\}1\\in\\\{0,\\frac\{1\}\{4\},\\frac\{1\}\{3\},\\frac\{1\}\{2\},\\frac\{2\}\{3\},\\frac\{3\}\{4\},1\\\};RCAEvalandPetShopuse strict\{0,1\}\\\{0,1\\\}scoring\. We retain these native scores rather than binarizingOpenRCA, so the audit respects each benchmark’s released scoring convention; the case\-level interaction tests use a binomial\-GLM fractional logit\(Papke and Wooldridge,[1996](https://arxiv.org/html/2606.29159#bib.bib15)\)\.

#### Full\-coverage method inclusion criterion\.

The main question is whether a pooled leaderboard provides enough evidence for subsystem\-level method selection within the audited benchmark set\. That question requires every pair to be evaluated on identical cases\. We therefore include a method or comparator only if it provides per\-case outputs for all1111subsystems and all778778matched scoring units\. This rule is applied before any primary pairwise result is read, so missing coverage cannot be mistaken for subsystem\-level performance\.[Table˜1](https://arxiv.org/html/2606.29159#S4.T1)reports the resulting main audit comparison set\. The four main methods or comparators arebaro\(Pham et al\.,[2024a](https://arxiv.org/html/2606.29159#bib.bib17)\), cross\-benchmark max\-\|Z\|\|Z\|, per\-servicealert\-count, and CD\-1min\.circa\(Li et al\.,[2022](https://arxiv.org/html/2606.29159#bib.bib13)\)and an internal causal\-guidedOpenRCAadapter are retained as appendix diagnostics because their cross\-benchmark runs do not satisfy the same inclusion criterion\. The full inclusion ledger is in[Appendix˜D](https://arxiv.org/html/2606.29159#A4)\.

Table 1:Full\-coverage inclusion criterion for the main audit comparison set\.Main\-claim evidence uses only methods or comparators with matched outputs on all 11 audited subsystems and all 778 scoring units\. CIRCA and an internal causal\-guided adapter remain useful diagnostics, but they do not satisfy the same inclusion criterion and are therefore excluded from the primary full\-coverage pairwise analysis\.Method / comparator11/11 subsystems778 matched scoring unitsMain\-audit rolebaroYesYesPublished statistical RCA baselinemax\-\|Z\|\|Z\|YesYesCross\-benchmark comparatoralert\-countYesYesCross\-benchmark comparatorCD\-1minYesYesFull\-coverage cross\-benchmark adaptercircaNoNoDiagnostic only; incomplete matched coverage due to timeout / modal\-output collapse in non\-native settingscausal\-guided adapterNoNoInternalOpenRCAdiagnostic only; limited tests outsideOpenRCA

#### Diagnostics and estimands\.

The diagnostics expose information that pooled accuracy hides\. The pooled score identifies the benchmark\-level average winner, while the per\-subsystem paired effectΔs=ns−1​∑i\(scores,iA−scores,iB\)\\Delta\_\{s\}=n\_\{s\}^\{\-1\}\\sum\_\{i\}\(\\text\{score\}^\{A\}\_\{s,i\}\-\\text\{score\}^\{B\}\_\{s,i\}\)gives the local contrast on subsystemss\. Random\-effects summaries and prediction intervals estimate how stable that contrast is across subsystems and what sign or magnitude might appear on another subsystem under the fitted model\. LOSO regret measures the cost of applying the pooled recommendation from ten systems to the held\-out system\. The method\-by\-system interaction test checks whether method performance can be treated as exchangeable across systems\. We report these diagnostics alongside pooled accuracy; they are not replacement performance metrics\.

For each method pair, we estimatevs=Var⁡\(ds,i\)/nsv\_\{s\}=\\operatorname\{Var\}\(d\_\{s,i\}\)/n\_\{s\}and a paired\-bootstrap95%95\\%confidence interval using5,0005\{,\}000bootstrap samples with seed4242\. Random\-effects meta\-analysis uses the Paule\-Mandel iteratedτ2\\tau^\{2\}estimator\(Paule and Mandel,[1982](https://arxiv.org/html/2606.29159#bib.bib16)\), a Hartung\-Knapp\-Sidik\-Jonkman confidence interval\(Hartung and Knapp,[2001](https://arxiv.org/html/2606.29159#bib.bib9); Sidik and Jonkman,[2005](https://arxiv.org/html/2606.29159#bib.bib22)\), and a95%95\\%prediction intervalΔRE±tk−2,0\.975​τ2\+SE​\(ΔRE\)2\\Delta\_\{\\mathrm\{RE\}\}\\pm t\_\{k\-2,\\,0\.975\}\\sqrt\{\\tau^\{2\}\+\\mathrm\{SE\}\(\\Delta\_\{\\mathrm\{RE\}\}\)^\{2\}\}\(IntHout et al\.,[2016](https://arxiv.org/html/2606.29159#bib.bib10)\)\. For LOSO regret, we poolacc​@​1\\mathrm\{acc@\}1over ten systems, select the pooled winner, and evaluate that recommendation on the held\-out subsystem\. Case\-level interaction is tested by likelihood\-ratio comparison of fractional\-logit models with and without method\-by\-system terms, using cluster\-robust standard errors by case ID\.

#### Robustness and diagnostics\.

The robustness checks assess whether the qualitative conclusion depends on a single system, benchmark family, pooling rule, interaction model, label mapping, or CD\-1min adapter choice\. We run leave\-one\-system\-out, leave\-one\-family\-out, leave\-kk\-systems\-out atk=2k=2, mean/median/nn\-weighted/trimmed pooling, fractional\-logit variants,PetShoplabel canonicalization,PetShoptemporal\-subsystem removal, and CD\-1min adapter sweeps\. We use these checks to probe sensitivity, not to expand the claim beyond the 11\-system full\-coverage comparison set in[Figures˜2](https://arxiv.org/html/2606.29159#S5.F2)and[2](https://arxiv.org/html/2606.29159#S5.T2)\.

#### Reproducibility\.

All analyses use seed4242\. The artifact contains the matched per\-case score CSVs, figure/table scripts, and a compact audit module that recomputes per\-subsystem effects, random\-effects summaries, prediction intervals, LOSO regret, and interaction tests\. A verification harness in[Appendix˜N](https://arxiv.org/html/2606.29159#A14)maps each printed statistic back to its source CSV or JSON file\.

## 5Results

The main pattern is simple: no full\-coverage method pair keeps a stable sign across subsystems\. Below we show that pattern in paired effects, selection regret from pooled recommendations, and case\-level interaction tests\.

![Refer to caption](https://arxiv.org/html/2606.29159v1/x2.png)Figure 2:Complete pairwise comparison set over the four methods that satisfy the full\-coverage inclusion criterion\.Rows are the six pairwise comparisons amongbaro, max\-\|Z\|\|Z\|, alert\-count, and CD\-1min; columns are the 11 audited subsystems\. Each cell is the paired per\-system effectΔs\\Delta\_\{s\}in pp acc@1, oriented as row method minus second\-named method \(e\.g\., for BARO vs max\-\|Z\|\|Z\|,Δs=BARO−max⁡\-​\|Z\|\\Delta\_\{s\}=\\mathrm\{BARO\}\-\\max\\text\{\-\}\|Z\|\)\. Every method pair has positive and negative subsystem effects, so the subsystem dependence is not restricted to BARO\-centered comparisons with simple comparators\. The diverging color scale is centered at zero; vertical rules separateOpenRCA,PetShop, andRCAEval\. Asterisks mark cells whose 95% paired\-bootstrap CI excludes zero\. Cells with\|Δs\|<5\|\\Delta\_\{s\}\|<5pp are rendered near\-white to distinguish substantive effects from near\-ties\.Table 2:Heterogeneity diagnostics for the full\-coverage pairwise comparison set\. Sign counts report positive/negative/zero per\-system paired effects for the row orientation\. All six pairs have subsystem effects with both signs and random\-effects 95% prediction intervals that include zero\.PairSign counts \(\+/−/0\+/\-/0\)I2I^\{2\}RE 95% PI \(pp\)BARO vs max\-\|Z\|\|Z\|3/8/087\.8%\[\-38\.1, \+19\.9\]BARO vs alert\-count6/4/185\.1%\[\-27\.4, \+35\.4\]BARO vs CD\-1min5/6/092\.7%\[\-54\.9, \+41\.6\]max\-\|Z\|\|Z\|vs alert\-count10/1/078\.0%\[\-9\.6, \+35\.3\]max\-\|Z\|\|Z\|vs CD\-1min6/5/067\.0%\[\-21\.8, \+22\.3\]alert\-count vs CD\-1min3/8/087\.1%\[\-45\.1, \+21\.0\]

![Refer to caption](https://arxiv.org/html/2606.29159v1/x3.png)Figure 3:Leave\-one\-system\-out selection regret on the two motivating BARO\-centered comparisons\.Bars show the per\-subsystem regret incurred when the pooled recommendation from the other ten systems is applied to the held\-out subsystem\. Subsystems without a selection reversal have zero regret by definition\. Bars with black outlines mark subsystems with selection reversals; light\-gray placeholder bars mark subsystems with zero regret\. Both panels share the same y\-axis range to preserve the regret magnitude comparison\. The max\-\|Z\|\|Z\|comparison has three reversals \(Bank, High\-Traffic, Online\-Boutique\); the alert\-count comparison has four different reversals \(Market\-1, Telecom, Sock\-Shop, Train\-Ticket\), with a maximum regret of 24\.80 pp on Sock\-Shop\. The disjoint reversal sets show that LOSO selection errors are pair\-dependent rather than attributable to a fixed set of difficult subsystems\.Table 3:Leave\-one\-system\-out selection regret for the full\-coverage pairwise comparison set\. For each pair, pooled LOSO selection selects the lower\-scoring method on at least one held\-out subsystem, with mean regret up to 4\.72 pp acc@1 and max regret up to 24\.80 pp\.PairLOSO reversalsMean regret \(pp\)Max regret \(pp\)Mean\-regret CI \(pp\)BARO vs max\-\|Z\|\|Z\|3/111\.157\.69\[0\.24, 2\.52\]BARO vs alert\-count4/113\.1024\.80\[2\.09, 12\.24\]BARO vs CD\-1min4/114\.7215\.38\[1\.94, 17\.15\]max\-\|Z\|\|Z\|vs alert\-count1/110\.040\.49\[0\.00, 3\.33\]max\-\|Z\|\|Z\|vs CD\-1min5/113\.2616\.80\[2\.11, 10\.15\]alert\-count vs CD\-1min3/111\.9012\.50\[0\.03, 5\.89\]

### 5\.1All full\-coverage method pairs show subsystem\-level reversals

The four methods or comparators that satisfy the full\-coverage inclusion criterion—baro, max\-\|Z\|\|Z\|,alert\-count, and CD\-1min—produce six method pairs\. In[Figures˜2](https://arxiv.org/html/2606.29159#S5.F2)and[2](https://arxiv.org/html/2606.29159#S5.T2), each pair has at least one positive and one negative subsystem\-level effect\. This is not confined to BARO\-centered comparisons or to comparisons against simple heuristics\. The three pairs that exclude CD\-1min already show effects with both signs, prediction intervals crossing zero, and at least one LOSO reversal\.

The random\-effects summaries quantify the spread: across the six pairs,I2I^\{2\}ranges from67\.0%67\.0\\%to92\.7%92\.7\\%, and every95%95\\%prediction interval includes zero\. BARO vs CD\-1min has the strongest heterogeneity \(I2=92\.7%I^\{2\}=92\.7\\%, PI\[−54\.9,\+41\.6\]\[\-54\.9,\+41\.6\]pp,4/114/11LOSO reversals\)\. The simplest pair in the comparison set, max\-\|Z\|\|Z\|vs alert\-count, also has subsystem effects with both signs, a prediction interval that includes zero, and one LOSO selection reversal\.

### 5\.2Pooled selection incurs subsystem\-level regret

For a held\-out subsystemss, we pool scores over the other ten subsystems, select the pooled winner, and measure the regret of using that recommendation onss\.[Figure˜3](https://arxiv.org/html/2606.29159#S5.F3)visualizes this selection cost for the two BARO\-centered cross\-benchmark comparator comparisons used in[Figure˜1](https://arxiv.org/html/2606.29159#S1.F1);[Table˜3](https://arxiv.org/html/2606.29159#S5.T3)summarizes LOSO regret across all six method pairs\.

Against cross\-benchmark max\-\|Z\|\|Z\|, pooled selection selects the lower\-scoring method onOpenRCA/Bank,PetShop/High\-Traffic, andRCAEval/Online\-Boutique; mean regret is1\.151\.15ppacc​@​1\\mathrm\{acc@\}1, with a worst case of7\.697\.69pp onPetShop/High\-Traffic\. Againstalert\-count, selection reversals occur onOpenRCA/Market\-1,OpenRCA/Telecom,RCAEval/Sock\-Shop, andRCAEval/Train\-Ticket; mean regret is3\.103\.10pp, with a worst case of24\.8024\.80pp onRCAEval/Sock\-Shop\. Because the two reversal sets are disjoint, the LOSO errors cannot be attributed to a fixed group of difficult subsystems; they change with the method pair\.

Across the full comparison set, pooled LOSO selection selects the lower\-scoring method on at least one subsystem for all six pairs and on as many as5/115/11subsystems for max\-\|Z\|\|Z\|vs CD\-1min\. These counts describe the audited sample; we do not claim them as estimates of a population selection\-reversal rate\.

### 5\.3Interaction tests support non\-exchangeability

Case\-level interaction tests agree with the system\-level summaries in[Figures˜2](https://arxiv.org/html/2606.29159#S5.F2)and[2](https://arxiv.org/html/2606.29159#S5.T2)\. In the four\-method comparison set, the method\-by\-system interaction is significant \(LRT=114\.36\\mathrm\{LRT\}=114\.36,d​f=30df=30,p<10−10p<10^\{\-10\}\), and the coarser method\-by\-benchmark\-family interaction is also significant \(LRT=36\.98\\mathrm\{LRT\}=36\.98,d​f=6df=6,p=1\.78×10−6p=1\.78\\times 10^\{\-6\}; see[Appendix˜C](https://arxiv.org/html/2606.29159#A3)\)\. Five of six pairwise method\-by\-system tests are significant atα=0\.05\\alpha=0\.05\. The only non\-significant pair, max\-\|Z\|\|Z\|vs CD\-1min \(p=0\.067p=0\.067\), still shows subsystem effects with both signs, a prediction interval that includes zero, and5/115/11LOSO reversals\.

### 5\.4Within\-OpenRCA checks

One possible explanation is that the interaction is only a cross\-benchmark artifact, caused by protocol differences amongOpenRCA,RCAEval, andPetShop\. WithinOpenRCAalone, three method pairs show sign reversals between\{Bank,Telecom\}\\\{\\textsc\{Bank\},\\textsc\{Telecom\}\\\}and\{Market\-1,Market\-2\}\\\{\\textsc\{Market\-1\},\\textsc\{Market\-2\}\\\}:barovs causal\-guided adapter, causal\-guided adapter vsmax\-​\|Z\|\\textsc\{max\-\}\|Z\|, and CD\-1min vsmax\-​\|Z\|\\textsc\{max\-\}\|Z\|\([Table˜9](https://arxiv.org/html/2606.29159#A7.T9)in[Appendix˜G](https://arxiv.org/html/2606.29159#A7), Panel A\)\.

A structural difficulty proxy absorbs much of this within\-suite heterogeneity but does not eliminate it\. Adding the per\-case mean ofanomalous\_metric\_countwith method interactions reduces the residual “ZZversusYY” likelihood\-ratio statistic from143\.85143\.85to56\.6356\.63\(Δ≈61%\\Delta\\approx 61\\%absorbed\), but the residual remains significant \(p=4\.5×10−7p=4\.5\\times 10^\{\-7\};[Table˜9](https://arxiv.org/html/2606.29159#A7.T9), Panel B\)\. We treat this analysis as a within\-suite check, not as an additional claim scope\.

### 5\.5Robustness and boundary diagnostics

These checks do not remove the main pattern\. Single\-system deletion, benchmark\-family deletion, leave\-kk\-systems\-out atk=2k=2, and alternative pooling rules all leave substantial heterogeneity and prediction intervals that include zero for the original BARO\-centered comparisons\.[Table˜10](https://arxiv.org/html/2606.29159#A10.T10)reports the leave\-one\-benchmark\-family\-out diagnostic for the four\-method comparison set; because there are only three benchmark families, we treat it as a boundary analysis rather than main evidence\.[Table˜11](https://arxiv.org/html/2606.29159#A10.T11)shows that PetShop temporal\-subsystem removal, PetShop label canonicalization, and CD\-1min adapter\-threshold sweeps preserve subsystem effects with both signs and prediction intervals that include zero for CD\-1min\-involving pairs\.

circa’s modal\-output collapse on most non\-native subsystems is documented separately in[Appendix˜F](https://arxiv.org/html/2606.29159#A6)\. We treat this as an adapter\-boundary diagnostic and exclude it from the full\-coverage comparison set\.

## 6Discussion, Limitations, and Recommendations

#### Interpretation\.

Pooled accuracy is still a valid suite average\. The problem is what readers infer from it\. Without stability checks, a pooled RCA leaderboard drops the subsystem information needed to see which method scores higher for a given service graph\. In our audit, the affected subsystems change with the method pair\. Similar pooling problems have been documented in NLU benchmarks\(Bowman and Dahl,[2021](https://arxiv.org/html/2606.29159#bib.bib2)\), multi\-task suites\(Srivastava et al\.,[2023](https://arxiv.org/html/2606.29159#bib.bib24)\), and benchmark\-choice sensitivity\(Dehghani et al\.,[2021](https://arxiv.org/html/2606.29159#bib.bib5)\)\. The per\-method absolute scores in our audit come from thebaroauthors’ reference implementation packaged withRCAEval\(Pham et al\.,[2025](https://arxiv.org/html/2606.29159#bib.bib19)\); an independent concurrent audit\(Fang et al\.,[2026](https://arxiv.org/html/2606.29159#bib.bib7)\)reports comparable absolutebaroTop\-11numbers on a relatedRCAEvalrelease\.

#### Limitations and scope\.

The main limitation of this audit is structural:1111audited subsystems cluster within33benchmark families, so the effective independent\-unit count is closer to three than to eleven\. Leave\-one\-family\-out sensitivity \([Appendix˜J](https://arxiv.org/html/2606.29159#A10)\) keepsI2≥61\.7%I^\{2\}\\geq 61\.7\\%and keeps the95%95\\%PI on zero, but the lower end of that range is well below the full\-suite estimates\. We treat LOFO as a boundary diagnostic rather than a population estimate over benchmark families\. The small\-sample random\-effects analysis uses Paule\-Mandelτ2\\tau^\{2\}, Hartung\-Knapp\-Sidik\-Jonkman CIs, and IntHout prediction intervals\(IntHout et al\.,[2016](https://arxiv.org/html/2606.29159#bib.bib10); Sidik and Jonkman,[2005](https://arxiv.org/html/2606.29159#bib.bib22); Hartung and Knapp,[2001](https://arxiv.org/html/2606.29159#bib.bib9)\); these choices reduce overconfident summaries atk=11k=11, but the three benchmark families remain a convenience sample rather than a random draw from RCA deployments\.

The cross\-benchmark comparators have a narrow interpretation\. max\-\|Z\|\|Z\|andalert\-countscan the metric columns exposed by each benchmark release, while published methods may impose their own metric filters or candidate sets\. The comparators serve as reporting probes for protocol stability rather than algorithmic competitors\. CD\-1min has the same status: it is a full\-coverage adapter with threshold and label\-sensitivity checks \([Appendix˜J](https://arxiv.org/html/2606.29159#A10)\)\.

Finally, the score scale mixes native benchmark conventions\.OpenRCAuses partial\-credit scores;RCAEvalandPetShopuse strict\{0,1\}\\\{0,1\\\}top\-1 scoring\. All paired effects are computed within a subsystem on matched cases, so their signs are meaningful for method comparison\. Their magnitudes should not be read as a common difficulty scale across benchmark families\. The case\-level interaction tests use a Papke\-Wooldridge fractional logit\(Papke and Wooldridge,[1996](https://arxiv.org/html/2606.29159#bib.bib15)\)\. Our claims apply to the current public offline RCA benchmark releases under mode \(a\) of[Section˜3](https://arxiv.org/html/2606.29159#S3); closed\-loop agentic benchmarks and ad\-hoc single\-system deployments are outside scope\.

#### Recommendations\.

Benchmark authors should report per\-subsystemΔs\\Delta\_\{s\}with paired\-bootstrap CIs next to any pooled number, together withQQ,I2I^\{2\},τ2\\tau^\{2\}, and a95%95\\%PI\. A pooled state\-of\-the\-art claim should also state whether the ranking is stable across audited subsystems; when it is not, the claim should be qualified with subsystem\-level information, such as “best\-performing onXXofYYaudited subsystems\.” Benchmark users and reviewers should read a single pooled number as a benchmark\-level average, not as evidence for subsystem\-level method superiority\. Until benchmark releases include per\-subsystem reporting with heterogeneity statistics, pooled offline\-RCA leaderboards alone do not justify subsystem\-level method selection\.

## 7Conclusion

Offline RCA leaderboards are used at the subsystem level, but most reports still rank methods by pooled case\-level scores\. This audit shows what that aggregation misses\. On three public benchmark families, current reports do not test whether rankings stay stable across subsystems; in our matched audit, reversals appear in the two motivating BARO comparisons and across the four\-method pairwise set\. The released module makes the check cheap for future releases: given a matched score table, it computes the per\-subsystem effects, prediction intervals, and LOSO regret reported here\. The next step is to apply the same audit to closed\-loop agentic benchmarks once they expose matched per\-case outputs\.

## References

- Bender and Koller \[2020\]Emily M\. Bender and Alexander Koller\.Climbing towards NLU: On meaning, form, and understanding in the age of data\.In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics \(ACL\)*, pages 5185–5198\. Association for Computational Linguistics, 2020\.
- Bowman and Dahl \[2021\]Samuel R\. Bowman and George E\. Dahl\.What will it take to fix benchmarking in natural language understanding?In*Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(NAACL\-HLT\)*, pages 4843–4855\. Association for Computational Linguistics, 2021\.
- Chen et al\. \[2025\]Yinfang Chen, Manish Shetty, Gagan Somashekar, Minghua Ma, Yogesh Simmhan, Jonathan Mace, Chetan Bansal, Rujia Wang, et al\.AIOpsLab: A holistic framework to evaluate AI agents for enabling autonomous clouds\.*arXiv preprint arXiv:2501\.06706*, 2025\.
- Cochran \[1954\]William G\. Cochran\.The combination of estimates from different experiments\.*Biometrics*, 10\(1\):101–129, 1954\.
- Dehghani et al\. \[2021\]Mostafa Dehghani, Yi Tay, Alexey A\. Gritsenko, Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals\.The benchmark lottery\.*arXiv preprint arXiv:2107\.07002*, 2021\.
- DerSimonian and Laird \[1986\]Rebecca DerSimonian and Nan Laird\.Meta\-analysis in clinical trials\.*Controlled Clinical Trials*, 7\(3\):177–188, 1986\.
- Fang et al\. \[2026\]Aoyang Fang, Songhan Zhang, Yifan Yang, Haotong Wu, Junjielong Xu, Xuyang Wang, Rui Wang, Manyi Wang, Qisheng Lu, and Pinjia He\.Rethinking the evaluation of microservice RCA with a fault propagation\-aware benchmark\.In*Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering \(FSE\)*, 2026\.
- Hardt et al\. \[2024\]Michaela Hardt, William R\. Orchard, Patrick Blöbaum, Elke Kirschbaum, and Shiva Kasiviswanathan\.The PetShop dataset — finding causes of performance issues across microservices\.In*Proceedings of the Third Conference on Causal Learning and Reasoning*, volume 236 of*Proceedings of Machine Learning Research*, pages 957–978\. PMLR, 2024\.
- Hartung and Knapp \[2001\]Joachim Hartung and Guido Knapp\.On tests of the overall treatment effect in meta\-analysis with normally distributed responses\.*Statistics in Medicine*, 20\(12\):1771–1782, 2001\.
- IntHout et al\. \[2016\]Joanna IntHout, John P\. A\. Ioannidis, Maroeska M\. Rovers, and Jelle J\. Goeman\.Plea for routinely presenting prediction intervals in meta\-analysis\.*BMJ Open*, 6\(7\):e010247, 2016\.
- Jha et al\. \[2025\]Saurabh Jha, Rohan Arora, Yuji Watanabe, Takumi Yanagawa, Yinfang Chen, Jackson Clark, Bhavya Bhavya, Mudit Verma, et al\.ITBench: Evaluating AI agents across diverse real\-world IT automation tasks\.*arXiv preprint arXiv:2502\.05352*, 2025\.
- Kim et al\. \[2026\]Taeyoon Kim, Woohyeok Park, Hoyeong Yun, and Kyungyong Lee\.Why do AI agents systematically fail at cloud root cause analysis?*arXiv preprint arXiv:2602\.09937*, 2026\.
- Li et al\. \[2022\]Mingjie Li, Zeyan Li, Kanglin Yin, Xiaohui Nie, Wenchi Zhang, Kaixin Sui, and Dan Pei\.Causal inference\-based root cause analysis for online service systems with intervention recognition\.In Aidong Zhang and Huzefa Rangwala, editors,*KDD ’22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14–18, 2022*, pages 3230–3240\. ACM, 2022\.
- Liang et al\. \[2023\]Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, et al\.Holistic evaluation of language models\.*Transactions on Machine Learning Research \(TMLR\)*, 2023\.
- Papke and Wooldridge \[1996\]Leslie E\. Papke and Jeffrey M\. Wooldridge\.Econometric methods for fractional response variables with an application to 401\(k\) plan participation rates\.*Journal of Applied Econometrics*, 11\(6\):619–632, 1996\.
- Paule and Mandel \[1982\]Robert C\. Paule and John Mandel\.Consensus values and weighting factors\.*Journal of Research of the National Bureau of Standards*, 87\(5\):377–385, 1982\.
- Pham et al\. \[2024a\]Luan Pham, Huong Ha, and Hongyu Zhang\.BARO: Robust root cause analysis for microservices via multivariate Bayesian online change point detection\.*Proceedings of the ACM on Software Engineering*, 1\(FSE\):2214–2237, 2024a\.
- Pham et al\. \[2024b\]Luan Pham, Huong Ha, and Hongyu Zhang\.Root cause analysis for microservice system based on causal inference: How far are we?In*Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering \(ASE\)*, pages 706–715\. ACM, 2024b\.
- Pham et al\. \[2025\]Luan Pham, Hongyu Zhang, Huong Ha, Flora Salim, and Xiuzhen Zhang\.RCAEval: A benchmark for root cause analysis of microservice systems with telemetry data\.In*Companion Proceedings of the ACM on Web Conference 2025 \(WWW Companion\)*, pages 777–780\. ACM, 2025\.
- Ribeiro et al\. \[2020\]Marco Túlio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh\.Beyond accuracy: Behavioral testing of NLP models with CheckList\.In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics \(ACL\)*, pages 4902–4912\. Association for Computational Linguistics, 2020\.
- Riddell et al\. \[2026\]Evelien Riddell, James Riddell, Gengyi Sun, Michał Antkiewicz, and Krzysztof Czarnecki\.Stalled, biased, and confused: Uncovering reasoning failures in LLMs for cloud\-based root cause analysis\.In*Proceedings of the 2026 IEEE/ACM Third International Conference on AI Foundation Models and Software Engineering \(FORGE\)*\. ACM, 2026\.
- Sidik and Jonkman \[2005\]Kurex Sidik and Jeffrey N\. Jonkman\.Simple heterogeneity variance estimation for meta\-analysis\.*Journal of the Royal Statistical Society: Series C \(Applied Statistics\)*, 54\(2\):367–384, 2005\.
- Sidik and Jonkman \[2006\]Kurex Sidik and Jeffrey N\. Jonkman\.Robust variance estimation for random effects meta\-analysis\.*Computational Statistics & Data Analysis*, 50\(12\):3681–3701, 2006\.
- Srivastava et al\. \[2023\]Aarohi Srivastava et al\.Beyond the imitation game: Quantifying and extrapolating the capabilities of language models\.*Transactions on Machine Learning Research \(TMLR\)*, 2023\.
- Viechtbauer \[2010\]Wolfgang Viechtbauer\.Conducting meta\-analyses in R with the metafor package\.*Journal of Statistical Software*, 36\(3\):1–48, 2010\.
- Wang et al\. \[2026\]Yilun Wang, Guangba Yu, Haiyu Huang, Zirui Wang, Yujie Huang, Pengfei Chen, and Michael R\. Lyu\.Cloud\-OpsBench: A reproducible benchmark for agentic root cause analysis in cloud systems\.*arXiv preprint arXiv:2603\.00468*, 2026\.
- Xu et al\. \[2025\]Junjielong Xu, Qinan Zhang, Zhiqing Zhong, Shilin He, Chaoyun Zhang, Qingwei Lin, Dan Pei, Pinjia He, Dongmei Zhang, and Qi Zhang\.OpenRCA: Can large language models locate the root cause of software failures?In*Proceedings of the 13th International Conference on Learning Representations \(ICLR\)*, 2025\.
- Zheng et al\. \[2024\]Lecheng Zheng, Zhengzhang Chen, Dongjie Wang, Chengyuan Deng, Reon Matsuoka, and Haifeng Chen\.LEMMA\-RCA: A large multi\-modal multi\-domain dataset for root cause analysis\.*arXiv preprint arXiv:2406\.05375*, 2024\.
- Zhou et al\. \[2019\]Xiang Zhou, Xin Peng, Tao Xie, Jun Sun, Chao Ji, Dewei Liu, Qilin Xiang, and Chuan He\.Latent error prediction and fault localization for microservice applications by learning from system trace logs\.In*Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering \(ESEC/FSE\)*, pages 683–694\. ACM, 2019\.

## Appendix APer\-system audit ledger

[Table˜4](https://arxiv.org/html/2606.29159#A1.T4)gives the case counts, per\-methodacc​@​1\\mathrm\{acc@\}1, paired effects, and LOSO recommendation for each audited subsystem\. Rows marked with†are held\-out subsystems where the pooled LOSO choice selects the lower\-scoring method\.

Table 4:Per\-system audit ledger across the 11 audited subsystems\.nn: matched cases\. Acc@1 columns: held\-out per\-system accuracy \(fractional for OpenRCA, strict 0/1 elsewhere\)\.Δs\\Delta\_\{s\}in pp acc@1 with 95% paired\-bootstrap CI \(5000 iters, seed=42\)\. Pooled→\\rightarrowActual columns: leave\-one\-system\-out pooled recommendation versus actual best on the held\-out system\.†marks subsystems with selection reversals\.Per\-systemΔs\\Delta\_\{s\}\[95% CI\]Pooled→\\toActual \(max\-\|Z\|\|Z\|\)Pooled→\\toActual \(alert\-cnt\)SubsystemnnBAROmax\-\|Z\|\|Z\|alert\-cntvs max\-\|Z\|\|Z\|vs alert\-cntPooledActual \(regret\)PooledActual \(regret\)OpenRCA / Bank1360\.1600\.1270\.132\+3\.4\+3\.4\[\-2\.3, \+9\.1\]\+2\.9\+2\.9\[\-3\.7, \+9\.5\]max\-\|Z\|\|Z\|†BARO \(\+3\.35\)BAROBARO \(\+0\.00\)OpenRCA / Market\-1700\.0570\.1610\.080−10\.4\-10\.4\[\-16\.6, \-4\.5\]−2\.3\-2\.3\[\-8\.3, \+3\.3\]max\-\|Z\|\|Z\|max\-\|Z\|\|Z\|\(\+0\.00\)BARO†alert\-cnt \(\+2\.26\)OpenRCA / Market\-2780\.1060\.2000\.039−9\.4\-9\.4\[\-16\.9, \-2\.0\]\+6\.6\+6\.6\[\+1\.9, \+12\.2\]max\-\|Z\|\|Z\|max\-\|Z\|\|Z\|\(\+0\.00\)BAROBARO \(\+0\.00\)OpenRCA / Telecom510\.1830\.2320\.229−4\.9\-4\.9\[\-14\.7, \+3\.9\]−4\.6\-4\.6\[\-10\.5, \+0\.3\]max\-\|Z\|\|Z\|max\-\|Z\|\|Z\|\(\+0\.00\)BARO†alert\-cnt \(\+4\.59\)PetShop / High\-Traffic260\.1540\.0770\.038\+7\.7\+7\.7\[\+0\.0, \+19\.2\]\+11\.5\+11\.5\[\-3\.8, \+26\.9\]max\-\|Z\|\|Z\|†BARO \(\+7\.69\)BAROBARO \(\+0\.00\)PetShop / Low\-Traffic260\.1540\.1920\.000−3\.8\-3\.8\[\-11\.5, \+0\.0\]\+15\.4\+15\.4\[\+3\.8, \+30\.8\]max\-\|Z\|\|Z\|max\-\|Z\|\|Z\|\(\+0\.00\)BAROBARO \(\+0\.00\)PetShop / Temporal\-180\.2500\.3750\.000−12\.5\-12\.5\[\-37\.5, \+0\.0\]\+25\.0\+25\.0\[\+0\.0, \+62\.5\]max\-\|Z\|\|Z\|max\-\|Z\|\|Z\|\(\+0\.00\)BAROBARO \(\+0\.00\)PetShop / Temporal\-280\.2500\.3750\.250−12\.5\-12\.5\[\-37\.5, \+0\.0\]\+0\.0\+0\.0\[\-37\.5, \+37\.5\]max\-\|Z\|\|Z\|max\-\|Z\|\|Z\|\(\+0\.00\)BAROtie \(\+0\.00\)RCAEval / Online\-Boutique1250\.7280\.7120\.432\+1\.6\+1\.6\[\-4\.8, \+8\.0\]\+29\.6\+29\.6\[\+19\.2, \+40\.0\]max\-\|Z\|\|Z\|†BARO \(\+1\.60\)BAROBARO \(\+0\.00\)RCAEval / Sock\-Shop1250\.2000\.5440\.448−34\.4\-34\.4\[\-44\.0, \-24\.8\]−24\.8\-24\.8\[\-36\.0, \-13\.6\]max\-\|Z\|\|Z\|max\-\|Z\|\|Z\|\(\+0\.00\)BARO†alert\-cnt \(\+24\.80\)RCAEval / Train\-Ticket1250\.1600\.4640\.184−30\.4\-30\.4\[\-40\.0, \-20\.8\]−2\.4\-2\.4\[\-10\.4, \+5\.6\]max\-\|Z\|\|Z\|max\-\|Z\|\|Z\|\(\+0\.00\)BARO†alert\-cnt \(\+2\.40\)

## Appendix BCase\-level interaction\-test variants

[Table˜5](https://arxiv.org/html/2606.29159#A2.T5)reports the modeling variants used to check the BARO\-centered method\-by\-system interaction tests\. The variants change the response scale and separation handling; the interaction remains significant across the reported specifications\.

Table 5:Case\-level method×\\timessystem interaction tests under 4 modeling approaches for the BARO vs cross\-benchmark max\-\|Z\|\|Z\|comparator, plus the fractional GLM for the BARO vs alert\-count comparator\. All approaches reject exchangeability of method effects across systems atp<10−3p<10^\{\-3\}\. Strict variants for alert\-count are not reported because no cell triggers quasi\-separation in that comparator\.ComparatorModeling approachno​b​sn\_\{obs\}LRTdfppMcFaddenR2R^\{2\}max\-\|Z\|\|Z\|Fractional GLM \(primary\)155637\.18105\.3×10−55\.3\\\!\\times\\\!10^\{\-5\}0\.197 / 0\.177Firth penalized1556—102\.1×10−52\.1\\\!\\times\\\!10^\{\-5\}—L2\-regularized \(C=1\.0\)155635\.79109\.1×10−59\.1\\\!\\times\\\!10^\{\-5\}—Drop quasi\-sep cell148638\.31103\.4×10−53\.4\\\!\\times\\\!10^\{\-5\}—alert\-countFractional GLM \(primary\)155654\.51103\.9×10−83\.9\\\!\\times\\\!10^\{\-8\}0\.179 / 0\.146
## Appendix CFour\-method interaction tests

[Table˜6](https://arxiv.org/html/2606.29159#A3.T6)extends the interaction tests to the full four\-method comparison set\. The omnibus tests use all four full\-coverage methods or comparators; the pairwise rows show which method pairs drive the interaction signal\.

Table 6:Case\-level method\-by\-system interaction tests in the four\-method full\-coverage comparison set\. Likelihood\-ratio tests compare fractional\-logit models with and without method\-by\-group interaction terms\.ComparisonGroupLRTdfppResultOmnibus tests4\-method omnibussystem114\.3630<10−10<10^\{\-10\}significant4\-method omnibusbenchmark family36\.9861\.78×10−61\.78\\times 10^\{\-6\}significantPairwise method\-by\-system testsBARO vs max\-\|Z\|\|Z\|system37\.18105\.26×10−55\.26\\times 10^\{\-5\}significantBARO vs alert\-countsystem54\.51103\.90×10−83\.90\\times 10^\{\-8\}significantBARO vs CD\-1minsystem66\.55102\.04×10−102\.04\\times 10^\{\-10\}significantmax\-\|Z\|\|Z\|vs alert\-countsystem26\.27100\.0030\.003significantmax\-\|Z\|\|Z\|vs CD\-1minsystem17\.35100\.0670\.067n\.s\.alert\-count vs CD\-1minsystem28\.03100\.0020\.002significant
## Appendix DFull\-coverage method inclusion criterion

[Table˜7](https://arxiv.org/html/2606.29159#A4.T7)records which methods and adapters have matched outputs acrossOpenRCA,RCAEval, andPetShop\. Only rows with full matched coverage enter the primary pairwise comparison set\.

Table 7:Full\-coverage method inclusion criterion for the primary comparison set\. The primary analysis includes only methods with matched outputs on all 11 subsystems and all 778 scoring units\. CIRCA and an internal causal\-guided adapter are retained as diagnostics because their cross\-benchmark adapters do not satisfy the same inclusion criterion\.MethodOpenRCARCAEvalPetShopMainReasonBAROYesYesYesYesExisting 11/11 full\-coverage rows in the primary comparison set\.max\-\|Z\|\|Z\|YesYesYesYesUnified z\-family baseline covers all 11 subsystems\.alert\-countYesYesYesYesExisting alert\-count block aligns with all 778 scoring units\.CD\-1min adapterYesYesYesYesFull adapter run completed 11/11 subsystems and aligns with all 778 scoring units\.CIRCAadapter onlyPartialPartialNoRCAEval Train\-Ticket did not complete at limit=1; PetShop temporal subsystems did not complete in bounded diagnostic runs\.causal\-guided adapterYeslimited testslimited testsNoInternalOpenRCAgraph protocol is adapter\-defined and evaluated only in limited tests outsideOpenRCA, not a full 11/11 matched main run\.

## Appendix ELeave\-one\-system\-out regret, side\-by\-side

[Table˜8](https://arxiv.org/html/2606.29159#A5.T8)lists the held\-out recommendation and regret for each subsystem under the two BARO\-centered comparator pairs\. The table is the per\-subsystem source for the disjoint reversal sets summarized in the main text\.

Table 8:Leave\-one\-system\-out selection regret across the 11 audited subsystems, side\-by\-side for both cross\-benchmark comparators\. For each held\-out subsystem, the pooled winner is computed on the remaining 10 systems’ acc@1; regret = bestactual−\-pooled\-recommendedactualon the held\-out system\.†marks subsystems with selection reversals\. The two comparators producedisjointreversal sets: under max\-\|Z\|\|Z\|the reversals are \{Bank, High\-Traffic, Online\-Boutique\}; under alert\-count they are \{Market\-1, Telecom, Sock\-Shop, Train\-Ticket\}\. The presence of selection reversals is consistent across comparators; the identity of affected systems is comparator\-specific\.vs cross\-benchmark max\-\|Z\|\|Z\|vs per\-service alert\-countSubsystemnnPooledActualRegret \(pp\)PooledActualRegret \(pp\)OpenRCA / Bank136max\-\|Z\|\|Z\|BARO†3\.35BAROBARO0\.00OpenRCA / Market\-170max\-\|Z\|\|Z\|max\-\|Z\|\|Z\|0\.00BAROalert\-cnt†2\.26OpenRCA / Market\-278max\-\|Z\|\|Z\|max\-\|Z\|\|Z\|0\.00BAROBARO0\.00OpenRCA / Telecom51max\-\|Z\|\|Z\|max\-\|Z\|\|Z\|0\.00BAROalert\-cnt†4\.59PetShop / High\-Traffic26max\-\|Z\|\|Z\|BARO†7\.69BAROBARO0\.00PetShop / Low\-Traffic26max\-\|Z\|\|Z\|max\-\|Z\|\|Z\|0\.00BAROBARO0\.00PetShop / Temporal\-18max\-\|Z\|\|Z\|max\-\|Z\|\|Z\|0\.00BAROBARO0\.00PetShop / Temporal\-28max\-\|Z\|\|Z\|max\-\|Z\|\|Z\|0\.00BAROtie0\.00RCAEval / Online\-Boutique125max\-\|Z\|\|Z\|BARO†1\.60BAROBARO0\.00RCAEval / Sock\-Shop125max\-\|Z\|\|Z\|max\-\|Z\|\|Z\|0\.00BAROalert\-cnt†24\.80RCAEval / Train\-Ticket125max\-\|Z\|\|Z\|max\-\|Z\|\|Z\|0\.00BAROalert\-cnt†2\.40Selection\-reversal rate3/11 = 27\.27%4/11 = 36\.36%Mean regret \(pp\)1\.153\.10Max regret \(pp\)7\.6924\.80
## Appendix Fcircaadapter\-boundary diagnostics on non\-native subsystems

circa’s causal\-graph output collapses to a single modal prediction on99of1111non\-native subsystems \(modal\-prediction frequency≥95%\\geq 95\\%\)\. A schema\-repair adapter that accepts theOpenRCAdouble\-underscore metric naming reduces the modal frequency to1515–25%25\\%onOpenRCA, but post\-adapteracc​@​1\\mathrm\{acc@\}1remains in55–13%13\\%; on these four subsystems, the low accuracy is therefore not explained by the parser failure alone\. For the seven non\-OpenRCAsubsystems the adapter was not re\-run: pre\-adapter measurements onPetShop×4\\times 4andRCAEval/Sock\-Shopalready collapse to100%100\\%modal frequency, the only non\-collapsedcircaresult on a non\-OpenRCAsystem is its nativeRCAEval/Online\-Boutique\(modal frequency50\.4%50\.4\\%,acc​@​1=25\.6%\\mathrm\{acc@\}1=25\.6\\%\), andRCAEval/Train\-Ticket’s PC skeleton step does not complete within the timeout \(\>13\-day ETA\)\. Thus the off\-OpenRCAcells support a cross\-benchmark adapter limitation diagnosis, not a post\-repair algorithmic conclusion\.[Figure˜4](https://arxiv.org/html/2606.29159#A6.F4)summarizes the four\-column modal\-output\-collapse matrix\.

![Refer to caption](https://arxiv.org/html/2606.29159v1/x4.png)Figure 4:CIRCA modal\-output collapse across 11 audited subsystems\.Rows are subsystems grouped by benchmark family; columns report CIRCA’s modal\-prediction frequency \(red colormap; high = collapsed to a single output, bad\) and top\-1 accuracy \(green colormap; high = good\), each shown before and after the RHT schema\-repair adapter\. On all four OpenRCA subsystems the adapter drops modal\_freq from100%100\\%to≤25%\\leq 25\\%and CIRCA is no longer collapsed to a single modal output, but post\-adapter acc@1 stays in55–13%13\\%; on these four subsystems, the low accuracy is not explained by preprocessing alone\. Post\-adapter cells for the seven non\-OpenRCA subsystems are hatched \(“—”\) because the adapter was not re\-run off OpenRCA; pre\-adapter measurements for those rows show100%100\\%collapse on PetShop×4\\times 4and RCAEval / Sock\-Shop, and the only non\-collapsed CIRCA result on a non\-OpenRCA system is its native RCAEval / Online\-Boutique benchmark \(50\.4%50\.4\\%modal\_freq,25\.6%25\.6\\%acc@1\)\. These off\-OpenRCA rows diagnose cross\-benchmark adapter limitations rather than post\-repair behavior\. RCAEval / Train\-Ticket is fully hatched because the PC skeleton step did not complete within the timeout \(\>13\>13\-day ETA\)\.
## Appendix GWithin\-OpenRCA reversals and structural\-covariate analysis

The within\-OpenRCA evidence behind Section[5\.4](https://arxiv.org/html/2606.29159#S5.SS4)appears in Table[9](https://arxiv.org/html/2606.29159#A7.T9)\(preceding page\)\. Panel A gives the per\-system paired effectsΔs\\Delta\_\{s\}used in the within\-suite reversal claim: three method pairs reverse sign between \{Bank, Telecom\} and \{Market\-1, Market\-2\}, all fourΔs\\Delta\_\{s\}estimated on matched OpenRCA cases\. Panel B asks how much of this within\-suite heterogeneity is captured by a structural difficulty proxy—the per\-case anomalous\-metric count—through a nested fractional\-logit comparison\. The proxy absorbs about 61% of the per\-system dummy variance, and the residual method\-by\-system interaction is still significant atp=4\.5×10−7p=4\.5\\times 10^\{\-7\}\.

Table 9:Within\-OpenRCA reversals \(Panel A\) and structural\-covariate explanatory analysis \(Panel B\)\. Together these locate the heterogeneity: reversals exist within a single benchmark suite, and a single structural proxy \(anomalous\-metric count\) explains roughly 60% of the per\-system variance — the remaining 40% is residual system identity\.Panel A\.Per\-system paired effectsΔs=methoda−methodb\\Delta\_\{s\}=\\text\{method\}\_\{a\}\-\\text\{method\}\_\{b\}\(pp acc@1\) with 95% paired\-bootstrap CIs \(5000 iters, seed=42\)\. Three method pairs reverse sign between \{Bank, Telecom\} and \{Market\-1, Market\-2\}, arguing against cross\-benchmark protocol differences as the only driver of heterogeneity\.

Method pairBankMkt\-1Mkt\-2TelBARO vs causal\-guided\+3\.2\+3\.2\[\-2\.6, \+8\.8\]−10\.4\-10\.4\[\-17\.7, \-3\.9\]−8\.7\-8\.7\[\-16\.3, \-1\.0\]\+0\.0\+0\.0\[\-10\.8, \+10\.5\]causal\-guided vs max\-\|Z\|\|Z\|−10\.1\-10\.1\[\-16\.8, \-3\.7\]\+3\.3\+3\.3\[\-0\.1, \+7\.1\]\+2\.7\+2\.7\[\-1\.4, \+7\.1\]−2\.3\-2\.3\[\-7\.9, \+3\.2\]cd1min vs max\-\|Z\|\|Z\|−6\.5\-6\.5\[\-12\.2, \-1\.0\]\+3\.3\+3\.3\[\-0\.1, \+7\.1\]\+2\.7\+2\.7\[\-1\.4, \+7\.1\]−2\.3\-2\.3\[\-7\.9, \+3\.2\]

Panel B\.Structural\-covariate nested model comparison \(GLM\-Binomial fractional logit,no​b​s=1556n\_\{obs\}=1556,nc​a​s​e​s=778n\_\{cases\}=778\)\. ModelXX:correct ~ method \+ structural covariates;YY:XXwith method×\\timescovariate interactions;ZZ:YYwith full per\-system dummy interactions \(baseline\)\. The structural proxy \(anomalous\_metric\_count\_mean\) absorbs≈\\approx61% of per\-system dummy variance \(comparingZZvsYYbefore vs after adding the proxy\); the residualZZvsYYLRT remains highly significant \(56\.6356\.63on df=1414,p=4\.5×10−7p=4\.5\\\!\\times\\\!10^\{\-7\}\)\.

TestLRTdfppInterpretationYYvsXX\(struct\.×\\timesmethod\)17\.9234\.6×10−44\.6\\\!\\times\\\!10^\{\-4\}method effects depend on structural difficultyZZvsYY\(residual system var\.\)56\.63144\.5×10−74\.5\\\!\\times\\\!10^\{\-7\}residual heterogeneity beyond proxyZZvsYY\(no proxy, baseline\)143\.8516≈0\\approx 0structural proxy absorbs≈\\approx61%

## Appendix HPer\-modality metric\-column lists and schema notes

Per\-benchmark metric\-column manifests, anomaly\-window definitions, and the exactmetric\_idregex parsers used in the audit pipeline are documented in the artifact schema notes\. Briefly:OpenRCAuses double\-underscore naming<component\>\_\_<metric\_group\>\_<\.\.\.\>;RCAEvaluses single\-underscore<service\>\_<metric\>;PetShopuses dotted Prometheus\-native naming\. In thecircaimplementation we audited, the root\-cause hypothesis tester assumes single\-underscore metric names; the schema\-repair adapter adds a regex that admits double\-underscore\.

## Appendix IPer\-subsystem forest\-plot data in tabular form

The full per\-subsystemΔs\\Delta\_\{s\}, sampling variancevsv\_\{s\}, paired\-bootstrap CI, and meta\-analytic weightws=1/\(vs\+τ2\)w\_\{s\}=1/\(v\_\{s\}\+\\tau^\{2\}\)for both themax\-​\|Z\|\\textsc\{max\-\}\|Z\|andalert\-countcomparators are included in the artifact’s per\-system delta table\. For brevity, the main\-text[Table˜4](https://arxiv.org/html/2606.29159#A1.T4)reports the five most\-cited columns per subsystem\.

## Appendix JRobustness checks

#### Leave\-one\-system\-out \(LOSO\)\.

Across all1111LOSO sub\-samples for themax\-​\|Z\|\\textsc\{max\-\}\|Z\|comparator,I2I^\{2\}stays in\[0\.822,0\.890\]\[0\.822,0\.890\]and the95%95\\%PI includes zero in11/1111/11sub\-samples\. For thealert\-countcomparator,I2∈\[0\.770,0\.866\]I^\{2\}\\in\[0\.770,0\.866\]across1111LOSO sub\-samples, with PI including zero in11/1111/11\.

#### Leave\-one\-family\-out \(LOFO\)\.

Dropping each ofOpenRCA,RCAEval, andPetShopin turn keepsI2I^\{2\}in the substantial tier \(≥50%\\geq 50\\%\) under themax\-​\|Z\|\\textsc\{max\-\}\|Z\|comparator \(range\[0\.617,0\.918\]\[0\.617,0\.918\]across the four scenarios including the baseline\) and likewise underalert\-count\(range\[0\.609,0\.899\]\[0\.609,0\.899\]\)\. The95%95\\%PI includes zero in4/44/4LOFO scenarios for both comparators\. Detailed outputs are included in the released artifact\.

#### Leave\-kk\-systems\-out atk=2k=2\(LKSO\)\.

The full distribution of5555LKSO sub\-samples confirms that no two\-system removal collapsesI2I^\{2\}below the substantial threshold or pulls the PI off zero\. Detailed outputs are included in the released artifact\.

#### Alternative pooling rules\.

Mean / median /nn\-cases\-weighted /10%10\\%\-trimmed\-mean pooling rules all preserve the selection\-reversal sets for both comparators\. Detailed outputs are included in the released artifact\.

#### Case\-level bootstrap of reversal stability\.

For each subsystem with a selection reversal, the case\-level bootstrap \(5000 iterations, seed4242\) reports the probability that the actual higher\-scoring method is preserved under case\-resampling; for all such subsystems this probability exceeds0\.850\.85\. Detailed outputs are included in the released artifact\.

Table 10:Leave\-one\-benchmark\-family\-out and variance diagnostics for the four\-method comparison set\. Family holdout is diagnostic rather than primary evidence because there are only three benchmark families\. The variance columns separate between\-family share from within\-family sign changes\.PairLOFO reversalsMax regretMean regretReversal familiesBetween\-family SS \(%\)Internal sign changesMax family rangeBARO vs max\-\|Z\|\|Z\|0/30\.00\.0–31\.63/336\.0BARO vs alert\-count0/30\.00\.0–16\.82/354\.4BARO vs CD\-1min2/331\.714\.9petshop, rcaeval74\.91/344\.8max\-\|Z\|\|Z\|vs alert\-count0/30\.00\.0–33\.31/333\.7max\-\|Z\|\|Z\|vs CD\-1min2/318\.39\.6petshop, rcaeval83\.21/317\.3alert\-count vs CD\-1min0/30\.00\.0–76\.62/325\.0

Table 11:CD\-1min robustness checks\. Panel A removes PetShop’s two temporal subsystems and separately canonicalizes PetShop root\-cause labels; CD\-1min\-involving pairs retain subsystem effects with both signs and PIs that include zero\. Panel B sweeps the CD\-1min adapter thresholds on the seven non\-OpenRCA systems used for adapter diagnosis\.PairI2I^\{2\}fullI2I^\{2\}no\-temp\.PI no\-temp\.LOSO fullLOSO no\-temp\.I2I^\{2\}canon\.PI canon\.LOSO canon\.BARO vs CD\-1min92\.793\.9yes4291\.8yes2max\-\|Z\|\|Z\|vs CD\-1min67\.067\.4yes5753\.6yes8alert\-count vs CD\-1min87\.189\.5yes3284\.5yes1

Adapter configSystemsMacro acc@1Min acc@1Max acc@1Macro acc@3Mean modal freq\.default\_z3\_pct5\_win10733\.30\.079\.252\.825\.6z2\.5\_pct3\_win10732\.70\.079\.252\.825\.6z3\.5\_pct10\_win10733\.40\.079\.258\.625\.6z3\_pct5\_win20731\.50\.073\.652\.125\.6z3\_pct5\_win5734\.50\.087\.254\.125\.4

## Appendix KWithin\-OpenRCA3\-method exploratory analysis \(F6\-C\)

On theOpenRCAsubset we also inspected a three\-method design \(\{baro,zbaselegacy,zbase\_univ\}×4\\\{\\textsc\{baro\},\\texttt\{zbase\}\_\{\\text\{legacy\}\},\\texttt\{zbase\\\_univ\}\\\}\\times 4subsystems;335335cases\) to check whether the method\-by\-system interaction survives when multiple Z\-score variants appear as separate methods\. The omnibus 6\-df likelihood\-ratio test isp=0\.25p=0\.25on this small subset\. Three exploratory follow\-ups—a 1\-df contrast isolating theBank\-versus\-others difference for the legacy\-minus\-universal contrast, a2,0002\{,\}000\-permutation empiricalpp\-value for the 6\-df omnibus, and a paired bootstrap CI on theBank\-versus\-others contrast—all reject the null atα=0\.05\\alpha=0\.05\. We present these follow\-ups transparently as post\-hoc, hypothesis\-generating analyses; they are excluded from the main claims\. The primary evidence in the paper rests on the1111\-system meta\-analysis, the cross\-comparator replication \([Section˜5\.2](https://arxiv.org/html/2606.29159#S5.SS2)\), and the leave\-one\-system\-out selection\-reversal analysis, not on this within\-suite subset\.

## Appendix LAlert\-count threshold sensitivity and third comparator

We re\-ran thealert\-countpredictor at\|z\|\|z\|thresholds in\{2\.5,3,3\.5\}\\\{2\.5,3,3\.5\\\}\. The selection\-reversal set is unchanged across all three thresholds: pooled LOSO selection still selectsbarowhenalert\-countis higher\-scoring on\{Market\-1,Telecom,Sock\-Shop,Train\-Ticket\}\\\{\\textsc\{Market\-1\},\\textsc\{Telecom\},\\textsc\{Sock\-Shop\},\\textsc\{Train\-Ticket\}\\\}at every threshold\. Per\-systemΔs\\Delta\_\{s\}values are exactly identical across the three thresholds for all fourOpenRCAsubsystems \(alert\-count’sOpenRCApath is dominated by threshold\-invariant component ranking\); shifts onPetShopandRCAEvalrange from0\.80\.8to11\.611\.6pp \(mean absolute shift1\.511\.51pp att=2\.5t=2\.5vst=3t=3,1\.851\.85pp att=3\.5t=3\.5vst=3t=3\), with a single outlier \(RCAEval/Sock\-Shop\) accounting for the bulk of the variation\. Detailed outputs are included in the released artifact\.

We also report the legacyzbaseonOpenRCAas a third independent comparator: per\-systemΔs\\Delta\_\{s\}forbaroversus legacyzbaseagree in sign with thezbase\_univ\(cross\-benchmarkmax\-​\|Z\|\\textsc\{max\-\}\|Z\|\) comparison on33of44OpenRCAsubsystems and disagree onBank\(the10\.310\.3pp divergence noted in[Section˜5\.5](https://arxiv.org/html/2606.29159#S5.SS5)\)\. The legacy comparator is not cross\-benchmark and is reported as a label\-sensitivity check only\.

## Appendix MFormal definitions of cross\-benchmark comparators

This appendix expands the cross\-benchmark comparator definitions from Section[4\.1](https://arxiv.org/html/2606.29159#S4.SS1)with the full notation and release file names\.

Cross\-benchmark max\-\|Z\|\|Z\|predictor \(zbase\_univ\)\.For an injected fault caseiiwith anomaly window\[t0,t1\]\[t\_\{0\},t\_\{1\}\]on subsystemss, given a metric matrixMs∈ℝT×dM\_\{s\}\\in\\mathbb\{R\}^\{T\\times d\}over the case window, compute per\-columnzz\-scoresZs,j,tZ\_\{s,j,t\}using the pre\-fault interval as the baseline\. Predict the metric \(and its mapped service / component\) achievingarg​maxj⁡maxt∈\[t0,t1\]⁡\|Zs,j,t\|\\operatorname\*\{arg\\,max\}\_\{j\}\\max\_\{t\\in\[t\_\{0\},t\_\{1\}\]\}\|Z\_\{s,j,t\}\|\. No tuning, no training/validation split, no per\-benchmark hyperparameter; the only design choice is the metric\-name parser used to map a winning metric to a service identifier, which is shared across all audited methods in this audit\.

Per\-service alert\-count predictor \(alert\-count\)\.For each metricjjin caseiion subsystemss, define the indicatoras,j,i=𝟏​\{maxt∈\[t0,t1\]⁡\|Zs,j,t\|\>3\}a\_\{s,j,i\}=\\mathbf\{1\}\\\{\\max\_\{t\\in\[t\_\{0\},t\_\{1\}\]\}\|Z\_\{s,j,t\}\|\>3\\\}\. Aggregate to the service level: for each serviceSS, the alert count isAS=∑j∈Sas,j,iA\_\{S\}=\\sum\_\{j\\in S\}a\_\{s,j,i\}\. Predictarg​maxS⁡AS\\operatorname\*\{arg\\,max\}\_\{S\}A\_\{S\}, with alphabetical tiebreak\. The\|z\|\>3\|z\|\>3threshold is a standard anomaly\-detection default; sensitivity to the threshold is reported in[Appendix˜L](https://arxiv.org/html/2606.29159#A12)\.

Both predictors use the same anomaly\-window definition and the same metric\-name\-to\-service mapping as the audited methods; the∼40\{\\sim\}40\-line implementations are released alongside the paper\.

## Appendix NReproduction and verification package

Artifact components\.The anonymized artifact is available at[https://anonymous\.4open\.science/r/rca\-leaderboard\-audit\-artifact\-1FC2](https://anonymous.4open.science/r/rca-leaderboard-audit-artifact-1FC2)\. It contains four components: \(i\) a compact audit module that computes per\-subsystem effects, paired\-bootstrap confidence intervals, random\-effects summaries, prediction intervals, LOSO regret, and LOFO/LKSO sensitivity checks; \(ii\) implementations of the two cross\-benchmark comparators, max\-\|Z\|\|Z\|andalert\-count; \(iii\) matched per\-case score tables and generated result files for all figures and tables; and \(iv\) a reviewer\-facing verification harness that recomputes every main\-text statistic from the score tables and checks agreement to numerical tolerance\. The artifact README provides the exact directory layout and single\-command reproduction instructions\.

Environment\.Python3\.123\.12,statsmodels0\.14\.x0\.14\.x,numpy1\.26\.x1\.26\.x,scipy1\.13\.x1\.13\.x,pandas2\.x2\.x\. Arequirements\.txtand aconda env exportare included\. The full audit \(per\-system effects, both comparators, LOSO/LOFO/LKSO sensitivity, all interaction\-test variants\) runs in approximately two CPU\-minutes on a four\-core laptop with no GPU; we report no GPU compute\.

Verification\.A reviewer\-facing verification table lists, for every numeric statistic that appears in the main text: \(i\) the number printed in the paper, \(ii\) an independent recomputation, and \(iii\) the absolute difference\. The released verification harness produces this table from the raw matched score inputs; all reported main\-text statistics match to numerical tolerance\.

License\.The audit module, audit harness, and per\-system raw\-output tables are released under Apache 2\.0\. The audited benchmarks \(OpenRCA,RCAEval,PetShop\) are used under their respective licenses; we redistribute only our derived per\-method per\-case score CSVs, not the original benchmark releases\.