Evidence-State Reliability Under Controlled Degradation: Parser-Validity Divergence in a Multi-Stage LLM Pipeline

# Evidence-State Reliability Under Controlled Degradation:Parser-Validity Divergence in a Multi-Stage LLM Pipeline
Source: [https://arxiv.org/html/2608.21559](https://arxiv.org/html/2608.21559)
August 2026

###### Abstract

Multi\-stage LLM pipelines can remain structurally valid even when evidence available to downstream stages becomes incomplete, compressed, or conflicting\. This paper introduces and operationalises Evidence\-State Reliability \(ESR\), an evaluation layer concerned with whether intermediate evidence remains sufficiently complete, grounded, internally consistent, and usable for a stage’s assigned function\. ESR is evaluated separately from parser validity, which measures structural conformance\.

We evaluate the framework using GLM\-5\.2 on 60 sanitised base cases under four evidence conditions: clean, compressed\-lossy, partial\-dropout, and noisy\-conflicting\. Each condition was processed through decision, audit, and escalation stages\. The design comprised 720 planned and ledgered calls, with 713 retained sanitised execution rows\.

Across nine matched degraded\-minus\-clean condition\-stage comparisons, all operational stage\-success estimates were negative and all 95% bootstrap intervals remained below zero\. All nine parser\-validity point estimates were positive, although the three partial\-dropout intervals included zero\. Among parser\-valid degraded audit outputs, degradation detection was 1\.0 in each degraded condition while false\-assurance rates remained non\-zero; among parser\-valid degraded escalation outputs, recovery was 0\.0 in every degraded condition\.

The results show a bounded reliability\-layer divergence in the evaluated pipeline: structural conformance can improve directionally while evidence\-sensitive stage success deteriorates under the same controlled intervention\. They also separate detection of degraded evidence from recovery\. The conclusions are limited to the evaluated model configuration, pipeline design, selected sanitized cases, scoring procedure, and single scaled run\.

Keywords:Evidence\-State Reliability; large language models; multi\-stage AI pipelines; parser validity; evidence sufficiency; controlled degradation; computational audit; escalation; reliability cascade

## 1Introduction

Large language models are increasingly used as components of multi\-stage systems rather than only as isolated question\-answering tools\. A pipeline may receive or retrieve evidence, transform that evidence into an intermediate representation, generate a decision, inspect the decision through an audit component, and route difficult cases to an escalation mechanism\. In such systems, reliability depends not only on the final output but also on the evidence transferred between stages\.

Evidence can deteriorate before it reaches a downstream component\. Relevant information may be removed during compression, omitted through partial data loss, or made difficult to reconcile by conflicting signals\. A later stage may therefore receive an evidence state that is no longer adequate for its assigned function while continuing to produce fluent, syntactically valid, and machine\-readable output\.

This creates a reliability problem that structural validation alone cannot capture\. A parser\-valid response satisfies a formal output contract, such as syntactically valid JSON containing the required fields\. Parser validity does not establish that the available evidence was sufficient, appropriately grounded, internally consistent, or usable for the substantive function assigned to the stage\. A structurally valid object can therefore continue through an automated pipeline without producing a technical exception even when its evidential basis has deteriorated\.

This paper introduces and operationalises*Evidence\-State Reliability*\(ESR\) as an evaluation layer for multi\-stage AI pipelines\. ESR concerns whether the evidence supplied to a stage remains sufficiently complete, grounded, internally consistent, and usable for the evidence\-sensitive function assigned to that stage\. The intermediate evidence state is therefore treated as an explicit object of evaluation rather than being assessed only indirectly through the final output\.

ESR is not intended to replace accuracy, calibration, robustness, faithfulness, parser validation, or explanation assessment\. These measures evaluate different properties of system behaviour\. Parser validity asks whether an output satisfies a structural contract\. ESR concerns whether the evidence available to a stage can support the function assigned to that stage\. In the present study, stage\-specific success is used as an operational indicator of ESR, but it is not treated as an exhaustive measurement of the broader construct\.

The central research question is:

> *How can Evidence\-State Reliability be measured separately from parser validity in a multi\-stage LLM decision pipeline?*

Three supporting research questions guide the analysis:

1. RQ1\.Do parser validity and evidence\-sensitive stage success respond differently to controlled evidence degradation?
2. RQ2\.Does successful detection of degraded evidence at an audit stage lead to recovery at a subsequent escalation stage?
3. RQ3\.What sequence\-level failure patterns become visible when decision, audit, and escalation outcomes are analysed together rather than as isolated outputs?

To investigate these questions, the study uses a controlled within\-case experiment with 60 sanitised base cases derived from the context of the Consumer Financial Protection Bureau Consumer Complaint Database\. Each base case was represented under four evidence conditions: a clean experimental baseline, compressed\-lossy evidence, partial\-dropout evidence, and noisy\-conflicting evidence\. Each condition was processed through three pipeline stages: decision, computational audit, and escalation\. The planned experimental design therefore comprised

60​cases×4​evidence conditions×3​stages=720​model calls\.60\\text\{ cases\}\\times 4\\text\{ evidence conditions\}\\times 3\\text\{ stages\}=720\\text\{ model calls\}\.
Base\-case identity was preserved across conditions so that each degraded observation could be compared with the corresponding clean observation for the same case and pipeline stage\.

The experiment examines whether structural conformance and evidence\-sensitive stage success respond differently to the same evidence interventions\. Across the nine matched degraded\-minus\-clean condition\-stage comparisons, all operational stage\-success estimates were negative and all corresponding 95% bootstrap intervals remained below zero\. In contrast, all nine parser\-validity point estimates were positive, although the three partial\-dropout parser\-validity intervals included zero\. Within the evaluated pipeline, structural conformance therefore moved in a favourable direction while evidence\-sensitive stage success deteriorated under the same controlled interventions\.

The downstream stages revealed a second distinction\. Among parser\-valid degraded audit outputs, the implemented audit indicator detected degradation in every degraded condition\. Detection, however, did not imply recovery\. Audit false\-assurance rates remained non\-zero, and no parser\-valid degraded escalation output was classified as recovered\. Detection of an evidential problem and restoration of a successful evidence\-sensitive outcome are therefore treated as separate pipeline functions in this study\.

The study makes four bounded contributions\. First, it operationalises parser validity, evidence\-sensitive stage success, reliability\-layer divergence, and format\-layer false assurance as distinct measurements within a stage\-aware evaluation framework\. Second, it applies controlled evidence\-state interventions across a decision–audit–escalation sequence while preserving base\-case identity across experimental conditions\. Third, it reports matched condition\-stage comparisons and nonparametric bootstrap intervals rather than relying only on aggregate performance values\. Fourth, it connects stage\-level behaviour to sequence\-level reliability patterns that distinguish parser failure, detected\-but\-unrecovered degradation, audit false assurance, incomplete execution, and preserved success\.

The resulting framework is intended as a diagnostic approach to evaluating intermediate evidence states and stage transitions\. The findings do not establish universal LLM unreliability or show that evidence degradation generally improves parser validity\. They remain confined to the evaluated GLM\-5\.2 configuration, pipeline design, controlled evidence conditions, selected sanitised cases, scoring procedure, and single scaled experimental run\. The study likewise does not establish deployment safety, regulatory validity, complaint truth, consumer\-harm prevalence, provider\-independent behaviour, or cross\-model generality\.

## 2Related Work

### 2\.1Evaluation beyond aggregate correctness

Reliability cannot always be represented by a single aggregate performance measure\. Behavioural testing frameworks such as CheckList decompose model performance into capability\-oriented evaluations rather than assuming that held\-out accuracy captures every relevant aspect of behaviour\([15](https://arxiv.org/html/2608.21559#bib.bib15)\)\. Calibration research similarly distinguishes predictive correctness from the reliability of reported confidence estimates\([6](https://arxiv.org/html/2608.21559#bib.bib6)\)\.

Retrieval\-augmented generation evaluation follows a related multidimensional approach\. RAGAS separates properties including faithfulness, context relevance, and answer relevance, while ARES evaluates multiple components contributing to retrieval\-augmented generation performance\([3](https://arxiv.org/html/2608.21559#bib.bib3);[16](https://arxiv.org/html/2608.21559#bib.bib16)\)\. These approaches illustrate a broader principle relevant to the present study: an output may satisfy one evaluation criterion while remaining deficient under another\.

Evidence\-State Reliability adopts this layered perspective but focuses on the evidence available at intermediate stages of a pipeline\. Rather than treating final\-answer quality as the only object of evaluation, ESR examines whether a stage receives evidence capable of supporting its assigned evidence\-sensitive function\.

### 2\.2Structural validity and evidence sufficiency

Structured outputs are important when language\-model responses must be consumed automatically by downstream software\. Grammar\-constrained decoding and related approaches can improve adherence to formal output specifications\([5](https://arxiv.org/html/2608.21559#bib.bib5)\)\. Work on structured\-output evaluation further distinguishes formal validity from properties such as value correctness, executability, and task performance\([14](https://arxiv.org/html/2608.21559#bib.bib14);[17](https://arxiv.org/html/2608.21559#bib.bib17)\)\.

The present study does not claim to originate the distinction between structural validity and substantive adequacy\. Instead, it examines how structural validity and evidence\-sensitive stage success respond to controlled changes in the evidence supplied to the same pipeline\. Parser validity is measured independently at the decision, audit, and escalation stages, allowing its response to evidence degradation to be compared with evidence\-sensitive success under the same condition\-stage combinations\.

Evidence sufficiency has also been studied in fact checking, retrieval, and selective generation\. Work on fact checking with insufficient evidence demonstrates the importance of distinguishing claims that can be supported from those for which the available evidence is inadequate\([1](https://arxiv.org/html/2608.21559#bib.bib1)\)\. Related work on sufficiency\-aware retrieval and delayed ground truth further considers how evidence availability can affect whether a task is answerable or can be acted upon reliably\([12](https://arxiv.org/html/2608.21559#bib.bib12);[18](https://arxiv.org/html/2608.21559#bib.bib18)\)\.

ESR extends this concern from the evidence supporting an endpoint answer to evidence states within a multi\-stage sequence\. In the present experiment, controlled versions of the same base case are processed through decision, audit, and escalation stages, making it possible to examine not only whether degradation affects stage success, but also whether it is detected and whether a downstream mechanism recovers from it\.

### 2\.3Component\-aware evaluation, auditing, and recovery

Evaluation of multi\-stage and agentic systems increasingly considers component\-level behaviour rather than only final outputs\. AgentBench evaluates capabilities required by interactive language\-model agents\([8](https://arxiv.org/html/2608.21559#bib.bib8)\), while retrieval\-augmented generation frameworks separately assess components contributing to an eventual response\([3](https://arxiv.org/html/2608.21559#bib.bib3);[16](https://arxiv.org/html/2608.21559#bib.bib16)\)\. Component\-level evaluation can reveal where a failure appears in a sequence and how later stages respond to it\.

The term*audit*requires particular care\. Algorithmic\-audit research addresses broader organisational processes involving documentation, accountability, governance, and institutional practice\([13](https://arxiv.org/html/2608.21559#bib.bib13)\)\. The audit stage in this study is narrower: it is a computational, model\-output\-coded component that records whether evidence degradation was detected and whether an experimental false\-assurance indicator appeared\. It is not an independent institutional audit and does not establish regulatory assurance\.

Selective prediction and learning\-to\-defer research similarly distinguish prediction from abstention, referral, and expert intervention\([4](https://arxiv.org/html/2608.21559#bib.bib4);[10](https://arxiv.org/html/2608.21559#bib.bib10)\)\. The escalation stage evaluated here is not a learned optimal deferral policy\. Its role is to measure whether the implemented downstream mechanism recovered after degradation had entered the pipeline\. This distinction is central to the present design because detecting a reliability problem and restoring successful operation are separate outcomes\.

### 2\.4Reliability cascades

Cascading failure is a long\-standing systems concept in which local disruptions propagate through connected structures\([9](https://arxiv.org/html/2608.21559#bib.bib9)\)\. Recent work on multi\-agent language\-model systems has likewise examined how errors or hallucinations can propagate between interacting components or agents\([20](https://arxiv.org/html/2608.21559#bib.bib20);[7](https://arxiv.org/html/2608.21559#bib.bib7)\)\.

The term*reliability cascade*is used more narrowly in this study\. It refers to an operational sequence\-level pattern in which parser failure, evidence\-sensitive failure, detected\-but\-unrecovered degradation, false assurance, or incomplete execution prevents a preserved successful path through the evaluated decision–audit–escalation sequence\.

This taxonomy is diagnostic rather than a general theory of cascading failure\. It distinguishes structurally different sequence outcomes so that parser failure, detected\-but\-unrecovered degradation, false assurance, and incomplete execution are not collapsed into a single endpoint score\. The resulting cascade rate is therefore interpreted only as a summary of the evaluated experimental sequences, not as an estimate of failure prevalence in deployed AI systems\.

### 2\.5Positioning of this work

The literature above addresses several components relevant to Evidence\-State Reliability: multidimensional evaluation, structured\-output validity, evidence sufficiency, component\-level assessment, computational auditing, selective prediction, deferral, recovery, and cascading failure\. The present study does not claim that these individual ideas are new\.

Its contribution is their bounded integration within a single controlled and matched multi\-stage experiment\. The framework jointly evaluates controlled evidence\-state degradation, parser validity, evidence\-sensitive stage success, reliability\-layer divergence, audit detection, false assurance, escalation recovery, paired uncertainty, and sequence\-level failure patterns while preserving base\-case identity across evidence conditions\.

The resulting contribution is therefore methodological and empirical rather than a claim of global priority\. This work operationalises these distinctions as a stage\-aware evaluation framework for examining how structural and evidence\-sensitive reliability measures respond to the same controlled evidence interventions\.

## 3Evidence\-State Reliability Framework

### 3\.1Evidence states and stage\-specific objectives

LetEi,c,kE\_\{i,c,k\}denote the evidence state associated with base caseii, evidence conditioncc, and pipeline stagekk\.

An evidence state may contain original records, extracted fields, summaries, retrieved passages, intermediate conclusions, structured observations, or transformed representations\. Its reliability cannot be assessed independently of the objective assigned to the stage receiving it\. An evidence state that is adequate for a broad classification may, for example, be insufficient for a detailed audit or recovery action\.

In this study, Evidence\-State Reliability concerns whether the evidence available to a stage remains:

- •sufficiently complete for the assigned objective;
- •grounded in the available source material;
- •internally consistent enough to support interpretation; and
- •usable by the receiving stage for its intended function\.

These properties define the broader ESR construct\. They are not assumed to be exhaustively represented by any single experimental variable\. The primary operational indicator used in this experiment is stage\-specific success under the committed scoring contract\.

### 3\.2Parser validity

Let

Vi,c,k∈\{0,1\}V\_\{i,c,k\}\\in\\\{0,1\\\}
indicate whether the output associated with base caseii, conditioncc, and stagekkis parser\-valid\.

Parser validity means that the sanitised model output satisfies the required structural contract\. A parser\-valid output can, for example, contain syntactically valid JSON from which all required fields can be successfully parsed\.

LetIc,kI\_\{c,k\}denote the retained observations for conditionccand stagekk, with

Nc,k=\|Ic,k\|\.N\_\{c,k\}=\|I\_\{c,k\}\|\.
The parser\-validity rate is

PV^c,k=1Nc,k​∑i∈Ic,kVi,c,k\.\\widehat\{\\mathrm\{PV\}\}\_\{c,k\}=\\frac\{1\}\{N\_\{c,k\}\}\\sum\_\{i\\in I\_\{c,k\}\}V\_\{i,c,k\}\.
Parser validity is necessary for dependable automated consumption, but it does not measure evidence sufficiency, substantive correctness, complaint truth, regulatory validity, or real\-world decision quality\.

### 3\.3Operational ESR rate

Let

Ri,c,k∈\{0,1\}R\_\{i,c,k\}\\in\\\{0,1\\\}
denote evidence\-sensitive stage success under the committed scoring contract\.

A retained observation is classified as stage\-successful when both of the following conditions are satisfied:

1. 1\.the output is parser\-valid; and
2. 2\.the sanitisedvalidity\_judgmentfield is positive\.

Rows without a parser\-valid output or a positive validity judgment are not counted as successful\.

The operational ESR rate for conditionccand stagekkis

ESR^c,k=1Nc,k​∑i∈Ic,kRi,c,k\.\\widehat\{\\mathrm\{ESR\}\}\_\{c,k\}=\\frac\{1\}\{N\_\{c,k\}\}\\sum\_\{i\\in I\_\{c,k\}\}R\_\{i,c,k\}\.
The notationESR^\\widehat\{\\mathrm\{ESR\}\}refers specifically to the operational indicator used in this experiment\. It does not imply that stage success exhaustively measures every dimension of Evidence\-State Reliability\.

The analysis also retains a separate binary field,evidence\_state\_adequate\. Let

Ai,c,k∈\{0,1\}A\_\{i,c,k\}\\in\\\{0,1\\\}
denote this model\-output\-coded indicator\.Ai,c,kA\_\{i,c,k\}records whether the supplied evidence state was coded as adequate for the relevant stage, whereasRi,c,kR\_\{i,c,k\}records whether the complete stage\-success criterion was satisfied\.

The evidence\-adequacy indicator is therefore distinct from operational ESR\. It is also not an independent human adjudication of evidence quality\. Matched degraded\-minus\-clean changes in evidence\-state adequacy are calculated using the same paired procedure applied to parser validity and operational ESR\.

### 3\.4Matched changes under evidence degradation

Letc0c\_\{0\}denote the clean evidence condition andcca degraded condition\. For each base case available under both conditions at stagekk, define the case\-level change in parser validity as

δi,c,kPV=Vi,c,k−Vi,c0,k\.\\delta^\{\\mathrm\{PV\}\}\_\{i,c,k\}=V\_\{i,c,k\}\-V\_\{i,c\_\{0\},k\}\.
LetPc,kP\_\{c,k\}denote the set of base cases for which both the clean and degraded observations are retained at stagekk, and let

nc,k=\|Pc,k\|\.n\_\{c,k\}=\|P\_\{c,k\}\|\.
The matched mean change in parser validity is

Δ​PVc,k=1nc,k​∑i∈Pc,kδi,c,kPV\.\\Delta\\mathrm\{PV\}\_\{c,k\}=\\frac\{1\}\{n\_\{c,k\}\}\\sum\_\{i\\in P\_\{c,k\}\}\\delta^\{\\mathrm\{PV\}\}\_\{i,c,k\}\.
The corresponding case\-level change in operational ESR is

δi,c,kESR=Ri,c,k−Ri,c0,k,\\delta^\{\\mathrm\{ESR\}\}\_\{i,c,k\}=R\_\{i,c,k\}\-R\_\{i,c\_\{0\},k\},
with matched mean

Δ​ESRc,k=1nc,k​∑i∈Pc,kδi,c,kESR\.\\Delta\\mathrm\{ESR\}\_\{c,k\}=\\frac\{1\}\{n\_\{c,k\}\}\\sum\_\{i\\in P\_\{c,k\}\}\\delta^\{\\mathrm\{ESR\}\}\_\{i,c,k\}\.
This formulation preserves base\-case identity: each degraded observation is compared with the corresponding clean observation for the same case and pipeline stage\.

### 3\.5Reliability\-layer divergence

Reliability\-layer divergence \(RLD\) measures the difference between the response of structural validity and the response of evidence\-sensitive stage success to the same evidence intervention\.

For degraded conditionccat stagekk,

RLDc,k=Δ​PVc,k−Δ​ESRc,k\.\\mathrm\{RLD\}\_\{c,k\}=\\Delta\\mathrm\{PV\}\_\{c,k\}\-\\Delta\\mathrm\{ESR\}\_\{c,k\}\.
A positive RLD value indicates that parser validity changed more favourably than operational ESR\.

The pattern of primary interest in this study is

Δ​PVc,k\>0andΔ​ESRc,k<0\.\\Delta\\mathrm\{PV\}\_\{c,k\}\>0\\qquad\\text\{and\}\\qquad\\Delta\\mathrm\{ESR\}\_\{c,k\}<0\.
Under this pattern, structural conformance improves directionally while evidence\-sensitive stage success deteriorates under the same intervention\. The two evaluation layers therefore provide opposing assessments of the intervention\.

RLD does not replace either component measure\. Its purpose is to make divergence between structural and evidence\-sensitive behaviour explicit\.

### 3\.6False assurance at the format layer

False assurance at the format layer occurs when an output is parser\-valid but does not satisfy the operational evidence\-sensitive success criterion\.

The general format\-layer false\-assurance rate is

FA^c,k=1Nc,k​∑i∈Ic,kVi,c,k​\(1−Ri,c,k\)\.\\widehat\{\\mathrm\{FA\}\}\_\{c,k\}=\\frac\{1\}\{N\_\{c,k\}\}\\sum\_\{i\\in I\_\{c,k\}\}V\_\{i,c,k\}\\left\(1\-R\_\{i,c,k\}\\right\)\.
This measure identifies outputs that satisfy the structural contract while remaining unsuccessful under the committed stage\-specific scoring rule\.

The audit\-specific false\-assurance result reported later in the paper uses a different calculation\. Its denominator consists only of parser\-valid audit rows, and its numerator consists of those rows for which the committedaudit\_false\_assurancefield is positive\. The audit\-specific measure is therefore a model\-output\-coded pipeline indicator and should not be interpreted as independent adjudication that an output was incorrect or misleading\.

### 3\.7Audit detection and escalation recovery

Audit detection is defined among parser\-valid audit outputs\. It records the proportion for which

audit\_detected\_degradation=true\.\\texttt\{audit\\\_detected\\\_degradation\}=\\texttt\{true\}\.
Escalation recovery is defined among parser\-valid escalation outputs\. It records the proportion for which

escalation\_recovery=true\.\\texttt\{escalation\\\_recovery\}=\\texttt\{true\}\.
These variables represent different pipeline functions\. Audit detection records whether the computational audit stage identified evidence degradation\. Escalation recovery records whether the subsequent stage restored a successful evidence\-sensitive outcome\.

A detected problem is therefore not treated as equivalent to a recovered problem\. Keeping the two measurements separate allows the analysis to identify cases in which degradation is recognised while the downstream mechanism remains unable to restore successful operation\.

### 3\.8Sequence\-level reliability cascades

A sequence is constructed for each base\-case\-by\-condition combination using the corresponding decision, audit, and escalation outcomes\.

The operational cascade taxonomy assigns sequences to five mutually exclusive outcome families:

1. 1\.Parser\-failure cascade:one or more required stage outputs fail the structural contract\.
2. 2\.Detected\-but\-unrecovered degradation:degradation is detected, but the escalation stage does not restore operational success\.
3. 3\.Audit false assurance:the sequence contains a positive audit false\-assurance indicator under the committed scoring contract\.
4. 4\.Incomplete persisted sequence:one or more expected retained stage rows are missing\.
5. 5\.Preserved success:the sequence completes without meeting any of the operational cascade\-failure criteria\.

The taxonomy is diagnostic rather than causal\. It classifies the observed sequence pattern but does not independently establish the mechanism that produced it\.

The categories distinguish different reliability problems\. Parser failure concerns structural output or validation\. Detected\-but\-unrecovered degradation indicates that recognition was not accompanied by successful remediation\. Audit false assurance concerns a nominally valid audit output carrying the committed false\-assurance indicator\. Incomplete sequences represent execution or persistence failures\.

Separating these outcomes prevents structurally different reliability problems from being collapsed into a single end\-to\-end failure measure\.

## 4Methodology

### 4\.1Study design

The primary study used a paired, controlled design centred on 60 sanitised base cases\. Each base case was represented under four evidence conditions and processed through three pipeline stages\. The planned execution therefore comprised

60​cases×4​evidence conditions×3​pipeline stages=720​model calls\.60\\text\{ cases\}\\times 4\\text\{ evidence conditions\}\\times 3\\text\{ pipeline stages\}=720\\text\{ model calls\}\.
Preserving base\-case identity across conditions allowed each degraded observation to be compared with the corresponding clean observation for the same case and pipeline stage\.

The 60 cases were randomly selected from a pre\-specified, version\-controlled sanitised evidence\-state package\. Every selected case contained all four recognised evidence conditions\. The execution manifest therefore contained 240 condition\-linked evidence states:

- •60 clean;
- •60 compressed\-lossy;
- •60 partial\-dropout; and
- •60 noisy\-conflicting\.

The selected cases are not treated as a representative or probability sample of consumer complaints, financial decisions, or deployed AI\-system interactions\. The random selection was performed within the pre\-specified experimental package rather than from a defined population of real\-world consumer complaints\. The cases therefore form a controlled experimental set for evaluating how the implemented pipeline responds to defined evidence\-state interventions\.

### 4\.2Framework development and primary\-study boundary

The ESR framework was refined through preliminary simulation, real\-model, sensitivity, and deterministic\-domain studies\. These developmental studies informed the evidence\-condition definitions, the separation of decision, audit, and escalation functions, and the operational reliability\-cascade taxonomy\.

The developmental programme included a 750\-row simulation study, a 60\-chain GLM\-5\.2 pilot, a smaller Claude Opus 4\.8 comparison subset, and a 72\-chain deterministic second\-domain study with robustness checks\. These studies were exploratory and developmental\. Their observations are not pooled with the primary experiment, treated as independent replications, or used to increase the effective sample size reported in this paper\.

The principal empirical evidence analysed here is therefore the scaled 720\-call GLM\-5\.2 experiment\.

### 4\.3Evidence provenance and sanitisation

The evidence packets were derived from a local export of the Consumer Financial Protection Bureau Consumer Complaint Database downloaded on 7 July 2026\([2](https://arxiv.org/html/2608.21559#bib.bib2)\)\.

Case identifiers and evidence materials were sanitised before experimental execution and reporting\. The shared reporting boundary excluded:

- •raw CFPB records;
- •raw prompts;
- •raw model responses;
- •JSONL output archives;
- •API credentials; and
- •environment files\.

CFPB provenance was used to provide a substantively realistic complaint\-context substrate\. It was not used to establish whether an individual complaint was factually correct, whether a company acted unlawfully, whether a regulatory violation occurred, or whether an actual financial decision was appropriate\.

The experimental units are sanitised evidence states and model\-output\-coded stage outcomes\. The study therefore evaluates a controlled pipeline\-reliability mechanism rather than consumer outcomes, company conduct, regulatory compliance, misconduct, or population prevalence\.

### 4\.4Evidence conditions

#### 4\.4\.1Clean

The clean condition served as the non\-degraded experimental baseline\. The term clean does not imply that the evidence representation was exhaustive, technically perfect, or guaranteed to produce a parser\-valid model output\. It identifies the baseline representation against which the three controlled evidence interventions were compared\.

#### 4\.4\.2Compressed\-lossy

The compressed\-lossy condition represented an evidence state that had been condensed in a way capable of removing details required by downstream stages\. The intervention was designed to preserve broad case context while reducing potentially consequential detail\.

This condition represents information loss associated with processes such as summarisation, compression, filtering, or transformation before evidence reaches a downstream component\.

#### 4\.4\.3Partial\-dropout

The partial\-dropout condition represented the controlled absence of required evidence elements\. Unlike the noisy\-conflicting condition, partial\-dropout did not introduce an explicit contradiction\.

It modelled a setting in which missing information creates evidential insufficiency or uncertainty without necessarily providing a clear signal that the remaining evidence is unreliable\.

#### 4\.4\.4Noisy\-conflicting

The noisy\-conflicting condition introduced competing, distracting, or internally difficult\-to\-reconcile evidence\.

This intervention tested the response of the pipeline after evidential conflict had entered the sequence, including whether the conflict was detected and whether subsequent audit and escalation functions produced successful outcomes\.

The three degraded conditions do not represent every possible evidence failure\. They operationalise three controlled degradation families:

- •information loss through compression;
- •information loss through omission; and
- •evidential conflict through noise\.

They do not directly model temporal staleness, distribution shift, adversarial manipulation, policy ambiguity, retrieval failure, or delayed ground truth\.

### 4\.5Pipeline stages

#### 4\.5\.1Decision

The decision stage generated an assessment or recommendation from the supplied evidence state\. Its principal measurements were:

- •parser validity;
- •evidence adequacy as encoded in the sanitised output; and
- •evidence\-sensitive stage success\.

The decision\-stage outputs were evaluated only as computational outputs of the experimental pipeline\. They were not treated as real financial decisions or regulatory determinations\.

#### 4\.5\.2Audit

The audit stage examined whether the available evidence and preceding decision state indicated degradation or concern\.

Among parser\-valid audit rows, two additional indicators were evaluated:

- •degradation detection; and
- •audit false assurance\.

These indicators are computational, model\-output\-coded variables produced within the experimental pipeline\. They do not represent independent institutional auditing, professional assurance, or regulatory review\.

#### 4\.5\.3Escalation

The escalation stage evaluated the downstream response after evidence degradation had entered the sequence\.

Among parser\-valid escalation rows, the recovery indicator recorded whether the implemented mechanism restored operational stage success\. The corresponding failure indicator recorded continued lack of recovery under the committed scoring contract\.

Detection and recovery were therefore treated as separate functions\. An audit stage could identify an evidential problem while the escalation stage still failed to restore a successful evidence\-sensitive outcome\.

### 4\.6Model execution and accounting

The experiment used the model identifier GLM\-5\.2\.

The planned and ledgered execution contained 720 calls\. The complete ledger contained

calls, comprising 470 parser\-valid and 250 parser\-invalid ledger records\.

The retained sanitised execution dataset contained 713 rows:

comprising 470 parser\-valid and 243 parser\-invalid retained rows\. Every parser\-valid ledger output was retained\.

The difference between the planned ledger and retained execution data was therefore

Seven attempted calls had ledger records but no retained sanitised execution row because of recorded URL or timeout errors\. Three of these missing retained observations occurred under the clean condition and four under the noisy\-conflicting condition\. No compressed\-lossy or partial\-dropout execution row was missing\.

By pipeline stage, the seven missing rows consisted of:

- •three decision rows;
- •three audit rows; and
- •one escalation row\.

These seven ledger\-only failures were excluded from execution\-level analyses and were not imputed\. Condition\-stage rates were calculated using the actual number of retained rows in each cell, producing retained denominators ranging from 58 to 60\.

Matched degraded\-minus\-clean comparisons included only base cases for which both the clean observation and corresponding degraded observation were retained for the same pipeline stage\. The resulting paired sample sizes ranged from 57 to 60\.

The maximum cumulative estimated model\-execution cost recorded during the experiment was USD 2\.2731216\. The approved execution ceiling was USD 8\.00\.

### 4\.7Uncertainty analysis

Matched degraded\-minus\-clean mean differences were evaluated for three model\-output\-coded measurements:

- •parser validity;
- •evidence\-sensitive stage success; and
- •the committed evidence\-state\-adequacy indicator\.

Uncertainty was estimated using a nonparametric bootstrap over matched base cases\. For each condition\-stage\-metric comparison, matched case pairs were resampled together so that the relationship between the clean and degraded observations for the same base case was preserved\.

Each comparison used:

- •2,000 bootstrap resamples; and
- •fixed random seed 5205\.

Percentile\-based 95% bootstrap intervals were calculated for all 27 combinations of

3​degraded conditions×3​pipeline stages×3​measurements\.3\\text\{ degraded conditions\}\\times 3\\text\{ pipeline stages\}\\times 3\\text\{ measurements\}\.
The intervals characterise uncertainty across the selected sanitised base cases under the implemented pairing, execution, and scoring procedures\. They are not interpreted as population\-level confidence intervals for deployed financial systems, consumers, institutions, or model providers\.

The analysis records whether each interval includes zero\. It does not use the bootstrap results to make unsupported claims of statistical significance, population generalisability, or causal identification beyond the controlled within\-case intervention design\.

### 4\.8Reproducibility and reporting boundary

The committed research artifacts support deterministic verification of:

- •execution and retention counts;
- •parser accounting;
- •metric definitions;
- •condition\-stage tables;
- •matched comparisons;
- •bootstrap intervals;
- •cascade classifications;
- •figure source data; and
- •claim\-to\-evidence mappings\.

This reporting approach is intended to make the analysis inspectable and traceable, consistent with established machine\-learning reproducibility principles\([11](https://arxiv.org/html/2608.21559#bib.bib11)\)\.

The shared artifacts do not support exact prompt\-response replay because raw prompts, raw model responses, raw CFPB records, JSONL execution archives, API credentials, and environment files are excluded from the reporting package\.

The study therefore provides artifact\-level reproducibility and auditability rather than unrestricted reconstruction of every original hosted\-model interaction\.

## 5Results

### 5\.1Execution and parser accounting

The planned execution contained 720 model calls, all of which were represented in the execution ledger\. Of these, 713 produced retained sanitised execution rows\. Table[1](https://arxiv.org/html/2608.21559#S5.T1)summarises the accounting\.

Table 1:Execution and parser accounting\.The seven ledger\-only observations were not evenly distributed\. Three occurred under the clean condition and four under noisy\-conflicting evidence\. By stage, the missing observations consisted of three decision rows, three audit rows, and one escalation row\. No compressed\-lossy or partial\-dropout execution row was missing\. Retained condition\-stage denominators therefore ranged from 58 to 60\.

### 5\.2Parser validity and stage success by condition

Table[2](https://arxiv.org/html/2608.21559#S5.T2)reports the retained observations, parser\-valid outputs, and evidence\-sensitive stage\-success outcomes for every condition\-stage combination\.

Table 2:Condition\-by\-stage structural and evidence\-sensitive outcomes\.The clean\-condition parser\-validity rates were 0\.508475 at decision, 0\.517241 at audit, and 0\.416667 at escalation\. The clean condition therefore served as an evidence baseline rather than a technically perfect output condition\.

Every degraded condition\-stage cell had a higher parser\-validity rate than its corresponding clean cell\. At the same time, every degraded condition\-stage cell had an operational stage\-success rate of 0\.0\.

These cell\-wise comparisons are descriptive\. The matched analysis below preserves base\-case identity and therefore provides the primary basis for evaluating degraded\-minus\-clean changes\. Because only cases retained under both conditions enter each matched comparison, the paired means differ slightly from the unpaired cell\-rate differences in Table[2](https://arxiv.org/html/2608.21559#S5.T2)\.

Across the nine paired degraded comparisons, parser\-validity mean changes ranged from\+0\.067797\+0\.067797to\+0\.368421\+0\.368421, whereas operational stage\-success mean changes ranged from−0\.517241\-0\.517241to−0\.406780\-0\.406780\. The two measurements therefore moved in opposite directions under every evaluated degraded condition\-stage comparison\.

![Refer to caption](https://arxiv.org/html/2608.21559v1/figure1_parser_stage_divergence.png)Figure 1:Paired parser\-validity and evidence\-sensitive stage\-success changes under controlled degradation\. Points show degraded\-minus\-clean means and 95% bootstrap intervals from 2,000 resamples with fixed seed 5205\. All nine parser\-validity point estimates are positive, although the three partial\-dropout intervals include zero\. All nine stage\-success intervals remain below zero\.
### 5\.3Paired bootstrap results

Tables[3](https://arxiv.org/html/2608.21559#S5.T3)–[5](https://arxiv.org/html/2608.21559#S5.T5)report the matched degraded\-minus\-clean estimates and percentile\-based 95% bootstrap intervals\.

Table 3:Paired degraded\-minus\-clean changes in parser validity\.All nine parser\-validity point estimates were positive\. Six of the nine bootstrap intervals remained entirely above zero\. The three intervals containing zero were all partial\-dropout comparisons:

- •decision:\[−0\.118644,0\.271186\]\[\-0\.118644,\\ 0\.271186\];
- •audit:\[−0\.034483,0\.328017\]\[\-0\.034483,\\ 0\.328017\]; and
- •escalation:\[−0\.066667,0\.300000\]\[\-0\.066667,\\ 0\.300000\]\.

The parser\-validity results therefore support directional improvement in every paired comparison, but not uniform interval separation from zero under partial dropout\.

Table 4:Paired degraded\-minus\-clean changes in operational stage success\.All nine operational stage\-success estimates were negative, and all nine corresponding bootstrap intervals remained below zero\.

Table 5:Paired degraded\-minus\-clean changes in the model\-output\-coded evidence\-state\-adequacy indicator\.All nine evidence\-state\-adequacy estimates were negative, and all nine corresponding intervals remained below zero\. This provides a second model\-output\-coded indicator that changed in the same direction as operational stage success under the tested degradations\.

Together, the paired results show the central reliability\-layer divergence observed in this experiment: parser\-validity point estimates moved in a favourable direction while the two evidence\-sensitive indicators moved in the opposite direction\.

### 5\.4Audit detection and false assurance

Among parser\-valid degraded audit outputs, the degradation\-detection indicator was positive in every evaluated row:

- •compressed\-lossy: 48 of 48, or 1\.0;
- •partial\-dropout: 41 of 41, or 1\.0; and
- •noisy\-conflicting: 52 of 52, or 1\.0\.

The implemented audit field therefore consistently signalled degradation among parser\-valid degraded audit outputs\.

Detection did not eliminate the audit false\-assurance indicator\. It appeared in:

- •3 of 48 compressed\-lossy parser\-valid audit rows, or 0\.062500;
- •2 of 41 partial\-dropout parser\-valid audit rows, or 0\.048780; and
- •1 of 52 noisy\-conflicting parser\-valid audit rows, or 0\.019231\.

These rates were non\-zero despite degradation detection being 1\.0 in each degraded condition\. Detection, false assurance, and downstream recovery are therefore reported as distinct outcomes\. Both audit detection and audit false assurance remain model\-output\-coded computational indicators rather than independent external audit findings\.

![Refer to caption](https://arxiv.org/html/2608.21559v1/figure2_audit_detection_recovery.png)Figure 2:Audit detection and escalation recovery under controlled evidence degradation\. Among parser\-valid degraded outputs, the audit degradation\-detection rate was 1\.0 in each degraded condition, while escalation recovery was 0\.0 in every degraded condition\. These are model\-output\-coded indicators from the implemented computational pipeline and do not represent institutional audit effectiveness\.
### 5\.5Escalation recovery

In the clean condition, all 25 parser\-valid escalation outputs were stage\-successful, giving a success rate of 1\.0 when the denominator was restricted to parser\-valid clean escalation outputs\.

Under degradation, no parser\-valid escalation output was coded as recovered:

- •compressed\-lossy: 0 of 43;
- •partial\-dropout: 0 of 32; and
- •noisy\-conflicting: 0 of 37\.

Escalation recovery was therefore 0\.0 in each degraded condition under the implemented scoring contract\.

This result establishes detection without recovery within the evaluated pipeline\. It does not establish that escalation mechanisms are inherently ineffective or that recovery would remain absent under a different model, prompt, evidence source, retrieval mechanism, human\-review pathway, or recovery architecture\.

### 5\.6Sequence\-level reliability cascades

The sequence\-level analysis formed 240 groups, corresponding to 60 base cases under four evidence conditions\. Of these, 234 contained all three retained stage rows and six were incomplete\.

A total of 223 groups met the operational cascade\-failure definition, while 17 were classified as preserved successes\. The aggregate cascade\-failure rate was therefore

223240=0\.929167\.\\frac\{223\}\{240\}=0\.929167\.
This statistic includes both clean and degraded conditions and is not interpreted as a degraded\-condition treatment effect or as an estimate of failure prevalence in deployment\.

![Refer to caption](https://arxiv.org/html/2608.21559v1/figure3_cascade_failure_rate.png)Figure 3:Sequence\-level cascade\-failure rate across all evidence conditions\. The operational cascade definition classified 223 of 240 condition\-linked sequence groups as failures\. This aggregate includes both clean and degraded conditions and should not be interpreted as a degradation treatment effect or as an estimate of failure prevalence in deployed systems\.Table[6](https://arxiv.org/html/2608.21559#S5.T6)reports the sequence\-level composition\.

Table 6:Sequence\-level reliability\-cascade composition\.Across all 240 sequence groups, the mutually exclusive taxonomy contained:

- •143 parser\-failure cascades;
- •71 detected\-but\-unrecovered patterns;
- •3 audit\-false\-assurance patterns;
- •6 incomplete sequences; and
- •17 preserved successes\.

The five categories sum to 240\.

![Refer to caption](https://arxiv.org/html/2608.21559v1/figure4_failure_family_composition.png)Figure 4:Sequence\-level failure\-family composition\. Across 240 condition\-linked groups, the operational taxonomy contained 143 parser\-failure cascades, 71 detected\-but\-unrecovered patterns, three audit\-false\-assurance patterns, six incomplete sequences, and 17 preserved successes\.Clean evidence accounted for 43 cascade failures among 60 groups, including 40 parser\-failure cascades and three incomplete sequences; the remaining 17 clean groups were preserved successes\. Each degraded condition produced 60 groups meeting the operational cascade\-failure definition\.

The aggregate cascade statistic serves a diagnostic purpose that differs from the paired degradation analysis\. The matched condition\-stage comparisons quantify changes associated with controlled evidence degradation, whereas the cascade taxonomy describes the sequence\-level outcome family observed for each condition\-linked group\.

## 6Discussion

### 6\.1Reliability\-layer divergence

The central result is not simply that controlled evidence degradation reduced operational stage success\. The more informative finding is that structural and evidence\-sensitive indicators moved in opposite directions under the same interventions\.

Across all nine degraded condition\-stage comparisons, parser\-validity point estimates were positive, while operational stage\-success estimates were negative\. All nine stage\-success bootstrap intervals remained below zero\. By contrast, three parser\-validity intervals, all under partial dropout, included zero\. The observed pattern therefore supports directional reliability\-layer divergence across all tested comparisons, while providing stronger interval separation for some degradation conditions than others\.

This distinction matters because a system evaluated only through schema compliance could appear structurally improved even while its evidence\-sensitive performance deteriorated\. In the evaluated pipeline, parser validity and operational ESR therefore measured different properties of system behaviour rather than interchangeable notions of reliability\.

The experiment does not establish why this divergence occurred\. Compression or omission might reduce the amount or complexity of information that must be expressed within a structured response, while conflicting evidence might alter response behaviour in ways that affect parsing\. These possibilities are interpretations rather than demonstrated mechanisms\. The present design establishes the divergence itself, not its causal explanation\.

Parser validity nevertheless remains necessary\. Structural failure was the largest sequence\-level failure family, and automated pipelines require outputs that downstream components can parse reliably\. The result therefore does not support replacing structural validation\. It supports evaluating structural conformance alongside evidence\-sensitive measures rather than treating parser validity as a sufficient proxy for substantive reliability\.

### 6\.2Interpreting the clean baseline

The clean evidence condition was not a technically easy output condition\. Parser\-validity rates were 0\.508475 at decision, 0\.517241 at audit, and 0\.416667 at escalation\.

These relatively low baseline rates created substantial headroom for positive parser\-validity changes under degraded evidence\. The observed positive parser\-validity estimates should therefore not be interpreted as evidence that degradation intrinsically improves structural conformance\.

The paired design still provides a meaningful within\-case comparison because each degraded observation is matched to the clean observation for the same base case and stage\. However, the clean baseline constrains the scope of interpretation\. The supported result is that parser validity increased directionally relative to this particular baseline while evidence\-sensitive stage success deteriorated\.

Future replication should test whether the same divergence persists when the clean condition already has high structural validity, for example under different output contracts, schema\-constrained decoding, prompting strategies, or model configurations\.

### 6\.3Interpreting zero degraded stage success

Every degraded condition\-stage cell had an operational stage\-success rate of 0\.0 under the committed scoring contract\. Consistently, all nine paired stage\-success bootstrap intervals remained below zero\.

This is a strong within\-experiment pattern, but the operational definition requires careful interpretation\. Stage success required both a parser\-valid output and a positive model\-output\-codedvalidity\_judgment\. Rows without a parser\-valid output or a positive validity judgment were not counted as successful\.

Operational stage success is therefore an indicator of ESR within this experiment rather than ESR itself\. ESR is the broader construct concerning whether an evidence state remains sufficiently complete, grounded, internally consistent, and usable for the function assigned to a stage\.

The separately recordedevidence\_state\_adequatevariable provides convergent support because all nine corresponding degraded\-minus\-clean intervals also remained below zero\. However, both stage success and evidence\-state adequacy are derived from model\-output\-coded fields rather than independent human or expert adjudication\.

The supported conclusion is therefore that multiple committed indicators changed consistently under controlled degradation within the evaluated pipeline\. The experiment does not establish that these variables provide an exhaustive or universally valid measurement of Evidence\-State Reliability\.

Independent adjudication would be required to determine how closely the model\-output\-coded variables correspond to external assessments of evidence sufficiency, grounding, decision support, audit correctness, and successful recovery\.

### 6\.4Detection is not recovery

The audit and escalation results reveal a second important separation between reliability functions\.

Among parser\-valid degraded audit outputs, the degradation\-detection indicator was positive in every evaluated row\. Despite this detection rate of 1\.0, audit false\-assurance indicators remained non\-zero, and no parser\-valid degraded escalation output was coded as recovered\.

Recognition of a degraded evidence state therefore did not imply restoration of successful operation in the implemented pipeline\. A monitoring component may identify a problem without having access to the evidence, authority, or mechanism required to resolve it\.

This distinction is important for interpreting Figure[2](https://arxiv.org/html/2608.21559#S5.F2)\. A high detection rate alone does not characterise the success of an audit\-and\-escalation pathway\. Evaluation should distinguish at least:

- •degradation detection;
- •false assurance;
- •escalation action;
- •recovery;
- •safe deferral; and
- •unresolved failure\.

Effective recovery may require information or capabilities that are not available to a monitoring stage, such as retrieving missing evidence, resolving contradictory information, applying a different decision policy, involving an authorised reviewer, or declining to proceed\.

The present experiment establishes that recovery was absent under the implemented escalation mechanism\. It does not compare alternative recovery architectures or establish that recovery is impossible under other designs\.

### 6\.5Why sequence\-aware evaluation matters

The sequence\-level analysis shows why stage\-level metrics and endpoint metrics answer different questions\.

Across the 240 condition\-linked groups, the operational taxonomy contained 143 parser\-failure cascades, 71 detected\-but\-unrecovered patterns, three audit\-false\-assurance patterns, six incomplete sequences, and 17 preserved successes\.

These categories represent different technical problems\. Parser\-failure cascades concern structural conformance or validation\. Detected\-but\-unrecovered degradation indicates that recognition was not accompanied by successful remediation\. Audit false assurance identifies sequences containing the committed false\-assurance indicator\. Incomplete sequences reflect execution or persistence problems rather than only model reasoning\.

A single end\-to\-end failure statistic would obscure these distinctions\.

The aggregate cascade\-failure rate of 0\.929167 should also be interpreted carefully\. It includes a clean\-condition failure rate of 0\.716667 and is therefore not a pure degradation effect\. The paired condition\-stage comparisons provide the stronger basis for interpreting changes associated with controlled evidence degradation, whereas the cascade taxonomy provides a diagnostic description of the observed sequence outcomes\.

Taken together, the results support a layered approach to evaluating multi\-stage LLM systems\. Structural validity, evidence\-state adequacy, evidence\-sensitive stage success, degradation detection, false assurance, recovery, and sequence completion should be measured separately when they represent different system functions\.

The same principle has implications for pipeline design\. A structural contract specifies the format required for automated consumption, but an evidence\-sensitive system also requires rules governing what evidence is sufficient, how conflict or absence should be represented, when deferral should occur, and what changes after a problem has been detected\.

Without such distinctions, a pipeline can remain structurally operational while the evidence available to support its substantive function has deteriorated\. The present results therefore support treating intermediate evidence states and stage transitions as measurable parts of reliability evaluation rather than evaluating only the final response\.

## 7Threats to Validity and Limitations

The limitations of the study are considered across construct, internal, external, conclusion, measurement\-coverage, and reproducibility dimensions, following established empirical\-research validity principles\([19](https://arxiv.org/html/2608.21559#bib.bib19)\)\. These limitations constrain the interpretation and generalisability of the findings but do not alter the reported within\-design comparisons\.

### 7\.1Construct validity

Evidence\-State Reliability is broader than the operational variables used in this experiment\.

The primary operational ESR indicator, stage success, required both:

1. 1\.a parser\-valid output; and
2. 2\.a positive model\-output\-codedvalidity\_judgment\.

Rows without a parser\-valid output or a positive validity judgment were not counted as successful\. This rule was applied consistently, but it may combine genuine evidence inadequacy with behaviour induced by the model, prompt, or output contract\.

The additional indicators for evidence\-state adequacy, audit detection, audit false assurance, and escalation recovery were likewise derived from sanitised model\-output fields rather than independent human, expert, or rule\-based adjudication\.

Agreement among these indicators provides convergent evidence within the implemented experiment\. It does not establish that they exhaustively represent ESR or that the same operational definitions would remain valid across different models, prompts, domains, or scoring contracts\.

Future validation should compare these model\-output\-coded variables with independent assessments of evidence sufficiency, grounding, consistency, audit correctness, appropriate deferral, and successful recovery\.

### 7\.2Internal validity

The paired design controls for base\-case identity by comparing clean and degraded observations for the same case and pipeline stage\. Nevertheless, the observed reliability\-layer divergence may also depend on properties of the experimental implementation, including:

- •prompt design;
- •parser\-contract difficulty;
- •output schema;
- •pipeline ordering;
- •stage\-specific instructions;
- •scoring rules; and
- •interactions among these elements\.

The clean condition had relatively low parser\-validity rates, ranging from approximately 0\.42 to 0\.52 across stages\. This created substantial headroom for positive parser\-validity changes under degraded conditions\. The results therefore do not establish that evidence degradation intrinsically improves structural validity\.

The primary experiment also consisted of one scaled execution using the evaluated model configuration\. It therefore does not quantify run\-to\-run variability under repeated executions of the same configuration\.

Seven of the 720 ledgered calls had no retained sanitised execution row because of recorded URL or timeout errors\. These observations were excluded rather than imputed\. Every parser\-valid ledger output was retained, but the execution dataset was not perfectly complete\.

Matched analyses reduced the effect of these missing observations by including only base cases for which both clean and degraded observations were retained for the relevant stage\. Even so, paired sample sizes varied from 57 to 60, and the missing observations could have affected individual estimates\.

### 7\.3External validity

The primary study evaluated:

- •one GLM\-5\.2 model configuration;
- •one decision–audit–escalation pipeline;
- •one scaled execution run;
- •one selected set of 60 sanitised base cases;
- •one committed scoring contract; and
- •three controlled evidence\-degradation families\.

The findings therefore do not establish cross\-model generality, provider independence, domain independence, or universal behaviour across multi\-stage LLM systems\.

The 60 selected cases are not claimed to constitute a representative or probability sample of consumer complaints, financial decisions, or deployed AI interactions\.

CFPB provenance provides a realistic complaint\-context substrate, but it does not establish complaint truth, institutional misconduct, regulatory violations, consumer\-harm prevalence, population\-level frequencies, or the quality of actual financial decisions\.

The tested interventions represent information loss through compression, information loss through omission, and evidential conflict through noise\. They do not directly evaluate other reliability threats such as:

- •temporal staleness;
- •distribution shift;
- •retrieval failure;
- •adversarial manipulation;
- •policy ambiguity;
- •delayed ground truth;
- •malicious evidence injection; or
- •changes in model or provider behaviour\.

Replication across models, domains, output contracts, evidence sources, and recovery architectures is therefore required before broader generalisation\.

### 7\.4Conclusion validity

The paired bootstrap intervals characterise uncertainty across the selected sanitised base cases under the implemented model configuration, matching procedure, and scoring contract\.

All nine operational stage\-success intervals remained below zero\. In contrast, three parser\-validity intervals—the partial\-dropout comparisons at decision, audit, and escalation—included zero\.

The evidence therefore supports consistent negative changes in operational stage success within the evaluated design\. It does not support a claim of uniformly interval\-separated parser\-validity improvement under every degraded condition\.

The bootstrap intervals should not be interpreted as population\-level confidence intervals for deployed financial systems, consumers, organisations, or model providers\. They describe variability across the selected matched cases rather than sampling uncertainty from a defined real\-world population\.

The aggregate cascade\-failure rate also requires separate interpretation\. Because it includes clean\-condition failures, it is not:

- •a pure degradation effect;
- •a deployment failure probability;
- •an estimate of consumer risk; or
- •a prevalence estimate for real\-world AI systems\.

The paired condition\-stage comparisons provide the stronger basis for interpreting changes associated with controlled evidence degradation\. The cascade statistic serves a different purpose by describing the sequence\-level outcome families observed in the experimental run\.

### 7\.5Measurement coverage

The experiment did not directly measure:

- •latency;
- •condition\-specific cost variation;
- •calibration;
- •confidence quality;
- •human interpretability;
- •explanation faithfulness;
- •downstream user reliance;
- •user harm;
- •institutional decision quality;
- •alternative retrieval policies;
- •safe\-deferral quality;
- •human escalation performance; or
- •competing recovery architectures\.

These properties may interact with Evidence\-State Reliability but remain distinct evaluation targets\.

The three controlled degradation conditions also should not be interpreted as complete reproductions of naturally occurring compression, omission, or conflict processes\. The experiment establishes behaviour under defined interventions; it does not establish how frequently or severely equivalent evidence failures occur in deployed systems\.

Similarly, the zero\-recovery result applies only to the implemented escalation mechanism\. The experiment did not test whether additional retrieval, another model, a revised policy, or a human reviewer would have restored successful operation\.

### 7\.6Reproducibility limitations

The committed sanitised artifacts support deterministic verification of execution and retention counts, parser accounting, metric calculations, condition\-stage tables, matched comparisons, bootstrap intervals, cascade classifications, figure source values, and claim\-to\-evidence mappings\.

However, raw prompts, raw model responses, raw CFPB records, JSONL execution archives, API credentials, and environment files were excluded from the reporting package by design\.

Exact prompt\-response replay and complete reconstruction of every original model interaction are therefore not claimed\. The available artifacts support analysis\-level reproducibility and auditability rather than unrestricted computational replication\.

A future replication using the same model identifier may also produce different outputs if the hosted model, provider infrastructure, or inference environment changes over time\.

Finally, the audit stage evaluated here is computational rather than institutional\. The observed zero recovery rate applies to the implemented escalation mechanism and does not establish that recovery is impossible under alternative retrieval, deferral, human\-review, or decision\-authority arrangements\.

## 8Ethical and Data\-Use Considerations

The study used sanitised derivative evidence packets constructed from the public context of the Consumer Financial Protection Bureau Consumer Complaint Database\. These materials were used solely to evaluate a controlled reliability mechanism within a multi\-stage language\-model pipeline\.

The experiment did not evaluate live consumer decisions and did not produce actual credit, lending, regulatory, legal, or financial determinations\. No experimental output was used to affect an individual, organisation, account, complaint, or institutional process\.

CFPB provenance was used to provide a realistic complaint\-context substrate\. It was not used to establish:

- •whether an individual complaint was factually correct;
- •whether a named company engaged in misconduct;
- •whether a legal or regulatory violation occurred;
- •whether a consumer experienced harm;
- •whether a financial decision was appropriate; or
- •how frequently an observed failure pattern would occur in deployment\.

Consumer complaints are therefore not treated as adjudicated findings or as a representative sample of consumer experience\. The study does not draw conclusions about individual consumers, specific organisations, market prevalence, regulatory compliance, or the safety of deployed financial systems\.

The shared research boundary excluded:

- •raw CFPB records;
- •direct personal identifiers;
- •raw prompts;
- •raw model responses;
- •JSONL execution archives;
- •API credentials; and
- •environment files\.

Only sanitised, derived, and aggregate research artifacts were retained within the reported evidence package\.

The decision, audit, and escalation outputs analysed in this study are computational and model\-output\-coded\. They do not constitute professional financial advice, institutional auditing, regulatory assurance, or human expert adjudication\.

The findings should therefore be interpreted strictly as evidence about the behaviour of the evaluated experimental pipeline under controlled evidence\-state interventions\. They should not be used to make decisions or claims concerning individual consumers, named organisations, actual complaints, or the safety and compliance of deployed financial systems\.

## 9Reproducibility and Availability

The code, sanitised evidence\-state contracts, execution accounting, sanitised execution summaries, condition\-stage tables, matched\-comparison outputs, bootstrap intervals, audit and escalation metrics, cascade\-sequence classifications, figure source data, validation reports, and claim\-traceability materials are available in the project repository:

The principal empirical analysis was generated from the following repository checkpoint:

c3e802c71976faae34ac3f327b537e12916bc970

The final manuscript and submission\-gate materials were prepared at the following repository checkpoint:

3144d46e3c59b011f2939d7ab3a590c688543492

Reporting both checkpoints distinguishes the evidence state used for the principal analysis from the later repository state used for manuscript preparation and submission validation\.

The shared artifacts support deterministic verification of:

- •planned, ledgered, and retained execution counts;
- •parser\-valid and parser\-invalid accounting;
- •metric definitions and formulas;
- •condition\-stage estimates;
- •matched degraded\-minus\-clean comparisons;
- •bootstrap confidence intervals;
- •audit\-detection and false\-assurance results;
- •escalation\-recovery results;
- •cascade classifications;
- •figure source values; and
- •mappings between reported claims and supporting artifacts\.

The repository does not provide unrestricted reconstruction of every original model interaction\. Raw CFPB records, raw prompts, raw model responses, JSONL execution archives, API credentials, and environment files are excluded from the shared reporting boundary\.

Accordingly, the research package supports artifact\-level reproducibility, traceability, and independent verification of the reported analysis\. It does not support exact prompt\-response replay or complete computational replication of the original hosted\-model execution\.

## 10Conclusion

This study introduced and operationalised Evidence\-State Reliability as a distinct evaluation layer for multi\-stage LLM pipelines\. The framework evaluates structural parser validity alongside evidence\-sensitive stage success, controlled evidence\-state interventions, matched uncertainty analysis, audit detection, false assurance, escalation recovery, and sequence\-level reliability patterns\.

The framework was evaluated in one scaled GLM\-5\.2 experiment using 60 sanitised base cases represented under clean, compressed\-lossy, partial\-dropout, and noisy\-conflicting evidence conditions\. Each condition was processed through decision, audit, and escalation stages, producing 720 planned and ledgered model calls and 713 retained sanitised execution rows\.

The primary empirical finding was a consistent divergence between structural and evidence\-sensitive measurements\. Across all nine matched degraded\-minus\-clean condition\-stage comparisons, operational stage\-success estimates were negative and all corresponding 95% bootstrap intervals remained below zero\. All nine parser\-validity point estimates were positive, although the three partial\-dropout parser\-validity intervals included zero\. Within the evaluated pipeline, structural conformance could therefore move in a favourable direction while evidence\-sensitive stage success deteriorated under the same controlled intervention\.

The audit and escalation results revealed a second distinction\. Among parser\-valid degraded audit outputs, the degradation\-detection indicator was 1\.0 in each degraded condition, while audit false\-assurance rates remained non\-zero\. Among parser\-valid degraded escalation outputs, recovery was 0\.0 in every degraded condition\. Detection of a degraded evidence state therefore did not imply restoration of successful operation under the implemented mechanism\.

At the sequence level, 223 of 240 condition\-linked groups met the operational cascade\-failure definition\. The corresponding taxonomy separated parser failure, detected\-but\-unrecovered degradation, audit false assurance, incomplete execution, and preserved success\. This sequence\-level view complements the paired intervention analysis by distinguishing failure families that imply different technical responses\.

Taken together, the findings support the central argument of the paper: parser validity is necessary for automated pipeline operation but is insufficient as a proxy for Evidence\-State Reliability\. Evaluating whether an output satisfies a structural contract does not establish that the evidence available to support the substantive function of a pipeline stage remains adequate\.

The contribution is therefore a bounded methodological and empirical operationalisation of controlled evidence\-state intervention, stage\-aware reliability measurement, parser\-versus\-evidence divergence, separation of audit detection from recovery, matched uncertainty analysis, and sequence\-level reliability diagnosis\.

These conclusions remain limited to one model configuration, one pipeline design, one scaled execution run, 60 selected sanitised cases, three controlled degradation families, and model\-output\-coded scoring\. Cross\-model, multi\-run, multi\-domain, and independently adjudicated replication is required before broader claims concerning generality, deployment reliability, or institutional assurance can be supported\.

## References

- Atanasovaet al\.\(2022\)P\. Atanasova, J\. G\. Simonsen, C\. Lioma, and I\. AugensteinFact checking with insufficient evidence\.Transactions of the Association for Computational Linguistics10,pp\. 746–763\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00486),[Link](https://aclanthology.org/2022.tacl-1.43/)Cited by:[§2\.2](https://arxiv.org/html/2608.21559#S2.SS2.p3.1)\.
- Consumer Financial Protection Bureau \(2026\)Consumer Financial Protection BureauConsumer complaint database\.Note:Living database; page last modified 7 August 2026; accessed 21 August 2026External Links:[Link](https://www.consumerfinance.gov/data-research/consumer-complaints/)Cited by:[§4\.3](https://arxiv.org/html/2608.21559#S4.SS3.p1.1)\.
- Eset al\.\(2024\)S\. Es, J\. James, L\. Espinosa Anke, and S\. SchockaertRAGAs: automated evaluation of retrieval augmented generation\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations,St\. Julians, Malta,pp\. 150–158\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.eacl-demo.16),[Link](https://aclanthology.org/2024.eacl-demo.16/)Cited by:[§2\.1](https://arxiv.org/html/2608.21559#S2.SS1.p2.1),[§2\.3](https://arxiv.org/html/2608.21559#S2.SS3.p1.1)\.
- Geifman and El\-Yaniv \(2019\)Y\. Geifman and R\. El\-YanivSelectiveNet: a deep neural network with an integrated reject option\.InProceedings of the 36th International Conference on Machine Learning,K\. Chaudhuri and R\. Salakhutdinov \(Eds\.\),Proceedings of Machine Learning Research, Vol\.97,pp\. 2151–2159\.External Links:[Link](https://proceedings.mlr.press/v97/geifman19a.html)Cited by:[§2\.3](https://arxiv.org/html/2608.21559#S2.SS3.p3.1)\.
- Genget al\.\(2023\)S\. Geng, M\. Josifoski, M\. Peyrard, and R\. WestGrammar\-constrained decoding for structured NLP tasks without finetuning\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 10932–10952\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.674),[Link](https://aclanthology.org/2023.emnlp-main.674/)Cited by:[§2\.2](https://arxiv.org/html/2608.21559#S2.SS2.p1.1)\.
- Guoet al\.\(2017\)C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. WeinbergerOn calibration of modern neural networks\.InProceedings of the 34th International Conference on Machine Learning,D\. Precup and Y\. W\. Teh \(Eds\.\),Proceedings of Machine Learning Research, Vol\.70,pp\. 1321–1330\.External Links:[Link](https://proceedings.mlr.press/v70/guo17a.html)Cited by:[§2\.1](https://arxiv.org/html/2608.21559#S2.SS1.p1.1)\.
- Jamshidiet al\.\(2026\)S\. Jamshidi, A\. Moradi Dakhel, K\. W\. Nafi, and F\. KhomhHallucination cascade: analyzing error propagation in multi\-agent LLM systems\.Note:arXiv:2606\.07937 \[cs\.CR\]External Links:2606\.07937,[Document](https://dx.doi.org/10.48550/arXiv.2606.07937),[Link](https://arxiv.org/abs/2606.07937)Cited by:[§2\.4](https://arxiv.org/html/2608.21559#S2.SS4.p1.1)\.
- Liuet al\.\(2024\)X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang, S\. Zhang, X\. Deng, A\. Zeng, Z\. Du, C\. Zhang, S\. Shen, T\. Zhang, Y\. Su, H\. Sun, M\. Huang, Y\. Dong, and J\. TangAgentBench: evaluating LLMs as agents\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/e9df36b21ff4ee211a8b71ee8b7e9f57-Abstract-Conference.html)Cited by:[§2\.3](https://arxiv.org/html/2608.21559#S2.SS3.p1.1)\.
- Motter and Lai \(2002\)A\. E\. Motter and Y\. LaiCascade\-based attacks on complex networks\.Physical Review E66\(6\),pp\. 065102\.External Links:[Document](https://dx.doi.org/10.1103/PhysRevE.66.065102),[Link](https://journals.aps.org/pre/abstract/10.1103/PhysRevE.66.065102)Cited by:[§2\.4](https://arxiv.org/html/2608.21559#S2.SS4.p1.1)\.
- Mozannar and Sontag \(2020\)H\. Mozannar and D\. SontagConsistent estimators for learning to defer to an expert\.InProceedings of the 37th International Conference on Machine Learning,H\. Daumé III and A\. Singh \(Eds\.\),Proceedings of Machine Learning Research, Vol\.119,pp\. 7076–7087\.External Links:[Link](https://proceedings.mlr.press/v119/mozannar20b.html)Cited by:[§2\.3](https://arxiv.org/html/2608.21559#S2.SS3.p3.1)\.
- Pineauet al\.\(2021\)J\. Pineau, P\. Vincent\-Lamarre, K\. Sinha, V\. Larivière, A\. Beygelzimer, F\. d’Alché\-Buc, E\. Fox, and H\. LarochelleImproving reproducibility in machine learning research \(a report from the NeurIPS 2019 reproducibility program\)\.Journal of Machine Learning Research22\(164\),pp\. 1–20\.External Links:[Link](https://jmlr.org/papers/v22/20-303.html)Cited by:[§4\.8](https://arxiv.org/html/2608.21559#S4.SS8.p3.1)\.
- Qiuet al\.\(2026\)J\. Qiu, Z\. Han, and C\. HuangSURE\-RAG: sufficiency and uncertainty\-aware evidence verification for selective retrieval\-augmented generation\.Note:arXiv:2605\.03534 \[cs\.CL\]External Links:2605\.03534,[Document](https://dx.doi.org/10.48550/arXiv.2605.03534),[Link](https://arxiv.org/abs/2605.03534)Cited by:[§2\.2](https://arxiv.org/html/2608.21559#S2.SS2.p3.1)\.
- Rajiet al\.\(2020\)I\. D\. Raji, A\. Smart, R\. N\. White, M\. Mitchell, T\. Gebru, B\. Hutchinson, J\. Smith\-Loud, D\. Theron, and P\. BarnesClosing the AI accountability gap: defining an end\-to\-end framework for internal algorithmic auditing\.InProceedings of the 2020 Conference on Fairness, Accountability, and Transparency,pp\. 33–44\.External Links:[Document](https://dx.doi.org/10.1145/3351095.3372873),[Link](https://doi.org/10.1145/3351095.3372873)Cited by:[§2\.3](https://arxiv.org/html/2608.21559#S2.SS3.p2.1)\.
- Ray \(2026\)J\. RayThe constraint tax: measuring validity\-correctness tradeoffs in structured outputs for small language models\.Note:arXiv:2605\.26128 \[cs\.LG\]External Links:2605\.26128,[Document](https://dx.doi.org/10.48550/arXiv.2605.26128),[Link](https://arxiv.org/abs/2605.26128)Cited by:[§2\.2](https://arxiv.org/html/2608.21559#S2.SS2.p1.1)\.
- Ribeiroet al\.\(2020\)M\. T\. Ribeiro, T\. Wu, C\. Guestrin, and S\. SinghBeyond accuracy: behavioral testing of NLP models with CheckList\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,Online,pp\. 4902–4912\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.442),[Link](https://aclanthology.org/2020.acl-main.442/)Cited by:[§2\.1](https://arxiv.org/html/2608.21559#S2.SS1.p1.1)\.
- Saad\-Falconet al\.\(2024\)J\. Saad\-Falcon, O\. Khattab, C\. Potts, and M\. ZahariaARES: an automated evaluation framework for retrieval\-augmented generation systems\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),Mexico City, Mexico,pp\. 338–354\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.20),[Link](https://aclanthology.org/2024.naacl-long.20/)Cited by:[§2\.1](https://arxiv.org/html/2608.21559#S2.SS1.p2.1),[§2\.3](https://arxiv.org/html/2608.21559#S2.SS3.p1.1)\.
- Singhet al\.\(2026\)A\. K\. Singh, H\. V\. Khurdula, Y\. D\. Khemlani, and V\. AgarwalThe structured output benchmark: a multi\-source benchmark for evaluating structured output quality in large language models\.Note:arXiv:2604\.25359 \[cs\.CL\]External Links:2604\.25359,[Document](https://dx.doi.org/10.48550/arXiv.2604.25359),[Link](https://arxiv.org/abs/2604.25359)Cited by:[§2\.2](https://arxiv.org/html/2608.21559#S2.SS2.p1.1)\.
- Solozobov \(2026\)O\. SolozobovEvidence sufficiency under delayed ground truth: proxy monitoring for risk decision systems\.Note:arXiv:2604\.15740 \[cs\.CY\]External Links:2604\.15740,[Document](https://dx.doi.org/10.48550/arXiv.2604.15740),[Link](https://arxiv.org/abs/2604.15740)Cited by:[§2\.2](https://arxiv.org/html/2608.21559#S2.SS2.p3.1)\.
- Wohlinet al\.\(2012\)C\. Wohlin, P\. Runeson, M\. Höst, M\. C\. Ohlsson, B\. Regnell, and A\. WesslénExperimentation in software engineering\.Springer Berlin, Heidelberg\.External Links:[Document](https://dx.doi.org/10.1007/978-3-642-29044-2),[Link](https://link.springer.com/book/10.1007/978-3-642-29044-2)Cited by:[§7](https://arxiv.org/html/2608.21559#S7.p1.1)\.
- Xieet al\.\(2026\)Y\. Xie, C\. Zhu, X\. Zhang, T\. Zhu, D\. Ye, M\. Qi, H\. Chen, and W\. ZhouFrom spark to fire: modeling and mitigating error cascades in LLM\-based multi\-agent collaboration\.Note:arXiv:2603\.04474 \[cs\.MA\]External Links:2603\.04474,[Document](https://dx.doi.org/10.48550/arXiv.2603.04474),[Link](https://arxiv.org/abs/2603.04474)Cited by:[§2\.4](https://arxiv.org/html/2608.21559#S2.SS4.p1.1)\.