Reproducibility is not construct validity: LLM measurement of institutionally situated communication
Summary
This paper tests the distinction between reproducibility and construct validity in LLM annotations, using data from the European Commission's AI Act consultation to demonstrate high reproducibility but limited convergence with survey measures, with systematic variations across stakeholder groups.
View Cached Full Text
Cached at: 09/18/26, 09:26 AM
# Reproducibility is not construct validity: LLM measurement of institutionally situated communication
Source: [https://arxiv.org/abs/2609.19866](https://arxiv.org/abs/2609.19866)
[View PDF](https://arxiv.org/pdf/2609.19866)
> Abstract:High annotation reproducibility does not necessarily imply that an LLM\-inferred measure captures the construct it is intended to measure\. We test this distinction using a dataset from the European Commission's AI Act consultation, linking structured survey responses to free\-text consultation submissions from the same stakeholders\. LLM annotations of consultation submissions are highly reproducible \(intraclass correlations \> 0\.99\), yet show limited convergence with survey\-reported measures of the nominal construct they were intended to approximate\. Divergence between survey\-and LLM\-inferred text\-based measures varies systematically across stakeholder groups: business associations express greater concern about AI risks in text\-based consultations than in survey responses \(\{g\} = \+1\.0\), whereas public authorities and several nonbusiness groups show smaller or negative divergences\. Divergences between scores suggest positive spatial autocorrelation across European countries \(Moran's I = 0\.347, p = 0\.036\), indicating that stakeholders from neighboring countries tend toward more similar text\-based stances towards AI safety concerns\. Despite divergence, survey\-reported concerns remain strongly associated with support for explainability across all divergence levels\. These results demonstrate that LLM annotation reproducibility can coexist with poor construct correspondence and motivate validation procedures that distinguish reproducibility, construct validity, and communication context variation when LLMs are used as measurement instruments\.
## Submission history
From: Carlo Romano Marcello Alessandro Santagiustina \[[view email](https://arxiv.org/show-email/a1ec6f39/2609.19866)\] \[via CCSD proxy\] **\[v1\]**Thu, 17 Sep 2026 08:19:58 UTC \(1,675 KB\)Similar Articles
How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure
This paper audits the reproducibility of LLM evaluation conclusions, finding that small-sample evaluations are unstable and can mislead, and it recommends reporting additional metrics for reliability.
Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructs
This paper examines the gap between reliability and construct validity when using LLMs as coding instruments for theoretical constructs, and proposes grain calibration as a method to decompose constructs into clause-level components for more valid measurement.
Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents
This paper presents a psychometric audit of large language models as synthetic survey respondents, finding that they fail to match human joint distributions and reliability, making them unsuitable replacements.
Validity of LLMs as data annotators: AMALIA on authority
This paper examines the validity of using LLMs as data annotators, focusing on Portugal's AMALIA-9B model for European Portuguese. It finds that while the model shows agreement with human coders, its reliance on surface correlates raises concerns about construct validity, with implications for sovereign LLM programs.
When Consistency Does Not Mean Reliability: Evaluating Local LLM Judges Against Human Ratings
The paper evaluates local open-weight LLM judges against human ratings, finding high self-consistency but limited agreement with human judgments, highlighting the need for dual assessment.