Can Coding Agents Reproduce Official Statistics? Metadata, Retry Budget and the Limits of Execution Feedback in a Controlled Eurostat Benchmark
Summary
This study benchmarks coding agents on reproducing Eurostat statistics, finding that semantic validation and a retry budget are crucial for reliability, not just execution diagnostics.
View Cached Full Text
Cached at: 09/22/26, 09:20 AM
# Can Coding Agents Reproduce Official Statistics? Metadata, Retry Budget and the Limits of Execution Feedback in a Controlled Eurostat Benchmark Source: [https://arxiv.org/abs/2609.22222](https://arxiv.org/abs/2609.22222) [View PDF](https://arxiv.org/pdf/2609.22222) > Abstract:Large language models can generate executable data\-analysis code, but successful execution is not equivalent to a valid official\-statistics result\. This study asks whether authoritative metadata and execution feedback improve the reproducibility of Eurostat answers produced by a coding agent, and isolates what execution feedback actually contributes\. A benchmark of 30 natural\-language tasks covering seven domains, seven Eurostat datasets and four difficulty tiers was run under four conditions: task only \(A\), task plus a frozen dataset metadata card \(B\), metadata plus a repair loop driven by sanitized execution feedback \(C\), and metadata plus the same attempt budget with no diagnostics of any kind \(D\)\. Claude Sonnet 5 generated Python through the Anthropic Messages API in three independent replicates, yielding 360 task\-runs\. Exact correctness required successful execution, the correct dataset, filters, output shape, values and unit\. A companion experiment run under an under\-specified output contract, in which the required ranking key and unit representation were never stated to the model, understated condition C by 23\.4 points, showing that evaluator and contract design can dominate measured agent error\. Reliable statistical coding agents need semantic validation against frozen specifications, a fully specified output contract, and a retry budget \- not execution diagnostics\. ## Submission history From: Sabina\-Cristiana Necula \[[view email](https://arxiv.org/show-email/f09cef7b/2609.22222)\] **\[v1\]**Wed, 2 Sep 2026 13:21:14 UTC \(305 KB\)
Similar Articles
Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?
This paper audits three performance-optimization benchmarks (GSO, SWE-Perf, SWE-efficiency) for coding agents, finding that runtime instability, scoring rules, and task coverage significantly affect reliability, and that many tasks are already solved by at least one public submission.
AI Coding Agents Can Reproduce Social Science Findings
This paper introduces SocSci-Repro-Bench, a benchmark of 221 tasks to evaluate AI coding agents' ability to reproduce social science findings from original data and code. It finds that frontier agents like Claude Code and Codex can reproduce a large share of results, with Claude substantially outperforming Codex, and that results are not primarily driven by memorization.
EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions
Introduces EvoCode-Bench, a benchmark of 26 stateful coding tasks across 227 rounds that evaluates coding agents in multi-turn iterative interactions, revealing that single-round performance overestimates multi-round capabilities by 22–40 points.
REAP: Automatic Curation of Coding Agent Benchmarks from Interactive Production Usage [R]
REAP is an automated pipeline that curates production-derived benchmarks for coding agents from real developer-agent sessions, using LLM-based classification and stability checks to ensure reliable evaluation without manual labeling.
Are coding agents getting expensive, or are we measuring cost the wrong way?
The article questions whether the real cost of coding agents includes hidden human oversight and debugging, arguing that true value should be measured by trusted output rather than raw token consumption.