编码代理能否重现官方统计数据？元数据、重试预算与执行反馈的局限性：一项受控Eurostat基准测试

arXiv cs.LG 论文
coding-agents large-language-models reproducibility statistics benchmark execution-feedback eurostat

摘要

本研究以编码代理重现Eurostat统计数据为基准进行评估，发现语义验证和重试预算对可靠性至关重要，远不止于执行诊断。

arXiv:2609.22222v1 Announce Type: new Abstract: Large language models can generate executable data-analysis code, but successful execution is not equivalent to a valid official-statistics result. This study asks whether authoritative metadata and execution feedback improve the reproducibility of Eurostat answers produced by a coding agent, and isolates what execution feedback actually contributes. A benchmark of 30 natural-language tasks covering seven domains, seven Eurostat datasets and four difficulty tiers was run under four conditions: task only (A), task plus a frozen dataset metadata card (B), metadata plus a repair loop driven by sanitized execution feedback (C), and metadata plus the same attempt budget with no diagnostics of any kind (D). Claude Sonnet 5 generated Python through the Anthropic Messages API in three independent replicates, yielding 360 task-runs. Exact correctness required successful execution, the correct dataset, filters, output shape, values and unit. A companion experiment run under an under-specified output contract, in which the required ranking key and unit representation were never stated to the model, understated condition C by 23.4 points, showing that evaluator and contract design can dominate measured agent error. Reliable statistical coding agents need semantic validation against frozen specifications, a fully specified output contract, and a retry budget - not execution diagnostics.
查看原文
查看缓存全文

缓存时间: 2026/09/22 09:20 

# Can Coding Agents Reproduce Official Statistics? Metadata, Retry Budget and the Limits of Execution Feedback in a Controlled Eurostat Benchmark
Source: [https://arxiv.org/abs/2609.22222](https://arxiv.org/abs/2609.22222)
[View PDF](https://arxiv.org/pdf/2609.22222)

> Abstract:Large language models can generate executable data\-analysis code, but successful execution is not equivalent to a valid official\-statistics result\. This study asks whether authoritative metadata and execution feedback improve the reproducibility of Eurostat answers produced by a coding agent, and isolates what execution feedback actually contributes\. A benchmark of 30 natural\-language tasks covering seven domains, seven Eurostat datasets and four difficulty tiers was run under four conditions: task only \(A\), task plus a frozen dataset metadata card \(B\), metadata plus a repair loop driven by sanitized execution feedback \(C\), and metadata plus the same attempt budget with no diagnostics of any kind \(D\)\. Claude Sonnet 5 generated Python through the Anthropic Messages API in three independent replicates, yielding 360 task\-runs\. Exact correctness required successful execution, the correct dataset, filters, output shape, values and unit\. A companion experiment run under an under\-specified output contract, in which the required ranking key and unit representation were never stated to the model, understated condition C by 23\.4 points, showing that evaluator and contract design can dominate measured agent error\. Reliable statistical coding agents need semantic validation against frozen specifications, a fully specified output contract, and a retry budget \- not execution diagnostics\.

## Submission history

From: Sabina\-Cristiana Necula \[[view email](https://arxiv.org/show-email/f09cef7b/2609.22222)\] **\[v1\]**Wed, 2 Sep 2026 13:21:14 UTC \(305 KB\)

相似文章

性能优化基准是否可靠地衡量编码代理？

Hugging Face Daily Papers

本文审计了针对编码代理的三个性能优化基准（GSO、SWE-Perf、SWE-efficiency），发现运行时不稳定、评分规则和任务覆盖率显著影响可靠性，并且许多任务已经至少有一个公开提交解决了。

AI编程代理可复现社会科学发现

arXiv cs.CL

本文介绍了SocSci-Repro-Bench，这是一个包含221个任务的基准测试，用于评估AI编程代理从原始数据和代码中复现社会科学发现的能力。研究发现，像Claude Code和Codex这样的前沿代理可以复现大部分结果，其中Claude明显优于Codex，并且结果并非主要由记忆驱动。