HSPR: Recovering Software-Agent Benchmark Scores via Hierarchical Posterior Inference

Hugging Face Daily Papers Papers
agent-evaluation benchmark bayesian-inference software-agents active-learning swe-bench

Summary

该论文提出层级分数后验恢复方法(HSPR)，通过结合历史响应模式与自适应任务选择，仅用小子集即可高效估计软件代理在完整基准测试上的分数，在SWE-bench系列任务库上取得最低的平均与最大绝对误差。

Evaluating software agents on a full benchmark is expensive. A small task subset can preserve agent rankings while yielding inaccurate scores. Historical responses provide useful structure, but tasks with identical source responses can have different outcomes for a new target agent. We introduce Hierarchical Score Posterior Recovery (HSPR), which combines source-fitted response factors with uncertain group success rates. Observed target outcomes update the posteriors over the target latent state and group rates; posterior-guided acquisition selects subsequent tasks. Its score-based criterion equals the expected one-step reduction in full-score posterior variance. Observed successes and posterior-predictive counts for unqueried tasks yield a full-score posterior. The final estimate uses its median with source-selected calibration. At a 10% task budget on four task banks from the SWE-bench family, HSPR achieves the lowest mean absolute error (MAE) on three banks and the lowest maximum absolute error (MaxErr) on all four among the methods compared. Removing group uncertainty or replacing adaptive queries increases macro MAE in the respective controls. Macro MAE decreases across 5-20% budgets.
Original Article
View Cached Full Text

Cached at: 10/20/26, 07:34 AM

Paper page - HSPR: Recovering Software-Agent Benchmark Scores via Hierarchical Posterior Inference

Source: https://huggingface.co/papers/2610.24611

Abstract

Hierarchical Score Posterior Recovery improves agent evaluation efficiency by combining historical response patterns with adaptive task selection to estimate full benchmark scores from small subsets.

Evaluating software agents on a full benchmark is expensive. A small task subset can preserve agent rankings while yielding inaccurate scores. Historical responses provide useful structure, but tasks with identical source responses can have different outcomes for a new target agent. We introduceHierarchical Score Posterior Recovery(HSPR), which combines source-fittedresponse factorswith uncertaingroup success rates. Observed target outcomes update the posteriors over the target latent state and group rates;posterior-guided acquisitionselects subsequent tasks. Itsscore-based criterionequals the expected one-step reduction in full-scoreposterior variance. Observed successes andposterior-predictive countsfor unqueried tasks yield a full-score posterior. The final estimate uses its median withsource-selected calibration. At a 10% task budget on four task banks from theSWE-benchfamily, HSPR achieves the lowest mean absolute error (MAE) on three banks and the lowest maximum absolute error (MaxErr) on all four among the methods compared. Removing group uncertainty or replacing adaptive queries increases macro MAE in the respective controls. Macro MAE decreases across 5-20% budgets.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2610\.24611

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.24611 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.24611 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.24611 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

SearchHorizonBench: A Benchmark for Search Value in AI Agents

Hugging Face Daily Papers

论文提出 SearchHorizonBench，一个用于评估 AI 智能体从执行经验中估计搜索价值的基准数据集，包含 45,632 次执行记录，发现历史校准可将 Brier 误差降低 16.2%，但历史估计器仅能解释不到 2% 的搜索增益方差。

Benchmarking and Enhancing Agentic Causal Discovery

arXiv cs.LG

论文提出 Heuresis，一个使用匿名模拟数据的因果发现基准，用于评估 LLM 智能体在数据驱动下进行细粒度、证据驱动的因果发现调整能力。结果显示 GPT-6.1 Sol 在全部六个数据族上超过最佳算法选择，最难族上邻接 F1 提升 14%。

Historical-Feature Residual Adaptation for Sequential Screening

arXiv cs.LG

该论文提出 acquisition-time residual adaptation 方法，在固定实验预算下保留可复用的基础打分器并学习排名修正，使 20 个测试筛选任务的命中回收率从 22.58% 提升至 28.61%（相对提升 26.7%），并优于多种对照方法。

Benchmarking Task Takeover Across Agent Handoffs

arXiv cs.CL

论文提出 Handoff State Continuity Benchmark (HSC-Bench)，包含 1,039 个任务，用于评估 AI 代理在交接时的任务接管能力，涵盖仓库证据恢复与从既有对话、工具和文件状态继续工作的两个方面，并对比了六个接收模型的表现。