Tag
An empirical arXiv study examines /retro, a skill that lets coding agents review completed sessions and propose harness changes, finding that while many proposals are generated, only some reduce repeated work on later tasks — showing harness self-evolution needs mechanisms to verify and revise changes.
An empirical study finds that continual harness self-evolution of coding agents across multiple software engineering tasks causes severe catastrophic forgetting, with SWE-Bench performance dropping from 94.0% to 32.0% after evolving on FeatureBench, and offers insights for future continual agent-harness evolution.
该研究探讨 LLM 如何结合用户的言语承诺（words）与历史决策记录（deeds）来预测用户在人机对话中的合作/背叛决策，发现 LLM 高度依赖声明的意图但几乎不受对手智能体人格特征影响，导致对被违背的承诺预测准确率大幅下降。
本文对 LLM 智能体在执行 Web 与计算机任务时的错误及自我恢复能力进行了实证研究，构建了包含 1,259 条轨迹、1,403 个错误事件的 ReInBench 数据集，并提出 ReCoF 方法来预测错误是否可自恢复，从而决定是否需要外部干预。
这篇实证系统经验报告分析了小型开源权重编码模型在真实编码工作中的表现，指出成功不仅取决于文本连贯性，还依赖于服务配置、协议设计和隐藏行为验证。研究基于4,165个网关轮次和多项委托实验，发现直接编码通常比委托给量化小模型具有更高的验证通过率和更低的每次成功成本。
The paper analyzes why most RL post-training methods for language models never make it into open-source frameworks, tracing seven frameworks against 6,939 arXiv papers and finding only 37 of 1,598 proposed methods were adopted, while releasing a tracker dataset linking training failures to framework implementations.
An empirical study from Nanjing University investigating whether integrating faster, lower-cost decision models (Jev by TypeSafe AI) into LLM agent workflows actually improves task-level efficiency and reliability, evaluating five agents across ALFWorld and AndroidWorld with 4,500 task executions.
An empirical study crashing four production AI agent codebases (Cua, Skyvern, Corsair, Inconvo) mid-action and measuring duplicate side effects on retry, finding a shared root cause: none of them records which actions already completed, so retries re-execute work from an empty history.
An empirical post-mortem of 109 AI agent security incidents finds that 38% of container breakouts exploited trivial configuration mistakes (e.g., mounted docker.sock, leaked env vars, missing taint tracking) rather than kernel 0-days. The authors release an open dataset and an open-source Multi-Agent Supervisor Security Harness with taint propagation and execution boundary controls.
该论文对12款主流的EU AI Act合规检查工具（AIACCs）进行了首次实证评估，发现这些工具质量参差不齐，只能作为早期参考，过度简化关键义务且无法提供确定、可执行的指导，可能造成虚假的合规感。
This paper empirically studies Jev, a decision-oriented "System One Model," for personalized recommendation reranking, finding it occupies a distinct quality–latency operating regime compared with recommendation-specific models and pointwise/listwise LLM rerankers across multiple domains and candidate-set sizes.
This paper finds that latent world action models fail to generalize and proposes Simple-WAM, which improves generalization performance while maintaining efficiency by simplifying future modeling into a single forward pass.
This paper studies developer productivity at Google, finding that code quality and other factors causally affect productivity, with code quality showing a strong impact through panel data analysis.
This article summarizes an empirical study on harness design for coding agents, showing that context management prevents overflow failures, rule-based elision before LLM summarization is cost-effective, and planning's role varies with model strength.
A user thanks AK for sharing their paper titled 'An Empirical Study of Harness Design for Coding Agents', which focuses on AI research in software engineering.
This paper presents a unified empirical evaluation of methods for revising travel itineraries under resource disruptions, comparing full replanning, plan repair, and LLM-based revision approaches in terms of effectiveness, plan stability, and computational cost.
The article introduces or discusses the concept of a scaling law in robotics, which may provide insights for advancing robotic systems and AI research.
This paper empirically studies harness design for coding agents, evaluating components like planning and context management to improve performance in software engineering tasks.
This paper challenges the distortion hypothesis for few-shot degradation in language models by introducing a random-text control, showing that representation shift is largely due to prompt length, and models with higher content delta benefit more from few-shot prompting.
This paper presents the first comprehensive empirical study on multi-task learning for predictive process monitoring, showing improvements over single-task learning in next-activity prediction and class imbalance mitigation.