Tag
该论文提出 ExecutionRAG，一种无需训练的需求寄存器机制，用于分析多跳 RAG 中 LLM 未能检索到证据的原因，发现许多检索失败实为搜索漂移，并指出剩余瓶颈从搜索引导转向证据充分性验证。
We present ToolBottleneckBench, a benchmark measuring tool-agent performance under controlled pressures (tool catalog, visible menu, certified workflow, record encoding) and fixed harness budgets, finding that only the visible tool menu heavily penalizes models — largely via truncation — and most failures end at harness limits.
OmicsBench introduces a long-horizon, multi-turn benchmark of 97 DAG-structured bioinformatics tasks (662 subtasks) for evaluating AI agents over research-study-scale workflows, revealing significant performance gaps and introducing an unsupervised failure mode discovery pipeline.
RoboQuest introduces a benchmark for goal-directed embodied exploration in mobile manipulation, testing frontier multimodal agents on search, inspection, and interactive testing tasks. Results show the best agent succeeds in only 23% of episodes, with failures mainly attributed to premature stopping of exploration rather than execution skills.
This paper presents a differential analysis framework that attributes skill-induced failures and efficiency regressions in LLM agents, introducing SkillTriage and finding that seemingly relevant skills are the dominant cause of functional failures and excessive verification/procedure overhead.
A DocInsights 2026 shared-task system paper describing EviCalc, a PDF document QA pipeline that pairs evidence passage selection with LLM-generated arithmetic expressions evaluated in local code, reporting 8.61% joint accuracy on DocSem's test set along with descriptive failure analysis, including OCR and block-grouping errors that merged relevant passages.
论文提出 Agent Error Dataset（AED），包含 50,228 个错误-诊断对，覆盖 33 个环境与 23 个策略模型，并通过 AET 流水线用于失败分析与错误感知的后训练，显著提升修正通过率与诊断准确率。
Big Mess O Wires published a detailed failure analysis of 70 QA-failing Floppy Emu boards, breaking down root causes: damaged clock crystals (42 boards), fragile Xilinx CPLD chips (6), unprogrammed microcontrollers (5), and assorted soldering/assembly issues.
Running a Discord agent powered by OpenClaw and GLM 5.3 Flash for 23 hours led to failures at around 65% of the 1M context window, exposing issues with caching and model performance under heavy use.
This paper introduces comprehensive taxonomies for analyzing failures in retrieval-based factuality evaluation in medical AI, revealing fundamental limitations of the retrieve-then-verify paradigm that persist despite model scaling and fine-tuning.
The author describes building a small business operations system using only an iPhone and AI tools like ChatGPT, detailing the architecture and seeking feedback on potential failure modes.
The author compared two 9B models for local agent seats, finding one model silently fails by writing tool calls as prose and making other errors that bypass detection, emphasizing the need to test for distinct failure modes.
The author discusses common challenges in debugging AI agent evaluation failures, seeking insights on efficient investigation methods and the reliability of comparison runs versus other evidence sources.
An operations professional tested eight AI agent platforms with the same job, finding that only two completed successfully, and highlighted the issue of agents reporting success when failures occur, suggesting that verifying the output destination is key.
The author has built Global Fail Map, an open-source interactive tool documenting historical failures in science, technology, and business to serve as a research layer for AI agents and developers.
The article discusses how teams often fix bad AI outputs individually, which doesn't scale. It advocates for logging failures to identify patterns, enabling systematic improvements instead of ad-hoc patches.
Anthropic uses a method of running the same AI agent multiple times in parallel, keeping successful runs, analyzing failures, and iteratively improving the process to enhance performance.
Filip Panoski shares his seven-year journey of building failed SaaS side projects, highlighting the importance of deciding to pivot, persist, or kill a project to eventually find success.
AutoSaddler is an automatic harness optimization framework that improves LLM agent performance on long-horizon tasks by iteratively updating harnesses using failure signals, achieving substantial gains on benchmarks like GAIA2 and SWE-Bench.
This article critiques the failure of Khanmigo, an AI tutoring chatbot, arguing that its design as a 'teaching by telling' tool was inherently flawed, contrasting it with effective learning through making and exploration.