Tag
The article discusses how teams often fix bad AI outputs individually, which doesn't scale. It advocates for logging failures to identify patterns, enabling systematic improvements instead of ad-hoc patches.
Anthropic uses a method of running the same AI agent multiple times in parallel, keeping successful runs, analyzing failures, and iteratively improving the process to enhance performance.
Filip Panoski shares his seven-year journey of building failed SaaS side projects, highlighting the importance of deciding to pivot, persist, or kill a project to eventually find success.
AutoSaddler is an automatic harness optimization framework that improves LLM agent performance on long-horizon tasks by iteratively updating harnesses using failure signals, achieving substantial gains on benchmarks like GAIA2 and SWE-Bench.
This article critiques the failure of Khanmigo, an AI tutoring chatbot, arguing that its design as a 'teaching by telling' tool was inherently flawed, contrasting it with effective learning through making and exploration.
This article discusses fundamental principles of how complex systems fail, emphasizing that failures are intrinsic and catastrophe requires multiple simultaneous failures.
This article recommends paying attention to AI Scientist and discusses how research agents can learn from failures by analogizing to fuzz testing, thereby mapping the unknown and guiding subsequent experiments.
ComponentBench introduces a benchmark and diagnostic pipeline for evaluating computer-use agents on component-level interactions in modern web UIs, addressing gaps in current evaluation methods by focusing on realistic, short interactions to diagnose failures across models.
This position paper argues that failures in LLM-based multi-agent systems are fundamentally concurrency control problems and advocates for prioritizing explicit concurrency control mechanisms as a core design concern.
The author analyzed 155 AI agent jobs and discovered that most failures stemmed from infrastructure issues like timeouts and false success signals, not model errors, leading to practices such as asserting on effects and using multiple verification paths.
This paper introduces AutoResearchEval, an evaluation framework for AI agents in automated scientific research, revealing a critical lack of metacognitive abilities as a recurring failure pattern across models.
Running 11 research agents in parallel for a data sweep cut processing time in half but revealed failures like empty briefs and API rate limits, emphasizing that orchestration and verification are more critical than the specific tool used.
A former employee looks back on the severe outage at Fubao Cloud (Alibaba Cloud) on November 12, 2023, and says the outage at Chrysanthemum Cloud (Huawei Cloud) on July 26, 2026 was more than a hundred times worse, involving millions of instances locked and stopped, underscoring the absence of circuit breakers and manual confirmation mechanisms at cloud providers.
An empirical attribution study of failures in multi-page visually rich document understanding (MP-VRDU), isolating representation, selection, and reasoning failure modes and providing guidance for building such systems under fixed compute budgets.
This paper introduces SearchAuditBench, a benchmark of 1,243 failed long-horizon search-agent trajectories with expert annotations, and SearchAuditor, a multi-perspective auditing framework that localizes, attributes, and repairs agent failures. Experiments show SearchAuditor outperforms baselines, achieving a 32.3% end-to-end pass rate with frontier models like GPT-5.5.
This paper presents a model-agnostic framework for per-modality failure analysis in multimodal clinical AI, distinguishing loud vs silent failures when a modality is dropped. Validated on planted ground truth and applied to EchoJEPA and HuBERT-ECG embeddings for LVEF prediction, it shows that dropping echo nearly doubles error.
This paper investigates the claim that 4-bit weight quantization is nearly lossless for LLM agents, showing that while aggregate benchmark scores stay flat, quantization amplifies existing tool-call failures (e.g., hallucination), hidden by the benchmark's error budget. The authors suggest reporting per-channel error rates and success under shrinking budgets to reveal the masked damage.
This article highlights the recurring issue of AI agents inadvertently modifying or deploying real infrastructure, prompting a discussion on what types of failures should be caught before they occur.
The author argues that pretrained LLMs fail on real company work because they lack a fast learning system akin to the hippocampus to capture specific, scattered work procedures, unlike the human brain's complementary learning systems. They propose consolidating work episodes into approved, versioned procedures with governance as a potential fix.
An analysis of the gap between episodic and procedural memory in LLM agents, citing a new paper (Memp) from Zhejiang University and Alibaba that builds procedural memory from agent trajectories and uses failure signals to revise stored procedures.