Tag
This article discusses fundamental principles of how complex systems fail, emphasizing that failures are intrinsic and catastrophe requires multiple simultaneous failures.
This article recommends paying attention to AI Scientist and discusses how research agents can learn from failures by analogizing to fuzz testing, thereby mapping the unknown and guiding subsequent experiments.
ComponentBench introduces a benchmark and diagnostic pipeline for evaluating computer-use agents on component-level interactions in modern web UIs, addressing gaps in current evaluation methods by focusing on realistic, short interactions to diagnose failures across models.
This position paper argues that failures in LLM-based multi-agent systems are fundamentally concurrency control problems and advocates for prioritizing explicit concurrency control mechanisms as a core design concern.
The author analyzed 155 AI agent jobs and discovered that most failures stemmed from infrastructure issues like timeouts and false success signals, not model errors, leading to practices such as asserting on effects and using multiple verification paths.
This paper introduces AutoResearchEval, an evaluation framework for AI agents in automated scientific research, revealing a critical lack of metacognitive abilities as a recurring failure pattern across models.
Running 11 research agents in parallel for a data sweep cut processing time in half but revealed failures like empty briefs and API rate limits, emphasizing that orchestration and verification are more critical than the specific tool used.
A former employee looks back on the severe outage at Fubao Cloud (Alibaba Cloud) on November 12, 2023, and says the outage at Chrysanthemum Cloud (Huawei Cloud) on July 26, 2026 was more than a hundred times worse, involving millions of instances locked and stopped, underscoring the absence of circuit breakers and manual confirmation mechanisms at cloud providers.
An empirical attribution study of failures in multi-page visually rich document understanding (MP-VRDU), isolating representation, selection, and reasoning failure modes and providing guidance for building such systems under fixed compute budgets.
This paper introduces SearchAuditBench, a benchmark of 1,243 failed long-horizon search-agent trajectories with expert annotations, and SearchAuditor, a multi-perspective auditing framework that localizes, attributes, and repairs agent failures. Experiments show SearchAuditor outperforms baselines, achieving a 32.3% end-to-end pass rate with frontier models like GPT-5.5.
This paper presents a model-agnostic framework for per-modality failure analysis in multimodal clinical AI, distinguishing loud vs silent failures when a modality is dropped. Validated on planted ground truth and applied to EchoJEPA and HuBERT-ECG embeddings for LVEF prediction, it shows that dropping echo nearly doubles error.
This paper investigates the claim that 4-bit weight quantization is nearly lossless for LLM agents, showing that while aggregate benchmark scores stay flat, quantization amplifies existing tool-call failures (e.g., hallucination), hidden by the benchmark's error budget. The authors suggest reporting per-channel error rates and success under shrinking budgets to reveal the masked damage.
This article highlights the recurring issue of AI agents inadvertently modifying or deploying real infrastructure, prompting a discussion on what types of failures should be caught before they occur.
The author argues that pretrained LLMs fail on real company work because they lack a fast learning system akin to the hippocampus to capture specific, scattered work procedures, unlike the human brain's complementary learning systems. They propose consolidating work episodes into approved, versioned procedures with governance as a potential fix.
An analysis of the gap between episodic and procedural memory in LLM agents, citing a new paper (Memp) from Zhejiang University and Alibaba that builds procedural memory from agent trajectories and uses failure signals to revise stored procedures.
AgentDebugX is an open-source debugging framework for LLM agents that organizes failure debugging as a closed loop of detect, attribute, recover, and rerun, with a core component DeepDebug achieving strong attribution accuracy and repair capability on benchmarks.
Discusses how even a highly capable AI agent can fail in production if its underlying operational truth is weak, highlighting challenges in real-world deployment.
This paper synthesizes 27 benchmark, taxonomy, and audit papers from 2023-2026 into a unified taxonomy of LLM agent limitations, identifying six failure clusters including tool invocation errors, planning failures, long-horizon degradation, multi-agent coordination issues, safety concerns, and measurement validity problems.
An auditor finds that 0 of 32 published "discoveries" by an autonomous research agent were novel as framed, with the dominant failure being over-labeling rather than bad measurement.
The article argues that diagnosis—explaining why an agent failed in operational terms and what is safe to do next—is a missing first-class skill in production agent stacks, more critical than making agents sound smart.