Tag
AgentDebugX is an open-source debugging framework for LLM agents that organizes failure debugging as a closed loop of detect, attribute, recover, and rerun, with a core component DeepDebug achieving strong attribution accuracy and repair capability on benchmarks.
Discusses how even a highly capable AI agent can fail in production if its underlying operational truth is weak, highlighting challenges in real-world deployment.
This paper synthesizes 27 benchmark, taxonomy, and audit papers from 2023-2026 into a unified taxonomy of LLM agent limitations, identifying six failure clusters including tool invocation errors, planning failures, long-horizon degradation, multi-agent coordination issues, safety concerns, and measurement validity problems.
An auditor finds that 0 of 32 published "discoveries" by an autonomous research agent were novel as framed, with the dominant failure being over-labeling rather than bad measurement.
The article argues that diagnosis—explaining why an agent failed in operational terms and what is safe to do next—is a missing first-class skill in production agent stacks, more critical than making agents sound smart.
Experiences building 6 AI agents reveal that model performance was not the bottleneck; instead, other practical issues consistently caused failures.
A new paper introduces Self-Harness, a method where AI agents self-improve by analyzing their own failures, generating fixes, and testing them, leading to up to 21 percentage point improvements in pass rates.
This paper investigates holographic reduced representations for zero-shot compositional reasoning in knowledge graphs, finding that while single-hop performance is strong, composition fails due to retrieval capacity and interference effects in the superposed memory, not the bind-unbind algebra.
A critique of AI agent loops that continue without reasoning, suggesting that agents should pause periodically to analyze failures and propose theories before retrying.
FailureScope is a behavioral-diagnosis method that clusters evaluation probes by cross-model pass/fail patterns to create interpretable failure taxonomies, validated across single-turn benchmarks, multi-turn dialogue, and adversarial agent attacks.
This paper introduces a white-box diagnostic framework that localizes instruction hierarchy failures in reasoning language models into identification, conflict resolution, and response realization stages. It evaluates several models and proposes two training-free self-monitoring mechanisms that reduce non-compliance by 81–99%.
A 6-week real-world experiment using an open-source desktop agent shell with a three-model split (Haiku triager, Sonnet reviewer, Opus executor) reports a 64% cost reduction and details failure modes like context bloat and runaway sub-agents.
The paper introduces GTBench, a curriculum-grounded benchmark for evaluating LLMs as mathematical research assistants in graph theory, containing 63 problems across three difficulty levels. It evaluates five frontier models and finds that performance degrades with difficulty, with GPT-5 achieving near-perfect results on basic problems but only 82% on graduate-level proofs.
ToolFailBench, a diagnostic benchmark for tool-using agents, has been accepted at two ICML 2026 workshops, FAGEN and AIWILD.
BenchTrace is a benchmark for evaluating the self-evolution abilities of LLM agents, focusing on reflection and controlled evolution through a dataset of 1,821 annotated episodes and two evaluation tasks: Reflection Evaluation and Evolution Evaluation. Experiments with Qwen3-32B and GPT-4.1 show both models struggle, with a main bottleneck in diagnosis and issues in generalization and forgetting.
This paper introduces the Insights Generator, a multi-agent system for systematic corpus-level trace diagnostics of LLM agents, which generates evidence-backed insights by proposing and testing hypotheses across execution traces. Experiments show that using Insights Generator reports improves scaffold performance by 30.4 percentage points.
Introduces LinAlg-Bench, a diagnostic benchmark evaluating 10 frontier LLMs on structured linear algebra computation across matrix dimensions, revealing that LLM mathematical failure is structurally constrained and transitions from execution errors to computational abandonment at 4x4 scale.