Tag
Experiences building 6 AI agents reveal that model performance was not the bottleneck; instead, other practical issues consistently caused failures.
A new paper introduces Self-Harness, a method where AI agents self-improve by analyzing their own failures, generating fixes, and testing them, leading to up to 21 percentage point improvements in pass rates.
This paper investigates holographic reduced representations for zero-shot compositional reasoning in knowledge graphs, finding that while single-hop performance is strong, composition fails due to retrieval capacity and interference effects in the superposed memory, not the bind-unbind algebra.
A critique of AI agent loops that continue without reasoning, suggesting that agents should pause periodically to analyze failures and propose theories before retrying.
FailureScope is a behavioral-diagnosis method that clusters evaluation probes by cross-model pass/fail patterns to create interpretable failure taxonomies, validated across single-turn benchmarks, multi-turn dialogue, and adversarial agent attacks.
This paper introduces a white-box diagnostic framework that localizes instruction hierarchy failures in reasoning language models into identification, conflict resolution, and response realization stages. It evaluates several models and proposes two training-free self-monitoring mechanisms that reduce non-compliance by 81–99%.
A 6-week real-world experiment using an open-source desktop agent shell with a three-model split (Haiku triager, Sonnet reviewer, Opus executor) reports a 64% cost reduction and details failure modes like context bloat and runaway sub-agents.
The paper introduces GTBench, a curriculum-grounded benchmark for evaluating LLMs as mathematical research assistants in graph theory, containing 63 problems across three difficulty levels. It evaluates five frontier models and finds that performance degrades with difficulty, with GPT-5 achieving near-perfect results on basic problems but only 82% on graduate-level proofs.
ToolFailBench, a diagnostic benchmark for tool-using agents, has been accepted at two ICML 2026 workshops, FAGEN and AIWILD.
BenchTrace is a benchmark for evaluating the self-evolution abilities of LLM agents, focusing on reflection and controlled evolution through a dataset of 1,821 annotated episodes and two evaluation tasks: Reflection Evaluation and Evolution Evaluation. Experiments with Qwen3-32B and GPT-4.1 show both models struggle, with a main bottleneck in diagnosis and issues in generalization and forgetting.
This paper introduces the Insights Generator, a multi-agent system for systematic corpus-level trace diagnostics of LLM agents, which generates evidence-backed insights by proposing and testing hypotheses across execution traces. Experiments show that using Insights Generator reports improves scaffold performance by 30.4 percentage points.
Introduces LinAlg-Bench, a diagnostic benchmark evaluating 10 frontier LLMs on structured linear algebra computation across matrix dimensions, revealing that LLM mathematical failure is structurally constrained and transitions from execution errors to computational abandonment at 4x4 scale.