error-analysis

The article discusses hidden states that an AI agent should track when diagnosing CI failures, such as flaky tests, real bugs, and configuration errors, and seeks feedback on weaknesses and missing states.

A developer describes running AI agents in a simulated restaurant world and finds that across different models and harnesses, the same three API-related mistakes dominate: guessing nonexistent endpoints, exceeding budgets, and acting on stale IDs. Shows persistent failure patterns despite fixing the system.

A guide on evaluating speech-to-text APIs by ranking transcript mistakes based on their actual impact on users, rather than raw accuracy metrics.

This paper characterizes the error that arises when substituting the Hadamard transform (which diagonalizes dyadic convolution) for the DFT (which diagonalizes circular convolution), identifying universal zero-error positions, a nearly full-rank error operator, and a closed-form expression for expected error magnitude governed by a single alignment scalar.

A blog post from Parlance Labs tests automated AI evaluation tools (Braintrust Loop, Arize Alyx, LangSmith Engine) on real production data, finding they catch 87% of issues humans flag but miss domain-specific failures and add noise, recommending iterative human-in-the-loop use.

Introduces how to use Phoenix's Pixie assistant to automatically classify error spans, batch annotate, and generate system prompt repair suggestions based on failure patterns.

Hamel Husain and Shreya present a new session on correctly automating AI evaluations, covering common mistakes, iterative error analysis, and building a failure mode taxonomy. The session includes a live demo of a review interface and emphasizes the importance of finding unknown unknowns.

MetaHOPE is a metaphor-oriented evaluation framework for analyzing translation errors in machine translation and large language models. The paper proposes an error severity-aware annotation framework and evaluates models like GoogleMT, GPT5.4, and Hunyuan-7b on English-Chinese metaphor translation.

A half-day tutorial at ISC High Performance 2026 on using compiler-assisted tools (FPChecker/LLVM) for floating-point error analysis and profiling in C/C++ scientific codes.

The paper presents a taxonomy of factual errors in human-written text, derived from newspaper corrections, and evaluates LLMs' performance on detecting these errors, finding that even top models like GPT-5.4 achieve only 52% word-level F1 score, highlighting the task's difficulty.

This paper argues that universal LLM reliability is impossible, but within operationally bounded patches (e.g., legal review, medical RAG), failures are sparse and repetitive, making reliability a local catalogue-discovery problem. It formalizes this with propositions and a corollary, relocating rather than dissolving the difficulty of long-context generation.

This paper presents a framework using LLMs to generate targeted synthetic misconceptions aligned to a five-class taxonomy adapted from Bloom's taxonomy, addressing the scarcity of labeled student error data in education research.

This paper decomposes MXFP4 quantization error into three additive components—scale bias, deadzone truncation, and grid noise—and proposes targeted corrections that recover BF16 accuracy to within 0.7 pp on Qwen2.5-3B and 3.0 pp on Qwen3-30B-A3B-Base for LLM reinforcement learning post-training.

This paper provides the first systematic analysis of error sources in trajectory-based data attribution methods, identifies optimizer mismatch as the dominant error, proposes AdamW-influence to address it, and offers practical guidelines for data selection via a K-step look-ahead framework.

This update to the RLM arXiv paper adds depth>1 experiments with recursive RLM calls, showing significant performance gains on OOLONG-Pairs and other benchmarks, along with new comparisons to OpenCode and Claude Code, additional training results on MRCRv2, and an expanded error analysis.

This paper introduces the Context-Contaminated Restart Model (CCRM) to formally analyze how failed attempts in LLM agent pipelines contaminate context and increase error rates during retries. It provides theoretical proofs and validates the model against SWE-bench data, showing significant discrepancies with standard independent models.

A comprehensive dual-aspect evaluation framework for large language models on Vietnamese legal text simplification, combining quantitative benchmarking (Accuracy, Readability, Consistency) with qualitative error analysis across GPT-4o, Claude 3 Opus, Gemini 1.5 Pro, and Grok-1.

DELEGATE-52 is a new benchmark revealing that current LLMs, including frontier models like GPT-5.4 and Claude 4.6 Opus, corrupt an average of 25% of document content during long delegated workflows across 52 professional domains. The research demonstrates that LLMs introduce sparse but severe errors that compound over interactions, raising concerns about their reliability for delegated work paradigms.