failure-detection

This paper identifies spectral drift in internal activations of neural networks during misclassifications and introduces Self-Detecting Neural Networks (SDNN) that monitors spectral dynamics to detect failures, achieving 79% AUROC on CIFAR-10, outperforming confidence-based methods by 25-30 percentage points.

This paper proposes a two-stage online learning framework for detecting service-affecting failures in mobile core networks by modeling normal traffic dynamics and analyzing residuals, achieving improved precision-recall trade-off over static thresholds.

Discusses the phenomenon where AI agents appear to succeed at tasks but later reveal failures, highlighting challenges in agent evaluation and monitoring.

This paper proposes a recall-controlled abort cascade that uses lightweight probes on LLM agent internal representations to detect and abort doomed episodes early, saving up to 47% inference compute while maintaining high recall of successful episodes.

Foresight is a failure detection framework for long-horizon robotic manipulation that uses action-conditioned world model latents and functional conformal prediction to monitor trajectories, trained only with final task labels. It demonstrates state-of-the-art performance across simulation and real robot tasks.

A community discussion on agent memory reveals that while various patches exist for what to write down (e.g., plain files, layered memory, post-mortems), the unsolved problem is what to keep—detecting failures is tractable, but deciding which lessons persist still needs human judgment.

This paper characterizes two distinct processes by which language models fail in reasoning—committed failure and persistent uncertainty—using token-level uncertainty signals, and demonstrates implications for self-consistency and failure detection strategies.

AEGIS uses activation-probe early warning to switch to a stronger policy before failures compound in long-horizon robot manipulation, recovering twice as many failures as budget-matched escalation.

Hide-and-Seek is a framework that detects robot execution failures in VLA models by localizing failure-indicative actions through contrastive learning without step-level annotations, achieving state-of-the-art multi-task failure detection.

This paper compares cross-validation ensembles to deep ensembles for uncertainty estimation in medical image segmentation. Deep ensembles outperform cross-validation ensembles in calibration and failure detection, while cross-validation ensembles better approximate inter-rater variability.