Tag
An analysis of 31,352 hourly LLM benchmark scores shows between-day variation is about three times greater than within-day variation, emphasizing the importance of continuous monitoring for performance drift, leading to the creation of the AIStupidLevel system.
ReWeaver AI DriftDetector is a tool that provides drift scores for GitHub repositories, aiding in monitoring AI model performance.
The article introduces DriftGuard, an open-sourced tool for detecting when AI agents drift off-task and waste tokens, using relevance and self-drift metrics with a focus on minimizing false alarms.
This paper presents a novel framework for detecting performance drift in Machine Learning as a Service (MLaaS) within IoT environments, demonstrating significant accuracy improvements over baseline methods through adaptive monitoring.
TRACE is a new multi-layer benchmark for diagnosing drift and failures in human-AI controller coordination, built from ALFRED traces with 1,918 drifted samples annotated across five execution layers. Baseline results show drift identification and attribution well above random baselines across classical, recurrent, and attention-based model families.
This paper presents EaaS, a cloud-native microservices architecture for scalable AI monitoring that provides conformal prediction, calibration assessment, drift detection, and fairness monitoring with statistical guarantees.
Introduces DriftGuard, a safety-aware adaptive moderation framework that uses multiple monitors to detect subtle, safety-relevant distribution shifts and selectively updates models with a hard-mix adaptation set, improving toxic recall on evolving datasets.
The author shares a methodology for building an external LLM drift detection system that continuously probes model behavior (schema adherence, instruction-following, refusal rates, etc.) to catch silent degradations in API performance, and invites feedback on the approach, pricing, and use cases.
Proposes an anytime-valid attribution method that uses a human-labeled anchor set and a betting e-process to distinguish whether score drift in LLM evaluation pipelines comes from the system or the judge, resolving the ambiguity caused by silent judge changes.
PRISM is a closed-loop framework that treats prompt engineering as a continuous reliability problem for enterprise conversational AI. It automates test generation, simulation, evaluation, and repair, achieving 99% reliability and reducing authoring time from days to minutes.
This paper investigates disagreement-based drift detection in ensembles of incremental decision trees, finding that while effective in neural networks, the method underperforms loss-based detectors for tree ensembles due to limited model plasticity.
The article discusses measuring 'undeclared-intent spend' in agent workflows, quantifying compute tokens spent outside the declared intent to reveal behavioral costs like drift and off-task execution.
This paper introduces geometric stability measures—based on pairwise distance consistency in representations—to predict language model steerability and detect structural drift. Supervised variants achieve near-perfect correlation (ρ=0.89-0.97) with linear steerability across 35-69 embedding models, while unsupervised variants outperform CKA and Procrustes for post-deployment drift detection.