Tag
A developer has been running daily fixed probes against 34 LLM APIs from 15 labs since August, hashing readings into a public transparency log to detect silent model changes. The project's first notable finding is that DeepSeek's reasoner began using 10-12x more thinking tokens on Sept 10 with no announcement, making it slower and more expensive rather than worse.
Livenerf is a deterministic benchmark tool designed to detect if AI models degrade after launch, currently running a 30-day study on Claude Opus 5.5 to measure performance drift.
The author describes running weekly audits on a production AI agent, observing that its responses gradually drifted and eventually violated policy without model updates, emphasizing the need for continuous monitoring in real-world AI deployments.
An internal HR chatbot with strict safety boundaries gradually started answering forbidden questions due to model drift, revealing a gap in continuous testing for AI systems.
AI Stupid Level provides real-time drift detection for AI agents, helping monitor model performance changes and maintain reliability.
Analysis of a recurring failure pattern in production AI systems where technically correct decisions become contextually wrong as underlying assumptions shift, framed as the 'Formalisation Trap' where meaning gets locked into outdated structures.