I've been measuring 34 LLM APIs every day since August to see when they quietly change. One did.

Reddit r/ArtificialInteligence 09/30/26, 12:48 PM Tools

llm-evaluation model-drift api-monitoring observability open-source benchmarking transparency

Summary

A developer has been running daily fixed probes against 34 LLM APIs from 15 labs since August, hashing readings into a public transparency log to detect silent model changes. The project's first notable finding is that DeepSeek's reasoner began using 10-12x more thinking tokens on Sept 10 with no announcement, making it slower and more expensive rather than worse.

Anyone building on a hosted model has had the moment where it feels different and nobody said anything. I wanted something more concrete than a feeling, so since Aug 20 I've been running the same private set of probes every day against 34 models from 15 labs, over their APIs, and comparing each model only with its own past. No rankings, no LLM judging the answers, just fixed checks graded by code. Every reading is hashed into a public transparency log the moment it's taken, so nobody can argue later about when the baseline existed. The first real catch: on Sept 10, DeepSeek's reasoner started using roughly 10 to 12x the thinking tokens it had used for the previous three weeks, on the same probes. I couldn't find any announcement. That's not the model getting worse, it's the model getting slower and more expensive, which is the kind of change people describe as "it feels off" and then can't prove. Nothing on the other 33 looks like a quiet drop in capability so far, and that's a result too. Opus 5.5 has been on it since it launched. Code and methodology: https://github.com/piperoll/seismograph Live readings: https://seismo.piperoll.org Still early. Happy to answer questions on how it works or what it can't detect yet.

Original Article

Similar Articles

We use LLMs to analyze every file in your codebase. Everyone told us this was a stupid idea because of cost but it wasnt.

Reddit r/ArtificialInteligence

A benchmark study demonstrates that using LLMs to analyze entire codebases is cost-effective, identifying DeepSeek V4 Flash as the optimal default model due to its low cost and comparable accuracy to premium options like Claude Opus.

DeepSeek LLM: Scaling Open-Source Language Models with Longtermism

Papers with Code Trending

DeepSeek LLM is an open-source language model project that develops a large dataset and employs SFT and DPO to achieve performance surpassing LLaMA-2 70B and GPT-3.5 in various benchmarks and open-ended evaluations.

Building independent LLM drift detection - sharing the methodology, looking for feedback on the approach

Reddit r/artificial

The author shares a methodology for building an external LLM drift detection system that continuously probes model behavior (schema adherence, instruction-following, refusal rates, etc.) to catch silent degradations in API performance, and invites feedback on the approach, pricing, and use cases.

Open-source LLM benchmark runs 147 coding tasks every 4 hours, 5-trial median with 95% CI, and uses CUSUM for change-point detection. Curious what people think of the methodology

Reddit r/AI_Agents

An open-source LLM benchmark with 147 coding tasks runs every 4 hours, using 5-trial median with 95% confidence intervals and CUSUM for change-point detection, sparking discussion on its methodology.

I analyzed 31,352 hourly LLM benchmark scores: within-day variation was 2.8 points, while between-day variation was 8.4 [P]

Reddit r/MachineLearning

An analysis of 31,352 hourly LLM benchmark scores shows between-day variation is about three times greater than within-day variation, emphasizing the importance of continuous monitoring for performance drift, leading to the creation of the AIStupidLevel system.