A developer created LLM Canary, an open-source quiz program that sends randomized tasks to multiple LLMs to track performance over time. After a week of hourly testing across seven models, the results show all models fluctuate throughout the day with no consistent pattern, and no clear evidence of degradation was found.
I've been noticing an increasing number of posts and comments on Reddit claiming that LLM models are either becoming dumber over time or have varying performance throughout the day. I tried to find long-form, over-time performance graphs or repos that tracked this but came up empty after a 5-minute search across GitHub and Google. So I ended up building LLM Canary. **What it is and how it works**: the program fires a pseudo-randomized questionnaire at a set of LLMs, scores every answer programmatically, and logs the results. There are 25 questions per run: arithmetic tasks, counting letters, reversing a word, predicting JavaScript output, a chained password game with 5, 10, and 15 simultaneous rules, and more. I ran it for a week with crontab every hour across 7 models: Claude Haiku 4.5, Claude Sonnet 4.6, GPT-4.1, GPT-4.1 Mini, GPT-4o Mini, GPT-4.1 Nano, Gemini 2.5 Flash Lite. The most consistent data came from Claude, since I only introduced the other providers partway through — and Gemini's expensive flagships burned through budget too quickly to collect enough data. Check the readme in the repo if you want to learn more. **Note**: One week is not enough to prove or disprove the degradation claim yet — I need to run it longer and review performance week over week or month over month. What I have is a project capable of asking questions and establishing an ELO score. # FINDINGS First things first — *ALL* models fluctuate throughout the day and not in any consistent pattern. Some are more volatile, like Gemini 2.5 Flash Lite, while others like GPT-4.1 Nano show an island of steady, predictable performance with smaller deviations between 6 AM and 1 PM GMT+0. If API load were driving degradation at specific hours, you'd expect the same hours to look bad across multiple providers simultaneously — but that's not what we see here. With the data collected so far, there's no "smoking gun" clearly showing a model becoming dumber. Models struggle with hard questions, some more than others. So that's one immediate finding — a model that successfully answers a question once isn't guaranteed to pass it the next hour. What matters is consistency and question difficulty. Next: It isn't really fair to compare model to model by question since some are naturally better at math while others are designed for language and writing — but let's do it anyway. Take \`*letter\_count*\` for example. The prompt is something like: How many times does the letter 'c' appear in the word 'ecophysiologies'? Reply with just the number. Pretty much all models pass this with 40–60% accuracy. However, GPT-4.1 Nano and Gemini 2.5 Flash Lite embarrassingly score 16.8% and 17.76% respectively. Another interesting find: Claude Haiku 4.5, the cheaper Anthropic model, outperforms Claude Sonnet 4.6 at counting vowels in a paragraph (71.58% vs 64.74%). Almost everywhere else, Sonnet 4.6 takes the lead. \`*count\_f*\` is a prompt where the program takes random excerpts from the Bible and asks an LLM to count the letter 'f'. Pretty much ALL models fail here with around a 7.5% pass rate — they tend to skip stopwords like "of" and "for" — but Claude Sonnet 4.6, the most capable model in this list, manages 45.79%. \`*word\_count*\` is a similar test: the prompt takes a random paragraph from the Bible and asks the LLM to count the words. Again, most models skip stopwords and the average hovers around a 5.5% pass rate, though GPT-4o Mini manages 16.54%. GPT-4.1 Nano is the weakest of the bunch. Its total average score is only 45% with an ELO of 965.98 — and it had the lowest scores on 9 out of 25 questions — while Claude Sonnet 4.6 leads at a 75% average and ELO 1293.29. A 327-point ELO gap might not sound dramatic on paper, but the per-question breakdowns make the performance difference pretty hard to ignore. Finally, going back to the within-day fluctuations (min-max deltas per hour), you're looking at roughly a 150-point swing except for Claude (both Haiku and Sonnet). Their fluctuation delta SUM is around 4.4k. Divide that by 24 and you get \~183.3 ELO points. That's probably what tips people off — it makes it feel like "Claude is dumber this morning than yesterday."
An analysis of 31,352 hourly LLM benchmark scores shows between-day variation is about three times greater than within-day variation, emphasizing the importance of continuous monitoring for performance drift, leading to the creation of the AIStupidLevel system.
This paper investigates whether team-based interaction improves LLM performance in the quiz game 'What? Where? When?' (ChGK). Using six recent open LLMs on a 2025 dataset of 572 questions, they show that team strategies (voting, silent captain, talkative captain) outperform single models by up to 20 percentage points, with the best team achieving 44.23% accuracy, approaching human performance.
A new study tests LLMs across 28 real-world studies and finds they match human majority only 53% of the time, no better than random, challenging the trend of using LLMs to replace human feedback.
This paper evaluates whether off-the-shelf LLMs can reliably simulate human responses in controlled behavioral experiments by comparing LLM-generated data with human survey responses on accuracy perception. The findings show that while LLMs capture directional effects and aggregate belief-updating patterns, they do not consistently match human-scale effect magnitudes, clarifying when synthetic LLM data can serve as behavioral proxies.
The article discusses a research paper revealing that many science-based LLM benchmarks have incorrect answers, and when corrected, LLM performance scores rise significantly, suggesting current evaluations may underestimate AI capabilities in physics.