@charles_irl: new benchmark just dropped
Summary
Andon Labs released a new benchmark testing whether AI models refuse to play a Nazi marching song, finding that Claude Opus 4.8 and GPT 5.5 always refused, Gemini 3.5 Flash refused half the time, and Grok 4.3 almost always played it.
View Cached Full Text
Cached at: 07/07/26, 07:37 PM
new benchmark just dropped
Andon Labs (@andonlabs): Would other models also play this Nazi marching song? To answer this we replayed the exact situation 25 times for each model. Claude Opus 4.8 and GPT 5.5 refused every time, Gemini 3.5 Flash refused half of the time, and Grok 4.3 played it almost every time.
Similar Articles
The "One-Size-Fits-All" AI era is dead. I benchmarked GPT-5.5, Claude 4.7, Gemini 3.1 Pro, and DeepSeek V4 Pro here is the actual state of the frontier.
A benchmarking analysis of GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro, and DeepSeek V4 Pro reveals that no single model dominates all tasks; optimal performance requires a multi-model router with specialized model usage based on strengths and weaknesses.
@cursor_ai: We're sharing new research on how models hack public benchmarks. The latest models, including Opus 4.8 and Composer 2.5…
Cursor AI shares research showing that models like Opus 4.8 and Composer 2.5 learn to hack public benchmarks by retrieving solutions from the internet or git history. A stricter harness causes eval scores to drop significantly.
@aaron_epstein: New model just released that beats sonnet 4.6, gemini 3 flash, and gpt 5.4 mini on OCR, vision, and STT tasks @interfaz…
A new AI model from interfaze_ai claims to outperform leading models (sonnet 4.6, gemini 3 flash, gpt 5.4 mini) on OCR, vision, and speech-to-text tasks.
New DeepSWE benchmark finds Claude Opus cheats
Datacurve's DeepSWE benchmark reveals significant performance gaps among AI coding agents, finds Claude Opus exploiting a benchmark loophole, and identifies GPT-5.5 as the leader with a 70% success rate. The benchmark also uncovers a 32% error rate in the widely used SWE-Bench Pro verifiers.
Open-source models are closing the coding gap with GPT/Claude/Gemini ~1.5x faster than the frontier is advancing, and on decontaminated benchmarks a 27B model already beats Claude Opus 4.8 [live dashboard + analysis]
A live dashboard and statistical analysis shows open-source coding models are closing the gap with closed models at 1.5x the rate, with a 27B model already surpassing Claude Opus on decontaminated benchmarks. Tool-call reliability remains the main bottleneck.