The author shares hands-on comparisons showing Gemma 4 outperforming larger models like Gemini 3.5 Flash and Claude Opus 5 on practical instruction-following, arguing that current LLM benchmarks fail to capture real-world usability.
Disclaimer, this was kinda written with AI (Gemma 4 again) but it also did really well here, it outputted what I wanted, when I asked it to refine stuff or improve on certain areas it did that without compromising others or making things bulky I’ve been noticing a massive disconnect lately between how models perform on technical leaderboards and how they actually perform in real-world tasks. Recently, I’ve been running some side-by-side tests, and I’m finding that Gemma 4 (26B A4B) is consistently outperforming much "larger" or more "advanced" models like Gemini 3.5 Flash and even Claude Opus 5 in practical instruction following. Here are two specific examples: Email Composition: I asked Gemini to write a response to an email with specific ideas. It failed to follow my instructions and missed the tone entirely. I tried Claude Opus 5, and while it was "smart," the output was sloppy, overly verbose, and sounded incredibly "AI-ish." Gemma 4, however, was able to nail the subtleties. It understands when I want something expressed subtly rather than just being blunt—it actually understands the subtext and the layers of intent. Prompt Refining & Engineering: I tried to have Gemini refine a prompt for me, and it failed my instructions every single time. This goes beyond just refining; even when I'm engineering a new prompt and tell a model, "Do X, Y, and Z, but avoid A, B, and C," Gemini is too literal—it just outputs: "Do X, Y, and Z and don't do A, B, and C." It's clunky and obvious. Gemma 4 handles this naturally; it writes the prompt in a way that pushes it away from A, B, and C without needing to explicitly mention them. It just understands. My takeaway: I’m starting to think that metrics used by sites like Artificial Analysis don't actually align with what the average user (or even a developer) needs. We don't just need high scores on math or coding benchmarks; we need models that actually listen, understand nuance, and don't hallucinate simple instructions. I want to hear from you guys: Has anyone else experienced this "intelligence gap" where smaller/different models feel more capable than the heavy hitters? Do you know of any leaderboards or benchmarks that more accurately represent real-world utility and instruction-following rather than just raw technical metrics? I’m not just looking for "use Gemma" because I like it, I know it isn't the "smartest" model overall. I want to know if there are other models (maybe the ChatGPT family?) that are actually stronger or bigger in this specific niche of nuance and instruction following?
A user tested Gemini-3.5-flash-lite on grading classwork and found it performed worse than its predecessor Gemini-2.5-flash-lite, suggesting newer models are not always better.
A user shares anecdotal findings that Gemma 4 31B outperforms Qwen 3.6 models and matches Opus 4.7 in understanding and refactoring messy academic code, highlighting a benchmark (SciCode) where Gemma excels.
A developer benchmarked 21 local LLMs on MacBook Air M5 using HumanEval+ and found Qwen 3.6 35B-A3B (MoE) leads at 89.6% with 16.9 tok/s, while Qwen 2.5 Coder 7B offers the best RAM-to-performance ratio at 84.2% in 4.5 GB. Notably, Gemma 4 models significantly underperformed expectations (31.1% for 31B), possibly due to Q4_K_M quantization effects.
Google updates Android Bench, a benchmark for LLMs in Android development tasks, adding new models like Claude Fable 5 and Qwen 3.7 Max, but its own Gemini models still trail behind competitors.
A comparison suggesting that Google's Gemini 3.1 Pro underperforms relative to Opus 4.7 in real-world usage, with the article highlighting Artificial Analysis as a go-to benchmarking resource.