we benchmark models nobody actually runs

Reddit r/LocalLLaMA 08/17/26, 09:53 PM News

Summary

The article critiques the lack of systematic benchmarking for AI models in different quantization formats, highlighting discrepancies between benchmark results and real-world usage, and calls for more thorough evaluation.

qwen3.8-27b looks genuinely impressive on the benchmark tables - beating models many times its size on some of them. but those numbers come from bf16 weights, and nobody here is running a 27b at bf16. we're running the 4-bit at ~17gb because that's what fits on a 4090 or a 24gb mac. so the model that got measured and the model that got downloaded three million times are not the same artifact. what i can't find anywhere: one model, one harness, swept across bf16 / Q8 / Q6_K / Q5_K_M / Q4_K_M / IQ4_XS, with error bars. perplexity charts exist, but ppl can stay nearly flat while something specific - long-context recall, multi-step math, strict tool-call json - quietly falls over. and with a vision model and a 256k window there's a lot more surface area for that to happen than usual. the question i actually care about: is Q4 of the 27b better than a higher-precision smaller model at the same vram? that's the decision everyone makes and i've never seen it measured. yes, red hat/neural magic publish evals for their quants and llama.cpp has kld tooling. nobody does it systematically, at release, on the harness the model card used. does this exist and i've missed it? and if it doesn't - is anyone with spare compute interested? agreeing on a harness and prompt set first is the only hard part; after that it's just a lot of hours.

Original Article

Similar Articles

Unsteady Metrics and Benchmarking Cultures of AI Model Builders

arXiv cs.AI

This paper introduces Benchmarking-Cultures-25, a dataset analyzing how AI model builders selectively highlight benchmarks in press releases. It finds a fragmented evaluation landscape with limited cross-model comparability, arguing that benchmarks are used as narrative devices for market positioning rather than standardized scientific measurement.

AI models provided by big AI corporate labs constitutes fraud by FTC's definition

Reddit r/ArtificialInteligence

The article argues that AI labs commit fraud by advertising high benchmark scores from ideal model versions while shipping heavily degraded versions (e.g., quantized, safety-stacked) that perform 50-60% worse, and proposes mandatory third-party re-benchmarking as a solution.

I stopped trusting model benchmarks and started running my own eval set, here is what changed[D]

Reddit r/MachineLearning

The author describes losing faith in public AI model benchmarks due to vendor-created metrics, self-reported parameters, and lack of independent verification, and advocates for building custom evaluation sets from real production traffic to make more relevant model comparisons.

Benchmarks compare open models against closed products, not closed models. We might be missing what were actually paying for

Reddit r/artificial

Argues that benchmarks comparing open models against closed API products are misleading because they measure raw inference vs. hidden tooling and preprocessing, suggesting the actual model quality gap may be smaller than reported.

@svpino: I don't trust benchmarks. We've all seen this movie: New model beats everyone else on a benchmark. People hype it. Then…

X AI KOLs Timeline

The tweet critiques traditional AI benchmarks and introduces TRACES, a new benchmark that evaluates AI's discovery process by focusing on how models reach answers, including tool usage, error correction, and evidence tracing.