benchmarking

This paper tests whether commonsense benchmark scores predict real-world downstream task performance by evaluating 23 LLMs across four benchmarks and their reworked variants, finding that revisions preserve rankings but only offer task-dependent predictive validity.

A user shares experiments running a 4-bit quantized DeepSeek v4 Flash on a 32GB M5 MacBook Air, achieving roughly 50 tokens/s prefill and 1 token/s decode using streamed experts and other tricks.

A detailed benchmark of DeepSeek V4 Flash on a dual GH200 workstation, comparing SGLang and vLLM at 1M context with DSpark speculative decoding, finding SGLang faster at ~317 vs ~276 tok/s.

A tweet argues that LLM benchmarks should report token usage alongside accuracy, introducing VulcanBench, an open-source benchmark for realistic software engineering tasks that tracks token efficiency and cost.

Introduces PAST-Bench, a benchmark for evaluating whether personal AI agents improve from retained experience across sessions, and Hermes+, an extension with targeted interventions. Finds improvement is real but uneven across capabilities and models.

CalibratedRubric is a task-adaptive framework for building compact, measurable rubric banks for open-ended LLM evaluation, using Bayesian measurability filtering and IRT-based selection to improve human-gold agreement and rank fidelity across financial, healthcare, general, and legal benchmarks.

This paper surveys and benchmarks NLP-based automatic deception detection in legal contexts, comparing fine-tuned transformers and seven LLMs with various prompting strategies across seven datasets. Results show domain sensitivity, with fine-tuned models excelling in general domains and few-shot LLMs competitive in low-resource legal settings.

This paper proposes a benchmarking protocol using automated multi-model LLM review to evaluate AI Scientist systems, comparing frameworks like Sakana AI, CycleResearcher, and Data-to-Paper, and finds that FARS benchmark papers significantly outperform other systems.

Simon Willison's keynote at AI Engineer World's Fair reviews the last six months in LLMs, highlighting over 30 significant model releases and his 'pelican on a bicycle' SVG benchmark as a practical evaluation tool.

This position paper argues that long-horizon benchmark failures must be compared against baselines built from matched short stages, introducing the 'horizon residual' metric to distinguish task size from task difficulty in LLM agent evaluation.

This position paper argues that language model evaluation scores should be treated as perishable knowledge claims, not ground truth, and proposes explicit metadata such as formality tier, scope declaration, and expiration date to counter 'trust inflation' caused by averaging weak and strong signals.

An agent skill that forces LLMs to write documentation in ASD-STE100 Simplified Technical English, reducing STE violations by 72.9% across six Claude models.

Benchmark results for MindControl, a sampler-level guided reasoning budget for llama.cpp, show multi-stage signaling reduces token consumption without accuracy loss, and sometimes improves it on easier tasks.

A user shares benchmark results for running large language models (Qwen 27B-122B) on a dual-GPU setup with RTX 4090 and RTX 5060 Ti, achieving high token generation speeds (e.g., 206 t/s on 35B-A3B, 37-41 t/s on 122B). The post includes setup details and a link to a GitHub repo with scripts and raw data.

Benchmarks ConvLSTM against simpler models for one-day-ahead rainfall-field prediction across four Indian cities using IMDAA reanalysis data, finding that ConvLSTM does not consistently outperform alternatives like FC-LSTM or persistence.

This paper studies the efficacy of various Graph Neural Network message-passing layers in regression contexts, finding that deep convolutional GNNs, particularly GEN, outperform attention-based GNNs.

The authors introduce an AI Security Leaderboard that benchmarks frontier model robustness by running models through 1500 automated jailbreak attempts, highlighting gaps in security across models and inviting community feedback on methodology and next steps.

A team benchmarked routing different stages of an AI agent workflow to different models versus sending every request to Claude Opus 5 across 89 Terminal-Bench 2.1 tasks, and found surprising results.

A Reddit user shares results from testing proven orchestration techniques on small local LLMs, finding that 90% failed but the surviving 10% roughly doubled task completion across models like LFM 1.2B and Gemma 4 26B-A4B.

Ahmad announces a Local AI Hardware Arena using ODS to benchmark LLMs on hardware like RTX PRO 6000, DGX Spark, Strix Halo, M5 MacBook Pro, and ChatGPT, inviting community input for future comparisons.