benchmarking

This paper proposes a sample-efficient framework using the cross-entropy method to estimate extreme reliability ('five-nines') in LLMs, addressing the limitations of standard benchmarks in detecting rare failures.

This paper argues that simple averaging in AI benchmarks fails under data sparsity and difficulty heterogeneity, proposing Item Response Theory (IRT) as a robust alternative to recover ground truth rankings.

This paper introduces EvalAgent, a system that automates the evaluation of AI agents by encoding domain-specific expertise, addressing the limitations of standard coding assistants in this task. It also presents AgentEvalBench, a benchmark for testing evaluation pipelines, and demonstrates significant improvements in evaluation reliability.

A user discusses the trade-offs between using vLLM and llama.cpp for local, single-user inference on AMD hardware, questioning if vLLM's performance benefits justify the complexity in non-enterprise settings.

The article highlights a performance rank-order flip between Claude Opus and Gemini Pro on a forecasting benchmark, depending on whether models perform their own web research or are given fixed evidence. This suggests that Opus excels at the research phase while Gemini is superior at judgment over fixed evidence, exposing a mismatch between standard benchmarks and actual deployment conditions.

The article highlights how ColBERT models, despite being smaller and older, outperform larger models like Qwen3-embed-8B when coupled with late interaction techniques and minimal fine-tuning.

MagicQuant v2.0 is a pipeline for creating hybrid mixed GGUF quant models, learning from Unsloth and other methods to find optimal quant configurations based on KLD benchmarks, with a focus on nonlinear wins and anomaly detection.

A user benchmarks token generation speed on llama.cpp with the GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 flag, comparing performance with and without MTP (Multi-Token Prediction). Results show a significant speedup from 49 tok/s to 64 tok/s when MTP is enabled on an RTX5090 with a Qwen3.6-27B model.

The author critiques the lack of transparent benchmarking in emerging context layer and MCP optimizer tools that promise drastic token savings, noting that real-world tests fail to replicate claimed efficiencies. They urge developers to demand open, reproducible benchmarks and ask for recommendations of tools that actually deliver measurable results.

Community testers evaluate quantized versions of Qwen3.6, ZAYA1, and other models for SVG chessboard generation accuracy using local inference frameworks like MLX.

This paper argues that log analysis is essential for credible AI agent evaluation, as outcome-only benchmarks often fail to reveal underlying capabilities, safety risks, or failure modes.

This paper presents a structured framework for benchmarking generative, multimodal, and agentic AI in healthcare, addressing the gap between high benchmark scores and real-world clinical reliability, safety, and relevance.

A developer toolkit providing configurations, wheels, and benchmarks for running large language models with NVFP4 precision on Nvidia Blackwell GPUs using TensorRT-LLM.

The article highlights a lack of benchmarks for evaluating the reliability of agent harnesses, specifically focusing on how MCP implementations handle tool calls and errors compared to the models themselves.

MemoryOS is an open-source, self-hosted AI agent memory tool using a temporal knowledge graph, achieving 86.2% accuracy on LongMemEval-s with fast 78ms retrieval speeds.

Eldar Kurtic presents a comprehensive study on TurboQuant, revealing its real-world effects on accuracy, latency, and throughput beyond initial evaluations.

This paper argues that Generative AI evaluation should shift from static benchmarks to measuring real-world utility and human outcomes. It introduces the SCU-GenEval framework and supporting instruments to address the disconnect between benchmark performance and deployment success.

This study presents a 33-model atlas analyzing domain-level metacognitive monitoring in frontier LLMs using MMLU benchmarks, revealing significant variations in confidence calibration across different knowledge domains that are obscured by aggregate metrics.

The author introduces a web-based script designed to help users intuitively understand token-per-second speeds in local LLM setups by simulating text, code, and reasoning generation rates.

The author introduces LLM Win, a tool that visualizes LLM benchmark results as a directed graph to analyze transitive relationships and ranking reversals. Experimental findings suggest that LLM rankings function more like a capability graph with high weak-to-strong reachability rather than a linear ladder.