Introducing BetterBench - more accurate PP and TPS measurement
Summary
BetterBench is a new benchmarking tool that provides more accurate PP and TPS measurements by ensuring content consistency within 1% and testing across different content types, addressing variability in LLM performance benchmarks.
Similar Articles
Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks
This paper identifies systemic measurement bias in production LLM inference benchmarks caused by single-process Python clients using asyncio, and proposes a multi-process evaluation framework and a new metric (NTPOT) to accurately profile serving engines at scale.
PostgresBench: A Reproducible Benchmark for Postgres Services
ClickHouse has released PostgresBench, a public and reproducible benchmark for comparing managed Postgres services, using the standard pgbench tool with TPC-B-like workloads at multiple scale factors.
CPI-Bench: A Comprehensive,Practical and Intelligent Benchmark for Real-World Image Editing
CPI-Bench is a comprehensive benchmark for real-world image editing that evaluates multi-image tasks, practical applications, and reasoning-based editing to better differentiate model performance.
PTXBench: Benchmark and Adapt LLMs for GPU Kernel Optimization with Architecture-specific PTX
PTXBench is introduced as a benchmark to evaluate and adapt large language models for optimizing GPU kernels using architecture-specific PTX, showing uneven performance and fine-tuning insights.
ParBench: A Benchmark for Reliable Evaluation of LLM Parallel Code Translation
ParBench is a benchmark framework for evaluating LLM-based parallel API translation across CUDA, OpenMP, OpenCL, and OpenMP target offload, focusing on executable, reproducible conditions and robustness testing.