Benchmarking LLMs
Summary
A study or report on benchmarking large language models, likely comparing performance across various tasks.
Similar Articles
LLM Ass Bench
LLM Ass Bench is a benchmark or tool for evaluating Large Language Models, with a focus on prompts.
Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation
This paper proposes a dataset-centric meta-evaluation framework that audits LLM benchmarks at the sample level across five latent dimensions, exposing internal heterogeneity and enabling criterion-driven composition of benchmark subsets for targeted model evaluation.
RepBench: Compiling Benchmarks into Capability Representations for Large Language Models
RepBench is a benchmark-grounded data layer for representation probing in LLMs, built from 13,427 benchmark papers and 353 public datasets, yielding 46,149 probe texts across 94 capabilities. It evaluates capability vector transfer across models and readout methods.
Ask Which, Not How Good: Sizing Benchmarks Scored by an LLM
This paper investigates the measurement limitations of LLM-judged benchmarks, showing that generalizability is constrained by judge variability and that protocol design significantly affects performance ceilings, with implications for AI benchmarking methodologies.
Benchmarking Language Models for Statistical Problem Formulation
The paper introduces StatFormBench, a benchmark for evaluating LLMs on statistical problem formulation, and finds that current models have significant limitations in classifying problems and identifying variables.