Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness

Hugging Face Daily Papers Papers
factual-completeness evaluation benchmark open-ended-generation meta-rubric gamut long-form-generation

Summary

This paper introduces a two-level meta-rubric framework for evaluating factual completeness in open-ended generation, instantiated as the GAMUT benchmark. It features 1,813 questions across 10 domains and finds the benchmark challenging and discriminative, with top models scoring 58.7%.

Evaluating the factuality of long-form generations has focused predominantly on precision, measuring whether the claims a model makes are correct. The dominant decompose-search-verify pipeline catches incorrect claims well but says little about whether a response contains all the information it should. Measuring factual completeness, the missing half of factuality, is harder: it requires enumerating the full set of facts a complete answer should contain, and these facts rarely form a flat list. They often involve open-ended sets where coverage is what matters, ordered processes, and relationships among facts that a list of independent boolean checks fails to capture. We introduce a two-level meta-rubric framework for evaluating open-ended generation, and instantiate it as Gamut (Grounded Assessment of Multimodal Factuality), a benchmark for factual completeness in long-form generation. The framework rests on a two-level rubric representation: a structured meta-rubric captures the organization and importance of the required content, which is then mechanically compiled into a flat checklist of binary, machine-gradable rubrics that an LLM judge scores reliably. We construct 1,813 questions grounded in real wearable imagery across 10 diverse domains, each paired with an evidence-backed rubric verified by expert human annotators. Because the framework is modality-agnostic, we also release a text-only variant. Evaluating 14 frontier and open-weight models, we find the benchmark genuinely challenging (best score 58.7% from Gemini 3.1 Pro), highly discriminative, and robust to the choice of judge.
Original Article
View Cached Full Text

Cached at: 07/22/26, 06:41 AM

Paper page - Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness

Source: https://huggingface.co/papers/2607.19322

Abstract

Evaluatingthefactualityoflong-formgenerationshasfocusedpredominantlyonprecision,measuringwhethertheclaimsamodelmakesarecorrect.Thedominantdecompose-search-verifypipelinecatchesincorrectclaimswellbutsayslittleaboutwhetheraresponsecontainsalltheinformationitshould.Measuringfactualcompleteness,themissinghalfoffactuality,isharder:itrequiresenumeratingthefullsetoffactsacompleteanswershouldcontain,andthesefactsrarelyformaflatlist.Theyofteninvolveopen-endedsetswherecoverageiswhatmatters,orderedprocesses,andrelationshipsamongfactsthatalistofindependentbooleanchecksfailstocapture.Weintroduceatwo-levelmeta-rubricframeworkforevaluatingopen-endedgeneration,andinstantiateitasGamut(GroundedAssessmentofMultimodalFactuality),abenchmarkforfactualcompletenessinlong-formgeneration.Theframeworkrestsonatwo-levelrubricrepresentation:astructuredmeta-rubriccapturestheorganizationandimportanceoftherequiredcontent,whichisthenmechanicallycompiledintoaflatchecklistofbinary,machine-gradablerubricsthatanLLMjudgescoresreliably.Weconstruct1,813questionsgroundedinrealwearableimageryacross10diversedomains,eachpairedwithanevidence-backedrubricverifiedbyexperthumanannotators.Becausetheframeworkismodality-agnostic,wealsoreleaseatext-onlyvariant.Evaluating14frontierandopen-weightmodels,wefindthebenchmarkgenuinelychallenging(bestscore58.7%fromGemini3.1Pro),highlydiscriminative,androbusttothechoiceofjudge.

View arXiv pageView PDFGitHub0Add to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.19322 in a model README.md to link it from this page.

Datasets citing this paper1

#### facebook/GAMUT Viewer• Updatedabout 1 hour ago • 3.62k • 1 • 1

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.19322 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproduction

arXiv cs.CL

This paper presents the first systematic meta-evaluation of LLM-generated rubrics for reproducing experiments from research papers. It reformulates rubrics into a checklist format and evaluates generation settings both intrinsically (semantic similarity) and extrinsically (score alignment), finding that augmented settings improve downstream evaluation alignment but generated rubrics are often overly fine-grained and biased toward high scores.