Tag
该论文提出了一种基于回归的元评估框架，用于评估多语言风格个性化文本生成中的自动指标，发现经过校准的简单字符串和嵌入指标在40种评估设置下可媲美LLM-judge。
PADMÉ reformulates meta-evaluation of LM agent evaluators as a preference alignment problem and introduces a data synthesis method using only small language models, boosting agreement with human judgment from 73% to 85% over a naive baseline while enabling evaluation of 25 common models across agentic domains.
This paper introduces RM-EVAL, a reward model trained on human preference data for reference-free meta-evaluation of grammatical error correction, and shows how it can improve GEC systems via reward-guided text generation.
MEGA is a self-evolving infrastructure for coding-agent optimization that distills reusable wisdom from sessions, composes it via a typed Wisdom Graph, and uses operational evidence to continuously improve both the agents and the knowledge guiding their optimization.
This paper proposes a dataset-centric meta-evaluation framework that audits LLM benchmarks at the sample level across five latent dimensions, exposing internal heterogeneity and enabling criterion-driven composition of benchmark subsets for targeted model evaluation.
This paper presents the first systematic meta-evaluation of LLM-generated rubrics for reproducing experiments from research papers. It reformulates rubrics into a checklist format and evaluates generation settings both intrinsically (semantic similarity) and extrinsically (score alignment), finding that augmented settings improve downstream evaluation alignment but generated rubrics are often overly fine-grained and biased toward high scores.
Introduces Eval-Pair Matrix, a controlled meta-evaluation protocol for source-grounded RAG that induces hidden contradictions to detect self-leniency in LLM judges. The study finds minimal same-model effects and emphasizes methodological improvements for RAG judge studies.
Counsel is the first public dataset of human meta-evaluations of LLM critiques for agentic tasks, designed to improve the calibration and reliability of automated evaluation methods.
This paper introduces REFLECT, a meta-evaluation benchmark for assessing the reliability of LLM judges in evaluating deep research agents. Experiments show current LLM judges remain unreliable, with overall accuracies below 55% across reasoning, tool-use, and report-quality failures.
This paper identifies the 'evaluation trap' where AI benchmarks inadvertently stabilize dominant paradigms by narrowing what counts as progress, and introduces Epistematics, a meta-evaluative methodology to ensure evaluation criteria discriminate true capability from proxy behaviors.