Tag
MEGA is a self-evolving infrastructure for coding-agent optimization that distills reusable wisdom from sessions, composes it via a typed Wisdom Graph, and uses operational evidence to continuously improve both the agents and the knowledge guiding their optimization.
This paper proposes a dataset-centric meta-evaluation framework that audits LLM benchmarks at the sample level across five latent dimensions, exposing internal heterogeneity and enabling criterion-driven composition of benchmark subsets for targeted model evaluation.
This paper presents the first systematic meta-evaluation of LLM-generated rubrics for reproducing experiments from research papers. It reformulates rubrics into a checklist format and evaluates generation settings both intrinsically (semantic similarity) and extrinsically (score alignment), finding that augmented settings improve downstream evaluation alignment but generated rubrics are often overly fine-grained and biased toward high scores.
Introduces Eval-Pair Matrix, a controlled meta-evaluation protocol for source-grounded RAG that induces hidden contradictions to detect self-leniency in LLM judges. The study finds minimal same-model effects and emphasizes methodological improvements for RAG judge studies.
Counsel is the first public dataset of human meta-evaluations of LLM critiques for agentic tasks, designed to improve the calibration and reliability of automated evaluation methods.
This paper introduces REFLECT, a meta-evaluation benchmark for assessing the reliability of LLM judges in evaluating deep research agents. Experiments show current LLM judges remain unreliable, with overall accuracies below 55% across reasoning, tool-use, and report-quality failures.
This paper identifies the 'evaluation trap' where AI benchmarks inadvertently stabilize dominant paradigms by narrowing what counts as progress, and introduces Epistematics, a meta-evaluative methodology to ensure evaluation criteria discriminate true capability from proxy behaviors.