Tag
This paper investigates the measurement limitations of LLM-judged benchmarks, showing that generalizability is constrained by judge variability and that protocol design significantly affects performance ceilings, with implications for AI benchmarking methodologies.
RULER introduces instance-aware rubric rewards for SVG generation, using a vision-language judge to optimize reinforcement learning and significantly improve performance over previous methods.
Grading Assistant is an AI agent that scores student work against rubrics and drafts growth-oriented feedback to help teachers provide timely, specific, and consistent assessments.
The paper introduces K-Bench 01, a benchmark for evaluating AI agents on real scientific requests, revealing that no model consistently meets the threshold for acceptable performance, with overclaiming as a common failure.
EviRank reformulates multimodal image re-ranking as semantic constraint satisfaction using structured evidence packages, achieving state-of-the-art performance without training.
This paper proposes a framework for sentence-level interpretability of rubric-based scoring, comparing SHAP and LLM-generated rationales. It finds that fine-tuned pretrained language models outperform LLMs in prediction accuracy, and SHAP provides more faithful and transferable explanations.