Tag
This paper diagnoses 'pass@k inversion' in reinforcement learning with verifiable rewards (RLVR), where training improves one-shot accuracy but reduces performance under repeated sampling, particularly on boundary prompts where correct trajectories are rare. It proposes a per-problem base anchoring (PBA) method to mitigate this issue.
Presents FiT, a diagnostic framework to evaluate small LLMs on cybersecurity QA capabilities before fine-tuning, showing that fine-tuning can degrade vocabulary and parametric knowledge depending on the regime. Provides guidance to avoid unnecessary fine-tuning.
SAAG proposes a cascaded diagnostic framework for evaluating LLM agent function calling by decomposing evaluation into registry conformance, structural completeness, and argument grounding stages, enabling interpretable diagnostics and iterative self-repair. Experiments with sub-4B models show improved argument precision and reduced value hallucination compared to single-pass evaluation.
ToolSense is an open-source diagnostic framework that generates three benchmarks (realistic retrieval, MCQ probing, QA probing) to audit LLMs' parametric tool knowledge, revealing a knowledge-retrieval dissociation where strong retrieval performance can coexist with poor factual understanding.
This paper presents a corpus-centric diagnostic framework for analyzing biomedical NER and EL benchmarks, revealing substantial differences across nine corpora and arguing that standard statistics are insufficient for characterizing evaluation demands.