Tag
提出一个统一框架，将学生对题目的作答与辅导对话中的子任务建模在共享的IRT量表上，通过从文本估计题目难度并利用辅导策略正则化难度值，在真实数学教育数据集上超越强基线。
The paper introduces reverse Item Response Theory to pharmacogenomics, treating cancer types as latent "subjects" and drugs as "items" to achieve sparsity-robust ranking recovery in fragmented drug-response matrices, outperforming simple averaging on GDSC2 data across multiple missingness regimes.
The paper introduces Rubric Response Theory (RRT), which replaces additive rubric scoring with a two-parameter item response model plus a Response Parameter Network (RPN) updated via online EM, achieving higher macro criterion scores than GRPO on Medical and Science benchmarks while halving judge requests.
This paper presents efficient methods for benchmarking production LLM agents, demonstrating that adaptive testing based on item response theory can reduce evaluation costs while maintaining accuracy, with practical deployment insights for recurring assessments.
IXPLORE is a bounded ideal point estimation algorithm that improves reconstruction and imputation performance in political data analysis, particularly for sparse responses, with grid-based uncertainty quantification.
This research investigates whether near-tied rankings of large language models remain robust to changes in benchmark item composition using psychometric methods, finding that while overall rankings are stable, individual close orderings can reverse based on item selection.
BenchMIRT introduces a method to audit LLM benchmarks at the individual prompt level using multidimensional item response theory, separating underlying capabilities like safety and general reasoning to reveal what benchmarks actually measure.
This paper describes using a fine-tuned multimodal LLM based on Qwen3.5 to simulate student response probabilities and estimate item difficulty parameters for multiple-choice assessments, approximating 3PL and MCM curves.
This paper applies Item Response Theory to eight safety benchmarks across 192 language models, identifying three latent factors, enabling 97-99% cost reduction via adaptive testing, and supporting sandbagging detection and model auditing.
This paper introduces LLM-NRM, an option-level psychometric framework for multiple-choice benchmarks that models the full distribution over answer choices rather than binary correctness, showing that incorrect responses carry useful measurement information and improving ability estimation and benchmarking efficiency.
This paper introduces CurveShift, an analysis method that separates overall ability gains from difficulty-specific improvements in LLM agents. Using METR time-horizon data and LiveCodeBench, it finds that most apparent shifts toward harder tasks are ceiling effects, though a genuine hard-task effect exists for reasoning models in competitive programming.
CalibratedRubric is a task-adaptive framework for building compact, measurable rubric banks for open-ended LLM evaluation, using Bayesian measurability filtering and IRT-based selection to improve human-gold agreement and rank fidelity across financial, healthcare, general, and legal benchmarks.
This paper investigates how LLM-generated difficulty ratings for math items align with actual student performance, finding that LLMs systematically underestimate difficulty for items driven by learner misconceptions, a phenomenon termed the 'Easy Trap'.
This paper proposes an evaluation framework for predicting item parameters from text embeddings using regularized regression and reliability/design ceilings. Results show that difficulty is substantially predictable from text, while discrimination and pseudo-guessing are limited by reliability, not text signal.
This paper introduces Multilingual-IRT, a statistical framework extending Item Response Theory with per-language difficulty deviations and split discriminability, enabling efficient prediction of unobserved evaluations, detection of translation errors, and recovery of culture-specific items across 29 languages.
Introduces Item Response Scaling Laws (IRSL) that integrates Item Response Theory to efficiently estimate neural scaling laws, reducing required evaluation questions by 99.9% while achieving comparable accuracy.
This paper introduces an Item Response Theory-based method to detect mislabeled examples in LLM benchmarks at 95% precision, tracing errors to labeling heuristics and annotation issues.
This paper introduces a Multi-Group Item Response Theory framework to decouple factors behind safety degradation in non-English languages, revealing that safety is primarily unidimensional and that low-resource languages produce more uncertain responses.
This paper argues that simple averaging in AI benchmarks fails under data sparsity and difficulty heterogeneity, proposing Item Response Theory (IRT) as a robust alternative to recover ground truth rankings.