Tag
Emory University researchers introduce PsyWorld, a profile-conditioned state world model that uses observed environment–behavior state transitions to learn transition-informed representations for predicting human behavior, achieving 0.904 Top-1 accuracy on the VEET behavioral simulator with participant-disjoint generalization.
This paper introduces the Schema Coherence Inventory (SCI), a 96-item forced-choice psychometric tool measuring whether beliefs assigned to persona-conditioned language models actually organize their cognition versus merely inducing a mood, validated across 107 conditions and five model backbones.
The paper introduces SkillEval, a framework that combines LLM-discovered interpretable skills with cognitive diagnosis models to produce fine-grained ability profiles of LLMs across benchmarks, improving on average-score evaluation and enabling LLM routing and adaptive testing.
EviDial introduces a benchmark that separates label agreement from evidence-grounded assessment in depression dialogue evaluation, revealing that 82.8% of participant-symptom cells lack sufficient textual evidence and that 69.2–96.9% of seemingly correct LLM predictions are actually unsupported by the dialogue.
该论文对决策模型(decision models)和最新生成式 LLM 作为合成问卷受访者的能力进行了心理测量审计,在 14 个决策模型与 47 个生成式模型上测试后发现,模型在样本层面的模拟保真度尚可,但无法替代真实受访者的个体回答,且新一代模型未必优于旧版本。
该论文从心理测量学视角评估 LLM-as-a-judge,使用 Many-Facet Rasch 模型将评分分解为潜在质量、评分者严苛度与残差难度,发现人类与 LLM 在汇总对齐之外仍存在明显的残差难度结构错配。
IQ Tester is an AI agent that estimates cognitive profiles with 30 adaptive questions, delivering detailed assessment reports based on psychometric frameworks across multiple reasoning domains.
This research investigates whether near-tied rankings of large language models remain robust to changes in benchmark item composition using psychometric methods, finding that while overall rankings are stable, individual close orderings can reverse based on item selection.
BenchMIRT introduces a method to audit LLM benchmarks at the individual prompt level using multidimensional item response theory, separating underlying capabilities like safety and general reasoning to reveal what benchmarks actually measure.
Introduces an analytically exact framework for controlled behavioral evaluation of LLMs, using fully crossed factorial experiments and exact token-level probability mass functions to isolate causal biases that aggregate benchmarks obscure.
This paper describes using a fine-tuned multimodal LLM based on Qwen3.5 to simulate student response probabilities and estimate item difficulty parameters for multiple-choice assessments, approximating 3PL and MCM curves.
This paper introduces NLP Psychometrics, a framework that treats psychological prediction from text as a psychometric problem. Using LLM personas, emotional profiles, and syntactic-semantic networks with random forest regressors, it explains up to 76% of variance in mental health scores and shows promise and limits of synthetic data for psychometric prediction.
This paper applies Item Response Theory to eight safety benchmarks across 192 language models, identifying three latent factors, enabling 97-99% cost reduction via adaptive testing, and supporting sandbagging detection and model auditing.
This paper introduces LLM-NRM, an option-level psychometric framework for multiple-choice benchmarks that models the full distribution over answer choices rather than binary correctness, showing that incorrect responses carry useful measurement information and improving ability estimation and benchmarking efficiency.
CogArena introduces a procedurally generated 13-paradigm benchmark to evaluate whether LLMs exhibit separable cognitive abilities or a single general competence, finding only weak support for stable five-dimensional profiles across 55 models.
This paper validates the Spanish version of the Large Language Models Dependency Scale (LLM-D12-SP), confirming its two-factor structure and good psychometric properties.
This paper proposes a fine-tuned multi-agent framework for detecting OCEAN personality traits from life narratives, using LLM sub-agents conditioned to adopt high, low, or neutral perspectives and a judge LLM that aggregates outputs to mitigate biases and improve interpretability.
This paper proposes an evaluation framework for predicting item parameters from text embeddings using regularized regression and reliability/design ceilings. Results show that difficulty is substantially predictable from text, while discrimination and pseudo-guessing are limited by reliability, not text signal.
This paper introduces a psychometric battery to separate framing artifacts from genuine moral judgment in LLMs, finding that frontier models have a coherent internal moral scale but display a yes/no bias that is purely a surface-level artifact of answer order and wording, not a real disposition to reject.
This paper presents a psychometric audit of large language models as synthetic survey respondents, finding that they fail to match human joint distributions and reliability, making them unsuitable replacements.