psychometrics

Tag

Cards List
#psychometrics

PsyWorld: Profile-Conditioned World Modeling for Human Behavior Prediction

arXiv cs.LG ↗ · 2d ago Cached

Emory University researchers introduce PsyWorld, a profile-conditioned state world model that uses observed environment–behavior state transitions to learn transition-informed representations for predicting human behavior, achieving 0.904 Top-1 accuracy on the VEET behavioral simulator with participant-disjoint generalization.

0 favorites 0 likes
#psychometrics

The Schema Coherence Inventory: Measuring Whether an Assigned Belief Organizes a Language Model's Cognition

arXiv cs.CL ↗ · 2026-10-26

This paper introduces the Schema Coherence Inventory (SCI), a 96-item forced-choice psychometric tool measuring whether beliefs assigned to persona-conditioned language models actually organize their cognition versus merely inducing a mood, validated across 107 conditions and five model backbones.

0 favorites 0 likes
#psychometrics

SkillEval: Learning Interpretable Ability Profiles of LLMs via Cognitive Diagnosis Models

arXiv cs.CL ↗ · 2026-10-22 Cached

The paper introduces SkillEval, a framework that combines LLM-discovered interpretable skills with cognitive diagnosis models to produce fine-grained ability profiles of LLMs across benchmarks, improving on average-score evaluation and enabling LLM routing and adaptive testing.

0 favorites 0 likes
#psychometrics

EviDial: When Correct Depression Predictions Lack Evidence

arXiv cs.CL ↗ · 2026-10-21 Cached

EviDial introduces a benchmark that separates label agreement from evidence-grounded assessment in depression dialogue evaluation, revealing that 82.8% of participant-symptom cells lack sufficient textual evidence and that 69.2–96.9% of seemingly correct LLM predictions are actually unsupported by the dialogue.

0 favorites 0 likes
#psychometrics

Probable but Not Valid: A Psychometric Audit of Decision Models as Synthetic Survey Respondents

Hugging Face Daily Papers ↗ · 2026-10-10 Cached

该论文对决策模型(decision models)和最新生成式 LLM 作为合成问卷受访者的能力进行了心理测量审计,在 14 个决策模型与 47 个生成式模型上测试后发现,模型在样本层面的模拟保真度尚可,但无法替代真实受访者的个体回答,且新一代模型未必优于旧版本。

0 favorites 0 likes
#psychometrics

Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty

arXiv cs.CL ↗ · 2026-10-05 Cached

该论文从心理测量学视角评估 LLM-as-a-judge,使用 Many-Facet Rasch 模型将评分分解为潜在质量、评分者严苛度与残差难度,发现人类与 LLM 在汇总对齐之外仍存在明显的残差难度结构错配。

0 favorites 0 likes
#psychometrics

@JenovaAIAgent: IQ Tester is an AI agent that estimates your cognitive profile with 30 adaptive questions across seven reasoning domain…

X AI KOLs Following ↗ · 2026-09-23 Cached

IQ Tester is an AI agent that estimates cognitive profiles with 30 adaptive questions, delivering detailed assessment reports based on psychometric frameworks across multiple reasoning domains.

0 favorites 0 likes
#psychometrics

Are Near-Tied LLM Rankings Robust to Family-DIF-Guided Benchmark Recomposition?

arXiv cs.CL ↗ · 2026-09-02 Cached

This research investigates whether near-tied rankings of large language models remain robust to changes in benchmark item composition using psychometric methods, finding that while overall rankings are stable, individual close orderings can reverse based on item selection.

0 favorites 0 likes
#psychometrics

BenchMIRT: What are LLM benchmarks actually measuring?

Hugging Face Blog ↗ · 2026-09-01 Cached

BenchMIRT introduces a method to audit LLM benchmarks at the individual prompt level using multidimensional item response theory, separating underlying capabilities like safety and general reasoning to reveal what benchmarks actually measure.

0 favorites 0 likes
#psychometrics

Every Token Counts: Exact Likert-Scale Distributions for Measuring LLM Attitudes and Biases

arXiv cs.CL ↗ · 2026-08-12 Cached

Introduces an analytically exact framework for controlled behavioral evaluation of LLMs, using fully crossed factorial experiments and exact token-level probability mass functions to isolate causal biases that aggregate benchmarks obscure.

0 favorites 0 likes
#psychometrics

Multimodal Item Parameter Estimation using Simulated Response Probabilitie

arXiv cs.CL ↗ · 2026-08-12 Cached

This paper describes using a fine-tuned multimodal LLM based on Qwen3.5 to simulate student response probabilities and estimate item difficulty parameters for multiple-choice assessments, approximating 3PL and MCM curves.

0 favorites 0 likes
#psychometrics

Natural Language Processing Psychometrics

arXiv cs.CL ↗ · 2026-08-10 Cached

This paper introduces NLP Psychometrics, a framework that treats psychological prediction from text as a psychometric problem. Using LLM personas, emotional profiles, and syntactic-semantic networks with random forest regressors, it explains up to 76% of variance in mental health scores and shows promise and limits of synthetic data for psychometric prediction.

0 favorites 0 likes
#psychometrics

Item Response Theory for AI Safety

arXiv cs.AI ↗ · 2026-08-06 Cached

This paper applies Item Response Theory to eight safety benchmarks across 192 language models, identifying three latent factors, enabling 97-99% cost reduction via adaptive testing, and supporting sandbagging detection and model auditing.

0 favorites 0 likes
#psychometrics

Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks

arXiv cs.CL ↗ · 2026-08-05 Cached

This paper introduces LLM-NRM, an option-level psychometric framework for multiple-choice benchmarks that models the full distribution over answer choices rather than binary correctness, showing that incorrect responses carry useful measurement information and improving ability estimation and benchmarking efficiency.

0 favorites 0 likes
#psychometrics

CogArena: A Multimethod Evaluation of Cognitive Ability Structure in Large Language Models

arXiv cs.CL ↗ · 2026-07-29 Cached

CogArena introduces a procedurally generated 13-paradigm benchmark to evaluate whether LLMs exhibit separable cognitive abilities or a single general competence, finding only weak support for stable five-dimensional profiles across 55 models.

0 favorites 0 likes
#psychometrics

Developing and Validating the Spanish Version of the Large Language Models Dependency Scale (LLM-D12-SP)

arXiv cs.CL ↗ · 2026-07-27 Cached

This paper validates the Spanish version of the Large Language Models Dependency Scale (LLM-D12-SP), confirming its two-factor structure and good psychometric properties.

0 favorites 0 likes
#psychometrics

Fine-Tuned Multi-Agent Framework for Detecting OCEAN in Life Narratives

arXiv cs.CL ↗ · 2026-07-15 Cached

This paper proposes a fine-tuned multi-agent framework for detecting OCEAN personality traits from life narratives, using LLM sub-agents conditioned to adopt high, low, or neutral perspectives and a judge LLM that aggregates outputs to mitigate biases and improve interpretability.

0 favorites 0 likes
#psychometrics

From Text to Parameters: Predicting Item Parameters from Embedding Regularization with Reliability and Design Ceilings

arXiv cs.CL ↗ · 2026-07-09 Cached

This paper proposes an evaluation framework for predicting item parameters from text embeddings using regularized regression and reliability/design ceilings. Results show that difficulty is substantially predictable from text, while discrimination and pseudo-guessing are limited by reliability, not text signal.

0 favorites 0 likes
#psychometrics

The yes-no bias of large language models reflects answer order and wording, not shifts in moral judgment

arXiv cs.CL ↗ · 2026-07-08 Cached

This paper introduces a psychometric battery to separate framing artifacts from genuine moral judgment in LLMs, finding that frontier models have a coherent internal moral scale but display a yes/no bias that is purely a surface-level artifact of answer order and wording, not a real disposition to reject.

0 favorites 0 likes
#psychometrics

Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents

Hugging Face Daily Papers ↗ · 2026-07-06 Cached

This paper presents a psychometric audit of large language models as synthetic survey respondents, finding that they fail to match human joint distributions and reliability, making them unsuitable replacements.

0 favorites 0 likes
Next →
← Back to home

Submit Feedback