human-alignment

该论文提出一种无标签的评分标准（rubric）修复流程：通过跨评审模型的分歧自动定位有争议的评分条款并由编辑模型重写。实验表明该方法提升了评审间一致性（Krippendorff's α 提升约 0.07），但并未提升与人类专家评分的一致性，甚至点估计略低于随机条款编辑对照组。

This paper evaluates large language models' politeness judgments against human pragmatic norms, finding that inter-model agreement exceeds model-human alignment and reveals systematic biases like overproducing neutral labels.

The paper evaluates local open-weight LLM judges against human ratings, finding high self-consistency but limited agreement with human judgments, highlighting the need for dual assessment.

A paper finds that reasoning models and humans struggle with similar commonsense problems, and that chain-of-thought token count correlates with human response time when averaged across multiple runs, making it a more reliable signal of problem difficulty.

SVG-Score introduces a human-aligned evaluation framework for text-to-SVG generation, addressing limitations of current metrics like CLIPScore by developing specialized evaluators and benchmarks.

This paper examines the alignment of thinking effort between humans and large reasoning models in abductive reasoning, finding evidence of shared effort and similar errors, and demonstrates that decoding methods increase this alignment.

This paper presents an LLM-as-a-Judge evaluation framework for agentic AI in drug discovery, validated through human alignment studies with expert annotators. It optimizes the judge to improve alignment with human judgment and provides insights for reusable evaluation in scientific domains.

This paper proposes a longitudinal memory framework called LifeMem to mitigate identity essentialism in LLM agents, improving their alignment with human data in social simulations by enhancing diversity and dynamic evolution.

This paper investigates when and why LLMs and humans converge or diverge in evaluating creativity, finding that alignment depends on the dimension (stronger for novelty, weaker for context) and that different LLMs apply different standards.

This paper introduces TheraJudge and TheraAgent, a framework that uses multi-dimensional human-aligned evaluation to improve therapeutic response generation in LLMs, showing significant gains in quality and safety.

Counsel is the first public dataset of human meta-evaluations of LLM critiques for agentic tasks, designed to improve the calibration and reliability of automated evaluation methods.

This paper presents simple prompting strategies that help large language models better capture the full distribution of human judgments, improving alignment on moral scenarios and beliefs. The authors show that asking models to report standard deviations and response proportions, along with ensuring scenario clarity, yields better agreement with human responses.

This paper geometrically analyzes why LLMs acting as judges agree strongly with each other but weakly with humans, finding that inter-LLM consensus reflects a collapsed subspace rather than true human alignment on subjective rubrics. Post-hoc calibration on human data improves alignment, but even calibrated LLMs fall short of human reliability.

This paper investigates how similar large language model uncertainty is to human uncertainty, exploring alignment, calibration, and activation patterns in LLMs across multiple datasets and the impact of instruction fine-tuning.

This paper empirically evaluates the alignment between LLM-generated and human reviews for scientific papers, finding limited and variable alignment. It also shows that authors can 'game' LLM reviews by iteratively revising papers to improve scores, with up to 35% of papers seeing statistically significant score increases.

JobBench is a benchmark built from worker surveys to evaluate AI agents on tasks that workers most want automated, covering 130 tasks across 35 professions with detailed rubrics.

This paper investigates the alignment of LLM-generated reviews with human judgment using 1k real ACL 2025 submissions, finding limited agreement, instability across models/prompts, and a method to artificially inflate scores without meaningful changes. The authors advise against relying solely on LLM reviews and call for discussion on their use in handling increasing submission volumes.

This paper studies the problem of learning to make optimal decisions with AI assistance under human-alignment, showing that alignment can reduce the complexity of learning, and provides regret bounds.

Google DeepMind published a paper in Nature detailing a method to align AI visual representations with human cognitive structures, improving model robustness and reliability.