Toward a Theory of Value in AI Alignment

# Toward a Theory of Value in AI Alignment
Source: [https://arxiv.org/html/2608.10327](https://arxiv.org/html/2608.10327)
###### Abstract

Can AI systems be aligned to human values? The popularization of large language models \(LLMs\) and multi\-modal foundation models has seen a rise harms spanning from toxic speech and hallucinations to AI agents executing unauthorized actions\. Within the field of AI safety, these harmful instances are often framed as “the alignment problem,” of models being “misaligned” with human values\. Researchers have responded by pursuing applied and theoretical AI “value alignment” efforts, often without specifying what they mean by human values\. How does the field of AI value alignment conceive of human values? How are these conceptions of values technically operationalized and evaluated? What does the emergent theory of value from this field signify for the future of AI?

We annotated 94 AI value alignment research papers to discern their implicit theory of values in AI\. The majority do not define values, relying heavily on “preferences” as a stand\-in that runs the risk of reducing complex, culturally situated concepts down to binary choices\. As researchers dispense with using human annotators for model training and evaluation, turning instead to synthetic data and LLM\-as\-a\-judge approaches to aligning and evaluating models, we identify the potential to close off alternative methods for contesting and enacting values in foundation models\. In making AI value alignment’s philosophical commitments explicit, we seek to bring greater specificity and under\-explored perspectives into the debate on whether and how AI can address human values\.

## Introduction

Large language models \(LLMs\) and multi\-modal AI models have transformed almost all domains of social, economic, and scientific practices from research projects to consumer products\(Gillespieet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib168)\)\. One area of concern is that, as these systems become more complex and widespread, it can be more difficult to understand, predict, and control their actions, potentially leading them to exhibit harmful outputs that were unintended by the models’ human creators\. However, as Guest et al point out, model designers do know the mechanistic structure of these models because they designed and built them, and to claim ignorance of how neural networks function is a form of mysticism\(Guest[2026](https://arxiv.org/html/2608.10327#bib.bib252)\)\. Within the research field of AI safety, a wide swath of instances of models deviating from human intentions to cause harm is referred to as ”the alignment problem,” often treated as the most important unsolved problem in machine learning\. In recent years, AI alignment research has emerged as a leading strand of AI safety research designed to enact a contested belief: that catastrophes where misaligned AI systems will evade human control and inflict irreversible damage can be averted if ML systems are ‘aligned’with human values\(Gabriel[2020](https://arxiv.org/html/2608.10327#bib.bib4); Jiet al\.[2023](https://arxiv.org/html/2608.10327#bib.bib117); Askellet al\.[2021](https://arxiv.org/html/2608.10327#bib.bib118); Russell[2019](https://arxiv.org/html/2608.10327#bib.bib96)\)\. A large technical literature now concerns ”value alignment” in AI, yet fundamental questions about what human values are, and with whose values AI should be aligned, remain under\-examined\. In this paper, we conducted a analysis of 94 AI alignment papers to reveal the emerging literature’s underlying theory of what constitutes human values in ML models\. To do so, we developed a rubric that draws from the study of values in anthropology, philosophy and other social sciences, which we then used to assess the papers in our dataset\. We build on recent recognition of these foundational problems in AI alignment research\(Zhi\-Xuanet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib5); Arzbergeret al\.[2024](https://arxiv.org/html/2608.10327#bib.bib240)\)\. Before norms and practices in the value alignment subfield begin to ossify, our paper serves as a critical intervention to reassess the epistemic claims, methods, and oversights that are coming to define this work, and as an invitation to include the perspectives of more interdisciplinary scholarship and perspectives\.

## Related Work

### What is value alignment?

Why should alignment be necessary in the first place? Fundamentally, the training objective of LLMs is to correctly predict how an incomplete piece of text will continue\. However, success in this training objective does not mean that the model can perform well in any arbitrary downstream task, or that the model is ‘safe’to use\(Hooker[2025](https://arxiv.org/html/2608.10327#bib.bib197)\)\. Value alignment focuses on ‘steering’an AI model toward outputs that humans \(and increasingly other AI models\(Sharmaet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib124)\)\) judge as in accordance with certain preferences, e\.g\., helpfulness, ‘not being racist’, safety, or fairness, through fine\-tuning methods including reinforcement learning from human feedback \(RLHF\)\(Christianoet al\.[2017a](https://arxiv.org/html/2608.10327#bib.bib112); Baiet al\.[2022](https://arxiv.org/html/2608.10327#bib.bib64); Chaudhariet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib137)\), inverse reinforcement learning \(IRL\)\(Hadfield\-Menellet al\.[2016](https://arxiv.org/html/2608.10327#bib.bib55)\), direct preference optimization \(DPO\)\(Amirlooet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib207)\), Bradley\-Terry\-based models, and related techniques\(Baiet al\.[2022](https://arxiv.org/html/2608.10327#bib.bib64); Hofmannet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib63); Hadfield\-Menellet al\.[2016](https://arxiv.org/html/2608.10327#bib.bib55); Ouyanget al\.[2022](https://arxiv.org/html/2608.10327#bib.bib123); Sunet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib133); Chaudhariet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib137)\)\. Within this paradigm, value alignment is reformulated as encoding the preferences demonstrated by a human into a reward function, and updating the parameters of the language model to produce output that maximizes this reward\. The assumption is that the AI can learn what human values and utility functions are by observing their behavior\(Hadfield\-Menellet al\.[2016](https://arxiv.org/html/2608.10327#bib.bib55)\)\.

In public\-facing discourse, AI safety researchers project the future possibility that misaligned foundation models could invoke damage ranging from unauthorized financial transactions to taking over power grids\. In worst\-case scenarios referred to as ‘existential risk’\(x\-risk\), some in the AI safety community fear that rogue LLMs could kill most of the human population\(Kasirzadeh[2024b](https://arxiv.org/html/2608.10327#bib.bib193)\)\. Others believe that such an event could lead to human extinction\(Nauer and van Doren[2025](https://arxiv.org/html/2608.10327#bib.bib194)\)\. On the flip side, many of these same people claim that if AI systems can be ‘aligned’, this could bring about a utopia where human beings would no longer have to work\. Yet despite the high stakes that alignment researchers place on ‘solving’alignment, research on value alignment rarely makes an effort to define values, let alone to center the analysis of pre\-existing human values as part of AI safety research\.

Regardless of whether one believes in x\-risk scenarios, value alignment is not merely a theoretical issue to be resolved within the machine learning community\. The current state of value alignment research has real\-world implications\. Corporations are marketing their ‘value\-aligned’AI products to governments around the world\. For example, the European Parliament uses Anthropic’s LLMs to provide access to its archives\. The Parliament’s choice was based on Anthropic’s claim that its Constitutional AI is value\-aligned\. Although there has been no independent evaluation of Anthropic’s approach, the claim of value alignment has been sufficient for governments to accept that these LLMs are legally compliant\(Shrishak[2025](https://arxiv.org/html/2608.10327#bib.bib172)\)\. Elsewhere, governments are funding alignment research, such as the GBP £15 million Alignment Project within the UK’s AI Security Institute\(Institute[2025](https://arxiv.org/html/2608.10327#bib.bib173)\)\.

To date, value alignment is predominantly a technical field even though its underlying premise is sociotechnical\. Yet foundation models function as epistemic technology that embeds normative models of knowledge within computational design\(Alvarado[2023](https://arxiv.org/html/2608.10327#bib.bib249)\)while also serving as a product of the social and cultural milieus that produce them\. Alignment proponents advocate for encoding human values, moral principles or objectives in AI systems, most often via incorporating human feedback\. For instance, Anthropic, the AI firm that produced the chatbot Claude, has published influential research positing that LLMs should exhibit traits such as helpfulness, honesty and harmlessness \(HHH\)\(Askellet al\.[2021](https://arxiv.org/html/2608.10327#bib.bib118)\)\. Others in the field attempt to guide LLMs to adhere to principles such as robustness, interpretability, controllability, and ethicality\(Jiet al\.[2023](https://arxiv.org/html/2608.10327#bib.bib117)\), or safety, quality and groundedness\(Thoppilanet al\.[2022](https://arxiv.org/html/2608.10327#bib.bib157)\)\. Many researchers default to HHH and other off\-the\-shelf values that are associated with training datasets and benchmarks, rarely questioning whether these values are indeed enacted in these technical artifacts\.

### Alignment and its discontents

As alignment crystallizes into a core research wing of AI safety, it is crucial to examine the roots of this work through fundamental questions around the very idea of value alignment\. What are the core assumptions and feasibility of this project? Is it a meaningful avenue for fair, transparent, and accountable AI development? Most alignment research has examined how to encode moral values into AI models to guide their behavior\(Bergmanet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib122)\)\. The question ofwhat moral values are, however, is left undefined or under\-specified\(Gabriel[2020](https://arxiv.org/html/2608.10327#bib.bib4)\)\. Existing alignment approaches tend to rely on universal framings of human values that obscure the question of which values the systems should capture and align with, despite the variety of operational situations in which these systems are deployed\(Arzbergeret al\.[2024](https://arxiv.org/html/2608.10327#bib.bib240)\)\.

Value alignment efforts suffer from vagueness, lack of internal consistency, and lack of commitment to establishing clear guidelines for how to determine what is acceptable AI system behavior\(Lindströmet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib119); Arvan[2024](https://arxiv.org/html/2608.10327#bib.bib158)\)\. The term ‘values’is often left undefined in alignment literature, its meaning assumed to stem from taken\-for\-granted background knowledge\. Moreover, crucial questions around ‘whose values’are hardly addressed\. The absence of transparency around these implicit ‘values’, how they are derived, and how they are implemented in and itself a problem\. Even though the importance of the diversity of views \(‘pluralism’\) is sometimes acknowledged, ”to date the research and policy proposals coming out of this \[AI safety\] community have converged around a narrow set of technical solutions that do not engage with work that falls outside of the \[its\] ideological and disciplinary boundaries”\(Ahmedet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib161)\)\.

Tech industry alignment research markets the mission of alignment as ”benefiting humanity” without acknowledging the vast diversity of the human experience\. Major AI firms have in\-house alignment experts, blogs\(OpenAI[2025](https://arxiv.org/html/2608.10327#bib.bib175)\), and teams\(Anthropic[2025](https://arxiv.org/html/2608.10327#bib.bib176)\)\. In response to how corporate approaches to AI alignment reverberate across the field, one critique of alignment’s shortcomings pertains to the profit motive driving the companies behind the most widely used models: ”the existence of financial incentives means that alignment work often turns into product development in disguise rather than actually making progress on mitigating long\-term harms”\(Dai[2025](https://arxiv.org/html/2608.10327#bib.bib174)\)\.

Furthermore,\(Lindströmet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib119)\)contend that even when objectives such ‘harmlessness’are identified as a chief aim, the nuanced nature of harm is oversimplified and operationalized in a way that is internally inconsistent and vague\. Due to superficial understanding of ethical behavior in AI systems, what is often sought is the ”least harmful option rather than striving to understand the deeper roots of harm and addressing these to prevent it”\(Lindströmet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib119)\)\. The phrase ’value alignment’ lacks an agreed\-upon meaning\. Measurements of the degree to which a system is value\-aligned are subjective at best\(Khlaaf[2023](https://arxiv.org/html/2608.10327#bib.bib160)\)\. According toKirket al\.\([2023b](https://arxiv.org/html/2608.10327#bib.bib159)\), ’alignment’ is an empty signifier that serves as a ”rhetorical placeholder for an aspirational conceptualisation”\. A lack of cohesion according toKhlaaf \([2023](https://arxiv.org/html/2608.10327#bib.bib160)\)has led to contradictory approaches that conflate safety properties with system requirements\.

The narrow focus on internal technical components and emphasis on the mathematical formulation, including the objective or reward function is another limitation of value alignment efforts\. Contrary to common assumptions within value alignment research, harms and failures due to accidents are notunanticipated emergent behaviorsbut rather abyproduct of basic design choice, resource requirements data, and compute requirements, as well as designers’ and developers’ API model delivery decisions\(Raji and Dobbe[2023](https://arxiv.org/html/2608.10327#bib.bib140); Dobbe[2022](https://arxiv.org/html/2608.10327#bib.bib141)\)\. Given that AI systems are sociotechnical systems that consist of technical AI artifacts, human agents, and institutions, comprehensively addressing safety issues requires thorough measures that include addressing how a system is used in practice and interacts with other \(human\) agents, systems, and its broader environment\(Dobbe[2022](https://arxiv.org/html/2608.10327#bib.bib141)\)\. Value alignment’s current technical focus, along with the framing that treats ’misaligned’ AI as a danger to human survival if not ’controlled and steered,’ serves those who are developing and deploying AI systems to evade accountability\(Gebru and Torres[2024](https://arxiv.org/html/2608.10327#bib.bib162)\)\.

### Preferences are not all you need

Although researchers are beginning to contend with problematic assumptions and conceptions of value underlying AI value alignment\(Johnson[2023](https://arxiv.org/html/2608.10327#bib.bib44); Lindströmet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib119); Casperet al\.[2023](https://arxiv.org/html/2608.10327#bib.bib206)\), one overlooked flaw we explicate is the field’s implicit grounding in economic theory\. In practice, value alignment’s reliance on economic theory manifests as an emphasis on rational choice theory and ”expected utility” \(which is the average ”reward” or ”payoff” of a decision or choice\)\(Becker[1976](https://arxiv.org/html/2608.10327#bib.bib254); Russell and Norvig[1995](https://arxiv.org/html/2608.10327#bib.bib195); Chibnik[2011](https://arxiv.org/html/2608.10327#bib.bib102)\)\. Model developers’ decision to substitute values with preferences in reinforcement learning from human feedback \(RLHF\), direct preference optimization \(DPO\)\(Rafailovet al\.[2023](https://arxiv.org/html/2608.10327#bib.bib208); Amirlooet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib207)\)or via AI feedback \(RLAIF\)\(Jiet al\.[2023](https://arxiv.org/html/2608.10327#bib.bib117)\)is likewise based on revealed preference theory from economics\. This theory posits that an agent’s values or goals can be inferred from observing their behavior–an assumption that has AI researchers have largely adopted without considering alternatives\(Casperet al\.[2023](https://arxiv.org/html/2608.10327#bib.bib206)\)\. More recent approaches such as DPO obviate the need for a reward model, but nonetheless use preference datasets\(Rafailovet al\.[2023](https://arxiv.org/html/2608.10327#bib.bib208)\)\.

If the assumptions underlying economic theory are correct, this process should ”align” AI with human values\. However, relying on highly abstract, reductionist, fictionalized, and idealized concepts of rational agents in lieu of a deep understanding of human values has been central to AI for decades\(Russell and Norvig[1995](https://arxiv.org/html/2608.10327#bib.bib195)\)\. This dominant technical paradigm in AI alignment is whatZhi\-Xuanet al\.\([2024](https://arxiv.org/html/2608.10327#bib.bib5)\)term the ”preferentist” approach to value alignment, where people’s preferences are translated into data points to provide aggregate guidance about preferred outcomes\(Zhi\-Xuanet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib5); Gabriel[2020](https://arxiv.org/html/2608.10327#bib.bib4)\)\. This approach has three core assumptions: 1\) that preferences are an adequate representation of human values, 2\) that rationality consists of maximizing the satisfaction of preferences, and 3\) that aligning AI models to data about these preferences is a sound technique for making AI safe\(Zhi\-Xuanet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib5)\)\.

A widely recognized problem is that within the preferentist paradigm, even with pluralist frameworks, current methods for fine\-tuning language models from human preferences treat these preferences \(and the unexamined values shaping the preferences\) as if they were homogeneous and static\(Bakkeret al\.[2022](https://arxiv.org/html/2608.10327#bib.bib165); Klassenet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib30); Sorensenet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib2); Lindströmet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib119)\)\.

More fundamentally, the very project of reliably aligning LLM behavior with human values has been criticized as provably impossible\(Arvan[2024](https://arxiv.org/html/2608.10327#bib.bib158)\)\.Arvan \([2024](https://arxiv.org/html/2608.10327#bib.bib158)\)argues that not only is there persistent human disagreement over moral values and principles, both among the public and moral theorists, but also that LLMs are complex given that they contain finite series of observational data and an infinite number of ‘misaligned’ functions\. In other words, there is always a vastly higher empirical probability that an LLM will diverge from what appears to be an ‘aligned function’ to some ‘misaligned’ one later\. ”‘Alignment’, thus is not a ”solvable” engineering or safety problem, but rather a quixotic and potentially dangerous fantasy based on a series of philosophical misunderstandings about empirical evidence”\(Arvan[2024](https://arxiv.org/html/2608.10327#bib.bib158)\)\.

## Statement of Contributions

Motivated by the issues outlined in the above discussion, we construct and analyze a sample of AI value alignment literature to critically assess how researchers typically operationalize value\. The question we raise in this paper is: what implicit theories of human values underpin the field of AI value alignment?

Our contributions are:

- •We develop a framework for analyzing the theory of value in AI alignment, inspired by interdisciplinary literature in the social sciences and philosophy\.
- •Using this framework, we conduct a qualitative study of how value is theorized from a snowballed sample of 100 commonly cited AI alignment papers, seeded by four canonical papers in the field\.
- •We form an initial hypothesis of where AI value alignment’s theory of value sits, based on our critical observations of the field\.
- •We briefly outline alternative theories of human values as potential paths forward for thinking about how to mitigate harms from large AI systems\.

## Methods

To characterize the dominant paradigm of value in AI research, we conducted a study of influential AI alignment papers, quantifying their philosophical positions using a scoring rubric developed from our above provocations\.

We chose 4 canonical ‘seed papers’ on AI value alignment based on our expert judgment and high Google Scholar citations counts \(\>1,000\>1,000\)\. The seed papers have been highly influential in defining the field of technical AI alignment\. Regarding the sample size and representativeness, we note that in the social sciences, when conducting close readings and qualitative analyses of texts, 94 papers is an acceptable sample\. We did not use automated methods to evaluate these papers\.

These 4 seed papers are:

- •Deep Reinforcement Learning from Human Preferences\(Christianoet al\.[2017b](https://arxiv.org/html/2608.10327#bib.bib135)\),
- •A General Language Assistant as a Laboratory for Alignment\(Askellet al\.[2021](https://arxiv.org/html/2608.10327#bib.bib118)\),
- •Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback\(Baiet al\.[2022](https://arxiv.org/html/2608.10327#bib.bib64)\),
- •Training language models to follow instructions with human feedback\(Ouyanget al\.[2022](https://arxiv.org/html/2608.10327#bib.bib123)\)

### Snowball sampling

Based on the 4 seed papers, we used a snowball sampling method\(Parker[2019](https://arxiv.org/html/2608.10327#bib.bib196)\)to automatically download papers that met the inclusion criteria based on ”value alignment”\. Using the Semantic Scholar API, all papers from 2016\-24 that cite these 4, with an initial corpus of 576 papers\.We sorted these from highest to lowest Google Scholar citations\. We sorted the papers by Google Scholar citation count as a weak proxy for the quality and impact of each paper\. During annotation, papers were discarded if they used the term ’value alignment’ only in a cursory sense, rather than addressing values as a central consideration of the paper\.After reaching consensus on which papers were about AI value alignment, out of the original 576 papers, we kept 94 for annotation\. The sample size was also chosen due to resource constraints: querying Google Scholar citation counts is time\-consuming, as is the paper annotation process itself\.

Paper selection, scoring, and annotation was conducted by the authors\. We used the annotation rubric to guide reading of the papers for scoring each question\. We kept detailed qualitative notes in one column of the dataset, using comments and selecting illustrative quotes and passages from each paper to support our qualitative analysis of this literature\.

### Annotation rubric

*Rubric\.*In order to characterize the theory of value underlying top cited AI alignment literature, we developed a rubric of questions based on a review of relevant literature in philosophy, sociology and anthropology that has not been considered within value alignment research\. All questions are ternary: the answer can be one of two specific options, or ”not applicable” if the answer is ambiguous or cannot be determined from the available information\.

Development of the rubric was iterative, responding to emergent themes we noticed early in the annotation process\. For example, we began developing our rubric with the assumption that papers ostensibly about ”value alignment” would engage with human values, and our initial rubric draft did not include the question of whether the papers under consideration even defined or engaged with human values\. Yet from a first pass at annotating papers, it became clear that we needed to first include the question of whether the paper defined human values because so few papers met this expectation\. We also realized that many of the papers in our sample relied on ”autoraters” or LLMs as stand\-ins for humans, and therefore we revised the rubric to include a question about whether actual humans were consulted about their values\.

#### Justification for rubric questions

One of the most fundamental distinctions philosophers make about values is betweenmonismandpluralism\. Monism is the view that there is asingle valuethat explains the value of everything \(e\.g\., happiness, or pleasure\) and that this single value can be ordered on acardinal scaleor a hierarchy of ends\(Korsgaard[1986](https://arxiv.org/html/2608.10327#bib.bib248)\)\. Strong monists claim that on this cardinal scale there are units of pleasure \(hedons or utils\) and the intervals between them is constant\. Monism about value corresponds with utilitarianism, economic utility and rational choice theory\(Mill[2016](https://arxiv.org/html/2608.10327#bib.bib247); Becker[1976](https://arxiv.org/html/2608.10327#bib.bib254)\)\.

Pluralismin contrast holds that there is not just a single value, that human actions might have multiple final ends, and that there are multiple things that are good in and of themselves such as love, knowledge, piety, and justice\(Haslanger[2023](https://arxiv.org/html/2608.10327#bib.bib19),[2022](https://arxiv.org/html/2608.10327#bib.bib143)\)\. Pluralism recognizes that values are culturally relative and fundamentally vary across societies\(Schmer\-Galunder[2026](https://arxiv.org/html/2608.10327#bib.bib251)\)\. Thus, a significant development in alignment research recently has been what we call ”the pluralist turn,” which seeks to develop approaches to accommodate diverse viewpoints and values\(Sorensenet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib2); Klassenet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib30); Kasirzadeh[2024a](https://arxiv.org/html/2608.10327#bib.bib33)\)\.

We incorporated questions about the quantifiability and measurability of values, and assessed papers for whether they view values as measurable\. The assumption that values are the equivalent of maximizing a utility function allows for the ostensible quantification of human values\. From anthropology we adapted the evaluative distinction betweenthinandthickconceptions of values\. Thick evaluative concepts and descriptions of human values view them not as mere abstractions, but as involving intentional, purposive detail that helps us understand those activities in their cultural and social contexts\(Geertz[2008](https://arxiv.org/html/2608.10327#bib.bib116)\)\.

Given the central role that economic theory plays in AI alignment, we also annotated papers for whether the alignment framework was rooted in utility theory\.

#### Rubric questions used to annotate the papers

1. 1\.Does the paper define or describevalue?
2. 2\.Does the paper’s theory of value subscribe to valuemonismorpluralism? Monism holds that there is a single, intrinsic dimension to value, while pluralism states that there is more than one dimension or consideration of value, which cannot be flattened\.
3. 3\.Does the paper consider valuesmeasurableorimmeasurable? Measurable values can be quantified, operationalized, and recorded as data with negligible loss of precision, while immeasurable values cannot be precisely operationalized and quantified\.
4. 4\.Does the paper consider values as revealedpreferences, or prescribedprinciples? Preferences are measured from behavioral decisions, and may involve some process of ranking preferred actions, outcomes, or things, while principles\-based values correspond to abstract ideals or standards, e\.g\. valuing beauty, truth, knowledge, human rights declarations, etc\.
5. 5\.Is the approach to valuesabstractorconcrete? Abstract approaches can be theoretical, idealized, philosophical, name no specific values in the paper but present a set of principles for extracting them, whereas concrete approaches may pertain to a specific existing system, culture, or group\.
6. 6\.Are values apparent inindividualinteractions, or formed from acollectiveof agents? That is, are values observed, measured, or determined from individual or one\-on\-one interactions, or do they arise over communities, cultures, nations, or groups?
7. 7\.Are valuesdynamicorstatic? Dynamic values can change over time or in different situations, while static values do not change \(or the change of values is not explained\)\.
8. 8\.Is the characterization of valuesthinorthick? We take a ”thin” characterization of value to be entirely instrumental, evaluative and calculative, while a ”thick” characterization is descriptive and situated in a societal context\.
9. 9\.Does the paper assume or explicitly state that values can emerge in AI autonomously from its creators?
10. 10\.Does the paper take the position that AI should follow human values? A normative question which asks if it is morally justified, ethically correct, and/or strategically important for the field to develop AI systems which ”uphold” values, resembling the processes by which humans uphold values \(according to the paper’s implicit or explicit theory of value\)\.
11. 11\.Does the paper identifywhich humans’values the research is aligning to?
12. 12\.When preferences are elicited from human annotators, does the paper provide information about thesample size of raters?
13. 13\.Does the paper useutility maximizationas an approach to value alignment?

#### Annotation process

We ensured that each paper was annotated by two annotators among the five of the co\-authors of this paper\. We used a Google form with the rubrics, and filled them in after reading each paper\. We divided the papers into groups so that a subset of annotators read approximately fifty papers each\. After an initial pass at reading the collected papers, we isolated disagreements and discussed them through resolutions and consensus\. This was, however, not done for every disagreement and some disagreement remains among the annotations\. However, we achieved a high degree of inter\-annotator agreement, which averages overall around 85%\.

### Qualitative and interpretive analysis

In addition to scoring the papers according to our rubric for a quantitative estimate of the prevalence of the concepts, we also collected quotes from each paper if they addressed human values in non\-mathematical terms\. We did this in order to characterize the beliefs about human values stated, where applicable, by the authors of the papers we analyze\.

## Findings

### Quantitative results

### Inter\-rater agreement

In Table[1](https://arxiv.org/html/2608.10327#Sx5.T1)we report measures of inter\-rater agreement\.

Table 1:Inter\-Rater Reliability StatisticsNotes:PABAK =2×Agreement−12\\times\\text\{Agreement\}\-1\(useful when class distributions are skewed\); Krippendorff’sα\\alphaconsiders all raters simultaneously and handles missing data\. Limitations of quantitative analysis: we report raw agreement, Cohen’s kappa, Krippendorff’sα\\alphaand Bias\-Adjusted Kappa \(PABAK\)\. Given that, we do expect high agreement on the binary yes/no questions, but some disagreement on extent or categorical questions; these results could indicate issues with design or an attribute of the setting in which we are not the research subjects\. Some subjectivity may not be fully specified by the rubric questions\. We may have different criteria for what counts as answers to our questions treating papers as the annotation item\. We are not claiming that we have developed an objective survey\(Aroyo and Welty[2015](https://arxiv.org/html/2608.10327#bib.bib198)\)\. We as the annotators are not the research subjects, which is one methodological assumption baked into inter\-rater reliability; another is that the raters are interchangeable, which we are not\.

We present visualizations of the quantitative results in Figure[1](https://arxiv.org/html/2608.10327#Sx5.F1)and Figure[2](https://arxiv.org/html/2608.10327#Sx5.F2)\.

![Refer to caption](https://arxiv.org/html/2608.10327v1/binary_questions.png)Figure 1:Summary of answers to our rubric aggregated across all raters for binary yes/no answers\.![Refer to caption](https://arxiv.org/html/2608.10327v1/categorical_questions.png)Figure 2:Summary of answers to our rubric aggregated across all raters for categorical questions\. INS = ”information not stated”
### Qualitative Analysis

We found that79% of the papers in our dataset neither defined nor described what they mean by ”values”despite ostensibly seeking to align models with values\. In many papers, it was common to encounter the word ”values” used interchangeably with ”preferences” and ”intentions,” and as we address below, elision between values and preferences has become common in this subfield\. In one rare example, values were also interchangeably used with ”ideologies”\(Kirket al\.[2023a](https://arxiv.org/html/2608.10327#bib.bib177)\)\. In another, researchers trained a model on a corpus they built using Chinese laws and morality texts\(Xuet al\.[2023](https://arxiv.org/html/2608.10327#bib.bib178)\)\.

Despite largely not defining what values mean, 90% of the papers treat values as measurable\. This most often originated with different means of determining human preferences\. Typically, researchers created datasets where LLMs were fed a prompt, generated two responses, and used either human raters, other LLMs \(sometimes referred to as ”autoraters”\)\(Luet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib185); Zhenget al\.[2024](https://arxiv.org/html/2608.10327#bib.bib180); Chakrabortyet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib188); Richemondet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib221)\), or a mix of both\(Yuet al\.[2023](https://arxiv.org/html/2608.10327#bib.bib184); Liuet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib222); Wanget al\.[2024a](https://arxiv.org/html/2608.10327#bib.bib223)\)to evaluate which of the two options better adhered to researchers’ choice of alignment criteria; in other cases, one response might be treated as ’chosen’ while the other is ’rejected\.’ Some papers drew from preference datasets and survey data that researchers who were not the authors of these papers generated\(Zhenget al\.[2024](https://arxiv.org/html/2608.10327#bib.bib180); Chakrabortyet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib188); Louet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib189); Durmuset al\.[2023](https://arxiv.org/html/2608.10327#bib.bib190); Zhaoet al\.[2023](https://arxiv.org/html/2608.10327#bib.bib220); Mirandaet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib211)\), while others identified preferences they sought to optimize for including ”conciseness\.\.being humorous, philosophical, sycophantic, helpful, concise, creative, formal, expert, pleasant, and uplifting”\(Zhonget al\.[2024](https://arxiv.org/html/2608.10327#bib.bib191)\)\.

Preferences have become such a dominant framework that82% of the papers treat preferences as a stand\-in for “values\.”By contrast, 9% treated principles as a basis of values, for instance drawing from social choice theory\(Siththaranjanet al\.[2023](https://arxiv.org/html/2608.10327#bib.bib199); Chakrabortyet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib188)\)\. In papers where researchers created preference datasets through reliance on human annotators to rate LLMs’ paired responses to prompts, only 29 of the 94 papers sampled provided a number for how many people comprised the rater pools\. The number of raters tended to be<100<100people, with the exception of one paper that used a 13,000\-person rater pool\(Köpfet al\.[2023](https://arxiv.org/html/2608.10327#bib.bib234)\)\. When human rater pools were used, 77% of that subset of papersdid not state the size of the rater pool\.

A growing debate within alignment is the extent to which pluralism of values can be achieved\. One paper acknowledged that ”different humans have different values, as it is nearly impossible to train a new large language model from scratch for individual preference”\(Zhouet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib200)\)\. Despite this, most papers took individual rather than collective approaches to values\. Rather than selecting values that are already practiced across societies, states, and other collectives, they rely on individual user interactions with LLMs \(or LLM\-to\-LLM interactions\), and in one case on collectives of 4\-5 people\(Bakkeret al\.[2022](https://arxiv.org/html/2608.10327#bib.bib165)\)\. In addition,53% of papers treated values as static, whether implicitly or through direct acknowledgment of doing so despite recognizing that values evolve\(Liuet al\.[2023](https://arxiv.org/html/2608.10327#bib.bib201)\)\.

We found 66% of papers presented “thin” descriptions and examples of what they meant by values\. By contrast, “thick” descriptions \(which accounted for 3 percent\) draw from relational, observed values that are difficult to cleave apart from the lived context of the people who enact these values\(Geertz[2008](https://arxiv.org/html/2608.10327#bib.bib116)\)\.

Likewise,91% of the papers did not refer to specific groups of humans whose values should be followed, underscoring the taken\-for\-granted idea of values as universal\. One paper noted ”We claim that the approach described is agnostic to the ethical paradigm, the user’s preferences, and the legal or social framework, provided we can supply enough feedback”\(Leikeet al\.[2018](https://arxiv.org/html/2608.10327#bib.bib182)\), while another professed ”No need to solve human values\. We assume we do not need to solve hard philosophical questions of human values and value aggregation before we can align a superhuman researcher model well enough that it avoids egregiously catastrophic outcomes”\(Burnset al\.[2023](https://arxiv.org/html/2608.10327#bib.bib186)\)\. Another reckoned with the difficulties of treating preference datasets as representative: ”This procedure aligns the behavior of GPT\-3 to the stated preferences of a specific group of people \(mostly our labelers and researchers\), rather than any broader notion of ’human values’”\(Ouyanget al\.[2022](https://arxiv.org/html/2608.10327#bib.bib123)\)\.

Finally, we noticed a trend in which a few papers suggest that AI models can develop their own value systems\(Ngoet al\.[2022](https://arxiv.org/html/2608.10327#bib.bib210)\)and ”internal meta\-objectives”\(Phelps and Ranson[2023b](https://arxiv.org/html/2608.10327#bib.bib202)\)\. Some researchers have begun to refer to models’ ”self\-alignment”\(Guoet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib218); Liet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib224)\)described as when ”a new paradigm emerges where LLMs can achieve value alignment by themselves… transforming an unaligned LLM into one adhering to societal norms, independently of external resources”\(Panget al\.[2024](https://arxiv.org/html/2608.10327#bib.bib209)\)\.

## Discussion

Our systematic review of 94 research papers on AI value alignment reveals how the field is erasing human inputs from an endeavor whose success rests on purportedly steering AI systems to respect human values\. Values are core components of how societies organize themselves; any attempt at value alignment is inextricable from society\(Graeber[2001](https://arxiv.org/html/2608.10327#bib.bib24); Haslanger[2023](https://arxiv.org/html/2608.10327#bib.bib19); Anderson[1995](https://arxiv.org/html/2608.10327#bib.bib155); Zhi\-Xuanet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib5); Birhaneet al\.[2022](https://arxiv.org/html/2608.10327#bib.bib41)\)\. Recent social philosophy argues that societies are complex systems – or clusters of interacting systems – that reproduce themselves: their hierarchies, culture, and structures\. These systems are also dynamic and constantly evolving\(Haslanger[2023](https://arxiv.org/html/2608.10327#bib.bib19)\)\. Yet we observe how research in this field abstracts away questions of whose values are represented, or what qualities constitute pluralism and dynamism in values, instead relegating these fundamental, unavoidable concerns to be taken up by others \(if at all\)\. The risk of repeatedly delaying explorations of what values are and how they can be accountably represented is that certain practices of design, deployment, and evaluation will ossify\. At worst, this may result in normalizing widespread use of AI systems that have been trained on a narrow conception of value and claiming that this is the best the field can offer\.

The pressure to neglect these essential debates is reflected in a subset of papers that treat alignment as a step toward achieving ”artificial superintelligence”\(Kimet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib183)\), or as an essential component of averting catastrophic and existential risks from AI\(Shenet al\.[2023](https://arxiv.org/html/2608.10327#bib.bib187); Ngoet al\.[2022](https://arxiv.org/html/2608.10327#bib.bib210); Burnset al\.[2023](https://arxiv.org/html/2608.10327#bib.bib186)\)\. This typifies the turn toward what we call ”alignment without humans,” or treatment of humans as an unreliable source of information on their own preferences\(Mirandaet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib211); Wanget al\.[2024a](https://arxiv.org/html/2608.10327#bib.bib223)\), as too expensive\(Wanget al\.[2024c](https://arxiv.org/html/2608.10327#bib.bib217); Zhenget al\.[2024](https://arxiv.org/html/2608.10327#bib.bib180); Liuet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib222)\)and time\-consuming\(Daiet al\.[2023](https://arxiv.org/html/2608.10327#bib.bib219)\)to use, therefore justifying the use of synthetic means of preference elicitation, feedback, and evaluations\(Honget al\.[2024](https://arxiv.org/html/2608.10327#bib.bib192); Cuiet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib226); Tianet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib227)\)\. We observe studies that rely on no human input for pre\-determining value alignment criteria\(Meiet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib212); Chenet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib213)\), often through the use of simulated humans\(Chakrabortyet al\.[2023](https://arxiv.org/html/2608.10327#bib.bib214); Poddaret al\.[2024](https://arxiv.org/html/2608.10327#bib.bib39); Liuet al\.[2023](https://arxiv.org/html/2608.10327#bib.bib201)\)\. Perhaps in part due to the desire to race toward solving ”the alignment problem,” one trend that arose from the corpus of papers is the dominance of Anthropic’s Helpful, Harmless, Honest \(HHH\) approach\. This ranged from papers that explicitly used the HH\-RLHF dataset for training and/or evaluations\(Yuet al\.[2023](https://arxiv.org/html/2608.10327#bib.bib184); Louet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib189); Huanget al\.[2023](https://arxiv.org/html/2608.10327#bib.bib215); Zhenget al\.[2024](https://arxiv.org/html/2608.10327#bib.bib180); Zenget al\.[2024](https://arxiv.org/html/2608.10327#bib.bib225)\), to others that name HHH as a guiding principle\(Dinget al\.[2024](https://arxiv.org/html/2608.10327#bib.bib179); Gaoet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib181); Luet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib185); Shenet al\.[2023](https://arxiv.org/html/2608.10327#bib.bib187); Shiet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib216)\)\.

Graeber’s critique of applying economic theory to diverse cultures is apt for AI value alignment: ”It \[economics\] also has the advantage of joining an extremely simple model of human nature with extremely complicated mathematical formulae that non\-specialists can rarely understand, much less criticize\.”\(Graeber[2001](https://arxiv.org/html/2608.10327#bib.bib24)\)\. Our review of the literature reveals how RLHF and other technical alignment methods likewise join extremely simplistic models of human nature with complicated mathematical formulae that are difficult for non\-specialists to understand or critique\.

Even though the majority of papers in our dataset do not define what they mean by values, adopting the framework of economic preference optimization to align models necessarily requires a commitment to a set of beliefs about what human values are\. At a minimum this entails the belief that values are individual, rational, monist, static, maximizing utility functions, and abstract\. The idea is that an agent that obeys some minimal conditions of rationality can be modeled as if the agent has an ordered set of preferences along with some probabilistic beliefs about what states of the world and itself will maximize the agent’s utility \(however defined\)\(Shea[2018](https://arxiv.org/html/2608.10327#bib.bib255)\)\. This particular set of assumptions about human nature and human values, drawn largely from economic decision theory and rational choice theory, has been empirically and theoretically challenged from multiple perspectives within and outside of economics\(Chibnik[2011](https://arxiv.org/html/2608.10327#bib.bib102); Graeber[2011](https://arxiv.org/html/2608.10327#bib.bib81),[2001](https://arxiv.org/html/2608.10327#bib.bib24); Gigerenzer[1997](https://arxiv.org/html/2608.10327#bib.bib132); Franket al\.[2024](https://arxiv.org/html/2608.10327#bib.bib91)\)and AI alignment\(Bakkeret al\.[2022](https://arxiv.org/html/2608.10327#bib.bib165); Kasirzadeh[2024a](https://arxiv.org/html/2608.10327#bib.bib33); Zhi\-Xuanet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib5)\)\. Rational or social choice theory do not describe the values or behavior of real human beings or cultures, but only the behavior of highly abstract, reductionist, and idealized fictional agents\(Franket al\.[2024](https://arxiv.org/html/2608.10327#bib.bib91)\)\.

Rather than asking ”what do human beings actually care about?”, rational choice theory asks \- and presumes there is a normatively correct answer to \- ”what should human beings do in order to make the utility\-maximizing decision?” This meta\-theoretical commitment risks foreclosing alternative conceptions of human values, and limits the ability of AI systems to align to diverse cultural values\(Prabhakaranet al\.[2022](https://arxiv.org/html/2608.10327#bib.bib1); Khanet al\.[2025](https://arxiv.org/html/2608.10327#bib.bib205); Gabriel[2020](https://arxiv.org/html/2608.10327#bib.bib4)\)\. Poddaret al\.point out, ”Current RLHF approaches rely on a prescriptive set of values curated by a small set of AI researchers\. Moreover, they typically assume that all end\-users share the same set of values\. Given the concerning lack of diversity in AI, it is clear that this approach cannot account for the range of social, moral, and political values that inform preferences in human populations”\(Poddaret al\.[2024](https://arxiv.org/html/2608.10327#bib.bib39)\)\. We call on the field to take seriously the fundamental weaknesses of utility maximization, rational choice theory and reductionist economic frameworks in AI alignment\.

#### Utility maximization

Economists and philosophers have generally assumed that people are trying to maximizesomething: money, or love, or sometimes something else \(most often, expected utility\) with their choices\(Glimcher[2022](https://arxiv.org/html/2608.10327#bib.bib253); Graeber[2001](https://arxiv.org/html/2608.10327#bib.bib24)\)\.Utility maximization treats all human behavior as the maximization of expected utility from a stable set of preferences, and holds that humans accumulate an optimal amount of information inmarkets\(Chibnik[2011](https://arxiv.org/html/2608.10327#bib.bib102); Russell and Norvig[1995](https://arxiv.org/html/2608.10327#bib.bib195); Becker[1976](https://arxiv.org/html/2608.10327#bib.bib254)\)\. Thus, preferences are seen as being derived from a single utility function that each individual is somehow computing in their brain\(Gigerenzer[1997](https://arxiv.org/html/2608.10327#bib.bib132)\)\. Taken to its extreme, utility theory holds that our behavior is entirely determined by maximizing this utility function, and preferences are assumed to not change substantially over time, nor to be very different between people of different socioeconomic classes, societies, and cultures\(Becker[1976](https://arxiv.org/html/2608.10327#bib.bib254)\)\. Finally, AI alignment assumes that an individual’s reward function \(and therefore their values\), or preferences about the future, can be inferred, or approximated, by observing how humans behave by collecting data on how humans interact with AI outputs\(Hadfield\-Menellet al\.[2016](https://arxiv.org/html/2608.10327#bib.bib55)\)\.

Despite the empirical inadequacy of this ambiguous economic approach to AI and value alignment, creating an artificial agent that embodies economic theory has been the almost unquestioned north star in the field until recently\(Zhi\-Xuanet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib5); Gabriel[2020](https://arxiv.org/html/2608.10327#bib.bib4); Russell[2019](https://arxiv.org/html/2608.10327#bib.bib96)\)\. Influential work on value alignment explicitly argues that alignment should be formulated as a cooperative and interactivereward maximization process\(Hadfield\-Menellet al\.[2016](https://arxiv.org/html/2608.10327#bib.bib55)\), with most alignment work implicitly adopting this approach\(Christianoet al\.[2017b](https://arxiv.org/html/2608.10327#bib.bib135); Ouyanget al\.[2022](https://arxiv.org/html/2608.10327#bib.bib123); Wanget al\.[2024b](https://arxiv.org/html/2608.10327#bib.bib120)\)the vast majority of technical approaches to alignment from our sample of papers implementing some form of utility maximization\. These often unstated philosophical commitments to an economic worldview determine the methodology that alignment researchers use to measure and model human values\. We make this often implicit metatheorectical commitment explicit so that it can be critiqued and improved\(Maxwell[2005](https://arxiv.org/html/2608.10327#bib.bib42)\)\.

We also evaluated our sample of papers for whether they adopted a framework ofutility maximization, discovering that 87 % of papers implicitly or explicitly use utility maximization as part of the technical framework to align models\. Adopting the framework of utility maximization necessarily involves normative assumptions that are value\-laden\(Graeber[2001](https://arxiv.org/html/2608.10327#bib.bib24)\)\. However, given that utility maximization is an inherent part of reinforcement learning from human feedback \(RLHF\) and reward modeling it is not surprising that the majority of alignment papers in our sample adopt this framework\. This also stems from framing AI alignment in terms of the ”principal\-agent” framework from economic theory\(Phelps and Ranson[2023a](https://arxiv.org/html/2608.10327#bib.bib238); Stanczaket al\.[2025](https://arxiv.org/html/2608.10327#bib.bib239)\)\.

Moreover, utility maximization is cited as a property of models, e\.g\.,\(Mazeikaet al\.[2025](https://arxiv.org/html/2608.10327#bib.bib230)\)notes ”it was shown that recent LLMs have structurally coherent, broad value systems\. As they become more capable, their value systems increasingly conform to the axioms of utility theory, meaning they can be described as maximizing a utility function\.” Given that models are trained and fine\-tuned according to the axioms of rational choice theory, should it be surprising that they behave according to utility theory?

A small number of the papers recognize the inherent limitations of utility maximization\.\(Phelps and Ranson[2023a](https://arxiv.org/html/2608.10327#bib.bib238)\)points out: ”Rather than seeking to impose a monolithic utility function on artificial agents, we propose a strategy of reducing information asymmetry and aligning interests through external incentives, much like the approach used in traditional economic solutions to principal\-agent problems\. ”Furthermore,\(Poddaret al\.[2024](https://arxiv.org/html/2608.10327#bib.bib39)\)writes, ”These insights suggest that human preferences are not derived from a single utility function, but are affected by unobserved, hidden user context\.”

One of the starting points of our meta\-review isZhi\-Xuanet al\.\([2024](https://arxiv.org/html/2608.10327#bib.bib5)\)’s critique of the reliance on eliciting preferences as representations of human values, in which they ask: ”What would AI alignment look like if it took these challenges seriously? It would move away from naive rational choice models of human decision making, towards richer models that include how we evaluate, commensurate, and act upon our values in boundedly rational ways\. It would no longer take for granted expected utility theory, and instead explore systems for reasoning about the normativity of our preferences and values\.”

### Alternative conceptions of value

> ”It’s values all the way down”\- Alondra Nelson\(Nelson[2023](https://arxiv.org/html/2608.10327#bib.bib246)\)

The narrow economic conception of human values that determines the methodology and philosophy of alignment is by no means the only way to understand what humans value\. As anthropologists point out, for 99 % of humanity’s history we did not live in market economies\(Graeber[2011](https://arxiv.org/html/2608.10327#bib.bib81)\)\. Economic theory implicitly posits that we’ve always been capitalists and that capitalism is somehow natural\(Hornborg[1998](https://arxiv.org/html/2608.10327#bib.bib250)\)\. We argue that even recent work on pluralistic alignment is still fundamentally rooted in the orthodox economic framework\.

However, this is a distorted view of both societies and individual humans\. Part of the challenge of rethinking alignment’s approach to value comes from the social positioning of the field that ’doing anything is better than nothing,’ without recognizing that over\-utilizing narrow disciplinary approaches has the potential to make matters worse\(Dai[2025](https://arxiv.org/html/2608.10327#bib.bib174)\)\. A fundamental limitation of existing alignment methods is that they reinforce shallow behavioral dispositions rather than endowing LLMs with a genuine capacity for normative deliberation\(Millière[2025](https://arxiv.org/html/2608.10327#bib.bib242)\)\. We find that the abstractions and idealizations which alignment postulates to describe human values become confused for real phenomena\.

Papers in our sample did in some cases engage directly with alternatives to utility theory, for example Huang et\.al\., \(2025\) who state, ”We are interested in values not as abstract entities, but as operational priorities that influence how the system navigates its possible space of outputs\.” We expand others’ calls to expand alignment approaches\(Casperet al\.[2023](https://arxiv.org/html/2608.10327#bib.bib206); Lindströmet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib119); Zhi\-Xuanet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib5)\)with our exploration of psychological and anthropological theories of value\.

#### Cognitive or psychological theories of value

Papers in our dataset referenced psychological theories of value that are accepted in mainstream psychology literature, such as Schwartz’s theory of basic values\(Sierraet al\.[2021](https://arxiv.org/html/2608.10327#bib.bib105); Schwartz[2012](https://arxiv.org/html/2608.10327#bib.bib115)\)\. Schwartz theorizes value as a set of beliefs, closely linked to emotions, which inform the selection of actions\. He further claims that there is a fixed set of ”basic values” or value categories, which areuniversalacross cultures, and tend to remain fixed over time within an adult individual\. These values are conceptualized as guiding principles which can come into conflict, but are ultimately resolved through a hierarchy of importance\. This leads to our characterization of the psychological theory of value as ultimately monistic\. This individualistic view of values is challenged by empirical psychological and ethnographic work which shows a much more complex, situated, holistic and dynamic view of values\(Chibnik[2011](https://arxiv.org/html/2608.10327#bib.bib102); Anderson[1995](https://arxiv.org/html/2608.10327#bib.bib155); Haslanger[2023](https://arxiv.org/html/2608.10327#bib.bib19)\)\. This field is also complex, and we encourage engagement with existing debates in the cross\-cultural study of values\(Schwartz[2012](https://arxiv.org/html/2608.10327#bib.bib115)\)\.

#### Anthropological, situated theories of value

The field with perhaps the most theoretical work and empirical data on human values \- anthropology \- has to date been ignored in AI alignment \(cf\.\(Schmer\-Galunder[2026](https://arxiv.org/html/2608.10327#bib.bib251)\)\. Graeber suggests that what people call “value” may be thought of as “the way people represent the importance of their own actions to themselves, as reflected in one or another socially recognized form”\(Graeber[2001](https://arxiv.org/html/2608.10327#bib.bib24)\)\. Economic theory’s view of human nature is cynical, assuming that humans are entirely self\-interested and will calculate the most effective way to obtain what they desire\(Becker[1976](https://arxiv.org/html/2608.10327#bib.bib254)\)\. These are values derived from living in market\-based societies\(Hornborg[2014](https://arxiv.org/html/2608.10327#bib.bib20)\)\. However, markets are not natural phenomena that arise spontaneously out of large complex societies as the economic narrative goes, but rather they require the enforcement of state violence to impose property rights, coercion, and the rule of law\(Graeber[2011](https://arxiv.org/html/2608.10327#bib.bib81); Mau[2023](https://arxiv.org/html/2608.10327#bib.bib126)\)\.

Economic anthropology recognizes that economic notions of value, articulated in objective monetary representations and utility theory, and ethical notions of value, articulated in moral terms, are “inextricably connected”\(Lambek[2013](https://arxiv.org/html/2608.10327#bib.bib233)\)\. Graeber\(Graeber[2001](https://arxiv.org/html/2608.10327#bib.bib24)\)outlines three ways in which values have been theorized in the social sciences that go beyond the narrow conception of value in the economic sense:

1. 1\.”values” in the sociological sense: conceptions of what is ultimately good, proper, or desirable in human life
2. 2\.”value” in the economic sense: the degree to which objects are desired, particularly, as measured by how much others are willing to give up to get them
3. 3\.”value” in the linguistic sense, or how meaning is derived from the differences in the usage of words\.

However, in our dataset it is unusual for researchers to be specific about which humans, which values, and which cultures they target for alignment\. We follow\(Arzbergeret al\.[2024](https://arxiv.org/html/2608.10327#bib.bib240)\), who argue that values acquire situated meaning; hence, their concrete interpretations, means of actualization, and hierarchies vary between people depending on the situations in which they guide the notion of what people really value\. This is closely aligned with situated\-constructivist views of emotions, which see emotions not as basic universal categories, but as arising in specific socially\- and culturally\- salient circumstances\. In other words, emotions are a category populated with highly variable instances\(Barrett[2017](https://arxiv.org/html/2608.10327#bib.bib241)\)\. Thus we can analogously we think of human values as a category populated with highly variable instances tied to specific cultural circumstances\.

## Limitations

One limitation of this work is the relatively small sample size of papers in our dataset compared with the vast number of research papers on this topic\. This is especially challenging given the widespread practice of rapidly uploading research to arXiv, avoiding peer\-review\(Gyevnar and Kasirzadeh[2025](https://arxiv.org/html/2608.10327#bib.bib231)\)\. However, we attempt to correct for this by estimating the relative impact of our dataset on the field using both quantitative and qualitative measures\. While we aimed to gather a representative subsample of available papers on AI value alignment, we make no claims to the completeness or comprehensiveness of the sampled papers\. The sample size was also limited due to time constraints on our small team of co\-authors\. We also acknowledge that the scoring rubric has many limitations\. The questions are highly nuanced and difficult to boil down to a binary or even ternary answer\. We considered a multi\-point Likert scale, which was ultimately abandoned due to concerns about calibrating relative scales\.

### Implications for future work

We suggest building on our analysis to apply concepts from humanistic social sciences such as anthropology and psychology to operationalize human values in AI in a more representative and holistic way that explicitly accounts for: concreteness/abstraction, the thick/thin distinction, distinctions between monism and pluralism, and embracing a view of values not as abstract universal things human possess, but as constructed in specific social and cultural contexts\(Kasirzadeh[2024a](https://arxiv.org/html/2608.10327#bib.bib33); Sorensenet al\.[2024](https://arxiv.org/html/2608.10327#bib.bib2); Benkleret al\.[2023](https://arxiv.org/html/2608.10327#bib.bib235); Arzbergeret al\.[2024](https://arxiv.org/html/2608.10327#bib.bib240)\)\.

To deepen the critique of ”preferentism,” we also recommend an examination of the content of prompts and paired responses in preference training datasets, as many contain largely nonsensical conversations with an LLM and involve choosing the less absurd of two unrealistic examples in each pair\. Communicating the shaky foundations of these unquestioned defaults is of particular value to informing policymakers and the public about the limits of value alignment\. Moreover, it can contribute to more effective sociotechnical and policy measures derived from realistic conceptions of AI’s risks\.

## Conclusion

We began with an examination of the theory of value\(s\) to which AI value alignment research subscribes, drawing from interdisciplinary philosophy and social scientific research on human values to develop a schema for answering this question\. We argued that AI alignment lacks a concrete theory of human values and implicitly adopts beliefs about human values rooted in economic theory\. Through our review, we have shown that the assumptions guiding technical AI alignment practices limit the possible range of human values with which AI can be aligned by translating alignment into utility theory\. As Lisa Feldman Barrett argues, ”Even a scientist who believes they are purely driven by data is doing philosophy”\(Barrett and Theriault[2025](https://arxiv.org/html/2608.10327#bib.bib243)\)\. Tech companies and AI value alignment researchers enact philosophy every day as they engage in value alignment work when choosing training data, setting up model architectures, and especially when running RLHF experiments\. Having laid bare AI value alignment’s philosophical commitments, we invite researchers from a wider variety of disciplines to bring greater specificity and under\-explored perspectives into the debate on whether and how AI can address human values\.

## Generative AI Usage Statement

Generative AI was not used in the research or writing of this publication\.

## References

- S\. Ahmed, K\. Jaźwińska, A\. Ahlawat, A\. Winecoff, and M\. Wang \(2024\)Field\-building and the epistemic culture of ai safety\.First Monday\.Cited by:[Alignment and its discontents](https://arxiv.org/html/2608.10327#Sx2.SSx2.p2.1)\.
- R\. Alvarado \(2023\)AI as an epistemic technology\.Science and Engineering Ethics29\(5\),pp\. 32\.Cited by:[What is value alignment?](https://arxiv.org/html/2608.10327#Sx2.SSx1.p4.1)\.
- E\. Amirloo, J\. Fauconnier, C\. Roesmann, C\. Kerl, R\. Boney, Y\. Qian, Z\. Wang, A\. Dehghan, Y\. Yang, Z\. Gan,et al\.\(2024\)Understanding alignment in multimodal llms: a comprehensive study\.arXiv preprint arXiv:2407\.02477\.Cited by:[What is value alignment?](https://arxiv.org/html/2608.10327#Sx2.SSx1.p1.1),[Preferences are not all you need](https://arxiv.org/html/2608.10327#Sx2.SSx3.p1.1)\.
- E\. Anderson \(1995\)Value in ethics and economics\.2\. print edition,Harvard Univ\. Press,Cambridge, Mass\.\(eng\)\.External Links:ISBN 9780674931909 9780674931893Cited by:[Cognitive or psychological theories of value](https://arxiv.org/html/2608.10327#Sx6.SSx1.SSSx1.p1.1),[Discussion](https://arxiv.org/html/2608.10327#Sx6.p1.1)\.
- Anthropic \(2025\)Anthropic Alignment Research Team\.\(en\-US\)\.External Links:[Link](https://www.anthropic.com/research/team/alignment)Cited by:[Alignment and its discontents](https://arxiv.org/html/2608.10327#Sx2.SSx2.p3.1)\.
- L\. Aroyo and C\. Welty \(2015\)Truth is a lie: crowd truth and the seven myths of human annotation\.AI Magazine36\(1\),pp\. 15–24\.Cited by:[Table 1](https://arxiv.org/html/2608.10327#Sx5.T1.5)\.
- M\. Arvan \(2024\)‘Interpretability’and ‘alignment’are fool’s errands: a proof that controlling misaligned large language models is the best anyone can hope for\.AI & SOCIETY,pp\. 1–16\.Cited by:[Alignment and its discontents](https://arxiv.org/html/2608.10327#Sx2.SSx2.p2.1),[Preferences are not all you need](https://arxiv.org/html/2608.10327#Sx2.SSx3.p4.1)\.
- A\. Arzberger, S\. Buijsman, M\. L\. Lupetti, A\. Bozzon, and J\. Yang \(2024\)Nothing comes without its world–practical challenges of aligning llms to situated human values through rlhf\.InProceedings of the AAAI/ACM Conference on AI, Ethics, and Society,Vol\.7,pp\. 61–73\.Cited by:[Introduction](https://arxiv.org/html/2608.10327#Sx1.p1.1),[Alignment and its discontents](https://arxiv.org/html/2608.10327#Sx2.SSx2.p1.1),[Anthropological, situated theories of value](https://arxiv.org/html/2608.10327#Sx6.SSx1.SSSx2.p3.1),[Implications for future work](https://arxiv.org/html/2608.10327#Sx7.SSx1.p1.1)\.
- A\. Askell, Y\. Bai, A\. Chen, D\. Drain, D\. Ganguli, T\. Henighan, A\. Jones, N\. Joseph, B\. Mann, N\. DasSarma,et al\.\(2021\)A general language assistant as a laboratory for alignment\.arXiv preprint arXiv:2112\.00861\.Cited by:[Introduction](https://arxiv.org/html/2608.10327#Sx1.p1.1),[What is value alignment?](https://arxiv.org/html/2608.10327#Sx2.SSx1.p4.1),[2nd item](https://arxiv.org/html/2608.10327#Sx4.I2.i2.p1.1)\.
- Y\. Bai, A\. Jones, K\. Ndousse, A\. Askell, A\. Chen, N\. DasSarma, D\. Drain, S\. Fort, D\. Ganguli, T\. Henighan, N\. Joseph, S\. Kadavath, J\. Kernion, T\. Conerly, S\. El\-Showk, N\. Elhage, Z\. Hatfield\-Dodds, D\. Hernandez, T\. Hume, S\. Johnston, S\. Kravec, L\. Lovitt, N\. Nanda, C\. Olsson, D\. Amodei, T\. Brown, J\. Clark, S\. McCandlish, C\. Olah, B\. Mann, and J\. Kaplan \(2022\)Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback\.arXiv\.Note:arXiv:2204\.05862External Links:[Link](http://arxiv.org/abs/2204.05862),[Document](https://dx.doi.org/10.48550/arXiv.2204.05862)Cited by:[What is value alignment?](https://arxiv.org/html/2608.10327#Sx2.SSx1.p1.1),[3rd item](https://arxiv.org/html/2608.10327#Sx4.I2.i3.p1.1)\.
- M\. Bakker, M\. Chadwick, H\. Sheahan, M\. Tessler, L\. Campbell\-Gillingham, J\. Balaguer, N\. McAleese, A\. Glaese, J\. Aslanides, M\. Botvinick,et al\.\(2022\)Fine\-tuning language models to find agreement among humans with diverse preferences\.Advances in Neural Information Processing Systems35,pp\. 38176–38189\.Cited by:[Preferences are not all you need](https://arxiv.org/html/2608.10327#Sx2.SSx3.p3.1),[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p4.1),[Discussion](https://arxiv.org/html/2608.10327#Sx6.p4.1)\.
- L\. F\. Barrett and J\. Theriault \(2025\)What’s real? a philosophy of science for social psychology\.The handbook of social psychology\.Cited by:[Conclusion](https://arxiv.org/html/2608.10327#Sx8.p1.1)\.
- L\. F\. Barrett \(2017\)The theory of constructed emotion: an active inference account of interoception and categorization\.Social cognitive and affective neuroscience12\(1\),pp\. 1–23\.Cited by:[Anthropological, situated theories of value](https://arxiv.org/html/2608.10327#Sx6.SSx1.SSSx2.p3.1)\.
- G\. S\. Becker \(1976\)The economic approach to human behavior\.Vol\.803,University of Chicago press\.Cited by:[Preferences are not all you need](https://arxiv.org/html/2608.10327#Sx2.SSx3.p1.1),[Justification for rubric questions](https://arxiv.org/html/2608.10327#Sx4.SSx2.SSSx1.p1.1),[Anthropological, situated theories of value](https://arxiv.org/html/2608.10327#Sx6.SSx1.SSSx2.p1.1),[Utility maximization](https://arxiv.org/html/2608.10327#Sx6.SSx3.SSSx1.p1.1)\.
- N\. Benkler, D\. Mosaphir, S\. Friedman, A\. Smart, and S\. Schmer\-Galunder \(2023\)Assessing llms for moral value pluralism\.arXiv preprint arXiv:2312\.10075\.Cited by:[Implications for future work](https://arxiv.org/html/2608.10327#Sx7.SSx1.p1.1)\.
- S\. Bergman, N\. Marchal, J\. Mellor, S\. Mohamed, I\. Gabriel, and W\. Isaac \(2024\)STELA: a community\-centred approach to norm elicitation for ai alignment\.Scientific Reports14\(1\),pp\. 6616\.Cited by:[Alignment and its discontents](https://arxiv.org/html/2608.10327#Sx2.SSx2.p1.1)\.
- A\. Birhane, P\. Kalluri, D\. Card, W\. Agnew, R\. Dotan, and M\. Bao \(2022\)The Values Encoded in Machine Learning Research\.In2022 ACM Conference on Fairness, Accountability, and Transparency,Seoul Republic of Korea,pp\. 173–184\(en\)\.External Links:ISBN 978\-1\-4503\-9352\-2,[Link](https://dl.acm.org/doi/10.1145/3531146.3533083),[Document](https://dx.doi.org/10.1145/3531146.3533083)Cited by:[Discussion](https://arxiv.org/html/2608.10327#Sx6.p1.1)\.
- C\. Burns, P\. Izmailov, J\. H\. Kirchner, B\. Baker, L\. Gao, L\. Aschenbrenner, Y\. Chen, A\. Ecoffet, M\. Joglekar, J\. Leike, I\. Sutskever, and J\. Wu \(2023\)Weak\-to\-strong generalization: eliciting strong capabilities with weak supervision\.\.External Links:[Link](https://arxiv.org/abs/2312.09390)Cited by:[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p6.1),[Discussion](https://arxiv.org/html/2608.10327#Sx6.p2.1)\.
- S\. Casper, X\. Davies, C\. Shi, T\. K\. Gilbert, J\. Scheurer, J\. Rando, R\. Freedman, T\. Korbak, D\. Lindner, P\. Freire,et al\.\(2023\)Open problems and fundamental limitations of reinforcement learning from human feedback\.arXiv preprint arXiv:2307\.15217\.Cited by:[Preferences are not all you need](https://arxiv.org/html/2608.10327#Sx2.SSx3.p1.1),[Alternative conceptions of value](https://arxiv.org/html/2608.10327#Sx6.SSx1.p4.1)\.
- S\. Chakraborty, J\. Qiu, H\. Yuan, A\. Koppel, F\. Huang, D\. Manocha, A\. S\. Bedi, and M\. Wang \(2024\)MaxMin\-rlhf: alignment with diverse human preferences\.\.External Links:[Link](https://arxiv.org/abs/2402.08925)Cited by:[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p2.1),[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p3.1)\.
- S\. Chakraborty, A\. Singh, A\. Bhaskar, P\. Tokekar, D\. Manocha, and A\. S\. Bedi \(2023\)REBEL: reward regularization\-based approach for robotic reinforcement learning from human feedback\.\.External Links:[Link](https://arxiv.org/abs/2312.14436)Cited by:[Discussion](https://arxiv.org/html/2608.10327#Sx6.p2.1)\.
- S\. Chaudhari, P\. Aggarwal, V\. Murahari, T\. Rajpurohit, A\. Kalyan, K\. Narasimhan, A\. Deshpande, and B\. C\. da Silva \(2024\)RLHF deciphered: a critical analysis of reinforcement learning from human feedback for llms\.arXiv preprint arXiv:2404\.08555\.Cited by:[What is value alignment?](https://arxiv.org/html/2608.10327#Sx2.SSx1.p1.1)\.
- Z\. Chen, K\. Zhou, W\. X\. Zhao, J\. Wang, and J\. Wen \(2024\)Low\-redundant optimization for large language model alignment\.\.External Links:[Link](https://arxiv.org/html/2406.12606v1)Cited by:[Discussion](https://arxiv.org/html/2608.10327#Sx6.p2.1)\.
- M\. Chibnik \(2011\)Anthropology, Economics, and Choice\.University of Texas Press\.External Links:ISBN 978\-0\-292\-72676\-5,[Link](https://www.jstor.org/stable/10.7560/726765),[Document](https://dx.doi.org/10.7560/726765)Cited by:[Preferences are not all you need](https://arxiv.org/html/2608.10327#Sx2.SSx3.p1.1),[Cognitive or psychological theories of value](https://arxiv.org/html/2608.10327#Sx6.SSx1.SSSx1.p1.1),[Utility maximization](https://arxiv.org/html/2608.10327#Sx6.SSx3.SSSx1.p1.1),[Discussion](https://arxiv.org/html/2608.10327#Sx6.p4.1)\.
- P\. F\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. Amodei \(2017a\)Deep Reinforcement Learning from Human Preferences\.InAdvances in Neural Information Processing Systems,Vol\.30\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2017/hash/d5e2c0adad503c91f91df240d0cd4e49-Abstract.html)Cited by:[What is value alignment?](https://arxiv.org/html/2608.10327#Sx2.SSx1.p1.1)\.
- P\. F\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. Amodei \(2017b\)Deep reinforcement learning from human preferences\.Advances in neural information processing systems30\.Cited by:[1st item](https://arxiv.org/html/2608.10327#Sx4.I2.i1.p1.1),[Utility maximization](https://arxiv.org/html/2608.10327#Sx6.SSx3.SSSx1.p2.1)\.
- G\. Cui, L\. Yuan, N\. Ding, G\. Yao, W\. Zhu, Y\. Ni, G\. Xie, Z\. Liu, and M\. Sun \(2024\)UltraFeedback: boosting language models with high\-quality feedback\.\.External Links:[Link](https://arxiv.org/pdf/2310.01377)Cited by:[Discussion](https://arxiv.org/html/2608.10327#Sx6.p2.1)\.
- J\. Dai \(2025\)The Artificiality of Alignment\.\(en\-US\)\.External Links:[Link](https://joinreboot.org/p/alignment)Cited by:[Alignment and its discontents](https://arxiv.org/html/2608.10327#Sx2.SSx2.p3.1),[Alternative conceptions of value](https://arxiv.org/html/2608.10327#Sx6.SSx1.p3.1)\.
- J\. Dai, X\. Pan, R\. Sun, J\. Ji, X\. Xu, M\. Liu, Y\. Wang, and Y\. Yang \(2023\)Safe rlhf: safe reinforcement learning from human feedback\.\.External Links:[Link](https://arxiv.org/abs/2310.12773)Cited by:[Discussion](https://arxiv.org/html/2608.10327#Sx6.p2.1)\.
- Y\. Ding, B\. Li, and R\. Zhang \(2024\)ETA: evaluating then aligning safety of vision language models at inference time\.\.External Links:[Link](https://arxiv.org/abs/2410.06625)Cited by:[Discussion](https://arxiv.org/html/2608.10327#Sx6.p2.1)\.
- R\. Dobbe \(2022\)System safety and artificial intelligence\.InProceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency,pp\. 1584–1584\.Cited by:[Alignment and its discontents](https://arxiv.org/html/2608.10327#Sx2.SSx2.p5.1)\.
- E\. Durmus, K\. Nguyen, T\. I\. Liao, N\. Schiefer, A\. Askell, A\. Bakhtin, C\. Chen, Z\. Hatfield\-Dodds, D\. Hernandez, N\. Joseph, L\. Lovitt, S\. McCandlish, O\. Sikder, A\. Tamkin, J\. Thamkul, J\. Kaplan, J\. Clark, and D\. Ganguli \(2023\)Towards measuring the representation of subjective global opinions in language models\.\.External Links:LinkCited by:[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p2.1)\.
- A\. Frank, M\. Gleiser, and E\. Thompson \(2024\)The blind spot: why science cannot ignore human experience\.The MIT press,Cambridge, Mass\(eng\)\.External Links:ISBN 978\-0\-262\-04880\-4Cited by:[Discussion](https://arxiv.org/html/2608.10327#Sx6.p4.1)\.
- I\. Gabriel \(2020\)Artificial Intelligence, Values, and Alignment\.Minds and Machines30\(3\),pp\. 411–437\(en\)\.External Links:ISSN 1572\-8641,[Link](https://doi.org/10.1007/s11023-020-09539-2),[Document](https://dx.doi.org/10.1007/s11023-020-09539-2)Cited by:[Introduction](https://arxiv.org/html/2608.10327#Sx1.p1.1),[Alignment and its discontents](https://arxiv.org/html/2608.10327#Sx2.SSx2.p1.1),[Preferences are not all you need](https://arxiv.org/html/2608.10327#Sx2.SSx3.p2.1),[Utility maximization](https://arxiv.org/html/2608.10327#Sx6.SSx3.SSSx1.p2.1),[Discussion](https://arxiv.org/html/2608.10327#Sx6.p5.1)\.
- C\. Gao, Y\. Wu, D\. Chen, Q\. Zhang, Fu,Zhengyan, Y\. Wan, L\. Sun, and X\. Zhang \(2024\)HonestLLM: toward an honest and helpful large language model\.\.External Links:[Link](https://arxiv.org/abs/2406.00380)Cited by:[Discussion](https://arxiv.org/html/2608.10327#Sx6.p2.1)\.
- T\. Gebru and É\. P\. Torres \(2024\)The tescreal bundle: eugenics and the promise of utopia through artificial general intelligence\.First Monday\.Cited by:[Alignment and its discontents](https://arxiv.org/html/2608.10327#Sx2.SSx2.p5.1)\.
- C\. Geertz \(2008\)“Thick Description: Toward an Interpretive Theory of Culture”\.InThe Cultural Geography Reader,Note:Num Pages: 11External Links:ISBN 978\-0\-203\-93195\-0Cited by:[Justification for rubric questions](https://arxiv.org/html/2608.10327#Sx4.SSx2.SSSx1.p3.1),[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p5.1)\.
- G\. Gigerenzer \(1997\)Bounded rationality: models of fast and frugal inference\.Swiss Journal of Economics and Statistics133\(2/2\),pp\. 201–218\.Cited by:[Utility maximization](https://arxiv.org/html/2608.10327#Sx6.SSx3.SSSx1.p1.1),[Discussion](https://arxiv.org/html/2608.10327#Sx6.p4.1)\.
- T\. Gillespie, R\. Shaw, M\. L\. Gray, and J\. Suh \(2024\)AI red\-teaming is a sociotechnical system\. now what?\.arXiv preprint arXiv:2412\.09751\.Cited by:[Introduction](https://arxiv.org/html/2608.10327#Sx1.p1.1)\.
- P\. W\. Glimcher \(2022\)Efficiently irrational: deciphering the riddle of human choice\.Trends in cognitive sciences26\(8\),pp\. 669–687\.Cited by:[Utility maximization](https://arxiv.org/html/2608.10327#Sx6.SSx3.SSSx1.p1.1)\.
- D\. Graeber \(2001\)Toward an Anthropological Theory of Value: The False Coin of Our Own Dreams\.Palgrave\-Macmillan\.Cited by:[Anthropological, situated theories of value](https://arxiv.org/html/2608.10327#Sx6.SSx1.SSSx2.p1.1),[Anthropological, situated theories of value](https://arxiv.org/html/2608.10327#Sx6.SSx1.SSSx2.p2.1),[Utility maximization](https://arxiv.org/html/2608.10327#Sx6.SSx3.SSSx1.p1.1),[Utility maximization](https://arxiv.org/html/2608.10327#Sx6.SSx3.SSSx1.p3.1),[Discussion](https://arxiv.org/html/2608.10327#Sx6.p1.1),[Discussion](https://arxiv.org/html/2608.10327#Sx6.p3.1),[Discussion](https://arxiv.org/html/2608.10327#Sx6.p4.1)\.
- D\. Graeber \(2011\)Debt: the first 5,000 years\.Melville House,Brooklyn, N\.Y\.External Links:ISBN 978\-1\-933633\-86\-2Cited by:[Anthropological, situated theories of value](https://arxiv.org/html/2608.10327#Sx6.SSx1.SSSx2.p1.1),[Alternative conceptions of value](https://arxiv.org/html/2608.10327#Sx6.SSx1.p2.1),[Discussion](https://arxiv.org/html/2608.10327#Sx6.p4.1)\.
- O\. Guest \(2026\)What does ‘human\-centred ai’mean?\.Behavioral Sciences16\(4\),pp\. 583\.Cited by:[Introduction](https://arxiv.org/html/2608.10327#Sx1.p1.1)\.
- H\. Guo, Y\. Yao, W\. Shen, J\. Wei, X\. Zhang, Z\. W\. Wang, and Y\. Liu \(2024\)Human\-instruction\-free llm self\-alignment with limited samples\.\.External Links:[Link](https://arxiv.org/abs/2401.06785)Cited by:[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p7.1)\.
- B\. Gyevnar and A\. Kasirzadeh \(2025\)AI safety for everyone\.Nature Machine Intelligence,pp\. 1–12\.Cited by:[Limitations](https://arxiv.org/html/2608.10327#Sx7.p1.1)\.
- D\. Hadfield\-Menell, A\. Dragan, P\. Abbeel, and S\. Russell \(2016\)Cooperative inverse reinforcement learning\.InProceedings of the 30th International Conference on Neural Information Processing Systems,NIPS’16,Red Hook, NY, USA,pp\. 3916–3924\.External Links:ISBN 978\-1\-5108\-3881\-9Cited by:[What is value alignment?](https://arxiv.org/html/2608.10327#Sx2.SSx1.p1.1),[Utility maximization](https://arxiv.org/html/2608.10327#Sx6.SSx3.SSSx1.p1.1),[Utility maximization](https://arxiv.org/html/2608.10327#Sx6.SSx3.SSSx1.p2.1)\.
- S\. Haslanger \(2022\)Failures of methodological individualism: the materiality of social systems\.\.Journal of Social Philosophy53\(4\)\.Cited by:[Justification for rubric questions](https://arxiv.org/html/2608.10327#Sx4.SSx2.SSSx1.p2.1)\.
- S\. Haslanger \(2023\)Situated Knowledge and Situated Values\.\(en\)\.Cited by:[Justification for rubric questions](https://arxiv.org/html/2608.10327#Sx4.SSx2.SSSx1.p2.1),[Cognitive or psychological theories of value](https://arxiv.org/html/2608.10327#Sx6.SSx1.SSSx1.p1.1),[Discussion](https://arxiv.org/html/2608.10327#Sx6.p1.1)\.
- V\. Hofmann, P\. R\. Kalluri, D\. Jurafsky, and S\. King \(2024\)AI generates covertly racist decisions about people based on their dialect\.Nature633\(8028\),pp\. 147–154\(en\)\.External Links:ISSN 1476\-4687,[Link](https://www.nature.com/articles/s41586-024-07856-5),[Document](https://dx.doi.org/10.1038/s41586-024-07856-5)Cited by:[What is value alignment?](https://arxiv.org/html/2608.10327#Sx2.SSx1.p1.1)\.
- I\. Hong, Z\. Li, A\. Bukharin, Y\. Li, H\. Jiang, T\. Yang, and T\. Zhao \(2024\)Adaptive preference scaling for reinforcement learning with human feedback\.Advances in Neural Information Processing Systems37,pp\. 107249–107269\.Cited by:[Discussion](https://arxiv.org/html/2608.10327#Sx6.p2.1)\.
- S\. Hooker \(2025\)On the slow death of scaling\.Available at SSRN 5877662\.Cited by:[What is value alignment?](https://arxiv.org/html/2608.10327#Sx2.SSx1.p1.1)\.
- A\. Hornborg \(1998\)Ecological embeddedness and personhood: have we always been capitalists?\.Anthropology today14\(2\),pp\. 3–5\.Cited by:[Alternative conceptions of value](https://arxiv.org/html/2608.10327#Sx6.SSx1.p2.1)\.
- A\. Hornborg \(2014\)Technology as Fetish: Marx, Latour, and the Cultural Foundations of Capitalism\.Theory, Culture & Society31\(4\),pp\. 119–140\(en\)\.Note:Publisher: SAGE Publications LtdExternal Links:ISSN 0263\-2764,[Link](https://doi.org/10.1177/0263276413488960),[Document](https://dx.doi.org/10.1177/0263276413488960)Cited by:[Anthropological, situated theories of value](https://arxiv.org/html/2608.10327#Sx6.SSx1.SSSx2.p1.1)\.
- Y\. Huang, S\. Gupta, M\. Xia, K\. Li, and D\. Chen \(2023\)Catastrophic jailbreak of open\-source llms via exploiting generation\.\.External Links:[Link](https://arxiv.org/pdf/2310.06987)Cited by:[Discussion](https://arxiv.org/html/2608.10327#Sx6.p2.1)\.
- U\. A\. S\. Institute \(2025\)UK AI Security Institute Alignment Project\.\(en\-US\)\.External Links:[Link](https://alignmentproject.aisi.gov.uk/about)Cited by:[What is value alignment?](https://arxiv.org/html/2608.10327#Sx2.SSx1.p3.1)\.
- J\. Ji, T\. Qiu, B\. Chen, B\. Zhang, H\. Lou, K\. Wang, Y\. Duan, Z\. He, J\. Zhou, Z\. Zhang,et al\.\(2023\)Ai alignment: a comprehensive survey\.arXiv preprint arXiv:2310\.19852\.Cited by:[Introduction](https://arxiv.org/html/2608.10327#Sx1.p1.1),[What is value alignment?](https://arxiv.org/html/2608.10327#Sx2.SSx1.p4.1),[Preferences are not all you need](https://arxiv.org/html/2608.10327#Sx2.SSx3.p1.1)\.
- G\. M\. Johnson \(2023\)Are Algorithms Value\-Free?\.Journal Moral Philosophy21\(1\-2\),pp\. 1–35\.External Links:[Link](https://philarchive.org/rec/JOHAAV),[Document](https://dx.doi.org/10.1163/17455243-20234372)Cited by:[Preferences are not all you need](https://arxiv.org/html/2608.10327#Sx2.SSx3.p1.1)\.
- A\. Kasirzadeh \(2024a\)Plurality of value pluralism and AI value alignment\.\(en\)\.External Links:[Link](https://openreview.net/forum?id=AOokh1UYLH)Cited by:[Justification for rubric questions](https://arxiv.org/html/2608.10327#Sx4.SSx2.SSSx1.p2.1),[Discussion](https://arxiv.org/html/2608.10327#Sx6.p4.1),[Implications for future work](https://arxiv.org/html/2608.10327#Sx7.SSx1.p1.1)\.
- A\. Kasirzadeh \(2024b\)Two types of ai existential risk: decisive and accumulative\.arXiv preprint arXiv:2401\.07836\.Cited by:[What is value alignment?](https://arxiv.org/html/2608.10327#Sx2.SSx1.p2.1)\.
- A\. Khan, S\. Casper, and D\. Hadfield\-Menell \(2025\)Randomness, not representation: the unreliability of evaluating cultural alignment in llms\.InProceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency,pp\. 2151–2165\.Cited by:[Discussion](https://arxiv.org/html/2608.10327#Sx6.p5.1)\.
- H\. Khlaaf \(2023\)Toward comprehensive risk assessments and assurance of ai\-based systems\.Trail of Bits7\.Cited by:[Alignment and its discontents](https://arxiv.org/html/2608.10327#Sx2.SSx2.p4.1)\.
- H\. Kim, X\. Yi, J\. Yao, J\. Lian, M\. Huang, S\. Duan, J\. Bak, and X\. Xie \(2024\)The road to artificial superintelligence: a comprehensive survey of superalignment\.\.External Links:[Link](https://arxiv.org/abs/2412.16468)Cited by:[Discussion](https://arxiv.org/html/2608.10327#Sx6.p2.1)\.
- H\. R\. Kirk, B\. Vidgen, P\. Röttger, and S\. A\. Hale \(2023a\)Personalisation within bounds: a risk taxonomy and policy framework for the alignment of large language models with personalised feedback\.\.External Links:[Link](https://arxiv.org/pdf/2303.05453)Cited by:[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p1.1)\.
- H\. R\. Kirk, B\. Vidgen, P\. Röttger, and S\. A\. Hale \(2023b\)The empty signifier problem: towards clearer paradigms for operationalising” alignment” in large language models\.arXiv preprint arXiv:2310\.02457\.Cited by:[Alignment and its discontents](https://arxiv.org/html/2608.10327#Sx2.SSx2.p4.1)\.
- T\. Q\. Klassen, P\. A\. Alamdari, and S\. A\. McIlraith \(2024\)Pluralistic Alignment Over Time\.\(en\)\.External Links:[Link](https://openreview.net/forum?id=BLhVcPbEo8)Cited by:[Preferences are not all you need](https://arxiv.org/html/2608.10327#Sx2.SSx3.p3.1),[Justification for rubric questions](https://arxiv.org/html/2608.10327#Sx4.SSx2.SSSx1.p2.1)\.
- A\. Köpf, Y\. Kilcher, D\. Von Rütte, S\. Anagnostidis, Z\. R\. Tam, K\. Stevens, A\. Barhoum, D\. Nguyen, O\. Stanley, R\. Nagyfi,et al\.\(2023\)Openassistant conversations\-democratizing large language model alignment\.Advances in neural information processing systems36,pp\. 47669–47681\.Cited by:[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p3.1)\.
- C\. M\. Korsgaard \(1986\)Aristotle and kant on the source of value\.Ethics96\(3\),pp\. 486–505\.Cited by:[Justification for rubric questions](https://arxiv.org/html/2608.10327#Sx4.SSx2.SSSx1.p1.1)\.
- M\. Lambek \(2013\)The value of \(performative\) acts\.HAU: Journal of Ethnographic Theory3\(2\),pp\. 141–160\.Cited by:[Anthropological, situated theories of value](https://arxiv.org/html/2608.10327#Sx6.SSx1.SSSx2.p2.1)\.
- J\. Leike, D\. Krueger, T\. Everitt, M\. Martic, V\. Maini, and S\. Legg \(2018\)Scalable agent alignment via reward modeling: a research direction\.\.External Links:[Link](https://arxiv.org/abs/1811.07871)Cited by:[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p6.1)\.
- X\. Li, P\. Yu, C\. Zhou, T\. Schick, L\. Zettlemoyer, O\. Levy, Z\. Luke, J\. Weston, and M\. Lewis \(2024\)Self\-alignment with instruction backtranslation\.\.External Links:[Link](https://arxiv.org/pdf/2308.06259)Cited by:[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p7.1)\.
- A\. D\. Lindström, L\. Methnani, L\. Krause, P\. Ericson, Í\. M\. d\. R\. de Troya, D\. C\. Mollo, and R\. Dobbe \(2024\)AI alignment through reinforcement learning from human feedback? contradictions and limitations\.arXiv preprint arXiv:2406\.18346\.Cited by:[Alignment and its discontents](https://arxiv.org/html/2608.10327#Sx2.SSx2.p2.1),[Alignment and its discontents](https://arxiv.org/html/2608.10327#Sx2.SSx2.p4.1),[Preferences are not all you need](https://arxiv.org/html/2608.10327#Sx2.SSx3.p1.1),[Preferences are not all you need](https://arxiv.org/html/2608.10327#Sx2.SSx3.p3.1),[Alternative conceptions of value](https://arxiv.org/html/2608.10327#Sx6.SSx1.p4.1)\.
- J\. Liu, D\. Ge, and R\. Zhu \(2024\)Reward learning from preference with ties\.\.External Links:[Link](https://arxiv.org/abs/2410.05328)Cited by:[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p2.1),[Discussion](https://arxiv.org/html/2608.10327#Sx6.p2.1)\.
- R\. Liu, R\. Yang, C\. Jia, G\. Zhang, D\. Zhou, A\. M\. Dai, D\. Yang, and S\. Vosoughi \(2023\)Training socially aligned language models on simulated social interactions\.\.External Links:[Link](https://arxiv.org/abs/2305.16960)Cited by:[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p4.1),[Discussion](https://arxiv.org/html/2608.10327#Sx6.p2.1)\.
- X\. Lou, J\. Zhang, J\. Xie, L\. Liu, D\. Yan, and K\. Huang \(2024\)SPO: multi\-dimensional preference sequential alignment with implicit reward modeling\.\.External Links:[Link](https://arxiv.org/abs/2405.12739)Cited by:[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p2.1),[Discussion](https://arxiv.org/html/2608.10327#Sx6.p2.1)\.
- K\. Lu, B\. Yu, F\. Huang, Y\. Fan, R\. Lin, and C\. Zhou \(2024\)Online merging optimizers for boosting rewards and mitigating tax in alignment\.\.External Links:[Link](https://arxiv.org/abs/2405.17931)Cited by:[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p2.1),[Discussion](https://arxiv.org/html/2608.10327#Sx6.p2.1)\.
- S\. Mau \(2023\)Mute compulsion: a marxist theory of the economic power of capital\.Verso books\.Cited by:[Anthropological, situated theories of value](https://arxiv.org/html/2608.10327#Sx6.SSx1.SSSx2.p1.1)\.
- N\. Maxwell \(2005\)The comprehensibility of the universe: a new conception of science\.Clarendon press,Oxford\(eng\)\.External Links:ISBN 978\-0\-19\-926155\-0Cited by:[Utility maximization](https://arxiv.org/html/2608.10327#Sx6.SSx3.SSSx1.p2.1)\.
- M\. Mazeika, X\. Yin, R\. Tamirisa, J\. Lim, B\. W\. Lee, R\. Ren, L\. Phan, N\. Mu, A\. Khoja, O\. Zhang,et al\.\(2025\)Utility engineering: analyzing and controlling emergent value systems in ais\.arXiv preprint arXiv:2502\.08640\.Cited by:[Utility maximization](https://arxiv.org/html/2608.10327#Sx6.SSx3.SSSx1.p4.1)\.
- L\. Mei, S\. Liu, Y\. Wang, B\. Bi, R\. Yuan, and X\. Cheng \(2024\)HiddenGuard: fine\-grained safe generation with specialized representation router\.\.External Links:[Link](https://arxiv.org/abs/2410.02684)Cited by:[Discussion](https://arxiv.org/html/2608.10327#Sx6.p2.1)\.
- J\. S\. Mill \(2016\)Utilitarianism\.InSeven masterpieces of philosophy,pp\. 329–375\.Cited by:[Justification for rubric questions](https://arxiv.org/html/2608.10327#Sx4.SSx2.SSSx1.p1.1)\.
- R\. Millière \(2025\)Normative conflicts and shallow ai alignment: r\. millière\.Philosophical Studies,pp\. 1–44\.Cited by:[Alternative conceptions of value](https://arxiv.org/html/2608.10327#Sx6.SSx1.p3.1)\.
- L\. J\. V\. Miranda, Y\. Wang, Y\. Elazar, S\. Kumar, V\. Pyatkin, F\. Brahman, N\. A\. Smith, H\. Hajishirzi, and Dasigi,Pradeep \(2024\)Hybrid preferences: learning to route instances for human vs\. ai feedback\.\.External Links:[Link](https://arxiv.org/abs/2410.19133)Cited by:[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p2.1),[Discussion](https://arxiv.org/html/2608.10327#Sx6.p2.1)\.
- C\. Nauer and Z\. van Doren \(2025\)EXISTENTIAL risk from ai\.Cited by:[What is value alignment?](https://arxiv.org/html/2608.10327#Sx2.SSx1.p2.1)\.
- A\. Nelson \(2023\)FAccT’23 Keynote: ”Thick Alignment”\.\(en\-US\)\.External Links:[Link](https://www.youtube.com/watch?v=Sq_XwqVTqvQ)Cited by:[Alternative conceptions of value](https://arxiv.org/html/2608.10327#Sx6.SSx1.p1.1.1)\.
- R\. Ngo, L\. Chan, and Mindermann,Sören \(2022\)The alignment problem from a deep learning perspective\.\.External Links:[Link](https://arxiv.org/abs/2209.00626)Cited by:[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p7.1),[Discussion](https://arxiv.org/html/2608.10327#Sx6.p2.1)\.
- OpenAI \(2025\)OpenAI Alignment Research Blog\.\(en\-US\)\.External Links:[Link](https://alignment.openai.com/)Cited by:[Alignment and its discontents](https://arxiv.org/html/2608.10327#Sx2.SSx2.p3.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[What is value alignment?](https://arxiv.org/html/2608.10327#Sx2.SSx1.p1.1),[4th item](https://arxiv.org/html/2608.10327#Sx4.I2.i4.p1.1),[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p6.1),[Utility maximization](https://arxiv.org/html/2608.10327#Sx6.SSx3.SSSx1.p2.1)\.
- X\. Pang, S\. Tang, R\. Ye, Y\. Xiong, B\. Zhang, Y\. Wang, and S\. Chen \(2024\)Self\-alignment of large language models via monopolylogue\-based social scene simulation\.\.External Links:[Link](https://arxiv.org/abs/2402.05699)Cited by:[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p7.1)\.
- C\. Parker \(2019\)Snowball sampling\.SAGE Research Methods Foundations\.Cited by:[Snowball sampling](https://arxiv.org/html/2608.10327#Sx4.SSx1.p1.1)\.
- S\. Phelps and R\. Ranson \(2023a\)Of models and tin men: a behavioural economics study of principal\-agent problems in ai alignment using large\-language models\.arXiv preprint arXiv:2307\.11137\.Cited by:[Utility maximization](https://arxiv.org/html/2608.10327#Sx6.SSx3.SSSx1.p3.1),[Utility maximization](https://arxiv.org/html/2608.10327#Sx6.SSx3.SSSx1.p5.1)\.
- S\. Phelps and R\. Ranson \(2023b\)Of models and tin men: a behavioural economics study of principal\-agent problems in ai alignment using large\-language models\.\.External Links:[Link](https://arxiv.org/abs/2307.11137)Cited by:[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p7.1)\.
- S\. Poddar, Y\. Wan, H\. Ivison, A\. Gupta, and N\. Jaques \(2024\)Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning\.\(en\)\.External Links:[Link](https://openreview.net/forum?id=zsXKtupv2I)Cited by:[Utility maximization](https://arxiv.org/html/2608.10327#Sx6.SSx3.SSSx1.p5.1),[Discussion](https://arxiv.org/html/2608.10327#Sx6.p2.1),[Discussion](https://arxiv.org/html/2608.10327#Sx6.p5.1)\.
- V\. Prabhakaran, M\. Mitchell, T\. Gebru, and I\. Gabriel \(2022\)A Human Rights\-Based Approach to Responsible AI\.arXiv\.Note:arXiv:2210\.02667 \[cs\]Comment: Presented as a \(non\-archival\) poster at the 2022 ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization or \(EAAMO ’22\)External Links:[Link](http://arxiv.org/abs/2210.02667),[Document](https://dx.doi.org/10.48550/arXiv.2210.02667)Cited by:[Discussion](https://arxiv.org/html/2608.10327#Sx6.p5.1)\.
- R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.Advances in neural information processing systems36,pp\. 53728–53741\.Cited by:[Preferences are not all you need](https://arxiv.org/html/2608.10327#Sx2.SSx3.p1.1)\.
- I\. D\. Raji and R\. Dobbe \(2023\)Concrete problems in ai safety, revisited\.arXiv preprint arXiv:2401\.10899\.Cited by:[Alignment and its discontents](https://arxiv.org/html/2608.10327#Sx2.SSx2.p5.1)\.
- P\. H\. Richemond, Y\. Tang, D\. Guo, D\. Calandriello, M\. G\. Azar, R\. Rafailov, B\. A\. Pires, E\. Tarassov, L\. Spangher, W\. Ellsworth, A\. Severyn, J\. Mallinson, L\. Shani, G\. Shamir, R\. Joshi, T\. Liu, R\. Munos, and B\. Piot \(2024\)Offline regularised reinforcement learning for large language models alignment\.\.External Links:[Link](https://arxiv.org/abs/2405.19107)Cited by:[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p2.1)\.
- S\. J\. Russell and P\. Norvig \(1995\)Artificial intelligence: a modern approach;\[the intelligent agent book\]\.Prentice hall\.Cited by:[Preferences are not all you need](https://arxiv.org/html/2608.10327#Sx2.SSx3.p1.1),[Preferences are not all you need](https://arxiv.org/html/2608.10327#Sx2.SSx3.p2.1),[Utility maximization](https://arxiv.org/html/2608.10327#Sx6.SSx3.SSSx1.p1.1)\.
- S\. J\. Russell \(2019\)Human compatible: artificial intelligence and the problem of control\.Viking,New York\(eng\)\.External Links:ISBN 978\-0\-525\-55861\-3Cited by:[Introduction](https://arxiv.org/html/2608.10327#Sx1.p1.1),[Utility maximization](https://arxiv.org/html/2608.10327#Sx6.SSx3.SSSx1.p2.1)\.
- S\. Schmer\-Galunder \(2026\)Culture in the code: anthropological concepts decoding ai’s hidden assumptions\.AI and Ethics6\(3\),pp\. 272\.Cited by:[Justification for rubric questions](https://arxiv.org/html/2608.10327#Sx4.SSx2.SSSx1.p2.1),[Anthropological, situated theories of value](https://arxiv.org/html/2608.10327#Sx6.SSx1.SSSx2.p1.1)\.
- S\. Schwartz \(2012\)An Overview of the Schwartz Theory of Basic Values\.Online Readings in Psychology and Culture2\(1\)\.External Links:ISSN 2307\-0919,[Link](https://scholarworks.gvsu.edu/orpc/vol2/iss1/11),[Document](https://dx.doi.org/10.9707/2307-0919.1116)Cited by:[Cognitive or psychological theories of value](https://arxiv.org/html/2608.10327#Sx6.SSx1.SSSx1.p1.1)\.
- A\. Sharma, S\. Keh, E\. Mitchell, C\. Finn, K\. Arora, and T\. Kollar \(2024\)A critical evaluation of ai feedback for aligning large language models\.arXiv preprint arXiv:2402\.12366\.Cited by:[What is value alignment?](https://arxiv.org/html/2608.10327#Sx2.SSx1.p1.1)\.
- N\. Shea \(2018\)Representation in cognitive science\.Oxford University Press\.Cited by:[Discussion](https://arxiv.org/html/2608.10327#Sx6.p4.1)\.
- T\. Shen, R\. Jin, Y\. Huang, C\. Liu, W\. Dong, Z\. Guo, X\. Wu, Y\. Liu, and Xiong,Deyi \(2023\)Large language model alignment: a survey\.\.External Links:[Link](https://arxiv.org/pdf/2309.15025)Cited by:[Discussion](https://arxiv.org/html/2608.10327#Sx6.p2.1)\.
- Z\. Shi, Z\. Wang, H\. Fan, Z\. Zhang, L\. Li, Y\. Zhang, Z\. Yin, L\. Sheng, Y\. Qiao, and J\. Shao \(2024\)Assessment of multimodal large language models in alignment with human values\.\.External Links:[Link](https://arxiv.org/abs/2403.17830)Cited by:[Discussion](https://arxiv.org/html/2608.10327#Sx6.p2.1)\.
- K\. Shrishak \(2025\)How not to deploy generative AI: the Story of the European Parliament\.\(en\-US\)\.External Links:[Link](https://www.iccl.ie/press-release/how-not-to-deploy-generative-ai-the-story-of-the-european-parliament/)Cited by:[What is value alignment?](https://arxiv.org/html/2608.10327#Sx2.SSx1.p3.1)\.
- C\. Sierra, N\. Osman, P\. Noriega, J\. Sabater\-Mir, and A\. Perelló \(2021\)Value alignment: a formal approach\.arXiv\.Note:arXiv:2110\.09240 \[cs\] version: 1Comment: accepted paper at the Responsible Artificial Intelligence Agents Workshop, of the 18th International Conference on Autonomous Agents and MultiAgent Systems \(AAMAS 2019\)External Links:[Link](http://arxiv.org/abs/2110.09240),[Document](https://dx.doi.org/10.48550/arXiv.2110.09240)Cited by:[Cognitive or psychological theories of value](https://arxiv.org/html/2608.10327#Sx6.SSx1.SSSx1.p1.1)\.
- A\. Siththaranjan, C\. Laidlaw, and D\. Hadfield\-Menell \(2023\)Distributional preference learning: understanding and accounting for hidden context in rlhf\.\.External Links:[Link](https://arxiv.org/abs/2312.08358)Cited by:[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p3.1)\.
- T\. Sorensen, J\. Moore, J\. Fisher, M\. Gordon, N\. Mireshghallah, C\. M\. Rytting, A\. Ye, L\. Jiang, X\. Lu, N\. Dziri, T\. Althoff, and Y\. Choi \(2024\)A Roadmap to Pluralistic Alignment\.arXiv\.Note:arXiv:2402\.05070 \[cs\]Comment: ICML 2024External Links:[Link](http://arxiv.org/abs/2402.05070),[Document](https://dx.doi.org/10.48550/arXiv.2402.05070)Cited by:[Preferences are not all you need](https://arxiv.org/html/2608.10327#Sx2.SSx3.p3.1),[Justification for rubric questions](https://arxiv.org/html/2608.10327#Sx4.SSx2.SSSx1.p2.1),[Implications for future work](https://arxiv.org/html/2608.10327#Sx7.SSx1.p1.1)\.
- K\. Stanczak, N\. Meade, M\. Bhatia, H\. Zhou, K\. Böttinger, J\. Barnes, J\. Stanley, J\. Montgomery, R\. Zemel, N\. Papernot,et al\.\(2025\)Societal alignment frameworks can improve llm alignment\.InICLR 2025 Workshop on Bidirectional Human\-AI Alignment,Cited by:[Utility maximization](https://arxiv.org/html/2608.10327#Sx6.SSx3.SSSx1.p3.1)\.
- H\. Sun, Y\. Shen, and J\. Ton \(2024\)Rethinking bradley\-terry models in preference\-based reward modeling: foundations, theory, and alternatives\.arXiv preprint arXiv:2411\.04991\.Cited by:[What is value alignment?](https://arxiv.org/html/2608.10327#Sx2.SSx1.p1.1)\.
- R\. Thoppilan, D\. De Freitas, J\. Hall, N\. Shazeer, A\. Kulshreshtha, H\. Cheng, A\. Jin, T\. Bos, L\. Baker, Y\. Du,et al\.\(2022\)Lamda: language models for dialog applications\.arXiv preprint arXiv:2201\.08239\.Cited by:[What is value alignment?](https://arxiv.org/html/2608.10327#Sx2.SSx1.p4.1)\.
- R\. Tian, C\. Xu, M\. Tomizuka, J\. Malik, and A\. V\. Bajcsy \(2024\)What matters to you? towards visual representation alignment for robot learning\.\.External Links:[Link](https://arxiv.org/pdf/2310.07932)Cited by:[Discussion](https://arxiv.org/html/2608.10327#Sx6.p2.1)\.
- B\. Wang, R\. Zheng, L\. Chen, Y\. Liu, S\. Dou, C\. Huang, W\. Shen, S\. Jin, E\. Zhou, C\. Shi, S\. Gao, N\. Xu, Y\. Zhou, Z\. Fan, J\. Zhao, X\. Wang, T\. Ji, H\. Yan, L\. Shen, Z\. Chen, T\. Gui, Q\. Zhang, X\. Qiu, X\. Huang, Z\. Wu, and Y\. Jiang \(2024a\)Secrets of rlhf in large language models part ii: reward modeling\.\.External Links:[Link](https://arxiv.org/abs/2401.06080)Cited by:[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p2.1),[Discussion](https://arxiv.org/html/2608.10327#Sx6.p2.1)\.
- C\. Wang, D\. Zhao, B\. Wang, R\. He, and Y\. Hou \(2024b\)Do llms have the generalization ability in conducting causal inference?\.arXiv preprint arXiv:2410\.11385\.Cited by:[Utility maximization](https://arxiv.org/html/2608.10327#Sx6.SSx3.SSSx1.p2.1)\.
- Z\. Wang, W\. He, Z\. Liang, X\. Zhang, C\. Bansal, Y\. Wei, W\. Zhang, and H\. Yao \(2024c\)CREAM: consistency regularized self\-rewarding language models\.\.External Links:[Link](https://arxiv.org/abs/2410.12735)Cited by:[Discussion](https://arxiv.org/html/2608.10327#Sx6.p2.1)\.
- C\. Xu, S\. Chern, E\. Chern, G\. Zhang, Z\. Wang, R\. Liu, J\. Li, J\. Fu, and P\. Li \(2023\)Align on the fly: adapting chatbot behavior to established norms\.\.External Links:[Link](https://arxiv.org/abs/2312.15907)Cited by:[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p1.1)\.
- T\. Yu, T\. Lin, Y\. Wu, M\. Yang, F\. Huang, and Y\. Li \(2023\)Constructive large language models alignment with diverse feedback\.\.External Links:[Link](https://arxiv.org/abs/2310.06450)Cited by:[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p2.1),[Discussion](https://arxiv.org/html/2608.10327#Sx6.p2.1)\.
- Y\. Zeng, G\. Liu, W\. Ma, N\. Yang, H\. Zhang, and J\. Wang \(2024\)Token\-level direct preference optimization\.\.External Links:[Link](https://arxiv.org/abs/2404.11999)Cited by:[Discussion](https://arxiv.org/html/2608.10327#Sx6.p2.1)\.
- S\. Zhao, J\. Dang, and A\. Grover \(2023\)Group preference optimization: few\-shot alignment of large language models\.\.External Links:[Link](https://arxiv.org/abs/2310.11523)Cited by:[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p2.1)\.
- C\. Zheng, K\. Sun, H\. Wu, C\. Xi, and X\. Zhou \(2024\)Balancing enhancement, harmlessness, and general capabilities: enhancing conversational llms with direct rlhf\.\.External Links:[Link](https://arxiv.org/abs/2403.02513)Cited by:[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p2.1),[Discussion](https://arxiv.org/html/2608.10327#Sx6.p2.1)\.
- T\. Zhi\-Xuan, M\. Carroll, M\. Franklin, and H\. Ashton \(2024\)Beyond Preferences in AI Alignment\.Philosophical Studies\.Note:arXiv:2408\.16984 \[cs\]Comment: 26 pages \(excl\. references\), 5 figuresExternal Links:ISSN 0031\-8116, 1573\-0883,[Link](http://arxiv.org/abs/2408.16984),[Document](https://dx.doi.org/10.1007/s11098-024-02249-w)Cited by:[Introduction](https://arxiv.org/html/2608.10327#Sx1.p1.1),[Preferences are not all you need](https://arxiv.org/html/2608.10327#Sx2.SSx3.p2.1),[Alternative conceptions of value](https://arxiv.org/html/2608.10327#Sx6.SSx1.p4.1),[Utility maximization](https://arxiv.org/html/2608.10327#Sx6.SSx3.SSSx1.p2.1),[Utility maximization](https://arxiv.org/html/2608.10327#Sx6.SSx3.SSSx1.p6.1),[Discussion](https://arxiv.org/html/2608.10327#Sx6.p1.1),[Discussion](https://arxiv.org/html/2608.10327#Sx6.p4.1)\.
- Y\. Zhong, C\. Ma, X\. Zhang, Z\. Yang, H\. Chen, Q\. Zhang, S\. Qi, and Y\. Yang \(2024\)Panacea: pareto alignment via preference adaptation for llms\.\.External Links:[Link](https://arxiv.org/abs/2402.02030)Cited by:[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p2.1)\.
- Z\. Zhou, Z\. Liu, J\. Liu, Z\. Dong, C\. Yang, and Y\. Qiao \(2024\)Weak\-to\-strong search: align large language models via searching over small language models\.\.External Links:[Link](https://arxiv.org/abs/2405.19262)Cited by:[Qualitative Analysis](https://arxiv.org/html/2608.10327#Sx5.SSx3.p4.1)\.