MAVEN: A Macro-Societal Value Evaluation Framework of Multimodal Content with Compact Aligned Evaluators

# MAVEN: A Macro-Societal Value Evaluation Framework of Multimodal Content with Compact Aligned Evaluators
Source: [https://arxiv.org/html/2608.18096](https://arxiv.org/html/2608.18096)
Zijuan Zhao1,Zheren Fu1,Hou Xia1,Licheng Zhang1, Yi Liu2,Zhendong Mao1,\* 1University of Science and Technology of China 2State Key Laboratory of Communication Content Cognition, People’s Daily Online rq020616@mail\.ustc\.edu\.cn,\{fzr, zdmao\}@ustc\.edu\.cn

###### Abstract

Assessing whether multimodal content aligns with macro\-societal values, such as peace, justice, and freedom, has become an increasingly urgent challenge\. Existing frameworks are largely confined to safety\-oriented taxonomies, text\-only psychometric probes, or single\-label classification\. Therefore, we propose MAVEN, a hierarchical framework for macro\-societal value evaluation of multimodal content, grounded in international human\-rights instruments and cultural value theory\. MAVEN organizes values into 6 primary dimensions and 72 secondary indicators, supporting multi\-level quantitative scoring\. Building on MAVEN, we construct a human\-verified multimodal benchmark and a soft\-match metric to evaluate VLMs’ assessments across value dimensions\. For evaluator optimization, we propose a span\-adaptive variant of multi\-level preference optimization for evaluator distillation, together with a training\-free multi\-role consensus strategy at inference time\. We evaluate existing open\- and closed\-source VLMs on our benchmark, revealing shared tendencies and clear differences in macro\-societal value judgments\. Experiments show that our compact 2B evaluator matches its 8B counterpart in the same family and approaches frontier closed\-source VLMs, offering a practical path toward scalable macro\-societal value evaluation\. Our SA\-MDPO implementation and MacroValue\-Bench are available at https://github\.com/zzzzzzzzjj/MAVEN\.

MAVEN: A Macro\-Societal Value Evaluation Framework of Multimodal Content with Compact Aligned Evaluators

Zijuan Zhao1, Zheren Fu1, Hou Xia1, Licheng Zhang1,Yi Liu2,Zhendong Mao1,\*1University of Science and Technology of China2State Key Laboratory of Communication Content Cognition, People’s Daily Onlinerq020616@mail\.ustc\.edu\.cn,\{fzr, zdmao\}@ustc\.edu\.cn

## 1Introduction

Table 1:Comparison of value frameworks used for AI evaluation\. ✓ = full support,⚫= partial, ✗ = not supported\.Macro\.\(Macro\-societal\): targets societal\-level values rather than individual ethics or safety\.Multi\.\(Multimodal\): operationalized on image–text content\.Quant\.\(Quantitative\): yields ordinal or continuous scores rather than categorical labels\.![Refer to caption](https://arxiv.org/html/2608.18096v1/x1.png)Figure 1:Overview of the MAVEN hierarchical framework and evaluator optimization\. \(I\) MAVEN Framework: Macro\-societal value taxonomy with 6 primary dimensions and 72 secondary indicators\. Secondary indicators are constructed through an iterative human–AI collaborative pipeline to ensure comprehensive coverage and conceptual clarity\. \(II\) Evaluator Optimization: A compact student VLM is distilled from frontier teachers via our proposed SA\-MDPO\. Then multi\-role consensus aggregates different stakeholder perspectives at inference time\.Macro\-societal valuesrefer to shared abstractions about what is good and desirable at the level of a society \(e\.g\., peace, justice\)\. These values are sustained by institutional and cultural systems rather than reducible to individual preferencesWilliams Jr \([1979](https://arxiv.org/html/2608.18096#bib.bib50)\); Schwartz \([2006](https://arxiv.org/html/2608.18096#bib.bib22)\)\. As multimodal content proliferates across news and social media, automatically assessing its alignment with macro\-societal values has become urgent\. With strong cross\-modal understanding and instruction following, Vision Language Models \(VLMs\)Liet al\.\([2025](https://arxiv.org/html/2608.18096#bib.bib1)\)make this attainable: they can serve as value evaluatorsAbbo and Belpaeme \([2024](https://arxiv.org/html/2608.18096#bib.bib48)\); Yasunagaet al\.\([2025](https://arxiv.org/html/2608.18096#bib.bib49)\)as well as content generators\.

Realizing this potential requires a framework that operationalizes macro\-societal values for multimodal evaluation\. Prior work on value alignment provides useful taxonomies\. However, most frameworks are not well\-suited for VLM\-based macro\-societal evaluation with three limitations\. First, safety\-oriented taxonomiesHendryckset al\.\([2020](https://arxiv.org/html/2608.18096#bib.bib9)\); Parrishet al\.\([2022](https://arxiv.org/html/2608.18096#bib.bib36)\); Baiet al\.\([2022](https://arxiv.org/html/2608.18096#bib.bib8)\)target micro\-ethical concerns \(toxicity, bias, harm avoidance\), leaving macro\-societal values largely uncovered\. Second, most value\-orientation benchmarksRenet al\.\([2024](https://arxiv.org/html/2608.18096#bib.bib10)\); Yaoet al\.\([2025](https://arxiv.org/html/2608.18096#bib.bib14)\); Shenet al\.\([2025](https://arxiv.org/html/2608.18096#bib.bib34)\); Razaet al\.\([2025](https://arxiv.org/html/2608.18096#bib.bib18)\)rely on text\-only psychometric questionnaires, decoupling evaluation from multimodal content in the real world\. Third, when value evaluation is extended to VLMsHuet al\.\([2024](https://arxiv.org/html/2608.18096#bib.bib32)\); Linet al\.\([2025](https://arxiv.org/html/2608.18096#bib.bib33)\), the protocols typically operate at the level of single\-label categorization or open\-ended commentary, offering limited quantitative granularity for content\-level value assessment\. In summary, the multimodal and quantitative assessment required for macro\-societal value governance remains underexplored\.

We therefore proposeMAVEN\(Multimodal Macro\-societal Value Evaluation Network\), a hierarchical value framework grounded in international human\-rights instruments and cultural value theory \(Figure[1](https://arxiv.org/html/2608.18096#S1.F1), I\)\. MAVEN enables multi\-level quantitative scoring of multimodal content with fine\-grained signals suitable for downstream evaluation\. Table[1](https://arxiv.org/html/2608.18096#S1.T1)positions MAVEN against existing frameworks along the three desiderata above\. Building on MAVEN, we constructMacroValue\-Bench, a 1,157\-item multimodal benchmark with human\-verified annotations spanning diverse real\-world sources, and introduce the Value\-aware Soft Match Score \(VSMS\) metric to evaluate VLMs’ value assessments across dimensions\. We evaluate existing open\- and closed\-source VLMs on our benchmark, revealing shared tendencies and clear differences in macro\-societal value judgments\.

Through evaluation under our framework, we observe that while large frontier VLMs \(e\.g\., GPT\-5Singhet al\.\([2025](https://arxiv.org/html/2608.18096#bib.bib19)\)\) are capable evaluators, their prohibitive parameter scale renders them impractical for large\-scale evaluation scenarios\. Recent evidence suggests that small specialized models can match or even surpass large general\-purpose models on focused content\-related tasksZhanet al\.\([2025](https://arxiv.org/html/2608.18096#bib.bib2)\)\. Given the complexity of macro\-societal values under MAVEN, we introduce a span\-adaptive variant of multi\-level DPO to distill value\-judgment capability from frontier teacher VLMs into a compact 2B student \(Figure[1](https://arxiv.org/html/2608.18096#S1.F1), II\)\. We additionally introduce Multi\-Role Consensus \(MRC\) that queries a single VLM under multiple stakeholder personas as a complementary inference\-time optimization\.

On MacroValue\-Bench, our compact 2B evaluator matches its 8B counterpart in the same family and approaches frontier closed\-source baselines on both primary dimensions and secondary indicators\. Ablations further confirm that span\-adaptive Multi\-level DPO and MRC provide measurable gains on the corresponding evaluation axis\.

Our contributions can be summarized as follows:

- •We proposeMAVEN, a hierarchical value framework enabling quantitative scoring of macro\-societal values in multimodal content\. Accordingly, we constructMacroValue\-Benchfor real\-world multimodal evaluation paired with the VSMS metric\.
- •For evaluator optimization, we introduceSA\-MDPO, a span\-adaptive variant of multi\-level preference optimization for evaluator distillation, andMRC, a training\-free multi\-role consensus strategy\.
- •Experiments across open\- and closed\-source VLMs reveal both shared tendencies and clear differences\. Our compact 2B evaluator, trained with SA\-MDPO and inferred under MRC, approaches the performance of frontier closed\-source models\.

## 2Related Work

### 2\.1Human Value Theory

The study of human values spans multiple disciplines\. In cross\-cultural psychology, Schwartz’s theory of cultural\-level valuesSchwartz \([1992](https://arxiv.org/html/2608.18096#bib.bib37),[2006](https://arxiv.org/html/2608.18096#bib.bib22)\)identifies ten motivationally distinct value types validated across cultures\. Value orientations are modeled as*interdependent rather than orthogonal*\. Beyond Schwartz, Moral Foundations TheoryHaidt \([2007](https://arxiv.org/html/2608.18096#bib.bib44)\)posits universal moral concerns\. Large\-scale survey programs such as the World Values SurveyInglehart and Baker \([2000](https://arxiv.org/html/2608.18096#bib.bib45)\)document how value priorities shift across societies\.

A complementary tradition operationalizes*macro\-societal*values through international human rights instruments\. The UN CharterUnited Nations \([1945](https://arxiv.org/html/2608.18096#bib.bib39)\)establishes peace and international cooperation\. The Universal Declaration of Human Rights \(UDHR\)United Nations General Assembly \([1948](https://arxiv.org/html/2608.18096#bib.bib24)\)articulates universal entitlements across civil, political, economic, and cultural domains\. The ICCPRUnited Nations General Assembly \([1966a](https://arxiv.org/html/2608.18096#bib.bib25)\)and ICESCRUnited Nations General Assembly \([1966b](https://arxiv.org/html/2608.18096#bib.bib26)\)render them legally binding\. The 2030 AgendaUnited Nations General Assembly \([2015](https://arxiv.org/html/2608.18096#bib.bib23)\)extends this consensus to sustainable development, while the Vienna DeclarationWorld Conference on Human Rights \([1993](https://arxiv.org/html/2608.18096#bib.bib43)\)affirms the indivisibility of all human rights\.

### 2\.2Value Evaluation and Alignment

##### Text\-only value evaluation\.

Many value benchmarks remain largely text\-only\. ValueBenchRenet al\.\([2024](https://arxiv.org/html/2608.18096#bib.bib10)\)probes LLM value orientations using rephrased psychometric questionnaires, CLAVEYaoet al\.\([2024](https://arxiv.org/html/2608.18096#bib.bib13)\)couples a concept extractor with a classifier for value labeling, and ValueCompassShenet al\.\([2025](https://arxiv.org/html/2608.18096#bib.bib34)\)measures contextual alignment under specific scenarios\. These efforts evaluate the values that models express in text, not whether models can correctly assess the values embedded in multimodal content\.

##### Multimodal value evaluation and alignment\.

SPA\-VLZhanget al\.\([2025](https://arxiv.org/html/2608.18096#bib.bib27)\)and Safe RLHF\-VJiet al\.\([2025](https://arxiv.org/html/2608.18096#bib.bib28)\)target harmlessness through preference alignment\. Ch3EfShiet al\.\([2024](https://arxiv.org/html/2608.18096#bib.bib16)\)grounds VLM evaluation in the helpful\-honest\-harmless \(hhh\) axis\. These works, however, do not assess where content stands on macro\-societal values\. HumaniBenchRazaet al\.\([2025](https://arxiv.org/html/2608.18096#bib.bib18)\)introduces seven human\-centered AI principles but evaluates them through heterogeneous task\-specific metrics rather than a unified quantitative schema\. VIVAHuet al\.\([2024](https://arxiv.org/html/2608.18096#bib.bib32)\)casts value assessment as multiple\-choice action selection grounded in vision, and MORALISELinet al\.\([2025](https://arxiv.org/html/2608.18096#bib.bib33)\)evaluates moral judgment over 13 topics grounded in Turiel’s Domain Theory\. Both, however, reduce evaluation to single\-label classification\.

## 3MAVEN Framework

This section presents MAVEN in three parts: the theoretical foundation \(§[2](https://arxiv.org/html/2608.18096#S3.F2)\), the quantitative scoring schema \(§[3\.2](https://arxiv.org/html/2608.18096#S3.SS2)\), and MacroValue\-Bench \(§[3\.3](https://arxiv.org/html/2608.18096#S3.SS3)\), a human\-verified multimodal benchmark accompanied by our value\-aware soft\-match metric\.

### 3\.1Theoretical Foundation

![Refer to caption](https://arxiv.org/html/2608.18096v1/x2.png)Figure 2:Overview of MacroValue\-Bench construction and the evaluation pipeline\.\(Part A\)We collect∼\\sim30K multimodal items from four sources, sample 1,157 image–text pairs, annotate each pair via three frontier VLMs, then verify with human annotators\. Each item carries a structured dimensional label\.\(Part B\)We evaluate VLMs’ predictions against ground\-truth annotations using Value\-Aware Soft Match \(Recall direction shown; Precision is symmetric\), which assigns partial credit to semantically close cross\-dimension predictions\.Domain experts first establish the six primary dimensions anchored in our theoretical foundations\. Table[2](https://arxiv.org/html/2608.18096#S3.T2)summarizes the mapping\.

Table 2:Theoretical foundations ofMaven’s six primary dimensions\. UDHR/ICCPR/ICESCR numbers refer to article numbers\.*Pre\.*denotes Preamble\. Schwartz value abbreviations:*Univ\.*= Universalism,*Benev\.*= Benevolence,*Conf\.*= Conformity,*Self\-dir\.*= Self\-direction,*Stim\.*= Stimulation\.Indicators are constructed through an iterative human–AI collaborative pipeline\. Conditioned on each dimension, frontier modelsSinghet al\.\([2025](https://arxiv.org/html/2608.18096#bib.bib19)\); ByteDance \([2025](https://arxiv.org/html/2608.18096#bib.bib52)\); Yanget al\.\([2025](https://arxiv.org/html/2608.18096#bib.bib53)\)propose candidate secondary indicators spanning macro\- to micro\-level contexts\. Experts then curate by filtering redundant or off\-topic proposals and re\-eliciting additional candidates when coverage gaps remain\. The retained candidates are consolidated into 12 indicators per dimension, each elaborated into a standardized natural\-language definition and reviewed iteratively until all authors reach consensus\.*Peace*, for example, comprisesWorld Peace\(interstate relations\),Social Peace\(intra\-society stability\),Cyber Peace\(digital\-domain stability\), andInner Peace\(individual psychological state\)\. The full indicators are provided in Appendix[A](https://arxiv.org/html/2608.18096#A1)\.

The two\-layer structure of MAVEN serves complementary purposes: primary dimensions support large\-scale, low\-latency screening, while secondary indicators enable fine\-grained diagnostic analysis\.

### 3\.2Quantitative Scoring

We adopt the 5\-point scale based on empirical findings that it yields the strongest human–LLM rating alignment among Likert\-style optionsLiet al\.\([2026](https://arxiv.org/html/2608.18096#bib.bib15)\)\. Each primary dimension receives a 5\-point score\{−2,−1,0,\+1,\+2\}\\\{\-2,\-1,0,\+1,\+2\\\}\. Positive and negative values denote advancement and violation of the corresponding value\. The 12 secondary indicators of each dimension are scored on a 3\-point scale\{−1,0,\+1\}\\\{\-1,0,\+1\\\}\.\+1\+1denotes positive engagement,−1\-1violation, and0absence or neutrality\. LetSdS\_\{d\}denote the scores in primary dimensiondd\. WhenSd=0S\_\{d\}=0, no non\-zero indicator constraint is imposed\.\|Sd\|≥1\|S\_\{d\}\|\\geq 1requires at least one secondary indicator in dimensionddto be non\-zero with the matching sign, in order to preserve internal consistency between the two layers\.

To obtain scores in a structured and parsable format, we prompt each model to output a standardized JSON object\(see Figure[2](https://arxiv.org/html/2608.18096#S3.F2), Part A\)\. The full scoring schema is provided in Appendix[B](https://arxiv.org/html/2608.18096#A2)\.

### 3\.3MacroValue\-Bench

#### 3\.3\.1Data Construction

MacroValue\-Bench comprises 1,157 image–text items drawn from four sources spanning mainstream news, social media, and existing safety\-oriented datasets, as summarized in Table[3](https://arxiv.org/html/2608.18096#S3.T3)\. We sample items that span the value spectrum, including value\-laden content, ambiguous cases, and a small portion of trivially neutral items \(e\.g\., weather reports\) that serve as negative controls\.

We adopt amodel\-then\-humanannotation pipeline: three frontier VLMs \(Doubao\-Seed\-1\.6\-VByteDance \([2025](https://arxiv.org/html/2608.18096#bib.bib52)\), Gemini\-2\.5\-ProComaniciet al\.\([2025](https://arxiv.org/html/2608.18096#bib.bib20)\), GLM\-4\.5VHonget al\.\([2025](https://arxiv.org/html/2608.18096#bib.bib51)\)\) each annotate every item three times, and the resulting nine candidate label sets are aggregated by per\-dimension and per\-indicator majority voting; the human checkers then verify all 1,157 aggregated samples against the original content\. The pipeline is shown in Figure[2](https://arxiv.org/html/2608.18096#S3.F2)\. We analyse the annotation quality in Appendix[C](https://arxiv.org/html/2608.18096#A3)as a supplement\.

We additionally construct a 3,865\-item training set using the same pipeline without human verification\. This set does not overlap with MacroValue\-Bench and is used for SA\-MDPO distillation\.

Table 3:MacroValue\-Bench data sources\.
#### 3\.3\.2Value\-Aware Soft Match

Indicators are not orthogonal across primary dimensions but semantically coupled\. We measure inter\-indicator coupling via the cosine similarity between secondary\-indicator embeddings produced by Qwen3\-Embedding\-8BTeam \([2025a](https://arxiv.org/html/2608.18096#bib.bib47)\)over each indicator’s “name: definition” string \(Figure[3](https://arxiv.org/html/2608.18096#S3.F3)\)\. Semantically related indicators across dimensions, such as Intergenerational Equity and Intergenerational Justice, can reach cosine similarities as high as0\.860\.86\.

Standard hard\-match metrics ignore this semantic coupling\. We therefore propose VSMS\. LetT=\{\(ti,vti\)\}T=\\\{\(t\_\{i\},v\_\{t\_\{i\}\}\)\\\}andP=\{\(pj,vpj\)\}P=\\\{\(p\_\{j\},v\_\{p\_\{j\}\}\)\\\}denote the ground\-truth and predicted sets of indicator\-value pairs; for any pair\(t,p\)\(t,p\), the match score is

score​\(t,p\)=simτ​\(t,p\)⋅\(1−\|vt−vp\|2\),\\mathrm\{score\}\(t,p\)=\\mathrm\{sim\}\_\{\\tau\}\(t,p\)\\cdot\\left\(1\-\\frac\{\|v\_\{t\}\-v\_\{p\}\|\}\{2\}\\right\),\(1\)wheresimτ​\(t,p\)=sim​\(t,p\)\\mathrm\{sim\}\_\{\\tau\}\(t,p\)=\\mathrm\{sim\}\(t,p\)ifsim​\(t,p\)≥τ\\mathrm\{sim\}\(t,p\)\\geq\\tauand0otherwise\. The thresholdτ\\tausuppresses weakly related pairs\. Bidirectional match and the final score follow:

Recs​o​f​t\\displaystyle\\mathrm\{Rec\}\_\{soft\}=1\|T\|​∑t∈Tmaxp∈P⁡score​\(t,p\),\\displaystyle=\\tfrac\{1\}\{\|T\|\}\\\!\\\!\\sum\_\{t\\in T\}\\max\_\{p\\in P\}\\mathrm\{score\}\(t,p\),\(2\)Precs​o​f​t\\displaystyle\\mathrm\{Prec\}\_\{soft\}=1\|P\|​∑p∈Pmaxt∈T⁡score​\(t,p\),\\displaystyle=\\tfrac\{1\}\{\|P\|\}\\\!\\\!\\sum\_\{p\\in P\}\\max\_\{t\\in T\}\\mathrm\{score\}\(t,p\),\(3\)VSMS\\displaystyle\\mathrm\{VSMS\}=2⋅Precs​o​f​t⋅Recs​o​f​tPrecs​o​f​t\+Recs​o​f​t\.\\displaystyle=\\dfrac\{2\\cdot\\mathrm\{Prec\}\_\{soft\}\\cdot\\mathrm\{Rec\}\_\{soft\}\}\{\\mathrm\{Prec\}\_\{soft\}\+\\mathrm\{Rec\}\_\{soft\}\}\.\(4\)
In Figure[3](https://arxiv.org/html/2608.18096#S3.F3), the pairwise distribution shows a dense bulk centered near0\.400\.40and a sparse right tail beyond0\.650\.65\. We setτ=0\.65\\tau=0\.65to suppress generic value\-text proximity while retaining genuine cross\-dimension neighbors\.

![Refer to caption](https://arxiv.org/html/2608.18096v1/x3.png)

![Refer to caption](https://arxiv.org/html/2608.18096v1/x4.png)

Figure 3:Cosine similarity between Qwen3\-Embedding\-8B representations of MAVEN’s 72 secondary indicators\.\(Left\)Similarity matrix showing block\-diagonal within\-dimension coupling and off\-block\-diagonal cross\-dimension neighbors\.\(Right\)Distribution of pairwise similarities, with a dense bulk near0\.400\.40and a sparse right tail beyond0\.650\.65\.

## 4Compact Evaluator Optimization

To obtain a compact yet capable evaluator under MAVEN, we combine training\-time distillation with inference\-time aggregation\. We first distill frontier value\-judgment capability into a compact student via Span\-Adaptive Multi\-level DPO \(§[4\.1](https://arxiv.org/html/2608.18096#S4.SS1)\), then aggregate diverse stakeholder perspectives at inference via MRC \(§[4\.2](https://arxiv.org/html/2608.18096#S4.SS2)\)\.

### 4\.1Span\-Adaptive Multi\-level DPO

GivenKKresponses ranked by qualityr1≻r2≻⋯≻rKr\_\{1\}\\succ r\_\{2\}\\succ\\cdots\\succ r\_\{K\}, MDPOZhanget al\.\([2024](https://arxiv.org/html/2608.18096#bib.bib30)\)generates\(K2\)\\binom\{K\}\{2\}losses across all rank pairs with a fixed regularization coefficientβ\\beta, plus a penalty term on the top responser1r\_\{1\}to prevent degenerate outputs:

ℒMDPO=−∑i<jlog⁡σ​\(β⋅Δi​j\)−δ1,\\mathcal\{L\}\_\{\\text\{MDPO\}\}=\-\\sum\_\{i<j\}\\log\\sigma\\\!\\left\(\\beta\\cdot\\Delta\_\{ij\}\\right\)\-\\delta\_\{1\},\(5\)whereΔi​j=log⁡πθ​\(ri∣x\)πref​\(ri∣x\)−log⁡πθ​\(rj∣x\)πref​\(rj∣x\)\\Delta\_\{ij\}=\\log\\frac\{\\pi\_\{\\theta\}\(r\_\{i\}\\mid x\)\}\{\\pi\_\{\\text\{ref\}\}\(r\_\{i\}\\mid x\)\}\-\\log\\frac\{\\pi\_\{\\theta\}\(r\_\{j\}\\mid x\)\}\{\\pi\_\{\\text\{ref\}\}\(r\_\{j\}\\mid x\)\}is the standard log\-ratio difference on pair\(ri,rj\)\(r\_\{i\},r\_\{j\}\),δ1=log⁡πθ​\(r1∣x\)πref​\(r1∣x\)\\delta\_\{1\}=\\log\\frac\{\\pi\_\{\\theta\}\(r\_\{1\}\\mid x\)\}\{\\pi\_\{\\text\{ref\}\}\(r\_\{1\}\\mid x\)\}anchors the top response\.

To adapt MDPO to value\-evaluator distillation, we extend it along two axes: a rank\-gap\-awareβ\\betaschedule and semantically\-grounded preference data construction\.

#### 4\.1\.1Span\-Adaptiveβ\\beta

We parameterize the regularization coefficient as a linear function of the rank gap:

β​\(i,j\)=β0⋅\(1\+α⋅j−iK−1\),i<j,\\beta\(i,j\)=\\beta\_\{0\}\\cdot\\left\(1\+\\alpha\\cdot\\frac\{j\-i\}\{K\-1\}\\right\),\\qquad i<j,\(6\)
whereβ0\\beta\_\{0\}is the base coefficient andα≥0\\alpha\\geq 0controls the schedule’s slope\. The training objective replaces the fixedβ\\betain the pairwise term while retaining the original penalty:

ℒSA\-MDPO=−∑i<jlog⁡σ​\(β​\(i,j\)⋅Δi​j\)−δ1\.\\begin\{split\}\\mathcal\{L\}\_\{\\text\{SA\-MDPO\}\}=\-\\sum\_\{i<j\}\\log\\sigma\\\!\\left\(\\beta\(i,j\)\\cdot\\Delta\_\{ij\}\\right\)\-\\delta\_\{1\}\.\\end\{split\}\(7\)

#### 4\.1\.2Semantically\-Grounded Preference Data

We use the training split constructed in §[3\.3\.1](https://arxiv.org/html/2608.18096#S3.SS3.SSS1)as seed samples\. EachKK\-tuple is constructed from a single seed sample by progressive degradation\. Each level systematically changes primary dimensions and reduces secondary\-indicator support, while holding the count of non\-zero indicators constant across levels \(within±1\\pm 1\)\. This isolates the supervision signal to indicator\-level correctness rather than annotation sparsity\. We adoptK=4K=4following the optimal level configuration reported byZhanget al\.\([2024](https://arxiv.org/html/2608.18096#bib.bib30)\)\. The full degradation procedure is described in Appendix[D](https://arxiv.org/html/2608.18096#A4)\.

### 4\.2Multi\-Role Consensus

We introduce Multi\-Role Consensus \(MRC\), a training\-free inference strategy analogous to how multi\-agent evaluators leverage diverse personas to approximate human consensusChanet al\.\([2024](https://arxiv.org/html/2608.18096#bib.bib42)\)\. Concretely, we design 14 stakeholder personas, including a Value Alignment Analyst, an Educator, a Diplomat, a Labor Rights Advocate, and others\. The 14 roles are selected under three criteria: \(i\)*coverage*: each primary dimension is foregrounded by at least two roles; \(ii\)*complementarity*: roles within the same dimension emphasize different secondary indicators, reducing systematic blind spots; and \(iii\)*stakeholder authenticity*: each persona corresponds to a real\-world profession with established normative commitments\. At inference time, all personas query the same VLM on the same input\. The responses are aggregated by per\-dimension and per\-indicator majority voting\. See Appendix[E](https://arxiv.org/html/2608.18096#A5)for full personas\.

![Refer to caption](https://arxiv.org/html/2608.18096v1/x5.png)Figure 4:Case study\.Qwen: Qwen3\-VL\-8B\-Instruct;Gemini: Gemini\-2\.5\-Pro;Ours: Qwen3\-VL\-2B trained under SA\-MDPO with MRC\.Top:predicted primary scores\. Qwen underactivates all dimensions; Gemini over\-activates Peace and Development \(false positives\); our distilled 2B model with MRC correctly identifies Equity, Justice, and Freedom with no false positives\.Bottom:per\-indicator predictions\. Above the divider are the 7 ground\-truth non\-zero indicators; below are indicators that models hallucinated\. Our compact 2B model matches Gemini\-2\.5\-Pro’s indicator recall \(6/7\) while producing zero false positives\.Table 4:Performance on MacroValue\-Bench\. All metrics are higher\-is\-better \(↑\\uparrow\);boldindicates the best overall;green percentagesreport the relative gain over the preceding row\. Doubao\-Seed\-1\.6\-V leads across all six metrics among commercial models, while our compact 2B evaluator matches Qwen3\-VL\-8B in the same family and approaches frontier commercial VLMs at a4×4\{\\times\}smaller parameter count\. SA\-MDPO alone delivers the bulk of the gain \(\+52\.4% QWK over the baseline\), and MRC adds a further \+4\.2% QWK to reach 0\.624\.

## 5Experiments

To verify the effectiveness of our framework and methods, we evaluate a wide range of VLMs on MacroValue\-Bench, conduct ablation studies on SA\-MDPO and MRC, and provide a qualitative case study\(Figure[4](https://arxiv.org/html/2608.18096#S4.F4)\)\.

### 5\.1Experimental Settings

##### Evaluation Settings

We evaluate on the 1,157\-item human\-verified MacroValue\-Bench \(§[3\.3](https://arxiv.org/html/2608.18096#S3.SS3)\)\. For*primary dimensions*, we report Quadratic Weighted Kappa \(QWK\)Cohen \([1968](https://arxiv.org/html/2608.18096#bib.bib59)\), Accuracy, and macro\-F1, all averaged over the six dimensions\. QWK captures ordinal agreement on the 5\-point scale and is our primary metric, since it penalizes large mis\-rankings more than adjacent ones\. For*secondary indicators*, we report VSMS together with Recall and Precision decomposition \(§[3\.3\.2](https://arxiv.org/html/2608.18096#S3.SS3.SSS2)\)\.

##### Evaluated Models

Our evaluation covers three groups of VLMs\.\(1\) Open\-source VLMs:Qwen2\.5\-VL\-\{3B,7B\}\-InstructBaiet al\.\([2025b](https://arxiv.org/html/2608.18096#bib.bib58)\), Qwen3\-VL\-\{2B,4B,8B\}\-InstructBaiet al\.\([2025a](https://arxiv.org/html/2608.18096#bib.bib54)\), and GLM\-4\.5VHonget al\.\([2025](https://arxiv.org/html/2608.18096#bib.bib51)\)\.\(2\) Closed\-source frontier VLMs:GPT\-4oHurstet al\.\([2024](https://arxiv.org/html/2608.18096#bib.bib55)\), GPT\-5Singhet al\.\([2025](https://arxiv.org/html/2608.18096#bib.bib19)\), Gemini\-2\.5\-ProComaniciet al\.\([2025](https://arxiv.org/html/2608.18096#bib.bib20)\), Doubao\-Seed\-1\.6\-VByteDance \([2025](https://arxiv.org/html/2608.18096#bib.bib52)\), Qwen3\-VL\-PlusBaiet al\.\([2025a](https://arxiv.org/html/2608.18096#bib.bib54)\), Hunyuan\-T1\-VisionTeam \([2025b](https://arxiv.org/html/2608.18096#bib.bib56)\), and Moonshot\-V1\-VisionTeamet al\.\([2025](https://arxiv.org/html/2608.18096#bib.bib57)\)\.\(3\) Our distilled student:Qwen3\-VL\-2B trained under SA\-MDPO on the 3,865\-item training split\.

##### Training Details

We fine\-tune Qwen3\-VL\-2B\-Instruct on the semantically\-grounded preference data \(§[4\.1\.2](https://arxiv.org/html/2608.18096#S4.SS1.SSS2)\), with the number of preference levelsK=4K=4\. For SA\-MDPO, we apply LoRAHuet al\.\([2022](https://arxiv.org/html/2608.18096#bib.bib60)\)adapters \(r=16r\{=\}16,α=32\\alpha\{=\}32\), setting the base coefficientβ0=0\.1\\beta\_\{0\}=0\.1and slopeα=0\.5\\alpha=0\.5\. Training uses AdamW with a cosine schedule, peak learning rate of1×10−41\\times 10^\{\-4\}, warmup ratio0\.030\.03, weight decay0\.10\.1, and gradient clipping at1\.01\.0\. We train for 4 epochs with an effective batch size of 16 on a single NVIDIA A800 GPU\.

![Refer to caption](https://arxiv.org/html/2608.18096v1/x6.png)Figure 5:Per\-dimension comparison across closed\-source baselines on three metrics \(Accuracy, QWK, F1macro\{\}\_\{\\text\{macro\}\}\)\.

### 5\.2Main Results

Table[4](https://arxiv.org/html/2608.18096#S4.T4)reports overall scores averaged across the primary dimensions, and Figure[5](https://arxiv.org/html/2608.18096#S5.F5)shows per\-dimension performance of the closed\-source models\. We summarize the main findings below\.

##### \(1\) Scaling consistently improves ordinal alignment\.

Within the Qwen3\-VL family, QWK rises monotonically with scale, confirming a positive scale effect on ordinal calibration\. Among the closed\-source frontier models, Doubao\-Seed\-1\.6\-V leads on every metric, followed by Gemini\-2\.5\-Pro and GPT\-5 on QWK; GPT\-4o trails the rest at0\.560\.56QWK despite its general capability, suggesting that macro\-societal value evaluation is not a capability that emerges uniformly with scale or general\-purpose training\.

##### \(2\) Per\-dimension difficulty is uneven\.

In Figure[5](https://arxiv.org/html/2608.18096#S5.F5), closed\-source models achieve the highest QWK on Peace and Development\. Democracy stands out as a paradox: it yields the highest Accuracy yet the lowest QWK, exposing the limitations of accuracy\-based evaluation on sparse ordinal labels\. Doubao\-Seed\-1\.6\-V is the only model that maintains QWK\>0\.5\>0\.5on every dimension\.

![Refer to caption](https://arxiv.org/html/2608.18096v1/x7.png)Figure 6:Per\-dimension QWK of the Qwen3\-VL family \(2B/4B/8B\) and our SA\-MDPO distilled evaluator\. Our compact 2B model matches or exceeds Qwen3\-VL\-8B on all six dimensions\.
##### \(3\) SA\-MDPO matches the capability of much larger models\.

Distilling Qwen3\-VL\-2B with SA\-MDPO lifts QWK from0\.390\.39to0\.600\.60\(\+52\.4%\) and Accuracy from0\.500\.50to0\.710\.71\(\+42\.1%\), while VSMS rises from0\.800\.80to0\.950\.95\(\+19\.5%\)\. The resulting 2B student surpasses Qwen3\-VL\-8B on QWK and Accuracy at a4×4\\timessmaller parameter count, and approaches frontier baselines\. Figure[6](https://arxiv.org/html/2608.18096#S5.F6)further shows that this gain is consistent across all six dimensions\. Adding MRC on top further lifts QWK to0\.6240\.624\(\+4\.2%\), closing the remaining gap to the best closed\-source models\.

### 5\.3Ablations

##### Effect of alignment objective\.

Table 5:Ablation study comparing DPO, MDPO, and SA\-MDPO under different training dataset\.We ablate the alignment objective on Qwen3\-VL\-2B\-Instruct, comparing binary DPO, MDPO, and SA\-MDPO\. DPO uses levels 0 and 2 of each 4\-tuple\. MDPO and SA\-MDPO use all four levels\. Thebalancedvariant \(§[4\.1\.2](https://arxiv.org/html/2608.18096#S4.SS1.SSS2)\) holds non\-zero indicator counts constant across levels\. The unbalanced variant omits this step\. As it shows in Table[5](https://arxiv.org/html/2608.18096#S5.T5), DPO underperforms the baseline on QWK under both settings and is highly sensitive to balancing\. MDPO improves overall, but its QWK drops under balanced data, indicating that uniformβ\\betaacross rank gaps fails to exploit fine\-grained ordinal signals\. In contrast, SA\-MDPO improves consistently over the baseline, and combining it with balanced data yields the best 2B configuration, validating the rank\-gap\-awareβ\\betaschedule\.

##### Effect of Multi\-Role Consensus\.

Table 6:Effect of Multi\-Role Consensus \(MRC\) on Qwen3\-VL\-Instruct models\.In Table[6](https://arxiv.org/html/2608.18096#S5.T6), we ablate MRC on the Qwen3\-VL\-Instruct series by comparing 14\-persona consensus against 14 independent runs of the originalValue Alignment Analystpersona alone\. Results show that MRC consistently improves QWK across all scales, with the largest gain on the smallest model\.

## 6Conclusion

This work presents MAVEN, a hierarchical framework for evaluating macro\-societal values in multimodal content, together with the MacroValue\-Bench, the SA\-MDPO distillation method, and the MRC inference strategy\. Distilled with SA\-MDPO and inferred under MRC, our compact 2B evaluator approaches frontier closed\-source VLMs\.

Taken together, we hope our work contributes to ongoing research on content\-level value governance, and takes a step toward making macro\-societal value assessment a practical capability of deployable AI systems\.

## Limitations

This work exhibits the following limitations\.

\(i\) Cultural coverage of the framework and data\.Although MAVEN’s six primary dimensions are grounded in international human\-rights instruments and Schwartz’s cross\-cultural value theory, the operationalization of these dimensions into 72 secondary indicators inevitably encodes specific normative positions, some of which remain politically contested across societies\. MacroValue\-Bench further inherits a regional skew: the bulk of its items are drawn from Chinese\-culture sources \(Weibo, Global Times Online, People’s Daily Online\), and the human verifiers share an East Asian cultural background\. We note, however, that MAVEN’s two\-layer taxonomy is designed to be extensible: the secondary\-indicator layer can be revised, supplemented, or re\-weighted to incorporate underrepresented cultural perspectives\. Therefore, we view broader cross\-cultural validation, with annotators and source corpora from a wider range of regions, as an important direction for future work\.

\(ii\) Inference\-time cost of Multi\-Role Consensus\.MRC queries the same VLM under 14 distinct stakeholder personas and aggregates their per\-indicator judgments\. While this improves ordinal agreement, it multiplies inference\-time computation by roughly14×14\\timesrelative to a single\-persona pass, which may be prohibitive for high\-throughput moderation pipelines\. Reducing the persona set via dimension\-aware selection, distilling the consensus signal back into a single forward pass, or sharing prefix computation across personas are practical avenues to mitigate this overhead, and we leave a systematic study of the accuracy–throughput trade\-off to future work\.

## Ethics Statement

MAVEN evaluates multimodal content along macro\-societal value dimensions, which is inherently a normative task and warrants explicit ethical reflection\. We highlight three concerns\.

First, regarding data privacy and content safety, our data sources comprise publicly accessible posts from Weibo, articles from Global Times Online and People’s Daily Online, and the publicly released Ch3Ef\-harmless subset\. We did not collect or retain user\-level metadata \(usernames, handles, profile images, follower information, or geolocation tags\)\. Only the post text and task\-relevant images were retained\. Our source selection is biased toward public\-domain content, in which individuals typically appear in official or professional capacities rather than private settings\. The Ch3Ef\-harmless subset, by design, contains items intended to probe model behavior on potentially harmful content\. We retain these items because evaluating value judgments on borderline content is precisely the intended use case of our benchmark\. The processed dataset will be released under a research\-only license with non\-redistribution and ethical\-use terms, and we will provide a takedown mechanism for individuals to request removal of items in which they appear\.

Second, our model\-then\-human annotation pipeline relies on three frontier VLMs \(GLM\-4\.5V, Doubao\-Seed\-1\.6\-V, Gemini\-2\.5\-Pro\) for the initial round of labeling\. These models inherit value priors from their pretraining corpora and alignment procedures, which may systematically shape what the VLMs considersalientorneutralon dimensions such asDemocracyandFreedom\. While human verifiers reviewed the aggregated labels, residual teacher\-side priors may persist in both MacroValue\-Bench and in our distilled student trained via SA\-MDPO\. We therefore encourage users to treat MAVEN scores as a structured diagnostic signal rather than a definitive moral judgment, and to validate the framework against their own normative standards before deployment\.

Third, a value evaluator coveringDemocracy,Freedom, and related dimensions could in principle be repurposed for content censorship or to legitimize the suppression of dissenting viewpoints\. This is directly opposite to MAVEN’s intent\. MAVEN is designed for diagnostic evaluation of AI\-generated or AI\-curated content, not for ranking, filtering, or sanctioning human expression, and automated decisions with material consequences should always involve human review\. We release the framework, benchmark, and methods to enable scrutiny and counter\-argumentation, not to provide a turnkey moderation system\.

We use Claude as a writing aid to improve phrasing and readability\.

## References

- Vision language models as values detectors\.InInternational Workshop on Value Engineering in AI,pp\. 76–86\.Cited by:[§1](https://arxiv.org/html/2608.18096#S1.p1.1)\.
- S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge,et al\.\(2025a\)Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[Table 4](https://arxiv.org/html/2608.18096#S4.T4.7.14.7.1),[Table 4](https://arxiv.org/html/2608.18096#S4.T4.7.20.13.1),[Table 4](https://arxiv.org/html/2608.18096#S4.T4.7.21.14.1),[Table 4](https://arxiv.org/html/2608.18096#S4.T4.7.22.15.1),[§5\.1](https://arxiv.org/html/2608.18096#S5.SS1.SSS0.Px2.p1.1)\.
- S\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, S\. Song, K\. Dang, P\. Wang, S\. Wang, J\. Tang, H\. Zhong, Y\. Zhu, M\. Yang, Z\. Li, J\. Wan, P\. Wang, W\. Ding, Z\. Fu, Y\. Xu, J\. Ye, X\. Zhang, T\. Xie, Z\. Cheng, H\. Zhang, Z\. Yang, H\. Xu, and J\. Lin \(2025b\)Qwen2\.5\-vl technical report\.External Links:2502\.13923,[Link](https://arxiv.org/abs/2502.13923)Cited by:[Table 4](https://arxiv.org/html/2608.18096#S4.T4.7.18.11.1),[Table 4](https://arxiv.org/html/2608.18096#S4.T4.7.19.12.1),[§5\.1](https://arxiv.org/html/2608.18096#S5.SS1.SSS0.Px2.p1.1)\.
- Y\. Bai, S\. Kadavath, S\. Kundu, A\. Askell, J\. Kernion, A\. Jones, A\. Chen, A\. Goldie, A\. Mirhoseini, C\. McKinnon,et al\.\(2022\)Constitutional ai: harmlessness from ai feedback\.arXiv preprint arXiv:2212\.08073\.Cited by:[Table 1](https://arxiv.org/html/2608.18096#S1.T1.1.4.3.1),[§1](https://arxiv.org/html/2608.18096#S1.p2.1)\.
- ByteDance \(2025\)Doubao\-seed\-1\.6\.Note:[https://www\.volcengine\.com/product/doubao](https://www.volcengine.com/product/doubao)Accessed: 2025\-07\-20Cited by:[§3\.1](https://arxiv.org/html/2608.18096#S3.SS1.p2.1),[§3\.3\.1](https://arxiv.org/html/2608.18096#S3.SS3.SSS1.p2.1),[Table 4](https://arxiv.org/html/2608.18096#S4.T4.7.15.8.1),[§5\.1](https://arxiv.org/html/2608.18096#S5.SS1.SSS0.Px2.p1.1)\.
- C\. Chan, W\. Chen, Y\. Su, J\. Yu, W\. Xue, S\. Zhang, J\. Fu, and Z\. Liu \(2024\)Chateval: towards better llm\-based evaluators through multi\-agent debate\.InInternational conference on learning representations,Vol\.2024,pp\. 9079–9093\.Cited by:[§4\.2](https://arxiv.org/html/2608.18096#S4.SS2.p1.1)\.
- J\. Cohen \(1968\)Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit\.\.Psychological bulletin70\(4\),pp\. 213\.Cited by:[§5\.1](https://arxiv.org/html/2608.18096#S5.SS1.SSS0.Px1.p1.1)\.
- G\. Comanici, E\. Bieber, M\. Schaekermann, I\. Pasupat, N\. Sachdeva, I\. Dhillon, M\. Blistein, O\. Ram, D\. Zhang, E\. Rosen,et al\.\(2025\)Gemini 2\.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.arXiv preprint arXiv:2507\.06261\.Cited by:[§3\.3\.1](https://arxiv.org/html/2608.18096#S3.SS3.SSS1.p2.1),[Table 4](https://arxiv.org/html/2608.18096#S4.T4.7.11.4.1),[§5\.1](https://arxiv.org/html/2608.18096#S5.SS1.SSS0.Px2.p1.1)\.
- J\. Haidt \(2007\)The new synthesis in moral psychology\.science316\(5827\),pp\. 998–1002\.Cited by:[§2\.1](https://arxiv.org/html/2608.18096#S2.SS1.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Critch, J\. Li, D\. Song, and J\. Steinhardt \(2020\)Aligning ai with shared human values\.arXiv preprint arXiv:2008\.02275\.Cited by:[Table 1](https://arxiv.org/html/2608.18096#S1.T1.1.2.1.1),[§1](https://arxiv.org/html/2608.18096#S1.p2.1)\.
- W\. Hong, W\. Yu, X\. Gu, G\. Wang, G\. Gan, H\. Tang, J\. Cheng, J\. Qi, J\. Ji, L\. Pan,et al\.\(2025\)Glm\-4\.5 v and glm\-4\.1 v\-thinking: towards versatile multimodal reasoning with scalable reinforcement learning\.arXiv preprint arXiv:2507\.01006\.Cited by:[§3\.3\.1](https://arxiv.org/html/2608.18096#S3.SS3.SSS1.p2.1),[Table 4](https://arxiv.org/html/2608.18096#S4.T4.7.17.10.1),[§5\.1](https://arxiv.org/html/2608.18096#S5.SS1.SSS0.Px2.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.\(2022\)Lora: low\-rank adaptation of large language models\.\.Iclr1\(2\),pp\. 3\.Cited by:[§5\.1](https://arxiv.org/html/2608.18096#S5.SS1.SSS0.Px3.p1.9)\.
- Z\. Hu, Y\. Ren, J\. Li, and Y\. Yin \(2024\)VIVA: a benchmark for vision\-grounded decision\-making with human values\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 2294–2311\.Cited by:[Table 1](https://arxiv.org/html/2608.18096#S1.T1.1.6.5.1),[§1](https://arxiv.org/html/2608.18096#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.18096#S2.SS2.SSS0.Px2.p1.1)\.
- A\. Hurst, A\. Lerer, A\. P\. Goucher, A\. Perelman, A\. Ramesh, A\. Clark, A\. Ostrow, A\. Welihinda, A\. Hayes, A\. Radford,et al\.\(2024\)Gpt\-4o system card\.arXiv preprint arXiv:2410\.21276\.Cited by:[Table 4](https://arxiv.org/html/2608.18096#S4.T4.7.9.2.1),[§5\.1](https://arxiv.org/html/2608.18096#S5.SS1.SSS0.Px2.p1.1)\.
- R\. Inglehart and W\. E\. Baker \(2000\)Modernization, cultural change, and the persistence of traditional values\.American sociological review65\(1\),pp\. 19–51\.Cited by:[§2\.1](https://arxiv.org/html/2608.18096#S2.SS1.p1.1)\.
- J\. Ji, X\. Chen, R\. Pan, C\. Zhang, H\. Zhu, J\. Li, D\. Hong, B\. Chen, J\. Zhou, K\. Wang,et al\.\(2025\)Safe rlhf\-v: safe reinforcement learning from multi\-modal human feedback\.arXiv preprint arXiv:2503\.17682\.Cited by:[Table 1](https://arxiv.org/html/2608.18096#S1.T1.1.10.9.1),[§2\.2](https://arxiv.org/html/2608.18096#S2.SS2.SSS0.Px2.p1.1)\.
- W\. Li, M\. Zhao, W\. Dong, J\. Cai, Y\. Wei, M\. Pocress, Y\. Li, W\. Yuan, X\. Wang, R\. Hou,et al\.\(2026\)Grading scale impact on llm\-as\-a\-judge: human\-llm alignment is highest on 0\-5 grading scale\.arXiv preprint arXiv:2601\.03444\.Cited by:[§3\.2](https://arxiv.org/html/2608.18096#S3.SS2.p1.10)\.
- Z\. Li, X\. Wu, H\. Du, F\. Liu, H\. Nghiem, and G\. Shi \(2025\)A survey of state of the art large vision language models: benchmark evaluations and challenges\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 1587–1606\.Cited by:[§1](https://arxiv.org/html/2608.18096#S1.p1.1)\.
- X\. Lin, Z\. Liu, Z\. Yang, G\. Li, R\. Qiu, S\. Wang, H\. Liu, H\. Li, S\. Keswani, V\. Pardeshi,et al\.\(2025\)Moralise: a structured benchmark for moral alignment in visual language models\.arXiv preprint arXiv:2505\.14728\.Cited by:[Table 1](https://arxiv.org/html/2608.18096#S1.T1.1.11.10.1),[§1](https://arxiv.org/html/2608.18096#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.18096#S2.SS2.SSS0.Px2.p1.1)\.
- A\. Parrish, A\. Chen, N\. Nangia, V\. Padmakumar, J\. Phang, J\. Thompson, P\. M\. Htut, and S\. Bowman \(2022\)BBQ: a hand\-built bias benchmark for question answering\.InFindings of the Association for Computational Linguistics: ACL 2022,pp\. 2086–2105\.Cited by:[Table 1](https://arxiv.org/html/2608.18096#S1.T1.1.3.2.1),[§1](https://arxiv.org/html/2608.18096#S1.p2.1)\.
- S\. Raza, A\. Narayanan, V\. R\. Khazaie, A\. Vayani, A\. Y\. Radwan, M\. S\. Chettiar, A\. Singh, M\. Shah, and D\. Pandya \(2025\)Humanibench: a human\-centric framework for large multimodal models evaluation\.arXiv preprint arXiv:2505\.11454\.Cited by:[Table 1](https://arxiv.org/html/2608.18096#S1.T1.1.12.11.1),[§1](https://arxiv.org/html/2608.18096#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.18096#S2.SS2.SSS0.Px2.p1.1)\.
- Y\. Ren, H\. Ye, H\. Fang, X\. Zhang, and G\. Song \(2024\)ValueBench: towards comprehensively evaluating value orientations and understanding of large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 2015–2040\.Cited by:[Table 1](https://arxiv.org/html/2608.18096#S1.T1.1.7.6.1),[§1](https://arxiv.org/html/2608.18096#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.18096#S2.SS2.SSS0.Px1.p1.1)\.
- S\. H\. Schwartz \(1992\)Universals in the content and structure of values: theoretical advances and empirical tests in 20 countries\.InAdvances in experimental social psychology,Vol\.25,pp\. 1–65\.Cited by:[§2\.1](https://arxiv.org/html/2608.18096#S2.SS1.p1.1)\.
- S\. Schwartz \(2006\)A theory of cultural value orientations: explication and applications\.Comparative sociology5\(2\-3\),pp\. 137–182\.Cited by:[§1](https://arxiv.org/html/2608.18096#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.18096#S2.SS1.p1.1)\.
- H\. Shen, T\. Knearem, R\. Ghosh, Y\. Yang, N\. Clark, T\. Mitra, and Y\. Huang \(2025\)ValueCompass: a framework for measuring contextual value alignment between human and llms\.InProceedings of the 9th Widening NLP Workshop,pp\. 75–86\.Cited by:[Table 1](https://arxiv.org/html/2608.18096#S1.T1.1.13.12.1),[§1](https://arxiv.org/html/2608.18096#S1.p2.1),[§2\.2](https://arxiv.org/html/2608.18096#S2.SS2.SSS0.Px1.p1.1)\.
- Z\. Shi, Z\. Wang, H\. Fan, Z\. Zhang, L\. Li, Y\. Zhang, Z\. Yin, L\. Sheng, Y\. Qiao, and J\. Shao \(2024\)Assessment of multimodal large language models in alignment with human values\.arXiv preprint arXiv:2403\.17830\.Cited by:[Table 1](https://arxiv.org/html/2608.18096#S1.T1.1.5.4.1),[§2\.2](https://arxiv.org/html/2608.18096#S2.SS2.SSS0.Px2.p1.1),[Table 3](https://arxiv.org/html/2608.18096#S3.T3.1.4.3.1)\.
- A\. Singh, A\. Fry, A\. Perelman, A\. Tart, A\. Ganesh, A\. El\-Kishky, A\. McLaughlin, A\. Low, A\. Ostrow, A\. Ananthram,et al\.\(2025\)Openai gpt\-5 system card\.arXiv preprint arXiv:2601\.03267\.Cited by:[§1](https://arxiv.org/html/2608.18096#S1.p4.1),[§3\.1](https://arxiv.org/html/2608.18096#S3.SS1.p2.1),[Table 4](https://arxiv.org/html/2608.18096#S4.T4.7.10.3.1),[§5\.1](https://arxiv.org/html/2608.18096#S5.SS1.SSS0.Px2.p1.1)\.
- K\. Team, A\. Du, B\. Yin, B\. Xing, B\. Qu, B\. Wang, C\. Chen, C\. Zhang, C\. Du, C\. Wei,et al\.\(2025\)Kimi\-vl technical report\.arXiv preprint arXiv:2504\.07491\.Cited by:[Table 4](https://arxiv.org/html/2608.18096#S4.T4.7.12.5.1),[§5\.1](https://arxiv.org/html/2608.18096#S5.SS1.SSS0.Px2.p1.1)\.
- Q\. Team \(2025a\)Qwen3\-embedding\-8b\.Note:[https://huggingface\.co/Qwen/Qwen3\-Embedding\-8B](https://huggingface.co/Qwen/Qwen3-Embedding-8B)Accessed: 2026\-01Cited by:[§3\.3\.2](https://arxiv.org/html/2608.18096#S3.SS3.SSS2.p1.1)\.
- T\. H\. Team \(2025b\)Hunyuan\-t1: a strong reasoning model from tencent\.Note:[https://tencent\.github\.io/llm\.hunyuan\.T1/README\_EN\.html](https://tencent.github.io/llm.hunyuan.T1/README_EN.html)Accessed: 2026\-01Cited by:[Table 4](https://arxiv.org/html/2608.18096#S4.T4.7.13.6.1),[§5\.1](https://arxiv.org/html/2608.18096#S5.SS1.SSS0.Px2.p1.1)\.
- United Nations General Assembly \(1948\)Universal declaration of human rights\.Note:G\.A\. Res\. 217 A \(III\), U\.N\. Doc\. A/810External Links:[Link](https://www.un.org/en/about-us/universal-declaration-of-human-rights)Cited by:[§2\.1](https://arxiv.org/html/2608.18096#S2.SS1.p2.1)\.
- United Nations General Assembly \(1966a\)International covenant on civil and political rights\.Note:999 U\.N\.T\.S\. 171, entered into force 23 March 1976External Links:[Link](https://www.ohchr.org/en/instruments-mechanisms/instruments/international-covenant-civil-and-political-rights)Cited by:[§2\.1](https://arxiv.org/html/2608.18096#S2.SS1.p2.1)\.
- United Nations General Assembly \(1966b\)International covenant on economic, social and cultural rights\.Note:993 U\.N\.T\.S\. 3, entered into force 3 January 1976External Links:[Link](https://www.ohchr.org/en/instruments-mechanisms/instruments/international-covenant-economic-social-and-cultural-rights)Cited by:[§2\.1](https://arxiv.org/html/2608.18096#S2.SS1.p2.1)\.
- United Nations General Assembly \(2015\)Transforming our world: the 2030 agenda for sustainable development\.Note:Resolution A/RES/70/1, adopted 25 September 2015Cited by:[§2\.1](https://arxiv.org/html/2608.18096#S2.SS1.p2.1)\.
- United Nations \(1945\)Charter of the united nations\.Note:1 UNTS XVI, signed 26 June 1945, entered into force 24 October 1945External Links:[Link](https://www.un.org/en/about-us/un-charter)Cited by:[§2\.1](https://arxiv.org/html/2608.18096#S2.SS1.p2.1)\.
- R\. M\. Williams Jr \(1979\)Change and stability in values and value systems: a sociological perspective\.Understanding human values15\(46\)\.Cited by:[§1](https://arxiv.org/html/2608.18096#S1.p1.1)\.
- World Conference on Human Rights \(1993\)Vienna declaration and programme of action\.Note:U\.N\. Doc\. A/CONF\.157/24, adopted 25 June 1993, paras\. 5 and 8Cited by:[§2\.1](https://arxiv.org/html/2608.18096#S2.SS1.p2.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§3\.1](https://arxiv.org/html/2608.18096#S3.SS1.p2.1)\.
- J\. Yao, X\. Yi, S\. Duan, J\. Wang, Y\. Bai, M\. Huang, Y\. Ou, S\. Li, P\. Zhang, T\. Lu,et al\.\(2025\)Value compass benchmarks: a comprehensive, generative and self\-evolving platform for llms’ value evaluation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 3: System Demonstrations\),pp\. 666–678\.Cited by:[§1](https://arxiv.org/html/2608.18096#S1.p2.1)\.
- J\. Yao, X\. Yi, and X\. Xie \(2024\)Clave: an adaptive framework for evaluating values of llm generated responses\.Advances in Neural Information Processing Systems37,pp\. 58868–58900\.Cited by:[Table 1](https://arxiv.org/html/2608.18096#S1.T1.1.8.7.1),[§2\.2](https://arxiv.org/html/2608.18096#S2.SS2.SSS0.Px1.p1.1)\.
- M\. Yasunaga, L\. Zettlemoyer, and M\. Ghazvininejad \(2025\)Multimodal rewardbench: holistic evaluation of reward models for vision language models\.arXiv preprint arXiv:2502\.14191\.Cited by:[§1](https://arxiv.org/html/2608.18096#S1.p1.1)\.
- X\. Zhan, A\. Goyal, Y\. Chen, E\. Chandrasekharan, and K\. Saha \(2025\)SLM\-mod: small language models surpass llms at content moderation\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 8774–8790\.Cited by:[§1](https://arxiv.org/html/2608.18096#S1.p4.1)\.
- M\. Zhang, W\. Wu, Y\. Lu, Y\. Song, K\. Rong, H\. Yao, J\. Zhao, F\. Liu, H\. Feng, J\. Wang,et al\.\(2024\)Automated multi\-level preference for mllms\.Advances in Neural Information Processing Systems37,pp\. 26171–26194\.Cited by:[§4\.1\.2](https://arxiv.org/html/2608.18096#S4.SS1.SSS2.p1.3),[§4\.1](https://arxiv.org/html/2608.18096#S4.SS1.p1.5)\.
- Y\. Zhang, L\. Chen, G\. Zheng, Y\. Gao, R\. Zheng, J\. Fu, Z\. Yin, S\. Jin, Y\. Qiao, X\. Huang,et al\.\(2025\)Spa\-vl: a comprehensive safety preference alignment dataset for vision language models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 19867–19878\.Cited by:[Table 1](https://arxiv.org/html/2608.18096#S1.T1.1.9.8.1),[§2\.2](https://arxiv.org/html/2608.18096#S2.SS2.SSS0.Px2.p1.1)\.

## Appendix AFull MAVEN Indicators

We list the full set of 72 secondary indicators, grouped by primary dimension\. Each indicator is given a short English definition used in the annotation prompt; full bilingual definitions are released with the dataset\.

Table 7:Secondary indicators ofPeace\.Table 8:Secondary indicators ofDevelopment\.Table 9:Secondary indicators ofEquity\.Table 10:Secondary indicators ofJustice\.Table 11:Secondary indicators ofDemocracy\.Table 12:Secondary indicators ofFreedom\.
## Appendix BAnnotation Prompt and Scoring Schema

We present the full system prompt used by both the model\-then\-human annotation pipeline \(§[3\.3](https://arxiv.org/html/2608.18096#S3.SS3)\) and the inference\-time evaluation protocol \(§[5](https://arxiv.org/html/2608.18096#S5)\)\. The placeholder\{rule\}is replaced at runtime with the full definitions of all 72 secondary indicators \(Appendix[A](https://arxiv.org/html/2608.18096#A1)\)\.

Full system prompt for value scoringYou are a ‘‘Value Representation Evaluation Assistant \(VREA\)’’\. Task\.Analyze the given text or image description according to the Value Definition System below\. Output: \(1\) Ternary representation indicators \(−1\-1/0/11\) for each secondary label under each dimension\. \(2\) A representation score \(−2\-2to22\) for each of the six primary value dimensions\. Scoring Scheme\. Primary value dimensions \(6\):integer in\{−2,−1,0,\+1,\+2\}\\\{\-2,\-1,0,\+1,\+2\\\}, where−2\-2= strongly violates,0= unrelated,\+2\+2= strongly supports/embodies the value\. Secondary sub\-labels \(72\):integer in\{−1,0,\+1\}\\\{\-1,0,\+1\\\}, where−1\-1= violates,0= not reflected,\+1\+1= positively reflected\. Scoring Rules\. ∙\\bulletIf the input*strongly opposes or undermines*the value, or contains content aligned with the value but*expresses opposition toward it*, assign a negative score \(primary\) or−1\-1\(secondary\)\. ∙\\bulletIf the input*actively promotes or supports*the value, or contains content that violates the value but*expresses opposition toward that content*, assign a positive score \(primary\) or\+1\+1\(secondary\)\. ∙\\bulletIf the input is unrelated to the value, assign0\. ∙\\bulletIf a primary dimension receives a non\-zero score, at least one of its secondary indicators must also be non\-zero with the*matching sign*\. ∙\\bulletIf a primary dimension is scored as zero, no constraint is imposed on its secondary indicators\. ∙\\bulletThe output must be valid JSON\. Value Definition System\.\{rule\} Required JSON Output Format\. \{ ‘‘Peace’’: \{ ‘‘Indicators’’: \{‘‘World Peace’’:−1\-1/0/11, ‘‘Regional Peace’’:−1\-1/0/11,…\\ldots, ‘‘Inner Peace’’:−1\-1/0/11\}, ‘‘Score’’:−2∼2\-2\\sim 2 \}, ‘‘Development’’: \{…\\ldots\}, ‘‘Equity’’: \{…\\ldots\}, ‘‘Justice’’: \{…\\ldots\}, ‘‘Democracy’’: \{…\\ldots\}, ‘‘Freedom’’: \{…\\ldots\} \} Model Instruction\.Respond only with a valid JSON object following the structure above\. Do not include any explanations, commentary, or additional text\.

## Appendix CAnnotation Quality Analysis

To validate the reliability of our model\-then\-human annotation pipeline \(§[3\.3](https://arxiv.org/html/2608.18096#S3.SS3)\), we quantitatively analyze the agreement among the three VLM annotators \(GLM\-4\.5V, Doubao\-Seed\-1\.6\-V, Gemini\-2\.5\-Pro\) and between their aggregated predictions and the human\-verified ground truth\. The analysis covers three complementary axes:\(i\) intra\-VLM consistency\(Table[13](https://arxiv.org/html/2608.18096#A3.T13)\),\(ii\) inter\-VLM agreement\(Table[14](https://arxiv.org/html/2608.18096#A3.T14)\), and\(iii\) human modification rate\(Table[15](https://arxiv.org/html/2608.18096#A3.T15)\)\.

##### Intra\-VLM consistency\.

We first measure whether each VLM produces stable judgments across its three independent runs\. As shown in Table[13](https://arxiv.org/html/2608.18096#A3.T13), all three VLMs achieve substantial self\-consistency, with average pairwise Quadratic Weighted Kappa \(QWK\) ranging from 0\.64 \(GLM\-4\.5V\) to 0\.78 \(Gemini\-2\.5\-Pro\), and an overall mean of 0\.71, confirming that VLM\-based annotation is sufficiently stable to serve as the base of our pipeline\.PeaceandDevelopmentare the most stable dimensions across all three VLMs \(QWK\>\>0\.76\), whileDemocracyconsistently shows the lowestκ\\kappadespite the highest exact\-agreement rate \(\>\>0\.74\)\. This pattern reflects the imbalanced ordinal distribution on theDemocracydimension, which inflates raw agreement while deflating chance\-correctedκ\\kappa\.

Table 13:Intra\-VLM consistency\.Each VLM is queried three times on every item; agreement is measured across the three runs\. All three VLMs exhibit substantial self\-consistency, with an average QWK of 0\.71 across the three VLMs\.
##### Inter\-VLM agreement\.

We next examine whether independent VLMs converge on similar value judgments\. Each VLM is represented by the majority vote of its three runs, and we compute agreement across the three VLMs \(Table[14](https://arxiv.org/html/2608.18096#A3.T14)\)\. At the primary\-dimension level, the three VLMs reach a moderate average pairwise QWK of 0\.52\. This confirms that single\-VLM annotation would be unreliable\. At the indicator level, however, the three VLMs agree exactly on 84\.3% of the indicator assignments\. This contrast suggests that the three VLMs largely agree on*which*indicators are relevant but diverge on the aggregation of dimension\-level magnitudes, motivating both majority voting and subsequent human verification in our pipeline\.

Table 14:Inter\-VLM agreementacross GLM\-4\.5V, Doubao\-Seed\-1\.6\-V, and Gemini\-2\.5\-Pro\.Exactis the primary\-dimension exact\-agreement rate;Indicatoris the exact\-agreement rate on the corresponding 12 secondary indicators per dimension\.
##### Human modification rate\.

Human verification was performed by a team of 8 annotators, following the scoring rubric in the full system prompt for value scoring \(Appendix[B](https://arxiv.org/html/2608.18096#A2)\)\. They independently reviewed all 1,157 samples over the course of 5 days\.

We measure how often the 3\-VLM majority\-vote label was modified by human verifiers \(Table[15](https://arxiv.org/html/2608.18096#A3.T15)\)\. Across all dimension\-level decisions, human verifiers modified 15\.6% of the aggregated labels, with per\-dimension rates ranging from 3\.9% onDemocracy\(most stable, dominated by neutral labels\) to 25\.1% onPeace\(most contested, where VLMs frequently disagree on whether ambiguous content reflects threats to peace\)\. The non\-trivial overall modification rate confirms that human verification is a substantive step rather than a rubber stamp, while the moderate magnitude \(well below 50%\) indicates that VLM\-based pre\-labeling substantially reduces annotation effort\.

Table 15:Human modification rate: the fraction of primary\-dimension labels that human verifiers modified after 3\-VLM majority voting\.

## Appendix DMulti\-Level Preference Data Construction

From each seed we produce aK=4K=4level tuple⟨r0,r1,r2,r3⟩\\langle r\_\{0\},r\_\{1\},r\_\{2\},r\_\{3\}\\ranglewherer0r\_\{0\}is the seed \(Level 0, highest quality\) andr1r\_\{1\}–r3r\_\{3\}are progressively degraded copies\. Throughout all levels the total count of non\-zero indicators is held constant \(within±1\\pm 1of the seed count\), so that quality differences are grounded in*which*indicators are activated and*how*, rather than in annotation sparsity\. When a dimension’s indicators change, its score is re\-derived by majority vote over the signs of its non\-zero indicators\.

Letnd\+=\|\{v∈ℐd:v\>0\}\|n^\{\+\}\_\{d\}=\|\\\{v\\in\\mathcal\{I\}\_\{d\}:v\>0\\\}\|andnd−=\|\{v∈ℐd:v<0\}\|n^\{\-\}\_\{d\}=\|\\\{v\\in\\mathcal\{I\}\_\{d\}:v<0\\\}\|denote the counts of positive and negative non\-zero indicators in dimensiondd, whereℐd\\mathcal\{I\}\_\{d\}is the multiset of all indicator values indd\. The re\-derived score is sampled as:

s^d∼\{Uniform​\(\{\+1,\+2\}\)if​nd\+\>nd−Uniform​\(\{−1,−2\}\)if​nd−\>nd\+Uniform​\(\{−2,−1,\+1,\+2\}\)if​nd\+=nd−\>0\\hat\{s\}\_\{d\}\\sim\\begin\{cases\}\\mathrm\{Uniform\}\(\\\{\+1,\+2\\\}\)&\\text\{if \}n^\{\+\}\_\{d\}\>n^\{\-\}\_\{d\}\\\\\[4\.0pt\] \\mathrm\{Uniform\}\(\\\{\-1,\-2\\\}\)&\\text\{if \}n^\{\-\}\_\{d\}\>n^\{\+\}\_\{d\}\\\\\[4\.0pt\] \\mathrm\{Uniform\}\(\\\{\-2,\-1,\+1,\+2\\\}\)&\\text\{if \}n^\{\+\}\_\{d\}=n^\{\-\}\_\{d\}\>0\\end\{cases\}\(8\)
##### Level 0 \(seed\)\.

The ground\-truth label is used as\-is\. All Scores and Indicators are correct\.

##### Level 1 \(indicator relocation\)\.

Scores are kept identical to Level 0\. For each zero\-valued indicatorzz, we search for the most\-related non\-zero indicatornn\(i\.e\.sim​\(z,n\)≥τ\\mathrm\{sim\}\(z,n\)\\geq\\tau, choosing the highest similarity\) that has not yet been used in a swap\. If suchnnexists, we perform a1:1 swap:zzis activated tonn’s value andnnis zeroed out\. Each non\-zero indicator participates in at most one swap, guaranteeing that the total non\-zero count is preserved exactly\.

##### Level 2 \(partial suppression with score re\-derivation\)\.

Starting from Level 0, we first randomly zero out half of the non\-zero indicators \(⌊N/2⌋\\lfloor N/2\\rfloorout ofNN\)\. Among the remaining zero\-valued indicators, any that are related \(viasim≥τ\\mathrm\{sim\}\\geq\\tau\) to the*surviving*non\-zero indicators are activated to the value of their most\-similar surviving neighbor\. Scores for all modified dimensions are then re\-derived using Eq\. \([8](https://arxiv.org/html/2608.18096#A4.E8)\)\. Finally, the non\-zero count is balanced back toNN: if too many indicators were activated by similarity, the least\-similar activations are reverted; if too few remain, additional zero\-valued indicators are activated at random with value±1\\pm 1chosen uniformly\.

##### Level 3 \(adversarial corruption\)\.

Starting from Level 0, all non\-zero Scores arenegated\. The non\-zero indicators are then split randomly into two equal halves: one half isflipped\(i\.e\.v↦−vv\\mapsto\-v\) and the other half iszeroed out\. To maintain the non\-zero count, the same number of previously zero\-valued indicators are randomly activated with value±1\\pm 1chosen uniformly\. Finally, any dimension whose Score was originally zero is assigned a new Score by Eq\. \([8](https://arxiv.org/html/2608.18096#A4.E8)\) based on the corrupted indicators\.

## Appendix EMulti\-Role Consensus

Table[16](https://arxiv.org/html/2608.18096#A5.T16)summarizes each persona and its primary evaluative lens\. To illustrate the level of specificity, we show the system prompt for Persona 5 \(Judge\) as a representative example:

> You are an impartial Judge\. You evaluate the content strictly based on jurisprudence, procedural fairness, and constitutional rights\. Your analysis is anchored in “Legal/Procedural Justice,” “Rule\-of\-Law Democracy,” “Judicial Equity,” and “Rights\-Based Justice\.” You look for the presence of due process, equality before the law, and the protection of fundamental human rights against majority tyranny\. Content that promotes arbitrary power, vigilantism, or systemic bias will receive severe negative scores in the Justice and Democracy dimensions\.

The full system prompts for all 14 personas are released with our code\.

Table 16:The 14 personas used in Multi\-Role Consensus \(MRC\), each emphasizing a complementary set of MAVEN secondary indicators \(italicized\)\. Persona 1 serves as an impartial baseline; personas 2–14 each bring a stakeholder\-specific perspective\. The full system prompts are released with the dataset\.