MAREval: A Multi-Agent Framework for Evaluating Natural Language Recommendation Explanations

# MAREval: A Multi-Agent Framework for Evaluating Natural Language Recommendation Explanations
Source: [https://arxiv.org/html/2610.36389](https://arxiv.org/html/2610.36389)
\\workshoptitle

Multi\-Turn Interactions in Large Language Models

Jayesh Uddhav Kudase\*Affiliation:Jason Cho Evren Korpeoglu Sushant KumarAysenur Inan\*Affiliation:Walmart Global Tech, Sunnyvale, CA, USA

###### Abstract

Evaluating natural language explanations in recommender systems is essential for fostering user trust, transparency, and engagement\. However, existing evaluation approaches like human evaluations, while accurate, are resource\-intensive and impractical at the scale required by modern recommendation platforms\. Also, automated methods using single\-agent LLMs suffer from prompt sensitivity and inconsistent outputs\. To address these challenges, we propose MAREval, a structured multi\-agent framework for evaluating recommendation explanations using large language models\. MAREval orchestrates \(i\) a planner agent that uses a novel Chain of Debate \(CoD\) prompting strategy to coordinate agent roles and enforce logically consistent evaluation plans; \(ii\) a moderator agent that regulates discussions by mitigating prompt drift; and \(iii\) an arbitrator agent that aggregates outputs from multiple evaluation rounds and \(iv\) a Monte Carlo sampling method, improving robustness and alignment with human judgment\. We conduct comprehensive evaluations on both public \(TopicalChat\) and proprietary recommendation datasets, demonstrating that MAREval outperforms state\-of\-the\-art baselines\. Comprehensive experiments on a public benchmark and a proprietary e\-commerce dataset show that MAREval improves alignment with human judgments over strong single\-and multi\-agent baselines\. Stability analyses indicate substantially lower variability across repeated trials\. In a large human\-annotation gate, MAREval meets production quality thresholds where prior evaluators fall short, and online A/B testing demonstrates statistically significant improvements in engagement and revenue metrics\. These results establish MAREval as a scalable and reliable solution for human\-aligned evaluation of recommendation explanations in real\-world systems\.

11footnotetext:Equal contribution\.22footnotetext:Corresponding author:reza\.yousefimaragheh@walmart\.com## 1Introduction

Millions of users daily interact with recommendation systems to discover tailored products, movies, or music\. System effectiveness hinges not just on recommendations but on accompanying explanations that build trust and guide informed decisions\[[1](https://arxiv.org/html/2610.36389#bib.bib36),[2](https://arxiv.org/html/2610.36389#bib.bib21),[3](https://arxiv.org/html/2610.36389#bib.bib22),[4](https://arxiv.org/html/2610.36389#bib.bib23)\]\. Poor\-quality explanations with misleading or unlear message lead to user frustration and decreased platform reliability\[[5](https://arxiv.org/html/2610.36389#bib.bib13),[6](https://arxiv.org/html/2610.36389#bib.bib6)\]\. Figure[1](https://arxiv.org/html/2610.36389#S1.F1)illustrates this: case \(A\) shows a budget camera misleadingly described for professional photography, case \(B\) presents a smartwatch with poorly phrased explanation, while only case \(C\) demonstrates ideal explanation quality\. Traditional recommendation explanation evaluation relies on expert annotators manually assessing explanations for relevance, phrasing, and informativeness\. While ensuring quality, this approach is expensive, time\-consuming, and impractical for large datasets\[[7](https://arxiv.org/html/2610.36389#bib.bib5),[5](https://arxiv.org/html/2610.36389#bib.bib13)\]\.

Recent advances in large language models have led to automated evaluation methods\[[8](https://arxiv.org/html/2610.36389#bib.bib8)\]\. This is achieved through single model judges or sophisticated strategies like chain\-of\-thought prompting\[[9](https://arxiv.org/html/2610.36389#bib.bib1),[10](https://arxiv.org/html/2610.36389#bib.bib24),[11](https://arxiv.org/html/2610.36389#bib.bib25)\], self\-refinement\[[12](https://arxiv.org/html/2610.36389#bib.bib26),[13](https://arxiv.org/html/2610.36389#bib.bib27)\], and multi\-agent frameworks\[[14](https://arxiv.org/html/2610.36389#bib.bib2)\]\. However, existing solutions suffer critical issues: single agent evaluators exhibit bias and inconsistency\[[15](https://arxiv.org/html/2610.36389#bib.bib3)\], multi\-agent frameworks lack structured coordination, leading to noisy debates\[[16](https://arxiv.org/html/2610.36389#bib.bib4)\]\. These frameworks may also lead to evaluation variability, which means that identical explanations may receive different scores in the same framework\[[17](https://arxiv.org/html/2610.36389#bib.bib14)\]\.

We introduce MAREval, a structured multi\-agent evaluation framework enhancing recommendation explanation evaluation through orchestrated planning, structured debate, and statistical refinement\. MAREval leverages a novel “Chain of Debate” \(CoD\) mechanism, a planning\-first prompting strategy structuring evaluation in multi\-agent frameworks\. Unlike traditional multi\-agent systems where agents interact in an ad hoc or loosely coordinated fashion, MAREval introduces a planner agent that first analyzes the overall flow and then generates synchronized, role\-specific plans for each agent involved in the discussion\. This prestructuring promotes logical consistency in the dialogue and encourages complementary reasoning between agents\. This ensures less evaluation output consistency with ground truth data\.

A dedicated moderator agent orchestrates the conversation flow, distilling the evaluations into concise summaries that capture agreement and contention points\. This grounds subsequent evaluations in shared context, preventing redundancy and noise\. In doing so, the moderator serves as a dynamic memory mechanism, enabling more effective information sharing without overloading the prompt context\. This structured regulation contributes to more stable and interpretable evaluations\.

Finally, MAREval employs an arbitrator agent to judge outcomes across multiple debate rounds and to counteract variability in LLM outputs\. In addition to this arbitration agent, and given the inherent randomness in language model generation MAREval uses a Monte Carlo sampling strategy that aggregates multiple independently sampled runs of the agent debates to arrive at a final decision\. Together with arbitration, the Monte Carlo sampling improve alignment with human annotations and reduce the likelihood of outlier judgments\. Extensive experiments demonstrate MAREval achieves higher consistency with human evaluations than existing methods\. Online A/B tests validate effectiveness, revealing measurable improvements in user engagement and conversion metrics\.

![Refer to caption](https://arxiv.org/html/2610.36389v1/figures/figures_updated.png)Figure 1:Natural Language Evaluation of Generated Recommendation Explanations for eCommerce Products\.
## 2Related Works

With LLM advancements, researchers utilize them for natural language evaluation tasks\[[8](https://arxiv.org/html/2610.36389#bib.bib8),[18](https://arxiv.org/html/2610.36389#bib.bib7),[19](https://arxiv.org/html/2610.36389#bib.bib18),[20](https://arxiv.org/html/2610.36389#bib.bib19)\]\.[Du et al\. \[21\]](https://arxiv.org/html/2610.36389#bib.bib11)systematically showed that using LLMs in debate settings improves reasoning capabilities and output quality\.[Liang et al\. \[22\]](https://arxiv.org/html/2610.36389#bib.bib12)identifies the "Degeneration of Thought" problem in frameworks like Self\-Refine\[[12](https://arxiv.org/html/2610.36389#bib.bib26)\], where LLMs struggle to self\-refine already generated evaluations\. This led to the "Multi\-Agent Debate" \(MAD\) framework, where agents discuss answers to generate refined responses\[[22](https://arxiv.org/html/2610.36389#bib.bib12)\]\.

After demonstrating multi\-agent framework efficacy, researchers applied them to evaluation tasks\.[Li et al\. \[23\]](https://arxiv.org/html/2610.36389#bib.bib9)proposes Peer Discussion, utilizing two LLM agents to discuss evaluation tasks and reach agreement\. Under CHATEVAL\[[14](https://arxiv.org/html/2610.36389#bib.bib2)\], evaluators collectively conduct evaluation tasks with different communication strategies\. Agentic pipelines have been applied to recommendation domains\[[24](https://arxiv.org/html/2610.36389#bib.bib20)\], focusing on: \(i\) context\-aware recommendations\[[25](https://arxiv.org/html/2610.36389#bib.bib29),[26](https://arxiv.org/html/2610.36389#bib.bib30),[27](https://arxiv.org/html/2610.36389#bib.bib28),[28](https://arxiv.org/html/2610.36389#bib.bib38)\], \(ii\) enhancing recommendations through interactive user engagement\[[29](https://arxiv.org/html/2610.36389#bib.bib31),[30](https://arxiv.org/html/2610.36389#bib.bib32),[31](https://arxiv.org/html/2610.36389#bib.bib33)\], \(iii\) simulating user behavior\[[32](https://arxiv.org/html/2610.36389#bib.bib34),[33](https://arxiv.org/html/2610.36389#bib.bib35)\], and \(iv\) recommendation explanation\[[34](https://arxiv.org/html/2610.36389#bib.bib37)\]\.

While prior works like CHATEVAL, Peer Discussion, and MAD demonstrated multi\-agent debate promise, they often struggle with prompt drift, redundancy, and inconsistency in evaluation outputs\. MAREval introduces a novel Chain of Debate planning mechanism that explicitly coordinates agent roles and reasoning sequences\. Additionally, MAREval uniquely integrates a moderator agent and an arbitration agent for communication regulation and evaluation aggregation respectively\. This significantly reduces variability and improving human judgment alignment\. Also, through a Monte\-Carlo sampling method MAREval stabilizes outlier output and improves consistency with ground truth judgments\. These innovations make MAREval the first end\-to\-end multi\-agent evaluation framework optimized for both general dialogue tasks and recommendation\-specific explanation evaluation\.

## 3Methodology

In this section, we review the major elements of MAREval including Chain of Debate \(CoD\) used for planning, Context Moderation \(MoD\) realized by moderation agent, arbitration, and Multi\-Agent Monte\-Carlo Sampling \(MAMCS\)\. We introduce formal notation and discuss the high\-level intuition and logic for each component\. We show a schematic view of the evaluation task in MAREval framework \(see Figure[2](https://arxiv.org/html/2610.36389#S3.F2)\)\.

### 3\.1Chain of Debate

Chain of Debate \(CoD\) extends Chain of Thought prompting to multi\-agent settings by introducing a planner agent that generates synchronized, step\-wise plans for all agents while considering the complete framework flow\. Unlike independent agent planning, CoD ensures coordinated collaboration where each agent’s steps depend on inter\-agent information flow\.

The planner receives evaluation criteria, contextual information, and framework flow description to generate comprehensive plans for evaluator agents, moderator, and arbitrator\. This synchronized planning promotes coherent step\-by\-step coordination across all agents, addressing the challenge of inconsistent and independent agents’ plans in multi\-agent environments\.

Formally, letCCbe evaluation criteria \(e\.g\.,\{relevance,phrasing\}\\\{\\text\{relevance\},\\text\{phrasing\}\\\}\),IIbe contextual information,FFbe the multi\-agent pipeline flow description, andTTbe total evaluation rounds\. Denotemmevaluator agents as𝒜=\{α1,…,αm\}\\mathcal\{A\}=\\\{\\alpha\_\{1\},\\dots,\\alpha\_\{m\}\\\}, moderator agent asαM\\alpha\_\{M\}, and arbitrator agent asαA\\alpha\_\{A\}\.

CoD is defined as:Planner​\(C,I,F\)=\(\{Pte\}t=1T,\{Ptm\}t=1T,Pa\)\\text\{Planner\}\(C,I,F\)=\\Bigl\(\\\{P^\{e\}\_\{t\}\\\}\_\{t=1\}^\{T\},\\\{P^\{m\}\_\{t\}\\\}\_\{t=1\}^\{T\},P^\{a\}\\Bigr\), wherePteP^\{e\}\_\{t\}is the evaluator agent plan for roundtt,PtmP^\{m\}\_\{t\}is the moderator plan for roundtt, andPaP^\{a\}is the arbitrator plan\.

During execution, the planner dispatches appropriate plans to respective agents in each round\. Agents produce intermediate outputs following their instructions while information flows according toFF\. Since the plans\{Pte\}t=1T\\\{P^\{e\}\_\{t\}\\\}\_\{t=1\}^\{T\},\{Ptm\}t=1T\\\{P^\{m\}\_\{t\}\\\}\_\{t=1\}^\{T\}, andPaP^\{a\}are generated in a synchronized manner with full knowledge of\(C,I,F\)\(C,I,F\), CoD ensures coherent, step\-by\-step coordination across all agents\. Although we present CoD in the context of an evaluation task, it can be extended naturally to other NLG workflows by adapting the agent roles accordingly\.

### 3\.2Context Moderation and Arbitration

The*moderator agent*governs information flow among evaluator agents by distilling key discussion points from previous rounds\. Inspired by human\-style debates and recent studies on memory mechanisms in multi\-agent frameworks\[[35](https://arxiv.org/html/2610.36389#bib.bib15)\], the moderator manages context without excessive verbosity or noise, storing essential points for subsequent rounds\.

![Refer to caption](https://arxiv.org/html/2610.36389v1/figures/evaltown_v8_light_purple.png)Figure 2:MAREval Framework\. Under MAREval, \(1\) the Planner Agent is prompted to generate synchronized chain of steps for individual agents of the pipelines given the overall flow design of the pipeline\. \(2\) The discussion among the evaluator agents is moderated by a moderator agent to better conduct the evaluation task\. \(3\) The result of discussions is then passed to the Arbitrator Agent to generate the final ruling on the evaluation task\. \(4\) Monte Carlo Sampling method applied to aggregate different rounds of MAREval block*Evaluator agents*apply plan segments to ongoing discussions, critiquing and refining judgments while flagging errors or hallucinations\. At debate roundtt, thejj\-th evaluator generates evaluationej,te\_\{j,t\}based on planPteP^\{e\}\_\{t\}and discussion historyHt−1H\_\{t\-1\}:

ej,t=f⁡\(Pte,Ht−1\),Ht−1=\{S1,S2,…,St−1\},e\_\{j,t\}=f\(P^\{e\}\_\{t\},H\_\{t\-1\}\),\\quad H\_\{t\-1\}=\\\{S\_\{1\},S\_\{2\},\\ldots,S\_\{t\-1\}\\\},whereSτS\_\{\\tau\}is the moderator summary after roundτ\\tau\.

The moderator combines partial evaluationse1,t,…,em,te\_\{1,t\},\\ldots,e\_\{m,t\}into noise\-reduced summaryStS\_\{t\}, capturing essential disagreements and refinements while compressing repetitive details:

St=M⁡\(e1,t,e2,t,…,em,t\|St−1\)\.S\_\{t\}=M\(e\_\{1,t\},e\_\{2,t\},\\ldots,e\_\{m,t\}\|S\_\{t\-1\}\)\.
While we focus on textual summarization, the moderation mechanism can be generalized\. An embedding\-based approach might maintain vector representations of evaluations, whereM⁡\(⋅\)M\(\\cdot\)operates on both textual inputs and embeddings:

\(St,Zt\)=M⁡\(\{ei,t\}i=1m,St−1,Zt−1\),\(S\_\{t\},Z\_\{t\}\)=M\(\\\{e\_\{i,t\}\\\}\_\{i=1\}^\{m\},S\_\{t\-1\},Z\_\{t\-1\}\),withZt−1Z\_\{t\-1\}representing internal memory\. This enables similarity detection for redundancy filtering, argument clustering, and noise reduction\. In the final round, the moderator provides conclusive summarySTS\_\{T\}to the arbitrator for final decision\-making\.

AfterTTrounds of debate and moderation, the*arbitrator agent*receives the final moderator outputSTS\_\{T\}to produce the ultimate evaluation outcomeYY\. The arbitrator employs structured decision\-making to enhance consistency and reduce variability in final evaluations\. For scalar evaluation scores, the arbitrator aggregates the final summarySTS\_\{T\}with the planner’s arbitration planPaP^\{a\}, evaluation criteriaCC, and instance contextII:

Y=g⁡\(Pa,C,I,ST\),Y=g\(P^\{a\},C,I,S\_\{T\}\),whereg⁡\(⋅\)g\(\\cdot\)converts the moderated summary into the conclusive evaluation outcome\. The framework can be generalized to incorporate richer information\. If the moderator maintains internal representationZTZ\_\{T\}\(e\.g\., neural encoding of the discussion\), the arbitrator can leverage both textual synopsis and semantic relationships:

Y=g⁡\(Pa,C,I,ST,ZT\)\.Y=g\(P^\{a\},C,I,S\_\{T\},Z\_\{T\}\)\.

### 3\.3Monte Carlo Sampling and Score Function

Language models’ probabilistic token sampling introduces high variability in multi\-agent settings, where single\-agent variance compounds in final outputs\. Inspired by\[[36](https://arxiv.org/html/2610.36389#bib.bib10)\], we propose Monte Carlo sampling for multi\-agent evaluation frameworks to mitigate outlier results through multiple pipeline runs and result aggregation\. To the best of our knowledge, the effect of sampling methods in the performance of multi\-agent evaluation frameworks has not been studied\.

For models providing token\-level probabilities, we compute expected outcomes directly\. LetY=\{y1,…,yn\}Y=\\\{y\_\{1\},\\dots,y\_\{n\}\\\}be possible final tokens or labels, withP⁡\(yi\|Pa,C,I,ST\)P\(y\_\{i\}\|P^\{a\},C,I,S\_\{T\}\)as the probability of outputtingyiy\_\{i\}\. The Monte Carlo weighted outcome is:

yM​C=∑i=1nyi×P⁡\(yi\|Pa,C,I,ST\)\.y^\{MC\}=\\sum\_\{i=1\}^\{n\}y\_\{i\}\\times P\(y\_\{i\}\|P^\{a\},C,I,S\_\{T\}\)\.\(1\)
For black\-box models without probability access, we apply pipeline\-level sampling by running the complete multi\-agent systemNNtimes independently\. The final outcome from runkkisYk=\{g⁡\(Pa,C,I,ST\)\}kY\_\{k\}=\\\{g\(P^\{a\},C,I,S\_\{T\}\)\\\}\_\{k\}, aggregated via arithmetic mean:

\{Yk=\{g⁡\(Pa,C,I,ST\)\}k,k=1,…,N,YfinalN=1N​∑k=1NYk\.\\begin\{cases\}Y\_\{k\}=\\\{g\(P^\{a\},C,I,S\_\{T\}\)\\\}\_\{k\},&k=1,\\dots,N,\\\\\[6\.0pt\] Y\_\{\\text\{final\}\}^\{N\}=\\frac\{1\}\{N\}\\sum\_\{k=1\}^\{N\}Y\_\{k\}\.\\end\{cases\}\(2\)
Both strategies address LLM sampling randomness, reducing reliance on single trajectories to achieve more stable evaluation outcomes across multi\-agent system runs \(See Section[4\.4](https://arxiv.org/html/2610.36389#S4.SS4)\)\. Refer to Appendix[A](https://arxiv.org/html/2610.36389#A1),[B](https://arxiv.org/html/2610.36389#A2)and[C](https://arxiv.org/html/2610.36389#A3)for MAREval’s sample outputs and inputs\.

## 4Experiments

### 4\.1Implementation Details and Datasets

We perform evaluation experiments on two datasets: recommendation explanation evaluation on a proprietary e\-commerce dataset and response generation evaluation on the public TopicalChat dataset\[[37](https://arxiv.org/html/2610.36389#bib.bib16),[38](https://arxiv.org/html/2610.36389#bib.bib17)\]: \(i\)Proprietary E\-commerce Dataset: Our Proprietary E\-commerce \(PE\) dataset contains 108 recommendation explanation phrases for target items, describing product "use cases" \(e\.g\., "gaming" for laptops, "heart\-rate monitoring" for smartwatches\)\. Phrases are manually evaluated by product domain experts on two binary criteria: relevance to the item and phrasing quality \(grammatical correctness\)\. We compare MAREval and benchmark model results with these expert annotations\. \(ii\)TopicalChat Dataset: TopicalChat\[[38](https://arxiv.org/html/2610.36389#bib.bib17)\]contains responses generated for chat histories with contextual considerations\. The dataset includes manual annotations across six criteria: naturalness, understandability, groundedness, engagingness, maintaining context, and overall quality\. While not directly containing product recommendations, TopicalChat provides conversational contexts mirroring recommendation scenarios requiring personalized responses based on user preferences and history\. All of the results are based on a gpt\-3\.5\-turbo model\.

### 4\.2Metrics and Benchmarks

We evaluate using Cohen’s Kappa \(κ\\kappa\) and Pearson’s correlation \(ρ\\rho\) for binary outcomes, and Pearson \(ρ\\rho\) and Kendall\-Tau \(τ\\tau\) correlations for integer range outcomes\[[14](https://arxiv.org/html/2610.36389#bib.bib2),[36](https://arxiv.org/html/2610.36389#bib.bib10)\]\. These metrics are standard for measuring agreement between evaluation frameworks and manual annotators\.

We compare against the following benchmark models: \(i\)Vanilla\-gpt: Simple instruction prompting asking the LLM to conduct evaluation based on given criteria\. \(ii\)Chain\-of\-Thought\[[9](https://arxiv.org/html/2610.36389#bib.bib1)\]: CoT prompting where the LLM generates step\-by\-step evaluation plans before conducting assessment\. \(iii\)Self\-Refine\[[12](https://arxiv.org/html/2610.36389#bib.bib26)\]: Response refinement technique where a single LLM generates then refines its evaluation response\. \(iv\)G\-Eval\[[36](https://arxiv.org/html/2610.36389#bib.bib10)\]: Combines CoT prompting with score function sampling, prompting GPT multiple times and averaging results\. \(v\)ChatEval\[[14](https://arxiv.org/html/2610.36389#bib.bib2)\]: State\-of\-the\-art multi\-agent framework using sequential evaluator chats with majority vote for final outcomes\.

Table 1:Performance of different frameworks across TopicalChat and PE datasets\. The variants of MAREval are compared to five benchmark models\. The table shows Pearson Correlation \(ρ\\rho\), Kendall\-Tau \(τ\\tau\), and Cohen’s kappa \(κ\\kappa\) metrics for different evaluation criteria\.TopicalChat \(TC\)MetricsNaturalnessEngagingnessMaintains ContextUnderstandabilityρ\\rhoτ\\tauρ\\rhoτ\\tauρ\\rhoτ\\tauρ\\rhoκ\\kappaCoT0\.3510\.3510\.3170\.3170\.3090\.3090\.2920\.2920\.4900\.4900\.4690\.4690\.2700\.2700\.2150\.215Self\-Refine0\.2210\.2210\.2000\.2000\.3550\.3250\.4080\.4080\.3840\.3840\.1670\.1670\.1440\.144G\-eval0\.3670\.3320\.2840\.2840\.2700\.2700\.4930\.4930\.4700\.4700\.2810\.2810\.2090\.209Vanilla0\.2910\.2910\.2660\.2660\.2030\.2030\.1930\.1930\.5220\.4990\.3360\.304ChatEval0\.2900\.2900\.2690\.2690\.2890\.2890\.2680\.2680\.4820\.4820\.4660\.4660\.2530\.2530\.228MAREval0\.3890\.3500\.3410\.3270\.5240\.5080\.2840\.1810\.181TopicalChat \(TC\)PE DatasetMetricsGroundedness \(TC\)Overall Chat Quality \(TC\)Relevance \(PE\)Phraseness \(PE\)ρ\\rhoκ\\kappaρ\\rhoτ\\tauρ\\rhoκ\\kappaρ\\rhoκ\\kappaCoT0\.4850\.4850\.4850\.4850\.4420\.4420\.3880\.3880\.0840\.0840\.0620\.0620\.0400\.0400\.0290\.029Self\-Refine0\.4120\.4120\.4120\.4120\.4690\.4690\.4010\.4010\.0750\.0750\.0460\.0460\.0260\.0260\.0120\.012G\-eval0\.5210\.5020\.4870\.4870\.4310\.4310\.0700\.0700\.0450\.045−0\.035\-0\.035−0\.023\-0\.023Vanilla0\.4550\.4550\.4550\.4550\.5050\.4380\.1020\.1020\.0580\.0580\.0180\.0180\.0110\.011ChatEval0\.4770\.4770\.4390\.4390\.4550\.4550\.3900\.3900\.2380\.2250\.2030\.196MAREval0\.5530\.5360\.5480\.4780\.3380\.3340\.2160\.208

### 4\.3Overall Performance

In this section, we investigate the performance of different frameworks with human annotators\. Results are reported in Tables[1](https://arxiv.org/html/2610.36389#S4.T1)\. Vanilla and G\-eval demonstrates strong performance on the TopicalChat dataset but struggles significantly with the PE \(recommendation\) dataset\. Vanilla achieves the second\-best performance in Overall Chat Quality \(ρ=0\.505\\rho=0\.505,τ=0\.438\\tau=0\.438\), while G\-eval follows closely \(ρ=0\.487\\rho=0\.487,τ=0\.431\\tau=0\.431\) for TopicalChat\. However, both models show dramatically reduced effectiveness when evaluating PE metrics This pattern suggests the lack of robustness for these two models when performing on recommendation evaluation tasks\.

Self\-Refine and CoT show middling performance across both datasets\. Self\-Refine ranks fourth in Overall Chat Quality \(ρ=0\.469\\rho=0\.469,τ=0\.401\\tau=0\.401\) but falls to the bottom tier for PE metrics\. Similarly, CoT shows the weakest performance in TC Overall Chat Quality among all models tested \(ρ=0\.442\\rho=0\.442,τ=0\.388\\tau=0\.388\) and only marginally better results than Self\-Refine and G\-eval on PE metrics\. This is due to the effect of the “degeneration of thought”\[[21](https://arxiv.org/html/2610.36389#bib.bib11)\]with Self\-Refine and CoT, where a single agent cannot correct their opinion once biased from the beginning\.

ChatEval show better performance on TopicalChat’s Overall Chat Quality but emerging as the clear second\-best performer on both PE dataset metrics\. ChatEval demonstrates considerably better cross\-domain capability than other benchmarks\. This suggest ChatEval and multi\-agent systems in general can achieve robust performance across evaluation tasks native to recommendation setting and they can safely be employed without drastic performance degradation across board\.

MAREval consistently outperforms all benchmarks across key metrics\. In Overall Chat Quality for the TopicalChat dataset, MAREval achieves the highest correlations \(ρ=0\.548\\rho=0\.548,τ=0\.478\\tau=0\.478\), demonstrating its superior ability to assess conversational quality\. This performance advantage becomes even more pronounced in the PE dataset, where MAREval significantly outperforms all benchmarks in both Relevance \(ρ=0\.338\\rho=0\.338,κ=0\.334\\kappa=0\.334\) and Phraseness \(ρ=0\.216\\rho=0\.216,κ=0\.208\\kappa=0\.208\), suggesting robust cross\-domain evaluation capabilities\. The results reveal MAREval’s significant advancement over existing evaluation frameworks, particularly in its balanced excellence across both conversational and recommendation evaluation tasks\. In addition to the performance analysis above, we study the variability of the evaluation output by different models\.

### 4\.4Stability and Reliability Analysis

We conduct an analysis of output variability across different evaluation models to assess the stability and reliability of MAREval compared to benchmark approaches\. Output variability is measured using the standard deviation of evaluation scores for identical instances across multiple runs, providing insight into the consistency of each framework’s decision\-making process\. High variability indicates unreliable evaluation behavior, while low variability suggests more stable and predictable outcomes that are crucial for practical deployment in production recommendation systems\.

Table[2](https://arxiv.org/html/2610.36389#S4.T2)presents the standard deviation results across all six evaluation criteria of the TopicalChat dataset\. MAREval demonstrates superior stability with the lowest average standard deviation \(0\.0197\), outperforming all baseline models\. Notably, G\-EVAL, the best\-performing benchmark model in terms of variability, still exhibits 15\.22% higher average standard deviation compared to MAREval \(0\.0227 vs 0\.0197\)\. This substantial improvement highlights MAREval’s enhanced reliability in producing consistent evaluation outcomes\. Self\-Refine exhibits the highest variability with an average standard deviation of 0\.1563, particularly problematic for overall chat quality \(0\.3429\) and engagingness \(0\.1784\), indicating severe inconsistency that would be unsuitable for automated evaluation systems\. We speculate the reason for this bad performance of Self\-Refine to be associated with longer token trajectory without any moderation\.

The superior stability of MAREval can be attributed to its structured multi\-agent coordination mechanisms and Monte Carlo sampling strategy\. The Chain of Debate component ensures synchronized planning across agents, reducing conflicting evaluations that contribute to output variance\. The moderator agent filters noise and maintains contextual coherence across evaluation rounds, preventing drift that often leads to inconsistent judgments\. Most importantly, the Monte Carlo sampling aggregates multiple independent runs, effectively averaging out the inherent randomness of token\-based language model generation\. This comprehensive approach to variability reduction makes MAREval particularly suitable for real\-world applications where consistent and reliable evaluation is paramount for maintaining user trust and system performance\.

Table 2:Standard deviation of evaluation scores across multiple runs for different models on TopicalChat dataset criteria
### 4\.5Ablation Study

Table 3:Ablation study on variants of MAREval and their correlation performance with Human Evaluations Across TopicalChat and PE DatasetsTopicalChat \(TC\)MetricsNaturalnessEngagingnessMaintains ContextUnderstandabilityρ\\rhoτ\\tauρ\\rhoτ\\tauρ\\rhoτ\\tauρ\\rhoκ\\kappaMAREval w\.o\. MoD&MC0\.3140\.3140\.2890\.2890\.3330\.3330\.3190\.3190\.4930\.4930\.4620\.4620\.2460\.2460\.1340\.134MAREval w\.o\. MoD0\.3400\.3170\.3680\.3560\.4990\.4670\.2600\.1580\.158MAREval0\.3890\.3500\.3410\.3270\.5240\.5080\.2840\.181TopicalChat \(TC\)PE DatasetMetricsGroundedness \(TC\)Overall Chat Quality \(TC\)Relevance \(PE\)Phraseness \(PE\)ρ\\rhoκ\\kappaρ\\rhoτ\\tauρ\\rhoκ\\kappaρ\\rhoκ\\kappaMAREval w\.o\. MoD&MC0\.4840\.4840\.4840\.4840\.4560\.4560\.4040\.4040\.3060\.3010\.3010\.1740\.1740\.1670\.167MAREval w\.o\. MoD0\.4970\.4840\.5030\.4530\.3260\.3250\.2160\.208MAREval0\.5530\.5360\.5480\.4780\.3380\.3340\.2160\.208

Our ablation study investigates individual component contributions within MAREval\. As shown in Table[3](https://arxiv.org/html/2610.36389#S4.T3), the complete MAREval demonstrates superior performance across most metrics in the TopicalChat dataset, indicating effective generalization and positive contributions from all components\.

Removing the Moderation component \(MAREval w\.o\. MoD\) causes significant performance decreases\. Naturalness correlation drops fromρ=0\.389\\rho=0\.389toρ=0\.340\\rho=0\.340, and Maintains Context decreases fromρ=0\.524\\rho=0\.524toρ=0\.499\\rho=0\.499\. The MoD component proves particularly critical for groundedness evaluation, with correlation scores dropping substantially fromρ=0\.553,κ=0\.536\\rho=0\.553,\\kappa=0\.536toρ=0\.497,κ=0\.484\\rho=0\.497,\\kappa=0\.484\. This effectiveness stems from moderated chat context provision, allowing the arbitrator to access relevant information without prompt pollution\.

Removing both Moderation and Monte Carlo sampling \(MAREval w\.o\. MoD&MC\) produces more pronounced degradation across evaluation dimensions, particularly in Naturalness, Engagingness, and Overall Chat Quality\. The substantial performance gap in Overall Chat Quality \(ρ=0\.456\\rho=0\.456versusρ=0\.548\\rho=0\.548\) highlights Monte Carlo sampling’s essential stabilizing role in probabilistic token\-based LLM pipelines, alleviating outcome variance and increasing robustness\. Similar degradation occurs when the CoD component is removed\. Performance comparisons across datasets confirm CoD’s relative advantage, attributed to pipeline\-aware planning by the planner agent, leading to improved agent synchronization\.

![Refer to caption](https://arxiv.org/html/2610.36389v1/figures/hyper_3.png)Figure 3:\(a\) Effect of Number of Monte\-Carlo samples\. \(b\) Effect of Number of evaluator agents\.
### 4\.6Hyperparamers and Performance vs Latency

As we discuss in the next section, MAREval has enabled us to pass the minimum requirement for user facing explanations\. However, this entablement comes at a cost of latency and it will be beneficial to have a discussion on latency versus robustness and performance\. Figure[3](https://arxiv.org/html/2610.36389#S4.F3)illustrates the relationship between the correlation metrics and \(a\) number of MAMCS runs, and \(b\) number of evaluator agents in the proposed multi\-agent pipeline\. As shown, both Pearson correlation and Kendall\-tau correlations generally improve as the number of evaluator agents increases from22to44, with Pearson correlation rising from0\.5090\.509to0\.5480\.548and Kendall\-tau from0\.4420\.442to0\.4780\.478\. Beyond44agents, the correlation metrics exhibit slight fluctuations rather than monotonic growth, with Pearson correlation reaching another peak of0\.5480\.548at44agents and Kendall\-tau maintaining relatively stable values between0\.4680\.468and0\.4750\.475\. Same pattern happens for the number of MAMCS runs, as it reaches its higher performance in 5 sample runs and not showing significant marginal correlation improvement for more number of runs\. In our experiments with five evaluator agents, the average single thread run time of the gpt\-3\.5\-Turbo model in the isolated enterprise serving layer was 8\.16 seconds\. We also, conducted an analysis on the performance MAREval using different backbone models which we defer to Appendix[E](https://arxiv.org/html/2610.36389#A5)\.

### 4\.7Human\-Annotated Evaluation

To productionize the generated recommendation explanations and expose them to live traffic, we first ran a human annotation study\. We were required to satisfy predetermined minimum quality thresholds set by the quality\-control team dectated by our product launch protocols\. These proprietary requirements were divided into hard and soft constraints; following industry practice, we sought a near\-100% pass rate on the hard constraints prior to launch\. In the*first round*, we submitted only the explanations approved by G\-Eval, the state\-of\-the\-art LLM\-as\-a\-judge framework at the time of the experiment\. However, these G\-Eval\-approved explanations did not reach the near\-100% satisfaction of the hard requirements mandated for launch and exhibited lower\-than\-expected performance on soft constraints\. Motivated by these shortcomings, we designed MAREval to deliver reliable and robust judgments via structured multi\-agent planning, moderation, and arbitration\.

Table 4:Human\-Annotation Results\.In a second round of manual evaluation, which is an expensive procedure conducted on the scale𝒪⁡\(1000\)\\mathcal\{O\}\(1000\), we submitted only the explanations that passed MAREval\. This round achieved our internal launch thresholds for the first time andMAREvalsatisfied all hard requirements \(100% vs\. 91% for G\-Eval\) and substantially improved soft\-criterion satisfaction \(98\.6% vs\. 76\.1%\), over 5,000 MAREval\-filtered and 2,500 G\-Eval\-filtered samples, respectively \(see Table[4](https://arxiv.org/html/2610.36389#S4.T4)\)\. These results underscore the practical hurdles of deploying recommendation explanations in production and in our environment, no prior method met the reliability bar necessary for live user traffic, whereas MAREval did\.

### 4\.8Online Experiments

We conducted an online A/B test to assess MAREval’s impact on user behavior in live traffic\. A proprietary recommendation explanation model generated explanations for approximately 300,000 high\-traffic items across 14 product categories, describing potential use cases for each product\. Only explanations that passed MAREval’s evaluation were shown to users in the treatment variant; the control group saw the same items without explanations\. We report two standard business metrics: Add\-to\-Cart \(ATC\) rate and Gross Merchandise Value \(GMV\) lift, which evaluate whether explanations drive engagement and purchase behavior\. See Figure[6](https://arxiv.org/html/2610.36389#A4.F6)for an example of the live experience and sample MAREval approved evaluations\.

Table[5](https://arxiv.org/html/2610.36389#S4.T5)reports the online outcomes\. MAREval\-validated explanations significantly improved both business metrics\. ATC increased by 1\.18% \(p=0\.05p\{=\}0\.05\) and GMV increased by 1\.36% \(p<0\.01p\{<\}0\.01\), confirming that quality\-filtered explanations meaningfully affect user engagement and conversion in production settings \(see Appendix[D](https://arxiv.org/html/2610.36389#A4)for a sample of real world implementation\)\.

Table 5:Online A/B test results on live traffic\.All these results indicate that MAREval is able to provide accurate and stable evaluations in different settings\.

## 5Conclusion

In this work, we introduced MAREval, a structured multi\-agent framework for evaluating natural language explanations in recommender systems\. Unlike prior approaches that rely on single\-agent prompting or loosely coordinated agent interactions, MAREval brings together a planner agent with a novel Chain of Debate \(CoD\) prompting strategy, a moderator agent to regulate discourse and reduce context noise, and an arbitrator agent that is being coupled with Monte Carlo sampling for a more robust, consistent, and human\-aligned evaluation outcomes\. Comprehensive experiments on both public \(TopicalChat\) and proprietary recommendation datasets demonstrate that MAREval consistently outperforms state\-of\-the\-art baselines across a range of evaluation criteria\. Notably, MAREval achieves stronger alignment with human judgments, as well as less output variability, where existing models fall short\. Our ablation studies further validate the complementary benefits of components of MAREval\. Results from a large\-scale A/B test indicate that explanations filtered through MAREval yield significant improvements in user engagement, with a 1\.36% lift in GMV and 1\.18% increase in ATC rate\. Taken together, our findings position MAREval as a scalable, reliable, and practically effective framework for natural language evaluation in recommender systems\.

## References

- \[1\]R\. Y\. Maragheh and Y\. Deldjoo\(2025\)The future is agentic: definitions, perspectives, and open challenges of multi\-agent recommender systems\.arXiv preprint arXiv:2507\.02097\.Cited by:[§1](https://arxiv.org/html/2610.36389#S1.p1.1)\.
- \[2\]Y\. Zhang X\. Chenet al\.\(2020\)Explainable recommendation: a survey and new perspectives\.Foundations and Trends® in Information Retrieval14\(1\),pp\. 1–101\.Cited by:[§1](https://arxiv.org/html/2610.36389#S1.p1.1)\.
- \[3\]K\. Balog and F\. Radlinski\(2020\)Measuring recommendation explanation quality: the conflicting goals of explanations\.InProceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval,pp\. 329–338\.Cited by:[§1](https://arxiv.org/html/2610.36389#S1.p1.1)\.
- \[4\]H\. Chen, X\. Chen, S\. Shi, and Y\. Zhang\(2021\)Generate natural language explanations for recommendation\.arXiv preprint arXiv:2101\.03392\.Cited by:[§1](https://arxiv.org/html/2610.36389#S1.p1.1)\.
- \[5\]Z\. Ji, Y\. Tiezheng, Y\. Xu, N\. Lee, E\. Ishii, and P\. Fung\(2023\)Towards mitigating llm hallucination via self reflection\.InThe 2023 Conference on Empirical Methods in Natural Language Processing,Cited by:[§1](https://arxiv.org/html/2610.36389#S1.p1.1)\.
- \[6\]P\. Wang, L\. Li, L\. Chen, D\. Zhu, B\. Lin, Y\. Cao, Q\. Liu, T\. Liu, and Z\. Sui\(2023\)Large language models are not fair evaluators\.arXiv preprint arXiv:2305\.17926\.Cited by:[§1](https://arxiv.org/html/2610.36389#S1.p1.1)\.
- \[7\]C\. Lin\(2004\)Rouge: a package for automatic evaluation of summaries\.InText summarization branches out,pp\. 74–81\.Cited by:[§1](https://arxiv.org/html/2610.36389#S1.p1.1)\.
- \[8\]L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.\(2024\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in Neural Information Processing Systems36\.Cited by:[§1](https://arxiv.org/html/2610.36389#S1.p2.1),[§2](https://arxiv.org/html/2610.36389#S2.p1.1)\.
- \[9\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. Le, and D\. Zhou\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.NeurIPS\.Cited by:[§1](https://arxiv.org/html/2610.36389#S1.p2.1),[§4\.2](https://arxiv.org/html/2610.36389#S4.SS2.p2.1)\.
- \[10\]X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou\(2022\)Self\-consistency improves chain of thought reasoning in language models\.arXiv preprint arXiv:2203\.11171\.Cited by:[§1](https://arxiv.org/html/2610.36389#S1.p2.1)\.
- \[11\]A\. Chowdhery, S\. Narang, J\. Devlin, M\. Bosma, G\. Mishra, A\. Roberts, P\. Barham, H\. W\. Chung, C\. Sutton, S\. Gehrmann,et al\.\(2023\)Palm: scaling language modeling with pathways\.Journal of Machine Learning Research24\(240\),pp\. 1–113\.Cited by:[§1](https://arxiv.org/html/2610.36389#S1.p2.1)\.
- \[12\]A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang,et al\.\(2023\)Self\-refine: iterative refinement with self\-feedback\.Advances in Neural Information Processing Systems36,pp\. 46534–46594\.Cited by:[§1](https://arxiv.org/html/2610.36389#S1.p2.1),[§2](https://arxiv.org/html/2610.36389#S2.p1.1),[§4\.2](https://arxiv.org/html/2610.36389#S4.SS2.p2.1)\.
- \[13\]Y\. Zeng, X\. Cui, X\. Jin, G\. Liu, Z\. Sun, Q\. He, D\. Li, N\. Yang, J\. Hao, H\. Zhang,et al\.\(2025\)ARIES: stimulating self\-refinement of large language models by iterative preference optimization\.arXiv preprint arXiv:2502\.05605\.Cited by:[§1](https://arxiv.org/html/2610.36389#S1.p2.1)\.
- \[14\]C\. Chan, W\. Chen, Y\. Su, J\. Yu, W\. Xue, S\. Zhang, J\. Fu, and Z\. Liu\(2023\)ChatEval: towards better llm\-based evaluators through multi\-agent debate\.arXiv preprint arXiv:2308\.07201\.Cited by:[§1](https://arxiv.org/html/2610.36389#S1.p2.1),[§2](https://arxiv.org/html/2610.36389#S2.p2.1),[§4\.2](https://arxiv.org/html/2610.36389#S4.SS2.p1.1),[§4\.2](https://arxiv.org/html/2610.36389#S4.SS2.p2.1)\.
- \[15\]Y\. Liu, N\. S\. Moosavi, and C\. Lin\(2023\)LLMs as narcissistic evaluators: when ego inflates evaluation scores\.arXiv preprint arXiv:2311\.09766\.Cited by:[§1](https://arxiv.org/html/2610.36389#S1.p2.1)\.
- \[16\]Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, S\. Zhang, E\. Zhu, B\. Li, L\. Jiang, X\. Zhang, and C\. Wang\(2023\)AutoGen: enabling next\-gen llm applications via multi\-agent conversation framework\.arXiv preprint arXiv:2308\.08155\.Cited by:[§1](https://arxiv.org/html/2610.36389#S1.p2.1)\.
- \[17\]H\. Duan, Y\. Yang, and K\. Y\. Tam\(2024\)Do llms know about hallucination? an empirical investigation of llm’s hidden states\.arXiv preprint arXiv:2402\.09733\.Cited by:[§1](https://arxiv.org/html/2610.36389#S1.p2.1)\.
- \[18\]R\. Y\. Maragheh, C\. Fang, C\. C\. Irugu, P\. Parikh, J\. Cho, J\. Xu, S\. Sukumar, M\. Patel, E\. Korpeoglu, S\. Kumar,et al\.\(2023\)LLM\-take: theme\-aware keyword extraction using large language models\.In2023 IEEE International Conference on Big Data \(BigData\),pp\. 4318–4324\.Cited by:[§2](https://arxiv.org/html/2610.36389#S2.p1.1)\.
- \[19\]X\. Li, T\. Zhang, Y\. Dubois, R\. Taori, I\. Gulrajani, C\. Guestrin, P\. Liang, and T\. B\. Hashimoto\(2023\)Alpacaeval: an automatic evaluator of instruction\-following models\.Cited by:[§2](https://arxiv.org/html/2610.36389#S2.p1.1)\.
- \[20\]S\. Kim, J\. Shin, Y\. Cho, J\. Jang, S\. Longpre, H\. Lee, S\. Yun, S\. Shin, S\. Kim, J\. Thorne,et al\.\(2023\)Prometheus: inducing fine\-grained evaluation capability in language models\.InThe Twelfth International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2610.36389#S2.p1.1)\.
- \[21\]Y\. Du, S\. Li, A\. Torralba, J\. B\. Tenenbaum, and I\. Mordatch\(2023\)Improving factuality and reasoning in language models through multiagent debate\.arXiv preprint arXiv:2305\.14325\.Cited by:[§2](https://arxiv.org/html/2610.36389#S2.p1.1),[§4\.3](https://arxiv.org/html/2610.36389#S4.SS3.p2.1)\.
- \[22\]T\. Liang, Z\. He, W\. Jiao, X\. Wang, Y\. Wang, R\. Wang, Y\. Yang, Z\. Tu, and S\. Shi\(2023\)Encouraging divergent thinking in large language models through multi\-agent debate\.arXiv preprint arXiv:2305\.19118\.Cited by:[§2](https://arxiv.org/html/2610.36389#S2.p1.1)\.
- \[23\]R\. Li, T\. Patel, and X\. Du\(2023\)Prd: peer rank and discussion improve large language model based evaluations\.arXiv preprint arXiv:2307\.02762\.Cited by:[§2](https://arxiv.org/html/2610.36389#S2.p2.1)\.
- \[24\]Q\. Peng, H\. Liu, H\. Huang, Q\. Yang, and M\. Shao\(2025\)A survey on llm\-powered agents for recommender systems\.arXiv preprint arXiv:2502\.10050\.Cited by:[§2](https://arxiv.org/html/2610.36389#S2.p2.1)\.
- \[25\]Z\. Wang, Y\. Yu, W\. Zheng, W\. Ma, and M\. Zhang\(2024\)Macrec: a multi\-agent collaboration framework for recommendation\.InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 2760–2764\.Cited by:[§2](https://arxiv.org/html/2610.36389#S2.p2.1)\.
- \[26\]Y\. Zhao, J\. Wu, X\. Wang, W\. Tang, D\. Wang, and M\. De Rijke\(2024\)Let me do it for you: towards llm empowered recommendation via tool learning\.InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 1796–1806\.Cited by:[§2](https://arxiv.org/html/2610.36389#S2.p2.1)\.
- \[27\]Y\. Shu, H\. Zhang, H\. Gu, P\. Zhang, T\. Lu, D\. Li, and N\. Gu\(2024\)RAH\! recsys–assistant–human: a human\-centered recommendation framework with llm agents\.IEEE Transactions on Computational Social Systems\.Cited by:[§2](https://arxiv.org/html/2610.36389#S2.p2.1)\.
- \[28\]R\. Giahi, K\. Yao, S\. Kollipara, K\. Zhao, V\. Mirjalili, J\. Xu, T\. Biswas, E\. Korpeoglu, and K\. Achan\(2025\)VL\-clip: enhancing multimodal recommendations via visual grounding and llm\-augmented clip embeddings\.arXiv preprint arXiv:2507\.17080\.Cited by:[§2](https://arxiv.org/html/2610.36389#S2.p2.1)\.
- \[29\]L\. Friedman, S\. Ahuja, D\. Allen, Z\. Tan, H\. Sidahmed, C\. Long, J\. Xie, G\. Schubiner, A\. Patel, H\. Lara,et al\.\(2023\)Leveraging large language models in conversational recommender systems\.arXiv preprint arXiv:2305\.07961\.Cited by:[§2](https://arxiv.org/html/2610.36389#S2.p2.1)\.
- \[30\]J\. Fang, S\. Gao, P\. Ren, X\. Chen, S\. Verberne, and Z\. Ren\(2024\)A multi\-agent conversational recommender system\.arXiv preprint arXiv:2402\.01135\.Cited by:[§2](https://arxiv.org/html/2610.36389#S2.p2.1)\.
- \[31\]J\. Zhang, K\. Bao, W\. Wang, Y\. Zhang, W\. Shi, W\. Xu, F\. Feng, and T\. Chua\(2024\)Prospect personalized recommendation on large language model\-based agent platform\.arXiv preprint arXiv:2402\.18240\.Cited by:[§2](https://arxiv.org/html/2610.36389#S2.p2.1)\.
- \[32\]L\. Zhu, X\. Huang, and J\. Sang\(2024\)A llm\-based controllable, scalable, human\-involved user simulator framework for conversational recommender systems\.arXiv preprint arXiv:2405\.08035\.Cited by:[§2](https://arxiv.org/html/2610.36389#S2.p2.1)\.
- \[33\]S\. Yoon, Z\. He, J\. M\. Echterhoff, and J\. McAuley\(2024\)Evaluating large language models as generative user simulators for conversational recommendation\.arXiv preprint arXiv:2403\.09738\.Cited by:[§2](https://arxiv.org/html/2610.36389#S2.p2.1)\.
- \[34\]J\. Chen, K\. Yao, R\. Y\. Maragheh, K\. Zhao, J\. Xu, J\. Cho, E\. Korpeoglu, S\. Kumar, and K\. Achan\(2025\)CARTS: collaborative agents for recommendation textual summarization\.arXiv preprint arXiv:2506\.17765\.Cited by:[§2](https://arxiv.org/html/2610.36389#S2.p2.1)\.
- \[35\]G\. Chen, S\. Dong, Y\. Shu, G\. Zhang, J\. Sesay, B\. F\. Karlsson, J\. Fu, and Y\. Shi\(2023\)Autoagents: a framework for automatic agent generation\.arXiv preprint arXiv:2309\.17288\.Cited by:[§3\.2](https://arxiv.org/html/2610.36389#S3.SS2.p1.1)\.
- \[36\]Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. Zhu\(2023\)Gpteval: nlg evaluation using gpt\-4 with better human alignment\.arXiv preprint arXiv:2303\.16634\.Cited by:[§3\.3](https://arxiv.org/html/2610.36389#S3.SS3.p1.1),[§4\.2](https://arxiv.org/html/2610.36389#S4.SS2.p1.1),[§4\.2](https://arxiv.org/html/2610.36389#S4.SS2.p2.1)\.
- \[37\]M\. Zhong, Y\. Liu, D\. Yin, Y\. Mao, Y\. Jiao, P\. Liu, C\. Zhu, H\. Ji, and J\. Han\(2022\)Towards a unified multi\-dimensional evaluator for text generation\.arXiv preprint arXiv:2210\.07197\.Cited by:[§4\.1](https://arxiv.org/html/2610.36389#S4.SS1.p1.1)\.
- \[38\]K\. Gopalakrishnan, B\. Hedayatnia, Q\. Chen, A\. Gottardi, S\. Kwatra, A\. Venkatesh, R\. Gabriel, and D\. Hakkani\-Tur\(2023\)Topical\-chat: towards knowledge\-grounded open\-domain conversations\.arXiv preprint arXiv:2308\.11995\.Cited by:[§4\.1](https://arxiv.org/html/2610.36389#S4.SS1.p1.1)\.

MAREval: A Multi\-Agent Framework for Evaluating Natural Language Recommendation Explanations

## Appendix ACase Study I: MAREval Evaluation on Conversational Response

Following are the conversation output from various agents in MAREval for two examples from the Topical\-Chat dataset\. It first outlines the chat and then the corresponding response that was evaluated using MAREval for two criteria:UnderstandabilityandNaturalness\.

#### Chat:

did you know that facebook and twitter work together on some issues ? they are actually competitors\.

no , i did n’t know that \. what kind of issues do they work together on ?

the baby sitter \- rating system broke the social networks ’ rules \.

#### Response:

i think i heard about that now that you mention it \. was the about predictim ?

Criteria: Understandability

#### Evaluator 1:

The response is understandable because the person is acknowledging that they may have heard about something when the topic of Predictim is brought up\. It shows a level of recognition and understanding of the subject being discussed\. Score of Evaluation: 1

#### Evaluator 2:

Based on the response provided, the sentence structure is a bit confusing and lacks clarity\. The use of "now that you mention it" and the question "was the about predictim?" make it difficult to fully understand the message being conveyed\. Therefore, I would rate this response as not fully understandable\. Score of Evaluation: 0

#### Evaluator 3:

The response is understandable as the speaker is acknowledging that they may have heard about something when the topic of Predictim is brought up\. The response shows comprehension and clarity in communication\. Score of Evaluation: 1

#### Moderator:

Based on the evaluations provided, Evaluator 1 believes the response is understandable due to the acknowledgment and recognition shown by the speaker\. Evaluator 2, however, finds the sentence structure confusing and lacking clarity, leading to a score of 0 for understandability\. Evaluator 3 agrees with Evaluator 1, stating that the response shows comprehension and clarity\. Overall, there is a split in opinions with two evaluators finding the response understandable and one evaluator finding it not fully understandable\.

#### Arbitrator:

Score of Evaluation: 1

Criteria: Naturalness

#### Evaluator 1:

The response seems somewhat unnatural because of the lack of punctuation and the awkward phrasing\. It could be improved by adding a comma after "heard about that" and rephrasing the second sentence for better flow\. Score of Evaluation: 2

#### Evaluator 2:

The response seems somewhat unnatural because of the awkward phrasing and lack of punctuation\. The use of "now that you mention it" feels a bit forced in this context\. However, the overall message is still understandable\. Score of Evaluation: 2

#### Evaluator 3:

The response seems somewhat unnatural because of the lack of punctuation and the awkward phrasing\. It could be improved by adding a comma after "heard about that" and rephrasing the second sentence for better flow\. Score of Evaluation: 2

#### Moderator:

The evaluators found the response somewhat unnatural due to the lack of punctuation and awkward phrasing\. They suggested adding a comma after "heard about that" and rephrasing the second sentence for better flow\. Overall, they gave a score of 2 for the evaluation criteria of naturalness\.

#### Arbitrator:

Score of Evaluation: 2

#### Chat:

do you like football ?

i love it \. do you know what color is the u of iowa visitor locker room ?

yes it is pink but i a not sure why they chose that color

i wonder if it is supposed to have some effect on them

i wonder too \. bowling was more popular than football in the 60s

#### Response:

it was in the 1960s and the difference \. i guess football was not as popular as top bowlers back in the 1960s

Criteria: Understandability

#### Evaluator 1:

Based on the response provided, it seems like the individual is trying to convey that football was not as popular as top bowlers back in the 1960s\. The response is somewhat understandable, but it lacks clarity and coherence in connecting the two points\. Therefore, I would rate it as: Score of Evaluation: 0

#### Evaluator 2:

Based on the response provided, it seems like the individual is trying to convey that in the 1960s, football was not as popular as top bowlers\. The response is somewhat understandable, but it lacks clarity and coherence in connecting the two points\. Therefore, I would rate it as: Score of Evaluation: 0

#### Evaluator 3:

Based on the response provided, it seems like the individual is trying to convey that in the 1960s, football was not as popular as top bowlers\. The response is somewhat understandable, but it lacks clarity and coherence in connecting the two points\. Therefore, I would rate it as: Score of Evaluation: 0

#### Moderator:

The evaluators all agree that the response is somewhat understandable but lacks clarity and coherence in connecting the two points about football and top bowlers in the 1960s\. Therefore, they all rated it as 0 for understandability\.

#### Arbitrator:

Score of Evaluation: 0

Criteria: Naturalness

#### Evaluator 1:

The response provided is somewhat unnatural due to the lack of proper punctuation and clarity\. The sentence structure is a bit confusing and could be improved for better understanding\. Score of Evaluation: 2

#### Evaluator 2:

The response provided is somewhat unnatural due to the lack of clarity and coherence in the sentence structure\. The use of "the difference" without further explanation makes the response confusing\. Additionally, the abrupt transition from mentioning the 1960s to comparing football and top bowlers without any context or explanation also contributes to the unnaturalness of the response\. Score of Evaluation: 1

#### Evaluator 3:

The response provided is somewhat unnatural due to the lack of clarity and coherence in the sentence structure\. The use of "the difference" without further explanation makes the response confusing\. Additionally, the abrupt transition from mentioning the 1960s to comparing the popularity of football and top bowlers is not smoothly executed\. Score of Evaluation: 2

#### Moderator:

The evaluators found the response to be somewhat unnatural due to the lack of clarity and coherence in the sentence structure\. They noted issues with punctuation, lack of explanation for certain phrases like "the difference," and a confusing transition from mentioning the 1960s to comparing football and top bowlers\. Overall, the response could be improved for better understanding

#### Arbitrator:

Score of Evaluation: 2

## Appendix BCase Study II: MAREval Evaluation on Recommendation Explanations

![Refer to caption](https://arxiv.org/html/2610.36389v1/figures/picA.png)

![Refer to caption](https://arxiv.org/html/2610.36389v1/figures/picB.png)

![Refer to caption](https://arxiv.org/html/2610.36389v1/figures/picC.png)

![Refer to caption](https://arxiv.org/html/2610.36389v1/figures/picD.png)

![Refer to caption](https://arxiv.org/html/2610.36389v1/figures/picE.png)

![Refer to caption](https://arxiv.org/html/2610.36389v1/figures/picF.png)

Figure 4:Examples of MARVal evaluation for recommendation explanations![Refer to caption](https://arxiv.org/html/2610.36389v1/figures/picG.png)

![Refer to caption](https://arxiv.org/html/2610.36389v1/figures/picH.png)

![Refer to caption](https://arxiv.org/html/2610.36389v1/figures/picI.png)

![Refer to caption](https://arxiv.org/html/2610.36389v1/figures/picJ.png)

![Refer to caption](https://arxiv.org/html/2610.36389v1/figures/picK.png)

![Refer to caption](https://arxiv.org/html/2610.36389v1/figures/picL.png)

Figure 5:Examples of MARVal evaluation for recommendation explanations
## Appendix CEvaluation Prompts

In this section we list the various evaluation criterias and the prompts for MAREval agents\.

### C\.1Planner Prompt

Planner PromptYou are a discussion planner\. Your task is to generate a plan and chain of steps for an evaluator that is conducting an evaluation task in the following multi\-agent pipeline\.Pipeline flow:•Multiple evaluators independently assess a response using specific criteria•A Moderator agent receives these evaluations and creates a balanced summary•A judge receives the moderator’s summary and makes the final evaluation decisionThe evaluator will be asked to evaluate the generated response for a given chat, while considering chat context and evaluation criteria:The context of the chat:c​o​n​t​e​x​tc​h​a​t\{\\color\[rgb\]\{0,0,1\}\{context\}\_\{chat\}\}\[The messages from the conversation\]The response:r​e​s​p​o​n​s​e\{\\color\[rgb\]\{0,0,1\}response\}\[Response following the above conversation\]evaluation criteria:e​v​a​lc​r​i​t​e​r​i​a\{\\color\[rgb\]\{0,0,1\}\{eval\}\_\{criteria\}\}\[Evaluation criteria along with its definition and guidelines\]Propose evaluation steps\. The steps should include:•Algorithmic sketch of the evaluation process\.•One step for inclusion of the scoring rule\.•Consider the flow of the pipeline and Evaluators position\.Output your response in the following format only: \[\["Chain of Steps": Step 1\. …\. Step 2…\. \]\] Output your response in the above format only\.

### C\.2Evaluator Prompt

Evaluator PromptYou are a critic \- an evaluator\. You will evaluate the generated response for a chat, while considering chat’s context and evaluation criteria:The context of the chat:c​o​n​t​e​x​tc​h​a​t\{\\color\[rgb\]\{0,0,1\}\{context\}\_\{chat\}\}\[The messages from the conversation\]The response:r​e​s​p​o​n​s​e\{\\color\[rgb\]\{0,0,1\}response\}\[Response following the above conversation\]evaluation criteria:e​v​a​lc​r​i​t​e​r​i​a\{\\color\[rgb\]\{0,0,1\}\{eval\}\_\{criteria\}\}\[Evaluation criteria along with its definition and guidelines\]When evaluating let’s think according to following steps:s​t​e​p​se​v​a​l\{\\color\[rgb\]\{0,0,1\}\{\{steps\}\_\{eval\}\}\}\[Evaluation steps as outlined by the planner agent\]First explain your logic and then end the response with the following format strictly: "Score of Evaluation: \[score only\]"

### C\.3Moderator Prompt

Moderator PromptYou are a Debate Moderator\. Your task is to summarize the given generated opinion by the evaluators on an evaluation task pass it to a judge to rule about the evaluation task\. The evaluators have already evaluated the generated response for a chat, while considering chat’s context and evaluation criteria:The context of the chat:c​o​n​t​e​x​tc​h​a​t\{\\color\[rgb\]\{0,0,1\}\{context\}\_\{chat\}\}\[The messages from the conversation\]The generated response:r​e​s​p​o​n​s​e\{\\color\[rgb\]\{0,0,1\}response\}\[Response following the above conversation\]Evaluators has done this evaluation for the following evaluation criteria:e​v​a​lc​r​i​t​e​r​i​a\{\\color\[rgb\]\{0,0,1\}\{eval\}\_\{criteria\}\}\[Evaluation criteria along with its definition and guidelines\]Here is the evaluation of Evaluator 1:o​u​t​p​u​t1\{\\color\[rgb\]\{0,0,1\}\{output\}\_\{1\}\}\[Evaluation from Evaluator\-1\]And, here is the evaluation of Evaluator 2:o​u​t​p​u​t2\{\\color\[rgb\]\{0,0,1\}\{output\}\_\{2\}\}\[Evaluation from Evaluator\-2\]Summarize these opinions and so that the judge conduct their selection fairly\. "summary\_of\_evaluation": Summarized Reason for the evaluation\.

### C\.4Arbitrator Prompt

Arbitrator PromptYou are a judge who will decide about the result of an evaluation task\. Evaluators have already conducted the evaluation of the generated response for a chat, while considering chat’s context and evaluation criteria:The context of the chat:c​o​n​t​e​x​tc​h​a​t\{\\color\[rgb\]\{0,0,1\}\{context\}\_\{chat\}\}\[The messages from the conversation\]The response:r​e​s​p​o​n​s​e\{\\color\[rgb\]\{0,0,1\}response\}\[Response following the above conversation\]evaluation criteria:e​v​a​lc​r​i​t​e​r​i​a\{\\color\[rgb\]\{0,0,1\}\{eval\}\_\{criteria\}\}\[Evaluation criteria along with its definition and guidelines\]Consider these evaluations:s​u​m​m​a​r​ym​o​d​e​r​a​t​o​r\{\\color\[rgb\]\{0,0,1\}\{summary\}\_\{moderator\}\}\[Summary provided by the Moderator agent\]Select the best evaluation opinion given the responses generated by the evaluators\.End the response in the following format strictly: "Score of Evaluation: \[integer score only\]"

## Appendix DReal World Implementation

![Refer to caption](https://arxiv.org/html/2610.36389v1/figures/AB_test.png)Figure 6:Real\-world deployment of recommendation explanations usingMAREval\.
## Appendix EDifferent Backbone Model Performance

Table 6:MAREval score correlations across backbone models\. We report Pearson correlation \(ρ\\rho\) and Kendall–Tau \(τ\\tau\) with human judgments for Overall Score Criteria of TopicalChat Dataset\.Table[6](https://arxiv.org/html/2610.36389#A5.T6)compares the agreement between MAREval and human judgments across backbone models using both Pearson’sρ\\rhoand Kendall’sτ\\tau\. Results are consistent across metrics:*gemini\-2\.5\-pro*attains the strongest alignment \(ρ=0\.728\\rho=0\.728,τ=0\.632\\tau=0\.632\), followed by*Claude\-3\.5\-sonnet*\(ρ=0\.658\\rho=0\.658,τ=0\.564\\tau=0\.564\) and*gemini\-2\.0\-flash\-001*\(ρ=0\.601\\rho=0\.601,τ=0\.517\\tau=0\.517\), with*GPT 3\.5 Turbo*trailing \(ρ=0\.548\\rho=0\.548,τ=0\.478\\tau=0\.478\)\. The ordering indicates that backbone choice materially impacts evaluation reliability, and the higherτ\\taufor*gemini\-2\.5\-pro*suggests superior preservation of human\-preferred rankings\. While Pearson values exceed the corresponding Kendall coefficients \- as expected given their different sensitivities \- the relative gaps are stable across models, reinforcing the robustness of the observed ranking\.