Fantastic Metrics and How to Find Them: Meta-Evaluating Metrics for Multi-lingual Style-Personalized Generation

# Fantastic Metrics and How to Find Them: Meta-Evaluating Metrics for Multi-lingual Style-Personalized Generation
Source: [https://arxiv.org/html/2610.19166](https://arxiv.org/html/2610.19166)
###### Abstract

Meta\-evaluation of text generation metrics and LLM\-judges is primarily conducted using metric\-human correlations\. However, prior work has shown that correlation is not an appropriate estimate of a metric’s evaluation capability and can be unstable across evaluation settings, such as different datasets or languages\. Therefore, we propose a regression\-based meta\-evaluation of metrics that learns on calibrated metric outputs to predict human ratings\. We ground this measurement relative to a trivial constant\-mean baseline and human agreements to provide a comparable measure of metric capability across evaluation settings\. To test our meta\-evaluation framework, we developed a large\-scale multi\-lingual style\-personalized text generation \(SPTG\) benchmark, collecting multiple annotations across 10 topologically diverse languages and four evaluation dimensions, each with varying degrees of subjectivity\. We meta\-evaluated 32 metrics from three style evaluation paradigms \(string\-based, embedding\-based, and LLM\-judges\) and their ensembles over these 40 evaluation settings\. Our analysis revealed that even simple string\- and embedding\-based measurements can match strong LLM\-judges when calibrated to human ratings\. We also identified characteristics of human annotations that determine metric performance, as well as conditions under which a metric ensemble outperforms its individual constituents\.111Code and data is available at[https://github\.com/anubhav\-jangra/multilingual\-sptg\-eval](https://github.com/anubhav-jangra/multilingual-sptg-eval)\.

## 1Introduction

Evaluating text generation often requires multiple human judgments for inherently subjective qualities, where there is no single correct answer\([Plank, 2022](https://arxiv.org/html/2610.19166#bib.bib38);[Amidei et al\., 2019](https://arxiv.org/html/2610.19166#bib.bib39)\)\. As obtaining human annotations is expensive and time\-consuming, researchers rely on automatic metrics to validate a system’s performance on subjective generation tasks\. In standard practice, these automatic metrics are evaluated by comparing their agreement with human ratings, computed as correlation coefficients\. This is a widely adopted norm across several tasks including summarization\([Fabbri et al\., 2021](https://arxiv.org/html/2610.19166#bib.bib40);[Liu et al\., 2023](https://arxiv.org/html/2610.19166#bib.bib41)\), machine translation\([Freitag et al\., 2021](https://arxiv.org/html/2610.19166#bib.bib43)\), style transfer\([Mir et al\., 2019](https://arxiv.org/html/2610.19166#bib.bib44);[Briakou et al\., 2021a](https://arxiv.org/html/2610.19166#bib.bib1);[Ostheimer et al\., 2024](https://arxiv.org/html/2610.19166#bib.bib45);[Mukherjee et al\., 2025](https://arxiv.org/html/2610.19166#bib.bib5)\)and LLM\-as\-judges\([Chiang and Lee, 2023](https://arxiv.org/html/2610.19166#bib.bib42)\)\. However, correlation gives a poor estimate of a metric’s utility for subjective assessments due to three reasons: i\) it assumes a consistent relationship between metric and human scores, ii\) it is sensitive to ties and label skews\([Deutsch et al\., 2023](https://arxiv.org/html/2610.19166#bib.bib46)\), and iii\) it does not have a reference point for human agreement, making it incomparable across tasks, annotators, and datasets\. Prior work has shown that metric correlations can be unstable and misleading\([Novikova et al\., 2017](https://arxiv.org/html/2610.19166#bib.bib47);[Mathur et al\., 2020](https://arxiv.org/html/2610.19166#bib.bib48)\)\.

As an alternative to correlation\-driven meta\-evaluation of metrics, we instead propose a regression\-based meta\-evaluation framework to overcome the above\-mentioned limitations\. The proposed meta\-evaluation approach first calibrates each metric to the human rating scale, and then trains a regressor to predict the human consensus from the metric’s output\. The quality of the metric is then determined by the trained regressor’s performance on held\-out data, anchored between a constant predictor and the averaged agreement of human annotators\. This anchored quality estimate of the metric is comparable across evaluation settings, and more interpretable than the metric\-human correlations\. We apply this framework to style personalized text generation\([Jangra et al\., 2025](https://arxiv.org/html/2610.19166#bib.bib3)\), a task that requires a model to rewrite a source text in the style of a reference text\. We selected this task as it captures several evaluation dimensions with varying subjectivity \(factual correctness of source textvs\.stylistic representation of reference text\)\. As the metric behaviors and annotation properties may vary across languages\([Hada et al\., 2024b](https://arxiv.org/html/2610.19166#bib.bib49);[Hada et al\., 2024a](https://arxiv.org/html/2610.19166#bib.bib50);[Kocmi et al\., 2021](https://arxiv.org/html/2610.19166#bib.bib51)\), we conducted our study in 10 topologically diverse languages\. We curated a large\-scale multi\-lingual dataset for this task, collecting ratings from multiple native speakers across the 10 languages along four evaluation dimensions, namelyfactualaccuracy,styleaccuracy,leakageandusability\. We evaluated 32 metrics across a broad spectrum of evaluation paradigms, including string\-based metrics, semantic embeddings, style embeddings, and LLM judges\. Since the meta\-evaluation framework can accept several metrics as inputs, it can evaluate learned combinations of metrics\. Our systematic investigation of metric ensembles extends prior work\([Jangra et al\., 2025](https://arxiv.org/html/2610.19166#bib.bib3)\)by identifying when and how they improve the evaluation quality over constituent metrics\.

Our analysis showed that simple correlation\-based assessment hides several indicators of metric quality\. Across all four dimensions, the correlation between LLM\-judges and human annotations was significantly higher than that of other evaluation measures\. However, once calibrated to human ratings, much simpler embeddings and string\-based metrics were able to capture style at least as well as the strongest LLM\-judges\. The meta\-evaluation across languages revealed the metric’s performance depended on two properties of human annotations: the label skews and annotator agreement\. We also showed that metric ensembles improved the most on the style dimension \(∼2​x\{\\sim\}2\\mathrm\{x\}improvement over best individual metric\) where the constituent metrics were individually weak\. Our meta\-evaluation revealed no single metric to uniformly perform well across all languages or dimensions\.

Our work makes three key contributions: 1\) We proposed a regression\-based meta\-evaluation framework to evaluate agreement of evaluation metrics on subjective human annotations\. 2\) We curated a large\-scale style\-personalized text generation dataset, spanning 10 typologically diverse languages annotated by multiple native speakers across four evaluation dimensions\. 3\) We conducted an extensive meta\-evaluation study over 32 metrics over four evaluation paradigms, including several metric ensembles to examine their performance across different languages and evaluation dimensions\.

## 2Related Work

Meta\-evaluation of metrics\.Human evaluations are considered the gold\-standard in evaluating natural language generation models\([Celikyilmaz et al\., 2020](https://arxiv.org/html/2610.19166#bib.bib54)\), and are typically validated with their correlation with human ratings\([Fabbri et al\., 2021](https://arxiv.org/html/2610.19166#bib.bib40);[Mir et al\., 2019](https://arxiv.org/html/2610.19166#bib.bib44);[Chiang and Lee, 2023](https://arxiv.org/html/2610.19166#bib.bib42);[Freitag et al\., 2021](https://arxiv.org/html/2610.19166#bib.bib43)\)\. Correlation\-based meta\-evaluation can be a poor estimate of a metric’s utility for subjective human ratings due to their inconsistencies across datasets[Novikova et al\. \(2017\)](https://arxiv.org/html/2610.19166#bib.bib47), sensitivity to outliers\([Mathur et al\., 2020](https://arxiv.org/html/2610.19166#bib.bib48)\), high uncertainty\([Deutsch et al\., 2021](https://arxiv.org/html/2610.19166#bib.bib55)\), and difficulty processing ties in ranking\-based correlations\([Deutsch et al\., 2023](https://arxiv.org/html/2610.19166#bib.bib46)\)\.[Gehrmann et al\. \(2023\)](https://arxiv.org/html/2610.19166#bib.bib56)conducted a meta\-analysis of 66 language generation papers and found that metrics behave differently across datasets and systems, and that annotator disagreements can reflect genuine task subjectivity rather than noise\. They also noted that beyond machine translation, metrics are seldom designed for languages other than English\. Our work builds on these limitations in our meta\-evaluation, which examines metric performance across four distinct subjective evaluation dimensions across 10 languages, grounding it in annotator agreement and reliability\. Regression over human judgments has been previously explored to train new metrics such asBLEURT\([Sellam et al\., 2020](https://arxiv.org/html/2610.19166#bib.bib57)\)andCOMET\([Rei et al\., 2020](https://arxiv.org/html/2610.19166#bib.bib58)\), or to combine features to learn an ensemble of metrics\([Albrecht and Hwa, 2007](https://arxiv.org/html/2610.19166#bib.bib59);[Verga et al\., 2024](https://arxiv.org/html/2610.19166#bib.bib60)\)\. In contrast, we use regression as a means for meta\-evaluation, scoring calibrated metrics relative to a mean\-label baseline and human agreements\.

Evaluating style in text generation\.We evaluate our work on style\-personalized text generation \(SPTG\), a low\-resource\-constrained text generation task in which a text generation model rewrites a source text in the style of a reference style grounding\([Jangra et al\., 2025](https://arxiv.org/html/2610.19166#bib.bib3)\)\. We selected this task because it represents a genuine real\-world task in personalized text generation\([Salemi et al\., 2024](https://arxiv.org/html/2610.19166#bib.bib61);[Kumar et al\., 2024](https://arxiv.org/html/2610.19166#bib.bib62)\)and is inherently subjective\. Style transfer has long relied on several measurements to gauge the generation quality, from style strength, content preservation, and fluency[Krishna et al\. \(2020\)](https://arxiv.org/html/2610.19166#bib.bib64);[Protasov et al\. \(2026\)](https://arxiv.org/html/2610.19166#bib.bib4);[Mukherjee et al\. \(2025\)](https://arxiv.org/html/2610.19166#bib.bib5);[Protasov et al\. \(2026\)](https://arxiv.org/html/2610.19166#bib.bib4)\. As LLMs have become more fluent, we instead focus on four evaluation dimensions \-style accuracy: does the generated output reflect the reference style,factual accuracy: does it adhere to the original text,leakage: does it introduce unnecessary content from the style reference, andusability: is the generated output a useful rewrite of the original text\. These four evaluation dimensions collectively capture a wide spectrum of subjective measures in text generation\.

While most prior work in style transfer is limited to English\([Jin et al\., 2022](https://arxiv.org/html/2610.19166#bib.bib63)\), several works have studied style transfer in multi\-lingual settings in specific settings such as formality\([Krishna et al\., 2022](https://arxiv.org/html/2610.19166#bib.bib2);[Briakou et al\., 2021b](https://arxiv.org/html/2610.19166#bib.bib65)\)and detoxification\([Dementieva et al\., 2024](https://arxiv.org/html/2610.19166#bib.bib66)\)\. Recent efforts have also developed multi\-lingual style\-aware embeddings\([Qiu et al\., 2025](https://arxiv.org/html/2610.19166#bib.bib7);[Kim et al\., 2025](https://arxiv.org/html/2610.19166#bib.bib20)\), extending their English\-only predecessors\([Wegmann et al\., 2022](https://arxiv.org/html/2610.19166#bib.bib67);[Patel et al\., 2025](https://arxiv.org/html/2610.19166#bib.bib6)\)\. Prior work includes multi\-lingual meta\-evaluation benchmarks for LLM judges, such as MM\-Eval\([Son et al\., 2024](https://arxiv.org/html/2610.19166#bib.bib68)\)and the Cross\-Lingual Auto Evaluation suite\([Doddapaneni et al\., 2025](https://arxiv.org/html/2610.19166#bib.bib69)\)\. These benchmarks among other prior work\([Hada et al\., 2024b](https://arxiv.org/html/2610.19166#bib.bib49);[Hada et al\., 2024a](https://arxiv.org/html/2610.19166#bib.bib50)\)showed that the agreement of LLM\-judges with human ratings generally drop in low\-resource and non\-Latin\-script languages\. However, these benchmarks usually target general response quality or preference, which is usually under\-defined in text generation\([Gehrmann et al\., 2023](https://arxiv.org/html/2610.19166#bib.bib56)\)\. These works also limit the analysis to just evaluating LLM\-judges, often using correlations or pairwise accuracy\. To bridge this gap in prior work, we curated a multi\-lingual SPTG benchmark to meta\-evaluate metrics on the same task across 10 languages\. We also evaluate three other metric paradigms in addition to LLM\-judges, showing that, once calibrated, even simpler evaluation metrics can match LLM\-judgments on state\-of\-the\-art reasoning models\.

## 3Problem Definition

Let an originoobe a source content document and a referencerrbe a set of exemplars characterizing a target style\. Then a SPTG generator is a modelf:\(o,r\)↦gf:\(o,r\)\\mapsto gthat produces a candidateggre\-expressing the content ofooin the style ofrr\. For each languageL∈ℒL\\in\\mathcal\{L\}we are given an annotated corpus𝒞L=\{\(xL,i,𝐲L,i\)\}i=1NL\\mathcal\{C\}\_\{L\}\\;=\\;\\bigl\\\{\\,\(x\_\{L,i\},\\,\\mathbf\{y\}\_\{L,i\}\)\\,\\bigr\\\}\_\{i=1\}^\{N\_\{L\}\}\. Each instance pairs a reference and an origin with the outputs of two candidate generatorsfa,fb∈ℱf\_\{a\},f\_\{b\}\\in\\mathcal\{F\},xL,i=\(rL,i,oL,i,gL,ia,gL,ib\)x\_\{L,i\}\\;=\\;\\bigl\(\\,r\_\{L,i\},\\;o\_\{L,i\},\\;g^\{a\}\_\{L,i\},g^\{b\}\_\{L,i\}\)\. For each candidateg∈\{gL,ia,gL,ib\}g\\in\\\{g^\{a\}\_\{L,i\},g^\{b\}\_\{L,i\}\\\}, multiple humans judge the quality along a set of ordinal evaluation dimensionse∈ℰe\\in\\mathcal\{E\}, obtaining the ordinal labelsyL,ie,j​\(gL,i\)∈\{0,1,…,Ee\}y\_\{L,i\}^\{e,j\}\(g\_\{L,i\}\)\\in\\\{0,1,\.\.\.,E\_\{e\}\\\}from annotatorsj∈\{1,2,⋯K\}j\\in\\\{1,2,\\cdots K\\\}\. An automatic evaluation metric is a functionm:\(r,o,g\)↦ℝ,m∈ℳ,m:\\;\(r,o,g\)\\;\\mapsto\\;\\mathbb\{R\},m\\in\\mathcal\{M\},that returns a scalar quality estimate for a generationgggiven its reference and origin\. The goal is to find a meta\-evaluation functionalΨ\\Psithat scores how well a metric captures a human evaluation dimension and is comparable across different metrics and languages\. Ideally, this meta\-evaluation measure must capture how well a metricmreproduces the human assessment of a generatorff’s outputs on dimensioneefor a languageLL\(𝚿L\(e\)​\(m,f\)\\mathbf\{\\Psi\}^\{\(e\)\}\_\{L\}\\\!\\left\(m\\,;\\,f\\right\)\)\. Additionally, it should help quantify how much of a human constructmpm\_\{p\}recovers thatmqm\_\{q\}misses in languageLL\(ΨL\(e\)​\(mp\)−ΨL\(e\)​\(mq\)\\Psi^\{\(e\)\}\_\{L\}\\\!\\left\(m\_\{p\}\\right\)\-\\Psi^\{\(e\)\}\_\{L\}\\\!\\left\(m\_\{q\}\\right\)\)\.

## 4Multi\-lingual SPTG Dataset Curation

The development of our multilingual style\-personalized text generation benchmark comprised two stages\. First, we translated and corrected the English source texts\(r,o\)\(r,o\)to obtain their corresponding versions in the target languages\. Second, these multilingual source texts were used to generate candidate paraphrases, which were subsequently annotated on several evaluation dimensions to construct the final corpus\. We describe these two stages in detail in the following subsections\.

### 4\.1From English corpus to Multi\-lingual corpora

We started with 1,002 pairs of long\-form English text comprising anorigintextoothat is to be re\-expressed in the style of areferencetextrr\. The reference text was obtained from two domains, Redditr/ShortStories222[https://www\.kaggle\.com/datasets/trevordu/reddit\-short\-stories](https://www.kaggle.com/datasets/trevordu/reddit-short-stories)and Amazon product reviews for Digital Music and Subscription Boxes\([Hou et al\., 2026](https://arxiv.org/html/2610.19166#bib.bib37)\), whereas the origin text spanned Redditr/ShortStories, Wikipedia, IMDB, CNN/DailyMail, and Amazon reviews for a diverse content range \(refer to Table[3](https://arxiv.org/html/2610.19166#A2.T3)\)\. Using this English corpus as the basis for our multi\-lingual SPTG benchmark, we first machine\-translated the reference and source texts into the 36 target locales, and then conducted a human\-correction study to improve the quality of the translations\. Across the 36 locales, native annotators edited the machine translation of most items \(typically 80–99% of the 1,502 items per locale; Table[4](https://arxiv.org/html/2610.19166#A2.T4)\)\. For brevity, provide additional details about the translation study in Appendix[B](https://arxiv.org/html/2610.19166#A2)\.

Table 1:Dataset and annotation statistics for the curated multi\-lingual style personalized text generation benchmark\. For each evaluation dimension, we provide the aggregate human label mean \(standard deviation\), and the ordinal Krippendorffα\\alpha\. The arrows indicate the direction for improved generation quality in each dimension\. The final row provides an aggregate across all ten languages\.
### 4\.2Style Personalized Text for Individual Language corpus

Given the localized reference \(style exemplars;rr\) and origin \(content;oo\) for each instance, we generate two candidate paraphrases that re\-express the origin content in the grounding style, each accompanied by a free\-text rationale\. The generation models \(ff\) are drawn from a nine different models to avoid model\-specific biases from any single system333The candidate generation models includeddev\-gpt\-4o\-11\-20,dev\-gpt\-45\-preview,gpt\-41\-longco\-2025\-04\-14,gpt\-5\-mini,gpt\-51\-2025\-11\-13,gpt\-oss\-120b,anthropic\-claude\-opus\-4\-1,Phi\-4andQwen\-2\.5\-VL\-7B\. The researchers inspected the quality of generations from different models to select only the models that generated coherent outputs for each language, and for some instances, the same model was used to generate both paraphrases as well\.\. Obtaining two valid generations per instance enabled us to conduct our SPTG annotation study\. For each instance in a language corpus, each candidate paraphrase was independently rated by three human annotators along four ordinal dimensions:stylistic accuracy\(00–44; higher==better stylistic alignment with reference\),factual accuracy\(00–44; higher==more hallucination\),leakage\(00–22; higher==more information leaked from the reference\), andusability\(00–44; higher==less usable\)\. Factual accuracy captures content hallucination not present in the generation input\(o,r\)\(o,r\), whereas leakage captures the undesirable content introduced in the generation from the style referencerr\. Besides the paraphrase\-level annotations, each rater additionally annotated their pairwise preference between the two outputs \(paraphrase\-a,paraphrase\-b,neitherorboth\)\. Since the annotations for this second phase required much more effort than the translation correction study, we limited the annotations to 10 languages\. We leave further investigation into the other languages to future work\.

We provide the annotation statistics for all languages in Table[1](https://arxiv.org/html/2610.19166#S4.T1), including the distribution of human labels for the ordinal dimensions as well as the inter\-annotator agreements\. We observe a clear distinction in the annotations based on the subjectivity of the evaluation dimension\. The aggregate values of objective dimensions concentrate on the positive end, with factual ratings ranging from0\.58±1\.330\.58\\pm 1\.33and leakage at0\.39±0\.780\.39\\pm 0\.78\. On the contrary, the subjective dimension annotations are much more spread out on the ordinal scale, with style scores at1\.53±1\.371\.53\\pm 1\.37and usability at2\.47±1\.422\.47\\pm 1\.42\. Inter\-annotator reliability \(evaluated as the mean ordinal Krippendorff’sα\\alphaover languages\) is substantial across dimensions and also follows the expected objectivity gradient, with average IAA for leakage \(0\.85\) and factual \(0\.79\) higher than for style \(0\.76\) and usability \(0\.75\)\. The annotator agreement confirms that agreement on the subjective dimensions is genuine but not uniform either across languages or across specific instances\. Our final corpus comprises 10,038 instances, spanning 10 languages, each with two candidate paraphrases, three\-annotator ratings on four dimensions, and one side\-by\-side pairwise preference between the two paraphrases\.

## 5Proposed Methodology

We first describe the statistical diagnostics we conduct to assess the evaluation metrics’ capability relative to human performance, following typical evaluation strategies in prior work \(Section[5\.1](https://arxiv.org/html/2610.19166#S5.SS1)\)\. Then, in Section[5\.2](https://arxiv.org/html/2610.19166#S5.SS2), we describe our principled regression\-based meta\-metric, which provides a normalized score against a trivial baseline and human\-annotation performance\.

### 5\.1Statistical Diagnostics

Pointwise alignment\.To identify how good evaluation metrics are aligned with the human judgments; specifically their ordinal pointwise judgments for each of the generated paraphrase, we use Spearman’sρ\\rhoto measure the metric\-human alignment\. For an assessment over an evaluation dimensioneefor languageLL, we aggregate the human labels as the mean of the three annotator ratings,y¯L\(e\)​\(g\)\\bar\{y\}^\{\(e\)\}\_\{L\}\(g\), for a candidate paraphrasegg, to obtain the target rankings for the generations\. This correlation is equivalent to a standardized ordinary least squares \(OLS\) fit over ranked values\.

Pairwise alignment\.While pointwise evaluation measures how well tuned the metrics are against the human judgments over an evaluation dimension for a language; the pairwise alignment evaluates how good the metrics are at being able to distinguish between two texts within the same context\. To evaluate pairwise alignment of a metric with the human annotators, we used the difference in average ordinal ratings of the two paraphrases as the target reference444Since our pairwise annotations were limited to a single higher\-order preference of generation, we chose the difference in ordinal ratings as the pairwise ground truth\.; and used Cohen’sκ\\kappato compute the alignment\. Since we computed this alignment on only the decisive instances555For pairwise instance selection, an annotation pair was considered decisive if the pointwise difference in the aggregate human labels differed by at least 2 points on that dimension, and at least 2 of the 3 annotators individually rank the pair in the same direction as the aggregate labels\., dropping the instances yielded a smaller subset compared to pointwise evaluation across the four dimensions: style \(527\), factual \(251\), leakage \(269\), and usability \(558\)\.

Relative agreement across evaluation metrics\.This experiment investigates how much of a human construct does a metricmpm\_\{p\}recover, that another metricmqm\_\{q\}misses on an evaluation dimensioneefor languageLL\. We fit both metrics jointly against the ranked human labels, and measure the incremental variancempm\_\{p\}explains beyondmqm\_\{q\},i\.e\.,the gain inR2R^\{2\}from addingmpm\_\{p\}to a model that already containsmqm\_\{q\}:

Δ​Rp\|q2=R2​\(mp,mq\)−R2​\(mq\)\\Delta R^\{2\}\_\{p\\mid q\}\\;=\\;R^\{2\}\(m\_\{p\},m\_\{q\}\)\-R^\{2\}\(m\_\{q\}\)\(1\)
Since ordinary least squares regression finds the best possible linear combination of the metrics,R2​\(mp,mq\)R^\{2\}\(m\_\{p\},m\_\{q\}\)is the highest agreement with human labels that the two metrics can reach together\([Cohen et al\., 2013](https://arxiv.org/html/2610.19166#bib.bib21)\)\. Therefore,Δ​Rp\|q2\\Delta R^\{2\}\_\{p\\mid q\}is the extra agreement that onlympm\_\{p\}provides \(always≥\\geq0\)\. It is the squared semi\-partial correlation that characterizes the association betweeny¯\\bar\{y\}and the part ofmpm\_\{p\}that is uncorrelated withmqm\_\{q\}\. Symmetrically,Δ​Rq\|p2\\Delta R^\{2\}\_\{q\\mid p\}ismqm\_\{q\}’s unique contribution, and the remainderR2​\(mp\)\+R2​\(mq\)−R2​\(mp,mq\)R^\{2\}\(m\_\{p\}\)\+R^\{2\}\(m\_\{q\}\)\-R^\{2\}\(m\_\{p\},m\_\{q\}\)is the part shared by both\([Seibold and McPHEE, 1979](https://arxiv.org/html/2610.19166#bib.bib22)\)\. We limit this investigation to pointwise evaluation, for which the metrics assign a score to each item independently, obtainingmpm\_\{p\}andmqm\_\{q\}on the same per\-item units asy¯\\bar\{y\}\.

### 5\.2Meta\-metric: Predicting human labels from metric outputs

To estimate the capabilities of different evaluation metrics across languages and evaluation dimensions, we propose a principled meta\-evaluation of metrics by fitting a regression model to human labels derived from metric predictions\. Formally, for a specific languageLLand evaluation dimensionee, for each instance, we have two paraphrased generationsgL,ia,gL,ib\{g^\{a\}\_\{L,i\},g^\{b\}\_\{L,i\}\}annotated by three annotators\. To train our meta\-evaluation regressor, we aggregated the annotations using a reliability weighted strategy to obtain the regression target labels:y¯L,ie​\(gL,ia\)=∑j=13wje​\.​yL,ie,j​\(gL,ia\)\\bar\{y\}^\{e\}\_\{L,i\}\(g\_\{L,i\}^\{a\}\)=\\sum\_\{j=1\}^\{3\}w^\{e\}\_\{j\}\\text\{ \. \}y^\{e,j\}\_\{L,i\}\(g\_\{L,i\}^\{a\}\), wherewL,je≥0w^\{e\}\_\{L,j\}\\geq 0reflects annotatorjj’s agreement with the others, and the weights sum to one\. Individual annotator reliability was computed using Spearman’s correlation with a leave\-one\-out strategy \(Figure[4](https://arxiv.org/html/2610.19166#A2.F4)\)\.

Since metrics produce scores on different scales \(embeddings between\[−1,1\]\[\-1,1\], LLM\-judges in\{0,1,…​Ee\}\\\{0,1,\\dots E\_\{e\}\\\}\), we first map each score onto the human label scale using a monotone calibration, preserving the rankings within the candidates666LLM\-judges in this study are prompted to follow the ordinal scales that were used to collect the human annotation, and therefore are left unchanged in this processing step\.\. The meta\-metric of any arbitrary metricmmis then a regressor trained to predicty¯\\bar\{y\}frommm’s calibrated output\. This arbitrary metric can be a single metric, a specific prompt used for an LLM, or an ensemble of metrics\. We evaluated 16 regression models and selected the best\-performing regression approach, extremely randomized trees\([Geurts et al\., 2006](https://arxiv.org/html/2610.19166#bib.bib23)\), across all the 4×\\times10 dimension\-language combinations \(see Appendix[A](https://arxiv.org/html/2610.19166#A1)for further details\)\.

We evaluated each meta\-metric with the same 5\-fold cross\-validation, repeated five times\. The folds were split by instance, preventing data contamination from the two generations corresponding to the same instance\. The calibration and regression fitting were done only on the training folds, using both candidates per instance\. We measured the mean absolute error \(MAE\) of the held\-out predictions\. The MAE is not comparable across evaluation settings because it depends on the scale and label skew\. Therefore, we normalized the MAE scores on a trivial metric baseline that always predicts the mean label in the training fold \(MAE0\\mathrm\{MAE\}\_\{0\}\), and the oracle error of each annotator in predicting the weighted consensus of the other two, averaged over the three annotators \(MAEH\\mathrm\{MAE\}\_\{H\}\)\. The absolute metric quality for languageLLon an evaluation dimensioneeis thus the fraction of this gap metricmmcloses:

𝚿L\(e\)​\(m\)=MAE0−MAEmMAE0−MAEH,\\mathbf\{\\Psi\}^\{\(e\)\}\_\{L\}\(m\\,\)=\\frac\{\\mathrm\{MAE\}\_\{0\}\-\\mathrm\{MAE\}\_\{m\}\}\{\\mathrm\{MAE\}\_\{0\}\-\\mathrm\{MAE\}\_\{H\}\},\(2\)
A metric quality of 0 indicates that the metric does not add beyond the trivial chance baseline, and 1 means that it agrees with the consensus as well as a single annotator\. Note that values above 1 are possible when one annotator is noisier than the consensus\.

## 6Experimental Settings

### 6\.1Off\-the\-shelf Evaluation Metrics

We meta\-evaluate metrics across four paradigms, ranging from simple string\-based overlap to model\-based judges\. The reference\-based metrics \(first three paradigms\) were computed against each signal’s corresponding reference: the style referencerrfor stylistic accuracy and leakage, and the content sourceoofor factual accuracy\. LLM\-based judges, on the other hand, were provided with bothrrandoo, along with the SPTG outputgg, for all four evaluation dimensions\. We also evaluated an ensemble of metrics to assess compositional improvements across diverse metric paradigms \(denoted asmXm\_\{X\}\)\.

String\-based\.These metrics comprise surface\-level comparisons, including normalizedLevenshteinedit distance\([Lcvenshtcin, 1966](https://arxiv.org/html/2610.19166#bib.bib19)\), character33\-gram cosine similarity, verbatim overlap, length ratio, punctuation\-rate difference, and token\-entropy difference\. These metrics are inexpensive, language\-agnostic, and cover all ten languages\.

Text embeddings \(semantic and style\)\.To measure semantic information overlap, we use the language\-agnosticBERTembedding model \(LaBSE;[Feng et al\., 2022](https://arxiv.org/html/2610.19166#bib.bib18)\) as a semantic multi\-lingual sentence embedding model\. We also used embedding models specifically trained to encode authoring style rather than content, in which the distance in latent space reflects stylistic rather than semantic differences\. For English, we usedWegmann\([Wegmann et al\., 2022](https://arxiv.org/html/2610.19166#bib.bib67)\)andStyleDistance\([Patel et al\., 2025](https://arxiv.org/html/2610.19166#bib.bib6)\)embeddings\. We additionally evaluated French \(fr\-FR\), Japanese \(ja\-JP\), and Korean \(ko\-KR\) on its multi\-lingual counterpart,mStyleDistance\([Qiu et al\., 2025](https://arxiv.org/html/2610.19166#bib.bib7)\), and on French \(fr\-FR\), Japanese \(ja\-JP\) and Greek \(el\-GR\) using another multilingual style representation model,mStyleRepr\([Kim et al\., 2025](https://arxiv.org/html/2610.19166#bib.bib20)\)\. We used cosine distance to measure the semantic and style distance between the SPTG outputs \(gg\) and the reference \(rroroo\) in the latent space\.

LLM\-as\-a\-judge\.We evaluated a wide spread of models, spanning several proprietary and open\-weight LLMs\. The proprietary LLMs includeGPT\-5\.5andGPT\-5\.4\-mini\( , \),Claude\-Opus\-4\.8,Sonnet\-5, andHaiku\-4\.5\( , \),Gemini\-3\.1\-ProandFlash\-Lite\([Team et al\., 2023](https://arxiv.org/html/2610.19166#bib.bib17)\); while the opensource models includeAya\-8B\([Dang et al\., 2024](https://arxiv.org/html/2610.19166#bib.bib8)\),Gemma\-4\(12B,31B,E4B\)\([Team et al\., 2026](https://arxiv.org/html/2610.19166#bib.bib9)\),GPT\-OSS\-20B\([Agarwal et al\., 2025](https://arxiv.org/html/2610.19166#bib.bib10)\),Llama\-3\.1\-8BandLlama\-3\.2\-3B\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.19166#bib.bib11)\),Mistral\-7BandMistral\-Nemo\-12B\( , \),OLMo\-3\-7B\([Olmo et al\., 2025](https://arxiv.org/html/2610.19166#bib.bib12)\),Phi\-4\-14B\([Abdin et al\., 2024](https://arxiv.org/html/2610.19166#bib.bib13)\),Qwen\-2\.5\-7B\([Qwen et al\., 2025](https://arxiv.org/html/2610.19166#bib.bib15)\),Qwen\-3\-8B\([Yang et al\., 2025](https://arxiv.org/html/2610.19166#bib.bib14)\), andQwen\-3\.5\(4B/9B/27B\)\([Team, 2026](https://arxiv.org/html/2610.19166#bib.bib16)\)\. Unlike the other reference\-based metrics, the judges score all four dimensions, including usability, for which no meaningful reference distance exists\. We provide the prompts for pointwise and pairwise evaluations in Appendix[D](https://arxiv.org/html/2610.19166#A4)\. All the open\-source models were run on 40Gb NVIDIA A100 GPUs\.

## 7Findings

### 7\.1Statistical Diagnostics of Evaluation Metrics

Table 2:Pairwise and pointwise statistical diagnostics over the four evaluation dimensions and their overall aggregate\. The evaluation is performed on the test split of the dataset, averaged over human annotations from 10 languages\. The pointwise alignment is computed with Spearman’sρ\\rho, measured against the mean ordinal human ratings for each paraphrase on the respective evaluation dimension\. Whereas the pairwise alignment is measured with Cohen’sκ\\kappa, against the sign of the difference in ordinal ratings for the two paraphrases, dropping the instances where the difference is zero\. This table includes only results for the best\-performing metrics in our meta\-evaluation \(Figure[2](https://arxiv.org/html/2610.19166#S7.F2)\); for pointwise and pairwise results across all metrics, refer to Table[6](https://arxiv.org/html/2610.19166#A3.T6)\.Pointwise alignmentTable[2](https://arxiv.org/html/2610.19166#S7.T2)reports the Spearmanρ\\rhobetween each metric and the aggregate human label for selective metrics \(refer to Table[6](https://arxiv.org/html/2610.19166#A3.T6)for results on all the metrics\)\. The overall alignment between the metrics and the human annotators remained modest, with the best metric achieving a mean correlation of0\.560\.56\. The alignment varied sharply by dimensions, asleakagewas the most tractable dimension \(highestρ\\rho=0\.760\.76\) whereas stylistic accuracy was the hardest dimension for every metric \(highestρ\\rho=0\.400\.40\)\.

Among the reference\-based metrics,LaBSEwas the strongest metric \(overall0\.350\.35\), slightly outperforming the simpler metrics \(Levenshtein,Verbatim, overall∼0\.33\{\\sim\}0\.33\)\. On the other hand, the style embeddings performed comparatively well on thestyledimension but collapsed to near\-zero correlations with factual accuracy and leakage\. For leakage, the reference\-based metrics were computed as the distance between the generationggand the content originoo\(exceptVerbatim, which was similarity\-based\) and therefore showed negative correlations with human ratings\. LLM\-judges, on the other hand, significantly outperformed the reference\-based metrics, with the strongest overall alignment \(∼0\.53\{\\sim\}0\.53–0\.560\.56\) obtained by the open\-weightGemma4\-31B\(0\.520\.52\) andGeminiFlashLite\-3\.1\(0\.530\.53\) for the proprietary LLMs\.

Pairwise alignment\.In contrast, the pairwise agreements were higher than the pointwise alignment across the board\. However, the comparison between the two evaluation settings is not fair, as the pointwise evaluation was conducted over the entire test set, whereas the pairwise comparison only spanned the instances with a distinctly superior paraphrase, encompassing∼250\\sim 250\-530530instances across the dimensions\. Nonetheless, the pairwise alignment trends broadly aligned with the pointwise results\. The ordering of the evaluation dimensions was preserved from the pointwise evaluation and was even sharper\. Leakage and factual accuracy were close to the human ceiling for strong LLM\-judges \(κ≥0\.90\\kappa\\geq 0\.90\), whereas stylistic accuracy remained the hardest dimension for both views \(κ≤0\.52\\kappa\\leq 0\.52\)\.

Figure 1:The aggregateΔR2p\|\.\\Delta R^\{2\}\_\{p\|\.\}\(Eq\.[1](https://arxiv.org/html/2610.19166#S5.E1)\) for selective evaluation metrics over the four evaluation dimensions against the rest of the evaluation metrics \(excluding style embeddings due to their limited language\)\. For the results on all the metrics, refer to Figure[6](https://arxiv.org/html/2610.19166#A3.F6)in Appendix[C](https://arxiv.org/html/2610.19166#A3)\.Inter\-metric variance decomposition\.Beyond absolute alignment against human annotators, we investigated how much unique information each metric contributes relative to other metrics\. In Figure[1](https://arxiv.org/html/2610.19166#S7.F1), we report the averageΔ​R2\\Delta R^\{2\}for the corresponding metrics in Table[2](https://arxiv.org/html/2610.19166#S7.T2)across all dimensions, capturing the variance it adds relative to the pool of all other metrics\. Only the LLM\-judge metrics had a net positive contribution across all dimensions, while the surface\-level string\-based metrics and smaller open models were net redundant, providing negligible additional information to the metric pool\. Aligning with the pointwise and pairwise alignment trends, the spread is widest for the leakage dimension \(up to\+0\.35\+0\.35\) and near\-collapse on stylistic accuracy across all metrics, indicating that style is a dimension on which metrics differ least in the unique information they carry\. Comparing the variance across metric paradigms, we observed that proprietary judges added substantially over every other paradigm \(\+0\.22\+0\.22to0\.370\.37in leakage\), while the reverse added minimally \(∼\+0\.02\\sim\+0\.02\)\. Among the non\-LLM metrics,LaBSEstood out as an exception in the leakage dimension, where it added a non\-trivial\+0\.20\+0\.20over the open\-weight judges\. We provide the pairwiseΔ​Rmp\|mq2\\Delta R^\{2\}\_\{m\_\{p\}\|m\_\{q\}\}and paradigm\-level aggregates in Appendix[C](https://arxiv.org/html/2610.19166#A3)\(Figures[7](https://arxiv.org/html/2610.19166#A3.F7)and[8](https://arxiv.org/html/2610.19166#A3.F8)\)\.

### 7\.2Meta\-evaluation of metrics

How does calibration affect the performance of individual metrics?Our meta\-evaluation revealed that the metric performance varied significantly across evaluation dimensions \(Figure[2](https://arxiv.org/html/2610.19166#S7.F2)\)\. Across all evaluation dimensions, best\-performing open\-weight models performed equivalently to proprietary models, whereas the small open models scored up toΨ=0\.15\\Psi=0\.15on average \(e\.g\.,Aya\-8BandMistral\-7B\)\. On content\-based evaluation dimensions \(factualandleakage\), reference\-based metrics performed competitively against the best\-performing LLM\-judges\. Whereas learningstyleaccuracy from a single LLM\-judge output was the hardest, even the strongest LLMs underperformed simpler string\-based metrics andLaBSE\. This finding contradicts the metric rankings obtained from the aggregate correlation ofLaBSEin Table[2](https://arxiv.org/html/2610.19166#S7.T2)\. While the per\-language correlations partially explain the gap, asLaBSEcorrelates positively with style in most languages but strongly negatively in Frisian and Japanese, canceling out its signal in the averaged results \(Fig\.[5](https://arxiv.org/html/2610.19166#A3.F5), top left\); general trends in correlation still indicated it underperformed competitive LLM\-judges\. As the calibration for each language\-dimension pair learns each metric’s direction and scale separately, the meta\-evaluation recovers the signal to assess the metric’s predictive capabilities\. While this shows that reference\-based metrics can match LLM judgments across the evaluated dimensions, they still require human annotations to calibrate the measurements for general applicability\.

Figure 2:Meta\-evaluation of metrics evaluated using the normalized quality scoreΨ\\Psi\(Eq\.[2](https://arxiv.org/html/2610.19166#S5.E2)\), averaged over 10 languages\.Ψe=0\\Psi^\{e\}=0indicates the chance performance, whereasΨ=1\\Psi=1corresponds to the average human performance\. We report the results for the best\-performing individual metrics in the corresponding evaluation paradigm as well as several combinations of ensemble metrics \(ρ\\rho\)\. We provide the detailed results for all the metrics in Tables[7](https://arxiv.org/html/2610.19166#A3.T7)and[8](https://arxiv.org/html/2610.19166#A3.T8)in Appendix[C](https://arxiv.org/html/2610.19166#A3)\.How do annotations affect meta\-evaluation across languages?For individual metrics, the performance varied at least as much across languages as across different metrics\. Overall, English and French scored the lowest, while Japanese, Greek, and Frisian scored highest \(Table[8](https://arxiv.org/html/2610.19166#A3.T8)\)\. These trends could not be justified by a language’s resource availability for model training, nor by its scripts\. We found two properties of human labels that explained most of this variation\.Firstly, the spread of the labels per dimension determined the prediction space\. For the factual and leakage dimensions,Ψ\\Psiwas the lowest, with most candidates receiving the same ratings\. For instance, the factual and leakage scores for Bengali, Korean, French, and English were lower across all metrics, as their labels were skewed toward the best rating for 84\-94% of candidates\.Secondly, because we normalized the MAE loss of the meta\-evaluation regressor to the averaged inter\-annotator predictability \(Section[5\.2](https://arxiv.org/html/2610.19166#S5.SS2)\), the label spread determined how strict the human reference was for the normalized scores\. On style and usability,Ψ\\Psiwas lower for languages where the annotators agreed more closely\. There was a strong negative correlation \(Spearmanρ=−0\.71\\rho=\-0\.71\) between the human agreement \(Krippendorff’sα\\alpha; Table[1](https://arxiv.org/html/2610.19166#S4.T1)\) and the normalized scores\. English performance was the worst on all four dimensions because it combined both effects, with the highest annotator agreement and the least label spread\. Metrics performed poorly on factual consistency and leakage in Korean, even though the annotators had the lowest agreement and the most skewed labels, which relaxed the gap between the human and trivial baselines for normalization\. Conversely,Ψjau​s​a​b​i​l​i​t​y\\Psi\_\{\\text\{ja\}\}^\{usability\}was above one, as one annotator \(A2\) had high disagreement with the other two \(Figure[4](https://arxiv.org/html/2610.19166#A2.F4)\), implying that the metrics matched the annotator consensus more closely than one of the three annotators and not because the evaluation in the language was easy\.

When are ensemble metrics most effective?Every metric ensemble outperformed its constituents, with the largest gains observed for the weakest evaluation dimension, style accuracy \(Figure[2](https://arxiv.org/html/2610.19166#S7.F2)\)\. Ensemble of all metrics obtainedΨs​t​y​l​e​\(ρa​l​l\)=0\.78\\Psi^\{style\}\(\\rho\_\{all\}\)=0\.78, almost twice as much as the best performing single metricΨs​t​y​l​e​\(LaBSEorVerbatim\)=0\.43\\Psi^\{style\}\(\\texttt\{LaBSE\}\\text\{ or \}\\texttt\{Verbatim\}\)=0\.43\. Usability showed a similar but less pronounced trend\. On content\-based evaluation dimensions \(factual consistency and leakage\), where the individual metrics already performed better, ensembling metrics did not yield substantial gains\. We identified from our variance decomposition diagnostics \(Figure[6](https://arxiv.org/html/2610.19166#A3.F6)\) that no individual metric contributes more than∼0\.1\{\\sim\}0\.1unique variance on style\. However, combining them yielded the largest ensemble gain\. This indicates that the style signal is thinly spread across multiple metrics, and can only be recovered by combining them\. In contrast, for content\-based dimensions, this signal is concentrated in a few metrics; hence, ensembling them does not yield significant gains\. Examining the ensemble constituents forma​l​lm\_\{all\}also aligned with this trend, as a single metric typically contributed more than all other metrics \(Figure[9](https://arxiv.org/html/2610.19166#A3.F9), Appendix[C](https://arxiv.org/html/2610.19166#A3)\)\. This is more pronounced in leakage, where the ensemble effectively behaves similarly to a single constituent metric, whereas for the style dimension, the contribution is comparatively more spread out, leading to significant improvements over individual metrics\. For all languages, no constituent contributes universally to the ensemble’s performance, except for usability, where the ensemble relies on the same judges more often than others\.

## 8Conclusion

In this work, we proposed a regression\-based meta\-evaluation to assess metrics for subjective tasks, such as style evaluation\. For the meta\-evaluation, we curated a large\-scale multi\-lingual dataset for style\-personalized text generation, collecting annotations from multiple native speakers across 10 languages along four evaluation dimensions\. We argued against using uncalibrated metric correlations with human ratings and showed that raw correlations masked several indicators of metric quality\. We analyzed the meta\-metric across languages and found its assessment to depend on two properties of human annotations: their label skews and annotator agreement\. Finally, we examined the role constituent metrics play within metric ensembles\. Ensembles improved most on style, the dimension where the signal was spread thinly across many metrics\. The best\-performing metrics varied across languages, with no single metric important for all languages or evaluation dimensions\. Together, these findings suggest that metric evaluation for subjective text generation should align annotation properties with metric scores, and that learnt combinations of metrics offer a more faithful measure of metric capability\.

### AI use statement

We have used generative AI tools to aid in writing the code to conduct the experiments, to serve as synthetic data generators, and to generate the figures and tables\. We have not used generative AI for ideation, literature review, or end\-to\-end paper or section writing\. We have reviewed all AI\-assisted work, conducted extensive human evaluation studies to improve the quality of synthetic data, validated the code used for experimentation, cross\-checked the illustration artifacts, and ensured that all authors had full agency over the paper\-writing process\. We take responsibility for the final content of this work, including text, claims, or artifacts produced with the aid of generative AI\.

## References

- Abdinet al\.\(2024\)M\. Abdin, J\. Aneja, H\. Behl, S\. Bubeck, R\. Eldan, S\. Gunasekar, M\. Harrison, R\. J\. Hewett, M\. Javaheripi, P\. Kauffmann,et al\.Phi\-4 technical report\.arXiv preprint arXiv:2412\.08905\.Cited by:[§6\.1](https://arxiv.org/html/2610.19166#S6.SS1.p4.1)\.
- Agarwalet al\.\(2025\)S\. Agarwal, L\. Ahmad, J\. Ai, S\. Altman, A\. Applebaum, E\. Arbus, R\. K\. Arora, Y\. Bai, B\. Baker, H\. Bao,et al\.Gpt\-oss\-120b & gpt\-oss\-20b model card\.arXiv preprint arXiv:2508\.10925\.Cited by:[§6\.1](https://arxiv.org/html/2610.19166#S6.SS1.p4.1)\.
- Albrecht and Hwa \(2007\)J\. Albrecht and R\. HwaA re\-examination of machine learning approaches for sentence\-level mt evaluation\.InProceedings of the 45th Annual Meeting of the Association of Computational Linguistics,pp\. 880–887\.Cited by:[§2](https://arxiv.org/html/2610.19166#S2.p1.1)\.
- Altman \(1992\)N\. S\. AltmanAn introduction to kernel and nearest\-neighbor nonparametric regression\.The American Statistician46\(3\),pp\. 175–185\.External Links:[Document](https://dx.doi.org/10.1080/00031305.1992.10475879),[Link](https://www.tandfonline.com/doi/abs/10.1080/00031305.1992.10475879),https://www\.tandfonline\.com/doi/pdf/10\.1080/00031305\.1992\.10475879Cited by:[Appendix A](https://arxiv.org/html/2610.19166#A1.p1.1)\.
- Amideiet al\.\(2019\)J\. Amidei, P\. Piwek, and A\. WillisAgreement is overrated: a plea for correlation to assess human evaluation reliability\.InProceedings of the 12th International Conference on Natural Language Generation,pp\. 344–354\.Cited by:[§1](https://arxiv.org/html/2610.19166#S1.p1.1)\.
- Breiman \(2001\)L\. BreimanRandom forests\.Machine learning45\(1\),pp\. 5–32\.Cited by:[Appendix A](https://arxiv.org/html/2610.19166#A1.p1.1)\.
- Briakouet al\.\(2021a\)E\. Briakou, S\. Agrawal, J\. Tetreault, and M\. CarpuatEvaluating the evaluation metrics for style transfer: a case study in multilingual formality transfer\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 1321–1336\.Cited by:[§1](https://arxiv.org/html/2610.19166#S1.p1.1)\.
- Briakouet al\.\(2021b\)E\. Briakou, D\. Lu, K\. Zhang, and J\. TetreaultOlá, bonjour, salve\! xformal: a benchmark for multilingual formality style transfer\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 3199–3216\.Cited by:[§2](https://arxiv.org/html/2610.19166#S2.p3.1)\.
- Celikyilmazet al\.\(2020\)A\. Celikyilmaz, E\. Clark, and J\. GaoEvaluation of text generation: a survey\.arXiv preprint arXiv:2006\.14799\.Cited by:[§2](https://arxiv.org/html/2610.19166#S2.p1.1)\.
- Chiang and Lee \(2023\)C\. Chiang and H\. LeeCan large language models be an alternative to human evaluations?\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 15607–15631\.Cited by:[§1](https://arxiv.org/html/2610.19166#S1.p1.1),[§2](https://arxiv.org/html/2610.19166#S2.p1.1)\.
- Cohenet al\.\(2013\)J\. Cohen, P\. Cohen, S\. G\. West, and L\. S\. AikenApplied multiple regression/correlation analysis for the behavioral sciences\.routledge\.Cited by:[§5\.1](https://arxiv.org/html/2610.19166#S5.SS1.p4.1)\.
- Danget al\.\(2024\)J\. Dang, S\. Singh, D\. D’souza, A\. Ahmadian, A\. Salamanca, M\. Smith, A\. Peppin, S\. Hong, M\. Govindassamy, T\. Zhao,et al\.Aya expanse: combining research breakthroughs for a new multilingual frontier\.arXiv preprint arXiv:2412\.04261\.Cited by:[§6\.1](https://arxiv.org/html/2610.19166#S6.SS1.p4.1)\.
- Dementievaet al\.\(2024\)D\. Dementieva, N\. Babakov, and A\. PanchenkoMultiParaDetox: extending text detoxification with parallel data to new languages\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 2: Short Papers\),pp\. 124–140\.Cited by:[§2](https://arxiv.org/html/2610.19166#S2.p3.1)\.
- Deutschet al\.\(2021\)D\. Deutsch, R\. Dror, and D\. RothA statistical analysis of summarization evaluation metrics using resampling methods\.Transactions of the Association for Computational Linguistics9,pp\. 1132–1146\.Cited by:[§2](https://arxiv.org/html/2610.19166#S2.p1.1)\.
- Deutschet al\.\(2023\)D\. Deutsch, G\. Foster, and M\. FreitagTies matter: meta\-evaluating modern metrics with pairwise accuracy and tie calibration\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 12914–12929\.Cited by:[§1](https://arxiv.org/html/2610.19166#S1.p1.1),[§2](https://arxiv.org/html/2610.19166#S2.p1.1)\.
- Doddapaneniet al\.\(2025\)S\. Doddapaneni, M\. S\. U\. R\. Khan, D\. Venkatesh, R\. Dabre, A\. Kunchukuttan, and M\. M\. KhapraCross\-lingual auto evaluation for assessing multilingual llms\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 29297–29329\.Cited by:[§2](https://arxiv.org/html/2610.19166#S2.p3.1)\.
- Druckeret al\.\(1996\)H\. Drucker, C\. J\. Burges, L\. Kaufman, A\. Smola, and V\. VapnikSupport vector regression machines\.Advances in neural information processing systems9\.Cited by:[Appendix A](https://arxiv.org/html/2610.19166#A1.p1.1)\.
- Fabbriet al\.\(2021\)A\. R\. Fabbri, W\. Kryściński, B\. McCann, C\. Xiong, R\. Socher, and D\. RadevSummeval: re\-evaluating summarization evaluation\.Transactions of the Association for Computational Linguistics9,pp\. 391–409\.Cited by:[§1](https://arxiv.org/html/2610.19166#S1.p1.1),[§2](https://arxiv.org/html/2610.19166#S2.p1.1)\.
- Fenget al\.\(2022\)F\. Feng, Y\. Yang, D\. Cer, N\. Arivazhagan, and W\. WangLanguage\-agnostic bert sentence embedding\.InProceedings of the 60th annual meeting of the association for computational linguistics \(volume 1: Long papers\),pp\. 878–891\.Cited by:[§6\.1](https://arxiv.org/html/2610.19166#S6.SS1.p3.1)\.
- Freitaget al\.\(2021\)M\. Freitag, G\. Foster, D\. Grangier, V\. Ratnakar, Q\. Tan, and W\. MachereyExperts, errors, and context: a large\-scale study of human evaluation for machine translation\.Transactions of the Association for Computational Linguistics9,pp\. 1460–1474\.Cited by:[§1](https://arxiv.org/html/2610.19166#S1.p1.1),[§2](https://arxiv.org/html/2610.19166#S2.p1.1)\.
- Friedman \(2001\)J\. H\. FriedmanGreedy function approximation: a gradient boosting machine\.Annals of statistics,pp\. 1189–1232\.Cited by:[Appendix A](https://arxiv.org/html/2610.19166#A1.p1.1)\.
- Gehrmannet al\.\(2023\)S\. Gehrmann, E\. Clark, and T\. SellamRepairing the cracked foundation: a survey of obstacles in evaluation practices for generated text\.Journal of Artificial Intelligence Research77,pp\. 103–166\.Cited by:[§2](https://arxiv.org/html/2610.19166#S2.p1.1),[§2](https://arxiv.org/html/2610.19166#S2.p3.1)\.
- Geurtset al\.\(2006\)P\. Geurts, D\. Ernst, and L\. WehenkelExtremely randomized trees\.Machine learning63\(1\),pp\. 3–42\.Cited by:[Appendix A](https://arxiv.org/html/2610.19166#A1.p1.1),[§5\.2](https://arxiv.org/html/2610.19166#S5.SS2.p2.1)\.
- Golubet al\.\(1979\)G\. H\. Golub, M\. Heath, and G\. WahbaGeneralized cross\-validation as a method for choosing a good ridge parameter\.Technometrics21\(2\),pp\. 215–223\.Cited by:[Appendix A](https://arxiv.org/html/2610.19166#A1.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.The Llama\-3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§6\.1](https://arxiv.org/html/2610.19166#S6.SS1.p4.1)\.
- Hadaet al\.\(2024a\)R\. Hada, V\. Gumma, M\. Ahmed, K\. Bali, and S\. SitaramMETAL: towards multilingual meta\-evaluation\.InFindings of the Association for Computational Linguistics: NAACL 2024,pp\. 2280–2298\.Cited by:[§1](https://arxiv.org/html/2610.19166#S1.p2.1),[§2](https://arxiv.org/html/2610.19166#S2.p3.1)\.
- Hadaet al\.\(2024b\)R\. Hada, V\. Gumma, A\. De Wynter, H\. Diddee, M\. Ahmed, M\. Choudhury, K\. Bali, and S\. SitaramAre large language model\-based evaluators the solution to scaling up multilingual evaluation?\.InFindings of the Association for Computational Linguistics: EACL 2024,pp\. 1051–1070\.Cited by:[§1](https://arxiv.org/html/2610.19166#S1.p2.1),[§2](https://arxiv.org/html/2610.19166#S2.p3.1)\.
- Hastieet al\.\(2009\)T\. Hastie, R\. Tibshirani, J\. H\. Friedman, and J\. H\. FriedmanThe elements of statistical learning: data mining, inference, and prediction\.Vol\.2,Springer\.Cited by:[Appendix A](https://arxiv.org/html/2610.19166#A1.p1.1)\.
- Hermannet al\.\(2015\)K\. M\. Hermann, T\. Kocisky, E\. Grefenstette, L\. Espeholt, W\. Kay, M\. Suleyman, and P\. BlunsomTeaching machines to read and comprehend\.Advances in neural information processing systems28\.Cited by:[Table 3](https://arxiv.org/html/2610.19166#A2.T3)\.
- Hoerl and Kennard \(1970\)A\. E\. Hoerl and R\. W\. KennardRidge regression: biased estimation for nonorthogonal problems\.Technometrics12\(1\),pp\. 55–67\.Cited by:[Appendix A](https://arxiv.org/html/2610.19166#A1.p1.1)\.
- Houet al\.\(2026\)Y\. Hou, J\. Li, X\. Fu, Z\. He, A\. Yan, X\. Chen, and J\. McAuleyBridging language and items for retrieval and recommendation: benchmarking llms as semantic encoders\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3251–3265\.Cited by:[Table 3](https://arxiv.org/html/2610.19166#A2.T3),[§4\.1](https://arxiv.org/html/2610.19166#S4.SS1.p1.1)\.
- Huber \(1992\)P\. J\. HuberRobust estimation of a location parameter\.InBreakthroughs in statistics: Methodology and distribution,pp\. 492–518\.Cited by:[Appendix A](https://arxiv.org/html/2610.19166#A1.p1.1)\.
- Jangraet al\.\(2025\)A\. Jangra, B\. Sarrafzadeh, S\. Cucerzan, A\. de Wynter, and S\. K\. JauharEvaluating style\-personalized text generation: challenges and directions\.arXiv preprint arXiv:2508\.06374\.Cited by:[§1](https://arxiv.org/html/2610.19166#S1.p2.1),[§2](https://arxiv.org/html/2610.19166#S2.p2.1)\.
- Jinet al\.\(2022\)D\. Jin, Z\. Jin, Z\. Hu, O\. Vechtomova, and R\. MihalceaDeep learning for text style transfer: a survey\.Computational Linguistics48\(1\),pp\. 155–205\.Cited by:[§2](https://arxiv.org/html/2610.19166#S2.p3.1)\.
- Kimet al\.\(2025\)J\. Kim, H\. Zhang, and D\. JurgensLeveraging multilingual training for authorship representation: enhancing generalization across languages and domains\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 34855–34880\.Cited by:[§2](https://arxiv.org/html/2610.19166#S2.p3.1),[§6\.1](https://arxiv.org/html/2610.19166#S6.SS1.p3.1.7)\.
- Kocmiet al\.\(2021\)T\. Kocmi, C\. Federmann, R\. Grundkiewicz, M\. Junczys\-Dowmunt, H\. Matsushita, and A\. MenezesTo ship or not to ship: an extensive evaluation of automatic metrics for machine translation\.InProceedings of the sixth conference on machine translation,pp\. 478–494\.Cited by:[§1](https://arxiv.org/html/2610.19166#S1.p2.1)\.
- Krishnaet al\.\(2022\)K\. Krishna, D\. Nathani, X\. Garcia, B\. Samanta, and P\. TalukdarFew\-shot controllable style transfer for low\-resource multilingual settings\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 7439–7468\.Cited by:[§2](https://arxiv.org/html/2610.19166#S2.p3.1)\.
- Krishnaet al\.\(2020\)K\. Krishna, J\. Wieting, and M\. IyyerReformulating unsupervised style transfer as paraphrase generation\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 737–762\.Cited by:[§2](https://arxiv.org/html/2610.19166#S2.p2.1)\.
- Kumaret al\.\(2024\)I\. Kumar, S\. Viswanathan, S\. Yerra, A\. Salemi, R\. A\. Rossi, F\. Dernoncourt, H\. Deilamsalehy, X\. Chen, R\. Zhang, S\. Agarwal,et al\.Longlamp: a benchmark for personalized long\-form text generation\.arXiv preprint arXiv:2407\.11016\.Cited by:[§2](https://arxiv.org/html/2610.19166#S2.p2.1)\.
- Lcvenshtcin \(1966\)V\. LcvenshtcinBinary coors capable or ‘correcting deletions, insertions, and reversals\.InSoviet physics\-doklady,Vol\.10,pp\. 707–710\.Cited by:[§6\.1](https://arxiv.org/html/2610.19166#S6.SS1.p2.1)\.
- Liuet al\.\(2023\)Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. ZhuG\-eval: nlg evaluation using gpt\-4 with better human alignment\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 2511–2522\.Cited by:[§1](https://arxiv.org/html/2610.19166#S1.p1.1)\.
- Maaset al\.\(2011\)A\. Maas, R\. E\. Daly, P\. T\. Pham, D\. Huang, A\. Y\. Ng, and C\. PottsLearning word vectors for sentiment analysis\.InProceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies,pp\. 142–150\.Cited by:[Table 3](https://arxiv.org/html/2610.19166#A2.T3)\.
- Mathuret al\.\(2020\)N\. Mathur, T\. Baldwin, and T\. CohnTangled up in bleu: reevaluating the evaluation of automatic machine translation evaluation metrics\.InProceedings of the 58th annual meeting of the association for computational linguistics,pp\. 4984–4997\.Cited by:[§1](https://arxiv.org/html/2610.19166#S1.p1.1),[§2](https://arxiv.org/html/2610.19166#S2.p1.1)\.
- Miret al\.\(2019\)R\. Mir, B\. Felbo, N\. Obradovich, and I\. RahwanEvaluating style transfer for text\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),pp\. 495–504\.Cited by:[§1](https://arxiv.org/html/2610.19166#S1.p1.1),[§2](https://arxiv.org/html/2610.19166#S2.p1.1)\.
- Mukherjeeet al\.\(2025\)S\. Mukherjee, A\. K\. Ojha, J\. P\. McCrae, and O\. DušekEvaluating text style transfer evaluation: are there any reliable metrics?\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 4: Student Research Workshop\),pp\. 418–434\.Cited by:[§1](https://arxiv.org/html/2610.19166#S1.p1.1),[§2](https://arxiv.org/html/2610.19166#S2.p2.1)\.
- Novikovaet al\.\(2017\)J\. Novikova, O\. Dušek, A\. C\. Curry, and V\. RieserWhy we need new evaluation metrics for nlg\.InProceedings of the 2017 conference on empirical methods in natural language processing,pp\. 2241–2252\.Cited by:[§1](https://arxiv.org/html/2610.19166#S1.p1.1),[§2](https://arxiv.org/html/2610.19166#S2.p1.1)\.
- Olmoet al\.\(2025\)T\. Olmo, A\. Ettinger, A\. Bertsch, B\. Kuehl, D\. Graham, D\. Heineman, D\. Groeneveld, F\. Brahman, F\. Timbers, H\. Ivison,et al\.Olmo 3\.arXiv preprint arXiv:2512\.13961\.Cited by:[§6\.1](https://arxiv.org/html/2610.19166#S6.SS1.p4.1)\.
- Ostheimeret al\.\(2024\)P\. S\. Ostheimer, M\. Nagda, M\. Kloft, and S\. FellenzText style transfer evaluation using large language models\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),pp\. 15802–15822\.Cited by:[§1](https://arxiv.org/html/2610.19166#S1.p1.1)\.
- Papke and Wooldridge \(1996\)L\. E\. Papke and J\. M\. WooldridgeEconometric methods for fractional response variables with an application to 401 \(k\) plan participation rates\.Journal of applied econometrics11\(6\),pp\. 619–632\.Cited by:[Appendix A](https://arxiv.org/html/2610.19166#A1.p1.1)\.
- Patelet al\.\(2025\)A\. Patel, J\. Zhu, J\. Qiu, Z\. Horvitz, M\. Apidianaki, K\. McKeown, and C\. Callison\-BurchStyleDistance: stronger content\-independent style embeddings with synthetic parallel examples\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 8662–8685\.External Links:[Link](https://aclanthology.org/2025.naacl-long.436/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.436),ISBN 979\-8\-89176\-189\-6Cited by:[§2](https://arxiv.org/html/2610.19166#S2.p3.1),[§6\.1](https://arxiv.org/html/2610.19166#S6.SS1.p3.1)\.
- Plank \(2022\)B\. PlankThe “problem” of human label variation: on ground truth in data, modeling and evaluation\.InProceedings of the 2022 conference on empirical methods in natural language processing,pp\. 10671–10682\.Cited by:[§1](https://arxiv.org/html/2610.19166#S1.p1.1)\.
- Protasovet al\.\(2026\)V\. Protasov, N\. Babakov, D\. Dementieva, and A\. PanchenkoEvaluating text style transfer: a nine\-language benchmark for text detoxification\.InProceedings of the Fifteenth Language Resources and Evaluation Conference \(LREC 2026\),Vol\.11,pp\. 4560–4574\.Cited by:[§2](https://arxiv.org/html/2610.19166#S2.p2.1)\.
- Qiuet al\.\(2025\)J\. Qiu, J\. Zhu, A\. Patel, M\. Apidianaki, and C\. Callison\-BurchMstyledistance: multilingual style embeddings and their evaluation\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 16917–16931\.Cited by:[§2](https://arxiv.org/html/2610.19166#S2.p3.1),[§6\.1](https://arxiv.org/html/2610.19166#S6.SS1.p3.1)\.
- Qwenet al\.\(2025\)Qwen, :, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. QiuQwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[§6\.1](https://arxiv.org/html/2610.19166#S6.SS1.p4.1)\.
- Reiet al\.\(2020\)R\. Rei, C\. Stewart, A\. C\. Farinha, and A\. LavieCOMET: a neural framework for mt evaluation\.InProceedings of the 2020 conference on empirical methods in natural language processing \(emnlp\),pp\. 2685–2702\.Cited by:[§2](https://arxiv.org/html/2610.19166#S2.p1.1)\.
- Rumelhartet al\.\(1986\)D\. E\. Rumelhart, G\. E\. Hinton, and R\. J\. WilliamsLearning representations by back\-propagating errors\.nature323\(6088\),pp\. 533–536\.Cited by:[Appendix A](https://arxiv.org/html/2610.19166#A1.p1.1)\.
- Salemiet al\.\(2024\)A\. Salemi, S\. Mysore, M\. Bendersky, and H\. ZamaniLamp: when large language models meet personalization\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 7370–7392\.Cited by:[§2](https://arxiv.org/html/2610.19166#S2.p2.1)\.
- Saunderset al\.\(1998\)C\. Saunders, A\. Gammerman, and V\. VovkRidge regression learning algorithm in dual variables\.Cited by:[Appendix A](https://arxiv.org/html/2610.19166#A1.p1.1)\.
- Seibold and McPHEE \(1979\)D\. R\. Seibold and R\. D\. McPHEECommonality analysis: a method for decomposing explained variance in multiple regression analyses\.Human Communication Research5\(4\),pp\. 355–365\.Cited by:[§5\.1](https://arxiv.org/html/2610.19166#S5.SS1.p4.1)\.
- Sellamet al\.\(2020\)T\. Sellam, D\. Das, and A\. ParikhBLEURT: learning robust metrics for text generation\.InProceedings of the 58th annual meeting of the association for computational linguistics,pp\. 7881–7892\.Cited by:[§2](https://arxiv.org/html/2610.19166#S2.p1.1)\.
- Sonet al\.\(2024\)G\. Son, D\. Yoon, J\. Suk, J\. Aula\-Blasco, M\. Aslan, V\. T\. Kim, S\. B\. Islam, J\. Prats\-Cristià, L\. Tormo\-Bañuelos, and S\. KimMM\-eval: a multilingual meta\-evaluation benchmark for llm\-as\-a\-judge and reward models\.arXiv preprint arXiv:2410\.17578\.Cited by:[§2](https://arxiv.org/html/2610.19166#S2.p3.1)\.
- Teamet al\.\(2023\)G\. Team, R\. Anil, S\. Borgeaud, J\. Alayrac, J\. Yu, R\. Soricut, J\. Schalkwyk, A\. M\. Dai, A\. Hauth, K\. Millican,et al\.Gemini: a family of highly capable multimodal models\.arXiv preprint arXiv:2312\.11805\.Cited by:[§6\.1](https://arxiv.org/html/2610.19166#S6.SS1.p4.1)\.
- Teamet al\.\(2026\)G\. Team, S\. E\. Abd, V\. Aggarwal, R\. Algayres, A\. Andreev, O\. Bachem, I\. Ballantyne, C\. Brick, V\. Cărbune, M\. Casbon,et al\.Gemma 4 technical report\.arXiv preprint arXiv:2607\.02770\.Cited by:[§6\.1](https://arxiv.org/html/2610.19166#S6.SS1.p4.1)\.
- Team \(2026\)Q\. TeamQwen3\.5\-omni technical report\.arXiv preprint arXiv:2604\.15804\.Cited by:[§6\.1](https://arxiv.org/html/2610.19166#S6.SS1.p4.1)\.
- Tibshirani \(1996\)R\. TibshiraniRegression shrinkage and selection via the lasso\.Journal of the Royal Statistical Society Series B: Statistical Methodology58\(1\),pp\. 267–288\.Cited by:[Appendix A](https://arxiv.org/html/2610.19166#A1.p1.1)\.
- Vergaet al\.\(2024\)P\. Verga, S\. Hofstatter, S\. Althammer, Y\. Su, A\. Piktus, A\. Arkhangorodsky, M\. Xu, N\. White, and P\. LewisReplacing judges with juries: evaluating llm generations with a panel of diverse models\.arXiv preprint arXiv:2404\.18796\.Cited by:[§2](https://arxiv.org/html/2610.19166#S2.p1.1)\.
- Wegmannet al\.\(2022\)A\. Wegmann, M\. Schraagen, and D\. NguyenSame author or just same topic? towards content\-independent style representations\.InProceedings of the 7th Workshop on Representation Learning for NLP,pp\. 249–268\.Cited by:[§2](https://arxiv.org/html/2610.19166#S2.p3.1),[§6\.1](https://arxiv.org/html/2610.19166#S6.SS1.p3.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. QiuQwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§6\.1](https://arxiv.org/html/2610.19166#S6.SS1.p4.1)\.
- Zou and Hastie \(2005\)H\. Zou and T\. HastieRegularization and variable selection via the elastic net\.Journal of the Royal Statistical Society Series B: Statistical Methodology67\(2\),pp\. 301–320\.Cited by:[Appendix A](https://arxiv.org/html/2610.19166#A1.p1.1)\.

## Appendix ASelecting the regressor for meta\-evaluation

Our formulation of the meta\-metric \(Eq\.[2](https://arxiv.org/html/2610.19166#S5.E2)\) is applicable to any regression model, and to select one for our meta\-metric evaluation, we compared 16 models across 4 families\. All metrics were trained on the same calibrated model outputs and evaluated on the same folds \(as described in Section[5\.2](https://arxiv.org/html/2610.19166#S5.SS2)\) over the test split of the dataset comprising roughly 100 instances \(Table[1](https://arxiv.org/html/2610.19166#S4.T1)\)\. Thelinearregression models comprised of least squares \(LR\)\([Hastie et al\., 2009](https://arxiv.org/html/2610.19166#bib.bib24)\), ridge with a fixed or cross\-validated penalty \(Ridge,Ridge\-CV\)\([Hoerl and Kennard, 1970](https://arxiv.org/html/2610.19166#bib.bib25);[Golub et al\., 1979](https://arxiv.org/html/2610.19166#bib.bib26)\),Lasso\([Tibshirani, 1996](https://arxiv.org/html/2610.19166#bib.bib27)\),ElasticNet\([Zou and Hastie, 2005](https://arxiv.org/html/2610.19166#bib.bib28)\), Huber\-loss regression \(Huber\)\([Huber, 1992](https://arxiv.org/html/2610.19166#bib.bib29)\), and two logit\-link models that rescale the label to\(0,1\)\(0,1\)and fit it on the logit scale \(LogitLink,LogitLink\-OLS\)\([Papke and Wooldridge, 1996](https://arxiv.org/html/2610.19166#bib.bib30)\)\.Kernel and instance\-basedmodels included support\-vector machines and kernel ridge regression with an RBF kernel \(SVR,KernelRidge\)\([Drucker et al\., 1996](https://arxiv.org/html/2610.19166#bib.bib31);[Saunders et al\., 1998](https://arxiv.org/html/2610.19166#bib.bib32)\), and distance\-weightedkk\-nearest neighbors \(kNN\)\([Altman, 1992](https://arxiv.org/html/2610.19166#bib.bib33)\)\.Tree ensemblesmodels includedRandomForest\([Breiman, 2001](https://arxiv.org/html/2610.19166#bib.bib34)\), extremely randomized trees \(ExtraTrees\)\([Geurts et al\., 2006](https://arxiv.org/html/2610.19166#bib.bib23)\), andGradientBoosting\([Friedman, 2001](https://arxiv.org/html/2610.19166#bib.bib35)\)\. We also evaluated simpleneural modelsmultilayer perceptrons with one or two hidden layers \(MLP\-1,MLP\-2\)\([Rumelhart et al\., 1986](https://arxiv.org/html/2610.19166#bib.bib36)\)\.

Figure 3:Performance of 16 regression models spanning four regression approaches\. The regression models were trained to fit the metric outputs from all 32 evaluation metrics \(i\.e\.,theρa​l​l\\rho\_\{all\}ensemble\) to predict the reliability\-weighted human annotations in a 5\-fold cross\-validation repeated five times on 10 languages for four evaluation dimensions \(styleaccuracy,factualaccuracy,leaksandusability\)\.Figure[3](https://arxiv.org/html/2610.19166#A1.F3)shows the mean absolute error \(MAE\) of all 16 regression models\. Overall, tree ensemble\-based approaches outperformed other regressors, withExtraTreesandRandomForestachieving the lowest mean ranks across languages and evaluation dimensions \(2\.4 and 2\.9\)\. Due to the small size of the training features \(32 metrics\) and training data \(∼\{\\sim\}160 generations for 80 instances\), the simple least\-squares approach overfitted the training data \(rank 13\.2\), with regularized models performing better \(rankLasso=\{\}\_\{\\texttt\{Lasso\}\}=7\.0\)\. Neural MLP models were among the weakest models \(rankMLP1=\{\}\_\{\\texttt\{MLP\}\_\{1\}\}=15\.5 and rankMLP2=\{\}\_\{\\texttt\{MLP\}\_\{2\}\}=14\.5\)\. We selected the best\-ranking model,ExtraTrees\(300 trees, minimum leaf size 3\), as the regression model for the meta\-evaluation in this work\.

## Appendix BDataset Curation \(Additional Details\)

To extend the SPTG corpus to other languages, we sampled 1,002 pairs of long\-form English text, comprising an original content text and a reference\-style text \(Table[3](https://arxiv.org/html/2610.19166#A2.T3)\)\. The raw translations were then reviewed by a native\-speaking annotator, who corrected the accuracy, fluency, and locale\-appropriateness of the output, with an optional note explaining their edits\. After this step, every instance consequently had three versions of a textttfor a languageLL: the standard English sourcete​nt\_\{en\}, the raw machine translation outputtLM​Tt\_\{L\}^\{MT\}, and the human\-corrected translationtLh​ct\_\{L\}^\{hc\}\.

The amount of human corrections varied widely across locales \(Table[4](https://arxiv.org/html/2610.19166#A2.T4)\), from under 1% of characters edited in five locales to 33\.9% for French, 23\.2% for German, and∼\{\\sim\}22% for Zulu and Vietnamese, with the large majority of translations altered in most locales \(e\.g\. 97\-99% of items edited in French, Vietnamese, and Turkish\)\. Annotators left 5,461 free\-text notes documenting their edits, mostly in Maltese, Welsh, Albanian and Irish\. Categorizing these notes surfaces six recurring types of corrections:fluency,lexical,grammatical,spelling,mistranslation, textit dialect\-specific, and others \(see Table[5](https://arxiv.org/html/2610.19166#A2.T5)for examples\)\.

We also provide the annotator reliability for each annotator across evaluation dimensions and languages in Figure[4](https://arxiv.org/html/2610.19166#A2.F4)\. This leave\-one\-out correlation between one annotator and the mean rating of the other two was used as the reliability weight for each annotator in our meta\-metric implementation \(Section[5\.2](https://arxiv.org/html/2610.19166#S5.SS2)\)\.

Table 3:Composition of the English corpus from which the multilingual SPTG corpus is built \(N=1,002N\{=\}1\{,\}002annotated instances, drawn from the1,5021\{,\}502translated source instances\)\. Each instance pairs a reference \(a set of style exemplars\) with an independently sampled origin \(the content to be re\-expressed\), and the two are matched at random\. References are drawn mostly from creative short\-form writing \(r/ShortStories;[https://www\.kaggle\.com/datasets/trevordu/reddit\-short\-stories](https://www.kaggle.com/datasets/trevordu/reddit-short-stories)\), with the remainder from Amazon product reviews\([Hou et al\., 2026](https://arxiv.org/html/2610.19166#bib.bib37)\); origins span fiction, encyclopedic text \(Wikipedia\), movie reviews \(IMDB;[Maas et al\., 2011](https://arxiv.org/html/2610.19166#bib.bib52)\), product reviews, and news \(CNN/DailyMail;[Hermann et al\., 2015](https://arxiv.org/html/2610.19166#bib.bib53)\)\.Composition of the English corpusCount%Reference \(style source\) domainr/ShortStories85785\.5Amazon Reviews \(Digital Music\)12112\.1Amazon Reviews \(Subscription Boxes\)242\.4Origin \(content source\) domainr/ShortStories34134\.0Wikipedia32031\.9IMDB14614\.6Amazon Reviews \(Digital Music\)12912\.9CNN/DailyMail343\.4Amazon Reviews \(Subscription Boxes\)323\.2Total instances1,002100\.0Table 4:Post\-editing statistics for LLM\-generated translations\. The native speakers were given the original English and model\-translated outputs to make corrections\. For each locale, we report the percentage of translations that the post\-editors changed \(*Edited*\), the extent of these changes as the character\-level edit distance between the machine translation and its corrected version, normalized by length \(*Post\-edit*; mean±\\pms\.d\. in %\), and the total number of free\-text comments left by the annotators \(*Notes*\)\. Locales are sorted by post\-edit distance\.Table 5:Types of corrections noted by post\-editors, with examples from different locales\. We assigned each free\-text note to one type by keyword matching \(e\.g\.*typo*→\\toSpelling,*tense*→\\toGrammar,*register*→\\toDialect\)\. Most notes were too short to classify and fall under*Other*, so the breakdown is only indicative\. The number of notes also varies widely by locale \(Maltese post\-editors wrote most of them\), reflecting differences in annotation habits as much as in translation quality\.![Refer to caption](https://arxiv.org/html/2610.19166v1/annotators_reliability.png)Figure 4:Per\-language annotator reliability for all three annotators computed using Spearman’s correlation with a leave\-one\-out strategy,i\.e\.,computing correlation between one annotator and the mean rating from the other two\.
## Appendix CExtended Results

We provide the extended results from our meta\-evaluation study in this section\. Table[6](https://arxiv.org/html/2610.19166#A3.T6)presents the pointwise and pairwise correlations of all the evaluation metrics\. Figure[5](https://arxiv.org/html/2610.19166#A3.F5)shows the per\-language Spearman correlation for the pointwise evaluation on the best\-performing metrics\. Figures[6](https://arxiv.org/html/2610.19166#A3.F6),[7](https://arxiv.org/html/2610.19166#A3.F7)and[8](https://arxiv.org/html/2610.19166#A3.F8)provide the extensive inter\-metric variation decomposition using theΔR2p\|\.\\Delta R^\{2\}\_\{p\|\.\}\(Eq\.[1](https://arxiv.org/html/2610.19166#S5.E1)\) for each metric, across different evaluation paradigms and for all individual metrics respectively\.

Figure[9](https://arxiv.org/html/2610.19166#A3.F9)illustrates the contribution of all constituent metrics for thema​l​lm\_\{all\}ensemble metric across the 40 evaluation dimension and language combinations\. We provide the aggregate results over all 10 languages from our meta\-evaluation analysis \(Eq\.[2](https://arxiv.org/html/2610.19166#S5.E2)\) in Table[7](https://arxiv.org/html/2610.19166#A3.T7), and the extensive per\-language results in Table[8](https://arxiv.org/html/2610.19166#A3.T8)\. Finally, we provide the parsing error distribution for all LLM\-judge outputs for pointwise and pairwise metric generations in Table[9](https://arxiv.org/html/2610.19166#A3.T9)\.

Table 6:Pairwise and pointwise statistical diagnostics of evaluation metrics over the four evaluation dimensions and their overall aggregate \(meanρ\\rhofor pointwise, and meanκ\\kappafor pairwise\)\. The evaluation is performed on the test split of the dataset, averaged over human annotations from 10 languages\. The pointwise alignment is computed with Spearman’sρ\\rho, measured against the mean ordinal human ratings for each paraphrase on the respective evaluation dimension\. Whereas the pairwise alignment is measured with Cohen’sκ\\kappa, against the sign of the difference in ordinal ratings for the two paraphrases, dropping the instances where the difference is zero\.![Refer to caption](https://arxiv.org/html/2610.19166v1/perlang_A_2x2.png)Figure 5:Per\-language Spearman’sρ\\rhofor the pointwise evaluation of the top\-2 overall best\-performing metrics for each paradigm for each evaluation dimension\.Figure 6:The aggregateΔR2p\|\.\\Delta R^\{2\}\_\{p\|\.\}\(Eq\.[1](https://arxiv.org/html/2610.19166#S5.E1)\) for all evaluation metrics over the four evaluation dimensions against the rest of the evaluation metrics \(excluding style embeddings due to their limited language\)\.![Refer to caption](https://arxiv.org/html/2610.19166v1/relative_per_paradigm.png)Figure 7:AverageΔ​Rp\|q2\\Delta R^\{2\}\_\{p\\mid q\}\(Eq\.[1](https://arxiv.org/html/2610.19166#S5.E1)\) over different evaluation metric paradigms\. Each entry\(x,y\)\(x,y\)denotes the average1\|x\|\+\|y\|​∑xi∑yjΔ​Rxi\|yj2\\frac\{1\}\{\|x\|\+\|y\|\}\\sum\_\{x\_\{i\}\}\\sum\_\{y\_\{j\}\}\\Delta R^\{2\}\_\{x\_\{i\}\\mid y\_\{j\}\}for evaluation paradigmsxxandyy\. The semantic embeddings paradigm has only one metric, so we leave the corresponding cell blank\. Similarly, usability is evaluated only with LLM\-specific metrics; hence, we leave the other comparisons blank\.![Refer to caption](https://arxiv.org/html/2610.19166v1/relative_dR2_2x2.png)Figure 8:Δ​Rp\|q2\\Delta R^\{2\}\_\{p\\mid q\}\(Eq\.[1](https://arxiv.org/html/2610.19166#S5.E1)\) for a metric pairmp,mqm\_\{p\},m\_\{q\}\. The brighter the cell, the more information it provides relative to the other metrics in the corresponding evaluation dimension\.Table 7:Meta\-evaluation of metrics calibrated on chance prediction \(Φ=0\\Phi=0\) and human ceiling \(Φ=0\\Phi=0\); see Section[5\.2](https://arxiv.org/html/2610.19166#S5.SS2)for implementation details\. We present the results for individual metrics covering string\-based metrics, semantic embeddings, style embeddings, open\-weight models, and proprietary models\. We also provide the performance of various ensemble metrics at the end \(denoted byρ\\rho\)\.![Refer to caption](https://arxiv.org/html/2610.19166v1/mm_contrib_extratrees_landscape.png)Figure 9:Contribution of each metric to the full ensemblemallm\_\{\\text\{all\}\}for all four evaluation dimensions on the 10 languages\. Each cell is the metric’s share of the ensemble’s out\-of\-fold permutation importance, computed as the metric’s normalized importance\. The importance of a metric is computed as the increase in out\-of\-fold MAE, computed thrice over all 25 out\-folds\. Negative importance values were clipped to zero before normalizing to obtain the final contribution of each constituent metric\. Gray cells indicate that the metric is unavailable for that language\. LLMs with less than half of the valid test set judgments are dropped out \(see Table[9](https://arxiv.org/html/2610.19166#A3.T9)\)\.Table 8:Metric quality𝚿\\mathbf\{\\Psi\}per language, for every single metric \(top\) and each ensemble of metrics \(bottom\)\. Each cell lists the scores for style, factual consistency, leakage, and usability \(S/F/L/U\)\. A score of 0 means the metric does no better than always predicting the mean label, and 1 means it matches the consensus as closely as a single annotator does; values are clipped to\[0,1\]\[0,1\]\. ‘–’ marks dimensions a metric cannot score \(usability has no reference text\) and languages a metric does not cover\.Table 9:Percentage of LLM\-judge predictions without a usable score, per language, pooled over the four evaluation dimensions\. A prediction counts as failed if the output could not be parsed, was cut off before the score, or was empty or a refusal\. Each cell lists pointwise / pairwise\. Pairwise rates are based on far fewer items \(20–40 per language and dimension, against about 200 pointwise\)\.
## Appendix DPrompts

Figures[10](https://arxiv.org/html/2610.19166#A4.F10)and[11](https://arxiv.org/html/2610.19166#A4.F11)provide the dataset construction prompts for translating English text into other languages and generating SPTG output, respectively\. Figures[12](https://arxiv.org/html/2610.19166#A4.F12)and[13](https://arxiv.org/html/2610.19166#A4.F13)provide the pointwise and pairwise evaluation prompts used to generate the LLM\-as\-judge responses for all the judge models\.

Figure 10:The zero\-shot prompt used to translate English text to other languages\.Figure 11:The zero\-shot prompt used to generate style\-personalized text for all languages\.Figure 12:The zero\-shot LLM\-as\-a\-judge prompt used to obtain the pointwise LLM responses\.Figure 13:The zero\-shot LLM\-as\-a\-judge prompt used to obtain the pairwise LLM responses\.