Evaluating RAG Metrics in Applied Contexts: An Experiment, Its Findings and Its Limitations

# Evaluating RAG Metrics in Applied Contexts: An Experiment, Its Findings and Its Limitations
Source: [https://arxiv.org/html/2607.07302](https://arxiv.org/html/2607.07302)
###### Abstract

This paper reports an empirical study evaluating the relevance of several RAG metrics\. The experiment is based on a question\-answering dataset created by human annotators from business data\. The generated responses and retrieved spans of a RAG system are scored using evaluation metrics from four libraries \(Ragas, DeepEval, RAGChecker, Opik\)\. These metrics are compared to scores given by two evaluators, as well as to standard metrics such as recall\. An analysis of correlations is conducted\. Finally, we highlight certain limitations of our methodology, compare it to those used in the literature, and suggest some avenues for future research\. This paper is an English translation of a paper originally published in the French\-speaking workshop EvalLLM 2026\(brabant\_2026\)\.

## 1Introduction

Evaluating and comparing RAG \(*Retrieval Augmented Generation*\) systems remains a challenging task today: even when a sufficiently large set of test questions with reference answers is available, automatically evaluating a system’s responses against these references is far from trivial\. A popular approach is to use so\-called LLM\-as\-a\-judge metrics to perform this evaluation\. Although these metrics generally seem more relevant than classical metrics such as BLEU, it is difficult to know in advance what the relevance of a specific metric will be on the dataset under consideration, especially since evaluation criteria can vary \(relevance, factuality, completeness of responses, etc\.\)\. It is therefore useful, when evaluating a RAG system under development, to conduct an evaluation of available metrics in order to verify that they provide acceptable approximations of the criterion considered, and that they will thus enable reliable comparison of different iterations of the RAG system being developed\. Generally, metrics are evaluated by measuring their correlation with scores given by humans\.

This article reports an experiment of this type\. This experiment is based on a question\-answering dataset created by annotators from business data\. The responses produced and the documents retrieved by a RAG system are scored using RAG metrics from four libraries: Ragas111[https://docs\.ragas\.io/](https://docs.ragas.io/)\(Eset al\.,[2024](https://arxiv.org/html/2607.07302#bib.bib2)\), DeepEval222[https://docs\.confident\-ai\.com/](https://docs.confident-ai.com/), RAGChecker333[https://github\.com/amazon\-science/RAGChecker](https://github.com/amazon-science/RAGChecker)\(Ruet al\.,[2024](https://arxiv.org/html/2607.07302#bib.bib1)\), and Opik444[https://www\.comet\.com/site/products/opik/](https://www.comet.com/site/products/opik/)\. These metrics are compared to reference evaluations: human evaluations for assessing generated responses, and recall for evaluating retrieval\.

Note that our objective is not to compare the different metrics evaluated, as the study results are strongly dependent on the choices made during its design\. The reported experiments rather aim to apply a methodology in order to test its advantages and limitations\. Unfortunately, the question\-answering dataset used cannot be made public; however, we share the raw scores and the code used for the statistical analyses555[https://github\.com/Orange\-OpenSource/evalllm2026\-metric\-correlation\-analysis](https://github.com/Orange-OpenSource/evalllm2026-metric-correlation-analysis)\.

The article is organized as follows\. Section[2](https://arxiv.org/html/2607.07302#S2)describes the application context and the data used, including manual annotation and evaluation processes\. Section[3](https://arxiv.org/html/2607.07302#S3)describes the methodology applied to compare reference evaluations with evaluations produced by metrics from the tested libraries\. Section[4](https://arxiv.org/html/2607.07302#S4)reports and analyzes the results of this experiment\. Certain limitations of our methodology are highlighted in Section[5](https://arxiv.org/html/2607.07302#S5)\. In Section[6](https://arxiv.org/html/2607.07302#S6), our methodology is compared to those employed in the literature\. We finally propose some research perspectives aimed at facilitating the application of reliable methodologies for evaluating RAG metrics, in Section[7](https://arxiv.org/html/2607.07302#S7)\.

## 2Context and Data

Our company is developing a RAG solution designed to answer questions in French about a business domain\. The developed system processes each given question via two key modules: first, a*retriever*, whose role is to retrieve relevant spans within a business document database \(also in French\), which combines a dense approach with BM25; then a*generator*, which injects the user’s question and the top 5 spans from the retriever into a prompt for GPT 3\.5, so that it generates the answer to the user’s question\. The retriever is a hybrid system combining a dense approach with BM25\.

In order to evaluate the performance of this system, a dataset was created from the corpus of business documents used by the RAG system\. This dataset is a set of question\-answer pairs accompanied by reference spans\.

### 2\.1The Question\-Answering Dataset

This dataset consists of 96 questions\. Each question is associated with a reference answer, as well as one or more spans from business documents; these spans contain the information necessary to answer the question with which they are associated\.

The question\-answering dataset is created based on the existing business document database on the telecommunications domain, containing information on, for instance, different offers and customer relations\. It contains 479 documents, with length ranging from 11 to 14,025 words \(1,112 on average\)\. The questions and answers are written by annotators who are company employees, as follows:

1. 1\.Documents are divided into “pages” of maximum 1,000 words, relying on the structure of titles and sections to preserve content coherence\. This size limit is chosen arbitrarily, with the aim of limiting the amount of information to process for each annotation\.
2. 2\.The obtained pages are distributed in random order to annotators\.
3. 3\.Each annotator annotates one by one the pages assigned to them\. The annotation of a page proceeds as follows\. 1. \(a\)The annotator writes a question related to the page content, as well as a correct answer to this question \(if the answer can be given from the information contained in the page\)\. 2. \(b\)They select one or more spans in the page that allow answering the question\. These steps can be repeated to produce up to 5 questions per page\.

Annotators were asked to diversify, as much as possible, the type of questions produced, and to indicate the type of each question within a dropdown list\. The possible question types and their numbers are summarized in Table[1](https://arxiv.org/html/2607.07302#S2.T1)\. Additionally, 15 questions are not associated with any answer, because the necessary information is not available in the business documents\. In this case, the system is expected to produce an answer such as ”I don’t know”\.

Table 1:Question types present in the question\-answering dataset\. Some questions belong to multiple types\.The number of spans per question varies from 0 \(for questions without answers\) to 4, with an average of 1\.3\. We observe that the size of spans associated with questions varies greatly, ranging from 1 word \(the annotator having selected a single keyword corresponding to the answer\) to 202 words\. The implications in terms of evaluation metric choice are discussed in Section[2\.3](https://arxiv.org/html/2607.07302#S2.SS3)\.

### 2\.2RAG System Evaluation Procedure

The question\-answering dataset is intended to enable evaluation of the RAG system: first, questions are given as input to the system, and the system outputs \(generated responses and retrieved spans\) are collected; then the system outputs are evaluated using different metrics\.

Comparing generated responses with reference answers produces an overall score, while comparing retrieved spans with reference spans produces a retriever performance score, which can be useful for estimating what proportion of system failures is due to the retriever and which is due to the generator, and thus targeting improvement efforts appropriately\.

### 2\.3Available Metrics

This subsection briefly describes the metrics used to evaluate the RAG system\. We follow the terminology of\(Ruet al\.,[2024](https://arxiv.org/html/2607.07302#bib.bib1)\), where three types of metrics are distinguished:

- •*retrieval*metrics, which evaluate the relevance of retrieved spans relative to a given question;
- •*overall*metrics, which evaluate the response generated by the system for the question;
- •*generation*metrics, which evaluate the generator’s behavior relative to the content of spans from the retrieval\.

#### Retrieval Metrics\.

In the context of information retrieval, several metrics are traditionally used to evaluate retrieval quality: recall, precision, nDCG, MAP, etc\. In this study, we choose to report recall scores, which will be used as reference scores to evaluate retrieval metrics from the tested libraries\.*Recall*is defined as the proportion of relevant elements actually present in the topkkspans returned by the retriever\. A common approach is to calculate recall at the document level, i\.e\., by considering that a reference span is present in the retriever results if at least one span from the same document is present\. However, this condition does not imply that the reference span is actually contained in a returned span: the reference span can be disjoint from the retrieved span or partially overlapping\. This problem arises particularly here, since our references are made of short spans compared to the documents from which they originate\. We therefore calculate recall at the word level, i\.e\.: the percentage of words from reference spans present in the top k retrieved spans\. Note that words are identified by their positions in the text, not by their value: returned spans completely disjoint from reference spans will therefore give a score of 0, even if they contain identical words\. The choice of recall over other metrics \(such as precision\) is justified by the fact that \(1\) its scores are easy to interpret, \(2\) since reference spans tend to be relatively small, we seek to know if they are included in the fixed\-size spans returned by the retriever, which is a reasonable expectation and corresponds to the definition of recall; conversely, precision at the word level would give necessarily low scores\. Finally, the choice of this metric is supported by the fact that it correlates much better with evaluators’ average scores \(r=0\.35r=0\.35\) than other cited metrics, and notably than document\-level recall \(r=0\.05r=0\.05\)\.

The retrieval metrics proposed by the tested libraries differ from traditional metrics on two main respects: first, some of them do not require reference spans, so they can be applied even when these are not available; second, they often use language models, which allows them to calculate the final score based on semantically important elements of the considered spans\. Like recall, these metrics evaluate retrieval quality from the topkkretrieved spans\. We choose to fixk=5k=5for all metrics, so that the spans on which retrieval is evaluated correspond to those actually inserted into the generator’s prompt\.

#### Overall Metrics\.

Overall metrics evaluate the quality of the generated response according to specific criteria\. Many overall metrics, corresponding to different criteria, are proposed by the tested libraries\. Although these criteria are diverse, it can be noted that most are concerned with the factuality of the generated response, generally decomposed into two aspects:*precision*\(“is all the information given in the response correct?”\), and*recall*\(“is all the expected information given in the response?”\)\. In addition to factuality,*relevance*is sometimes considered: “is all the information given in the response related to the question?” In addition to metrics evaluating factuality and relevance, we integrate Opik’s*moderation*and*usefulness*metrics: the former checks for the absence of harmful or inappropriate content, while the latter combines various criteria to obtain a general score\.

#### Generation Metrics\.

Generation metrics evaluate the generator’s behavior relative to the content of spans from the retrieval\. The primary purpose of these metrics is to study certain generator behaviors and not its performance in the strict sense; however, one may want to use them to approximate overall criteria\. For example, DeepEval’s*faithfulness*metric, which aims to measure the response’s faithfulness to retrieved spans, can be used to approximate the precision criterion\. We therefore integrate into our study some generation metrics, which will be evaluated as overall metrics \(these are the four metrics named*hallucination*or*faithfulness*, see Figure[1](https://arxiv.org/html/2607.07302#S4.F1)\)\.

## 3Metric Evaluation: Methodology

We seek to evaluate to what extent the metrics mentioned in the previous section provide a good approximation of a given evaluation criterion\. We choose to define an overall criterion simultaneously measuring aspects of factuality and relevance\. This criterion is evaluated on a scale of 1 to 5 and defined by the rubric in Table[2](https://arxiv.org/html/2607.07302#S3.T2)\.

Table 2:Rubric used during human evaluation of RAG system responses\.This rubric is then applied by two evaluators \(company employees with expertise in natural language processing and generative models, one of whom participated in the annotation phase during dataset creation\), to score each RAG system output on the 96 instances of the question\-answering dataset\. We calculate the Pearson correlation666We calculate Pearson correlation rather than Cohen’s or Fleiss’s kappa, as these are poorly suited when annotations are ordinal in nature, as is the case here\.between the two series of scores obtained, to verify the robustness of the rubric and estimate the maximum performance that can be expected from an automatic metric\. The correlation obtained is 0\.85\. To simplify subsequent analyses, we base ourselves on the average scores obtained per response\. The scores thus obtained are called*reference scores*\.

We then proceed to the correlation analysis intended to evaluate the metrics\. The following section reports and analyzes:

- •correlations between overall metrics and reference scores;
- •correlations of retrieval metrics with recall: although recall is itself an imperfect metric, it is expected that a stronger correlation of a retrieval metric with recall indicates better reliability;
- •correlations between retrieval metrics and reference scores: indeed, since better retrieval correlates with better responses, retrieval metrics should also correlate with scores from our rubric\.

All reported correlations correspond to Pearson’s coefficient \(performing analyses with Spearman correlation gives results similar to those reported\)\.

## 4Results

\-1\.00\-0\.75\-0\.50\-0\.250\.000\.250\.500\.751\.00DeepEvalRagasOpikRAGCheckerbleurougeLsummeteorbertscore\-f1hallucinationanswer relevancyfaithfulnessfaithfulnessfactual correctness:f1factual correctness:recallfactual correctness:accuracyhallucinationmoderationanswer relevanceusefulnesscontext precisioncontext recallf1recallprecisionFigure 1:Pearson correlation of overall metrics with average human rates\.Figure[1](https://arxiv.org/html/2607.07302#S4.F1)summarizes the correlations obtained for overall metrics\. We first note that the width of confidence intervals is substantial, due to the modest size of our sample \(96\)\. However, it is possible to see certain trends emerge\.

First, we note that metrics sometimes considered obsolete such as METEOR correlate surprisingly well with reference scores\. Next, we observe that generation metrics used as overall metrics correlate weakly with reference scores, which is expected, since they do not have access to the reference answer\. Similarly, Opik’s*moderation*metric does not correlate with reference scores\. Conversely, we observe a very strong correlation for RAGChecker metrics, particularly recall\.

\-1\.00\-0\.75\-0\.50\-0\.250\.000\.250\.500\.751\.00DeepEvalRagasRAGCheckercontextual precision∗contextual recall∗contextual relevancy∗LLM context precision w/o ref∗LLM context precision w/ ref∗non\-LLM context precision w/ refLLM context recall∗non\-LLM context recallcontext entity recall∗claim recallcontext precisionFigure 2:Pearson correlation of overall metrics with recall\. Metrics that do not rely of reference spans are marked with an asterisk\.Figure[2](https://arxiv.org/html/2607.07302#S4.F2)summarizes the correlations of retrieval metrics with recall\. We observe larger overlaps of confidence intervals\. However, it is interesting to note that a metric such as DeepEval’s*contextual precision*obtains a correlation above 0\.5 with recall, without using reference spans\. We observe that conversely, Ragas’s non\-LLM metrics obtain very weak correlation\. This is probably explained by the fact that it compares retrieved spans to reference spans via Levenshtein distance, which does not give a relevant value when compared spans are of very different sizes, as is the case here\.

\-1\.00\-0\.75\-0\.50\-0\.250\.000\.250\.500\.751\.00DeepEvalRagasRAGCheckercontextual precision∗contextual recall∗contextual relevancy∗LLM context precision w/o ref∗LLM context precision w/ ref∗non\-LLM context precision w/ refLLM context recall∗non\-LLM context recallcontext entity recall∗claim recallcontext precisionFigure 3:Pearson correlations of retrieval metrics with average human rates\. Metrics that do not rely on reference spans are marked with an asterisk\.If we observe Figure[3](https://arxiv.org/html/2607.07302#S4.F3), we note a surprising phenomenon: several metrics show very strong correlation with reference scores\. This correlation is sometimes stronger than the correlation with recall\. RAGChecker’s*claim recall*correlation appears close to 0\.7\. It does not seem plausible that such a correlation is due solely to this metric’s ability to measure the relevance of retrieved documents\. We comment on this phenomenon in more detail in the following section\.

## 5Limitations

The main limitation of our study is related to the interpretation of correlations\. It is difficult to know a priori what sophisticated metrics using LLMs rely on to produce their scores\. Consequently, observing that a given metric correlates well with human judgment is not sufficient to affirm that it measures what we want it to measure, given the configuration of our experiment\. This difficulty can be illustrated by imagining a metric measuring question difficulty, without knowledge of generated responses\. Such a metric would give higher scores to easy questions, and these indeed tend to obtain better scores\. It would therefore have a positive correlation with reference scores, while its relevance for evaluating and comparing RAG systems would be null \(it would give exactly the same scores to all systems\)\. A less extreme illustration of this phenomenon is probably RAGChecker’s*claim recall*metric, whose correlation with evaluators’ scores is very high\. It seems likely that this metric does not measure only retrieval quality; a plausible interpretation is that it partially captures other characteristics such as, for example, the ease with which response information can be extracted from retrieved spans, or question difficulty\. This limitation is partly due to the configuration of our experiment: since it involves only one RAG system, it is impossible to observe metric behavior as a function of system outputs independently of the input question\.

Note that, despite this weakness, our methodology produces certain exploitable results by eliminating certain candidate metrics: indeed, we can affirm that metrics with poor correlation with the evaluation criterion considered do not measure this criterion\.

## 6Related Work

Metric evaluations through correlation studies with human judgment have been reported in several recent publications\. Although most of them consider the evaluation of NLG \(Natural Language Generation\) tasks other than RAG, the evaluation of these tasks involves issues common to those of evaluating RAG system responses\. This section offers a \(non\-exhaustive\) overview of these publications, grouped according to their evaluation methodologies\.

First, some of these publications rely on a methodology similar to ours \(correlation analysis on a single system\)\. We can cite, for example:\(Liuet al\.,[2024](https://arxiv.org/html/2607.07302#bib.bib9)\)which evaluates a system dedicated to evaluating various aspects of various natural language generation tasks, or\(Yeginbergenet al\.,[2025](https://arxiv.org/html/2607.07302#bib.bib5)\)which observes the correlation of several LLM\-as\-a\-judge systems with human evaluations of automatically generated counter\-arguments\. These studies suffer from the same methodological limitation as ours\.

Other studies evaluate metrics by measuring their adequacy with a preference order expressed on pairs of outputs corresponding to the same input\. More precisely: each task input is associated with two alternative outputs, for which a preference order is available; all pairs for which the metric score conforms to the preference order are considered successes, and the success ratio forms the metric’s performance score\. This approach helps guard against confounding factors related to question characteristics that limit the interpretations of our results\. Among studies using this approach, we can cite:\(Keet al\.,[2024](https://arxiv.org/html/2607.07302#bib.bib26); Zhuet al\.,[2025](https://arxiv.org/html/2607.07302#bib.bib25); Lambertet al\.,[2025](https://arxiv.org/html/2607.07302#bib.bib27)\)\. Some studies combine this approach with correlation analysis on a single system, for example:\(Xuet al\.,[2023](https://arxiv.org/html/2607.07302#bib.bib8); Kimet al\.,[2024](https://arxiv.org/html/2607.07302#bib.bib7); Xionget al\.,[2025](https://arxiv.org/html/2607.07302#bib.bib6)\)\.

Finally, some publications report correlation analyses involving multiple systems\. The statistical treatments performed can then vary, since the scores generated by a metric, like reference scores, are then arranged into a matrix with one row per system and one column per evaluated output\. There are indeed several ways to calculate a correlation between two matrices:\(Gaoet al\.,[2025](https://arxiv.org/html/2607.07302#bib.bib4)\)studies four of them, two of which seem relevant to us in the context of evaluating RAG metrics\. The first is to calculate the correlation of each system’s average scores noted by the metric with each system’s average reference score\. The second is to calculate, for each input, the correlation of metric scores with reference scores across different systems, then to average the correlations thus obtained\. These two approaches are also studied by\(Deutschet al\.,[2021](https://arxiv.org/html/2607.07302#bib.bib3)\)\. Both studies empirically conclude that the second approach has greater power to discriminate among tested metrics\. Moreover, it has been demonstrated that this approach considerably reduces the importance of confounding factors in correlations measured between machine translation evaluation metrics and human judgment\(Perrellaet al\.,[2024](https://arxiv.org/html/2607.07302#bib.bib10)\)\. A similar approach, applied in\(Dinhet al\.,[2024](https://arxiv.org/html/2607.07302#bib.bib24)\), consists of calculating correlation on all scores, normalized by inputs\.

## 7Research Directions

The empirical results of our study are in line with some of the results mentioned in the previous section, and confirm the usefulness of integrating several \(at least two\) RAG systems into correlation studies aimed at evaluating RAG metrics\. It should be noted that the study results will then be partially dependent on the chosen RAG systems: it is therefore appropriate to choose a set of systems representative of all systems to which we wish to apply the metrics\. Moreover, integrating too many systems risks making the annotation task costly\.

These difficulties open interesting research perspectives\. For example, how to estimate the probable contribution of an additional model or question to a correlation study? Is it possible to develop procedures to choose and adapt the number of systems and questions to integrate into the empirical study, as human evaluations are collected? Answers to such questions could enable professionals implementing RAG systems to validate their evaluation metrics more reliably while minimizing the costs associated with this validation\.

## References

- D\. Deutsch, R\. Dror, and D\. Roth \(2021\)A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods\.Transactions of the Association for Computational Linguistics9,pp\. 1132–1146\.External Links:ISSN 2307\-387X,[Link](https://doi.org/10.1162/tacl_a_00417),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00417)Cited by:[§6](https://arxiv.org/html/2607.07302#S6.p4.1)\.
- T\. A\. Dinh, C\. Mullov, L\. Bärmann, Z\. Li, D\. Liu, S\. Reiß, J\. Lee, N\. Lerzer, J\. Gao, F\. Peller\-Konrad, T\. Röddiger, A\. Waibel, T\. Asfour, M\. Beigl, R\. Stiefelhagen, C\. Dachsbacher, K\. Böhm, and J\. Niehues \(2024\)SciEx: Benchmarking Large Language Models on Scientific Exams with Human Expert Grading and Automatic Grading\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 11592–11610\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.647/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.647)Cited by:[§6](https://arxiv.org/html/2607.07302#S6.p4.1)\.
- S\. Es, J\. James, L\. Espinosa Anke, and S\. Schockaert \(2024\)RAGAs: Automated Evaluation of Retrieval Augmented Generation\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations,N\. Aletras and O\. De Clercq \(Eds\.\),St\. Julians, Malta,pp\. 150–158\.External Links:[Link](https://aclanthology.org/2024.eacl-demo.16/),[Document](https://dx.doi.org/10.18653/v1/2024.eacl-demo.16)Cited by:[§1](https://arxiv.org/html/2607.07302#S1.p2.1)\.
- M\. Gao, X\. Hu, L\. Lin, and X\. Wan \(2025\)Analyzing and Evaluating Correlation Measures in NLG Meta\-Evaluation\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 2199–2222\.External Links:ISBN 979\-8\-89176\-189\-6,[Link](https://aclanthology.org/2025.naacl-long.111/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.111)Cited by:[§6](https://arxiv.org/html/2607.07302#S6.p4.1)\.
- P\. Ke, B\. Wen, A\. Feng, X\. Liu, X\. Lei, J\. Cheng, S\. Wang, A\. Zeng, Y\. Dong, H\. Wang, J\. Tang, and M\. Huang \(2024\)CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 13034–13054\.External Links:[Link](https://aclanthology.org/2024.acl-long.704/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.704)Cited by:[§6](https://arxiv.org/html/2607.07302#S6.p3.1)\.
- S\. Kim, J\. Suk, S\. Longpre, B\. Y\. Lin, J\. Shin, S\. Welleck, G\. Neubig, M\. Lee, K\. Lee, and M\. Seo \(2024\)Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 4334–4353\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.248/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.248)Cited by:[§6](https://arxiv.org/html/2607.07302#S6.p3.1)\.
- N\. Lambert, V\. Pyatkin, J\. Morrison, L\. Miranda, B\. Y\. Lin, K\. Chandu, N\. Dziri, S\. Kumar, T\. Zick, Y\. Choi, N\. A\. Smith, and H\. Hajishirzi \(2025\)RewardBench: Evaluating Reward Models for Language Modeling\.InFindings of the Association for Computational Linguistics: NAACL 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 1755–1797\.External Links:ISBN 979\-8\-89176\-195\-7,[Link](https://aclanthology.org/2025.findings-naacl.96/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.96)Cited by:[§6](https://arxiv.org/html/2607.07302#S6.p3.1)\.
- M\. Liu, Y\. Shen, Z\. Xu, Y\. Cao, E\. Cho, V\. Kumar, R\. Ghanadan, and L\. Huang \(2024\)X\-Eval: Generalizable Multi\-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspects\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 8560–8579\.External Links:[Link](https://aclanthology.org/2024.naacl-long.473/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.473)Cited by:[§6](https://arxiv.org/html/2607.07302#S6.p2.1)\.
- S\. Perrella, L\. Proietti, A\. Scirè, E\. Barba, and R\. Navigli \(2024\)Guardians of the Machine Translation Meta\-Evaluation: Sentinel Metrics Fall In\!\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 16216–16244\.External Links:[Link](https://aclanthology.org/2024.acl-long.856/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.856)Cited by:[§6](https://arxiv.org/html/2607.07302#S6.p4.1)\.
- D\. Ru, L\. Qiu, X\. Hu, T\. Zhang, P\. Shi, S\. Chang, C\. Jiayang, C\. Wang, S\. Sun, H\. Li, Z\. Zhang, B\. Wang, J\. Jiang, T\. He, Z\. Wang, P\. Liu, Y\. Zhang, and Z\. Zhang \(2024\)RAGChecker: A Fine\-grained Framework for Diagnosing Retrieval\-Augmented Generation\.External Links:[Link](http://arxiv.org/abs/2408.08067),[Document](https://dx.doi.org/10.48550/arXiv.2408.08067)Cited by:[§1](https://arxiv.org/html/2607.07302#S1.p2.1),[§2\.3](https://arxiv.org/html/2607.07302#S2.SS3.p1.1)\.
- T\. Xiong, X\. Wang, D\. Guo, Q\. Ye, H\. Fan, Q\. Gu, H\. Huang, and C\. Li \(2025\)LLLaVA\-Critic: Learning to Evaluate Multimodal Models\.2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 13618–13628\.Note:Conference Name: 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\) ISBN: 9798331543648 Place: Nashville, TN, USAExternal Links:[Link](https://ieeexplore.ieee.org/document/11093772/),[Document](https://dx.doi.org/10.1109/CVPR52734.2025.01271)Cited by:[§6](https://arxiv.org/html/2607.07302#S6.p3.1)\.
- W\. Xu, D\. Wang, L\. Pan, Z\. Song, M\. Freitag, W\. Wang, and L\. Li \(2023\)INSTRUCTSCORE: Towards Explainable Text Generation Evaluation with Automatic Feedback\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 5967–5994\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.365/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.365)Cited by:[§6](https://arxiv.org/html/2607.07302#S6.p3.1)\.
- A\. Yeginbergen, M\. Oronoz, and R\. Agerri \(2025\)Dynamic Knowledge Integration for Evidence\-Driven Counter\-Argument Generation with Large Language Models\.Note:Version Number: 2Other ACL 2025External Links:[Link](https://arxiv.org/abs/2503.05328),[Document](https://dx.doi.org/10.48550/ARXIV.2503.05328)Cited by:[§6](https://arxiv.org/html/2607.07302#S6.p2.1)\.
- L\. Zhu, X\. Wang, and X\. Wang \(2025\)JudgeLM: Fine\-tuned Large Language Models are Scalable Judges\.arXiv\.Note:arXiv:2310\.17631 \[cs\]External Links:[Link](http://arxiv.org/abs/2310.17631),[Document](https://dx.doi.org/10.48550/arXiv.2310.17631)Cited by:[§6](https://arxiv.org/html/2607.07302#S6.p3.1)\.