Comprehensive Evaluation of Large Language Model Responses: A Multi-Factor Scoring System
Summary
This paper proposes a multi-factor scoring system for evaluating LLM responses, integrating accuracy, conciseness, factual consistency, readability, and coherence. Applied to the TruthfulQA dataset, it reveals strengths and limitations of mainstream models, offering a transparent evaluation framework.
View Cached Full Text
Cached at: 07/09/26, 07:49 AM
# Comprehensive Evaluation of Large Language Model Responses: A Multi-Factor Scoring System Source: [https://arxiv.org/abs/2607.06940](https://arxiv.org/abs/2607.06940) [View PDF](https://arxiv.org/pdf/2607.06940) > Abstract:The remarkable performance of large language models \(LLMs\) in linguistic tasks underscores an urgent need for comprehensive evaluation of their response quality\. Prevailing methods, often confined to singular dimensions, fall short of capturing the full spectrum of model capabilities\. This study introduces a multifactor scoring paradigm, integrating accuracy, conciseness, factual consistency, readability, and coherence, complemented by a graphical user interface \(GUI\) for visualizing outcomes\. Evaluations on the TruthfulQA dataset unveil mainstream LLMs' strengths in reasoning tasks \(peaking at a composite score of 0\.6104\) alongside pervasive limitations in navigating complex facts and ambiguities\. Transcending the narrow lens of traditional metrics, this framework offers a transparent, adaptable avenue to illuminate model potential and deficiencies\. Though presently focused on English tasks, its horizons beckon toward multilingual domains\. This work carves a novel path for knowledge engineering and model refinement\. ## Submission history From: Yiming Gai \[[view email](https://arxiv.org/show-email/c9f2229a/2607.06940)\] **\[v1\]**Wed, 8 Jul 2026 03:10:33 UTC \(1,508 KB\)
Similar Articles
Beyond Accuracy: A Multidimensional Evaluation of Statistical Reasoning in Large Language Models
This paper proposes a multidimensional evaluation framework for assessing statistical reasoning in large language models, combining response accuracy, response behavior, structural topic modeling, and lexical similarity analysis across 15 LLMs and 90 exam questions. It finds that accuracy alone is insufficient to characterize LLM statistical reasoning and that vendor-specific stylistic differences exist.
(Towards) Scalable Reliable Automated Evaluation with Large Language Models
This paper proposes a scalable, domain-agnostic framework for automated LLM evaluation that uses pairwise comparisons by multiple LLMs and an Elo rating system to approximate expert judgments, reducing the need for human intervention.
Explain Like I'm 5 or Whatever I Choose: Evaluating the Interactive Potential of Language Model Responses
This paper proposes a framework for evaluating LLMs' ability to generate multiple responses to scientific queries at different language complexity levels. The study finds that models often vary complexity inconsistently, with Claude Sonnet 4.5 performing best but only shifting complexity correctly 46% of the time.
How Human-Like Are Large Language Models? A Register-Aware Linguistic Evaluation Framework
This paper introduces a register-aware linguistic evaluation framework to assess how human-like large language models (LLMs) are by comparing the distribution of 67 lexico-grammatical features between human and LLM-generated texts using Maximum Mean Discrepancy. Experiments across seven instruction-tuned open-source models and five registers show that no model perfectly matches human baselines, and closeness to human language varies by register rather than model size.
A Consensus-Based Framework for Relative Preference Evaluation of Large Language Models
This paper introduces a consensus-based framework for evaluating LLM responses by having a panel of models rank anonymized outputs, producing a Relative Intelligence Index (RII) as a proxy for relative quality across domains.