Comprehensive Evaluation of Large Language Model Responses: A Multi-Factor Scoring System
Summary
This paper proposes a multi-factor scoring system for evaluating LLM responses, integrating accuracy, conciseness, factual consistency, readability, and coherence. Applied to the TruthfulQA dataset, it reveals strengths and limitations of mainstream models, offering a transparent evaluation framework.
View Cached Full Text
Cached at: 07/09/26, 07:49 AM
# Comprehensive Evaluation of Large Language Model Responses: A Multi-Factor Scoring System Source: [https://arxiv.org/abs/2607.06940](https://arxiv.org/abs/2607.06940) [View PDF](https://arxiv.org/pdf/2607.06940) > Abstract:The remarkable performance of large language models \(LLMs\) in linguistic tasks underscores an urgent need for comprehensive evaluation of their response quality\. Prevailing methods, often confined to singular dimensions, fall short of capturing the full spectrum of model capabilities\. This study introduces a multifactor scoring paradigm, integrating accuracy, conciseness, factual consistency, readability, and coherence, complemented by a graphical user interface \(GUI\) for visualizing outcomes\. Evaluations on the TruthfulQA dataset unveil mainstream LLMs' strengths in reasoning tasks \(peaking at a composite score of 0\.6104\) alongside pervasive limitations in navigating complex facts and ambiguities\. Transcending the narrow lens of traditional metrics, this framework offers a transparent, adaptable avenue to illuminate model potential and deficiencies\. Though presently focused on English tasks, its horizons beckon toward multilingual domains\. This work carves a novel path for knowledge engineering and model refinement\. ## Submission history From: Yiming Gai \[[view email](https://arxiv.org/show-email/c9f2229a/2607.06940)\] **\[v1\]**Wed, 8 Jul 2026 03:10:33 UTC \(1,508 KB\)
Similar Articles
Explain Like I'm 5 or Whatever I Choose: Evaluating the Interactive Potential of Language Model Responses
This paper proposes a framework for evaluating LLMs' ability to generate multiple responses to scientific queries at different language complexity levels. The study finds that models often vary complexity inconsistently, with Claude Sonnet 4.5 performing best but only shifting complexity correctly 46% of the time.
How Human-Like Are Large Language Models? A Register-Aware Linguistic Evaluation Framework
This paper introduces a register-aware linguistic evaluation framework to assess how human-like large language models (LLMs) are by comparing the distribution of 67 lexico-grammatical features between human and LLM-generated texts using Maximum Mean Discrepancy. Experiments across seven instruction-tuned open-source models and five registers show that no model perfectly matches human baselines, and closeness to human language varies by register rather than model size.
Can Large Language Models Reliably Code Qualitative Humanitarian Data? A Benchmark Study Against Human Expert Adjudication
This benchmark study evaluates 46 large language models against human experts for coding qualitative humanitarian data, finding that LLMs can achieve comparable reliability with structured prompts and reasoning, but require careful oversight for nuanced themes.
A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluation
This paper presents a red teaming framework for LLMs that uses a multi-role architecture to systematically uncover vulnerabilities, particularly in faithfulness. The framework demonstrated a 7.9% increase in attack success rate in QA tasks and highlights the impact of architectural choices over parameter scaling on model safety.
Automated Scoring of Arabic Text Using Large Language Models: A Literature Review
A literature review examining LLM-based approaches for automatic scoring of Arabic text, covering short answer grading and essay scoring, with a proposed taxonomy and comparative analysis.