From Benchmarking to Reasoning: A Dual-Aspect, Large-Scale Evaluation of LLMs on Vietnamese Legal Text

# A Dual-Aspect, Large-Scale Evaluation of LLMs on Vietnamese Legal Text

Accepted at the FISU Joint Conference on Artificial Intelligence (FJCAI 2026), Vietnam.
Source: https://arxiv.org/html/2604.16270

## From Benchmarking to Reasoning: A Dual-Aspect, Large-Scale Evaluation of LLMs on Vietnamese Legal Text

###### Abstract

The complexity of Vietnam's legal texts presents a significant barrier to public access to justice. While Large Language Models offer a promising solution for legal text simplification, evaluating their true capabilities requires a multifaceted approach that goes beyond surface-level metrics. This paper introduces a comprehensive dual-aspect evaluation framework to address this need. First, we establish a performance benchmark for four state-of-the-art large language models (GPT-4o, Claude 3 Opus, Gemini 1.5 Pro, and Grok-1) across three key dimensions: Accuracy, Readability, and Consistency. Second, to understand the "why" behind these performance scores, we conduct a large-scale error analysis on a curated dataset of 60 complex Vietnamese legal articles, using a novel, expert-validated error typology. Our results reveal a crucial trade-off: models like Grok-1 excel in Readability and Consistency but compromise on fine-grained legal Accuracy, while models like Claude 3 Opus achieve high Accuracy scores that mask a significant number of subtle but critical reasoning errors. The error analysis pinpoints Incorrect Example and Misinterpretation as the most prevalent failures, confirming that the primary challenge for current LLMs is not summarization but controlled, accurate legal reasoning. By integrating a quantitative benchmark with a qualitative deep dive, our work provides a holistic and actionable assessment of LLMs for legal applications.

## I. Introduction

The advent of Large Language Models (LLMs) offers a transformative promise: to democratize access to justice by translating complex, codified legislation into language the public can understand. In civil law systems like Vietnam's, this promise is particularly profound. The nation's laws, while comprehensive, are often shrouded in "legalese"—a dense legal terminology that creates a significant barrier for citizens seeking to understand their fundamental rights and obligations. LLMs present a potential tool to remove this barrier.

However, this promise is shadowed by a significant danger: the risk of generating fluent, plausible, yet inaccurate legal simplifications. To harness the potential of LLMs responsibly, we must first be able to measure their capabilities with nuance and depth. Initial evaluations, including our own preliminary work, have focused on setting performance benchmarks using surface-level metrics such as legal accuracy, user-perceived readability, and output consistency. Although these metrics provide a valuable "what"—quantifying which models perform better on the surface—they fundamentally fail to explain the "why." A model might achieve a high accuracy score by correctly summarizing the general rule for inheritance, yet completely miss a critical exception for a specific circumstance, a subtle but catastrophic reasoning error that superficial scores would mask. This limitation was a key piece of feedback on our initial research, which was criticized for its small sample size and lack of novel insight into the models' failure modes.

This highlights a critical gap in the literature: the absence of a holistic evaluation framework that marries large-scale quantitative benchmarking with deep qualitative diagnostics of legal reasoning. To fill this gap, this paper introduces a comprehensive, dual-aspect evaluation framework designed to be both broad and deep. We combine two distinct but complementary research thrusts:

1. **A Large-Scale Performance Benchmark:** We expand upon our previous methodology, applying the metrics of Accuracy, Readability, and Consistency to a large and diverse dataset of 60 complex Vietnamese legal articles to ensure the generalizability and statistical significance of our findings.
2. **An In-Depth Legal Reasoning Error Analysis:** We introduce and apply a novel, expert-validated error typology to the same outputs. This allows us to move beyond scores, dissecting each model's performance to identify and categorize the specific root causes of their reasoning failures.

By integrating these quantitative and qualitative approaches, we provide more than just a leaderboard of models. Our work offers a detailed diagnostic of the current capabilities and, more importantly, the systemic weaknesses of state-of-the-art LLMs in the Vietnamese legal domain. The findings reveal crucial trade-offs between performance aspects—such as readability versus factual fidelity—and pinpoint the core challenges that must be overcome to build truly reliable and trustworthy legal AI.

## II. Related Work

The application of LLMs in the legal domain has rapidly evolved, yet significant gaps remain, particularly for non-English languages and civil law systems. Our work is situated within several key research streams.

### II-A. Domain Adaptation and Pre-trained Legal Models

A foundational line of research demonstrates that domain-specific pre-training significantly improves performance on legal tasks. Models like LEGAL-BERT show that continued pre-training on large legal corpora enables models to better understand legal vocabulary and context compared to general-purpose models. This highlights the importance of domain-specific data, a challenge for languages like Vietnamese where large, digitized legal corpora are less abundant.

### II-B. Legal Text Simplification

Simplifying legal texts is a recognized task aimed at improving access to justice. Research in this area has explored various techniques, from unsupervised methods combining lexical replacement and sentence splitting to supervised approaches. However, a recurring challenge, as noted by Garimella et al., is the scarcity of parallel complex-simple legal corpora, making supervised training difficult. This scarcity elevates the importance of evaluating the zero-shot simplification capabilities of modern LLMs, which is a focus of our study.

### II-C. Evaluation and Reliability of Legal LLMs

As LLMs become more powerful, evaluating their reliability in high-stakes domains is paramount. Surveys like Lai et al. provide a broad overview of LLM applications in law, from drafting assistance to legal Q&A, while also outlining risks related to reasoning, transparency, and factual accuracy. A critical aspect of evaluation is understanding and mitigating "hallucinations." Dahl et al., for instance, developed a typology for legal hallucinations focused primarily on factual inaccuracies and citation errors in case law summarization and legal Q&A tasks.

While such work is foundational for identifying fact-based errors, our research addresses a different, yet equally critical, challenge: reasoning failures within the context of text simplification. Our nine-category error typology is specifically designed to capture nuances unique to this task. Unlike typologies focused on hallucinations (i.e., fabricating non-existent facts), our framework also categorizes subtle but critical errors like Oversimplification, Misinterpretation of legal terms, and failure to apply principles correctly in Incorrect Examples. This focus on simplification-specific reasoning errors, rather than just factual correctness, represents a key contribution that complements existing evaluation frameworks. It underscores the inadequacy of surface-level accuracy metrics and motivates our deep dive into error analysis.

### II-D. Vietnamese Legal Language Processing

In the context of Vietnamese, efforts to develop specialized legal AI resources are accelerating. On the research front, projects like ViGPT-Law and legal Q&A datasets such as VLQA and VNLAWQC are building foundational models and benchmarks, primarily for information retrieval and question answering. In parallel, the Vietnamese government has deployed practical tools, such as the Ministry of Justice's "Virtual Legal Assistant," aimed at improving public access to legal information.

However, a critical gap exists between creating these applications and fundamentally understanding their reliability. While existing efforts often emphasize factual correctness, they may overlook subtle but critical reasoning failures in generative tasks like simplification. Our work directly addresses this gap. While applied tools provide valuable services, our research offers a fundamental, diagnostic evaluation of the underlying LLM capabilities, identifying systemic failure modes through a novel error typology. This large-scale, systematic evaluation of generative simplification, viewed through the lens of legal reasoning, is a contribution that complements existing work. It not only advances technical benchmarks but also provides an actionable framework for auditing and improving the safety and reliability of public-facing legal AI systems in Vietnam.

## III. A Dual-Aspect Evaluation Framework

To provide a holistic assessment, we designed a framework that evaluates LLM performance from two complementary perspectives: high-level performance metrics and low-level error analysis. The entire framework was applied to a large dataset to ensure statistical significance and generalizability, while also enabling a deeper understanding of how different models succeed or fail across various aspects of legal reasoning.

### III-A. Data and Model Selection

**Models:** We selected four state-of-the-art LLMs representing the current forefront of commercial AI: **GPT-4o** (OpenAI), **Claude 3 Opus** (Anthropic), **Gemini 1.5 Pro** (Google), and **Grok-1** (xAI). These models were chosen based on three criteria: (1) Market Dominance: They represent the flagship models from the leading AI laboratories; (2) Architectural Diversity: They embody different training philosophies and context window capabilities (e.g., Gemini's long context vs. Grok's MoE architecture); and (3) Accessibility: They are currently the most accessible high-performance tools for general users, making their evaluation highly relevant for public legal access.

**Dataset:** We curated a dataset of 60 legal articles via purposive sampling. This large-scale approach prioritizes both depth and breadth. The articles were selected from three core legislative texts: the Penal Code 2015 (20 articles), the Civil Code 2015 (20 articles), and the Land Law 2024 (20 articles). The selection criteria focused on articles known for their complexity, including those with: (a) multiple exceptions and conditions, (b) abstract legal terminology, and (c) requirements for sequential logical reasoning. This ensures a challenging and representative testbed for the LLMs. By combining civil, criminal, and land law, the dataset reflects a wide range of legal reasoning challenges encountered in both daily life and specialized legal contexts. Moreover, this carefully balanced corpus provides sufficient variation to stress-test the adaptability of different model families under comparable conditions.

**Task:** For each article, models were given a consistent, zero-shot prompt asking them to act as a legal assistant and explain the law in simple terms with a practical example for a layperson. The specific prompt used was: "Act as a legal assistant. Explain the following article [Article Content] in simple terms for a layperson and provide a practical example". Each model generated two responses per article (at temperature 0.2), creating a corpus of 480 outputs (4 models × 60 articles × 2 runs) for evaluation. This experimental design enables us to evaluate not only the accuracy and accessibility of the generated simplifications but also the stability of model behavior across repeated generations. Notably, this dual-component prompt—requiring both explanation (a summarization task) and example generation (an application task)—allowed us to observe that models consistently struggled more with the latter, highlighting a core challenge in legal reasoning over mere text simplification.

### III-B. Phase 1: Overall Performance Benchmarking

This phase quantifies the overall quality of the generated texts across three user-centric dimensions, providing a high-level comparison of the models.

#### III-B-1. Metric 1: Legal Accuracy

This metric assessed the legal precision of the outputs. It was rated by a team of five trained law students on a 1-5 Likert scale. To ensure reliability, each output was rated by at least two students, and any significant discrepancies were resolved through discussion. The metric is a weighted average of four sub-criteria:

- **Content Preservation (40%):** Fidelity to the core legal rule.
- **Completeness (30%):** Inclusion of all essential elements and conditions.
- **Clarity (20%):** How easy the legal definition is to understand.
- **Example Relevance (10%):** Appropriateness of the provided example.

The weighting scheme was established based on the principle of 'Safety First' in legal advice. Content Preservation is assigned the highest weight (40%) because a simplification that omits core legal meaning poses the greatest risk of misinformation. Completeness (30%) follows, ensuring no critical conditions are missed. Clarity (20%) and Example Relevance (10%) are weighted lower, as a legally accurate but dry explanation is preferable to a fluent but incorrect one.

#### III-B-2. Metric 2: Readability

This metric assessed the ease of understanding for a non-expert audience. It was rated by a diverse group of 253 non-expert participants recruited via university mailing lists and social media. Each participant rated a random subset of 10-15 outputs on a 1-5 Likert scale across three weighted sub-criteria:

- **Ease of Language (40%):** Simplicity of vocabulary and sentence structure.
- **Structural Coherence (30%):** Logical flow, formatting, and organization.
- **Utility of Examples (30%):** How helpful the example was in clarifying the concept.

For readability, Ease of Language is weighted highest (40%) as it directly addresses the barrier of 'legalese' for laypeople, followed equally by Structural Coherence (30%) and Utility of Examples (30%), which support comprehension and retention.

#### III-B-3. Metric 3: Consistency

This metric measured the stability of a model's output across two independent runs. This composite metric combined automated scores and human judgment:

- **Semantic Consistency (30%):** Cosine similarity between sentence embeddings (using paraphrase-multilingual-MiniLM-L12-v2).
- **Linguistic Consistency (30%):** Sentence-level BLEU score to measure phrasing overlap.
- **Length Consistency (10%):** Stability in output length (word count).
- **Manual Consistency Score (30%):** Human evaluator rating of meaning preservation across runs.

**Length Consistency (10%):** While output length is not a direct measure of reasoning quality, significant variance in length for identical prompts (e.g., producing a brief summary versus an extended explanation) indicates instability that could undermine user trust in the model's reliability.