L-MAD: A Systematic Evaluation of Multi-Agent Debate Structures in Legal Reasoning

# A Systematic Evaluation of Multi-Agent Debate Structures in Legal Reasoning
Source: [https://arxiv.org/html/2607.09099](https://arxiv.org/html/2607.09099)
Hoang\-Trung NguyenHuu\-Dong NguyenTruong Dinh DoThi\-Hai\-Yen VuongLe\-Minh Nguyen

###### Abstract

While multi\-agent debate \(MAD\) frameworks have shown significant potential in general reasoning, their effectiveness in highly structured, knowledge\-heavy legal domains remains under\-explored\. In this work, we introduce the Legal Multi\-Agent Debate \(L\-MAD\) framework to systematically evaluate different debate structures and aggregation methods within Legal Textual Entailment\. By assigning distinct expert personas to multiple agents, L\-MAD improves upon strong single\-agent baselines by up to 8%\. Furthermore, analyzing how debate scales reveals a clear trade\-off: increasing the agent population reduces inconsistency and improves accuracy, whereas extending discussion rounds induces a detrimentalover\-deliberation driftwhere agents reinforce each other’s mistakes\. Ultimately, our findings outline the practical boundaries and safety margins of deploying collaborative multi\-agent systems in high\-stakes legal reasoning environments\.

Machine Learning, ICML

## 1Introduction

Large language models \(LLMs\) show remarkable capabilities in general reasoning, but they frequently struggle in high\-stakes, knowledge\-heavy domains that demand strict logic\. Legal Textual Entailment \(LTE\)\(Aokiet al\.,[2022](https://arxiv.org/html/2607.09099#bib.bib29); Bilginet al\.,[2024](https://arxiv.org/html/2607.09099#bib.bib41)\)is a challenging test for these models, requiring them to determine whether specific legal statutes apply to a given factual scenario\(Aletraset al\.,[2016](https://arxiv.org/html/2607.09099#bib.bib39); Zhonget al\.,[2020](https://arxiv.org/html/2607.09099#bib.bib40)\)\. While specialized models like LegalBERT\(Chalkidiset al\.,[2020](https://arxiv.org/html/2607.09099#bib.bib44)\)and Lawformer\(Xiaoet al\.,[2021](https://arxiv.org/html/2607.09099#bib.bib45)\)outperform generalist baselines, standard single\-model inference is still limited by a model’s basic reasoning capacity and its tendency to make factual or logical errors\.

![Refer to caption](https://arxiv.org/html/2607.09099v1/figs/framework.png)

Figure 1:Illustration of decision\-making protocols used in this study\.Multi\-Agent Debate \(MAD\) frameworks offer a promising way to overcome these limitations by allowing models to spend more computing power during inference\(Nieet al\.,[2020](https://arxiv.org/html/2607.09099#bib.bib47); Choiet al\.,[2026](https://arxiv.org/html/2607.09099#bib.bib34)\)\. By assigning distinct expert personas to multiple LLM agents, these frameworks let models iterate and refine their arguments to arrive at a better final answer\(Kaesberget al\.,[2025b](https://arxiv.org/html/2607.09099#bib.bib48)\)\. However, how these debate frameworks perform in highly structured, rule\-bound fields like law remains underexplored\. While open\-ended debate typically helps in general tasks, it introduces unique challenges in legal reasoning\. The trade\-offs between forcing agents to agree versus letting them vote independently—and the risk of errors multiplying over several rounds—have not yet been systematically analyzed\.

To bridge this gap, we introduce Legal Multi\-Agent Debate \(L\-MAD\), a framework designed to analyze different multi\-agent debate structures under strict legal rules \(Figure[1](https://arxiv.org/html/2607.09099#S1.F1)\)\. Unlike prior work that mainly focuses on connecting LLMs to external tools, L\-MAD provides a controlled environment to evaluate core aggregation methods\. Specifically, we compareforced consensusstrategies \(where agents debate until they agree\) againstvotingapproaches \(where agents maintain independent reasoning paths\)\.

Our empirical results show that L\-MAD significantly improves performance, outperforming strong single\-agent baselines by up to 8% in some settings\. Crucially, we find that the best debate strategy depends heavily on the underlying model’s capability\. Forcing a consensus works well with highly capable models \(e\.g\., 30B\+ parameters\) by encouraging constructive debate, whereas independent voting protects smaller models \(e\.g\., 8B parameters\) from superficial agreement\. Furthermore, exploring how debate scales reveals a clear trade\-off: adding more agents to the population reduces inconsistency and improves accuracy, but adding more discussion rounds causesover\-deliberation drift—a behavior where agents echo and validate each other’s mistakes, ultimately degrading performance\.

In summary, our core contributions are:

- •A systematic evaluation of multi\-agent debate frameworks for decision\-making in high\-stakes, knowledge\-dense legal domains\.
- •A clear analysis of the scaling trade\-offs between increasing the number of participating agents versus extending the number of debate rounds\.
- •Actionable insights into the boundaries and safety margins of deploying multi\-agent LLM systems in collaborative legal reasoning environments\.

## 2Preliminaries

### 2\.1Legal Textual Entailment

We employ Legal Textual Entailment to assess the ability of LLMs to accurately deduce legal outcomes from factual descriptions and statutory rules\. By emphasizing subtle linguistic and doctrinal distinctions, this task rigorously evaluates whether a model possesses a genuine comprehension of legal theory\. An illustrative example of the task \(official translation from Japanese\) is presented below:

> Premise:\(Juridical Acts by Adult Wards under Guardianship\) Article 9 A juridical act performed by an adult ward is voidable; provided, however, that this does not apply to the purchase of daily necessities or to any other act involved in day\-to\-day life\. Hypothesis:If a person under adult guardianship purchases daily necessities without the consent of the adult guardian, the adult guardian may cancel the contract involving the purchase\. Entailment:No

More formally, let𝒮=\{S1,S2,…,Sn\}\\mathcal\{S\}=\\\{S\_\{1\},S\_\{2\},\\dots,S\_\{n\}\\\}denote a set of relevant statutory articles and letQQrepresent a legal hypothesis\. The Legal Textual Entailment task can be cast as a binary classification problem mapping the tuple\(𝒮,Q\)\(\\mathcal\{S\},Q\)to a labely∈\{1,0\}y\\in\\\{1,0\\\}, corresponding toE​n​t​a​i​l​s​\(𝒮,Q\)Entails\(\\mathcal\{S\},Q\)andE​n​t​a​i​l​s​\(𝒮,¬Q\)Entails\(\\mathcal\{S\},\\neg Q\), respectively\.

The primary evaluation metric for this task is classification accuracy\. Given that the benchmark datasets are rigorously balanced between positive and negative instances, a random guessing baseline or a naive uniform\-prediction heuristic yields an expected accuracy of approximately0\.500\.50\.

### 2\.2Prompting Techniques

##### Standard Prompting and In\-Context Learning\.

Standard prompting elicits an answeraafrom a model parameterized byθ\\thetagiven a queryqq, formally modeling the conditional distributionPθ​\(a∣q\)P\_\{\\theta\}\(a\\mid q\)\. In\-context learning\(Brownet al\.,[2020](https://arxiv.org/html/2607.09099#bib.bib21)\)enhances this generation process by conditioning the language model on a small set of exemplars𝒟=\{d1,d2,…,dn\}\\mathcal\{D\}=\\\{d\_\{1\},d\_\{2\},\\dots,d\_\{n\}\\\}, effectively modelingPθ​\(a∣𝒟,q\)P\_\{\\theta\}\(a\\mid\\mathcal\{D\},q\)without requiring parameter updates\.

##### Legal Reasoning CoT \(IRAC\)\.

Eliciting intermediate reasoning steps significantly improves the proficiency of LLMs on complex cognitive tasks\(Weiet al\.,[2022](https://arxiv.org/html/2607.09099#bib.bib25)\)\. Standard CoT extends the in\-context learning paradigm by augmenting each exemplar with a rationalerir\_\{i\}, such thatdi=\(qi,ri,ai\)d\_\{i\}=\(q\_\{i\},r\_\{i\},a\_\{i\}\)\. The complexity and structural rigor ofrir\_\{i\}directly correlate with downstream performance\(Fuet al\.,[2022](https://arxiv.org/html/2607.09099#bib.bib37)\)\. In the juridical domain,Yuet al\.\([2022](https://arxiv.org/html/2607.09099#bib.bib9)\)specialized this approach via the IRAC framework—a canonical legal reasoning structure comprising four discrete stages: Issue \(identifying the legal controversy\), Rule \(retrieving the governing statute\), Application \(mapping the rule to the factual matrix\), and Conclusion \(deriving the final legal verdict\)\.

##### Self\-Consistency\.

To mitigate the stochasticity of single\-path decoding, self\-consistency\(Wanget al\.,[2022](https://arxiv.org/html/2607.09099#bib.bib10)\)samples a diverse set of reasoning paths and marginalizes over them to find the most robust answer\. By drawing multiple independent completions from the model, the final predictiona∗a^\{\*\}is obtained via majority voting, defined asa∗=arg⁡maxa​∑i𝕀​\(ai=a\)a^\{\*\}=\\arg\\max\_\{a\}\\sum\_\{i\}\\mathbb\{I\}\(a\_\{i\}=a\)\. While this ensemble\-like method effectively increases the likelihood of finding the correct reasoning pathway, it remains upper\-bounded by the native inferential limits of the base model\.

### 2\.3Multi\-Agent Debate for Decision\-Making

Multi\-Agent Debate is a collaborative framework in which multiple language\-model agents engage in structured interaction—typically in the form of iterative exchanges or discussions —to solve tasks such as question answering or text generation\. In a typical MAD protocol, each agent independently generates an initial response and then engages in a series of debate rounds\. At roundtt, an agent receives the original question along with responses from its peers at roundt−1t\-1, prompting the model to update its response accordingly\. This iterative process is designed to leverage diverse reasoning paths and peer wisdom, potentially enhancing the overall decision quality\. After all rounds of debate, the final answer is typically derived through an aggregation mechanism, such as majority voting\.

## 3Methodology

In this section, we formalize the architecture of our multi\-agent framework, detailing the deliberation topology, the prompt\-based response conditioning, and the decision aggregation protocols\.

### 3\.1Framework Configuration and Topology

Our multi\-agent debate framework is parameterized by three foundational elements: a deliberation topology, a decision protocol, and an agent\-specific response generator\. We initialize each debate session with three distinct, automatically generated expert personas, adapting the methodology ofKimet al\.\([2024](https://arxiv.org/html/2607.09099#bib.bib20)\)\. Empirical evidence fromYinet al\.\([2023](https://arxiv.org/html/2607.09099#bib.bib14)\)demonstrates that a tripartite configuration offers an optimal theoretical trade\-off between viewpoint diversity and computational overhead\. By instantiating distinct personas, we effectively inject domain\-specific inductive biases into the reasoning process, ensuring a robust exploration of the solution space\.

During deliberation, agents exchange information over discrete communication rounds\. In each round, agents operate asynchronously, attending to the broadcasted messages of their peers from previous rounds based on a sliding context window \(restricted to the two most recent turns\)\. This truncated context serves as an attention regularization mechanism to limit context dilution and prevent catastrophic drift from the original hypothesis\. The exact termination condition of the debate is strictly governed by the chosen decision protocol\. For consensus protocols, agents explicitly signal agreement or disagreement per turn, iteratively refining their stance until the threshold is met\. For voting protocols, agents communicate for a fixed horizon \(e\.g\., three rounds\) before casting terminal votes\.

### 3\.2Response Generation and Decision Protocols

The semantic structure of the agent prompt heavily influences the trajectory of multi\-agent deliberation\. We employ aSimple Response Generatoras our default template, which conditions agents to evaluate preceding arguments neutrally and without adversarial bias, sustaining constructive dialogue while accommodating the persona formulation\.

To aggregate heterogeneous reasoning paths into a final output, we implement and evaluate two distinct classes of decision\-making protocols:

Consensus\-Based Protocols\.These protocols enforce active negotiation, driving agents to mathematically converge on a unified hypothesis\. A final decision is formalized only when a predefined threshold of inter\-agent agreement is satisfied\. We define three progressive thresholds: majority \(\>50%\>50\\%agreement\), supermajority \(\>66%\>66\\%agreement\), and unanimity \(100%100\\%agreement\)\. Building upon the standard majority\-vote mechanism ofYinet al\.\([2023](https://arxiv.org/html/2607.09099#bib.bib14)\), we employ a hybrid consensus approach that demands unanimity for initial convergence while defaulting to a supermajority threshold to prevent deadlock at the turn limit\.

Voting\-Based Protocols\.In contrast to forced consensus, voting protocols decouple the deliberation phase from the final aggregation step, permitting agents to maintain isolated, parallel reasoning tracks throughout the debate\. Following the communication horizon, the protocol halts argumentation and prompts a discrete voting mechanism\. In the event of a tie, the framework triggers a single supplementary deliberation round, followed by a tie\-breaking vote\. Inspired byYanget al\.\([2024](https://arxiv.org/html/2607.09099#bib.bib19)\), we utilize a ranked\-choice voting scheme wherein each agent issues a strict preference ordering over the candidate solutions \(e\.g\., Borda count equivalent\)\. The hypothesis maximizing the aggregated rank score is established as the final legal entailment\.

## 4Experiment

### 4\.1Benchmarks

Table 1:Statistics of COLIEE benchmarks for three years\.For the experiments, we used the training and test datasets provided in COLIEE 2026 Task 4\. Although no model fine\-tuning was performed in this study, the training data were utilized to select examples for few\-shot prompting\.

To analyze year\-by\-year performance variation, we used evaluation data from three years, spanning from 2024 to 2026\. The COLIEE 2026 formal run uses the 2025 test subset, while our methods comparisons are reported as a post\-hoc controlled comparison over 2024\-2025\. Problem IDs in the dataset follow the Japanese era naming convention, for example, R05 to 2023, and so on up to R07 for 2025\. Each sample in the training and test datasets consists of the following information: Relevant article, Query, Answer \(Y: correct/N: incorrect\)\. The statistics of benchmarks are reported in Table[1](https://arxiv.org/html/2607.09099#S4.T1)\.

### 4\.2Metric

We used accuracy as the evaluation metric to assess the performance of each method\. The model outputs were parsed from the fixed output labels: “Answer:Y” \(Correct/True\) and “Answer:N” \(Incorrect/False\)\. Accuracy was computed by comparing the parsed label with the ground truth label\.

### 4\.3Baselines and Implementation Details

We compare L\-MAD against various prompting baselines, including Zeroshot, Legal theory chain\-of\-thought \(IRAC\)\(Yuet al\.,[2022](https://arxiv.org/html/2607.09099#bib.bib9)\), Fewshot, and Self\-Consistency\(Wanget al\.,[2022](https://arxiv.org/html/2607.09099#bib.bib10)\)\.

The LLMs used for the experiments are Qwen3\-8/32B, Qwen3\-30B\-A3B\(Yanget al\.,[2025](https://arxiv.org/html/2607.09099#bib.bib11)\), Llama3\.1\-8B\(Dubeyet al\.,[2024](https://arxiv.org/html/2607.09099#bib.bib12)\), all instruction following versions\. All experiments are conducted on a single A6000 GPU with 49GB of memory\. For implementation, we set the temperature to 0\.1 for generating diverse reasoning paths\. All models are loaded in full precisionbfloat16\. The self\-consistency method is aggregated on three independent samples\. Few\-shot examples are retrieved using BM25 on the training set\. The number of examples is set at 6\.

## 5Result

Table 2:Task performance for two decision protocols \(voting and consensus\-based\) on four LLM backbones\. All values and standard deviations are multiplied by 100\. Bold and underline indicate the highest and second\-highest results per dataset\. Green cells indicate that the L\-MAD variant outperforms the Zeroshot baseline; red indicates it does not\. Standard deviation over three runs\.### 5\.1Performance of Multi\-Agent Debate framework

Table[2](https://arxiv.org/html/2607.09099#S5.T2)presents the efficacy of the L\-MAD framework, which scales non\-linearly with the inherent competence of the base language model\. Rather than providing a universal performance boost, L\-MAD acts as a cognitive amplifier that reveals three distinct behavioral regimes\.

First, for highly capable but non\-saturating models \(such as Qwen3\-8B and Qwen3\-30B\), L\-MAD serves as a powerful corrective mechanism\. On average, the consensus and voting protocols outperform robust single\-agent baselines like self\-consistency and IRAC\. In this ”sweet spot,” the debate structure successfully aggregates diverse reasoning paths, allowing agents to fact\-check one another and converge on more nuanced legal entailments\.

Second, we observe a ceiling effect with the most advanced architecture \(Qwen3\-32B\)\. As a model’s standalone reasoning approaches state\-of\-the\-art, advanced single\-agent prompting—particularly self\-consistency—saturates the task performance, matching or even marginally exceeding the multi\-agent setup\. This suggests that when a model possesses sufficiently robust internal semantic representations of legal logic, the computational overhead and complexity of multi\-agent collaboration may offer diminishing returns\.

Finally, the results expose a critical “competence threshold” for multi\-agent systems\. When deployed on less\-capable models \(e\.g\., Llama3\.1\-8B\), the L\-MAD protocols actively degrade performance relative to isolated single\-agent baselines\. From a theoretical standpoint, this indicates that without a minimum baseline of deductive reasoning, multi\-agent frameworks become vulnerable to collaborative hallucination\. Instead of correcting mistakes, weak agents blindly validate each other’s flawed premises, cascading errors, and forming a confidently incorrect echo chamber\.

### 5\.2Performance of Decision Protocols

Comparing consensus and voting mechanisms reveals a clear trade\-off between individual agent independence and collective alignment\. Consensus protocols require agents to actively negotiate and converge on a single shared conclusion across debate rounds\. In contrast, voting protocols preserve independent, parallel reasoning paths, delaying aggregation until the final step\.

Our empirical results indicate that the optimal decision protocol depends heavily on the reasoning capacity of the underlying model\. For larger models \(e\.g\., 30B\+ parameters\), consensus protocols generally outperform voting\. Because these models possess stronger internal logical consistency, they can engage in constructive evaluation—effectively identifying errors in competing arguments, correcting them, and synthesizing a more accurate final rationale\. In this regime, the forced alignment mechanism functions as a rigorous, iterative peer\-review process\.

Conversely, for smaller models \(e\.g\., 8B parameters\), the voting mechanism significantly outperforms the consensus approach\. Forced consensus in mid\-tier models frequently induces sycophancy or premature convergence, where agents tend to uncritically adopt the initial or dominant reasoning path, even if it is fundamentally flawed\. Voting mitigates this vulnerability by maintaining the independence of each agent’s analysis\. By delaying aggregation to the final stage, voting successfully leverages a ”wisdom of the crowd” effect, preventing the final verdict from being derailed by a single, incorrect agent during the discussion phase\.

### 5\.3Number of Agents and Discussion Rounds

![Refer to caption](https://arxiv.org/html/2607.09099v1/x1.png)Figure 2:Left: System accuracy as a function of the number of deliberation rounds prior to voting\.Right: System accuracy as a function of the number of participating agents\. The final answers are derived using the ranked voting decision protocol\. Trends are indicated by linear regression fits \(dotted lines\)\.Prior literature presents conflicting hypotheses on the scaling properties of multi\-agent debate\. While some studies suggest that scaling both the number of participating agents and deliberation rounds monotonically improves performance via test\-time computational scaling\(Yinet al\.,[2023](https://arxiv.org/html/2607.09099#bib.bib14); Wanget al\.,[2024](https://arxiv.org/html/2607.09099#bib.bib17)\), others warn that extended discourse may induce topic drift or echo\-chamber effects\(Becker,[2024](https://arxiv.org/html/2607.09099#bib.bib16)\)\. To optimize the compute\-accuracy trade\-off and rigorously evaluate these claims within the domain of legal reasoning, we systematically ablate the multi\-agent topology parameter space\.

Specifically, we independently scale the agent population size \(N∈\{2,3,4,5\}N\\in\\\{2,3,4,5\\\}\) and the deliberation depth constraint \(T∈\{1,2,3,4,5\}T\\in\\\{1,2,3,4,5\\\}\) on the Qwen3\-8B backbone using the ranked\-voting protocol\. To isolate the marginal effect of each dimension, accuracy with respect to the number of rounds is averaged across all agent configurations, and conversely, accuracy with respect to the agent count is marginalized across all round limits\.

The results reveal a clear divergence in how these two scaling axes govern task performance \(Figure[2](https://arxiv.org/html/2607.09099#S5.F2)\)\. Broadening the agent population yields a modest upward trend in accuracy \(Figure[2](https://arxiv.org/html/2607.09099#S5.F2), right\), analogous to the variance reduction and coverage maximization observed in self\-consistency techniques\(Wanget al\.,[2022](https://arxiv.org/html/2607.09099#bib.bib10)\)\. Expanding the diversity of initial propositions generally benefits the final aggregation step by providing a broader sample of reasoning paths\.

Conversely, scaling the deliberation depth exhibits a negative trajectory \(Figure[2](https://arxiv.org/html/2607.09099#S5.F2), left\)\. Rather than self\-refining toward the ground truth, agents subjected to extended discussion rounds demonstrate gradual performance degradation\. This empirical finding challenges the assumed universal benefits of iterative self\-correction\(Madaanet al\.,[2023a](https://arxiv.org/html/2607.09099#bib.bib18)\), aligning instead with emerging skepticism regarding LLM self\-refinement stability without external feedback\(Huanget al\.,[2023](https://arxiv.org/html/2607.09099#bib.bib13)\)\. In specialized domains like legal entailment, extended deliberation intuitively amplifies the risk of adversarial drift, where agents over\-critique valid logic and jointly regress toward suboptimal hypotheses\.

### 5\.4Qualitative Analysis

To better understand the quantitative results, we conducted a systematic review of the debate transcripts across both successful and failed entailment cases\. This evaluation categorizes the distinct behaviors of multi\-agent reasoning into its fundamental strengths and practical limitations within rule\-based environments\.

#### 5\.4\.1Advantages of Multi\-Agent Debate

##### Distributed Interpretation and Error Correction\.

In highly capable models \(e\.g\., Qwen3\-8B and Qwen3\-30B\), performance improvements are driven by substantive corrections rather than simple output aggregation\. When multi\-agent systems correct a unanimous failure of a single agent, the success typically stems from dividing the analytical effort\. Figure[3](https://arxiv.org/html/2607.09099#S5.F3)illustrates this on a complex statute\. A single agent often focuses prematurely on a primary clause and misses critical exceptions\. In contrast, assigning different roles allows agents to independently parse various parts of the legal text, enabling one agent to identify an overlooked exception and correct the group’s understanding\.

Article 102Gold: Yes Baseline: No/No/No MAD: Yes
Statement\.An act performed by a restricted\-capacity person as legal agent for another restricted\-capacity person may be cancelled on grounds of incapacity\.
Article\.An act by a restricted\-capacity person as agent cannot be cancelled on grounds of capacity; provided, however, that this does not apply when acting as legal agent for another restricted\-capacity person\.
Baseline \(R1, R3\): Reads only the main clause\. The act cannot be cancelled\. No
Agent 1 \(R1\): The exception clause carves out an exception that applies here\. Yes
Agent 2 \(R1\): Agree\. The exception explicitly allows cancellation in this case\. Yes
Agent 3 \(R1\): Agree\. Input matches the exception\. Yes
Final:Yes \(correct\)\. A clause that zeroshot reads past is surfaced by the multi\-agent parse\.Figure 3:MAD\-win on Civil Code Article 102\. The zeroshot baseline misses the statutory exception in all three runs; L\-MAD identifies it on the first turn\.
##### Multi\-Step Reasoning Convergence\.

Building on the ability to distribute reading, the debate framework also effectively connects multiple legal provisions\. The multi\-agent setup excels when a legal outcome requires combining two distinct articles \(e\.g\., Articles 253 and 254 in Figure[4](https://arxiv.org/html/2607.09099#S5.F4)\)\. While a single agent might stop after reading only one provision, collaborating agents can sequentially build upon each other’s points to bridge logical gaps\. This behavior constructs a chain of reasoning that single\-pass prompting often fails to produce, which explains the heightened performance on datasets with complex legislative requirements\.

Articles 253 & 254Gold: Yes Baseline: No/No/No MAD: Yes
Statement\.Co\-owner A paid management costs\. Co\-owner B then transferred their share to C\. A may claim reimbursement from C proportional to C’s share\.
Article 253\.Each co\-owner pays management costs in proportion to their share\.
Article 254\.A claim that a co\-owner holds against another co\-owner may also be exercised against a successor in interest\.
Baseline \(R1, R3\): Considers only Article 253\. Successor’s liability is unclear\. No
Agent 1 \(R1\): Article 253 imposes proportional liability; Article 254 transmits the claim to successors\. C inherits the obligation\. Yes
Agent 2 \(R1\): Agree\. The two articles together support the input\. Yes
Agent 3 \(R1\): Agree\. The chain is consistent with the statutory text\. Yes
Final:Yes \(correct\)\. The two\-article chain is resolved on the first turn of debate\.Figure 4:MAD\-win on co\-ownership reimbursement \(Articles 253 and 254\)\. Zeroshot fixates on a single article; L\-MAD integrates both provisions in one turn\.
##### Voting Divergence as a Reliability Signal\.

Evaluating the voting framework reveals that disagreement among agents serves as a practical, natural indicator of task difficulty\. For mid\-tier models, instances with unanimous votes achieved approximately 70% accuracy, while split votes—making up about 25% of queries—dropped to 48% accuracy\. Protocols that force consensus obscure this metric by requiring all agents to align\. By allowing agents to reason independently until the final decision, voting frameworks naturally expose when a model has reached the limits of its understanding\. In real\-world applications, this allows administrators to automatically route uncertain, split decisions to human experts while confidently automating unanimous cases\.

#### 5\.4\.2Failure Cases of Multi\-Agent Debate

##### Collaborative Hallucination and the Over\-Qualification Cascade\.

For less capable models, such as Llama3\.1\-8B, the debate format introduces a compounding failure mode where errors are validated rather than corrected\. As shown in Figure[5](https://arxiv.org/html/2607.09099#S5.F5), this cascade begins when an agent introduces a flawed interpretation or an incorrect hesitation\. Because subsequent agents lack the reasoning capacity to recognize the error, they adopt the incorrect premise and treat it as a factual constraint\. Our analysis shows that expressions of uncertainty or hesitation appear ten times more frequently in the failure logs of weaker models compared to capable ones\. This indicates that without a strong foundational reasoning ability, multi\-agent frameworks can act as echo chambers that reinforce mistakes\.

Statement:The benefit of a deadline cannot be waived when waiver harms the other party\.
Article 136\(2\):The benefit may be waived; provided, however, that the waiver may not harm the other party\.Gold:No\.
Agent 1 \(round 1\):The statement reflects the exception in Article 136\(2\), which prohibits waiver when it harms the other party\. Yes\.
Agent 2 \(round 1\):Agree\. The input mirrors the exception clause\. Yes\.
Agent 3 \(round 1\):Agree\. Direct restatement of the exception\. Yes\.
All round\-2 agents reaffirm without disagreement\.
Final:Yes \(incorrect\)\. The article restricts the effect of waiver, not the act itself\.Figure 5:Cascade on Civil Code Article 136\(2\)\. A literal misreading of the exception clause is adopted unchanged through the three remaining turns of the five\-turn debate\.
##### Over\-Deliberation Drift from Extended Debate

A key limitation of allocating more discussion rounds is the risk of continuous performance degradation, previously identified as over\-deliberation drift\. Because the framework instructs agents to critique previous answers, extending the discussion selectively encourages the creation of unnecessary doubts\. Figure[6](https://arxiv.org/html/2607.09099#S5.F6)demonstrates how a correct initial voting is overturned in the second round by a flawed objection, leading the group away from the correct answer\. When agents cannot resolve ambiguity, they tend to generate increasingly defensive responses instead of producing new insights\. Therefore, implementing a rule to stop the debate early when unanimous agreement is reached is essential to preserve correct conclusions and prevent topic drift\.

Statement:The pledgee of immovable property bears management costs unless the contract provides otherwise\.
Article 357:The pledgee shall pay management costs and other burdens related to the immovable\.Gold:Yes\.
Round 1 \(all three agents\):The statement aligns with Article 357’s assignment of costs to the pledgee\. Yes\.
Agent 1 \(round 2\):Disagree\. The clause unless provided otherwise is not in Article 357\. No\.
Agent 2 \(round 2\):Disagree\. The input introduces an exception not present in the article\. No\.
Agent 3 \(round 2\):Disagree\. Same reasoning\. No\.
Final:No \(incorrect\)\. A correct round\-1 voting is inverted by a second\-round critique\.Figure 6:Over\-deliberation drift on Civil Code Article 357\. Round 1 produces the correct answer; round 2 introduces a spurious objection that flips the entire group\.
##### Fundamental Knowledge Boundaries\.

Despite these collaborative benefits, the framework consistently struggles with an irreducible subset of queries \(roughly 17\-24%\)\. Manual inspection shows that most of these universally failed cases require legal knowledge that is not directly stated in the supplied text\. Because all participating agents share the same underlying model and knowledge constraints, they are equally unable to bridge these informational gaps\. Consequently, the agents reach the same incorrect conclusion without productive debate\. This highlights that simply increasing the amount of computation during inference is insufficient; multi\-agent systems must be supported by external knowledge sources—such as legal precedents and hierarchical document context—to surpass this inherent performance limit\.

## 6Related Work

##### Legal NLP\.

Legal NLP benchmarks such as LexGLUE\(Chalkidiset al\.,[2022](https://arxiv.org/html/2607.09099#bib.bib22)\), LegalBench\(Guhaet al\.,[2023](https://arxiv.org/html/2607.09099#bib.bib23)\), and LEXam\(Fanet al\.,[2026](https://arxiv.org/html/2607.09099#bib.bib32)\)show that legal language understanding requires models to handle domain\-specific terminology, long contexts, and rule\-based inference\. Our work focuses on COLIEE Task 4\(Rabeloet al\.,[2024](https://arxiv.org/html/2607.09099#bib.bib31); Goebelet al\.,[2026](https://arxiv.org/html/2607.09099#bib.bib35)\), a binary legal textual entailment task over Japanese Civil Code articles\. Prior COLIEE systems have explored retrieval, transformer\-based entailment models, data augmentation, hybrid symbolic\-neural methods, and Japanese LLM prompting for Task 4\(Goebelet al\.,[2024](https://arxiv.org/html/2607.09099#bib.bib28); Steging and Leeuwen,[2024](https://arxiv.org/html/2607.09099#bib.bib30); Onaga and Kano,[2026](https://arxiv.org/html/2607.09099#bib.bib36)\)\. More broadly, prompting methods such as in\-context learning\(Brownet al\.,[2020](https://arxiv.org/html/2607.09099#bib.bib21)\), chain\-of\-thought prompting\(Weiet al\.,[2022](https://arxiv.org/html/2607.09099#bib.bib25)\), self\-consistency\(Wanget al\.,[2022](https://arxiv.org/html/2607.09099#bib.bib10)\), and IRAC\-style legal prompting\(Yuet al\.,[2022](https://arxiv.org/html/2607.09099#bib.bib9)\)have been used to improve reasoning without model fine\-tuning\. However, iterative reasoning is not always reliable: self\-refinement can help in some settings\(Madaanet al\.,[2023b](https://arxiv.org/html/2607.09099#bib.bib26)\), but LLMs often fail to self\-correct without external feedback\(Huanget al\.,[2023](https://arxiv.org/html/2607.09099#bib.bib13)\)\. In contrast to prior single\-agent legal entailment systems, we study whether the final legal decision can be improved through multi\-agent deliberation\.

##### Multi\-agent Frameworks\.

Multi\-agent debate uses multiple LLM instances to propose, critique, and revise answers before aggregation\. Prior work shows that such interaction can improve factuality and reasoning\(Duet al\.,[2024](https://arxiv.org/html/2607.09099#bib.bib15)\), and communication\-based frameworks study topologies such as memory, relay, report, and debate\(Yinet al\.,[2023](https://arxiv.org/html/2607.09099#bib.bib14)\)\. However, these gains are not universal: strong single\-agent prompting can often match multi\-agent discussion\(Wanget al\.,[2024](https://arxiv.org/html/2607.09099#bib.bib17)\), and voting may explain a large part of the improvement attributed to debate\(Choiet al\.,[2026](https://arxiv.org/html/2607.09099#bib.bib34)\)\. MALLM frames multi\-agent systems as a design space involving personas, response generators, communication paradigms, and decision protocols\(Beckeret al\.,[2025](https://arxiv.org/html/2607.09099#bib.bib27)\)\. Persona prompting can diversify reasoning paths but may also introduce instability\(Kimet al\.,[2024](https://arxiv.org/html/2607.09099#bib.bib20)\)\. Aggregation protocols also matter: consensus forces agents to converge, while voting preserves independent judgments until the final decision\(Yanget al\.,[2024](https://arxiv.org/html/2607.09099#bib.bib19); Kaesberget al\.,[2025a](https://arxiv.org/html/2607.09099#bib.bib33)\)\. Our work connects these ideas to statute\-grounded legal entailment, comparing consensus and voting across several LLM backbones and analyzing when debate corrects errors, over\-deliberates, or amplifies flawed reasoning\.

## 7Conclusion

In this work, we introduced L\-MAD, a framework to evaluate multi\-agent debate for complex legal reasoning\. We found that simply adding more agents or discussion rounds does not guarantee better results\. Instead, MAD acts as a performance multiplier that is strictly limited by the base model’s capabilities: while it helps capable models collaborate and correct mistakes, it offers little benefit to state\-of\-the\-art models and causes weaker models to reinforce each other’s errors\. Furthermore, we demonstrated that forcing models to debate for too long actually harms accuracy, as agents begin to second\-guess correct answers\. Our results suggest that to improve performance, it is better to increase the number of diverse agents voting independently than to force extended, consensus\-driven discussions\.

These findings are crucial for safely deploying language models in high\-stakes environments like law\. When agents naturally disagree during a vote, it serves as a reliable signal to route the difficult problem to a human expert\. Ultimately, effective multi\-agent collaboration requires carefully balancing the capabilities of the models and the structure of the debate, rather than assuming that longer discussions will naturally yield better answers\.

For future work, we plan to explore how asymmetric power dynamics and hierarchical relationships among agents influence debate outcomes\. Specifically, investigating whether authoritative or collaborative leadership personas are more effective at guiding discussions could yield more robust decision protocols\. Additionally, integrating Explainable AI \(XAI\) techniques into the MAD framework should be explored to further enhance the transparency and reliability of LLMs in high\-stakes legal applications\.

## Acknowledgements

We sincerely thank the anonymous reviewers for their insightful comments and valuable feedback, which helped improve the quality of this manuscript\. This work was partly supported by the Japan Science and Technology Agency \(JST\) as part of the Adopting Sustainable Partnerships for Innovative Research Ecosystem \(ASPIRE\), Grant Number JPMJAP25B2\.

## Impact Statement

This paper presents work whose primary goal is to advance the reliability and interpretability of Machine Learning systems in high\-stakes, knowledge\-dense domains, specifically focusing on legal reasoning\. As Large Language Models are increasingly integrated into judicial, legal, and regulatory workflows, ensuring their deductive accuracy and epistemological robustness is of paramount societal importance\.

Our research highlights both the potential and the systemic risks of deploying multi\-agent systems in these environments\. On the positive side, our findings regarding vote disagreement provide a native, training\-free confidence signal\. This enables the design of safe, hybrid human\-in\-the\-loop systems where high\-uncertainty cases are reliably routed to human legal professionals, thereby mitigating the risk of fully autonomous legal malpractice\.

## References

- N\. Aletras, D\. Tsarapatsanis, D\. Preoţiuc\-Pietro, and V\. Lampos \(2016\)Predicting judicial decisions of the european court of human rights: a natural language processing perspective\.PeerJ computer science2,pp\. e93\.Cited by:[§1](https://arxiv.org/html/2607.09099#S1.p1.1)\.
- Y\. Aoki, M\. Yoshioka, and Y\. Suzuki \(2022\)Data\-augmentation method for bert\-based legal textual entailment systems in coliee statute law task\.The Review of Socionetwork Strategies16\(1\),pp\. 175–196\.Cited by:[§1](https://arxiv.org/html/2607.09099#S1.p1.1)\.
- J\. Becker, L\. B\. Kaesberg, N\. Bauer, J\. P\. Wahle, T\. Ruas, and B\. Gipp \(2025\)MALLM: multi\-agent large language models framework\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,I\. Habernal, P\. Schulam, and J\. Tiedemann \(Eds\.\),Suzhou, China,pp\. 418–439\.External Links:[Link](https://aclanthology.org/2025.emnlp-demos.29/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-demos.29),ISBN 979\-8\-89176\-334\-0Cited by:[§6](https://arxiv.org/html/2607.09099#S6.SS0.SSS0.Px2.p1.1)\.
- J\. Becker \(2024\)Multi\-agent large language models for conversational task\-solving\.arXiv preprint arXiv:2410\.22932\.Cited by:[§5\.3](https://arxiv.org/html/2607.09099#S5.SS3.p1.1)\.
- O\. Bilgin, L\. Fields, A\. Laverghetta Jr, Z\. Marji, A\. Nighojkar, S\. Steinle, and J\. Licato \(2024\)Exploring prompting approaches in legal textual entailment\.The Review of Socionetwork Strategies18\(1\),pp\. 75–100\.Cited by:[§1](https://arxiv.org/html/2607.09099#S1.p1.1)\.
- T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[§2\.2](https://arxiv.org/html/2607.09099#S2.SS2.SSS0.Px1.p1.6),[§6](https://arxiv.org/html/2607.09099#S6.SS0.SSS0.Px1.p1.1)\.
- I\. Chalkidis, M\. Fergadiotis, P\. Malakasiotis, N\. Aletras, and I\. Androutsopoulos \(2020\)LEGAL\-BERT: the muppets straight out of law school\.InFindings of the Association for Computational Linguistics: EMNLP 2020,T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 2898–2904\.External Links:[Link](https://aclanthology.org/2020.findings-emnlp.261/),[Document](https://dx.doi.org/10.18653/v1/2020.findings-emnlp.261)Cited by:[§1](https://arxiv.org/html/2607.09099#S1.p1.1)\.
- I\. Chalkidis, A\. Jana, D\. Hartung, M\. Bommarito, I\. Androutsopoulos, D\. Katz, and N\. Aletras \(2022\)LexGLUE: a benchmark dataset for legal language understanding in English\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 4310–4330\.External Links:[Link](https://aclanthology.org/2022.acl-long.297/),[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.297)Cited by:[§6](https://arxiv.org/html/2607.09099#S6.SS0.SSS0.Px1.p1.1)\.
- H\. K\. Choi, J\. Zhu, and S\. Li \(2026\)Debate or vote: which yields better decisions in multi\-agent large language models?\.Advances in Neural Information Processing Systems38,pp\. 101732–101764\.Cited by:[§1](https://arxiv.org/html/2607.09099#S1.p2.1),[§6](https://arxiv.org/html/2607.09099#S6.SS0.SSS0.Px2.p1.1)\.
- Y\. Du, S\. Li, A\. Torralba, J\. B\. Tenenbaum, and I\. Mordatch \(2024\)Improving factuality and reasoning in language models through multiagent debate\.InProceedings of the 41st International Conference on Machine Learning,ICML’24\.Cited by:[§6](https://arxiv.org/html/2607.09099#S6.SS0.SSS0.Px2.p1.1)\.
- A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan,et al\.\(2024\)The llama 3 herd of models\.arXiv e\-prints,pp\. arXiv–2407\.Cited by:[§4\.3](https://arxiv.org/html/2607.09099#S4.SS3.p2.1)\.
- Y\. Fan, J\. Ni, J\. Merane, E\. Salimbeni, Y\. Tian, Y\. Hermstrüwer, Y\. Huang, M\. Akhtar, F\. Geering, O\. Dreyer,et al\.\(2026\)LEXam: benchmarking legal reasoning on 340 law exams\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§6](https://arxiv.org/html/2607.09099#S6.SS0.SSS0.Px1.p1.1)\.
- Y\. Fu, H\. Peng, A\. Sabharwal, P\. Clark, and T\. Khot \(2022\)Complexity\-based prompting for multi\-step reasoning\.arXiv preprint arXiv:2210\.00720\.Cited by:[§2\.2](https://arxiv.org/html/2607.09099#S2.SS2.SSS0.Px2.p1.3)\.
- R\. Goebel, Y\. Kano, M\. Kim, C\. Kwan, J\. Rabelo, K\. Satoh, H\. Yamada, and M\. Yoshioka \(2026\)The coliee 2025 competition on legal information extraction and entailment: overview, discussion, and dataset expansion\.The Review of Socionetwork Strategies,pp\. 1–31\.Cited by:[§6](https://arxiv.org/html/2607.09099#S6.SS0.SSS0.Px1.p1.1)\.
- R\. Goebel, Y\. Kano, M\. Kim, J\. Rabelo, K\. Satoh, and M\. Yoshioka \(2024\)Overview of benchmark datasets and methods for the legal information extraction/entailment competition \(coliee\) 2024\.InNew Frontiers in Artificial Intelligence: JSAI International Symposium on Artificial Intelligence, JSAI\-IsAI 2024, Hamamatsu, Japan, May 28–29, 2024, Proceedings,Berlin, Heidelberg,pp\. 109–124\.External Links:ISBN 978\-981\-97\-3075\-9,[Link](https://doi.org/10.1007/978-981-97-3076-6_8),[Document](https://dx.doi.org/10.1007/978-981-97-3076-6%5F8)Cited by:[§6](https://arxiv.org/html/2607.09099#S6.SS0.SSS0.Px1.p1.1)\.
- N\. Guha, J\. Nyarko, D\. Ho, C\. Ré, A\. Chilton, A\. K, A\. Chohlas\-Wood, A\. Peters, B\. Waldon, D\. Rockmore, D\. Zambrano, D\. Talisman, E\. Hoque, F\. Surani, F\. Fagan, G\. Sarfaty, G\. Dickinson, H\. Porat, J\. Hegland, J\. Wu, J\. Nudell, J\. Niklaus, J\. Nay, J\. Choi, K\. Tobia, M\. Hagan, M\. Ma, M\. Livermore, N\. Rasumov\-Rahe, N\. Holzenberger, N\. Kolt, P\. Henderson, S\. Rehaag, S\. Goel, S\. Gao, S\. Williams, S\. Gandhi, T\. Zur, V\. Iyer, and Z\. Li \(2023\)LegalBench: a collaboratively built benchmark for measuring legal reasoning in large language models\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 44123–44279\.Cited by:[§6](https://arxiv.org/html/2607.09099#S6.SS0.SSS0.Px1.p1.1)\.
- J\. Huang, X\. Chen, S\. Mishra, H\. S\. Zheng, A\. W\. Yu, X\. Song, and D\. Zhou \(2023\)Large language models cannot self\-correct reasoning yet\.arXiv preprint arXiv:2310\.01798\.Cited by:[§5\.3](https://arxiv.org/html/2607.09099#S5.SS3.p4.1),[§6](https://arxiv.org/html/2607.09099#S6.SS0.SSS0.Px1.p1.1)\.
- L\. B\. Kaesberg, J\. Becker, J\. P\. Wahle, T\. Ruas, and B\. Gipp \(2025a\)Voting or consensus? decision\-making in multi\-agent debate\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 11640–11671\.External Links:[Link](https://aclanthology.org/2025.findings-acl.606/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.606),ISBN 979\-8\-89176\-256\-5Cited by:[§6](https://arxiv.org/html/2607.09099#S6.SS0.SSS0.Px2.p1.1)\.
- L\. B\. Kaesberg, J\. Becker, J\. P\. Wahle, T\. Ruas, and B\. Gipp \(2025b\)Voting or consensus? decision\-making in multi\-agent debate\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 11640–11671\.Cited by:[§1](https://arxiv.org/html/2607.09099#S1.p2.1)\.
- J\. Kim, N\. Yang, and K\. Jung \(2024\)Persona is a double\-edged sword: mitigating the negative impact of role\-playing prompts in zero\-shot reasoning tasks\.arXiv preprint arXiv:2408\.08631\.Cited by:[§3\.1](https://arxiv.org/html/2607.09099#S3.SS1.p1.1),[§6](https://arxiv.org/html/2607.09099#S6.SS0.SSS0.Px2.p1.1)\.
- P\. Langley \(2000\)Crafting papers on machine learning\.InProceedings of the 17th International Conference on Machine Learning \(ICML 2000\),P\. Langley \(Ed\.\),Stanford, CA,pp\. 1207–1216\.Cited by:[§A\.2\.3](https://arxiv.org/html/2607.09099#A1.SS2.SSS3.p3.1)\.
- A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang, S\. Gupta, B\. P\. Majumder, K\. Hermann, S\. Welleck, A\. Yazdanbakhsh, and P\. Clark \(2023a\)Self\-refine: iterative refinement with self\-feedback\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 46534–46594\.Cited by:[§5\.3](https://arxiv.org/html/2607.09099#S5.SS3.p4.1)\.
- A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang,et al\.\(2023b\)Self\-refine: iterative refinement with self\-feedback\.Advances in neural information processing systems36,pp\. 46534–46594\.Cited by:[§6](https://arxiv.org/html/2607.09099#S6.SS0.SSS0.Px1.p1.1)\.
- Y\. Nie, A\. Williams, E\. Dinan, M\. Bansal, J\. Weston, and D\. Kiela \(2020\)Adversarial NLI: a new benchmark for natural language understanding\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 4885–4901\.External Links:[Link](https://aclanthology.org/2020.acl-main.441/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.441)Cited by:[§1](https://arxiv.org/html/2607.09099#S1.p2.1)\.
- T\. Onaga and Y\. Kano \(2026\)Kis: coliee 2025 task 4 solver using japanese llm\.The Review of Socionetwork Strategies,pp\. 1–19\.Cited by:[§6](https://arxiv.org/html/2607.09099#S6.SS0.SSS0.Px1.p1.1)\.
- J\. Rabelo, R\. Goebel, M\. Kim, Y\. Kano, M\. Yoshioka, and K\. Satoh \(2024\)Overview and discussion of the competition on legal information extraction/entailment \(COLIEE\) 2023\.The Review of Socionetwork Strategies18\(1\),pp\. 27–47\.Cited by:[§6](https://arxiv.org/html/2607.09099#S6.SS0.SSS0.Px1.p1.1)\.
- C\. Steging and L\. Leeuwen \(2024\)A hybrid approach to legal textual entailment\.InProceedings of the Eighteenth International Workshop on Juris\-Informatics \(JURISIN 2024\),Cited by:[§6](https://arxiv.org/html/2607.09099#S6.SS0.SSS0.Px1.p1.1)\.
- Q\. Wang, Z\. Wang, Y\. Su, H\. Tong, and Y\. Song \(2024\)Rethinking the bounds of LLM reasoning: are multi\-agent discussions the key?\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 6106–6131\.External Links:[Link](https://aclanthology.org/2024.acl-long.331/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.331)Cited by:[§5\.3](https://arxiv.org/html/2607.09099#S5.SS3.p1.1),[§6](https://arxiv.org/html/2607.09099#S6.SS0.SSS0.Px2.p1.1)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou \(2022\)Self\-consistency improves chain of thought reasoning in language models\.arXiv preprint arXiv:2203\.11171\.Cited by:[§2\.2](https://arxiv.org/html/2607.09099#S2.SS2.SSS0.Px3.p1.2),[§4\.3](https://arxiv.org/html/2607.09099#S4.SS3.p1.1),[§5\.3](https://arxiv.org/html/2607.09099#S5.SS3.p3.1),[§6](https://arxiv.org/html/2607.09099#S6.SS0.SSS0.Px1.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[§2\.2](https://arxiv.org/html/2607.09099#S2.SS2.SSS0.Px2.p1.3),[§6](https://arxiv.org/html/2607.09099#S6.SS0.SSS0.Px1.p1.1)\.
- C\. Xiao, X\. Hu, Z\. Liu, C\. Tu, and M\. Sun \(2021\)Lawformer: a pre\-trained language model for chinese legal long documents\.AI Open2,pp\. 79–84\.Cited by:[§1](https://arxiv.org/html/2607.09099#S1.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§4\.3](https://arxiv.org/html/2607.09099#S4.SS3.p2.1)\.
- J\. C\. Yang, D\. Dailisan, M\. Korecki, C\. I\. Hausladen, and D\. Helbing \(2024\)Llm voting: human choices and ai collective decision\-making\.InProceedings of the AAAI/ACM Conference on AI, Ethics, and Society,Vol\.7,pp\. 1696–1708\.Cited by:[§3\.2](https://arxiv.org/html/2607.09099#S3.SS2.p4.1),[§6](https://arxiv.org/html/2607.09099#S6.SS0.SSS0.Px2.p1.1)\.
- Z\. Yin, Q\. Sun, C\. Chang, Q\. Guo, J\. Dai, X\. Huang, and X\. Qiu \(2023\)Exchange\-of\-thought: enhancing large language model capabilities through cross\-model communication\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 15135–15153\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.936/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.936)Cited by:[§3\.1](https://arxiv.org/html/2607.09099#S3.SS1.p1.1),[§3\.2](https://arxiv.org/html/2607.09099#S3.SS2.p3.3),[§5\.3](https://arxiv.org/html/2607.09099#S5.SS3.p1.1),[§6](https://arxiv.org/html/2607.09099#S6.SS0.SSS0.Px2.p1.1)\.
- F\. Yu, L\. Quartey, and F\. Schilder \(2022\)Legal prompting: teaching a language model to think like a lawyer\.arXiv preprint arXiv:2212\.01326\.Cited by:[§2\.2](https://arxiv.org/html/2607.09099#S2.SS2.SSS0.Px2.p1.3),[§4\.3](https://arxiv.org/html/2607.09099#S4.SS3.p1.1),[§6](https://arxiv.org/html/2607.09099#S6.SS0.SSS0.Px1.p1.1)\.
- H\. Zhong, C\. Xiao, C\. Tu, T\. Zhang, Z\. Liu, and M\. Sun \(2020\)How does NLP benefit legal system: a summary of legal artificial intelligence\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 5218–5230\.External Links:[Link](https://aclanthology.org/2020.acl-main.466/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.466)Cited by:[§1](https://arxiv.org/html/2607.09099#S1.p1.1)\.

## Appendix AAdditional Experiment Details\.

### A\.1Agents’ Personas

Judge: You are an impartial, highly analytical Japanese Presiding Judge in a civil chamber\. Your objective is to evaluate the arguments of the Plaintiff and Defendant, verify their statutory mapping against the Civil Code, and render a final verdict\.

Attorney: You are a precise, methodical Japanese civil litigator\. Your sole objective is to prove that the Premise legally ENTAILS the Hypothesis under the Japanese Civil Code\.

Lawyer: You are a sharp, analytical Japanese civil lawyer\. Your objective is to prove that the Premise does NOT legally entail the Hypothesis\.

### A\.2Prompt Templates

This section provides the exact prompt templates used for the baselines evaluated in our experiments\. All experiments were conducted using Japanese prompts, natively aligning with the language of the Japanese Civil Code corpus\. The text enclosed in brackets\{\.\.\.\}represents dynamic variables substituted at inference time\.

#### A\.2\.1Zeroshot Prompting

System Prompt \(Zeroshot\)``` あなたは日本の民法に精通した法律専門家です。 与えられた民法条文と法律問題を注意深く分析し、 条文が問題の答えを「はい」と裏付けているかどうかを判断してください。 回答は必ず以下の形式で出力してください： 判断: Y または N 理由: （簡潔な説明） ```

User Prompt \(Zeroshot\)``` 以下の民法条文と法律問題を読んでください。 【関連条文】 {article_text} 【法律問題】 {query} この条文を根拠として、問題の答えは「はい」と言えますか？ 判断: Y または N で答え、その理由を説明してください。 ```

#### A\.2\.2IRAC \(Legal Reasoning CoT\) Prompting

System Prompt \(IRAC\)``` あなたは日本の民法に精通した法律専門家です。 提示された法的推論の手法に従い、仮説が「真」か「偽」かを分析してください。 手法：争点、法則、適用、結論。 回答は必ず以下の形式で出力してください： 判断: Y または N 理由: （簡潔な説明） ```

User Prompt \(IRAC\)``` 以下の民法条文と法律問題を読んでください。 【関連条文】 {article_text} 【法律問題】 {query} この条文を根拠として、問題の答えは「はい」と言えますか？ 判断: Y または N で答え、その理由を説明してください。 ```

#### A\.2\.3Fewshot Prompting

System Prompt \(Fewshot\)``` あなたは日本の民法に精通した法律専門家です。 与えられた民法条文と法律問題を分析し、 条文が問題の答えを「はい」と裏付けているか判断してください。 回答は必ず以下の形式で出力してください： 判断: Y または N 理由: （簡潔な説明） ```

User Prompt \(Fewshot\)``` 以下の例を参考にして判断してください。 {examples} 今度はあなたが判断してください。 【関連条文】 {article_text} 【法律問題】 {query} 判断: Y または N で答え、その理由を説明してください。 ```