From Judgments to Issues: Structured Extraction of Legal Reasoning with Citation-Hallucination Control

# From Judgments to Issues: Structured Extraction of Legal Reasoning with Citation-Hallucination Control
Source: [https://arxiv.org/html/2607.03325](https://arxiv.org/html/2607.03325)
Giovanni Piccioli Quantitative and Digital Law Laboratory King’s College London Strand, London WC2R 2LS giovannipiccioli@gmail\.com &Alessia Fidelangeli CIRSFID \- Alma AI, Faculty of Law University of Bologna Bologna, Via Zamboni 27/29 &Piera Santin Robert Schuman Centre European University Institute Fiesole, Badia Fiesolana \- Via dei Roccettini 9 &Pierpaolo Vivo Quantitative and Digital Law Laboratory King’s College London Strand, London WC2R 2LS

###### Abstract

We present an automated pipeline that decomposes Italian tax\-court judgments into individual legal issues and extracts, for each issue, a structured XML representation grounded in the IRAC framework and the legal syllogism\. The pipeline targets a corpus of approximately330,000330\{,\}000first\- and second\-instance decisions of the Italian tax courts and is built around a capable yet cost\-efficient general\-purpose model \(DeepSeek V3\), a choice driven by the need to process several hundred thousand documents at a sustainable cost\. To address the well\-documented unreliability of large language models on legal citations, we couple the extraction step with an automatic hallucination\-detection filter that compares the references produced by the model with those identified in the judgment text by a dedicated parser \(Linkoln\), normalised to standard identifiers \(URN\-NIR, ECLI, CELEX\)\. We validate the pipeline on5050judgments annotated by two PhDs in tax law, computing inter\-annotator agreement and LLM\-vs\-expert agreement on both issue extraction and legal citations, together with a stand\-alone evaluation of the hallucination filter\. To the best of our knowledge, this is the first issue\-level, expert\-validated structured extraction pipeline with hallucination control for Italian tax\-court decisions, and it provides a concrete starting point for downstream applications such as issue\-level retrieval, citation\-network analysis, and the construction of large\-scale datasets of legal reasoning\.

## 1Introduction

Modern judicial systems generate court decisions at a scale that has long outpaced the capacity of legal professionals to read them individually\. In Italy, the database of first\- and second\-instance tax\-court decisions alone contains almost a million judgments, and grows by the hundreds of thousands every year\. The practical value of this volume lies almost entirely in secondary use: precedent search by practitioners, citation\-network analysis, evaluation of judicial behaviour, and the construction of large datasets for training and benchmarking legal NLP systems\. Each of these uses, however, requires a representation of the decisions that is more compact and structured, than the original PDFs the courts produce\. The central question this paper addresses is how to obtain such a representation automatically, at scale, and with quantified reliability\.

Italian tax\-court judgments are a particularly informative testbed for this question\. On the one hand, the domain stresses any extraction pipeline that claims to be general\. Tax disputes range from a few tens of euros to tens of millions, and routinely draw on civil, commercial, criminal, procedural, and European Union law in resolving a single case\. The first two instances are decided by panels that include non\-career honorary judges, whose drafting style is markedly less standardized than that of higher courts: even the names and ordering of the structural sections of a judgment vary from one decision to the next\. On the other hand, the domain also makes the problem tractable: an official open database is available, the volume is large enough that statistical evaluation is meaningful, and the parties are always the same pair \(taxpayer and tax authority\)\. A pipeline that performs well here is therefore informative both as a methodological contribution and as a usable tool for a jurisdiction whose case law has, so far, received little attention in the legal NLP literature\.

The central methodological choice of this work is to take the*legal issue*\(*questione giuridica*\), rather than the judgment, as the unit of representation\. This choice is consistent with long\-standing analyses of judicial justification, which view adjudication as the resolution of one or more legal questions through reasoning grounded in facts and legal norms\(MacCormick,[1994](https://arxiv.org/html/2607.03325#bib.bib64); Bench\-Caponet al\.,[2024](https://arxiv.org/html/2607.03325#bib.bib62)\)\. A single judgment typically resolves several issues that have little in common beyond belonging to the same case: a court may first decide on the admissibility of an appeal, then on the merits, and finally on the allocation of costs\. For practically all secondary uses we have in mind, the issue is the natural unit of analysis: a precedent search is a search for cases that have decided a specific question; a citation graph is informative to the extent that its edges reflect references actually used to decide a point of dispute, rather than incidental mentions; a dataset of legal reasoning is useful to the extent that its examples isolate a single argumentative chain\. We therefore design our pipeline to decompose each judgment into a set of autonomous issues so as to operate on these issues independently\.

The structure we extract for each issue is inspired by two widely used analytical frameworks for describing judicial reasoning: IRAC \(Issue, Rule, Application, Conclusion\) and the legal syllogism\. These models impose enough structure to make the representation machine\-readable and to anchor it in a recognized theoretical tradition, while remaining coarse\-grained enough to be extracted reliably by a language model\. Concretely, each issue is encoded as an XML record with aWhether\-clause statement of the question, a list of factual premises, a list of legal references with their citation reasons, the judge’s reasoning, the outcome, and an abstractive summary\. The schema deliberately sits at a level of granularity that users can interpret easily, and that the language model can populate consistently across the heterogeneous styles of the corpus\. The trade\-off between structural richness and reliable extractability is itself a design choice as discussed in detail in Section[3\.4](https://arxiv.org/html/2607.03325#S3.SS4)\.

Two practical constraints shape the rest of the pipeline\. The first is cost\. Extraction at the scale of the full database requires a model whose per\-document price is compatible with processing several hundred thousand judgments; this motivates our use of DeepSeek V3 which resulted in a cost of roughly $0\.0035 per judgment\. In comparison, the rates of top\-range proprietary models can be twenty times higher\. The second is reliability\. The LLM should be powerful enough to correctly segment the judgment into issues and fill the corresponding XML fields in a way that a tax law expert would find acceptable\. In addition, the model’s hallucination rate should be as low as possible\. This is a known failure mode of LLMs, consisting in the fabrication of plausible\-looking but non\-existent citations to legislation and case law\(Dahlet al\.,[2024](https://arxiv.org/html/2607.03325#bib.bib65); Hanet al\.,[2026](https://arxiv.org/html/2607.03325#bib.bib19)\)\.

Because cost\-efficiency and reliability represent competing priorities, we conducted an informal comparative assessment of multiple models and selected the most cost\-effective model that delivered results acceptable to legal experts; see Appendix[J](https://arxiv.org/html/2607.03325#A10)\. To mitigate the risk of hallucinations, we add an explicit verification layer after the extraction\. We parse the legal references in both the judgment text and the LLM output with the Linkoln library\(Bacciet al\.,[2019](https://arxiv.org/html/2607.03325#bib.bib1)\), normalise them to standard identifiers \(URN\-NIR, ECLI, CELEX\), and remove from the output any reference that cannot be located in the source\. This filter is conservative by design: it preserves the citations on which the pipeline is reliable and removes those it is not\.

We evaluate the resulting pipeline on a manually annotated subsample of5050judgments\. Two experts \(PhDs in tax law\) doubly annotated2020of these under double\-blind conditions and annotated the remaining3030in disjoint halves, following guidelines they had jointly developed on a separate set of judgments\. This protocol lets us report LLM\-vs\-expert agreement on issue extraction, on legal references extraction, and on a set of qualitative scores for the free\-text fields, and also inter\-annotator agreement on the same metrics\. Section[6](https://arxiv.org/html/2607.03325#S6)reports the results in detail; we additionally evaluate the hallucination filter as a stand\-alone classifier and analyze the residual hallucinations by citation type\.

### 1\.1Contributions

The contribution of this paper is threefold\.

First, we propose a structured XML schema for representing court judgments that is explicitly grounded in IRAC and in the legal syllogism, takes the legal issue as its unit, and is simultaneously useful for human consultation, machine\-readable for downstream tasks, and reliably instantiated by a mid\-range LLM\.

Second, we develop an extraction pipeline built around a cost\-efficient general\-purpose model \(DeepSeek V3\) and combine it with an automatic hallucination\-detection step on legal references\. By normalizing both the judgment text and the LLM output through the Linkoln library\(Bacciet al\.,[2019](https://arxiv.org/html/2607.03325#bib.bib1)\)and comparing the two sets, we remove citations that the model has fabricated while preserving the rest\. The pipeline is run and refined on a set of about330,000330\{,\}000processed \(of401,349401\{,\}349retrieved\), Italian tax court judgments\.

Third, we evaluate the pipeline on a corpus of5050Italian tax\-court judgments annotated by two legal experts, reporting inter\-annotator agreement and LLM\-vs\-expert agreement on issue extraction, on legal citations, and on qualitative dimensions of the free\-text fields, together with a stand\-alone analysis of the hallucination filter\.

The rest of the paper is organised as follows\. Section[2](https://arxiv.org/html/2607.03325#S2)reviews related work on structured extraction from court decisions, IRAC\- and syllogism\-based pipelines, citation handling, and LLM\-based legal annotation\. Section[3](https://arxiv.org/html/2607.03325#S3)presents the issue\-based XML schema and the design requirements that shape it\. Section[4](https://arxiv.org/html/2607.03325#S4)describes the corpus and the extraction pipeline, including the hallucination\-detection step\. Section[5](https://arxiv.org/html/2607.03325#S5)sets out the validation protocol, and Section[6](https://arxiv.org/html/2607.03325#S6)reports the evaluation results\. Section[7](https://arxiv.org/html/2607.03325#S7)concludes with a discussion of limitations and directions for future work\.

## 2Related Work

Our work sits at the intersection of several lines of research in legal NLP: \(i\) formal models of judicial reasoning, \(ii\) structured extraction from court decisions, \(iii\) legal citation extraction, normalization, and hallucination control, and \(iv\) the use of LLMs for extraction and annotation on legal corpora\.

#### IRAC, the legal syllogism, and structured judicial reasoning\.

A first body of work in AI and Law seeks to represent legal reasoning in structured forms that support computational analysis, legal knowledge extraction, and decision support\(Ashley,[2017](https://arxiv.org/html/2607.03325#bib.bib63)\)\. Within this tradition, several recent works formalize judicial reasoning along the lines of the IRAC framework or of the legal syllogism and exploit these structures in LLM\-based pipelines\(Bench\-Capon,[2020](https://arxiv.org/html/2607.03325#bib.bib72)\)\.Jiang and Yang \([2023](https://arxiv.org/html/2607.03325#bib.bib49)\)introduce*legal syllogism prompting*, instructing LLMs to articulate major premise, minor premise, and conclusion for judgment prediction;Yuet al\.\([2022](https://arxiv.org/html/2607.03325#bib.bib20)\)similarly show that IRAC\-style prompts outperform generic chain\-of\-thought on the legal entailment task in the COLIEE competition\(Rabeloet al\.,[2022](https://arxiv.org/html/2607.03325#bib.bib52)\)\.Kanget al\.\([2025](https://arxiv.org/html/2607.03325#bib.bib11)\)couple an IRAC benchmark of Malaysian contract\-law scenarios with a semi\-structured knowledge base, showing substantial gains on issue identification and rule retrieval\.Holzenberger and Van Durme \([2023](https://arxiv.org/html/2607.03325#bib.bib60)\)extend this approach by treating statutory reasoning as a pipeline in which legal text is first converted into structured representations through information extraction, and then processed by a symbolic Prolog\-based reasoner, showing that reasoning accuracy depends on the quality of the extracted structure\. While these works use IRAC or the syllogism to guide prompting or give LLMs a framework for legal reasoning, we take inspiration from them in designing the extraction schema\(Ruf and Yin,[2024](https://arxiv.org/html/2607.03325#bib.bib73)\)\.

#### Structured extraction and decomposition of judgments\.

A closely related strand uses LLMs to extract structured representations of decisions for secondary use\.Grundleret al\.\([2025](https://arxiv.org/html/2607.03325#bib.bib25)\)extract Judicial Interpretative Formulas from CJEU decisions on VAT, combining LLM\-based annotation with fine\-tuned BERT and expert validation; their tax\-law focus and expert\-validated setup parallel ours\.Adhikaryet al\.\([2024](https://arxiv.org/html/2607.03325#bib.bib10)\)define fine\-grained attributes for Indian criminal cases and extract them with few\-shot LLM prompting, demonstrating that attribute\-level representations help with judgment and statute prediction\.Costaet al\.\([2024](https://arxiv.org/html/2607.03325#bib.bib30)\)produce structured annotations of Brazilian judicial decisions tagged with*ratio decidendi*and outcome;Westermannet al\.\([2025](https://arxiv.org/html/2607.03325#bib.bib28)\)use LLMs to map court decisions onto a "rule tree" \(a logical breakdown of the conditions a claim must satisfy\) identifying which conditions the judge found met and explaining the reasoning;Janatianet al\.\([2023](https://arxiv.org/html/2607.03325#bib.bib29)\)convert legislation into structured if\-then rules that computer systems can use to answer legal questions\.Belfathiet al\.\([2023](https://arxiv.org/html/2607.03325#bib.bib35)\)andLombardiet al\.\([2023](https://arxiv.org/html/2607.03325#bib.bib36)\)use LLMs to segment and label judgments into logical parts\. The annotation of arguments in judgments and subsequent use for NLP tasks is explored inHabernalet al\.\([2024](https://arxiv.org/html/2607.03325#bib.bib22)\)andSantinet al\.\([2023](https://arxiv.org/html/2607.03325#bib.bib27)\)\. Finally, the Openjustice initiative\(Dahanet al\.,[2023](https://arxiv.org/html/2607.03325#bib.bib61); Bhambhoriaet al\.,[2026](https://arxiv.org/html/2607.03325#bib.bib67)\)aims to build a structured dataset or legal reasoning based on crowdsourcing reasoning flows composed of facts, logic conditions, and outcomes\. Compared with this body of work, our contribution differs in \(i\) anchoring the extraction template in IRAC/syllogism rather than in a flat attribute list, \(ii\) decomposing each judgment into autonomous*issues*, and \(iii\) integrating an explicit hallucination check on legal references\.

#### Legal holdings and summarization, with an Italian focus\.

The extraction of holdings,massime, andLeitsätzeand structured summaries is closely related to ours\.Licariet al\.\([2023](https://arxiv.org/html/2607.03325#bib.bib26)\)conducted experiments aiming to extractmassimefrom Italian judgments using Italian\-LEGAL\-BERT\.Zinet al\.\([2025](https://arxiv.org/html/2607.03325#bib.bib42)\)study LLMs for generating GermanLeitsätzeandArvin \([2025](https://arxiv.org/html/2607.03325#bib.bib45)\)systematically evaluate LLMs of different sizes for holding identification\.Xu and Ashley \([2023](https://arxiv.org/html/2607.03325#bib.bib32)\)propose evaluating legal summaries by whether they address every legal issue raised in the underlying judgment\. For Italian,Benedettoet al\.\([2023](https://arxiv.org/html/2607.03325#bib.bib47),[2025](https://arxiv.org/html/2607.03325#bib.bib21)\)benchmark and develop BART\-based summarization models adapted to Italian legal documents\. Most directly comparable,Dal Pontet al\.\([2023](https://arxiv.org/html/2607.03325#bib.bib18)\)apply GPT\-4 to Italian tax\-court judgments within the PRODIGIT project for summarization, identification of legal issues, and keyword extraction, with extensive evaluation by tax judges and lawyers\.Garzo and Palumbo \([2025](https://arxiv.org/html/2607.03325#bib.bib33)\)evaluate GPT\-4o on Italian vehicular homicide rulings, assessing both conceptual understanding and the ability to extract the*ratio decidendi*\. We extend these contributions by moving from free\-form abstractive summaries to a fully structured XML schema modelled on IRAC, decomposing each judgment into individual issues with their own fields and embedding citation\-level hallucination control\.

#### Citation extraction, normalization, and citation networks\.

Reliable handling of legal references is a fundamental element of our pipeline\. The Linkoln library\(Bacciet al\.,[2019](https://arxiv.org/html/2607.03325#bib.bib1)\)parses legal citations in Italian and resolves them to standard identifiers \(urn:nir, CELEX, ECLI\); the URN\-NIR scheme for Italian legal acts is described inFrancesconiet al\.\([2010](https://arxiv.org/html/2607.03325#bib.bib5)\)andGiovanniniet al\.\([2012](https://arxiv.org/html/2607.03325#bib.bib59)\)\. We rely on these tools to standardize citations in both the judgment text and the LLM output, and to flag references that do not match as candidate hallucinations\. Several works focused on building legal citation networks, see for exampleCoupetteet al\.\([2021](https://arxiv.org/html/2607.03325#bib.bib57)\),Ovádeket al\.\([2021](https://arxiv.org/html/2607.03325#bib.bib58)\)\. The construction of citation networks from judgments has been recently revisited byMumfordet al\.\([2025](https://arxiv.org/html/2607.03325#bib.bib40)\), who annotate each citation*instance*with its context; our`<citation\_reason\>`field is aligned with this idea\.

## 3An Issue\-Based Schema for Tax\-Court Judgments

### 3\.1The XML format

In this section we motivate and describe the different fields we extract from each judgment\. Our work in fact starts from the idea of transforming a judgment into another structured document\. The choice of this structure is crucial, as it determines the possible uses of the output format\. We selected a possible format based on the following criteria:

1. 1\.Usefulness for legal professionals\. A professional conducting a precedent search looking at the extracted fields should more easily grasp the essential information about the case, compared to having to read the original document\.
2. 2\.Possibility of extracting the information automatically using LLMs\. In other words, the format should be simple enough to be reliably filled by an LLM, when the text of a judgment is provided\.
3. 3\.Possibility to use the extracted format for computational tasks\. In particular, we value the machine readability of extracted fields\.
4. 4\.Generality\. The format should be general enough to apply to several types of judgments\.

In essence, our goal is to develop a structured format that can be extracted automatically from the text of judgments and that is useful for secondary use, both for human use and for computational analysis\. In practice, there can be tension between some of the requirements, for example machine and human readability\. To obtain the final structure, we iteratively refined an extraction prompt through several rounds of validation\. To design the structure, we were also inspired by the IRAC approach and by the legal syllogism\. These frameworks give a stylized view of the judge’s reasoning process\. In listing[1](https://arxiv.org/html/2607.03325#LST1)we report an English translation of the final prompt we used for the extraction\. The original Italian prompts are reported in Appendix[C](https://arxiv.org/html/2607.03325#A3)\.

Listing 1:Prompt \(translated in English\) used for XML extraction\.InsidethecurlybracesthereisanItaliantaxcourtdecision\.

Yourtaskistoidentifyandextractthemainlegalissues\(typicallyoneortwo\)thatthejudgeaddressesandresolvesinordertodecidethedispute\.Consideronlytheissuesonwhichthejudgerulesexplicitlyandwithreasoning\.

Foreachissue,performthefollowing:

\-Formulatethelegalissueclearly,asaself\-containedandautonomousquestion\.TheformulationmustfollowthestructureWhether\+clauseandmustbeunderstandableonitsown,withoutreferringtothefulldecisionorotherissues\.

\-Indicatetheoutcomeoftheissue,i\.e\.,theanswertothelegalquestion\.

\-Breakdownthejudge’sreasoningintothefollowingparts:

\-\-<factual\_premises\>:listthefacts,actions,events,ordataofthecasethatarerelevanttotheissue\.

\-\-<legal\_references\>:listALLlegalreferences\(legislation,caselaw,generalprinciples,administrativepractice\)relevanttotheissue,astheyappearinthedecision\.Orderthembydecreasingimportance\.

\-\-<citation\_reason\>:forthelegalreferencesdiscussedmoreindetail,explainwhythejudgecitesthem\.Includeonlyreferencesthatareanalyzedindepth\.

\-\-<judge\_reasoning\>:describeinfreetexthowthejudgeappliedthelegalreferencestotheconcretecase,leadingtothedecision\.Donotrepeatthefactualpremises\.

\-Writeanabstractivesummaryoftheissue\(100\-150words\)including:facts,law,reasoning,andconclusion\.Thesummarymustbeself\-containedandunderstandablewithoutreferringtothefulldecisionorotherissues\.

Generalrequirements:

\-Eachissuemustbeautonomous,self\-contained,andformulatedinpreciseanddetailedlegallanguage\.

\-Thefactsmustbedescribedwithsufficientprecisiontoallowanotherjudgetodecideasimilarissueconsistently\.

Requiredoutputformat:XMLonly,asinthefollowingschema:

<issues\>

<issuetitle="issuetitle"\>

<text\>textofthelegalissue\(~30words\)</text\>

<issue\_outcomein\_favor\_of="taxauthority\|taxpayer\|other"\>outcometext</issue\_outcome\>

<factual\_premises\>

<item\>factualpremise1</item\>

<item\>factualpremise2</item\>

</factual\_premises\>

<legal\_references\>

<itemid="D1"type="caselaw"ref="SupremeCourtno\.nnnn/yyyy"/\>

<itemid="D2"type="leg"ref="art\.xlawno\.nnn/yyyy"/\>

</legal\_references\>

<citation\_reason\>

<itemref\_id="D1"\>reasonforcitinglegalreference</item\>

<itemref\_id="D2"\>reasonforcitinglegalreference</item\>

</citation\_reason\>

<judge\_reasoning\>reasoningtext\(~100words\)</judge\_reasoning\>

<summary\>summarytext\(~100words\)</summary\>

</issue\>

</issues\>

Inthe"ref"fieldoflegalreferences,alwaysindicatetheexactcitationoftheact\.Usethe"type"fieldasfollows:

\-type="caselaw"forcaselaw\.Ifmultipledecisionsarecitedconsecutively,groupthemintoasingleitem;

\-type="leg"forlegislation;

\-type="princ"forgenerallegalprinciples\(inthiscase,"ref"containsonlythenameoftheprinciple\);

\-type="admin\_pract"foradministrativepractice\.

OutputONLYtheXMLblockandnothingelse\.

The prompt instructs the LLM to extract the main issues contained in the judgment\. In addition, the prompt also instructs the model to keep issues autonomous\. In other words each issue should be self\-standing and understandable on its own\. Our format therefore aims to decompose a judgment into a set of independent issues and to treat each separately\. This is justified by the fact that judgments are often heterogeneous entities\. Within one judgment, the judge might first decide on the admissibility of the appeal; then once this is declared admissible it goes on to decide on the merits of the case\. These would correspond to two separate issues that indeed have little reason to be grouped together for secondary use\. The XML format still preserves the unity of the judgment by grouping all issues belonging to the same judgment in the`<issues\></issues\>`tags\.

For each issue, the LLM is instructed to extract several fields:

- •`<text\>`, the question of the issue, formulated in a standardized way ’Whether \+ clause’\.
- •`<issue\_outcome\>`the outcome of the issue, i\.e\., the judge’s decision on the matter\. The attribute`in\_favor\_of`taking values in \(`taxpayer, tax authority, other`\) provides a simple classification of the outcome\.
- •`<factual\_premises\>`a list of relevant facts to understand the case
- •`<legal\_references\>`, and`<citation\_reason\>`a list of legal citations, each optionally accompanied by a short sentence describing why it was cited\. Each reference is assigned a type among caselaw, legislation, principle, administrative practice\.
- •`<judge\_reasoning\>`a textual field focused on reporting the application of the law to the particular case\.
- •`<summary\>`a textual summary of the overall issue, with potential overlaps with other fields\.

Listing[2](https://arxiv.org/html/2607.03325#LST2)gives a compact view of an issue extracted from judgment n\. 318/2024 of the Corte di Giustizia Tributaria di primo grado di Teramo; the full extraction, with all fields reported verbatim, is reproduced in Appendix[G](https://arxiv.org/html/2607.03325#A7)\.

Listing 2:Compact view of an extracted XML issue \(full version in Appendix[G](https://arxiv.org/html/2607.03325#A7)\)\.<issues\>

<issuetitle="VATdeductibilityforanagriculturalbusinessnotyetproductive"id="Q1"\>

<text\>Whetherthepurchaseofacapitalasset\(agriculturaltractor\)byanewlyestablished

agriculturalbusiness,notyetproductive,isconnectedtotheentrepreneurialactivity

andthereforedeductible\[\.\.\.\]\.</text\>

<issue\_outcomein\_favor\_of="taxpayer"\>VATisdeductiblebecausethepurchaseisrelatedto

theentrepreneurialactivity,evenintheabsenceofimmediatetaxableoperations\.</issue\_outcome\>

<factual\_premises\>

<item\>Theappellantbeganthesnail\-farmingactivityin2020,anactivitythatrequires

timetobecomeproductive\.</item\>

<item\>In2022,theappellantpurchasedanagriculturaltractorforlandpreparationand

forprocuringplant\-basedfeednecessaryforthefarmingactivity\.</item\>

<item\>\[\.\.\.2furtherfactualpremises\.\.\.\]</item\>

</factual\_premises\>

<legal\_references\>

<itemid="D1"type="caselaw"ref="Cassazionen\.26689/2022;n\.7440/2021;n\.15570/2023"/\>

<itemid="D4"type="leg"ref="Articles4and5DPR633/72"/\>

<item\>\[\.\.\.3furtherlegalreferences\.\.\.\]</item\>

</legal\_references\>

<citation\_reason\>

<itemref\_id="D1"\>NationalcaselawstatesthattherighttodeductVATislinkedtothe

relevanceoftheassettotheentrepreneurialactivity,evenintheabsenceof

immediatetaxableoperations,providedtheassetisfunctionaltotheorganisation

ofthebusiness\.</item\>

<item\>\[\.\.\.2furthercitationreasons\.\.\.\]</item\>

</citation\_reason\>

<judge\_reasoning\>Thejudgeconsidersthatthepurchaseofthetractorisconnectedtothe

taxpayer’sagriculturalactivity,despitetheabsenceofimmediatetaxableoperations

\[\.\.\.\]\.Therefore,thedenialisannulledandtheVATrefundisgranted\.</judge\_reasoning\>

<summary\>Thetaxpayer,whooperatesasnail\-farmingbusiness,purchasedatractorin2022

topreparetheland,despiteproductionnotyethavingbegun\.\[\.\.\.\]Theruling

annulledthedenialandrecognisedthedeductibility\.</summary\>

</issue\>

</issues\>

The example illustrates how each field of the schema is instantiated on a real decision\. A single`<issue\>`is extracted in this case\.

### 3\.2IRAC and the legal syllogism

We now provide a comparison between our structured format and the IRAC and legal syllogism frameworks, to which our format is inspired\. IRAC is a structured legal reasoning framework that stands for ’Issue’ \(the legal question at stake\), ’Rule’ \(the applicable law or precedent\), ’Application’ \(how the rule applies to the facts\), and ’Conclusion’ \(the decision\)\. The parallel of our format with IRAC starts with the idea of extracting issues\. Some of the different XML fields resemble the IRAC elements as follows\.

- •Issue: the`<text\>`field states the issue\. In addition`<factual\_premises\>`provides some additional context\.
- •Rule: the references in`<legal\_references\>`contain the relevant rules\.
- •Application: the field`<judge\_reasoning\>`details how the legal references are applied\. In addition, the items in`<citation\_reason\>`provide some additional information on the reasons why individual references were quoted\.
- •Conclusion:`<issue\_outcome\>`contains the decision on the issue\. The information is partially present also in the`<judge\_reasoning\>`field\.

However, there are differences between IRAC and our format\. First, the`<legal\_references\>`field disaggregates the ’Rule’ component by assigning each citation a type \(case law, legislation, principle, administrative practice\) and an optional`<citation\_reason\>`, making explicit the distinct role each source plays in the reasoning\. Second, IRAC does not distinguish between the facts of the case and the legal issue, while our format separates`<factual\_premises\>`from`<text\>`\. Finally, the field`<summary\>`has no direct counterpart in IRAC\.

Similarly, we can provide a parallel with the legal syllogism\. A legal syllogism on a given issue is composed of the following parts:

1. 1\.Major premise \(rule\): parallel with`<legal\_references\>`\.
2. 2\.Minor premise \(facts\): parallel with`<factual\_premises\>`\.
3. 3\.Conclusion \(judgment\): similar to`<issue\_outcome\>`\.

Although legal syllogism is a useful reference, the idea that judges strictly follow syllogistic reasoning is debated in the literature on legal theory\. Moreover, it is far from evident in unstructured texts, such as tax court decisions, where the reasoning is often implicit, fragmented, or intertwined with procedural matters\. Our format is thereforeinspired bythe syllogism: in designing the fields we had syllogism in mind, but the structure was refined in several respects\. First, the fields are more granular than the syllogistic parts\. Second, and most importantly, a key component of our format is the`<judge\_reasoning\>`field, which captures forms of reasoning that go beyond the syllogistic model, such as reasoning by analogy, balancing of competing principles, or teleological interpretation\. Unlike the major and minor premises of a syllogism, which are based on a fixed logical relationship, both`<legal\_references\>`and`<factual\_premises\>`can contain multiple elements on a single issue, without a one\-to\-one correspondence between them\. This is linked to our format adopting asingle\-stepmodel per legal issue\. In principle, the reasoning leading to the decision of an issue could be decomposed into achain of syllogisms, where each step uses the conclusions of the previous ones as a minor premise, and the conclusion of the last step coincides with the decision on the issue\. Such a multi\-step decomposition would yield a fine\-grained picture of the reasoning\. However, this structure comes at the price of increased ambiguity in extraction and lower reproducibility\. Hence, in our format, each legal issue is associated to a single structured representation, which captures the relevant rules, facts, reasoning, and outcome without decomposing the inferential path into intermediate steps\. This design choice prioritises consistency and extractability\.

### 3\.3Design pressures and potential uses

The schema described above is shaped by the requirements imposed by four anticipated uses of the extracted format\. We discuss each in turn, in each case naming the design choice that follows from the corresponding requirement\.

#### Human consultation\.

In the context of precedent research, as an alternative to reading the text of the judgment, the user may find it easier to grasp the central points of the judgment by looking at a graphical interface displaying the XML fields of one of its issues\. The decomposition into issues also means that the user can engage directly with the issue they are interested in, without having to deal with other potentially irrelevant parts of the judgment\. The format would then play a role similar to that of headnotes, ormassimein the case of Italian judgments\. However, differently frommassime, which select issues based on their novelty and relevance to the uniform interpretation of the law \(funzione nomofilattica\), our pipeline aims to extract all issues addressed in a judgment, including repetitive or well\-settled ones\. In practice, however, we instruct the LLM to focus on the most relevant issues, as extracting the full set of legal issues from every decision can be beyond its reliable capabilities\.

#### Issue\-level retrieval\.

The second use is automated retrieval over the corpus\. Existing legal information systems combine full\-text and semantic methods, both of which suffer when the unit of retrieval is the full judgment\. Judgments are long and contain substantial amounts of structurally necessary but retrieval\-irrelevant text \(standard procedural formulas, section headings, recurring boilerplate\) which introduces noise into the document representation and degrades the precision of full\-text search\. Semantic retrieval, in turn, requires documents to be segmented into \(often overlapping\) chunks and indexed at the chunk level, which increases index size and redundancy and may obscure long\-range dependencies when relevant information is distributed across chunks\. A representation that supports retrieval well must therefore \(i\) isolate single legal questions and \(ii\) produce self\-contained units of appropriate length\. The decomposition into issues and the use of an abstractive`<summary\>`for each issue address both requirements: each extracted issue is a self\-contained unit, several hundred tokens long, that strips boilerplate and structurally irrelevant content while preserving the legal context\.

#### Citation networks\.

The legal references attached to each extracted issue can be used to construct a citation network between judgments\. To this end, a parser is first employed to standardize citation formats\. The resulting network can be further enriched by associating each edge with the textual attributes from`<citation\_reason\>`, capturing the reasons for which a given document is cited\. When a document is cited multiple times across judgments, the aggregation of these attributes over incoming edges provides a useful indication of its practical relevance, highlighting the most common purposes for which the document is invoked\. The construction of such a graph also allows to implement advanced retrieval algorithms such as PageRank\(Brin and Page,[1998](https://arxiv.org/html/2607.03325#bib.bib66)\)\. A natural objection is that such a network could be constructed directly from judgments without relying on issue\-level extraction\. The key difference lies in the set of citations considered\. A network built from full judgments includes all references appearing in the decision, many of which are not central to the court’s reasoning\. In Italian tax law judgments, for example, parties often refer to case law or statutory provisions in the section reporting their submissions \(richieste delle parti\), which are not necessarily taken up in the court’s reasoning\. More generally, decisions frequently include references to standard or procedural provisions \(e\.g\. jurisdictional rules or general principles\) that serve mainly a formal function and add little to the resolution of the dispute\. By contrast, when citations are extracted in association with individual issues, the resulting network is still defined at the judgment level, but its edges are based only on references that are relevant to the resolution of those issues\. This selective inclusion improves the signal\-to\-noise ratio of the network, yielding a representation that more accurately reflects the relationships between decisions\. Appendix[I](https://arxiv.org/html/2607.03325#A9)assesses this argument empirically on the validation set, comparing the citations parsed by Linkoln from the full text and from the reasoning section with the expert\-validated issue\-level references\.

#### Legal reasoning dataset\.

The proposed format can be seen as a simplified and operational model of legal reasoning\. While numerous theoretical models of legal reasoning have been proposed in the literature, there is a lack of large\-scale, structured datasets that instantiate these models in practice\. By extracting issue\-level representations inspired by IRAC and syllogistic structures, our approach provides a first step toward the creation of such datasets\. A corpus of extracted issues could support the training and evaluation of machine learning models for tasks involving legal reasoning, offering a connection between abstract theoretical frameworks and data\-driven methods\.

### 3\.4Balancing Structure, Expressiveness, and Extractability

A central design challenge in defining the proposed XML schema concerns the trade\-off between structural rigidity and expressive adequacy\. On the one hand, a relatively rigid format is desirable to ensure machine readability and facilitate computational use: for instance, modeling each issue as a single\-step judicial syllogism enables consistent extraction of premises, legal references, and outcomes, at the cost of abstracting away from more complex reasoning patterns \(e\.g\. chains of interdependent syllogisms\)\. On the other hand, secondary use by human users often benefits from more flexible, narrative representations \(such as the summary field\) which can capture nuances that are difficult to encode in strictly structured components\. A further tension arises from the heterogeneity of the underlying corpus: Italian tax law judgments exhibit substantial variation in structure and style\. An overly restrictive schema risks forcing the extraction of elements that are not explicitly present in the text, thereby increasing the likelihood of hallucinations or unstable outputs\. Similarly, an overly rich or complex schema may introduce ambiguity, leading to inconsistent representations even across expert annotators\. These considerations are compounded by the practical limitations of LLM\-based extraction, as increasingly complex templates place greater demands on the model’s ability to produce reliable outputs\. The proposed format reflects the compromise objective to provide enough structure for computational use while maintaining flexibility to accommodate variability in the source material and to support meaningful human interpretation\.

## 4Extraction Pipeline

### 4\.1Dataset

The corpus used in this work consists of first\- and second\-instance Italian tax\-court judgments retrieved from the official database of tax\-court decisionsMinistero dell’Economia e delle Finanze – Dipartimento della Giustizia Tributaria \([2024](https://arxiv.org/html/2607.03325#bib.bib2)\)\. At the time of extraction we retrieved almost all the available decisions published between 2021 and 2023, amounting to401,349401\{,\}349documents\.

Judgments are distributed by the database in PDF format\. We converted each document to plain text using the PyMuPDF libraryArtifex Software, Inc\. and McKie \([2026](https://arxiv.org/html/2607.03325#bib.bib3)\)\. The resulting texts are heterogeneous in layout and typographic conventions, reflecting the absence of a standardized drafting template across tax courts\. To identify the main structural headings of each judgment \(in particular the section “Motivi della decisione”, which contains the judge’s reasoning\) we relied on a regex\-based matcher\. Headings are not entirely uniform across decisions, so the matcher succeeds only on a subset of documents\.

Before extraction we applied two filters intended to remove documents that do not contain legal issues and would therefore introduce noise into the subsequent analysis\. First, we discarded judgments whose “Motivi della decisione” section is shorter than680680characters when the section was identified by the aforementioned matcher, and judgments whose total length is below2,3002\{,\}300characters when it was not: in both cases the text is too short to contain any interesting reasoning\. Second, we filtered out judgments containing variants of “estinzione del giudizio” or “cessata materia del contendere”, as these expressions signal that the proceedings were terminated or that the case had become moot, so that the decision does not resolve any issue on the merits\. Of the401,349401\{,\}349judgments retrieved,328,605328\{,\}605\(81\.9%81\.9\\%\) passed both filters and were processed by the extraction pipeline described in Section[4\.3](https://arxiv.org/html/2607.03325#S4.SS3)\. For the manual validation reported in Section[5](https://arxiv.org/html/2607.03325#S5), we drew a subsample of5050judgments uniformly at random from the328,605328\{,\}605judgments in the final structured corpus\.

### 4\.2Overview of the extraction pipeline

The overall extraction process can be represented with the flow chart in Figure[1](https://arxiv.org/html/2607.03325#S4.F1)\. The text of each judgment is passed to LLM together with a prompt; an XML file, containing the structured information, is produced in output\. To verify that the XML outputted by the LLM conforms to what one asked for, the output is validated using an XSD \(schema\) file that specifies the fields that must be present and the values they are allowed to take\. The aim of the rest of the pipeline is to eliminate hallucinations from elements of`<legal\_references\>`\. To do so, the pipeline extracts and converts to a standardized format all references present in the text of the judgment\. This is accomplished with the aid of the Linkoln\(Bacciet al\.,[2019](https://arxiv.org/html/2607.03325#bib.bib1)\)library\. The same standardization is applied to references in the XML\. Then the two sets are compared and references found in the XML but not in the judgment are marked as potential hallucinations and eliminated\.

![Refer to caption](https://arxiv.org/html/2607.03325v1/x1.png)Figure 1:Stylized visualization of the extraction and verification pipeline\.
### 4\.3LLM extraction

In this section we detail the XML format extraction procedure\. To carry out the extraction we used the DeepSeek V3\-0324 model\(Liuet al\.,[2024](https://arxiv.org/html/2607.03325#bib.bib68)\)\. After informally reviewing several LLMs for the issue extraction task, we chose DeepSeek because among the models achieving a satisfactory performance, it was the one with the best balance between capabilities and price \($0\.20/M input tokens, $0\.77/M output tokens\)\. In contrast, top\-range models can cost up to twenty times as much: for example Claude Sonnet 4\.5 was priced at $3/M input tokens and $15/M output tokens\. Many of the potential uses outlined in Section[3\.3](https://arxiv.org/html/2607.03325#S3.SS3)can only be achieved if a large number of judgments is processed\. In our case, this amounts to processing several hundred thousand documents, and at this scale a ten\-fold increase in price is very relevant to the feasibility of the extraction\. The title\-correction step described below uses the slightly newer DeepSeek V3\.1 model \($0\.21/M input tokens, $0\.79/M output tokens\)\. At list prices, the per\-document cost amounts to approximately $0\.0032 for the main extraction with V3\-0324 and $0\.00034 for the title\-correction step with V3\.1; processing the full corpus of328,605328\{,\}605judgments cost approximately $1,150 in total\.

The pre\-processed corpus described in Section[4\.1](https://arxiv.org/html/2607.03325#S4.SS1)is the input to the extraction pipeline\. We detail below the prompting strategy and a post\-extraction correction step\.

1. 1\.If the “Motivi della decisione” section identified during preprocessing \(Section[4\.1](https://arxiv.org/html/2607.03325#S4.SS1)\) is shorter than41004100characters, the XML was extracted using prompt[1](https://arxiv.org/html/2607.03325#LST1), followed by the entire text of the judgment\.111When the section “Motivi della decisione” is not present we default to the single step prompt\.Otherwise, a two\-step approach was used\. In the first step, prompt[3](https://arxiv.org/html/2607.03325#LST3)is passed together with the judgment’s text to the LLM\. This first prompt identifies the issues within the judgment and extracts the`<text\>,<summary\>, <issue\_outcome\>`for each issue\. In the second step, the following procedure is repeated for all issues\. We pass prompt[4](https://arxiv.org/html/2607.03325#LST4), the fields`<text\>`and`<issue\_outcome\>`of the issue, and the text of the judgment to the LLM\. As output we collect the`<factual\_premises\>, <legal\_references\>, <citation\_reason\>, <judge\_reasoning\>`fields\. The new fields are then merged to ones extracted in step 1, obtaining the complete XML format, as presented in[1](https://arxiv.org/html/2607.03325#LST1)\. We chose to split the extraction of the issues because the LLM was not producing satisfactory outputs on longer judgments containing several issues\. We remarked that the LLM had a ’preferred’ output length range and that in the case of judgments requiring longer XMLs, the single prompt approach led to too concise outputs and missing issues\. We believe that using more powerful LLMs the single prompt approach would suffice\. In our analysis, we relied exclusively on zero\-shot prompting\. To implement few\-shot prompting one would need to provide the text of several judgments along with the extracted issues\. Given that judgments are usually several pages long, this is unfeasible\. Additionally, there is the risk of biasing the extraction towards the topics selected in the example judgments\.
2. 2\.Early experiments with this setting revealed that sometimes the`<text\>`field was not in the "Whether \+ clause" form, with the LLM mistakenly using a direct interrogative form, or other incorrect forms\. For example instead of writing`<text\>Whether the taxpayer correctly calculated the VAT due </text\>`the LLM could sometime formulate the issue as`<text\>Did the taxpayer correctly calculate the VAT due?</text\>`\. This problem affected about 10% of the extracted issues\. We therefore attempted to correct these fields automatically using few\-shot prompting\. Prompt[5](https://arxiv.org/html/2607.03325#LST5), followed by each of the`<text\>`fields, was fed into the LLM \(DeepSeek V3\.1 for this step\)\. Once again we expect this step not to be necessary if one is using a more powerful LLM\. After this additional processing basically all issues were in the correct form\.

The steps we described illustrate in detail the procedure employed to extract the XML\. We stress that irrespectively of whether prompt[1](https://arxiv.org/html/2607.03325#LST1)or[3](https://arxiv.org/html/2607.03325#LST3)and[4](https://arxiv.org/html/2607.03325#LST4)were used, the XML format is always the same\. The LLM was run at zero temperature and with a fixed random seed using API calls to third party providers\. Despite these measures the output of the model was not deterministic, yielding different results on identical inputs\. We quantify the impact of this non\-determinism in Appendix[H](https://arxiv.org/html/2607.03325#A8), where we run the pipeline twice on the validation set and compare the two runs with the same set\-based metrics used for the human validation\.

### 4\.4Verification procedure

We now move to describing the post extraction checks we performed on the output of the LLM\. The checks we are able to carry out automatically involve three aspects

1. 1\.Syntax of the XML\. Occasionally \(in less than 3% of cases\) the LLM misses an XML closing tag ’`\>`’ or makes another syntax error\. This causes the XML parser to throw an error\. In this case the corresponding judgment is rerun through the LLM until the XML is well formed\.
2. 2\.Schema conformity of the XML\. The prompt specifies a particular structure for the output XML, with constraints on elements and attributes \(e\.g\. the`type`attribute in`legal\_references`can only take values in`leg,caselaw, princ, admin\_pract`\. To check that the LLM respected these constraints we employ an XML Schema Definition \(XSD\) file\. This file allows to express the constraints in a machine\-readable form\. One can then automatically verify the conformity of the XMLs\. The fraction of non conformal XMLs is below 5%\. We rerun the corresponding judgments through the LLM pipeline until the output XML passes the XSD check\.
3. 3\.The most onerous check we carry out is on the presence of hallucinations within`<legal\_references\>`\. LLMs can in fact make up issues, facts, and legal references that are not present in the judgment\. Ideally one would like to put in place some system that allows to detect and then eliminate hallucinations automatically\. For free\-text fields, this is quite hard to accomplish: essentially one would need an additional LLM to evaluate the extraction outputs \(with the risk that also this second model hallucinates\)\. A more rigorous approach can be taken concerning legal references, since their presence in the text of the judgment can be more readily verified\. Citations can be of four types: legislation, case law, administrative practice, and principle of law\. Let us focus for the moment on non\-principle citations\. We use several methods in succession to determine whether a citation is present in the text\. 1. \(a\)Textual match\. If the reference text \(contained in the`ref`attribute\) appears in the judgment’s text, then the citation is verified\. To make matches more likely, we convert text to lowercase and remove spaces and non alphanumeric characters\. 2. \(b\)Unique Resource Identifier \(URI\) based matching\. If possible, we assign a URI \(for example ECLI or CELEX\) to the legal reference using Linkoln\. Then, we assign URIs to all legal citations appearing in the judgment\. Afterwards, we check whether the URI of the reference in the XML is among those found in the judgment\. 3. \(c\)Number\-year matching\. Legal documents can usually be uniquely identified by the following: type of document \(e\.g\. judgment, decree, law, order\), the issuing authority \(e\.g\. regional tax court, president of the republic\), number, and year\. It becomes significantly easier to match references based exclusively on number and year as these are numerical values and therefore not subject to abbreviation\. Given the number and year of a reference in the XML, we first look for the number in the judgment\. If this is found we look for the year in a window of±100\\pm 100characters from the number\. If also the year is found, the reference is considered verified\.222This approach potentially misses hallucinated references that have the same number and year as the ones in the judgment but different authority\. This is however very rare\. The above pipeline is unsatisfactory for citations of type`princ`, since principles of law \(e\.g\. principle of proportionality, principle of legality\) are usually not associated with a URI or a combination of number \+ year\. On top of this, we notice that LLMs do not have a clear understanding of what constitutes a principle, and tend to put other legal concepts \(e\.g\. taxability of building areas, fiscal residence\) under this category\. We therefore employ a different approach\. Based on the whole corpus of extracted issues \(i\.e\. the328,605328\{,\}605processed files\), minus the 50 test judgments, the experts jointly produced a list of 182 allowed legal principles\. For each principle in the list a canonical form was established together with a series of possible variations \(e\.g\. principle of legal certainty, principle of certainty of the law\)\. Each reference of type`princ`is kept if both of the following conditions are met\. \(1\) there is a textual match in the judgment \(computed in the same way as with other citations\) and \(2\) the reference matches one of the principles in the list\. The flowchart in figure[2](https://arxiv.org/html/2607.03325#S4.F2)illustrates the logic of the whole hallucination checker\. This procedure is applied to each`<item\>`in`<legal\_references\>`333It is possible, that one item contains more than one reference\. In this case we remove the item only if all references identified within the item are recognized as hallucinated\. ![Refer to caption](https://arxiv.org/html/2607.03325v1/x2.png)Figure 2:Flowchart of the hallucination check for items in <legal\_references\>\.

## 5Validation Protocol

To assess the quality of the extraction pipeline, we conducted a manual validation of the LLM outputs on the5050\-judgment subsample described in Section[4\.1](https://arxiv.org/html/2607.03325#S4.SS1)\. Two tax\-law experts \(PhD in tax law\), hereafterA1A\_\{1\}andA2A\_\{2\}, served as annotators\.

### 5\.1Development of the annotation guidelines

Prior to the validation task,A1A\_\{1\}andA2A\_\{2\}jointly developed a set of annotation guidelines through an iterative process consisting of two preparatory rounds, both carried out on judgments disjoint from the test set\. In a first round, the annotators jointly read and discussed1010judgments while examining the corresponding LLM outputs, in order to identify recurring ambiguities and edge cases \(e\.g\. partially overlapping issues, citations to general legal principles, judgments concerning exclusively costs\) and to converge on a common scoring scheme\. In a second round, they independently annotated1010additional judgments under blind conditions, then compared their outputs, resolved disagreements through discussion, and used the residual divergences to produce the final version of the guidelines\. The resulting document fixes both the scope of the annotation task and the criteria for each metric; the full guidelines are reported in Appendix[E](https://arxiv.org/html/2607.03325#A5)\.

The central outcome of this phase was the adoption of a working definition of a*legal issue*\(*questione giuridica*\): a legal question explicitly decided by the judge in the reasoning section of the judgment, requiring the interpretation or application of a legal source, and supported by argumentative content sufficient to fill the structured fields of the XML format\. This definition focuses on the reasoning of the judge as it emerges from the argumentative part of the decision, rather than on the requests of the parties, and was informed by the literature in civil procedural law and the philosophy of law\(Villa,[2023](https://arxiv.org/html/2607.03325#bib.bib69); Ricci,[1987](https://arxiv.org/html/2607.03325#bib.bib71); Liebmanet al\.,[2007](https://arxiv.org/html/2607.03325#bib.bib70)\)\. The choice is consistent with the intended use of the extracted format, which is to support access to legal issues as they are addressed and resolved by judges\. The definition deliberately admits a small residual margin of interpretation: as the inter\-annotator agreement results in Section[6](https://arxiv.org/html/2607.03325#S6)show, even two experts working under the same guidelines may segment a judgment into slightly different but largely overlapping sets of issues\.

### 5\.2Annotation procedure

For each judgment, the annotator carried out the following four steps, in order\. All annotations were recorded in a spreadsheet with one row per extracted issue\.

1. 1\.Independent issue identification\.Without consulting the LLM output, the annotator read the full text of the judgment and compiled a list of the legal issues present, formulating each as a*Whether*\+ clause \(Italian:*Se \+ congiuntivo*\), that is, in the same form used by the LLM in the`<text\>`field\.
2. 2\.Issue alignment and scoring\.The annotator then examined the LLM\-extracted issues and aligned them with their own list, on the basis of content rather than literal wording\. This step partitions the LLM issues into*present*\(those matching an issue in the annotator’s list\) and*absent*\(those that do not correspond to any issue the annotator considered present in the judgment\)\. The alignment also identifies issues in the annotator’s list that the LLM failed to extract\. Across the entire test set, fewer than five alignments were considered ambiguous and were resolved by discussion between the two annotators\. For each LLM issue marked as present, the annotator additionally rated, on a11–55Likert scale, the appropriateness of its*level of generality*\(neither too narrow nor too broad for a caselaw research\) and the quality of its*legal language*\.
3. 3\.Citation evaluation\.For each present issue, and adopting the LLM’s formulation of that issue, the annotator independently compiled the list of legal references they considered relevant to its resolution, without looking at`<legal\_references\>`\. This list was then aligned with the LLM\-extracted citations\. The annotator further scored the`<citation\_reason\>`entries on a binary scale \(11if the stated reason corresponds to the use the judge makes of the citation in the decision,0otherwise\)\.
4. 4\.Free\-text field scoring\.Finally, the annotator scored the remaining textual fields\. The`<summary\>`field was rated along four dimensions on11–55Likert scales:*correctness*\(absence of elements not present in the source\),*form*\(coherence, readability, legal terminology\),*completeness*\(coverage of the key points of the issue\), and overall*satisfaction*\. The`<factual\_premises\>`,`<judge\_reasoning\>`, and`<issue\_outcome\>`fields each received a single overall satisfaction score on the same scale\.

### 5\.3Two\-phase rollout

The5050judgments were annotated in two phases\. In the*first phase*, bothA1A\_\{1\}andA2A\_\{2\}independently annotated the same2020judgments under double\-blind conditions\. This shared subset serves the purpose of supporting the computation of inter\-annotator agreement \(IAA\)\. Where IAA is high, deviations of the LLM output from the human annotations can plausibly be attributed to model error; where IAA is lower, part of the gap should instead be read as an irreducible disagreement between experts on a task that even humans would struggle to resolve consistently\. In the*second phase*, the remaining3030judgments were split evenly between the two annotators, each annotating1515judgments independently\. Each annotator therefore produced annotations for a total of3535judgments, with2020in common\.

### 5\.4Evaluation metrics

For both issue extraction and citation extraction, the LLM output and the annotator output are sets of items \(issues or citations\), and we evaluate them with standard precision \(P\), recall \(R\), and F1\. To define these unambiguously, fix an annotatorA∈\{A1,A2\}A\\in\\\{A\_\{1\},A\_\{2\}\\\}\. Then

PLLM\|A=\|LLM∩A\|\|LLM\|,RLLM\|A=\|LLM∩A\|\|A\|,\\mathrm\{P\}\_\{\\mathrm\{LLM\}\|A\}=\\frac\{\|\\mathrm\{LLM\}\\cap A\|\}\{\|\\mathrm\{LLM\}\|\},\\qquad\\mathrm\{R\}\_\{\\mathrm\{LLM\}\|A\}=\\frac\{\|\\mathrm\{LLM\}\\cap A\|\}\{\|A\|\},i\.e\. the annotator’s set is treated as ground truth\. The same formulas, withA1A\_\{1\}andA2A\_\{2\}in turn playing the role of ground truth, are used to compute IAA on the shared subset; the two directions \(A1A\_\{1\}as ground truth forA2A\_\{2\}and vice versa\) yield different precision and recall but the same F1\.

All set\-level scores in the main text are computed by pooling items across the relevant set of judgments before computing the metric\. As a robustness check, Appendix[D](https://arxiv.org/html/2607.03325#A4)reports these metrics averaged per judgment rather than pooled, and Appendix[D\.3](https://arxiv.org/html/2607.03325#A4.SS3)against a combined\-annotator ground truth\. To quantify the sampling uncertainty of these pooled scores, we accompany the main tables with95%95\\%confidence intervals computed with a non\-parametric cluster bootstrap at the judgment level: the judgments, with all their issues and citations, are resampled with replacement \(B=10,000B\{=\}10\{,\}000\), the pooled metric is recomputed on each replicate, and we report the2\.52\.5th–97\.597\.5th percentile interval\. Resampling entire judgments preserves the dependence among items belonging to the same decision; the judgment is also the unit at which the validation sample was drawn\. Intervals are shown in brackets in the corresponding tables\. For the qualitative Likert scores, we report exact agreement, within\-one\-point agreement, and Gwet’sAC2\\mathrm\{AC2\}coefficient with quadratic weights\(Gwet,[2014](https://arxiv.org/html/2607.03325#bib.bib53)\), the latter being designed to remain stable in the high\-prevalence regime in which annotators agree on most items\. We do not adopt Cohen’sκ\\kappaas the primary agreement statistic becauseκ\\kappais known to collapse under marginal skew\(Feinstein and Cicchetti,[1990](https://arxiv.org/html/2607.03325#bib.bib54); Wongpakaranet al\.,[2013](https://arxiv.org/html/2607.03325#bib.bib55)\), a condition that holds for several of our dimensions; for completeness,κ\\kappavalues are reported in Appendix[F](https://arxiv.org/html/2607.03325#A6)together with a brief discussion of their interpretation here\.

For the hallucination filter, we additionally report its specificity \(the fraction of valid citations correctly retained\), its precision and recall as a binary classifier, and the residual hallucination rate among the citations it keeps\. These are defined and discussed in Section[6\.3](https://arxiv.org/html/2607.03325#S6.SS3)\.

After validation, hallucinated issues \(i\.e\. LLM\-extracted issues that the annotator marked as absent\) are dropped from the analysis of all other fields, since their associated facts, references, reasoning, and summary have no annotator counterpart\.

## 6Results

We organize the results as follows\. Section[6\.1](https://arxiv.org/html/2607.03325#S6.SS1)reports IAA and LLM\-vs\-annotator agreement on the extraction of legal issues; Section[6\.2](https://arxiv.org/html/2607.03325#S6.SS2)does the same for legal citations \(`<legal\_references\>`and`<citation\_reason\>`\); Section[6\.3](https://arxiv.org/html/2607.03325#S6.SS3)evaluates the hallucination filter as a stand\-alone component; and Section[6\.4](https://arxiv.org/html/2607.03325#S6.SS4)reports the qualitative Likert scores on the free\-text fields\.

### 6\.1Legal issues extraction

We start by considering the average number of legal issues per judgment extracted by the LLM and by each annotator\. From the results, displayed in Table[1](https://arxiv.org/html/2607.03325#S6.T1), a consistent ordering emerges:A1A\_\{1\}identifies more issues thanA2A\_\{2\}, who in turn identifies more than the LLM, both on the shared subset and on the full sets\. The LLM extracts on average1\.51\.5issues per judgment, against roughly22for the human annotators, indicating a tendency to under\-extract\.

Table 1:Average number of issues identified per judgment \(total issues in parentheses\)\.Next, we consider the inter annotator agreement on the shared set \(N=20N=20\)\. The two annotators show very strong agreement on issue identification: F1=88\.1%\{=\}88\.1\\%on the shared set \(Table[2](https://arxiv.org/html/2607.03325#S6.T2), top rows\)\. The residual disagreement, despite shared guidelines, sets the natural ceiling against which the LLM should be evaluated\. Table[2](https://arxiv.org/html/2607.03325#S6.T2)also shows the performance of the LLM’s extraction against the ground truth set by the experts\. The LLM achieves high precision against both annotators \(93\.3%93\.3\\%\), meaning that the model rarely produces an issue that the annotator considers absent \(only two such cases onN=20N\{=\}20, confirmed independently by both annotators\)\. Recall is lower and varies with the reference annotator: the LLM recovers about60%60\\%ofA1A\_\{1\}’s issues but76%76\\%ofA2A\_\{2\}’s\. The pattern is confirmed on the fullN=35N\{=\}35sets \(Table[3](https://arxiv.org/html/2607.03325#S6.T3)\), where precision remains above94%94\\%while recall ranges from69\.9%69\.9\\%\(againstA1A\_\{1\}\) to82\.3%82\.3\\%\(againstA2A\_\{2\}\)\. Across all comparisons, the dominant source of error is under\-extraction rather than hallucination\. This profile is partly by design\. By asking the model for the “one or two” main issues in the prompt, we deliberately trade recall for precision\. Hallucinated issues are indeed rare in absolute terms: on the whole dataset \(N=50N=50\), only44of the7878LLM\-extracted issues were marked as absent by either annotator\.

These44issues are excluded from the subsequent analyses\.

Table 2:Pairwise agreement on issue extraction \(N=20N\{=\}20shared judgments\)\. Each row treats the second item as ground truth\. In brackets:95%95\\%judgment\-level bootstrap confidence intervals\.Table 3:LLM issue extraction evaluated against each annotator’s full set \(N=35N\{=\}35\)\. In brackets:95%95\\%judgment\-level bootstrap confidence intervals\.
### 6\.2Legal citation extraction

All citation results in this section refer to the LLM output*after*the hallucination filter has been applied\. Unless otherwise noted, citation metrics are restricted to the issues marked as present by the relevant annotator\.

Table[4](https://arxiv.org/html/2607.03325#S6.T4)reports the average number of legal references per present issue\. We remark that the LLM extracts about the same number of references per issue as the annotators\.

Table 4:Average number of legal references identified per present issue \(total citations in parentheses\)\.The two annotators show near\-perfect agreement on the identification of citations relevant to each issue \(F1=97\.1%\{=\}97\.1\\%; Table[5](https://arxiv.org/html/2607.03325#S6.T5)\)\. This is coherent with the less discretionary nature of the task\. Against this reference, the LLM achieves F1 around74%74\\%, with precision around72%72\\%and recall around7575–77%77\\%\. Notice that most of the LLM citations identified as irrelevant by the annotators are still present in the judgment text \(and so are not hallucinations\); they are simply not deemed relevant to the resolution of the specific issue\. We remark that running Linkoln on the “Motivi” section of the judgment is a competitive baseline on recall \(Appendix[I](https://arxiv.org/html/2607.03325#A9)\); the LLM’s distinctive contribution is attributing each citation to the specific issue it resolves\.

Table 5:Pairwise agreement on citation extraction \(N=20N\{=\}20shared judgments,2828present issues\)\. Each row treats the second item as ground truth\. In brackets:95%95\\%judgment\-level bootstrap confidence intervals\.On each annotator’s fullN=35N\{=\}35set \(Table[6](https://arxiv.org/html/2607.03325#S6.T6)\), precision is stable at73%73\\%and recall is in the range7070–80%80\\%, depending on the annotator\.

Table 6:LLM citation extraction evaluated against each annotator’s full set \(N=35N\{=\}35\)\. In brackets:95%95\\%judgment\-level bootstrap confidence intervals\.#### Faithfulness of citation reasons\.

For each LLM\-extracted citation, the annotators also scored the corresponding`<citation\_reason\>`field on a binary scale \(Section[5](https://arxiv.org/html/2607.03325#S5), step 3\):11if the stated reason corresponds to the use that the judge makes of the citation in the decision,0otherwise\. The natural summary statistic is the fraction of LLM\-extracted reasons judged faithful, pooled across issues\. Table[7](https://arxiv.org/html/2607.03325#S6.T7)reports this fraction on the shared subset \(N=20N=20,4949scored reasons\)\. The two annotators agree on the count of faithful reasons in96\.4%96\.4\\%of issues\. On the full corpus \(N=50N=50\), the LLM achieves a faithfulness rate of80%80\\%, indicating that four out of five citation reasons produced by the model correctly summarize the role that the corresponding citation plays in the decision\.

Table 7:Citation\-reason faithfulness: fraction of LLM\-extracted citation reasons judged faithful by the annotators \(means\)\. Shared subset:N=20N\{=\}20,2828present issues,4949reasons\. Full corpus:N=50N=50\.

### 6\.3Hallucination filter

We now evaluate the hallucination detection step as a stand\-alone component, on the full corpus \(N=50N\{=\}50,264264initially extracted citations\)\. Before and after applying the filter, the annotators recorded whether each reference actually appears in the judgment text; this provides the ground truth against which the filter is assessed\. In the case of principles, we consider hallucinated references that do appear in the text but that clearly do not correspond to a recognized legal principle\.

Out of 264 total citations, 31 were found to be hallucinated\. Of the 31, 20 were of type principle showing that the LLM performs particularly poorly on this category, which over the test set of 50 judgments only counts 29 references, giving a hallucination rate of 69% on principles\.444One can break this further down into concepts that appear in the judgment’s text but are not principles \(12/20\) and proper principles that are not cited in the judgment \(8/20\)\.\. In comparison, the hallucination rate for other references is11/235=4\.7%11/235=4\.7\\%\. The filter removes3636of the264264extracted citations \(29 truly hallucinated and 7 not\), reducing the output set to228228\. Table[8](https://arxiv.org/html/2607.03325#S6.T8)shows the net effect: the overall hallucination rate drops from11\.7%11\.7\\%\(95%95\\%CI\[6\.6,17\.8\]\[6\.6,17\.8\]\) to0\.9%0\.9\\%\(\[0\.0,2\.3\]\[0\.0,2\.3\]\)\. Crucially, this comes at a very low cost in valid citations: of the233233non\-hallucinated citations, only77\(3%3\\%\) are incorrectly removed\.

Table 8:Effect of the hallucination filter \(N=50N\{=\}50\)\. In brackets:95%95\\%judgment\-level bootstrap confidence intervals\.The filter’s full confusion matrix is reported in Table[9](https://arxiv.org/html/2607.03325#S6.T9)\.

Table 9:Confusion matrix of the hallucination detector \(N=50N\{=\}50,264264citations\)\. Specificity=97\.0%\{=\}97\.0\\%\.There are only two residual hallucinations, both of type`caselaw`\. These correspond to made up references whose year and number match some combination of year and number found in the judgment, but the issuing authority is incorrect\.

### 6\.4Qualitative scores on free\-text fields

Table[10](https://arxiv.org/html/2607.03325#S6.T10)reports inter\-annotator agreement on the11–55Likert scales, computed on the shared subset \(N=20N\{=\}20,2828present issues\), alongside the average LLM score on each dimension, computed on the full corpus \(N=50N\{=\}50, with shared judgments contributing the mean of the two annotators’ scores\)\.

Agreement is uniformly high across all dimensions\. The exact agreement ranges from64\.3%64\.3\\%\(generality of the issue\) to100%100\\%\(outcome satisfaction\); the agreement within a point is at least89%89\\%in every dimension and reaches100%100\\%in legal language, form of the summary and outcome satisfaction\. The chance\-corrected Gwet’sAC2\\mathrm\{AC2\}never drops below0\.930\.93\. The two annotators thus converge on a shared expert evaluation of the LLM’s free\-text output\. Cohen’sκ\\kappavalues for the same data are reported and discussed in Appendix[F](https://arxiv.org/html/2607.03325#A6)\. We remark, however, that in our saturated regime, where most items receive the same ratingκ\\kappais unstable and not informative on its own\.

Table 10:Inter\-annotator agreement on Likert dimensions \(N=20N\{=\}20,2828present issues\) and average LLM scores \(N=50N\{=\}50\)\. EA: exact agreement; W1: within\-one\-point agreement; AC2: Gwet’sAC2\\mathrm\{AC2\}with quadratic weights\.

## 7Discussion and Conclusion

We have presented an extraction pipeline that decomposes court judgments into autonomous*issues*represented in a structured XML schema grounded in IRAC and in the legal syllogism\. The pipeline is paired with an automatic hallucination\-detection step that compares LLM\-extracted legal references against the judgment text after normalization to standard identifiers \(urn:nir, CELEX, ECLI\)\. The pipeline was developed and validated on a corpus of Italian first\- and second\-instance tax\-court judgments, a sector characterized by hypertrophic case\-law production, heterogeneous drafting styles \(partly due to the involvement of non\-career honorary judges\), and a thematic complexity that draws on civil, criminal, and commercial law\.

The validation on5050judgments, doubly annotated by two tax\-law experts on a shared subset of2020, supports four main claims\. First, issue extraction is high\-precision and moderate\-recall: of the7878issues extracted by the LLM, only44were marked as absent by either annotator, while the model recovers between69\.9%69\.9\\%and82\.3%82\.3\\%\(N=35N=35\) of the issues identified by the human annotators\. Under\-extraction is the dominant error\. Second, citation extraction shows moderate precision and recall, both above70%70\\%\. Third, the hallucination filter effectively removes most fabricated references: at the cost of incorrectly removing only77of233233valid citations \(3\.0%3\.0\\%\), it eliminates2929of3131hallucinated references, bringing the residual hallucination rate from11\.7%11\.7\\%to0\.9%0\.9\\%\. The type of reference most subject to hallucinations are principles, with a hallucination rate of69%69\\%\. We attribute this to LLMs having little knowledge of what constitutes a legal principle and therefore tending to include irrelevant elements\. Fourth, the qualitative Likert scores on the free\-text fields are uniformly above4\.6/54\.6/5on the summary, judge’s reasoning, and issue outcome, with slightly lower \(yet strong\) scores on factual premises and on the level of generality of the issue formulation, due to both being intrinsically more discretionary tasks\.

A methodological observation cuts across these results\. The two annotators converged on a shared expert evaluation across nearly all tasks, with the lowest \(yet still strong\) agreement on issue identification itself\. The choice of how to segment a decision into autonomous questions retains an irreducible subjective margin: part of the gap between LLM and humans on issue extraction is therefore a property of the task\.

Beyond the specific results, the contribution we wish to emphasize is methodological\. An issue\-based, IRAC\-aligned representation, validated by experts and equipped with citation\-level hallucination control, offers a template for structured extraction over case law that is independent of the specific jurisdiction and of the specific LLM\. Our validation effort demonstrates that current LLMs are able to reliably instantiate the proposed XML format from judgments\. This opens up a range of applications, from issue\-level retrieval, to the construction of citation networks whose edges reflect actual reasoning rather than incidental mentions, to the creation of large\-scale datasets that provide a structured representation of the reasoning contained in case law\.

#### Limitations and future work

The validation rests on5050judgments and two annotators, which, while consistent with the cost of expert annotation, limits the tightness of the reported estimates\. The results are tied to one model \(DeepSeek V3\) and one corpus \(Italian tax\-court judgments\); LLM outputs were moreover non\-deterministic even at zero temperature and fixed random seed, in line with recent findings\(Blair\-Stanek and Van Durme,[2025](https://arxiv.org/html/2607.03325#bib.bib46)\), so replication on other models and other branches of law is needed to confirm generality\.

Several directions naturally extend the present contribution\. First, one can apply the same format to other types of judgments, both within Italian jurisprudence and across jurisdictions, to test the generality of an issue\-based representation\. A second direction is to exploit the`<citation\_reason\>`field to enrich the description of cited documents, for instance by associating each piece of legislation with the list of issues that cite it and with the reasons for citation\. A third, related direction is to leverage the structured textual fields and the citation network for retrieval, possibly combining issue\-level vector representations with graph\-based ranking algorithms such as PageRank computed on the resulting issue\-to\-issue citation graph\.

## 8Acknowledgments

P\.V\. and G\.P\. acknowledge support from UKRI FLF Scheme \(No\. MR/X023028/1\)

## 9Code and Data Availability

The validation code and data are available at[https://github\.com/giovannipiccioli/legal\_issues\_extraction](https://github.com/giovannipiccioli/legal_issues_extraction)\. The repository contains the expert annotations, the raw and post\-filter XML extractions of the 50 test judgments for both runs, the list of 182 canonical legal principles, and five notebooks that reproduce every table in Section[6](https://arxiv.org/html/2607.03325#S6)and the appendices\.

## References

- A case study for automated attribute extraction from legal documents using large language models\.Artificial Intelligence and Law,pp\. 1–22\.Note:Online\-first pagination shown; later assigned to vol\. 34, pp\. 245–266External Links:[Document](https://dx.doi.org/10.1007/s10506-024-09425-7)Cited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px2.p1.1)\.
- Artifex Software, Inc\. and J\. X\. McKie \(2026\)PyMuPDF documentationNote:Accessed: 2026\-04\-29External Links:[Link](https://pymupdf.readthedocs.io/en/latest/)Cited by:[§4\.1](https://arxiv.org/html/2607.03325#S4.SS1.p2.1)\.
- C\. Arvin \(2025\)Identifying legal holdings with llms: a systematic study of performance, scale, and memorization\.InProceedings of the Twentieth International Conference on Artificial Intelligence and Law,pp\. 404–408\.External Links:[Document](https://dx.doi.org/10.1145/3769126.3769128)Cited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px3.p1.1)\.
- K\. D\. Ashley \(2017\)Artificial intelligence and legal analytics: new tools for law practice in the digital age\.Cambridge University Press\.External Links:[Document](https://dx.doi.org/10.1017/9781316761380)Cited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px1.p1.1)\.
- L\. Bacci, T\. Agnoloni, C\. Marchetti, and R\. Battistoni \(2019\)Improving public access to legislation through legal citations detection: the linkoln project at the italian senate\.Knowledge of the Law in the Big Data Age317,pp\. 149–158\.External Links:[Document](https://dx.doi.org/10.3233/FAIA190017)Cited by:[§1\.1](https://arxiv.org/html/2607.03325#S1.SS1.p3.2),[§1](https://arxiv.org/html/2607.03325#S1.p6.1),[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px4.p1.1),[§4\.2](https://arxiv.org/html/2607.03325#S4.SS2.p1.1)\.
- A\. Belfathi, N\. Hernandez, and L\. Monceaux \(2023\)Harnessing gpt\-3\.5\-turbo for rhetorical role prediction in legal cases\.arXiv preprint arXiv:2310\.17413\.Note:Also published as peer\-reviewed paper at JURIX 2023 \(DOI 10\.3233/FAIA230964\)External Links:[Document](https://dx.doi.org/10.48550/arXiv.2310.17413)Cited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px2.p1.1)\.
- T\. Bench\-Capon, K\. Atkinson, F\. Bex, H\. Prakken, and H\. Verheij \(2024\)Computational models of legal argument\.InHandbook of Formal Argumentation,pp\. 1–123\.Note:No DOI available; College Publications does not assign DOIs to this handbookCited by:[§1](https://arxiv.org/html/2607.03325#S1.p3.1)\.
- T\. Bench\-Capon \(2020\)Explaining legal decisions using irac\.InCEUR Workshop Proceedings,Vol\.2669,pp\. 74–83\.Cited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px1.p1.1)\.
- I\. Benedetto, L\. Cagliero, F\. Tarasconi, G\. Giacalone, and C\. Bernini \(2023\)Benchmarking abstractive models for italian legal news summarization\.InLegal Knowledge and Information Systems: JURIX 2023: The Thirty\-sixth Annual Conference, Maastricht, the Netherlands, 18–20 December 2023,pp\. 101–106\.Note:Pages corrected from 311–316 to 101–106; DOI could not be confirmed with certaintyCited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px3.p1.1)\.
- I\. Benedetto, M\. La Quatra, and L\. Cagliero \(2025\)LegItBART: a summarization model for italian legal documents\.Artificial Intelligence and Law,pp\. 1–31\.External Links:[Document](https://dx.doi.org/10.1007/s10506-025-09436-y)Cited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px3.p1.1)\.
- R\. Bhambhoria, S\. Dahan, J\. Li, and X\. Zhu \(2026\)Evaluating ai for law: bridging the gap with open\-source solutions\.InCompliance for Artificial Intelligence Systems,Lecture Notes in Computer Science, Vol\.14377,pp\. 59–74\.Note:Preprint available as arXiv:2404\.12349 \(2024\)External Links:[Document](https://dx.doi.org/10.1007/978-3-032-12795-2%5F5)Cited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Blair\-Stanek and B\. Van Durme \(2025\)Llms provide unstable answers to legal questions\.InProceedings of the Twentieth International Conference on Artificial Intelligence and Law,pp\. 425–429\.External Links:[Document](https://dx.doi.org/10.1145/3769126.3769245)Cited by:[Appendix H](https://arxiv.org/html/2607.03325#A8.p1.1),[§7](https://arxiv.org/html/2607.03325#S7.SS0.SSS0.Px1.p1.1)\.
- S\. Brin and L\. Page \(1998\)The anatomy of a large\-scale hypertextual web search engine\.Computer networks and ISDN systems30\(1\-7\),pp\. 107–117\.External Links:[Document](https://dx.doi.org/10.1016/S0169-7552%2898%2900110-X)Cited by:[§3\.3](https://arxiv.org/html/2607.03325#S3.SS3.SSS0.Px3.p1.1)\.
- M\. Z\. Costa, D\. F\. Robson, T\. B\. P\. Vieira, J\. Bourguet, G\. Guizzardi, and J\. P\. A\. Almeida \(2024\)Automated semantic annotation pipeline for brazilian judicial decisions\.In37th Annual Conference on Legal Knowledge and Information Systems, JURIX 2024,pp\. 226–238\.External Links:[Document](https://dx.doi.org/10.3233/FAIA241248)Cited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px2.p1.1)\.
- C\. Coupette, J\. Beckedorf, D\. Hartung, M\. Bommarito, and D\. M\. Katz \(2021\)Measuring law over time: a network analytical framework with an application to statutes and regulations in the united states and germany\.Frontiers in Physics9,pp\. 658463\.External Links:[Document](https://dx.doi.org/10.3389/fphy.2021.658463)Cited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px4.p1.1)\.
- S\. Dahan, R\. Bhambhoria, D\. Liang, and X\. Zhu \(2023\)Openjustice\. ai: a global open\-source legal language model\.InLegal Knowledge and Information Systems: JURIX 2023: The Thirty\-sixth Annual Conference, Maastricht, the Netherlands, 18–20 December 2023,pp\. 387–390\.External Links:[Document](https://dx.doi.org/10.3233/FAIA230995)Cited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Dahl, V\. Magesh, M\. Suzgun, and D\. E\. Ho \(2024\)Large legal fictions: profiling legal hallucinations in large language models\.Journal of Legal Analysis16\(1\),pp\. 64–93\.External Links:[Document](https://dx.doi.org/10.1093/jla/laae003)Cited by:[§1](https://arxiv.org/html/2607.03325#S1.p5.1)\.
- T\. Dal Pont, F\. Galli, A\. Loreggia, G\. Pisano, R\. Rovatti, and G\. Sartor \(2023\)Legal summarisation through llms: the prodigit project\.arXiv preprint arXiv:2308\.04416\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2308.04416)Cited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px3.p1.1)\.
- A\. R\. Feinstein and D\. V\. Cicchetti \(1990\)High agreement but low kappa: I\. the problems of two paradoxes\.Journal of Clinical Epidemiology43\(6\),pp\. 543–549\.External Links:[Document](https://dx.doi.org/10.1016/0895-4356%2890%2990158-L)Cited by:[Appendix F](https://arxiv.org/html/2607.03325#A6.p2.10),[§5\.4](https://arxiv.org/html/2607.03325#S5.SS4.p2.8)\.
- E\. Francesconi, C\. Marchetti, R\. Pietramala, and P\. Spinosa \(2010\)URN\-based identification of legal acts: the case of the italian senate\.Informatica e diritto19\(1\-2\),pp\. 233–252\.Note:No DOI available; article predates DOI assignment for this journalCited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px4.p1.1)\.
- G\. Garzo and A\. Palumbo \(2025\)Does chatgpt understand the law? a case study on road homicide in italy\.InJURIX 2025\-38th International Conference on Legal Knowledge and Information Systems,pp\. 74–85\.External Links:[Document](https://dx.doi.org/10.3233/FAIA251578)Cited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px3.p1.1)\.
- M\. P\. Giovannini, M\. Palmirani, and E\. Francesconi \(2012\)Linee guida per la marcatura dei documenti normativi secondo gli standard normeinrete\.European Press Academic Publishing Firenze\.External Links:ISBN 978\-88\-8398\-076\-3Cited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px4.p1.1)\.
- G\. Grundler, P\. Santin, A\. Fidelangeli, R\. Mignone, F\. Galli, A\. Galassi, G\. Contissa, P\. Torroni,et al\.\(2025\)Automated extraction of judicial interpretative formulas in eu case law on vat\.InFrontiers in Artificial Intelligence and Applications,pp\. 294–299\.Note:JURIX 2025; DOI could not be confirmed at time of verificationCited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px2.p1.1)\.
- K\. L\. Gwet \(2014\)Handbook of inter\-rater reliability: the definitive guide to measuring the extent of agreement among raters\.4 edition,Advanced Analytics, LLC,Gaithersburg, MD\.External Links:ISBN 978\-0\-9708062\-8\-4Cited by:[Appendix F](https://arxiv.org/html/2607.03325#A6.p2.10),[§5\.4](https://arxiv.org/html/2607.03325#S5.SS4.p2.8)\.
- I\. Habernal, D\. Faber, N\. Recchia, S\. Bretthauer, I\. Gurevych, I\. Spiecker genannt Döhmann, and C\. Burchard \(2024\)Mining legal arguments in court decisions\.Artificial Intelligence and Law32\(3\),pp\. 1–38\.External Links:[Document](https://dx.doi.org/10.1007/s10506-023-09361-y)Cited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Han, P\. Burgess, and E\. Shareghi \(2026\)Legal citation prediction with llms: a comparative evaluation of instruction tuning, retrieval, and jurisdiction\-specific pre\-training on the auslaw citation benchmark\.Artificial Intelligence and Law,pp\. 1–35\.External Links:[Document](https://dx.doi.org/10.1007/s10506-026-09506-9)Cited by:[§1](https://arxiv.org/html/2607.03325#S1.p5.1)\.
- N\. Holzenberger and B\. Van Durme \(2023\)Connecting symbolic statutory reasoning with legal information extraction\.InProceedings of the Natural Legal Language Processing Workshop 2023,pp\. 113–131\.Note:This entry previously appeared twice in the bibliography \(duplicate key\); the duplicate has been removedExternal Links:[Document](https://dx.doi.org/10.18653/v1/2023.nllp-1.12)Cited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Janatian, H\. Westermann, J\. Tan, J\. Savelka, and K\. Benyekhlef \(2023\)From text to structure: using large language models to support the development of legal expert systems\.InLegal Knowledge and Information Systems: JURIX 2023: The Thirty\-sixth Annual Conference, Maastricht, the Netherlands, 18–20 December 2023,pp\. 167–176\.Note:DOI could not be confirmed at time of verificationCited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px2.p1.1)\.
- C\. Jiang and X\. Yang \(2023\)Legal syllogism prompting: teaching large language models for legal judgment prediction\.InProceedings of the nineteenth international conference on artificial intelligence and law,pp\. 417–421\.External Links:[Document](https://dx.doi.org/10.1145/3594536.3595170)Cited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px1.p1.1)\.
- X\. Kang, L\. Qu, L\. Soon, Z\. Li, and A\. Trakic \(2025\)Automating irac analysis in malaysian contract law using a semi\-structured knowledge base\.Artificial Intelligence and Law,pp\. 1–44\.External Links:[Document](https://dx.doi.org/10.1007/s10506-025-09467-5)Cited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px1.p1.1)\.
- D\. Licari, P\. Bushipaka, G\. Marino, G\. Comandé, and T\. Cucinotta \(2023\)Legal holding extraction from italian case documents using italian\-legal\-bert text summarization\.InProceedings of the Nineteenth International Conference on Artificial Intelligence and Law,pp\. 148–156\.External Links:[Document](https://dx.doi.org/10.1145/3594536.3595177)Cited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px3.p1.1)\.
- E\. T\. Liebman, V\. Colesanti, E\. Merlin, and E\. F\. Ricci \(2007\)Manuale di diritto processuale civile\. principi\.Giuffrè Editore\.Cited by:[§5\.1](https://arxiv.org/html/2607.03325#S5.SS1.p2.1)\.
- A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan,et al\.\(2024\)Deepseek\-v3 technical report\.arXiv preprint arXiv:2412\.19437\.Cited by:[§4\.3](https://arxiv.org/html/2607.03325#S4.SS3.p1.1)\.
- A\. Lombardi, D\. Alfano, and R\. Abbruzzese \(2023\)Legal text segmentation through breakpoint detection\.InLegal Knowledge and Information Systems: JURIX 2023: The Thirty\-sixth Annual Conference, Maastricht, the Netherlands, 18–20 December 2023,pp\. 227–236\.Note:DOI could not be confirmed at time of verificationCited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px2.p1.1)\.
- N\. MacCormick \(1994\)Legal reasoning and legal theory\.Clarendon Press\.External Links:ISBN 978\-0\-19\-876384\-0Cited by:[§1](https://arxiv.org/html/2607.03325#S1.p3.1)\.
- Ministero dell’Economia e delle Finanze – Dipartimento della Giustizia Tributaria \(2024\)Note:Accessed: 2026\-04\-29External Links:[Link](https://bancadatigiurisprudenza.giustiziatributaria.gov.it/)Cited by:[§4\.1](https://arxiv.org/html/2607.03325#S4.SS1.p1.1)\.
- J\. Mumford, F\. Florimonte, K\. Atkinson, and K\. Dzehtsiarou \(2025\)Context\-aware citation networks: a human–ai dataset, analysis, and tool\.Frontiers in Artificial Intelligence and Applications\.External Links:[Document](https://dx.doi.org/10.3233/FAIA251584)Cited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px4.p1.1)\.
- M\. Ovádek, A\. Dyevre, and K\. Wigard \(2021\)Analysing eu treaty\-making and litigation with network analysis and natural language processing\.Frontiers in Physics9,pp\. 657607\.External Links:[Document](https://dx.doi.org/10.3389/fphy.2021.657607)Cited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px4.p1.1)\.
- J\. Rabelo, R\. Goebel, M\. Kim, Y\. Kano, M\. Yoshioka, and K\. Satoh \(2022\)Overview and discussion of the competition on legal information extraction/entailment \(coliee\) 2021\.The Review of Socionetwork Strategies16\(1\),pp\. 111–133\.External Links:[Document](https://dx.doi.org/10.1007/s12626-022-00105-z)Cited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px1.p1.1)\.
- F\. Ricci \(1987\)Accertamento giudiziale\.Indigesto discipline privatistiche,Cited by:[§5\.1](https://arxiv.org/html/2607.03325#S5.SS1.p2.1)\.
- A\. Ruf and K\. Yin \(2024\)Aristotle is long dead but his wisdom rules us from the grave: a comparison of the application of logic in legal reasoning in common law and civil law systems\.InComparative Law: Unraveling Global Legal Systems,pp\. 1–23\.Cited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px1.p1.1)\.
- P\. Santin, G\. Grundler, A\. Galassi, F\. Galli, F\. Lagioia, E\. Palmieri, F\. Ruggeri, G\. Sartor, and P\. Torroni \(2023\)Argumentation structure prediction in cjeu decisions on fiscal state aid\.InProceedings of the Nineteenth International Conference on Artificial Intelligence and Law,pp\. 247–256\.External Links:[Document](https://dx.doi.org/10.1145/3594536.3595174)Cited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Villa \(2023\)La graduazione delle questioni di merito\. ammissibilità e profili dinamici\.Vol\.149,Giappichelli\.Cited by:[§5\.1](https://arxiv.org/html/2607.03325#S5.SS1.p2.1)\.
- H\. Westermann, V\. R\. Walker, and J\. Savelka \(2025\)Automated mapping of legal criteria to the texts of adjudicatory decisions using llms\.InProceedings of the Twentieth International Conference on Artificial Intelligence and Law,pp\. 219–228\.External Links:[Document](https://dx.doi.org/10.1145/3769126.3769236)Cited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px2.p1.1)\.
- N\. Wongpakaran, T\. Wongpakaran, D\. Wedding, and K\. L\. Gwet \(2013\)A comparison of Cohen’s kappa and Gwet’s AC1 when calculating inter\-rater reliability coefficients: a study conducted with personality disorder samples\.BMC Medical Research Methodology13\(1\),pp\. 61\.External Links:[Document](https://dx.doi.org/10.1186/1471-2288-13-61)Cited by:[Appendix F](https://arxiv.org/html/2607.03325#A6.p2.10),[§5\.4](https://arxiv.org/html/2607.03325#S5.SS4.p2.8)\.
- H\. Xu and K\. Ashley \(2023\)Question\-answering approach to evaluating legal summaries\.arXiv preprint arXiv:2309\.15016\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2309.15016)Cited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px3.p1.1)\.
- F\. Yu, L\. Quartey, and F\. Schilder \(2022\)Legal prompting: teaching a language model to think like a lawyer\.arXiv preprint arXiv:2212\.01326\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2212.01326)Cited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px1.p1.1)\.
- M\. M\. Zin, K\. Satoh, and G\. Borges \(2025\)From court decisions to guiding principles: advancing complex legal summarization with llms\.InLegal Knowledge and Information Systems,pp\. 371–376\.Note:Verified as a real JURIX 2025 paper; DOI could not be confirmedCited by:[§2](https://arxiv.org/html/2607.03325#S2.SS0.SSS0.Px3.p1.1)\.

## Appendix ATwo\-step prompt

In this appendix we report the two\-step prompts used for the extraction\. As explained in section[4\.3](https://arxiv.org/html/2607.03325#S4.SS3), prompt[1](https://arxiv.org/html/2607.03325#LST1)was used for shorter judgments while a two\-step approach was used for longer ones\.

Listing 3:First part of the two\-step prompt \(translated in English\)\. The goal of this prompt is to extract and summarize all issues within the judgmentInsidethecurlybracesthereisanItaliantaxcourtdecision\.Yourtaskistoidentifyandextractthemainlegalissues\(generallyoneortwo\)thatthejudgeaddressesandresolvesinordertodecidethedispute\.Consideronlytheissuesonwhichthejudgerulesexplicitlyandwithreasoning\.

Foreachissue,performthefollowing:

\-Formulatethelegalissueclearly,asaself\-containedandautonomousquestion\.Theformulationmustfollowthestructure"Whether\+clause"andmustbeunderstandableonitsown,withoutreferringtothefulldecisionorotherissues\.Beconcise\.

\-Indicatetheoutcomeoftheissue,i\.e\.,theanswertothelegalquestion\.Includeonlytheresolutionoftheissuewithoutmentioningotherrulingsofthejudge\.

\-Writeanabstractivesummaryofapproximately100words,including:

\-\-thefactsrelevanttotheissue

\-\-themainlegislativeandcaselawreferencesrelevanttotheissue

\-\-asynthesisofthejudge’sreasoningindecidingtheissue

\-\-theconclusionreachedbythejudge

Thesummarymustbeself\-containedandunderstandablewithoutreferencetothefulldecision\.

Alwaysusepreciselegallanguage\.Includeonlytheinformationrelevanttoeachissue\.

XMLoutputformat:

<issues\>

<issuetitle="issuetitle"\>

<text\>textofthelegalissue\(~30words\)</text\>

<issue\_outcomein\_favor\_of="taxauthority\|taxpayer\|other"\>testoesito</issue\_outcome\>

<summary\>summary\(~100words\)oftheissue</summary\>

</issue\>

</issues\>

OutputONLYtheXMLblockandnothingelse\.

Listing 4:Second part of the two\-step prompt \(translated in English\)\. The text of the judgment and the <text\> field of one of the issues extracted in the previous step are passed to the LLM along with the prompt\.InsidecurlybracketsisanItaliantaxjudgment\.Insidesquarebrackets,alegalissuealreadyidentifiedinthesamejudgmentisalsoprovided\.Yourtaskistoextractthejudge’sreasoningfromthedecisionwithreferencetothatissue\.Structurethereasoningasfollows:

<factual\_premises\>:listthefacts,actions,events,ordataofthecaseconsideredrelevanttotheissue\.Eachfactmustbedescribedwithsufficientprecisiontoallowanotherjudgetodecideasimilarissueconsistently\.

<legal\_references\>:listexhaustivelyALLANDONLYthelegalreferences\(statutoryprovisions,caselaw,generalprinciples,administrativepractice\)usedbythejudgetoresolvetheissue,astheyappearinthejudgment\.Orderthembydecreasingimportance\.

<citation\_reason\>:foronlythoselegalreferencesdiscussedingreaterdetail,explainwhythejudgecitesthem\.Consideronlyreferencesthatarediscussedindepthinthejudgment\.

<judge\_reasoning\>:describeinfreetexthowthejudgeappliedthelegalreferencestothespecificcase,leadingtothedecision\.Donotincludeinformationalreadystatedinthefactualpremises\.Useapproximatelyfewerthan120words\.

Alwaysusepreciselegallanguage\.

XMLoutputformat:

<factual\_premises\>

<item\>factualpremise1</item\>

<item\>factualpremise2</item\>

</factual\_premises\>

<legal\_references\>

<itemid="D1"type="caselaw"ref="SupremeCourtno\.nnnn/yyyy"/\>

<itemid="D2"type="leg"ref="Art\.xx,para\.z,LegislativeDecreeno\.nnn/yyyy"/\>

<itemid="D3"type="caselaw"ref="SupremeCourtno\.nnnn/yyyy,SupremeCourtno\.mmmm/yyyy"/\>

</legal\_references\>

<citation\_reason\>

<itemref\_id="D1"\>reasonforcitinglegalreference</item\>

<itemref\_id="D2"\>reasonforcitinglegalreference</item\>

</citation\_reason\>

<judge\_reasoning\>reasoningtext\(fewerthan150words\)</judge\_reasoning\>

Inthereffieldoflegalreferences,alwaysindicatetheexactcitationdetailsofthesource\.Usethetypefieldtoclassifythereference:

type="caselaw"forcaselaw\.Ifmultipledecisionsarecitedconsecutively,grouptheminasingleitem;

type="leg"forlegislation;

type="princ"forgeneralprinciplesoflaw\(inthiscase,refcontainsonlythenameoftheprinciple\);

type="admin\_pract"foradministrativepractice\.

Includeonlyinformationrelevanttothegivenissue\.

GenerateonlytheXMLblockandnothingelse\.

## Appendix BIssue text correction

Listing 5:Few\-shot prompt used to correct the <text\> field\.YouareanexpertintheItalianlanguageandtaxlaw\.YourtaskistocorrectlegalissuequestionsextractedfromItaliantaxcourtdecisions\.

CORRECTIONRULES:

1\.Correctstructure:Thequestionsmustbeindirectinterrogativesstartingwith"Whether"followedbyasubjunctiveverb,expressingthelegalissue\.Theinitial"Whether"mustbefollowedbythesubjectofthesentence\.Theymustnotbeconditionalclausesintroducedby"if/whether"inahypotheticalsense\.

2\.Indirectinterrogativeform:"Whether"introducesthemainclause\(e\.g\."Whetherthetaxpayerisrequired\.\.\.","Whetherthereexists\.\.\."\),notasubordinateconditionalclause\.Donotuseafinalquestionmark\.

3\.Agreementandsyntax:Correctagreementerrorsandaddmissingrelativepronouns\.

EXAMPLESOFCORRECTION:

INCORRECT:

Whetherataxpayerproduceswastedisposedofindependentlythroughaprivatecompany,withouttheMunicipalityhavingprovidedanycollectionservice,isexemptfromthepaymentofthewastetax\.

CORRECT:

Whetherataxpayerwhoproduceswastedisposedofindependentlythroughaprivatecompany,withouttheMunicipalityhavingprovidedanycollectionservice,isexemptfromthepaymentofthewastetax\.

INCORRECT:

Whetherabusinessdisplaysasignidentifyingthepremisesoftheactivity,thereisexemptionfromthemunicipaladvertisingtax\.

CORRECT:

Whetherthereisanexemptionfromthemunicipaladvertisingtaxforabusinessthatdisplaysasignidentifyingthepremisesoftheactivity\.

INCORRECT:

TheTaxAuthority,nothavingappearedinthefirst\-instanceproceedings,hasthenecessarystandingtoappealthefirst\-instancejudgment?

CORRECT:

WhethertheTaxAuthority,nothavingappearedinthefirst\-instanceproceedings,hasthenecessarystandingtoappealthefirst\-instancejudgment\.

INCORRECT:

Whetheranassetissubjecttojudicialseizure,whetherthetaxpayerisrequiredtopayIMUdespitetheseizureandlackofavailabilityoftheasset\.

CORRECT:

Whether,inthecasewhereanassetissubjecttojudicialseizure,thetaxpayerisrequiredtopayIMUdespitetheseizureandlackofavailabilityoftheasset\.

INSTRUCTIONS:

IMPORTANT:Onlyabout15%ofquestionscontainerrors\.First,assesswhetherthequestionactuallyneedscorrection\.Ifitdoesnot,respondonly:QUESTIONALREADYCORRECT

CorrectthequestionONLYifitpresentsoneormoreofthefollowingissues:

\-Directinterrogativeform\("?"\)insteadofindirectform

\-Agreementorsyntacticerrors

\-Confusedstructure\(e\.g\.double"Whether",conditionalperiodsinsteadofindirectinterrogatives\)

\-Missingnecessaryrelativepronouns

OutputONLYthecorrectedquestion\.

Ifthequestionisalreadycorrect,write:QUESTIONALREADYCORRECT

QUESTIONTOEVALUATE:

## Appendix CPrompts in Italian

In this section we report all prompts in their original form, that is, in Italian\. All of the documents involved in our analysis are in fact in Italian, including judgments, prompts, and XML outputs\.

Listing 6:Single step promptAll’internodelleparentesigraffeèriportataunasentenzatributariaitaliana\.

Iltuocompitoèidentificareedestrarreleprincipaliquestionigiuridiche\(unaomassimodue\)cheilgiudiceaffrontaerisolveperdeciderelacontroversia\.Consideraesclusivamentelequestionisucuiilgiudicesipronunciainmodoesplicitoemotivato\.

Perciascunaquestione,svolgileseguentioperazioni:

\-Formulalaquestionegiuridicainmodochiaro,autosufficienteeautonomo\.Laformulazionedeveesserenellaforma"Se\+congiuntivo"edeveesserecomprensibiledasola,senzadoverconsultarel’interasentenzaoaltrequestioni\.

\-Indical’esitodellaquestione,cioèlarispostaalquesitogiuridico\.

\-Scomponiilragionamentodelgiudicenelleseguentiparti:

\-\-<premesse\_fatto\>:elencalecircostanze,azioni,eventiodatidelcasoritenutirilevantiperlaquestione\.

\-\-<lista\_riferimenti\_diritto\>:elencainmodocompletoTUTTIiriferimentigiuridici\(norme,giurisprudenza,principigenerali,prassiamministrativa\)rilevantiperlaquestione,cosìcomeappaiononellasentenza\.Ordinaliperimportanzadecrescente\.

\-\-<motivo\_citazione\>:perisoliriferimentigiuridicitrattatipiùindettagliospiegaperchéilgiudicelicita\.Considerasoloiriferimentidiscussiinmodoargomentatoeapprofonditonellasentenza\.

\-\-<ragionamento\_giudice\>:descriviintestoliberocomeilgiudicehaapplicatoiriferimentigiuridicialcasoconcreto,finoaarrivarealladecisione\.Nonincludereleinformazionigiàpresentinellepremessedifatto\.

\-Redigiunriassuntoastrattivodellaquestione,tra100e150parole,cheincluda:fatto,diritto,ragionamentoeconclusione\.Ilriassuntodeveessereautosufficienteecompleto,comprensibilesenzafareriferimentoallasentenzacompletaoadaltrequestioni\.

Requisitigenerali:

\-Ogniquestionedeveessereautonoma,autosufficienteeformulatainlinguaggiogiuridicoprecisoedettagliato\.

\-Ilfattodeveesseredescrittoconsufficienteprecisionedapermettereaunaltrogiudicedidecidereunaquestioneanalogainmodocoerente\.

Formatodioutputrichiesto:esclusivamenteinXML,comenelloschemaseguente:

<questioni\>

<questionetitle="titoloquestione"\>

<text\>testodellaquestionegiuridica</text\>

<esito\_questionefavorevole\_a="ufficio\|contribuente\|other"\>testoesito</esito\_questione\>

<premesse\_fatto\>

<item\>premessadifatto1</item\>

<item\>premessadifatto2</item\>

</premesse\_fatto\>

<lista\_riferimenti\_diritto\>

<itemid="D1"type="jur"ref="Cassazionen\.nnnn/yyyy"/\>

<itemid="D2"type="norm"ref="art\.xxcommazd\.lgsn\.nnn/yyyy"/\>

</lista\_riferimenti\_diritto\>

<motivo\_citazione\>

<itemid\_ref="D1"\>motivocitazionediritto</item\>

<itemid\_ref="D2"\>motivocitazionediritto</item\>

</motivo\_citazione\>

<ragionamento\_giudice\>testoragionamento\(~100parole\)</ragionamento\_giudice\>

<riassunto\>testodelriassunto\(~100parole\)</riassunto\>

</questione\>

</questioni\>

Nelcamporefdeiriferimentigiuridiciindicasempregliestremiesattidell’atto\.Usailcampotypeperclassificareilriferimento:

\-type="jur"pergiurisprudenza\.Sepiùsentenzevengonocitateconsecutivamente,raggruppaleinununicoitem;

\-type="norm"pernormedilegge;

\-type="princ"perprincipigeneralideldiritto\(inquestocaso,refcontieneesclusivamenteilnomedelprincipio\);

\-type="prassi\_amm"perlaprassiamministrativa\.

GenerasoloilbloccoXMLenullaoltre\.

Listing 7:First prompt of the two\-step approach\. This prompt extracts issues together with their respective outcomes and summariesAll’internodelleparentesigraffeèriportataunasentenzatributariaitaliana\.Iltuocompitoèidentificareedestrarreleprincipaliquestionigiuridiche\(generalmenteunaodue\)cheilgiudiceaffrontaerisolveperdeciderelacontroversia\.Consideraesclusivamentelequestionisucuiilgiudicesipronunciainmodoesplicitoemotivato\.

Perciascunaquestione,svolgileseguentioperazioni:

\-Formulalaquestionegiuridicainmodochiaro,autosufficienteeautonomo\.Laformulazionedeveesserenellaforma"Se\+congiuntivo"edeveesserecomprensibiledasola,senzadoverconsultarel’interasentenzaoaltrequestioni\.Siisintetico\.

\-Indical’esitodellaquestione,cioèlarispostaalquesitogiuridico\.Includisololasoluzionedellaquestionesenzaindicarealtredisposizionidelgiudice\.

\-Scriviunriassuntoastrattivodicirca100paroledellaquestione,cheincluda:

\-\-ifattirilevantiperlaquestione

\-\-iriferimentinormativiegiurisprudenzialiprincipalirilevantiperlaquestione

\-\-lasintesidelragionamentodelgiudiceperdeciderelaquestione

\-\-laconclusionedellaquestioneraggiuntadalgiudice

Ilriassuntodeveessereautonomoecomprensibilesenzariferimentoallasentenzacompleta\.

Usasempreunlinguaggiogiuridicopreciso\.Includisololeinformazionirilevantiperciascunaquestione\.

FormatodioutputXML:

<questioni\>

<questionetitle="titoloquestione"\>

<text\>testodellaquestionegiuridica\(~30parole\)</text\>

<esito\_questionefavorevole\_a="ufficio\|contribuente\|other"\>testoesito</esito\_questione\>

<riassunto\>riassunto\(~100parole\)dellaquestione</riassunto\>

</questione\>

</questioni\>

GenerasoloilbloccoXMLenullaoltre\.

Listing 8:Second prompt of the two\-step approach\. This prompt extracts the reasoning \(facts, legal references, citation reasons, judge reasoning\)\.All’internodelleparentesigraffeèriportataunasentenzatributariaitaliana\.Traparentesiquadreèinoltrefornitaunaquestionegiuridicagiàidentificataall’internodellastessasentenza\.Iltuocompitoèestrarreilragionamentodelgiudicedallasentenzainriferimentoallaquestione\.Suddividiilragionamentocomesegue:

\-<premesse\_fatto\>:elencalecircostanze,azioni,eventiodatidelcasoritenutirilevantiperlaquestione\.Ilfattodeveesseredescrittoconsufficienteprecisionedapermettereaunaltrogiudicedidecidereunaquestioneanalogainmodocoerente\.

\-<lista\_riferimenti\_diritto\>:elencainmodocompletoTUTTIESOLIiriferimentigiuridici\(norme,giurisprudenza,principigenerali,prassiamministrativa\)usatidalgiudiceperrisolverelaquestionedata,cosìcomeappaiononellasentenza\.Ordinaliperimportanzadecrescente\.

\-<motivo\_citazione\>:perisoliriferimentigiuridicitrattatipiùindettagliospiegaperchéilgiudicelicita\.Considerasoloiriferimentidiscussiinmodoargomentatoeapprofonditonellasentenza\.

\-<ragionamento\_giudice\>:descriviintestoliberocomeilgiudicehaapplicatoiriferimentigiuridicialcasoconcreto,finoaarrivarealladecisione\.Nonincludereleinformazionigiàpresentinellepremessedifatto\.Usamenodi120parolecirca\.

Usasempreunlinguaggiogiuridicopreciso\.

FormatodioutputXML:

<premesse\_fatto\>

<item\>premessadifatto1</item\>

<item\>premessadifatto2</item\>

</premesse\_fatto\>

<lista\_riferimenti\_diritto\>

<itemid="D1"type="jur"ref="Cassazionen\.nnnn/yyyy"/\>

<itemid="D2"type="norm"ref="art\.xxcommazdlgsn\.nnn/yyyy"/\>

<itemid="D3"type="jur"ref="Cassazionen\.nnnn/yyyy,Cassazionen\.mmmm/yyyy"/\>

</lista\_riferimenti\_diritto\>

<motivo\_citazione\>

<itemid\_ref="D1"\>motivocitazionediritto</item\>

<itemid\_ref="D2"\>motivocitazionediritto</item\>

</motivo\_citazione\>

<ragionamento\_giudice\>testoragionamento\(menodi150parole\)</ragionamento\_giudice\>

Nelcamporefdeiriferimentigiuridiciindicasempregliestremiesattidell’atto\.Usailcampotypeperclassificareilriferimento:

\-type="jur"pergiurisprudenza\.Sepiùsentenzevengonocitateconsecutivamente,raggruppaleinununicoitem;

\-type="norm"pernormedilegge;

\-type="princ"perprincipigeneralideldiritto\(inquestocaso,refcontieneesclusivamenteilnomedelprincipio\);

\-type="prassi\_amm"perlaprassiamministrativa\.

Includisololeinformazionirilevantiperlaquestionedata\.

GenerasoloilbloccoXMLenullaoltre\.

Listing 9:Few\-shot prompt used to correct the <text\> field\.Seiunespertodilinguaitalianaedirittotributario\.Iltuocompitoècorreggerequesitidiquestionigiuridicheestrattedasentenzetributarieitaliane\.

REGOLEDICORREZIONE:

1\.Strutturacorretta:Iquesitidevonoessereinterrogativeindirettecheinizianocon"Se"\+congiuntivoperporrelaquestionegiuridica\.Il"Se"inizialedeveessereseguitodalsoggettodellafrase\.Nondevonoessereperiodiipoteticicon"Se"cheintroduceunacondizione\.

2\.Formainterrogativaindiretta:"Se"introducelaproposizioneprincipale\(es\."Seilcontribuentesiatenuto\.\.\.","Sesussista\.\.\."\),nonunasubordinataipotetica\.Nonusareilpuntointerrogativofinale\.

4\.Concordanzaesintassi:Correggierroridiconcordanza,aggiungipronomirelativimancanti\.

ESEMPIDICORREZIONE:

SCORRETTO:

Seuncontribuenteproducarifiutismaltitiautonomamentetramitedittaprivata,senzacheilComuneabbiasvoltoalcunserviziodiraccolta,siaesentedalpagamentodellaTARI\.

CORRETTO:

Seuncontribuentecheproducerifiutismaltitiautonomamentetramitedittaprivata,senzacheilComuneabbiasvoltoalcunserviziodiraccolta,siaesentedalpagamentodellaTARI\.

SCORRETTO:

Seun’impresaesponeun’insegnadieserciziochecontraddistinguelasededell’attività,sussistal’esenzionedall’impostacomunalesullapubblicità\.

CORRETTO:

Sesussistal’esenzionedall’impostacomunalesullapubblicitàperun’impresacheesponeun’insegnadieserciziochecontraddistinguelasededell’attività\.

SCORRETTO:

L’AgenziadelleEntrate,nonessendosicostituitanelgiudiziodiprimogrado,halalegittimazionenecessariaperproporreappelloavversolasentenzadiprimogrado?

CORRETTO:

Sel’AgenziadelleEntrate,nonessendosicostituitanelgiudiziodiprimogrado,abbialalegittimazionenecessariaperproporreappelloavversolasentenzadiprimogrado\.

SCORRETTO:

Seunimmobilesiasoggettoasequestrogiudiziario,seilcontribuentesiatenutoalpagamentodell’IMUnonostanteilsequestroelamancatadisponibilitàdelbene\.

CORRETTO:

Se,nelcasoincuiunimmobilesiasoggettoasequestrogiudiziario,ilcontribuentesiatenutoalpagamentodell’IMUnonostanteilsequestroelamancatadisponibilitàdelbene\.

ISTRUZIONI:

IMPORTANTE:Soloil15%circadeiquesitipresentaerrori\.Primaditutto,valutaseilquesitonecessitadavverodicorrezioni\.Incasocontrarioscriviunicamente:QUESITOGIÀCORRETTO

CorreggiilquesitoSOLOsepresentaunoopiùdiquestiproblemi:

\-Formainterrogativadiretta\(con"?"\)invececheindiretta

\-Erroridiconcordanzaosintassi

\-Strutturaconfusa\(es\.doppi"Se",periodiipoteticiinvecediinterrogativeindirette\)

\-Mancanzadipronomirelativinecessari

InoutputproduciESCLUSIVAMENTEilquesitocorretto

Seilquesitoègiàcorretto,scrivi:QUESITOGIÀCORRETTO

QUESITODAVALUTARE:

## Appendix DExtraction results averaged per judgment and per issue

The main text reports precision, recall, and F1 computed by*pooling*all items \(issues or citations\) across the relevant judgments before computing the metric\. While this is a natural statistic, as it gives information about how accurate a single issue or citation is, it can be skewed by the judgments that contain many issues, or the issues with many associated references\. Here we report, alongside the pooled figures, the corresponding scores obtained by computing the metric separately on each unit \(each judgment for issues, each present issue for citations\) and then averaging, so that every unit contributes equally regardless of how many items it contains\. Each table places the two views side by side\.

### D\.1Issue extraction

Averaging per judgment raises every issue score relative to pooling \(Tables[11](https://arxiv.org/html/2607.03325#A4.T11)and[12](https://arxiv.org/html/2607.03325#A4.T12)\)\. First, we remark that the inter annotator agreement improves\. The LLM’s precision and recall also increase significantly \(recall goes from69\.9%69\.9\\%to86\.3%86\.3\\%and from82\.3%82\.3\\%to92\.4%92\.4\\%\) when averaged per\-judgment\. This confirms that the model performs well on typical judgments and is penalized mainly on a few harder ones\.

Table 11:Pairwise agreement on issue extraction \(N=20N\{=\}20shared judgments\): pooled vs\. per\-judgment averaged\. Each row treats the second system as ground truth\.Table 12:LLM issue extraction against each annotator’s full set \(N=35N\{=\}35\): pooled vs\. per\-judgment averaged\.
### D\.2Citation extraction

For citations the averaging unit is the present issue, and the effects are similar \(Tables[13](https://arxiv.org/html/2607.03325#A4.T13)and[14](https://arxiv.org/html/2607.03325#A4.T14)\)\. Once again, the precision and recall show some improvement in the per\-issue case\.

Table 13:Pairwise agreement on citation extraction \(N=20N\{=\}20,2828present issues\): pooled vs\. per\-issue averaged\.Table 14:LLM citation extraction against each annotator’s full set \(N=35N\{=\}35\): pooled vs\. per\-issue averaged\.
### D\.3Union and intersection ground truth \(N=50N\{=\}50\)

The main text reports LLM performance separately against each annotator\. An alternative is to combine the two annotators into a single ground truth over allN=50N\{=\}50judgments\. For the3030judgments evaluated by a single annotator, the ground truth is that annotator’s set\. For the2020shared judgments, we consider two strategies:

- •Union: an item \(issue or citation\) is in the ground truth if*either*annotator identified it\. This produces a larger reference set and thus a more demanding recall target\.
- •Intersection: an item is in the ground truth only if*both*annotators identified it\. This produces a stricter reference set that may increase measured recall but can also affect precision, since an LLM\-extracted item confirmed by only one annotator counts as a false positive\.

#### Issue extraction\.

Table[15](https://arxiv.org/html/2607.03325#A4.T15)reports the results under both definitions\. Precision is identical \(94\.9%94\.9\\%\) because both annotators agree on which LLM issues are present in every shared judgment\. The difference appears in recall: the union ground truth includes more annotator issues \(lowering recall to75\.5%75\.5\\%\), while the intersection ground truth is smaller \(recall rises to84\.1%84\.1\\%\)\.

Table 15:LLM issue extraction with union/intersection ground truth \(N=50N\{=\}50\)\.
#### Citation extraction\.

Table[16](https://arxiv.org/html/2607.03325#A4.T16)reports the analogous results for citations\. Here the precision values differ slightly between union and intersection, because the number of LLM citations matching the ground truth changes depending on the definition\.555For33of2828shared issues, the exact count of LLM citations falling in the intersection of both annotators’ lists \(\|LLM∩A1∩A2\|\|\\text\{LLM\}\\cap A\_\{1\}\\cap A\_\{2\}\|\) cannot be determined from the available data; we use the midpoint of tight inclusion\-exclusion bounds, with a maximum error of±1\{\\pm\}1citation per affected issue\.

Table 16:LLM citation extraction with union/intersection ground truth \(N=50N\{=\}50,7474present issues\)\.

## Appendix EAnnotators scoring criteria

In this appendix we present the guidelines followed by the annotators\. As a general rule, all fields extracted by the LLM must be validated\. For each field, both quantitative and qualitative metrics are defined\. Quantitative metrics are numerically measurable, although they may still involve a certain degree of expert judgment \(i\.e\. number of extracted legal issues, number of extracted relevant citations\)\. Qualitative metrics rely on the expert’s assessment, typically expressed as a numerical score corresponding to predefined, albeit arbitrary, thresholds\.

Independent issue identificationThe LLM\-generated output to be evaluated is the one provided in the section`<text\>`A legal issue is described as a legal question explicitly decided by the judge in the reasoning section of the judgment, requiring the interpretation or application of a legal source, and supported by argumentative content sufficient to fill the structured fields of the XML format\. The annotator shall manually extract from the argumentative part of the judgement all the relevant legal issues according to the previous definition\.

Issue alignment and scoring

The annotator should mark the presence of each LLM\-extracted issue as:

- •TRUE if it matches an issue in the annotator’s list;
- •FALSE if it does not correspond to any issue the annotator considered present in the judgment\.

The annotator shall indicate the number of legal issues present in its list but not extracted by the LLM \(the minimum number of legal issues which, in addition to those already extracted, would be required to cover the entire content of the judgment\)\. Any further step of the assessment shall concern only the legal issues present in the judgement\. For each LLM\-extracted legal issue, the annotator shall assess:

- •generality: whether the automatically extracted legal issue is neither overly specific nor excessively general; the optimal level is the one that a legal expert would expect to find among the results of a search and that most closely approximates the correspondent manually extracted legal issue;
- •legal language: whether legal terminology and the wording are appropriate;

The assessment is carried out by assigning a score ranging from 1 \(completely inappropriate\) to 5 \(optimal level\) for both generality and legal language of each extracted issue\.

#### Citation

Independent citation identificationThe LLM\-generated output to be evaluated is the one provided in the section`<legal\_references\>`\. Given the text of the legal issue extracted by the LLM, the annotator shall compile a list of the legal references contained in the portion of the judgment correspondent to the legal issue under evaluation\. The annotator shall select only the references that are used by the judge to solve the question\. That means that not all the legal references mentioned in the judgment shall be considered, but only those relevant\. Citations shall be listed in the order in which they appear in the judgment\. That list is meant to be compiled without looking at`<legal\_references\>`section, which is the list of relevant citations extracted by the LLM\.

Citation reason evaluationThe LLM\-generated output to be evaluated is the one provided in the section`<citation\_reason\>`Since not all the references quoted are relevant, the annotator shall also evaluate the motivation given by the LLM for their extraction\. Normally, not all the extracted references are motivated, the annotator shall assess the motivations actually present\. No assessment is required of the criteria the LLM used to select the reference to be explained\. Citation reason is assessed by the sum of citation rationale scores: for each rationale, the annotator assigns a score of 1 if the rationale is both contained in the judgment and faithfully represents its content, and 0 otherwise \(e\.g\. if the LLM presented the rationale for three out of five citation extracted, the annotator shall assign a maximum score of 3 and a minimum of 0\)

#### Summary

The LLM\-generated output to be evaluated is the one provided in the section`<summary\>`Assessment of the summary is carried out by assessing a score ranging from 1 \(completely inappropriate\) to 5 \(optimal level\) to each of the following aspects:

- •Correctness as the absence of elements that are not present in the source document;
- •Form as coherence, readability, syntactic and grammatical correctness, and adherence to legal terminology;
- •Completeness as the inclusion of all relevant elements of the decision \(key points, legal issues, and essential factual information\) and the comprehensive representation of the source’s content;
- •Satisfaction as the degree of satisfaction with the global quality of the summary\.

#### Fact

The LLM\-generated output to be evaluated is the one provided in the section`<factual\_premises\>`\. The annotator shall provide an overall assessment of the quality of the fact as summarised by the LLM\. The overall assessment is carried out by assessing a comprehensive score ranging from 1 \(completely inappropriate\) to 5 \(optimal level\)\. In assessing the overall satisfaction the annotator shall take into account:

- •the absence of factual elements that are not supported by the source document;
- •the coherence, readability, syntactic and grammatical accuracy, and appropriate use of legal terminology;
- •the inclusion of all elements of the decision that are relevant to the extracted content\.

#### Reasoning

The LLM\-generated output to be evaluated is the one provided in the section`<judge\_reasoning\>`The annotator shall provide an overall assessment of the quality of the reasoning of the judge as summarised by the LLM\. The general assessment is carried out by assessing a comprehensive satisfaction score ranging from 1 \(completely inappropriate\) to 5 \(optimal level\)\. In assessing the overall satisfaction the annotator shall take into account:

- •absence of elements that are not present in the source document \(e\.g\. the statement made by the judge in the decision is actually motivated based on those reasons\);
- •correct use of logical connections \(e\.g\. the reasoning steps extracted by the LLM actually bring to the conclusion\);
- •coherence, readability, syntactic and grammatical correctness, and adherence to legal terminology;
- •inclusion of all relevant elements of the decision and the comprehensive representation of the source’s content\.

#### Outcome

The LLM\-generated output to be evaluated is the one provided in the section`<issue\_outcome\>`\. The annotator shall provide an overall assessment of the outcome of the procedure as summarised by the LLM\. The general assessment is carried out by assigning a satisfaction score ranging from 1 \(completely inappropriate\) to 5 \(optimal level\)\. It refers to the degree of satisfaction with the global quality of the outcome formulation\.

## Appendix FCohen’sκ\\kappafor the Likert dimensions

For completeness, Table[17](https://arxiv.org/html/2607.03325#A6.T17)reports Cohen’sκ\\kappawith linear and quadratic weights and the Pearson correlationrrfor the same dimensions as Table[10](https://arxiv.org/html/2607.03325#S6.T10)\. These chance\-corrected statistics paint a substantially different picture from the saturation\-robust statistics reported in the main text\. On the two dimensions with appreciable rating variance, issue generality \(κℓ=0\.40\\kappa\_\{\\ell\}\{=\}0\.40,κq=0\.48\\kappa\_\{q\}\{=\}0\.48\) and legal language \(κℓ=0\.64\\kappa\_\{\\ell\}\{=\}0\.64,κq=0\.74\\kappa\_\{q\}\{=\}0\.74\),κ\\kappabehaves as expected and indicates moderate\-to\-strong agreement\. On the remaining dimensions, however,κ\\kappaeither drops close to zero or turns negative; in the case of summary form it reachesκℓ=κq=−0\.11\\kappa\_\{\\ell\}\{=\}\\kappa\_\{q\}\{=\}\-0\.11despite an exact\-agreement rate of78\.6%78\.6\\%and within\-one\-point agreement of100%100\\%\.

This pattern is a textbook instance of the so\-called*kappa paradox*\[Feinstein and Cicchetti,[1990](https://arxiv.org/html/2607.03325#bib.bib54), Wongpakaranet al\.,[2013](https://arxiv.org/html/2607.03325#bib.bib55)\]\. When one rating category accounts for the great majority of observations, the marginal\-based estimate of chance agreementpep\_\{e\}that entersκ=\(po−pe\)/\(1−pe\)\\kappa\{=\}\(p\_\{o\}\-p\_\{e\}\)/\(1\-p\_\{e\}\)approaches the observed agreementpop\_\{o\}, and small fluctuations in either term swingκ\\kappaover a wide range, including into negative values\. In our data, both annotators rate almost every issue 4 or 5 on the four summary dimensions and on the reasoning, outcome, and fact satisfaction scores, soκ\\kappain this regime measures the rarity of disagreement rather than its substance\. The same observation applies, less dramatically, to the Pearson statistics\. Gwet’sAC2\\mathrm\{AC2\}\[Gwet,[2014](https://arxiv.org/html/2607.03325#bib.bib53)\]is designed precisely to be robust to this regime, and the values reported in Table[10](https://arxiv.org/html/2607.03325#S6.T10)of the main text give a more faithful picture of the level of agreement reached by the two annotators\. For the citation\-reason metric reported in Table[7](https://arxiv.org/html/2607.03325#S6.T7), whose per\-issue counts are well spread across the integer range\{0,1,2,…\}\\\{0,1,2,\\ldots\\\}rather than saturated, the kappa paradox does not arise:κℓ=0\.91\\kappa\_\{\\ell\}\{=\}0\.91,κq=0\.85\\kappa\_\{q\}\{=\}0\.85, andr=0\.86r\{=\}0\.86, all consistent with the high agreement reported by the count\-based statistics\.

Table 17:Chance\-corrected agreement statistics for the Likert dimensions \(N=20N\{=\}20,2828present issues\)\.κℓ\\kappa\_\{\\ell\},κq\\kappa\_\{q\}: linearly and quadratically weighted Cohen’sκ\\kappa: two\-way random\-effects, absolute\-agreement, single\-measures intraclass correlation;rr: Pearson correlation\. In the case of outcome satisfaction, both annotators assigned55to every issue, making variance\-based statistics undefined\.
## Appendix GWorked extraction example

This appendix reproduces in full the XML extraction summarised in Listing[2](https://arxiv.org/html/2607.03325#LST2)in the main text, from judgment n\. 318/2024 of the Corte di Giustizia Tributaria di primo grado di Teramo\.

Listing 10:Full extraction for the worked example\.<issues\>

<issuetitle="VATdeductibilityforanagriculturalbusinessnotyetproductive"id="Q1"\>

<text\>Whetherthepurchaseofacapitalasset\(agriculturaltractor\)byanewlyestablishedagriculturalbusiness,notyetproductive,isconnectedtotheentrepreneurialactivityandthereforedeductiblepursuanttoArticles4and5ofPresidentialDecree633/72\.</text\>

<issue\_outcomein\_favor\_of="taxpayer"\>VATisdeductiblebecausethepurchaseisrelatedtotheentrepreneurialactivity,evenintheabsenceofimmediatetaxableoperations\.</issue\_outcome\>

<factual\_premises\>

<item\>Theappellantbeganthesnail\-farmingactivityin2020,anactivitythatrequirestimetobecomeproductive\.</item\>

<item\>In2022,theappellantpurchasedanagriculturaltractorforlandpreparationandforprocuringplant\-basedfeednecessaryforthefarmingactivity\.</item\>

<item\>TheRevenueAgencydeniedtheVATrefund,arguingthattheactivityhadnotyetstartedandthatthetractorwasnotincludedinthedepreciableassetslist\.</item\>

<item\>Theappellantprovideddocumentation\(companyregistryextract,invoicesforthetractorandforsnailpurchase\)todemonstratetherelevanceofthepurchasetotheactivity\.</item\>

</factual\_premises\>

<legal\_references\>

<itemid="D1"type="caselaw"ref="Cassazionen\.26689/2022;n\.7440/2021;n\.15570/2023"/\>

<itemid="D2"type="caselaw"ref="CJEUcaseC\-110/94;caseC\-37/95;caseC\-400/98;caseC\-263/15"/\>

<itemid="D3"type="caselaw"ref="CJEUcaseC\-672/16"/\>

<itemid="D4"type="leg"ref="Articles4and5DPR633/72"/\>

<itemid="D5"type="leg"ref="Article2135oftheCivilCode"/\>

</legal\_references\>

<citation\_reason\>

<itemref\_id="D1"\>NationalcaselawstatesthattherighttodeductVATislinkedtotherelevanceoftheassettotheentrepreneurialactivity,evenintheabsenceofimmediatetaxableoperations,providedtheassetisfunctionaltotheorganisationofthebusiness\.</item\>

<itemref\_id="D2"\>TheEUCourtofJusticereiteratesthattherighttodeductexistsforinvestmentexpensesincurredinviewoffutureentrepreneurialoperations,withoutwaitingfortheactualstartoftheactivity\.</item\>

<itemref\_id="D3"\>TheEUCourtofJusticeemphasisesthatVATdeductibilityisdeterminedbythepurchaseoftheasset,notbyitsimmediateuse,inordertoensuretaxneutrality\.</item\>

</citation\_reason\>

<judge\_reasoning\>Thejudgeconsidersthatthepurchaseofthetractorisconnectedtothetaxpayer’sagriculturalactivity,despitetheabsenceofimmediatetaxableoperations,asitisfunctionaltolandpreparationandthefarmingprocess\.BothnationalandEUcaselawconfirmthatVATdeductiondoesnotrequiretheactualstartoftheactivity,butonlytherelevanceoftheassettoentrepreneurialpurposes\.TheRevenueAgency’sobjections,basedonnon\-probativeindicators,donotoverridethisprinciple\.Therefore,thedenialisannulledandtheVATrefundisgranted\.

</judge\_reasoning\>

<summary\>Thetaxpayer,whooperatesasnail\-farmingbusiness,purchasedatractorin2022topreparetheland,despiteproductionnotyethavingbegun\.TheItalianRevenueAgencydeniedtheVATrefund,consideringtheactivitynotyetconcretelystarted\.Thejudge,referringtoArticle30ofPresidentialDecree633/72andcaselaw\(CivilSupremeCourtno\.26689/2022andEUCourtofJusticeC\-110/94\),heldthatdeductionisadmissibleforassetsnecessaryfortheorganisationofthebusiness,evenduringthepreparatoryphase,providedthereisanobjectivelinkwiththeplannedactivity\.Therulingannulledthedenialandrecognisedthedeductibility\.

</summary\>

</issue\>

</issues\>

## Appendix HSensitivity analysis: run\-to\-run consistency

As noted in Section[4\.3](https://arxiv.org/html/2607.03325#S4.SS3), the extraction model is not deterministic: even at zero temperature and with a fixed random seed, repeated calls on identical inputs can yield different outputs, a phenomenon also documented byBlair\-Stanek and Van Durme \[[2025](https://arxiv.org/html/2607.03325#bib.bib46)\]\. All results in the main text are based on a single run of the pipeline, and it is natural to ask how much they would change under replication\. In this appendix we quantify this run\-to\-run variability treating it in the same way as the inter\-annotator disagreement of Section[6](https://arxiv.org/html/2607.03325#S6), so that the two sources of variation can be compared\.

#### Protocol\.

We ran the extraction pipeline a second time on the5050test judgments, under identical settings \(same prompts, model version, temperature, and seed\), obtaining a second collection of XML extractions\. We denote byR1R\_\{1\}the run used throughout the main validation and byR2R\_\{2\}the replication\. A tax\-law expert then compared the two runs with the same set\-based procedure adopted for the human validation: for each judgment the issues ofR1R\_\{1\}andR2R\_\{2\}were aligned by content, and within each pair of aligned issues the corresponding legal references were aligned, counting individual cited documents \(a single`<item\>`may bundle several, e\.g\.*art\. 21, commi 1 e 2*\)\. TreatingR1R\_\{1\}andR2R\_\{2\}as two annotators, we report precision, recall, and F1 with the definitions of Section[5](https://arxiv.org/html/2607.03325#S5): for a system runSSand a ground\-truth runGG,PS∣G=\|S∩G\|/\|S\|\\mathrm\{P\}\_\{S\\mid G\}=\|S\\cap G\|/\|S\|andRS∣G=\|S∩G\|/\|G\|\\mathrm\{R\}\_\{S\\mid G\}=\|S\\cap G\|/\|G\|, so the two directions exchange precision and recall but share the same F1\. Citations are evaluated within the issues on which the two runs agree, the analogue of the*present issues*of the main text\.

Two features of this set\-up should be kept in mind\. First, the comparison is carried out on the*raw*LLM outputs, before the hallucination filter of Section[6\.3](https://arxiv.org/html/2607.03325#S6.SS3), whereas the citation results in the main text are computed*after*filtering; since the filter removes references that do not appear in the judgment, and different runs fabricate different references, part of the citation variability reported below would be eliminated by the filter\. We chose to compare the raw outputs since we want to measure the LLM’s sensitivity in isolation\. Second, the alignment is itself an imperfect, manual step: we have no control over exactly which issues each run extracts, and the annotator must match issues that are phrased differently and are not necessarily in a one\-to\-one correspondence\. In particular, the two runs do not extract identical issues, and even within an aligned pair the issue formulation can differ slightly between runs, shifting the set of references relevant to it\. Part of the disagreement on references is therefore inherited from the disagreement on issues rather than being a property of citation extraction alone\.

#### Extraction volume\.

The two runs are close in volume \(Table[18](https://arxiv.org/html/2607.03325#A8.T18)\), with the replicationR2R\_\{2\}slightly more prolific: it extracts1\.661\.66issues per judgment against1\.561\.56forR1R\_\{1\}\(8383vs7878issues in total\), and4\.114\.11vs3\.703\.70legal references per aligned issue\. The difference is small and does not, on its own, indicate a systematic shift between runs\.

Table 18:Average number of items per unit \(totals in parentheses\), per run\. The6363aligned issues are those identified by both runs\.
#### Issue extraction\.

The two runs identify the same legal issues with F1=78\.3%\{=\}78\.3\\%\(Table[19](https://arxiv.org/html/2607.03325#A8.T19)\)\. Of the5050judgments,3030\(60%60\\%\) are segmented identically; on the remaining2020at least one issue is found by only one of the two runs\. The two runs share6363issues\.

Table 19:Run\-to\-run agreement on issue extraction \(5050judgments\)\. \(ref\. system∣\\midground truth\)\. The two directions share the same F1\.
#### Citation extraction\.

Within the6363aligned issues, the two runs agree on legal references with F1=73\.2%\{=\}73\.2\\%\(Table[20](https://arxiv.org/html/2607.03325#A8.T20)\)\. Agreement is far from complete:77of the6363aligned issues \(11%11\\%\) share no reference at all\.

Table 20:Run\-to\-run agreement on citation extraction, within the6363aligned issues\. References are counted as individual cited documents\.
#### Comparison with inter\-annotator agreement\.

The interest of these figures lies in the comparison with the human results of Section[6](https://arxiv.org/html/2607.03325#S6), collected in Table[21](https://arxiv.org/html/2607.03325#A8.T21)\. Two runs of the same model agree with each other*less*than two human experts do: on issue extraction the run\-to\-run F1 \(78\.3%78\.3\\%\) sits below the inter\-annotator F1 \(88\.1%88\.1\\%\), and on citations the gap is much wider \(73\.2%73\.2\\%vs97\.1%97\.1\\%\)\. At the same time, the run\-to\-run agreement is of the same order as the agreement between the model and a human expert, falling within the LLM\-vs\-annotator range on both issues \(72\.772\.7–83\.6%83\.6\\%\) and citations \(73\.873\.8–74\.8%74\.8\\%\)\. A substantial part of the gap between the model and the human annotators is thus not a stable, systematic difference but the model’s own stochasticity; for citations in particular, run\-to\-run variance is as large as the model–human gap itself\.

Table 21:Agreement \(F1, %\) under three regimes: between the two human annotators \(IAA\), between the model and each annotator \(LLM–Human\), and between two runs of the model \(Run–Run\)\. Human figures are reproduced from Tables[2](https://arxiv.org/html/2607.03325#S6.T2)and[5](https://arxiv.org/html/2607.03325#S6.T5)\.These figures are conservative on two counts\. The citation comparison is run on raw outputs: the hallucination filter, which removes the fabricated references that differ from run to run, would raise the measured agreement, so the variability of the whole pipeline’s output is smaller than Table[20](https://arxiv.org/html/2607.03325#A8.T20)suggests\. And part of the reference disagreement is inherited from the imperfect alignment of issues across runs rather than from citation extraction proper\. The practical implication is, however, clear: single\-run scores are affected by the stochasticity of the model\. Averaging over several judgments provides a way of reducing this noise\.

## Appendix IRule\-based citation extraction as a baseline

Section[3\.3](https://arxiv.org/html/2607.03325#S3.SS3)argues that a citation network built directly from judgments would include many references that are not central to the court’s reasoning\. Here we quantify that claim, and at the same time ask how much of the benefit of the LLM extraction could be obtained by a purely rule\-based pipeline\. We compare three ways of attaching a set of cited documents to a judgment: \(i\) all citations parsed by Linkoln from the full text; \(ii\) all citations parsed by Linkoln from the “Motivi della decisione” section, located with the same regex\-based matcher used in preprocessing \(Section[4\.1](https://arxiv.org/html/2607.03325#S4.SS1)\); \(iii\) the citations extracted by our pipeline, after the hallucination filter\. The reference standard is the set of citations that the experts judged relevant to the issues of the judgment\.

#### Protocol\.

The comparison is restricted to judgments where \(a\) the annotators flagged no missing issues \(otherwise their per\-issue citation lists would not cover the entire judgment\) and \(b\) the section heading is detectable\. Of the5050validation judgments,3939satisfy \(a\),3939satisfy \(b\), and3131satisfy both \(1919annotated byA1A\_\{1\},2020byA2A\_\{2\}\)\. Because the expert citations are recorded as free text, we parse them with Linkoln as well, and match citations automatically at the document level \(number and year, completed with URN identifiers where needed\)\. This automatic matching differs from the manual matching underlying Table[5](https://arxiv.org/html/2607.03325#S6.T5), so absolute scores are not comparable between the two analyses; within this appendix, however, all three tiers are evaluated identically\. References that the parser cannot represent— general legal principles, administrative practice, and generic collective references \(e\.g\.*giurisprudenza consolidata*\)—are excluded from all tiers\.

Table 22:Citations per judgment and pooled precision/recall of three extraction tiers against the expert\-relevant citations, on the3131qualifying judgments\. Automatic document\-level matching; principles excluded\. In brackets:95%95\\%judgment\-level bootstrap confidence intervals\.
#### Results\.

Table[22](https://arxiv.org/html/2607.03325#A9.T22)shows a consistent gradient\. The full text contains the most citations per judgment and the lowest precision, confirming the dilution argument\. Restricting the parser to the reasoning section is a strong and essentially free improvement: it removes the references contained in the parties’ submissions at no cost in recall \(every relevant citation found in the full text already lies in the section\)\. The LLM extraction is the most selective tier and the most precise, and gives up1010–2020points of recall in exchange\. We believe the recall gap can be bridged by using more powerful LLMs\.

On the subset where the section heading can be located, a Linkoln\-only pipeline is a reasonable low\-cost alternative for*judgment\-level*citation networks: it requires no LLM calls and achieves higher recall than our pipeline\. The case for the LLM extraction rests on what no section\-level method provides: \(i\) the attribution of each citation to the specific issue it serves, which is what makes the edges of the network interpretable and is a prerequisite for the issue\-level uses of Section[3\.3](https://arxiv.org/html/2607.03325#S3.SS3); \(ii\) the presence of additional`<citation\_reason\>`elements attached to each citation; and \(iii\) applicability to the whole corpus, whereas the section heading is detected in only3939of the5050validation judgments \(78%78\\%\)\. These results are based on3131judgments and automatic matching, and should be read as indicative\.

## Appendix JModel selection

The choice of extraction model was made through an informal, qualitative comparison carried out during the development phase, prior to and separately from the validation of Section[5](https://arxiv.org/html/2607.03325#S5)\. Because the pipeline must run over several hundred thousand judgments, the goal was not to identify the single most capable model but a sufficiently cheap model whose output the legal experts judged acceptable\. The assessment was deliberately lightweight: we did not compute the set\-based metrics of the main text, which would have required a second full annotation effort, but relied on the two tax\-law experts reading and commenting on the extractions produced by each candidate on a small development set of 10 decisions, disjoint from the5050validation judgments\.

We compared a range of open\-weight, cost\-efficient models against a top\-range proprietary reference \(Claude Sonnet 4\.5\)\. The open\-weight candidates included several DeepSeek V3 variants \(V3\-0324, V3\.1, and V3\.2\-exp\), Llama 3\.3 70B, Gemma 3 27B, Qwen QwQ 32B, Mistral Devstral, and Kimi\-K2\. Models were accessed through third\-party API providers\.

For each candidate the experts considered, informally: \(i\) the quality of the segmentation of a judgment into issues, penalising both artificial splitting and the omission of genuine issues; \(ii\) the precision and appropriateness of the legal language in the free\-text fields; \(iii\) the faithfulness of the extracted`<legal\_references\>`, i\.e\. the tendency to fabricate or misattribute citations \(e\.g\. misattributing the references relevant for one issue to another of the same judgment\); and \(iv\) price\.

Among the models reaching a quality acceptable to the experts on criteria \(i\)–\(iv\), DeepSeek V3\-0324 offered the best trade\-off with cost and was selected for the bulk of the extraction\. Claude Sonnet 4\.5 produced better output but at roughly twenty times the price \(Section[4\.3](https://arxiv.org/html/2607.03325#S4.SS3)\), which is not sustainable at corpus scale\. The smaller open\-weight models \(Llama 3\.3 70B, Gemma 3 27B, QwQ 32B, Devstral\) were less reliable on issue segmentation and on the precision of legal language, while the remaining large open\-weight models did not offer a clear advantage over DeepSeek at comparable cost\. The slightly newer DeepSeek V3\.1 was adopted only for the lightweight title\-correction step \(Section[4\.3](https://arxiv.org/html/2607.03325#S4.SS3)\)\. We stress that this comparison is qualitative and was intended to support an engineering decision under a cost constraint, not to establish a ranking of the models on the extraction task\.