Shortcut Learning in Legal Judgment Prediction: Empirical Evidence from the UK Employment Tribunal

arXiv cs.AI Papers
legal-judgment-prediction shortcut-learning uk-employment-tribunal llm nlp leakage

Summary

This paper empirically investigates shortcut learning in legal judgment prediction using UK Employment Tribunal claims, finding that models exploit outcome-revealing cues in post-hoc judicial texts, but performance remains robust after masking such leakage features.

arXiv:2607.04261v1 Announce Type: new Abstract: Current Legal Judgment Prediction (LJP) is constrained by its reliance on post-hoc judicial materials, increasing the likelihood that models perform retrospective classification rather than true forecasting. This paper empirically investigates shortcut learning in this context by studying claim-level outcome prediction in UK Employment Tribunal (UKET) decisions. Using a corpus of 33,158 individual claims, we predict outcomes from claim texts and LLM-extracted case summaries, evaluating models ranging from interpretable TF-IDF-based classifiers to black-box LLMs. While headline predictive performance figures appear strong, we demonstrate that such performance in LJP systems trained on post-hoc judicial text can be driven by the retrospective nature of the source material. Stratifying the test data by human judgments of leakage reveals that performance increases where outcome-revealing cues are embedded in the narrative. Moreover, a model trained on just the 4% of features identified as leakage achieves high performance, outperforming human experts. These findings substantiate concerns that LJP performance may be exaggerated by linguistic artefacts. Yet this vulnerability is not fatal to the research agenda. Instead, post-hoc judgments might be treated as potentially contaminated texts, requiring active auditing. Retraining models after masking leakage features results in only a negligible reduction in Macro-F1. Hence, while models will opportunistically exploit shortcuts when available, they remain capable of extracting useful predictive signals when these artefacts are removed.
Original Article
View Cached Full Text

Cached at: 07/07/26, 04:36 AM 

# Shortcut Learning in Legal Judgment Prediction: Empirical Evidence from the UK Employment Tribunal
Source: [https://arxiv.org/abs/2607.04261](https://arxiv.org/abs/2607.04261)
[View PDF](https://arxiv.org/pdf/2607.04261)

> Abstract:Current Legal Judgment Prediction \(LJP\) is constrained by its reliance on post\-hoc judicial materials, increasing the likelihood that models perform retrospective classification rather than true forecasting\. This paper empirically investigates shortcut learning in this context by studying claim\-level outcome prediction in UK Employment Tribunal \(UKET\) decisions\. Using a corpus of 33,158 individual claims, we predict outcomes from claim texts and LLM\-extracted case summaries, evaluating models ranging from interpretable TF\-IDF\-based classifiers to black\-box LLMs\. While headline predictive performance figures appear strong, we demonstrate that such performance in LJP systems trained on post\-hoc judicial text can be driven by the retrospective nature of the source material\. Stratifying the test data by human judgments of leakage reveals that performance increases where outcome\-revealing cues are embedded in the narrative\. Moreover, a model trained on just the 4% of features identified as leakage achieves high performance, outperforming human experts\. These findings substantiate concerns that LJP performance may be exaggerated by linguistic artefacts\. Yet this vulnerability is not fatal to the research agenda\. Instead, post\-hoc judgments might be treated as potentially contaminated texts, requiring active auditing\. Retraining models after masking leakage features results in only a negligible reduction in Macro\-F1\. Hence, while models will opportunistically exploit shortcuts when available, they remain capable of extracting useful predictive signals when these artefacts are removed\.

## Submission history

From: Joe Watson \[[view email](https://arxiv.org/show-email/33342473/2607.04261)\] **\[v1\]**Sun, 5 Jul 2026 12:15:47 UTC \(1,152 KB\)

Similar Articles