IntLawNER: A Named Entity Recognition Dataset and Benchmark in International Law

# IntLawNER: A Named Entity Recognition Dataset and Benchmark in International Law
Source: [https://arxiv.org/abs/2609.22529](https://arxiv.org/abs/2609.22529)
[View PDF](https://arxiv.org/pdf/2609.22529)

> Abstract:International law provides the normative framework through which states coordinate action, regulate armed conflict, and protect human rights, yet its texts remain without token\-level named entity recognition \(NER\) resources\. We introduce IntLawNER, a NER dataset and benchmark for codified sources of international law, covering 2,987 gold\-annotated sentences and 8,094 entity spans from International Court of Justice \(ICJ\) decisions, UN Security Council resolutions, and European Court of Human Rights \(ECtHR\) judgments, annotated with seven institution\-specific entity types\. We construct IntLawNER with a cost\-effective hybrid algorithmic\-agentic pipeline that reduces 468k source sentences to a compact annotation set through candidate retrieval, LLM\-based vetting, and human review, with 89\.6% of gold spans accepted unchanged from the silver layer\. However, the silver\-to\-gold analysis reveals that human\-machine aggregate agreement metrics can be misleading in domain\-specific NER: Cohen's kappa=0\.964 on boundary\-matched spans masks a macro\-F1 of 0\.753 when missing entities, boundary errors, and label corrections are included\. The benchmark shows that zero\-shot span\-based GLiNER collapses on entity types dependent on institutional function rather than surface form \(0\.243 micro\-F1\), while fine\-tuned transformers struggle on rare labels\. Carefully selected few\-shot examples that demonstrate label contrasts improve every LLM over zero\-shot prompting, with Claude Opus 4\.6 reaching the best score of 0\.873 micro\-F1\. We release IntLawNER as a benchmark and reusable resource for extracting references in international legal texts\.

## Submission history

From: Genis Skura \[[view email](https://arxiv.org/show-email/b621d16a/2609.22529)\] **\[v1\]**Fri, 18 Sep 2026 19:36:30 UTC \(509 KB\)