IntLawNER: 国际法中的命名实体识别数据集与基准

arXiv cs.AI 论文
named-entity-recognition dataset benchmark international-law nlp legal-ai evaluation

摘要

IntLawNER 是一个新的用于国际法的命名实体识别数据集和基准，涵盖了来自法律文本的金标准标注句子，并评估了模型在少样本改进方面的性能。

arXiv:2609.22529v1 Announce Type: new Abstract: International law provides the normative framework through which states coordinate action, regulate armed conflict, and protect human rights, yet its texts remain without token-level named entity recognition (NER) resources. We introduce IntLawNER, a NER dataset and benchmark for codified sources of international law, covering 2,987 gold-annotated sentences and 8,094 entity spans from International Court of Justice (ICJ) decisions, UN Security Council resolutions, and European Court of Human Rights (ECtHR) judgments, annotated with seven institution-specific entity types. We construct IntLawNER with a cost-effective hybrid algorithmic-agentic pipeline that reduces 468k source sentences to a compact annotation set through candidate retrieval, LLM-based vetting, and human review, with 89.6% of gold spans accepted unchanged from the silver layer. However, the silver-to-gold analysis reveals that human-machine aggregate agreement metrics can be misleading in domain-specific NER: Cohen's kappa=0.964 on boundary-matched spans masks a macro-F1 of 0.753 when missing entities, boundary errors, and label corrections are included. The benchmark shows that zero-shot span-based GLiNER collapses on entity types dependent on institutional function rather than surface form (0.243 micro-F1), while fine-tuned transformers struggle on rare labels. Carefully selected few-shot examples that demonstrate label contrasts improve every LLM over zero-shot prompting, with Claude Opus 4.6 reaching the best score of 0.873 micro-F1. We release IntLawNER as a benchmark and reusable resource for extracting references in international legal texts.
查看原文
查看缓存全文

缓存时间: 2026/09/23 09:08 

# IntLawNER: A Named Entity Recognition Dataset and Benchmark in International Law
Source: [https://arxiv.org/abs/2609.22529](https://arxiv.org/abs/2609.22529)
[View PDF](https://arxiv.org/pdf/2609.22529)

> Abstract:International law provides the normative framework through which states coordinate action, regulate armed conflict, and protect human rights, yet its texts remain without token\-level named entity recognition \(NER\) resources\. We introduce IntLawNER, a NER dataset and benchmark for codified sources of international law, covering 2,987 gold\-annotated sentences and 8,094 entity spans from International Court of Justice \(ICJ\) decisions, UN Security Council resolutions, and European Court of Human Rights \(ECtHR\) judgments, annotated with seven institution\-specific entity types\. We construct IntLawNER with a cost\-effective hybrid algorithmic\-agentic pipeline that reduces 468k source sentences to a compact annotation set through candidate retrieval, LLM\-based vetting, and human review, with 89\.6% of gold spans accepted unchanged from the silver layer\. However, the silver\-to\-gold analysis reveals that human\-machine aggregate agreement metrics can be misleading in domain\-specific NER: Cohen's kappa=0\.964 on boundary\-matched spans masks a macro\-F1 of 0\.753 when missing entities, boundary errors, and label corrections are included\. The benchmark shows that zero\-shot span\-based GLiNER collapses on entity types dependent on institutional function rather than surface form \(0\.243 micro\-F1\), while fine\-tuned transformers struggle on rare labels\. Carefully selected few\-shot examples that demonstrate label contrasts improve every LLM over zero\-shot prompting, with Claude Opus 4\.6 reaching the best score of 0\.873 micro\-F1\. We release IntLawNER as a benchmark and reusable resource for extracting references in international legal texts\.

## Submission history

From: Genis Skura \[[view email](https://arxiv.org/show-email/b621d16a/2609.22529)\] **\[v1\]**Fri, 18 Sep 2026 19:36:30 UTC \(509 KB\)

相似文章

GLiNER-Relex：联合命名实体识别与关系提取的统一框架

Hugging Face Daily Papers

GLiNER-Relex 是一个用于联合命名实体识别（NER）与关系提取（RE）的统一框架，利用共享的 Transformer 编码器实现零样本能力。该论文展示了模型在标准基准测试中具有竞争力的性能，并将其作为开源 Python 包发布。

LegalPincite：多层级法律信息检索数据集

Hugging Face Daily Papers

介绍LegalPincite，一个基于欧盟法院（CJEU）判决构建的大规模法律信息检索数据集，包含掩码查询、完整语料库和段落级引文标注，以支持多层级检索评估。