NER-D:面向德语命名实体识别的跨领域基准
摘要
本文介绍了 NER-D,一个面向德语命名实体识别(NER)的跨领域基准,涵盖 9 个领域的 13 个数据集。该研究比较了专有大语言模型(GPT-5.4 mini)和开源模型在提示、上下文学习及 LoRA 微调设置下的表现,发现经 LoRA 微调的小型模型能够显著优于专有模型基线。
arXiv:2611.18257v1 Announce Type: new
Abstract: We present NER-D, a cross-domain benchmark for German NER comprising 13 datasets spanning 9 domains, including General/Politics, Wikipedia, Legal, Medical, Historical, Financial, Agriculture and Archaeology. We examine the performance of both proprietary LLMs and state-of-the-art small open-source models using normal prompting and further examine the performance gains achievable via in-context learning and LoRA fine-tuning. Our results show that without any sort of optimization a proprietary model (GPT-5.4 mini) outperforms the smaller open-source models on 6 out of 13 datasets, but LoRA fine-tuning enables the smaller models to substantially outperform the proprietary baseline. We further analyze the influence of the domain and the tagset used in each task on the performance achieved in the different evaluation settings. Our benchmark will be available as a resource for the development of future German NER methods on both general and specialized domains.\footnote{We make our code, data, and models publicly available upon acceptance of this paper.
查看缓存全文
缓存时间: 2026/11/16 08:27
# NER-D: A cross-domain benchmark for German NER Source: [https://arxiv.org/html/2611.18257](https://arxiv.org/html/2611.18257) Marc BrinnerAffiliation:Computational Linguistics, Department of Linguistics, Bielefeld UniversityEmail:[marc\.brinner@uni\-bielefeld\.de](mailto:[email protected])Philipp CimianoAffiliation:Center for Cognitive Interaction Technology, Faculty of Technology, Bielefeld UniversityEmail:[karolina\.czok@uni\-bielefeld\.de](mailto:[email protected])Karolina Hanna CzokAffiliation:Center for Cognitive Interaction Technology, Faculty of Technology, Bielefeld UniversityEmail:[regina\.stodden@uni\-bielefeld\.de](mailto:[email protected])Regina StoddenAffiliation:Center for Cognitive Interaction Technology, Faculty of Technology, Bielefeld UniversityEmail:[viju\.sudhi@uni\-bielefeld\.de](mailto:[email protected])Viju SudhiAffiliation:Center for Cognitive Interaction Technology, Faculty of Technology, Bielefeld UniversityEmail:[sina\.zarriess@uni\-bielefeld\.de](mailto:[email protected])Sina ZarrießAffiliation:Computational Linguistics, Department of Linguistics, Bielefeld UniversityEmail:[cimiano@techfak\.uni\-bielefeld\.de](mailto:[email protected]) ###### Abstract We present NER\-D, a cross\-domain benchmark for German NER comprising 13 datasets spanning 9 domains, including General/Politics, Wikipedia, Legal, Medical, Historical, Financial, Agriculture and Archaeology\. We examine the performance of both proprietary LLMs and state\-of\-the\-art small open\-source models using normal prompting and further examine the performance gains achievable via in\-context learning and LoRA fine\-tuning\. Our results show that without any sort of optimization a proprietary model \(GPT\-5\.4 mini\) outperforms the smaller open\-source models on 6 out of 13 datasets, but LoRA fine\-tuning enables the smaller models to substantially outperform the proprietary baseline\. We further analyze the influence of the domain and the tagset used in each task on the performance achieved in the different evaluation settings\. Our benchmark will be available as a resource for the development of future German NER methods on both general and specialized domains\.111We make our code, data, and models publicly available upon acceptance of this paper\. ## 1Introduction Large language models \(LMs\) have demonstrated strong performance across a variety of natural language processing tasks, including named entity recognition \(NER\)[Brown et al\. \(2020\)](https://arxiv.org/html/2611.18257#bib.bib14)\. Proprietary models, such as those provided through commercial APIs, often achieve superior performance compared to open\-weight alternatives\. However, their practical adoption is limited by concerns regarding data privacy, lack of transparency and interpretability, dependence on third\-party providers, and limited possibilities for model optimization[Liao and Wortman Vaughan \(2024\)](https://arxiv.org/html/2611.18257#bib.bib9);[Zhang et al\. \(2024\)](https://arxiv.org/html/2611.18257#bib.bib10);[Yan et al\. \(2025\)](https://arxiv.org/html/2611.18257#bib.bib11)\. In contrast, small open\-weight LMs can be deployed on private infrastructure, allowing sensitive data to remain local\. They can be customized through fine\-tuning and incur no costs beyond the required infrastructure, making them an attractive alternative for privacy\-sensitive applications and organizations requiring customizable, cost\-effective NLP solutions[Kukreja et al\. \(2024\)](https://arxiv.org/html/2611.18257#bib.bib12);[Manchanda et al\. \(2025\)](https://arxiv.org/html/2611.18257#bib.bib13)\. NER is a well\-researched task in natural language processing\. It can be formulated as an information extraction or sequence labeling task in which text spans are annotated with predefined entity labels \(e\.g\., person, location, organization\)[Li et al\. \(2022\)](https://arxiv.org/html/2611.18257#bib.bib7)\. While NER has been extensively studied for high\-resource settings and general domains, research in specialized domains and languages other than English remains limited[Krasadakis et al\. \(2024\)](https://arxiv.org/html/2611.18257#bib.bib16)\. One major challenge in specialized domains requiring expert knowledge is the lack of annotated training data, often caused by the high effort required to design annotation guidelines and manually create high\-quality annotations, as well as the high costs associated with expert annotation[Dao et al\. \(2025\)](https://arxiv.org/html/2611.18257#bib.bib15)\. Examples[1](https://arxiv.org/html/2611.18257#S1)and[1](https://arxiv.org/html/2611.18257#S1)show the differences in complexity of annotation between general\-domain and specialized\-domain NER\.\\ex\.Sundar PichaiPER\{\}\_\{\\text\{PER\}\}was born inIndiaLOC\{\}\_\{\\text\{LOC\}\}and is a CEO ofGoogleORG\{\}\_\{\\text\{ORG\}\}\. \\ex \. The patient withtype 2 diabetesDISEASE\{\}\_\{\\text\{DISEASE\}\}was treated withMetforminDRUG\{\}\_\{\\text\{DRUG\}\}but developedgastrointestinal intoleranceADVERSE\_EFFECT\{\}\_\{\\text\{ADVERSE\\\_EFFECT\}\}withpersistent nauseaSYMPTOM\{\}\_\{\\text\{SYMPTOM\}\}\. In addition, privacy constraints in domains such as medicine can limit the availability and sharing of annotated datasets\. For clinical German NLP, for example,[Frei and Kramer \(2022\)](https://arxiv.org/html/2611.18257#bib.bib8)have highlighted that only very scarce data is available due to privacy and other constraints\. Despite the growing availability of German NER datasets from both general and specialized domains, these datasets are typically evaluated independently due to differences in annotation schemes and domain characteristics – except[Pfister and Hotho \(2024\)](https://arxiv.org/html/2611.18257#bib.bib34)which provide a German NLP benchmark across several tasks\. Therefore, it remains unclear how well current LMs perform on German NER across specialized domains and in different settings \(i\.e\., zero\-shot, few\-shot, and fine\-tuning\)\. To address this question, we propose NER\-D as a new German NER benchmark that comprises 13 datasets from 9 different domains\. We examine the performance of small open\-source models compared to proprietary models \(GPT\-5\.4 mini in particular\) across all the datasets, investigating how big the gap is between proprietary and open\-source models under different training regimens \(zero\-shot, few\-shot and low\-rank fine\-tuning\)\. In particular, we address the following research questions: 1. RQ1How does NER performance vary across different domains? 2. RQ2How does the performance of small, open\-weight LMs compare to proprietary SLM? 3. RQ3To what extent can few\-shot learning and parameter\-efficient fine\-tuning reduce the performance gap between proprietary SLM and small, open\-weight LMs? ## 2Related Work Several recent studies have investigated the capabilities of LMs for NER across different languages and domains\. The Universal NER \(UNER\) benchmark[Blevins et al\. \(2026\)](https://arxiv.org/html/2611.18257#bib.bib17)combines NER datasets from multiple languages using consistent annotation guidelines and evaluates both encoder\-based models and LMs\. For German,[Pfister and Hotho \(2024\)](https://arxiv.org/html/2611.18257#bib.bib34)introduced SuperGLEBer, a broad benchmark for evaluating German LMs across multiple tasks, including sequence tagging tasks such as NER\. For NER, five datasets in German were included, covering different domains: news \(XGLUE;[Liang et al\., 2020](https://arxiv.org/html/2611.18257#bib.bib33)\), politics \(EUROPARL;[Faruqui and Padó, 2010](https://arxiv.org/html/2611.18257#bib.bib1)\), biomedical \(BIOfid;[Ahmed et al\., 2019](https://arxiv.org/html/2611.18257#bib.bib28)\), Wikipedia/news \([Benikova et al\., 2014](https://arxiv.org/html/2611.18257#bib.bib29)\), and legal texts \([Leitner et al\., 2019](https://arxiv.org/html/2611.18257#bib.bib2)\)\. However, this benchmark focuses only on the evaluation of German language models rather than comparing state\-of\-the\-art proprietary and open\-weight LMs\. Other studies have investigated LMs for specialized domains\. For example, in the comparative study of LMs for legal NER by[Deußer et al\. \(2024\)](https://arxiv.org/html/2611.18257#bib.bib18), the authors evaluated eleven models on legal NER tasks across seven datasets in five languages, including English, Portuguese, German, Turkish, and Ukrainian\.[Spiegel et al\. \(2025\)](https://arxiv.org/html/2611.18257#bib.bib35)investigated the use of 7\-billion\-parameter LMs for clinical NER in the German medical domain, evaluating their performance on three publicly available German gold\-standard datasets\. The authors additionally investigated instruction tuning with QLoRA and showed that it improves performance and reduces hallucinations\. For historical texts,[Hiltmann et al\. \(2025\)](https://arxiv.org/html/2611.18257#bib.bib19)evaluated GPT\-4o on a manually curated historical dataset, testing zero\-shot and few\-shot approaches under different prompting strategies\. However, no previous work comprehensively evaluates German NER across multiple domains using both proprietary and open\-weight models while comparing zero\-shot, few\-shot and fine\-tuning approaches\. This work addresses this gap by providing such an evaluation\. ## 3Methods ### 3\.1Data We collected 13 publicly available NER datasets covering a variety of sources, including general text, legal documents, medical texts, historical documents, and specialized domains such as finance, agriculture, and archeology\. The selection is not intended to be exhaustive but represents a diverse set of datasets that were identified through a review of available German NER resources\. We selected datasets based on their availability, annotation quality, domain diversity, and suitability for evaluating language models\. For multilingual datasets, we included only the German subset\. The datasets included in our benchmark are summarized in[Table 1](https://arxiv.org/html/2611.18257#S3.T1)and briefly described below: #### Europarl\-NER [Agerri et al\. \(2018\)](https://arxiv.org/html/2611.18257#bib.bib27);[Koehn \(2005\)](https://arxiv.org/html/2611.18257#bib.bib37)contains political proceedings from the European Parliament and provides general\-domain entity annotations\. #### SwissNER [Institute for Data Science \(2024\)](https://arxiv.org/html/2611.18257#bib.bib20)is a curated collection of 172 short, spoken\-style German sentences, focusing on Switzerland\.222As the dataset is very small with only 173 samples, we use it only for the experiments with zero\-shot setting\. #### NoSta\-D [Benikova et al\. \(2014\)](https://arxiv.org/html/2611.18257#bib.bib29)is a German NER dataset based on Wikipedia texts and annotated with general entity types as well as derivational and part\-of entity categories\. #### WikiBiography [Filippova and Strube \(2006\)](https://arxiv.org/html/2611.18257#bib.bib36)is also based on Wikipedia texts, but only on bibliography pages\. Compared to NoSta\-D it contains fewer entity types which are more general\. #### German LER [Leitner et al\. \(2020\)](https://arxiv.org/html/2611.18257#bib.bib32)is a legal NER dataset containing German legal documents from federal court decisions annotated with fine\-grained legal entities\. #### GerPS [Feddoul et al\. \(2024\)](https://arxiv.org/html/2611.18257#bib.bib4)contains German public service documents annotated with domain\-specific entities related to administrative processes\. #### GERNERMED [Frei and Kramer \(2023\)](https://arxiv.org/html/2611.18257#bib.bib5)is a German medical NER dataset focusing on clinical entities related to medication\. #### HUMADEX German NER [Sallauka et al\. \(2025\)](https://arxiv.org/html/2611.18257#bib.bib6)is a German medical NER dataset containing annotations for clinical entities such as problems, treatments, and tests\. #### NER Staatsarchiv [van der Lek et al\. \(2025\)](https://arxiv.org/html/2611.18257#bib.bib22)contains historical administrative documents from Swiss archives annotated with general entity types\. #### EHRI\-NER [Dermentzi and Scheithauer \(2024\)](https://arxiv.org/html/2611.18257#bib.bib30)is a historical dataset focused on Holocaust\-related documents, containing annotations for general entities along with holocaust specific entities such as dates, camps, and ghettos\. #### CO\-Fun [Foroutan et al\. \(2024\)](https://arxiv.org/html/2611.18257#bib.bib31)is a financial\-domain dataset annotated with entities related to company outsourcing\. #### FAIRagro [Abdelmalak et al\. \(2025\)](https://arxiv.org/html/2611.18257#bib.bib3)is an agriculture\-domain dataset created for metadata enrichment and contains fine\-grained annotations related to agricultural research\. #### Archaeo\-NER [Brandsen \(2025\)](https://arxiv.org/html/2611.18257#bib.bib21)is an archaeology\-domain dataset containing annotations for archeological entities\. To enable a consistent comparison across datasets, all resources were converted into a unified format, which includes splitting larger samples into sentences that are individually processed by the LLMs\. The processed datasets will be released in a HuggingFace collection\. After processing, each sample consists of an input sentence and an output JSON object containing all possible entity types defined for the respective dataset as keys\. The value of each key is a list of entity mentions found in the sentence; if no entity of a given type is present, the list remains empty\. This format is identical across all datasets, while the set of available entity types depends on the original annotation scheme of each resource\. In addition, duplicated entity mentions within individual labels were removed\. We used the original dataset splits whenever they were provided by the authors\. For datasets without predefined splits, we created training, validation, and test splits using an 80/10/10 ratio\. Table 1:Overview of datasets included in the NER\-D benchmark\. ### 3\.2Models We investigate a total of four models spanning different accessibility levels, architectures, sizes, and training objectives\. We explicitly chose not to evaluate any models developed specifically for German, as the available options are already outdated relative to these state\-of\-the\-art variants, so that the inferior general task\-solving capabilities would constitute a severely limiting factor\. Further, multilinguality \(thus including the comprehension of German texts\) has become standard across all frontier open\-weight models, including those evaluated in this work\. Finally, as an additional baseline, we include OpenAI’s GPT\-5\.4\-mini[OpenAI \(2026\)](https://arxiv.org/html/2611.18257#bib.bib25)to assess whether cost\-effective closed\-source models offer any substantial advantage over their open counterparts\. ### 3\.3Experimental Setup For each dataset, we created a German task\-specific prompt describing the available entity types and the required output format \(see Appendix[A\.1](https://arxiv.org/html/2611.18257#A1.SS1)\)\. All prompts followed the same template and differed only in the manually curated entity definitions corresponding to the specific dataset\. Models were instructed to return a valid JSON object containing all entity types, with empty lists for entity types not present in the input sentence\. We intentionally did not perform prompt optimization\. Instead, we used a simple, standardized prompting strategy to evaluate model performance under realistic conditions, where extensive dataset\-specific prompt development is often impractical and requires substantial manual effort and domain expertise\. All models were evaluated on the test split of each dataset\. We report results in both zero\-shot and few\-shot settings\. In the zero\-shot setting, models received only the task instructions and the input sentence\. In the few\-shot setting, five demonstration examples sampled from the validation split were included in the prompt\. The same prompt template, inference settings and evaluation procedure were used for all models to ensure a fair comparison\. ### 3\.4Fine\-Tuning For the three open\-weights models, we also evaluate the improvements in downstream performance that can be realized via supervised fine\-tuning\. Since all models possess basic capabilities for solving the different tasks already, we experiment with performing a low\-rank adaptation[Hu et al\. \(2022\)](https://arxiv.org/html/2611.18257#bib.bib26)of the original model weights on all linear layers \(r=16r=16,α=32\\alpha=32,dropout=0\.05\\textrm\{dropout\}=0\.05,lr=1e−4\\textrm\{lr\}=1e\-4,batch size=128\\textrm\{batch size\}=128\)\. Models are trained with the same prompt format used at inference, and the loss is computed on the target tokens only, thus optimizing the model solely for producing the correct target JSON\. Model selection and early stopping rely on determining the loss on the held\-out validation split\. This validation is done after every epoch, or \(for larger datasets\) after every 10k training examples, and training is stopped if the validation loss did not decrease for three consecutive evaluations\. ### 3\.5Evaluation For each class, the model is tasked to predict a JSON object containing all class names as keys, and the corresponding list of detected entities for that class as values\. To evaluate the prediction, we parse this JSON from the model’s response, treating a JSON object that cannot be parsed as predicting no entities\. The list of predicted entities are then compared to the ground\-truth annotations to calculate the number of entities a\) that were both predicted and annotated \(true positives\), b\) entities that were predicted but are not present in the ground truth list \(false positives\) and c\) entities that belong to the ground truth but were not predicted by the model \(false negatives\)\. These individual, sample\-specific true positives, false positives, and false negatives are then accumulated over all entities of all types across the entire test set, and both micro and macro F1 scores are computed from these corpus\-level counts\. To evaluate both the broad recognition, but also the exact, verbatim matching of each ground\-truth entity, we report two variants of this F1 score that differ only in how a predicted entity is matched to a gold entity\. In theexact\-match F1, a predicted string counts as a true positive only if it is identical to a gold string of the same type\. Because generative models frequently produce near\-misses that correctly identified the entity but differ superficially \(e\.g\. an included article or a slightly different span boundary\), we additionally report apartial\-match F1: within each type, predicted and gold strings are matched one\-to\-one \(greedily, highest similarity first\) and a pair counts as a match if their token\-set similarity \(a token\-level Levenshtein ratio\) is at least 0\.90\. The partial\-match score thus upper\-bounds the exact\-match score and quantifies how much of the gap is due to superficial string differences rather than genuine extraction errors\. Zero\-shotFew\-shotFine\-tuning000\.20\.20\.40\.40\.60\.60\.80\.8Average entity\-level F1Ministral 8BQwen 3\.5 9BGemma 4 12BGPT\-5\.4 miniFigure 1:Average entity\-level F1 across all datasets per model, grouped by prompting/training strategy\. The dark segment of each bar shows the exact\-match micro\-F1; the lighter extension shows the score under partial \(fuzzy\) matching\. GPT\-5\.4 mini was evaluated zero\-shot only\.Table 2:Micro\-averaged entity\-level F1 on the German NER benchmark\. Per dataset, the best score isboldand the second bestunderlined\.*0\-shot*: zero\-shot,*5\-shot*: 5\-shot in\-context,*LoRA*: fine\-tuned\. The final row reports per\-column averages over the datasets, excluding SwissNER, which is only evaluated in the 0\-shot setting due to the small dataset size that does not allow for creating train or validation splits\.Table 3:Partial\-match \(fuzzy\) entity\-level F1 on the German NER benchmark, using token\-set\-ratio matching at threshold 0\.90\. Per dataset, bestbold, second bestunderlined; final row = per\-column averages\. ## 4Results & Discussion The results for the exact\-match micro F1 and partial\-match micro F1 are listed in Tables[2](https://arxiv.org/html/2611.18257#S3.T2)and[3](https://arxiv.org/html/2611.18257#S3.T3), respectively, and visualized in Figure[1](https://arxiv.org/html/2611.18257#S3.F1)\. Macro F1 scores are included as additional reference in Appendix[B](https://arxiv.org/html/2611.18257#A2)\. ### 4\.1Domain\-specific Analysis First, we analyze difference of performance across different domains\. Interestingly, while partial\-match micro F1 scores vary widely from 0\.275 to 0\.867 in the zero\-shot setting, we do not observe a significantly better performance in more general domains like Wikipedia or political texts compared to more specialized domains like archaeology or agriculture, even though these specialized domains are likely covered less extensively during model training\. Further, the specified entity labels and jargon of the datasets of these specialized domains do not seem to harm the quality of the models’ prediction\. However, for some specialized domain datasets like GERPS \(and other legal datasets in general\) the scores are significantly lower compared to the other datasets, which we analyze more extensively in Section[4\.3](https://arxiv.org/html/2611.18257#S4.SS3)\. Further, the evaluation identifies some variance across datasets within the same domain\. For example, models perform much better on GERNERMED than on HUMADEX \(both medical\), although the latter has a lower number of different entity labels\. However, fine\-tuning on the large training dataset helps to reduce this performance gap between the two medical datasets to some extent, with a similar pattern arising in the legal domain\. Within other domains such as historical texts the scores are quite similar across datasets, thus ultimately suggesting that the models’ performance is more dependent on specific properties of the the dataset \(e\.g\., its label set and annotation procedure\) than on the domain \(RQ1\)\. ### 4\.2Small, open LMs vs\. closed LMs #### Zero\-Shot Performance Both the open models and the proprietary model show reasonable zero\-shot performance, with average exact\-match F1 scores being in the range of0\.4300\.430\-0\.5070\.507\. This demonstrates a basic capability of solving NER tasks even in German, even though substantial differences between the different tasks exist\. For a more detailed discussion of task\-specific difficulties, see Section[4\.3](https://arxiv.org/html/2611.18257#S4.SS3)\. Comparing the open\-models with the proprietary model \(RQ2\), GPT\-5\.4 mini interestingly does not outperform the \- likely smaller \- open weights models, with Gemma 4 12B surpassing GPT\-5\.4 mini in average zero\-shot exact\-match performance \(0\.507 vs\. 0\.504\)\. However, no model is able to consistently outperform the others: while GPT\-5\.4 mini performs best on 6 out of 13 datasets, Qwen manages to outperform the other models on 3 datasets, with Gemma and Ministral leading on 2 datasets each\. Thus, while general trends in average performance over all datasets are evident, individual datasets display high variance between models so that a dataset\-specific choice of model is mandatory for optimal zero\-shot performance\. #### Few\-Shot Performance Few\-shot prompting provides additional task\-specific information by including a small number of correct input\-output examples within the model’s context during generation\. In this way, the model behavior can be adapted to the target task without requiring any modification of its parameters\. Indeed, including five demonstrations in the prompt leads to substantial improvements in exact match performance \(e\.g\., Qwen: 0\.481→\\rightarrow0\.542, Gemma: 0\.507→\\rightarrow0\.560\), thus showing that the models are able to draw inferences from the examples that lead to reliable performance improvement\. Interestingly, these improvements are much less pronounced for the partial\-match F1 score, indicating that the model does not become substantially better at recognizing entities themselves\. Instead, the larger gains in exact\-match performance suggest that the demonstrations primarily help the model learn task\-specific annotation conventions, such as whether articles or other surrounding words should be included in entity spans\. Such conventions can often be learned from just a few examples, while precise understanding of classes is usually more difficult to acquire\. #### Effects of Fine\-Tuning In contrast to the few\-shot setting, fine\-tuning allows for stronger adaptation of model behavior to a specific task\. Further, it allows the model to implicitly take information from hundreds of samples into account, in contrast to the much more limited number in the few\-shot setting\. This manifests in substantially improved performance, with exact match scores increasing on average by 0\.264 across all three models\. \(Ministral: 0\.430→\\rightarrow0\.732, Qwen: 0\.481→\\rightarrow0\.747, Gemma: 0\.507→\\rightarrow0\.731\)\. In contrast, partial match scores increase substantially as well, but by a lower 0\.173 on average \(Ministral: 0\.607→\\rightarrow0\.800, Qwen: 0\.624→\\rightarrow0\.813, Gemma: 0\.663→\\rightarrow0\.799\)\. Ultimately, fine\-tuning therefore also leads to a narrower gap between exact\-match and partial\-match performance, thus indicating that the fine\-tuned model is better at matching span\-boundaries of ground\-truth entities exactly and has thus learned the annotation conventions from the training data\. To answer research question 3, fine\-tuning not only helps surpassing the simpler in\-context learning method, but also makes small, local models substantially outperform the GPT\-model by over 0\.2 points in exact\-match F1 if averaged across all datasets\. Further, this performance gap is consistent across all datasets, so that even small numbers of annotated training samples are sufficient to make local models the most reliable choice\. Lastly, a notable insight is that average performance after fine\-tuning is very similar for all three models\. Thus, while in the untrained zero\-shot setting choosing the right model was crucial, performance differences are often less pronounced after tuning parameters with supervised data\. ### 4\.3Error Analysis #### Difficult Datasets Although all models demonstrate reasonable performance even in the zero\-shot setting, thus indicating a baseline capability for solving German NER tasks, some datasets prove substantially more challenging than others\. The GerPS dataset, for instance, consistently yields comparatively poor results across both zero\-shot and few\-shot settings, with the best\-performing models achieving a partial\-match F1 score only slightly above 0\.3\. Moreover, fine\-tuning only leads to small improvements, which suggests that there are structural factors that substantially limit downstream performance\. When analyzing failure modes on this dataset, we found that subpar performance can, at least in part, be attributed to the standardized preprocessing pipeline employed in our experiments, which involved splitting longer documents into individual sentences for both training and evaluation\. However, the original GerPS task requires identifying semantic roles within German administrative texts, which often depend on information distributed across the full document\. By removing this broader context, sentence\-level processing imposes an inherent upper bound on achievable performance\. Further, some labels \(e\.g\., "Bedingung"; EN: "condition"\) expect very long sequences, often spanning several clauses\. Such labels prove to be more difficult, especially before fine\-tuning\. In comparison, high scoring datasets contain mostly annotations with single\-word sequences \(i\.e\. SwissNER\) or only a small label set \(i\.e\., CO\-FUN\)\. Another dataset with weaker performance, particularly prior to fine\-tuning, is German LER\. In contrast to GerPS, this behavior appears to be primarily caused by the inherent complexity of the task itself\. The dataset focuses on the legal domain and contains a fine\-grained entity schema that divides semantically similar entities into different classes\. In practice, we find that the zero\-shot models are not able to perform this precise distinction and tend to assign each entity to too many classes\. While recall therefore is very high, the precision drops and the resulting F1 score is mediocre\. Fine\-tuning then improves performance substantially by helping the models distinguish better between relevant and irrelevant classes for each entity\. #### Prompt Limitations Particularly in the zero\-shot and few\-shot settings, the prompt itself is a major driver of model performance, and thus also has the potential to be an important limiting factor\. Usually, prompts provide only a concise task description and therefore cannot capture every aspect of a dataset’s annotation guidelines or class definitions, which can negatively affect downstream performance\. One effect resulting from an under\-specification of annotation rules within the prompt is the consistently higher partial\-match F1 scores compared to their exact\-match counterparts\. As discussed previously, models often succeed in identifying the correct entity but fail to identify the precise span boundaries of the ground truth annotations, since the prompt does not contain sufficient information to decide between multiple different valid annotation conventions\. In some cases, relatively small prompt refinements could alleviate this issue, for example by explicitly specifying whether determiners should be included in entity spans \(e\.g\., "the European Court" versus "European Court"\) or whether honorifics and name prefixes should be annotated \(e\.g\., "Mr\. Smith" versus "Smith"\)\. However, precisely specifying annotation rules for all edge cases is usually infeasible\. As a result, despite the increasingly strong zero\-shot capabilities of modern LLMs, supervised fine\-tuning remains the most reliable approach for achieving optimal performance\. A second limitation concerns the entity class definitions themselves\. Some datasets contain intentionally broad or heterogeneous categories that cannot be adequately described within a single sentence\. For example, the NoSta\-D dataset includes an*Other*category that encompasses multiple types of entities not covered by the other labels\. As a result, zero\-shot models struggle to apply it consistently based on the under\-specified class description\. Again, supervised fine\-tuning enables the model to learn these implicit distinctions directly from annotated examples, resulting in substantially improved performance\. ## 5Conclusion & Future Work In this benchmark called NER\-D, we provided an overview of how well small, open language models perform on the task of German named entity recognition\. NER\-D is the first benchmark that focuses on German NER across various domains, spanning from a general domain like Wikipedia to specialized domains like agriculture or archaeology\. Overall, our results show that multilingual, small, open LMs already in a zero\-shot setting have at least basic capability of solving NER tasks even in German\. Our results show that open SLMs perform differently across datasets, but the performance differences cannot be explained based on the domain specialization\. For low\-resource domains such as agriculture or archaeology, the performance of the models is comparable to the performance on other more general domains\. The complexity of the NER tasks seems to be more dependent on the annotation procedure than on the jargon on specialized labels of the domains\. We also established that small, open LMs can compete with proprietary SLMs when enough samples are available for fine\-tuning\. Although creating training data on specialized domains is very time consuming, we have shown that it is worth to let the model learn the task\- and domain specific annotation conventions\. Thus, small, open models show benefits compared to closed SLMs not only with respect to data privacy and lower resource costs but also with respect to perfomance improvements\. In future work, we plan to compare different sizes of models within the same model family to see whether our results also hold across very small models \(e\.g\., smaller than 3B\)\. Further, we have focused more on generalizability across several domains than on the best performance on each dataset or domain\. In future work, we believe that the performance of the open SLMs can even be increased by optimizing the prompt per domain or set of entity labels\. ## Limitations In this section, we discuss a few limitations of our work\. First, we only considered multilingual language models\. It would be interesting to compare our results also to newer German\-only LMs\. Further, including always 5 randomly\-picked samples in the few\-shot setting or providing less details of annotation guideline in the prompt could have limited the performance of the LMs\. Lastly, we acknowledge a limitation of the number of specialized domains compared to more general domains in the benchmark, which is mostly due to the lack of publicly available German NER datasets on specialized domains\. ## References - A\. Abdelmalak, G\. Schneider, H\. Riegler, K\. Meier, X\. Specka, N\. Svoboda, M\. Husain, and J\. FluckFAIRagro NER Dataset: Increasing FAIRness of FAIRagro Data Through AI\-Supported Metadata Enrichment\.Fachrepositorium Lebenswissenschaften \(FRL\)\.Note:Version 1\.0External Links:[Document](https://dx.doi.org/10.4126/FRL01-6526458),[Link](https://doi.org/10.4126/FRL01-6526458)Cited by:[§3\.1](https://arxiv.org/html/2611.18257#S3.SS1.SSS0.Px12.p1.1)\. - Agerriet al\.\(2018\)R\. Agerri, Y\. Chung, I\. Aldabe, N\. Aranberri, G\. Labaka, and G\. RigauBuilding named entity recognition taggers via parallel corpora\.InProceedings of the Eleventh International Conference on Language Resources and Evaluation \(LREC 2018\),N\. Calzolari, K\. Choukri, C\. Cieri, T\. Declerck, S\. Goggi, K\. Hasida, H\. Isahara, B\. Maegaard, J\. Mariani, H\. Mazo, A\. Moreno, J\. Odijk, S\. Piperidis, and T\. Tokunaga \(Eds\.\),Miyazaki, Japan\.External Links:[Link](https://aclanthology.org/L18-1557/)Cited by:[§3\.1](https://arxiv.org/html/2611.18257#S3.SS1.SSS0.Px1.p1.1)\. - Ahmedet al\.\(2019\)S\. Ahmed, M\. Stoeckel, C\. Driller, A\. Pachzelt, and A\. MehlerBIOfid dataset: publishing a German gold standard for named entity recognition in historical biodiversity literature\.InProceedings of the 23rd Conference on Computational Natural Language Learning \(CoNLL\),M\. Bansal and A\. Villavicencio \(Eds\.\),Hong Kong, China,pp\. 871–880\.External Links:[Link](https://aclanthology.org/K19-1081/),[Document](https://dx.doi.org/10.18653/v1/K19-1081)Cited by:[§2](https://arxiv.org/html/2611.18257#S2.p2.1)\. - Benikovaet al\.\(2014\)D\. Benikova, C\. Biemann, and M\. ReznicekNoSta\-D named entity annotation for German: guidelines and dataset\.InProceedings of the Ninth International Conference on Language Resources and Evaluation \(LREC’14\),N\. Calzolari, K\. Choukri, T\. Declerck, H\. Loftsson, B\. Maegaard, J\. Mariani, A\. Moreno, J\. Odijk, and S\. Piperidis \(Eds\.\),Reykjavik, Iceland,pp\. 2524–2531\.External Links:[Link](https://aclanthology.org/L14-1251/)Cited by:[§2](https://arxiv.org/html/2611.18257#S2.p2.1),[§3\.1](https://arxiv.org/html/2611.18257#S3.SS1.SSS0.Px3.p1.1)\. - Blevinset al\.\(2026\)T\. Blevins, S\. Mayhew, M\. Suppa, H\. Gonen, S\. Mirkin, V\. Pais, K\. Dobrovoljc, V\. Giouli, J\. Kevin, E\. Jang, E\. Kim, J\. Seo, X\. Gialis, and Y\. PinterUniversal ner v2: towards a massively multilingual named entity recognition benchmark\.InProceedings of the Fifteenth Language Resources and Evaluation Conference \(LREC 2026\),S\. Piperidis, N\. Bel, H\. van den Heuvel, N\. Ide, S\. Krek, and A\. Toral \(Eds\.\),Palma, Mallorca, Spain,pp\. 6609–6618\.External Links:[Document](https://dx.doi.org/10.63317/4qhcjikvgeda)Cited by:[§2](https://arxiv.org/html/2611.18257#S2.p1.1)\. - Brandsen \(2025\)A\. BrandsenArchaeo\-ner\-data\-english\-dutch\-german\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.16691762),[Link](https://doi.org/10.5281/zenodo.16691762)Cited by:[§3\.1](https://arxiv.org/html/2611.18257#S3.SS1.SSS0.Px13.p1.1)\. - Brownet al\.\(2020\)T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. AmodeiLanguage models are few\-shot learners\.InAdvances in Neural Information Processing Systems,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.F\. Balcan, and H\. Lin \(Eds\.\),Vol\.33,pp\. 1877–1901\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf)Cited by:[§1](https://arxiv.org/html/2611.18257#S1.p1.1)\. - Daoet al\.\(2025\)A\. Dao, H\. Teranishi, Y\. Matsumoto, F\. Boudin, and A\. AizawaOvercoming data scarcity in named entity recognition: synthetic data generation with large language models\.InProceedings of the 24th Workshop on Biomedical Language Processing,D\. Demner\-Fushman, S\. Ananiadou, M\. Miwa, and J\. Tsujii \(Eds\.\),Viena, Austria,pp\. 328–340\.External Links:[Link](https://aclanthology.org/2025.bionlp-1.28/),[Document](https://dx.doi.org/10.18653/v1/2025.bionlp-1.28),ISBN 979\-8\-89176\-275\-6Cited by:[§1](https://arxiv.org/html/2611.18257#S1.p2.1)\. - Dermentzi and Scheithauer \(2024\)M\. Dermentzi and H\. ScheithauerRepurposing holocaust\-related digital scholarly editions to develop multilingual domain\-specific named entity recognition tools\.InProceedings of the First Workshop on Holocaust Testimonies as Language Resources \(HTRes\) @ LREC\-COLING 2024,I\. Anuradha, M\. Wynne, F\. Frontini, and A\. Plum \(Eds\.\),Torino, Italia,pp\. 18–28\.External Links:[Link](https://aclanthology.org/2024.htres-1.3/)Cited by:[§3\.1](https://arxiv.org/html/2611.18257#S3.SS1.SSS0.Px10.p1.1)\. - Deußeret al\.\(2024\)T\. Deußer, C\. Zhao, L\. Sparrenberg, D\. Uedelhoven, A\. Berger, M\. Pielka, L\. Hillebrand, C\. Bauckhage, and R\. SifaA comparative study of large language models for named entity recognition in the legal domain\.In2024 IEEE International Conference on Big Data \(BigData\),Vol\.,pp\. 4737–4742\.External Links:[Document](https://dx.doi.org/10.1109/BigData62323.2024.10825695)Cited by:[§2](https://arxiv.org/html/2611.18257#S2.p3.1)\. - Faruqui and Padó \(2010\)M\. Faruqui and S\. PadóTraining and evaluating a german named entity recognizer with semantic generalization\.InProceedings of KONVENS 2010,Saarbrücken, Germany\.External Links:[Link](https://www.manaalfaruqui.com/papers/konvens10_faruqui.pdf)Cited by:[§2](https://arxiv.org/html/2611.18257#S2.p2.1)\. - Feddoulet al\.\(2024\)L\. Feddoul, S\. T\. Bachinger, C\. Lachenmaier, S\. Apel, P\. Karg, N\. Klewer, D\. Forshayt, R\. Erd, and M\. MauchGerPS\-NER: A Dataset for Named Entity Recognition to Support Public Service Process Creation in Germany 1\-13\.InProceedings of the Second International Workshop on Semantic Technologies and Deep Learning Models for Scientific, Technical and Legal Data \(SemTech4STLD\),Hersonissos, Greece,pp\. 1–13\.External Links:[Link](https://ceur-ws.org/Vol-3697/paper1.pdf)Cited by:[§3\.1](https://arxiv.org/html/2611.18257#S3.SS1.SSS0.Px6.p1.1)\. - Filippova and Strube \(2006\)K\. Filippova and M\. StrubeUsing linguistically motivated features for paragraph boundary identification\.InProceedings of the 2006 Conference on Empirical Methods in Natural Language Processing,D\. Jurafsky and E\. Gaussier \(Eds\.\),Sydney, Australia,pp\. 267–274\.External Links:[Link](https://aclanthology.org/W06-1632/)Cited by:[§3\.1](https://arxiv.org/html/2611.18257#S3.SS1.SSS0.Px4.p1.1)\. - Foroutanet al\.\(2024\)N\. Foroutan, M\. Schröder, and A\. DengelCO\-fun: a German dataset on company outsourcing in fund prospectuses for named entity recognition and relation extraction\.InProceedings of the 20th Conference on Natural Language Processing \(KONVENS 2024\),P\. H\. Luz de Araujo, A\. Baumann, D\. Gromann, B\. Krenn, B\. Roth, and M\. Wiegand \(Eds\.\),Vienna, Austria,pp\. 117–122\.External Links:[Link](https://aclanthology.org/2024.konvens-main.13/)Cited by:[§3\.1](https://arxiv.org/html/2611.18257#S3.SS1.SSS0.Px11.p1.1)\. - Frei and Kramer \(2022\)J\. Frei and F\. KramerGERNERMED: an open german medical ner model\.Software Impacts11,pp\. 100212\.External Links:ISSN 2665\-9638,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.simpa.2021.100212),[Link](https://www.sciencedirect.com/science/article/pii/S2665963821000944)Cited by:[§1](https://arxiv.org/html/2611.18257#S1.p4.1)\. - Frei and Kramer \(2023\)J\. Frei and F\. KramerGerman medical named entity recognition model and data set creation using machine translation and word alignment: algorithm development and validation\.JMIR Form Res7,pp\. e39077\.External Links:ISSN 2561\-326X,[Document](https://dx.doi.org/10.2196/39077),[Link](https://doi.org/10.2196/39077)Cited by:[§3\.1](https://arxiv.org/html/2611.18257#S3.SS1.SSS0.Px7.p1.1)\. - Hiltmannet al\.\(2025\)T\. Hiltmann, M\. Dröge, N\. Dresselhaus, T\. Grallert, M\. Althage, P\. Bayer, S\. Eckenstaler, K\. Mendi, J\. M\. Schmitz, P\. Schneider, W\. Sczeponik, and A\. SkibbaNER4all or context is all you need: using llms for low\-effort, high\-performance ner on historical texts\. a humanities informed approach\.External Links:2502\.04351,[Link](https://arxiv.org/abs/2502.04351)Cited by:[§2](https://arxiv.org/html/2611.18257#S2.p3.1)\. - Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.\.InProceedings of the Tenth International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/pdf?id=nZeVKeeFYf9)Cited by:[§3\.4](https://arxiv.org/html/2611.18257#S3.SS4.p1.1)\. - Institute for Data Science \(2024\)Institute for Data ScienceSwissNER\.Hugging Face\.External Links:[Link](https://huggingface.co/datasets/i4ds/swiss-ner)Cited by:[§3\.1](https://arxiv.org/html/2611.18257#S3.SS1.SSS0.Px2.p1.1)\. - Koehn \(2005\)P\. KoehnEuroparl: a parallel corpus for statistical machine translation\.InProceedings of Machine Translation Summit X: Papers,Phuket, Thailand,pp\. 79–86\.External Links:[Link](https://aclanthology.org/2005.mtsummit-papers.11/)Cited by:[§3\.1](https://arxiv.org/html/2611.18257#S3.SS1.SSS0.Px1.p1.1)\. - Krasadakiset al\.\(2024\)P\. Krasadakis, E\. Sakkopoulos, and V\. S\. VerykiosA survey on challenges and advances in natural language processing with a focus on legal informatics and low\-resource languages\.Electronics13\(3\)\.External Links:[Link](https://www.mdpi.com/2079-9292/13/3/648),ISSN 2079\-9292,[Document](https://dx.doi.org/10.3390/electronics13030648)Cited by:[§1](https://arxiv.org/html/2611.18257#S1.p2.1)\. - Kukrejaet al\.\(2024\)S\. Kukreja, T\. Kumar, A\. Purohit, A\. Dasgupta, and D\. GuhaA literature survey on open source large language models\.InProceedings of the 2024 7th International Conference on Computers in Management and Business,Singapore Singapore,pp\. 133–143\(en\)\.External Links:ISBN 9798400716652,[Link](https://dl.acm.org/doi/10.1145/3647782.3647803),[Document](https://dx.doi.org/10.1145/3647782.3647803)Cited by:[§1](https://arxiv.org/html/2611.18257#S1.p1.1)\. - Leitneret al\.\(2019\)E\. Leitner, G\. Rehm, and J\. Moreno\-SchneiderFine\-grained named entity recognition in legal documents\.InSemantic Systems\. The Power of AI and Knowledge Graphs,M\. Acosta, P\. Cudré\-Mauroux, M\. Maleshkova, T\. Pellegrini, H\. Sack, and Y\. Sure\-Vetter \(Eds\.\),Cham,pp\. 272–287\.External Links:[Link](https://doi.org/10.1007/978-3-030-33220-4_20),ISBN 978\-3\-030\-33220\-4Cited by:[§2](https://arxiv.org/html/2611.18257#S2.p2.1)\. - Leitneret al\.\(2020\)E\. Leitner, G\. Rehm, and J\. Moreno\-SchneiderA dataset of German legal documents for named entity recognition\.InProceedings of the Twelfth Language Resources and Evaluation Conference,N\. Calzolari, F\. Béchet, P\. Blache, K\. Choukri, C\. Cieri, T\. Declerck, S\. Goggi, H\. Isahara, B\. Maegaard, J\. Mariani, H\. Mazo, A\. Moreno, J\. Odijk, and S\. Piperidis \(Eds\.\),Marseille, France,pp\. 4478–4485\(eng\)\.External Links:[Link](https://aclanthology.org/2020.lrec-1.551/),ISBN 979\-10\-95546\-34\-4Cited by:[§3\.1](https://arxiv.org/html/2611.18257#S3.SS1.SSS0.Px5.p1.1)\. - Liet al\.\(2022\)J\. Li, A\. Sun, J\. Han, and C\. LiA survey on deep learning for named entity recognition\.IEEE Transactions on Knowledge and Data Engineering34\(1\),pp\. 50–70\.External Links:[Document](https://dx.doi.org/10.1109/TKDE.2020.2981314)Cited by:[§1](https://arxiv.org/html/2611.18257#S1.p2.1)\. - Lianget al\.\(2020\)Y\. Liang, N\. Duan, Y\. Gong, N\. Wu, F\. Guo, W\. Qi, M\. Gong, L\. Shou, D\. Jiang, G\. Cao, X\. Fan, R\. Zhang, R\. Agrawal, E\. Cui, S\. Wei, T\. Bharti, Y\. Qiao, J\. Chen, W\. Wu, S\. Liu, F\. Yang, D\. Campos, R\. Majumder, and M\. ZhouXGLUE: a new benchmark dataset for cross\-lingual pre\-training, understanding and generation\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 6008–6018\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.484/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.484)Cited by:[§2](https://arxiv.org/html/2611.18257#S2.p2.1)\. - Liao and Wortman Vaughan \(2024\)Q\. V\. Liao and J\. Wortman VaughanAI Transparency in the Age of LLMs: a Human\-Centered Research Roadmap\.Harvard Data Science Review\(Special Issue 5: Grappling With the Generative AI Revolution\)\.External Links:[Link](https://doi.org/10.1162/99608f92.8036d03b)Cited by:[§1](https://arxiv.org/html/2611.18257#S1.p1.1)\. - Manchandaet al\.\(2025\)J\. Manchanda, L\. Boettcher, M\. Westphalen, and J\. JasserThe open source advantage in large language models \(llms\)\.External Links:2412\.12004,[Link](https://arxiv.org/abs/2412.12004)Cited by:[§1](https://arxiv.org/html/2611.18257#S1.p1.1)\. - OpenAI \(2026\)OpenAIIntroducing GPT‑5\.4\(Website\)External Links:[Link](https://openai.com/index/introducing-gpt-5-4/)Cited by:[§3\.2](https://arxiv.org/html/2611.18257#S3.SS2.p4.1)\. - Pfister and Hotho \(2024\)J\. Pfister and A\. HothoSuperGLEBer: German language understanding evaluation benchmark\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 7904–7923\.External Links:[Link](https://aclanthology.org/2024.naacl-long.438/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.438)Cited by:[§1](https://arxiv.org/html/2611.18257#S1.p5.1),[§2](https://arxiv.org/html/2611.18257#S2.p2.1)\. - Pintoet al\.\(2026\)A\. S\. Pinto, A\. Steiner, K\. Misiunas, K\. Roth, M\. Tschannen, and O\. SansevieroGemma 4 12b: the developer guide\(Website\)External Links:[Link](https://developers.googleblog.com/gemma-4-12b-the-developer-guide/)Cited by:[§3\.2](https://arxiv.org/html/2611.18257#S3.SS2.p2.1)\. - Qwen Team \(2026\)Qwen TeamQwen3\.5: towards native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§3\.2](https://arxiv.org/html/2611.18257#S3.SS2.p2.1)\. - Sallaukaet al\.\(2025\)R\. Sallauka, U\. Arioz, M\. Rojc, and I\. MlakarWeakly\-supervised multilingual medical ner for symptom extraction for low\-resource languages\.Applied Sciences15\(10\)\.External Links:[Link](https://www.mdpi.com/2076-3417/15/10/5585),ISSN 2076\-3417,[Document](https://dx.doi.org/10.3390/app15105585)Cited by:[§3\.1](https://arxiv.org/html/2611.18257#S3.SS1.SSS0.Px8.p1.1)\. - Spiegelet al\.\(2025\)S\. Spiegel, S\. M\. Yimam, P\. Breitfeld, and F\. ÜckertAdaption and evaluation of generative large language models for German medical information extraction\.InProceedings of the 21st Conference on Natural Language Processing \(KONVENS 2025\): Long and Short Papers,C\. Wartena and U\. Heid \(Eds\.\),Hannover, Germany,pp\. 33–47\.External Links:[Link](https://aclanthology.org/2025.konvens-1.4/)Cited by:[§2](https://arxiv.org/html/2611.18257#S2.p3.1)\. - van der Leket al\.\(2025\)A\. van der Lek, R\. Plüss, and P\. ArneckeProjekt «Named Entity Recognition für die Zentralen Serien des Staatsarchivs Kanton Zürich»\.GitHub\.External Links:[Link](https://github.com/machinelearningZH/named-entity-recognition_staatsarchiv)Cited by:[§3\.1](https://arxiv.org/html/2611.18257#S3.SS1.SSS0.Px9.p1.1)\. - Yanet al\.\(2025\)B\. Yan, K\. Li, M\. Xu, Y\. Dong, Y\. Zhang, Z\. Ren, and X\. ChengOn protecting the data privacy of Large Language Models \(LLMs\) and LLM agents: A literature review\.High\-Confidence Computing5\(2\)\.External Links:ISSN 2667\-2952,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.hcc.2025.100300),[Link](https://www.sciencedirect.com/science/article/pii/S2667295225000042)Cited by:[§1](https://arxiv.org/html/2611.18257#S1.p1.1)\. - Zhanget al\.\(2024\)G\. Zhang, Q\. Jin, Y\. Zhou, S\. Wang, B\. Idnay, Y\. Luo, E\. Park, J\. G\. Nestor, M\. E\. Spotnitz, A\. Soroush, T\. R\. Campion, Z\. Lu, C\. Weng, and Y\. PengClosing the gap between open source and commercial large language models for medical evidence summarization\.npj Digital Medicine7\(1\),pp\. 239\(en\)\.External Links:ISSN 2398\-6352,[Link](https://www.nature.com/articles/s41746-024-01239-w),[Document](https://dx.doi.org/10.1038/s41746-024-01239-w)Cited by:[§1](https://arxiv.org/html/2611.18257#S1.p1.1)\. ## Appendix AAppendix ### A\.1Prompt Template We used the following prompt template in German for all datasets\. > Du bist ein präzises System zur Named Entity Recognition \(NER\) für deutsche Texte\. Deine Aufgabe ist es, strukturierte Entitäten basierend auf den NoSta\-D\-Annotation\-Guidelines aus dem bereitgestellten Text zu extrahieren und ausschließlich als valides JSON\-Objekt zurückzugeben\. \#\#\# Entitätstypen: \* \*\*<tag\-name\>\*\*:<List of types for this tag also including some concrete examples\>\* \*\*location\*\*: Geografische Orte, Länder, Staaten, Städte, Regionen, Straßen, Meere oder Planeten \(z\. B\. "Stuttgart", "Hessen", "Frankreich"\)\. \* … \#\#\# Regeln: 1\. Extrahiere die Wörter exakt so, wie sie im Text stehen\. Modifiziere oder korrigiere die Sätze oder Schreibweisen nicht\. 2\. Wenn eine Entität im Text nicht vorkommt, gib ein leeres Array \[\] für diesen Schlüssel an\. 3\. WICHTIG: Das JSON muss absolut valide sein\. Setze NIEMALS ein Komma nach dem letzten Element \("person\_part"\)\. Vermeide jegliche Syntaxfehler\. \#\#\# Ziel\-Format: “‘json<tag\-name\>:\[<annotated span\>\], "location": \["Wert für location"\], … “‘ \#\#\# Eingabetext: "input\_text" \#\#\# Extrahierte NER\-Tags im JSON\-Format: ## Appendix BMacro F1 Results Table 4:Macro\-averaged entity\-level F1 on the German NER benchmark\. Per dataset, the best score isboldand the second bestunderlined\.*0\-shot*: zero\-shot,*5\-shot*: 5\-shot in\-context,*LoRA*: fine\-tuned\. The final row reports per\-column averages over the datasets, excluding SwissNER, which is only evaluated in the 0\-shot setting due to the small dataset size that does not allow for creating train or validation splits\.Table 5:Partial\-match \(fuzzy\) entity\-level macro F1 on the German NER benchmark, using token\-set\-ratio matching at threshold 0\.90\. Per dataset, bestbold, second bestunderlined; final row = per\-column averages\.
相似文章
RedBench:大型语言模型综合红队测试通用数据集
RedBench 引入了一个通用数据集,聚合了 37 个基准数据集,包含 29,362 个样本,涵盖 22 个风险类别和 19 个领域,用于实现大型语言模型的标准化和综合红队测试评估。该工作解决了现有红队测试数据集中的不一致问题,并提供了基准、评估代码和开源资源,用于评估 LLM 对对抗提示的鲁棒性。
DetectRL-X:面向可靠的多语言及真实世界的LLM生成文本检测
DetectRL-X是一个全面的多语言基准测试,用于评估跨8种语言和6个领域的LLM生成文本检测器,包括针对AI辅助写作操作和扰动的压力测试。它揭示了当前检测器在多语言场景中的优势与局限性。
生物医学命名实体识别与实体链接基准测试究竟衡量什么?一个语料库中心的诊断框架
本文提出了一种以语料库为中心的诊断框架,用于分析生物医学NER和EL基准测试,揭示了九个语料库之间的显著差异,并论证了标准统计量不足以描述评估需求。
KoNeoBench:一个用于评估 LLM 对韩语新词理解的精选评估数据集
本文介绍了 KoNeoBench,这是一个精选的基准数据集,用于评估大型语言模型对韩语新词的理解,基于自2020年以来的1,785个在线新闻条目。它揭示了当前 LLM 在处理近期词汇变化和特定韩语语言特性方面的局限性。
语言特异性与领域多样性:Bangla医疗命名实体识别中的Transformer基准测试
本文评测了Transformer模型在Bangla医疗命名实体识别任务上的表现,结果显示,经过微调的XLM-RoBERTa模型取得了最优性能,超越了语言特定模型,突显了在资源稀缺场景下领域多样性的重要性。