LC-ICL: Label-Guided Contrastive In-Context Learning for Robust Information Extraction

# LC-ICL: Label-Guided Contrastive In-Context Learning for Robust Information Extraction
Source: [https://arxiv.org/html/2606.29407](https://arxiv.org/html/2606.29407)
Xiao You1, Tianwei Yan2, Shan Zhao1 1Hefei University of Technology, Hefei, China 2Chongqing Jiaotong University, Chongqing, China

###### Abstract

There has been increasing interest in exploring the capabilities of advanced large language models \(LLMs\) in the field of information extraction \(IE\), specifically focusing on tasks related to named entity recognition \(NER\) and relation extraction \(RE\)\. Although researchers are exploring the use of few\-shot information extraction through in\-context learning with LLMs, they tend to focus only on using correct or positive examples for demonstration, neglecting the potential value of incorporating incorrect or negative examples into the learning process\. In this paper, we presentLC\-ICL, a novel few\-shot technique that leverages both correct and incorrect sample constructions to create in\-context learning demonstrations\. This approach enhances the ability of LLMs to extract entities and relations by combining positive samples with negative samples annotated by error\-cause labels\. These labels expose more detailed error features in erroneous examples, enabling the model to understand why similar predictions fail and avoid repeating such errors during inference\. Specifically, our proposed method taps into the inherent contextual information and valuable information in hard negative samples and the nearest positive neighbors to the test and then applies the in\-context learning demonstrations based on LLMs\. Our experiments on various datasets indicate thatLC\-ICLoutperforms previous few\-shot in\-context learning methods, delivering substantial enhancements in performance across a broad spectrum of related tasks\. These improvements are noteworthy, showcasing the versatility of our approach in diverse scenarios\.

## IIntroduction

Information extraction \(IE\) is an important task in natural language processing, aiming to obtain structured knowledge from plain text\. It can be applied across different domains, such as knowledge graph construction\[[54](https://arxiv.org/html/2606.29407#bib.bib22)\]and question answering systems\[[2](https://arxiv.org/html/2606.29407#bib.bib23)\]\. With the rise of large language models \(LLMs\)\[[3](https://arxiv.org/html/2606.29407#bib.bib24),[28](https://arxiv.org/html/2606.29407#bib.bib36),[39](https://arxiv.org/html/2606.29407#bib.bib38),[1](https://arxiv.org/html/2606.29407#bib.bib39)\], IE has achieved remarkable progress\[[16](https://arxiv.org/html/2606.29407#bib.bib40),[48](https://arxiv.org/html/2606.29407#bib.bib41)\]\. Recent advances in few\-shot IE have shifted the focus from traditional supervised fine\-tuning methods to leveraging LLMs for in\-context learning \(ICL\) demonstrations\[[4](https://arxiv.org/html/2606.29407#bib.bib46),[25](https://arxiv.org/html/2606.29407#bib.bib47)\]\.

![Refer to caption](https://arxiv.org/html/2606.29407v1/x1.png)Figure 1:Using the relation extraction \(RE\) task as an example, this figure illustrates the core mechanism ofLC\-ICL\. When performing direct inference, large models are prone to generating a large number of errors with specific patterns\.LC\-ICLintroduces an error labeling system that feeds back error types to the model, guiding it to refine its generation, thereby significantly reducing errors and improving inference accuracy\.Previous work\[[47](https://arxiv.org/html/2606.29407#bib.bib44),[4](https://arxiv.org/html/2606.29407#bib.bib46),[26](https://arxiv.org/html/2606.29407#bib.bib49),[40](https://arxiv.org/html/2606.29407#bib.bib50)\]has explored using natural language prompts or in\-context learning \(ICL\) demonstrations to guide LLMs in annotating test data under few\-shot settings, sometimes requiring additional pretraining or fine\-tuning steps\. To better align with the structured nature of information extraction tasks, more recent approaches\[[17](https://arxiv.org/html/2606.29407#bib.bib48),[35](https://arxiv.org/html/2606.29407#bib.bib51),[44](https://arxiv.org/html/2606.29407#bib.bib45),[45](https://arxiv.org/html/2606.29407#bib.bib57),[42](https://arxiv.org/html/2606.29407#bib.bib54)\]employ code\-like or structured prompts to enhance the consistency between pretraining and inference\. However, these methods have yet to fully unlock the potential of LLMs, partly because the models rely on limited positive data and cannot learn from their own errors\.

To address this issue, this paper proposes a contrastive in\-context learning approach that leverages both positive and negative examples to extend the learning process of LLMs, thereby exposing them to a wider range of scenarios, including typical errors\. The method is designed to exploit the value of often overlooked negative data, enabling more comprehensive and robust information extraction capabilities\.

Assume that the model has already learned task execution and problem\-solving patterns from the positive information extraction dataset, yet its predictions still contain errors\. In this case, the model should reflect on the causes of these errors, categorize the error types, and attempt to avoid them in subsequent reasoning\. Therefore, incorporating information related to negative samples can help address this issue\. Inspired by this idea, this paper integrates both correct/positive and incorrect/negative examples into ICL demonstrations to enhance the information extraction performance of in\-context learning\.

Specifically, we first employ a large\-scale model to generate labels for the annotated data in order to select hard negative samples\. Then, we select semantically similar positive samples from the training data for the current test instance, and design the most suitable in\-context demonstrations using different models \(natural language LLMs or code LLMs\)\. In the module for selecting error/negative samples that contain richer knowledge, we adopt a semantic\-similarity\-aware retrieval approach for ranking\.

To demonstrate the advantages of the proposed method, we conduct experiments on three named entity recognition \(NER\) and five relation extraction \(RE\) benchmark datasets, followed by a thorough analysis of the benefits brought by this approach\.

The main contributions of this paper are summarized as follows:

- •We proposeLC\-ICL, a contrastive in\-context learning approach that incorporates positive demonstrations and negative samples annotated with error\-cause labels\. These labeled negative samples provide more detailed error features, helping LLMs recognize failure patterns and avoid similar mistakes during information extraction\.
- •We design an effective retrieval strategy to select hard negative samples as part of in\-context learning, leveraging them to further strengthen information extraction capabilities\.
- •We conduct extensive experiments on benchmark datasets, demonstrating the effectiveness and broad applicability of the proposed method across NER and RE tasks\.

## IITask Formulation

Given a sentenceXXcontaininglltokensx1,x2,⋯,xlx\_\{1\},x\_\{2\},\\cdots,x\_\{l\}, the goal of IE tasks is to predict a structured outputYY\(named entities or relations\) fromxx\. In the NER task, the targetYYis entity spans with entity typesE​\(\(e,t\)\|xi,…,xj\)E\(\(e,t\)\|x\_\{i\},\\dots,x\_\{j\}\), whereeeis an entity in the sequence, andttis an entity type from a predefined set of entity types𝒯\\mathcal\{T\}\(e\.g\.,LOC,PER,ORG\)\.

In the RE task, the targetYYis a set of relations between entities, typically expressed as triples\(e1,r,e2\)\(e\_\{1\},r,e\_\{2\}\)\. This not only involves predicting the relationr∈ℛr\\in\\mathcal\{R\}but also includes the typest1t\_\{1\}andt2t\_\{2\}of entitiese1e\_\{1\}ande2e\_\{2\}, whereℛ\\mathcal\{R\}represents relation types \(e\.g\.,Work For,Live In,Located In\)\. The types of entitiese1e\_\{1\}ande2e\_\{2\}also need to be predicted, wheret∈𝒯t\\in\\mathcal\{T\}represents entity types\.

We formulate IE as a generation task, prompting large language models to perform inference for NER or RE\.

In the few\-shot in\-context learning setting for information extraction, the demonstrations are structured into four components: \(1\) negative examples, \(2\) positive examples, \(3\) instructions, and \(4\) test text\. The output of LLMs is a list of tuples, which in the NER task is\[\(e1,t1\),…,\(ej,tj\)\]\[\(e\_\{1\},t\_\{1\}\),\\dots,\(e\_\{j\},t\_\{j\}\)\], such as \[\(’Steve’, ’person’\)\]\.

![Refer to caption](https://arxiv.org/html/2606.29407v1/x2.png)Figure 2:The figure illustrates an overview of theLC\-ICLframework for information extraction tasks, using the Named Entity Recognition \(NER\) task as an example\. The method constructs positive and negative examples, which, along with the test samples and task instructions, are input into the large language model to generate predictions\. Steps 1–4 represent the process of obtaining labeled negative samples based on retrieved examples\.
## IIILC\-ICL

### III\-AModel Overview

As shown in Figure[2](https://arxiv.org/html/2606.29407#S2.F2), in the context examples, our method mainly consists of 4 parts: Instruction part, Test sentence part, Positive Samples part, and Negative Samples part\.

Unlike prior methods that rely exclusively on positive samples for in\-context learning, our approach,LC\-ICL, leverages both positive examples and negative instances annotated with explicit reasoning errors\. For the selection of positive samples, we employ different retrieval strategies to select samples from the training set that correspond to the test examples\. For negative sample selection, we first sample a subset of data from the training dataset of each dataset to form a negative sample collection\. Then,MinferM\_\{\\mathrm\{infer\}\}generates predictions for the data in the negative sample collection, andMlabelM\_\{\\mathrm\{label\}\}compares each prediction with its gold label to assign error labels, forming the final negative sample pool\. Finally, we also use different retrieval strategies to retrieve a negative sample from the negative sample pool that corresponds to the test sample\. Combined with the previously selected positive samples, we use both positive and negative samples together as examples for ICL to the large model\. The large model will learn correct patterns from positive examples, thereby improving accuracy, while negative samples with error labels will inform the model about similar error patterns, helping it avoid similar mistakes and reduce error rates\.

### III\-BLC\-ICL Demonstrations Construction

We construct a prompt for each given test sentence, and input it into the LLM\. Each prompt contains the following components:

Test SentenceXt\{X\_\{t\}\}represents the original text of the RE/NER test sample, which is used for the final evaluation of the large model’s performance in executing RE/NER tasks under the specified method\.

Instructionℐ\\mathcal\{I\}We constrain the large model to different expert roles according to different tasks, and based on the task differences, provide the relation setℛ\\mathcal\{R\}or entity type set𝒯\\mathcal\{T\}\. We use brief commands to require the large model to complete RE/NER tasks, and need it to return responses in a fixed list format\. The instruction further guides the model to attend to negative samples with error\-cause labels, so that it can acquire fine\-grained error\-feature signals from erroneous examples and avoid making similar mistakes on the test sentence\.

Positive Samples𝒫\\mathcal\{P\}Referring to previous work by\[[17](https://arxiv.org/html/2606.29407#bib.bib48)\], we use the KNN algorithm \(k\-nearest neighbors algorithm\) to select examples from the training set that have the highest semantic similarity with the test sampleXtX\_\{t\}in the embedding space, to serve as positive examples𝒫\\mathcal\{P\}\.

Negative Samples𝒩\\mathcal\{N\}LetMinferM\_\{\\mathrm\{infer\}\}denote the inference model that produces predictions, and letMlabelM\_\{\\mathrm\{label\}\}denote the label\-construction model that assigns error labels by comparing each prediction with its gold label\.

𝒳pred′=Minfer​\(ℐ,𝒳t′\)\\mathcal\{X\}^\{\\prime\}\_\{\\mathrm\{pred\}\}=M\_\{\\mathrm\{infer\}\}\(\\mathcal\{I\},\\mathcal\{X\}^\{\\prime\}\_\{t\}\)\(1\)𝒩label=Mlabel​\(ℐ,E,𝒳t′,𝒳pred′,𝒳g′\)\\mathcal\{N\}\_\{\\mathrm\{label\}\}=M\_\{\\mathrm\{label\}\}\(\\mathcal\{I\},E,\\mathcal\{X\}^\{\\prime\}\_\{t\},\\mathcal\{X\}^\{\\prime\}\_\{\\mathrm\{pred\}\},\\mathcal\{X\}^\{\\prime\}\_\{g\}\)\(2\)𝒩=\{\(𝒳t′,𝒳pred′,𝒳g′,𝒩label\)\}\\mathcal\{N\}=\\\{\(\\mathcal\{X\}^\{\\prime\}\_\{t\},\\mathcal\{X\}^\{\\prime\}\_\{\\mathrm\{pred\}\},\\mathcal\{X\}^\{\\prime\}\_\{g\},\\mathcal\{N\}\_\{\\mathrm\{label\}\}\)\\\}\(3\)Following the approach in previous work\[[17](https://arxiv.org/html/2606.29407#bib.bib48)\], we first sample 2500 labeled examples from the training set and useMinferM\_\{\\mathrm\{infer\}\}to generate predictions for these samples, as shown in Equation[1](https://arxiv.org/html/2606.29407#S3.E1)\. Here,𝒳t′\\mathcal\{X\}^\{\\prime\}\_\{t\}represents the sampled input examples,𝒳g′\\mathcal\{X\}^\{\\prime\}\_\{g\}represents their corresponding gold labels, and𝒳pred′\\mathcal\{X\}^\{\\prime\}\_\{\\mathrm\{pred\}\}represents the predictions produced byMinferM\_\{\\mathrm\{infer\}\}\. Next, we provide the input examples, predictions, and gold labels toMlabelM\_\{\\mathrm\{label\}\}to annotate incorrect predictions with error labels, as shown in Equation[2](https://arxiv.org/html/2606.29407#S3.E2)\. Here,EErepresents the set of error labels, and𝒩label\\mathcal\{N\}\_\{\\mathrm\{label\}\}represents the generated error label set\. Finally, using the same method used to filter positive samples, we employ the KNN algorithm to select negative samples corresponding to the test samples, and store𝒳t′\\mathcal\{X\}^\{\\prime\}\_\{t\},𝒳pred′\\mathcal\{X\}^\{\\prime\}\_\{\\mathrm\{pred\}\},𝒳g′\\mathcal\{X\}^\{\\prime\}\_\{g\}, and𝒩label\\mathcal\{N\}\_\{\\mathrm\{label\}\}together in the negative sample pool, as shown in Equation[3](https://arxiv.org/html/2606.29407#S3.E3)\.

Yt=Minfer​\(𝒫,𝒩,ℐ,Xt\)Y\_\{t\}=M\_\{\\mathrm\{infer\}\}\(\\mathcal\{P\},\\mathcal\{N\},\\mathcal\{I\},X\_\{t\}\)\(4\)Overall, the input and output of our proposed framework can be represented by formula[4](https://arxiv.org/html/2606.29407#S3.E4), whereYtY\_\{t\}represents the inference result produced byMinferM\_\{\\mathrm\{infer\}\}for the test sampleXtX\_\{t\}\.

ERE=\{E1,E2,E3,E4,E5,E6\}E\_\{\\mathrm\{RE\}\}=\\\{E\_\{1\},E\_\{2\},E\_\{3\},E\_\{4\},E\_\{5\},E\_\{6\}\\\}\(5\)ENER=\{E1′,E2′,E3′,E4′,E5′\}E\_\{\\mathrm\{NER\}\}=\\\{E\_\{1\}^\{\\prime\},E\_\{2\}^\{\\prime\},E\_\{3\}^\{\\prime\},E\_\{4\}^\{\\prime\},E\_\{5\}^\{\\prime\}\\\}\(6\)ERE∪ENER⊆EE\_\{\\mathrm\{RE\}\}\\cup E\_\{\\mathrm\{NER\}\}\\subseteq E\(7\)
We carefully analyze the causes of reasoning errors and formulate detailed error labels for RE and NER tasks respectively\. Errors in RE are categorized into 6 types, as shown in Equation[5](https://arxiv.org/html/2606.29407#S3.E5), while errors in NER are divided into 5 types, as shown in Equation[6](https://arxiv.org/html/2606.29407#S3.E6)\. In Equation[7](https://arxiv.org/html/2606.29407#S3.E7),EREE\_\{\\mathrm\{RE\}\}andENERE\_\{\\mathrm\{NER\}\}together constituteEE\.

The specific label types and their meanings in Equations[5](https://arxiv.org/html/2606.29407#S3.E5)and[6](https://arxiv.org/html/2606.29407#S3.E6)can be found in Appendix[B](https://arxiv.org/html/2606.29407#A2)\.

### III\-CLC\-ICL Retrieval Strategy

#### III\-C1KNN\-Based Retrieval

In few\-shot learning, selecting demonstrations that are semantically similar to the test sample is crucial\[[21](https://arxiv.org/html/2606.29407#bib.bib58)\]\. Research has shown that using KNN methods to retrieve the most similar examples from the training set can improve model performance\[[14](https://arxiv.org/html/2606.29407#bib.bib59),[21](https://arxiv.org/html/2606.29407#bib.bib58),[12](https://arxiv.org/html/2606.29407#bib.bib56)\]\. KNN relies on effective embedding spaces to encode inputs and examples; therefore, pre\-trained language models or improved sentence embedding methods have been proposed to optimize sentence representations\.

For information extraction tasks, we also adopt a retrieval method based on sentence embeddings, utilizing the k\-nearest neighbor algorithm to filter out demonstration sentences from the training set that are relevant to the target task\. By calculating semantic similarity, we select the top k sentences containing entities or relations, and use LLMs combined with cosine similarity for matching, thereby providing higher quality demonstrations for in\-context learning\.

#### III\-C2Error Label Retrieval

Algorithm 1Negative Sample Generation and Retrieval Algorithm0:Test sample

XtX\_\{t\}; labeled training dataset

𝒟train\\mathcal\{D\}\_\{\\text\{train\}\}; inference model

MinferM\_\{\\mathrm\{infer\}\}; label\-construction model

MlabelM\_\{\\mathrm\{label\}\}; error label set

EE; sample size

k=2500k=2500\.

0:Retrieved negative sample

𝒩\\mathcal\{N\}for test sample

XtX\_\{t\}
1:/\* Phase 1: Generate negative sample pool \*/

2:

𝒳′←Sample​\(𝒟train,k\)\\mathcal\{X\}^\{\\prime\}\\leftarrow\\text\{Sample\}\(\\mathcal\{D\}\_\{\\text\{train\}\},k\)\{Sample

kklabeled instances\}

3:Initialize negative sample pool

𝒮←∅\\mathcal\{S\}\\leftarrow\\emptyset
4:foreach labeled sample

\(xt′,xg′\)∈𝒳′\(x^\{\\prime\}\_\{t\},x^\{\\prime\}\_\{g\}\)\\in\\mathcal\{X\}^\{\\prime\}do

5:

xpred′←Minfer​\(ℐ,xt′\)x^\{\\prime\}\_\{\\mathrm\{pred\}\}\\leftarrow M\_\{\\mathrm\{infer\}\}\(\\mathcal\{I\},x^\{\\prime\}\_\{t\}\)\{Generate prediction as in Eq\.[1](https://arxiv.org/html/2606.29407#S3.E1)\}

6:

𝒩label←Mlabel​\(ℐ,E,xt′,xpred′,xg′\)\\mathcal\{N\}\_\{\\text\{label\}\}\\leftarrow M\_\{\\mathrm\{label\}\}\(\\mathcal\{I\},E,x^\{\\prime\}\_\{t\},x^\{\\prime\}\_\{\\mathrm\{pred\}\},x^\{\\prime\}\_\{g\}\)\{Generate error labels using the gold label\}

7:

𝒮←𝒮∪\{\(xt′,xpred′,xg′,𝒩label\)\}\\mathcal\{S\}\\leftarrow\\mathcal\{S\}\\cup\\\{\(x^\{\\prime\}\_\{t\},x^\{\\prime\}\_\{\\mathrm\{pred\}\},x^\{\\prime\}\_\{g\},\\mathcal\{N\}\_\{\\text\{label\}\}\)\\\}
8:endfor

9:/\* Phase 2: Retrieve relevant negative sample for test instance \*/

10:Compute semantic embedding for

XtX\_\{t\}and all samples in

𝒮\\mathcal\{S\}
11:

𝒩←KNNRetrieval​\(Xt,𝒮\)\\mathcal\{N\}\\leftarrow\\text\{KNNRetrieval\}\(X\_\{t\},\\mathcal\{S\}\)\{Retrieve most similar negative sample\}

12:return

𝒩\\mathcal\{N\}

Inspired by\[[9](https://arxiv.org/html/2606.29407#bib.bib12)\], in addition to using positive examples to help large models recognize correct answers, negative examples also serve an instructive role, helping models correct similar errors\. Mo et al\.\[[29](https://arxiv.org/html/2606.29407#bib.bib4)\]identified hard negative examples by querying the model multiple times and selecting high\-confidence erroneous outputs\. However, repeatedly calling the large model to obtain hard negative samples consumes significant time and incurs expensive costs\. Therefore, we propose an error label retrieval strategy\. We conducted a detailed analysis of the reasons for large model inference errors on NER/RE datasets and formulated detailed error labels based on these reasons, as shown in Algorithm[1](https://arxiv.org/html/2606.29407#alg1)\. After a single forward pass on the sampled training data usingMinferM\_\{\\mathrm\{infer\}\}, we provide each input, its prediction, and its gold label toMlabelM\_\{\\mathrm\{label\}\}to obtain error labels\. The resulting negative sample pool stores the input, prediction, gold label, and error labels together\. Finally, we use the kNN algorithm to select semantically similar labeled error samples as negative examples for test samples\.

## IVExperiments

### IV\-ADatasets

RE DatasetsFor relation extraction, we evaluate on datasets CoNLL04\[[33](https://arxiv.org/html/2606.29407#bib.bib75)\], NYT10\[[32](https://arxiv.org/html/2606.29407#bib.bib8)\],NYT11\[[37](https://arxiv.org/html/2606.29407#bib.bib9)\],SciERC\[[24](https://arxiv.org/html/2606.29407#bib.bib76)\]and ADE\[[13](https://arxiv.org/html/2606.29407#bib.bib10)\]\. We adopt the dataset splits from prior UIE work\[[23](https://arxiv.org/html/2606.29407#bib.bib43)\]for all these datasets\.

NER DatasetsWe evaluate our approach on NER task with ACE04\[[6](https://arxiv.org/html/2606.29407#bib.bib73)\], ACE05\[[41](https://arxiv.org/html/2606.29407#bib.bib74)\]and NCBI\[[7](https://arxiv.org/html/2606.29407#bib.bib7)\]\. and we split the datasets followed by the works\[[18](https://arxiv.org/html/2606.29407#bib.bib62),[30](https://arxiv.org/html/2606.29407#bib.bib63),[31](https://arxiv.org/html/2606.29407#bib.bib71),[17](https://arxiv.org/html/2606.29407#bib.bib48)\]\. Table[III](https://arxiv.org/html/2606.29407#A1.T3)shows the dataset statistics in Appendix[A](https://arxiv.org/html/2606.29407#A1)\.

### IV\-BExperiments Setting

We use Llama\[[11](https://arxiv.org/html/2606.29407#bib.bib6)\]as the backbone forMinferM\_\{\\mathrm\{infer\}\}on test sets and DeepSeek\[[20](https://arxiv.org/html/2606.29407#bib.bib5)\]as the backbone forMlabelM\_\{\\mathrm\{label\}\}when constructing error labels for negative samples\.

For the IE task, we construct both positive and negative samples to serve as contextual prompts\. For positive sample selection, we employ a KNN retrieval strategy to select the most matching positive sample from the training set for each test sample\. For negative samples, we first perform inference on the sampled training instances usingMinferM\_\{\\mathrm\{infer\}\}, then present each input, its prediction, and its gold label toMlabelM\_\{\\mathrm\{label\}\}to obtain error labels, forming a negative sample pool that stores the gold label together with the prediction and error labels\. Finally, using the same KNN retrieval strategy, we retrieve the most similar sample from the negative sample pool as the negative sample for each test case\. Due to the large number of samples in the training datasets, constructing error labels for all training data would be costly\. Following\[[17](https://arxiv.org/html/2606.29407#bib.bib48)\], we randomly sample 2,500 examples from each training dataset to build the negative sample pool for subsequent negative sample retrieval\.

Detailed experimental settings are provided in Appendix[A](https://arxiv.org/html/2606.29407#A1)\.

### IV\-CEvaluation

Following previous work by\[[23](https://arxiv.org/html/2606.29407#bib.bib43),[17](https://arxiv.org/html/2606.29407#bib.bib48)\], we employ a strict evaluation metric for both NER and RE F1 scores\. In the NER datasets, a prediction is considered correct only when both the entity name and type are correct\. Similarly, in the RE datasets, a prediction is counted as correct only when both the entities and their relationship are accurately identified\. Additionally, we introduce an instruction\-following metric, which allows us to observe whether errors in model reasoning stem from poor instruction adherence or from difficulties in the task execution itself\. To ensure the reliability of our results, we conducted three rounds of experiments with three different random seeds and reported the average scores\.

### IV\-DResults

We investigate the performance of our proposed method in comparison to traditional in\-context learning approaches and direct inference on NER and RE tasks\. In addition, we conduct experiments with two model scales, Llama\-3\.1\-8B\-Instruct and Llama\-3\.3\-70B\-Instruct, to further examine the impact of model size on performance\.

TABLE I:Performance comparison on RE benchmarks in terms of F1 score \(%\)\. ICL denotes the model performing in\-context learning without parameter updates\. PE refers to using only positive examples, which is the traditional ICL, while LC\-ICL, our method, includes both positive and negative examples\.ModelParadigmBackboneRECoNLL04NYT10NYT11SciERCADEDIw/o ICLLlama\-3\.1\-8B\-Instruct0\.350\.270\.090\.00111Repeated verification confirms that the model consistently returns empty results on this dataset, resulting in an F1 score of 0%\.0\.79PEICLLlama\-3\.1\-8B\-Instruct15\.1115\.738\.282\.0213\.60LC\-ICL \(ours\)ICLLlama\-3\.1\-8B\-Instruct32\.2738\.6921\.524\.1818\.10DIw/o ICLLlama\-3\.3\-70B\-Instruct7\.026\.336\.612\.780\.68PEICLLlama\-3\.3\-70B\-Instruct48\.4227\.7723\.019\.3549\.62LC\-ICL \(ours\)ICLLlama\-3\.3\-70B\-Instruct50\.7142\.0434\.1512\.2154\.13

TABLE II:Performance comparison on NER benchmarks in terms of F1 score \(%\)\. w/o ICL represents zero\-shot performance, while ICL denotes in\-context learning\. DI refers to direct inference, contrasting with PE, which uses only positive examples as in traditional ICL\. Our method, LC\-ICL, incorporates both positive and negative examples\.ModelParadigmBackboneNERACE04ACE05NCBIDIw/o ICLLlama\-3\.1\-8B\-Instruct11\.0116\.8839\.40PEICLLlama\-3\.1\-8B\-Instruct14\.7224\.5933\.30LC\-ICL \(ours\)ICLLlama\-3\.1\-8B\-Instruct21\.3625\.7634\.36DIw/o ICLLlama\-3\.3\-70B\-Instruct24\.6624\.0740\.44PEICLLlama\-3\.3\-70B\-Instruct37\.1838\.9938\.20LC\-ICL \(ours\)ICLLlama\-3\.3\-70B\-Instruct44\.5344\.6456\.36

RE ResultsTable[I](https://arxiv.org/html/2606.29407#S4.T1)shows the results of the RE task\.

Overall, our method and the traditional method that only uses positive examples as ICL prompts \(PE\) both significantly outperform the direct inference method \(DI\), demonstrating that effective contextual examples can significantly enhance the model’s ability to complete RE tasks\. On the same Llama\-3\.1\-8B\-Instruct base model, our method outperforms the PE method because our approach not only uses positive examples to guide the large model in identifying similar patterns but also informs the model about potential errors\. Through error label information, it avoids the hallucination content that might be produced when merely imitating positive example patterns using only positive samples\. Furthermore, as the model parameter size increases, our method remains effective\. On the Llama\-3\.3\-70B\-Instruct model, our method shows improvement over the ICL method across all RE datasets, further proving that our approach is effective not only on small\-parameter models but also on large\-parameter models\.

On the Llama\-3\.1\-8B\-Instruct model, we conducted a detailed comparison between the PE method and our proposed LC\-ICL method\. On the NYT10 dataset, the traditional ICL method achieved an F1 score of only 15\.73%, while our method directly improved it to38\.69%, representing a relative improvement of 146%\. On the NYT11 dataset, the LC\-ICL method \(F1=21\.52%\) showed a 159\.8% improvement compared to the PE method \(F1=8\.28%\), demonstrating our method’s significant advantage on the NYT series datasets\. On the SciERC scientific literature relation extraction task, the PE method performed extremely poorly with an F1 score of 2\.02%, while our method improved it to4\.18%\. Although the absolute value is not high, the improvement exceeds 100%, indicating that our Label design remains effective even in extremely low\-resource scenarios\. On the ADE medical relation extraction task, the PE method achieved an F1 score of 13\.60%, while our method improved it to18\.10%, demonstrating stronger relation modeling capabilities in the medical domain\. On the CoNLL04 task, our method achieved an F1 score of32\.27%, a 113\.7% improvement compared to PE \(15\.11%\), fully validating the applicability and effectiveness of the LC\-ICL strategy on classic small\-scale RE datasets\.

On the Llama\-3\.3\-70B\-Instruct model, we also compared the PE method with our proposed LC\-ICL method\. On the NYT10 dataset, the PE method achieved an F1 score of 27\.77%, while our method directly improved to42\.04%, representing a relative improvement of 51\.4%\. On the NYT11 dataset, the LC\-ICL method \(F1=34\.15%\) showed a 48\.4% improvement compared to the PE method \(F1=23\.01%\), continuing to maintain significant advantages on the NYT series datasets\. On the SciERC scientific literature relation extraction task, the PE method achieved an F1 score of 9\.35%, showing relatively weak performance, while the LC\-ICL method improved to12\.21%, a relative improvement of 30\.6%, indicating that the LC\-ICL design is also effective on specialized domain texts\. On the ADE medical relation extraction task, the PE method achieved an F1 score of 49\.62%, while the LC\-ICL method improved to54\.13%, a relative improvement of 9\.1%, demonstrating stronger relation modeling capabilities in the medical domain\. On the CoNLL04 task, the LC\-ICL method achieved an F1 score of50\.71%, showing a 4\.7% improvement compared to PE \(48\.42%\)\. Although the improvement margin is relatively small, it further verifies the generalization ability of the LC\-ICL strategy on large\-scale models\. Overall, the experimental results on the 70B model fully demonstrate that even on larger\-scale models, the LC\-ICL method incorporating negative examples can still bring significant performance improvements\.

NER ResultsTable[II](https://arxiv.org/html/2606.29407#S4.T2)shows the results of the NER task\.

Similarly, in the NER task, our method and the PE method mostly outperform direct inference approaches\. Compared with PE, LC\-ICL generally achieves better results, suggesting that incorporating incorrectly predicted entities and their error\-type labels as negative samples can help large language models avoid similar entity recognition errors\. However, the gains are not uniform across all baselines and datasets; for example, direct inference remains competitive on the NCBI dataset with Llama\-3\.1\-8B\-Instruct\.

Similarly, on the Llama\-3\.1\-8B\-Instruct model, we conducted a detailed comparison between the PE method and our approach\. In the ACE04 dataset, the PE method only achieved an F1 score of 14\.72%, while the LC\-ICL method significantly improved to21\.36%, representing a 45\.12% increase\. In the ACE05 dataset, the PE method scored 24\.59%, with LC\-ICL improving to25\.76%\. Although the improvement margin was limited, it still demonstrates the robustness of the LC\-ICL strategy across different datasets\. For the NCBI disease entity recognition task, the PE method reached 33\.30%, while LC\-ICL improved further to34\.36%\. This result indicates that LC\-ICL improves over PE on this medical\-domain task, although direct inference remains a strong baseline on NCBI\.

In the Llama\-3\.3\-70B\-Instruct model, our LC\-ICL method demonstrates significant performance advantages\. The experimental results show that our proposed LC\-ICL method achieves F1 scores of44\.53%,44\.64%, and56\.36%in the three datasets, comprehensively outperforming other methods\. Compared to PE, LC\-ICL improves by 7\.35 percentage points \(a relative increase of 19\.77%\) on the first dataset, 5\.65 percentage points \(a relative increase of 14\.49%\) on the second dataset, and an impressive 18\.16 percentage points \(a relative increase of 47\.54%\) on the third dataset\. By comparing performance across different datasets, we find that the LC\-ICL method exhibits strong robustness and adaptability, consistently maintaining its advantage on data with varying characteristics\. These results convincingly demonstrate that our method’s design philosophy effectively overcomes the limitations of traditional ICL methods, providing a more efficient paradigm for applying large language models to entity recognition tasks\. Particularly on large\-scale models like the 70B model, the LC\-ICL method shows even more pronounced performance gains, indicating its ability to better leverage potential as model scale increases\.

## VFurther Analysis

### V\-AAblation Study

![Refer to caption](https://arxiv.org/html/2606.29407v1/figures/icl_explanation_label_RE.png)\(a\)RE Task
![Refer to caption](https://arxiv.org/html/2606.29407v1/figures/icl_explanation_label_NER.png)\(b\)NER Task

Figure 3:Comparative experimental results based on NER and RE tasks, evaluating the effectiveness of two negative sample identification methods: Label annotation and natural language explanation \(Explanation\)\. Here, Explanation refers to summarizing error reasons in the form of a natural language sentence, while Label refers to annotating error types in a structured label format \(the method adopted in this paper\)\.To investigate the impact of structured features in mislabeled negative samples on experimental results, we conduct a comparative study between two types of negative samples: those annotated with structured labels \(denoted asLabel\) and those annotated with a one\-sentence explanation of the error reason generated by a large language model \(denoted asExplanation\)\. Experiments are conducted on both Named Entity Recognition \(NER\) and Relation Extraction \(RE\) datasets\.

In our approach, we categorize and label model inference errors in detail, and reintroduce them as negative samples in In\-Context Learning \(ICL\) demonstrations\. This is intended to enhance the model’s ability to recognize and correct error patterns\. In contrast, the baselineExplanationmethod provides only a single\-sentence summary of the error cause, without distinguishing between specific error types\.

In most cases, both theExplanationandLabelmethods can enhance the information extraction capabilities of large language models\. This can be attributed to the fact that negative samples provide certain error patterns, which help the model avoid similar mistakes and thereby reduce the overall error rate\. However, on the ACE05 and NCBI datasets, theICLmethod outperforms the Explanation approach\. This may be because, for relatively simple NER tasks, the longer explanations generated by the Explanation method could increase the risk of hallucinations in large models, ultimately decreasing accuracy\. Overall, theLabelmethod consistently outperforms both the Explanation and ICL approaches, which strongly supports the effectiveness of our label design\. In the following sections, we provide a detailed analysis of the Explanation and Label methods on each dataset\.

##### Performance on NER Datasets

Experimental results on three named entity recognition datasets—ACE04, ACE05, and NCBI—demonstrate that incorporating fine\-grained label information consistently improves model inference performance to varying degrees\. In the ACE04 dataset, the label\-based approach yields a slight improvement \(\+0\.28 percentage points\)\. However, more substantial gains are observed on ACE05 and NCBI, with performance increases of \+2\.13 and \+2\.29 percentage points, respectively\. These findings suggest that for NER tasks with complex structures or diverse entity types, detailed error annotations are more effective in helping large language models understand and correct their own inference biases\. This effect is particularly pronounced in medical\-domain datasets such as NCBI\.

##### Performance on RE Datasets

Except for the ADE dataset, all other RE datasets benefit significantly from the labeling approach, especially CoNLL04, NYT10, and NYT11, with F1 score improvements of \+12\.17, \+9\.42, and \+7\.56 percentage points, respectively\. These results suggest that error types in relation extraction tasks are more diverse and complex, making it difficult for simple error summaries to effectively guide model improvements\. In contrast, explicit error category labels help the model capture more fine\-grained relational patterns, thereby significantly enhancing its error correction capability\.

It is worth noting that on the ADE dataset, the labeling approach actually led to a performance drop of 2\.62 percentage points\. Upon deeper analysis, we find that the relation types in ADE are relatively homogeneous, primarily consisting of a single type: Drug\-Adverse Effect\. The task itself exhibits a simple structure, and the sources of errors are relatively concentrated\. Under such circumstances, the multi\-class error labels designed in the labeling method have limited utility\. In fact, due to the insufficient number of error categories, the overall information content may even be reduced, making the method less effective than theExplanationapproach, which directly provides a concise summary of the error\.

The one\-sentence explanation offered by theExplanationmethod is able to guide the model’s attention toward the core mistake in a direct manner under the simple scenario of ADE, significantly reducing redundant information and thereby improving inference accuracy\.

This observation suggests that the effectiveness of error label design in relation extraction tasks is closely tied to the complexity of the task\. For tasks with a single relation type and highly concentrated error patterns, concise and direct error explanations offer greater advantages\. In contrast, for tasks involving diverse relation types and complex error patterns, fine\-grained error labels are more beneficial in helping the model effectively reflect and optimize its reasoning process\.

### V\-BAnalysis of Label Definition Validity

![Refer to caption](https://arxiv.org/html/2606.29407v1/figures/icl_label_def_label_RE.png)\(a\)RE Task
![Refer to caption](https://arxiv.org/html/2606.29407v1/figures/icl_label_def_label_NER.png)\(b\)NER Task

Figure 4:Comparison of experimental results under two settings: using only labels \(Label\) versus using labels accompanied by definitional explanations \(Label\_Def\)\. The results are evaluated on both NER and RE datasets\. Here, Label\_Def refers to the setting where definitional explanations are provided for each label, while Label represents the setting where only the label itself is provided \(which is the approach adopted in this paper\)\.To verify the impact of error\-labeled in\-context learning \(ICL\) examples on the inference capability of large language models, we designed two strategies for comparison:LabelandLabel\_Def\. TheLabelstrategy marks the specific type of error for each incorrect sample, while theLabel\_Defstrategy further provides a detailed explanation of the meaning of each label on top of theLabelstrategy\.

Experiments were conducted on both Named Entity Recognition \(NER\) and Relation Extraction \(RE\) tasks, with F1 score used as the evaluation metric\. The experimental results are summarized as follows\.

Overall, bothLabelandLabel\_Defstrategies consistently improve the F1 scores of large language models on information extraction tasks across most datasets\. This suggests that providing suitable negative samples enables large models to learn certain error patterns and avoid making the same mistakes\. Detailed analysis for each dataset is presented below\.

##### NER

From the results of the NER task, theLabelmethod consistently outperforms theLabel\_Defmethod across all datasets, with an average F1 score improvement of over 4 percentage points, demonstrating a clear advantage\. Upon analysis, this can be attributed to the fact that errors in entity boundary recognition and type classification in NER are highly categorizable\. By directly using concise error labels, large language models are provided with clear correction directions\. In contrast, theLabel\_Defmethod introduces label definitions that may contain redundant information, potentially distracting the model from focusing on the core error categories, thereby weakening the effectiveness of inference enhancement\.

Moreover, named entity recognition inherently emphasizes fine\-grained boundary and type discrimination, where error types are relatively explicit\. In this context, lengthy explanations offer limited benefits and may even lead the model to over\-attend to irrelevant details, ultimately degrading performance\.

##### RE

In the relation extraction task, the Label method also achieves better performance across all datasets, with a particularly significant improvement in F1 score\. This improvement is especially notable in complex relation scenarios such as NYT10 and NYT11, where the increase reaches \+8\.28 and \+5\.24 percentage points, respectively\. This result indicates that the error patterns in relation extraction are complex and diverse\. Directly providing structured error labels helps the model effectively align with error categories, enhancing the model’s error correction and inference capabilities\.

In contrast, the Label\_Def method, which introduces label explanations, did not bring the expected benefits in complex scenarios\. Instead, it may have hindered the model’s ability to focus on core error categories due to lengthy descriptions that distracted attention\. Particularly in cross\-domain, multi\-relation datasets like the NYT series and SciERC,concise and clear error labels are more effective in helping the model extract useful patterns than lengthy explanations\.

The performance on the ADE dataset remained relatively stable, with the Label method bringing only a slight improvement\. Considering that the ADE relation type is singular \(drug\-adverse reaction\), errors are more concentrated, and the marginal effect of label explanations is smaller, reflecting the close relationship between label design and task complexity\.

### V\-CValidity of Negative Samples

![Refer to caption](https://arxiv.org/html/2606.29407v1/figures/random_ACE04_CoNLL04.png)Figure 5:Comparison of RE and NER task performance of various methods under random retrieval strategy\. We selected the ACE04 and CoNLL04 datasets to evaluate the performance of different methods\. w/o\_ICL indicates that ICL is not used, and the large model performs zero\-shot inference directly on the test text\.We employed a random retrieval strategy on the ACE04 and CoNLL04 datasets to observe the comparison between our method and others under random retrieval\.

To further validate the robustness and generalization ability of our proposed LC\-ICL method, we designed a random retrieval \(Random Retrieval\) scenario, where we randomly select examples from the training set as support samples to replace similarity\-based retrieval methods\. This was done to evaluate the performance differences of various methods without effective retrieval support\.

As shown in the figure, under the random retrieval strategy, our method still demonstrates significant advantages\. In particular, the label\-based method outperforms other baseline methods on both the NER and RE tasks:

In the ACE04 \(NER\) task, the F1 score of the Label method reached 16\.84%, significantly higher than Explanation \(16\.47%\), ICL \(13\.69%\), and w/o\_ICL \(11\.01%\)\. In the CoNLL04 \(RE\) task, the Label method performed especially well, with an F1 score of 13\.39%, far surpassing Explanation \(4\.62%\), ICL \(2\.45%\), and w/o\_ICL \(0\.56%\)\. Moreover,Label\_Def\(label with definition\) showed a slight decline compared to the pure Label method, indicating that under random retrieval, adding too much explanatory information may weaken the effectiveness of the contrastive signal\.

From the random retrieval experiment, it can be observed that the label\-based contrastive example construction strategy we proposed demonstrates good robustness\. Even in the absence of semantic retrieval support and with poor\-quality supporting sample information, the Label method still significantly improves the model’s reasoning and discrimination abilities\. This phenomenon further validates the core advantage of our method, which lies in explicit error signal labeling and structured supervision, rather than relying on the retrieved "good examples\."

In summary, the random retrieval experiment fully verifies the effectiveness and generalization of theLC\-ICLmethod, indicating that reasonably designed label information is more effective in helping large models understand task objectives and improve performance than solely relying on sample similarity\.

### V\-DCase study

![Refer to caption](https://arxiv.org/html/2606.29407v1/x3.png)Figure 6:A comparison of prompts betweenLC\-ICLand ICL methods on the same RE task\. The traditional ICL approach \(right\) includes positive examples, instructions, and a test instance\. In contrast,LC\-ICL\(left\) augments this setup by additionally incorporating a negative example, which not only presents an instance where the large language model makes a reasoning error, but also provides a set of labels indicating the reasons behind the error\. For the NER task and additional examples, please refer to the appendix\.We selected a number of representative test cases to further demonstrate the effectiveness of our proposed method, as shown in Figure[6](https://arxiv.org/html/2606.29407#S5.F6)\.

Taking Example 1 as an illustration, we present a contrastive set of positive and negative examples in the relation extraction \(RE\) task that are more similar to the test instance\.

In this case, using only the ICL method \(right side of the figure\) indeed enables the model to effectively identify the entities "Sony" and "Akio Morita", but it fails to accurately recognize their relationship\. In contrast, our method \(left side of the figure\) allows the large language model to learn that the reasoning outcome

”​A​k​i​o​M​o​r​i​t​a​”,”​f​o​u​n​d​e​d​”,”​S​o​n​y​”"AkioMorita","founded","Sony"corresponds to the label "Correct entities, incorrect relation" after being exposed to negative samples\. As a result, the model becomes more attentive to the correctness of the relation\.

By learning correct patterns from positive examples and simultaneously learning to avoid erroneous patterns from negative examples, the large model is better able to derive accurate conclusions\.

In addition, we also discuss typical cases in the named entity recognition \(NER\) task; more examples can be found in Appendix[C](https://arxiv.org/html/2606.29407#A3)\.

## VIRelated Work

##### Generative Information Extraction

In recent years, a large number of representative supervised modeling methods have been proposed for sub\-tasks in information extraction, such as named entity recognition \(NER\) and relation extraction \(RE\)\. Early works primarily focused on understanding\-based architectures, such as the LSTM\-CRF model\[[15](https://arxiv.org/html/2606.29407#bib.bib80)\]and RE methods based on dependency tree modeling\[[38](https://arxiv.org/html/2606.29407#bib.bib81)\], which effectively capture both local and syntactic features\.

\[[53](https://arxiv.org/html/2606.29407#bib.bib20)\]leveraged the mutual influence between RE and NER tasks to design a dynamic cross\-task model that further enhances the model’s information extraction capabilities\. However, supervised modeling methods typically require fine\-tuning on specific datasets, resulting in limited generalization ability\.

Meanwhile, the rise of generative Transformer architectures has introduced a new paradigm for information extraction, where the task is reformulated as a text generation problem\. Works such as the Crop model\[[50](https://arxiv.org/html/2606.29407#bib.bib28)\], the MCL\-NER framework\[[31](https://arxiv.org/html/2606.29407#bib.bib71)\], and zero\-shot IE methods based on ChatGPT\[[46](https://arxiv.org/html/2606.29407#bib.bib65)\]have all demonstrated the strong potential of large models in extraction tasks\.

Overall, although generative large language models exhibit strong potential and diverse developmental trends in information extraction tasks, their performance remains suboptimal when directly applied to extraction tasks without fine\-tuning\.

##### In\-context Learning

In recent years, in\-context learning \(ICL\) has garnered extensive attention and application in large\-scale language models\. By constructing high\-quality exemplars, ICL significantly enhances the models’ reasoning and task execution capabilities\[[51](https://arxiv.org/html/2606.29407#bib.bib67),[34](https://arxiv.org/html/2606.29407#bib.bib84),[8](https://arxiv.org/html/2606.29407#bib.bib85)\]\. To further improve the reasoning abilities of language models, existing studies have shown that relying solely on the models’ inherent reasoning mechanisms is insufficient to fully unleash their potential\. Instead, constructing detailed chains of thought can notably boost the performance of reasoning models\[[10](https://arxiv.org/html/2606.29407#bib.bib15)\], underscoring the critical role of context construction in influencing model performance\. In addition, methods such as nearest\-neighbor retrieval\[[21](https://arxiv.org/html/2606.29407#bib.bib58)\], task\-aware retrieval strategies, and label\-guided reasoning representations\[[42](https://arxiv.org/html/2606.29407#bib.bib54)\]have further optimized the process of exemplar selection\.

Recently, substantial progress has also been made in instruction\-following tasks\.\[[19](https://arxiv.org/html/2606.29407#bib.bib17)\]proposed the URIAL method, which achieves effective alignment for base language models using only three stylized in\-context examples\. Although URIAL demonstrates notable improvements in instruction\-following capabilities,\[[52](https://arxiv.org/html/2606.29407#bib.bib16)\]pointed out that it still underperforms fine\-tuned models in multi\-turn interactive scenarios\.

Overall, in\-context learning exhibits great potential in enhancing the reasoning ability and task performance of large language models\. However, its limitations have also gradually emerged, primarily due to the lack of negative feedback mechanisms, which restricts further performance improvement\.

##### Hard negative sample

Hard negative samples have a long\-standing history of applications in machine learning\. In the early days,\[[27](https://arxiv.org/html/2606.29407#bib.bib3)\]introduced the Word2Vec model, which employed negative sampling to train word embeddings, enabling more efficient learning of textual features\. Similarly,\[[5](https://arxiv.org/html/2606.29407#bib.bib2)\]enabled the model to predict masked tokens while treating other tokens as negative samples, significantly improving performance on a range of NLP tasks such as question answering, sentiment analysis, and text classification\. A current research focus is whether erroneous information generated by large language models \(LLMs\) during inference can be systematically categorized and utilized as negative samples to enhance model capabilities\. Relevant scholars have conducted systematic analyses and discussions on this topic\. For instance,\[[36](https://arxiv.org/html/2606.29407#bib.bib13)\]presented a detailed taxonomy of hallucinations in LLMs, distinguishing between "ignorance\-based errors" caused by knowledge gaps and "knowingly wrong" errors where the model possesses the correct knowledge yet generates incorrect outputs\. This distinction provides a crucial theoretical foundation for error detection and intervention\. In addition,\[[22](https://arxiv.org/html/2606.29407#bib.bib14)\]proposed a prompting method that integrates chain\-of\-thought reasoning with error analysis\. By simulating human evaluation systems to conduct fine\-grained error categorization in translation outputs, the method not only improves evaluation performance but also enhances model interpretability\. In the fields of Named Entity Recognition \(NER\) and Relation Extraction \(RE\),\[[43](https://arxiv.org/html/2606.29407#bib.bib21),[49](https://arxiv.org/html/2606.29407#bib.bib18)\]skillfully applied similar principles, leveraging comparison\-based learning to further boost accuracy in information extraction tasks\. Furthermore,\[[29](https://arxiv.org/html/2606.29407#bib.bib4)\]proposed a theoretical framework based on consistency learning to obtain high\-quality negative samples, thereby advancing the information extraction capabilities of LLMs\.

Motivated by these observations, the core goal of this study is to exploit specific error patterns in LLM\-generated content as learning signals\. These patterns are fed back to the model in the form of hard negative samples, with the aim of improving the model’s performance on targeted tasks\.

## VIIConclusion

In this work, we introduceLC\-ICL, contrastive in\-context learning for few\-shot information extraction, including right/positive and wrong/negative demonstrations\. In addition through type instruction demonstrations prompt mention tags in the IE task\. From the contrastive samples, the LLMs could obtain effective information and indirect but positive, valuable additional knowledge for IE tasks\. Besides, our method adopts semantic similarity retrieval strategies to retrieve in\-context examples better suited for the current sentence and task, significantly improving IE performance\. Extensive experiments prove the effectiveness ofLC\-ICLon various benchmarks\.

## Limitations

Despite its contributions, this study has the following limitations: \(1\) It primarily investigates the contextual learning capabilities in few\-shot Named Entity Recognition \(NER\) and Relation Extraction \(RE\) tasks, while the applicability of this paradigm to other Information Extraction \(IE\) tasks remains underexplored; \(2\) We employed a commonly used sentence embedding similarity approach to retrieve samples, yet there may exist other, more diverse strategies for selecting appropriate positive and negative samples; \(3\) Our evaluation was conducted using English datasets on large models trained in English, and further exploration on datasets or models in other languages, such as Chinese, was not carried out\.

## References

- \[1\]J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§I](https://arxiv.org/html/2606.29407#S1.p1.1)\.
- \[2\]D\. M\. Aliod, M\. van Zaanen, and D\. Smith\(2006\)Named entity recognition for question answering\.InProceedings of the Australasian Language Technology Workshop, ALTA 2006, Sydney, Australia, November 30\-December 1, 2006,L\. Cavedon and I\. Zukerman \(Eds\.\),pp\. 51–58\.External Links:[Link](https://aclanthology.org/U06-1009/)Cited by:[§I](https://arxiv.org/html/2606.29407#S1.p1.1)\.
- \[3\]T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. M\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. Amodei\(2020\)Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6\-12, 2020, virtual,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\. Balcan, and H\. Lin \(Eds\.\),External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html)Cited by:[§I](https://arxiv.org/html/2606.29407#S1.p1.1)\.
- \[4\]J\. Chen, Y\. Lu, H\. Lin, J\. Lou, W\. Jia, D\. Dai, H\. Wu, B\. Cao, X\. Han, and L\. Sun\(2023\)Learning in\-context learning for named entity recognition\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2023, Toronto, Canada, July 9\-14, 2023,A\. Rogers, J\. L\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),pp\. 13661–13675\.External Links:[Link](https://doi.org/10.18653/v1/2023.acl-long.764),[Document](https://dx.doi.org/10.18653/V1/2023.ACL-LONG.764)Cited by:[§I](https://arxiv.org/html/2606.29407#S1.p1.1),[§I](https://arxiv.org/html/2606.29407#S1.p2.1)\.
- \[5\]J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova\(2019\)Bert: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 \(long and short papers\),pp\. 4171–4186\.Cited by:[§VI](https://arxiv.org/html/2606.29407#S6.SS0.SSS0.Px3.p1.1)\.
- \[6\]G\. R\. Doddington, A\. Mitchell, M\. A\. Przybocki, L\. A\. Ramshaw, S\. M\. Strassel, and R\. M\. Weischedel\(2004\)The automatic content extraction \(ACE\) program \- tasks, data, and evaluation\.InProceedings of the Fourth International Conference on Language Resources and Evaluation, LREC 2004, May 26\-28, 2004, Lisbon, Portugal,External Links:[Link](http://www.lrec-conf.org/proceedings/lrec2004/summaries/5.htm)Cited by:[§IV\-A](https://arxiv.org/html/2606.29407#S4.SS1.p2.1)\.
- \[7\]R\. I\. Dogan, R\. Leaman, and Z\. Lu\(2014\)NCBI disease corpus: a resource for disease name recognition and concept normalization\.Journal of biomedical informatics47,pp\. 1–10\.Cited by:[§IV\-A](https://arxiv.org/html/2606.29407#S4.SS1.p2.1)\.
- \[8\]Q\. Dong, L\. Li, D\. Dai, C\. Zheng, Z\. Wu, B\. Chang, X\. Sun, J\. Xu, L\. Li, and Z\. Sui\(2023\)A survey on in\-context learning\.External Links:2301\.00234Cited by:[§VI](https://arxiv.org/html/2606.29407#S6.SS0.SSS0.Px2.p1.1)\.
- \[9\]F\. Faghri, D\. J\. Fleet, J\. R\. Kiros, and S\. Fidler\(2017\)Vse\+\+: improving visual\-semantic embeddings with hard negatives\.arXiv preprint arXiv:1707\.05612\.Cited by:[§III\-C2](https://arxiv.org/html/2606.29407#S3.SS3.SSS2.p1.2)\.
- \[10\]Y\. Ge, S\. Liu, Y\. Wang, L\. Mei, L\. Chen, B\. Bi, and X\. Cheng\(2025\)Innate reasoning is not enough: in\-context learning enhances reasoning large language models with less overthinking\.External Links:2503\.19602,[Link](https://arxiv.org/abs/2503.19602)Cited by:[§VI](https://arxiv.org/html/2606.29407#S6.SS0.SSS0.Px2.p1.1)\.
- \[11\]A\. Grattafiori, A\. Dubey, A\. Jauhri,et al\.\(2024\)The llama 3 herd of models\.External Links:2407\.21783Cited by:[§IV\-B](https://arxiv.org/html/2606.29407#S4.SS2.p1.2)\.
- \[12\]Y\. Guo, Z\. Li, X\. Jin, Y\. Liu, Y\. Zeng, W\. Liu, X\. Li, P\. Yang, L\. Bai, J\. Guo, and X\. Cheng\(2023\)Retrieval\-augmented code generation for universal information extraction\.CoRRabs/2311\.02962\.External Links:[Link](https://doi.org/10.48550/arXiv.2311.02962),[Document](https://dx.doi.org/10.48550/ARXIV.2311.02962),2311\.02962Cited by:[§III\-C1](https://arxiv.org/html/2606.29407#S3.SS3.SSS1.p1.1)\.
- \[13\]H\. Gurulingappa, A\. M\. Rajput, A\. Roberts, J\. Fluck, M\. Hofmann\-Apitius, and L\. Toldo\(2012\)Development of a benchmark corpus to support the automatic extraction of drug\-related adverse effects from medical case reports\.Journal of biomedical informatics45\(5\),pp\. 885–892\.Cited by:[§IV\-A](https://arxiv.org/html/2606.29407#S4.SS1.p1.1)\.
- \[14\]B\. J\. Gutierrez, N\. McNeal, C\. Washington, Y\. Chen, L\. Li, H\. Sun, and Y\. Su\(2022\)Thinking about GPT\-3 in\-context learning for biomedical ie? think again\.InFindings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7\-11, 2022,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),pp\. 4497–4512\.External Links:[Link](https://doi.org/10.18653/v1/2022.findings-emnlp.329),[Document](https://dx.doi.org/10.18653/V1/2022.FINDINGS-EMNLP.329)Cited by:[§III\-C1](https://arxiv.org/html/2606.29407#S3.SS3.SSS1.p1.1)\.
- \[15\]G\. Lample, M\. Ballesteros, S\. Subramanian, K\. Kawakami, and C\. Dyer\(2016\)Neural architectures for named entity recognition\.InNAACL HLT 2016,pp\. 260–270\.Cited by:[§VI](https://arxiv.org/html/2606.29407#S6.SS0.SSS0.Px1.p1.1)\.
- \[16\]B\. Li, G\. Fang, Y\. Yang, Q\. Wang, W\. Ye, W\. Zhao, and S\. Zhang\(2023\)Evaluating chatgpt’s information extraction capabilities: an assessment of performance, explainability, calibration, and faithfulness\.CoRRabs/2304\.11633\.External Links:[Link](https://doi.org/10.48550/arXiv.2304.11633),[Document](https://dx.doi.org/10.48550/ARXIV.2304.11633),2304\.11633Cited by:[§I](https://arxiv.org/html/2606.29407#S1.p1.1)\.
- \[17\]P\. Li, T\. Sun, Q\. Tang, H\. Yan, Y\. Wu, X\. Huang, and X\. Qiu\(2023\)CodeIE: large code generation models are better few\-shot information extractors\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2023, Toronto, Canada, July 9\-14, 2023,A\. Rogers, J\. L\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),pp\. 15339–15353\.External Links:[Link](https://doi.org/10.18653/v1/2023.acl-long.855),[Document](https://dx.doi.org/10.18653/V1/2023.ACL-LONG.855)Cited by:[§I](https://arxiv.org/html/2606.29407#S1.p2.1),[§III\-B](https://arxiv.org/html/2606.29407#S3.SS2.p4.3),[§III\-B](https://arxiv.org/html/2606.29407#S3.SS2.p6.12),[§IV\-A](https://arxiv.org/html/2606.29407#S4.SS1.p2.1),[§IV\-B](https://arxiv.org/html/2606.29407#S4.SS2.p2.2),[§IV\-C](https://arxiv.org/html/2606.29407#S4.SS3.p1.1)\.
- \[18\]X\. Li, J\. Feng, Y\. Meng, Q\. Han, F\. Wu, and J\. Li\(2020\)A unified MRC framework for named entity recognition\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5\-10, 2020,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. R\. Tetreault \(Eds\.\),pp\. 5849–5859\.External Links:[Link](https://doi.org/10.18653/v1/2020.acl-main.519),[Document](https://dx.doi.org/10.18653/V1/2020.ACL-MAIN.519)Cited by:[§IV\-A](https://arxiv.org/html/2606.29407#S4.SS1.p2.1)\.
- \[19\]B\. Y\. Lin, A\. Ravichander, X\. Lu, N\. Dziri, M\. Sclar, K\. Chandu, C\. Bhagavatula, and Y\. Choi\(2023\)The unlocking spell on base llms: rethinking alignment via in\-context learning\.arXiv preprint arXiv:2312\.01552\.Cited by:[§VI](https://arxiv.org/html/2606.29407#S6.SS0.SSS0.Px2.p2.1)\.
- \[20\]A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan,et al\.\(2024\)Deepseek\-v3 technical report\.arXiv preprint arXiv:2412\.19437\.Cited by:[§IV\-B](https://arxiv.org/html/2606.29407#S4.SS2.p1.2)\.
- \[21\]J\. Liu, D\. Shen, Y\. Zhang, B\. Dolan, L\. Carin, and W\. Chen\(2022\)What makes good in\-context examples for gpt\-3?\.InProceedings of Deep Learning Inside Out: The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, DeeLIO@ACL 2022, Dublin, Ireland and Online, May 27, 2022,E\. Agirre, M\. Apidianaki, and I\. Vulic \(Eds\.\),pp\. 100–114\.External Links:[Link](https://doi.org/10.18653/v1/2022.deelio-1.10),[Document](https://dx.doi.org/10.18653/V1/2022.DEELIO-1.10)Cited by:[§III\-C1](https://arxiv.org/html/2606.29407#S3.SS3.SSS1.p1.1),[§VI](https://arxiv.org/html/2606.29407#S6.SS0.SSS0.Px2.p1.1)\.
- \[22\]Q\. Lu, B\. Qiu, L\. Ding, K\. Zhang, T\. Kocmi, and D\. Tao\(2023\)Error analysis prompting enables human\-like translation evaluation in large language models\.arXiv preprint arXiv:2303\.13809\.Cited by:[§VI](https://arxiv.org/html/2606.29407#S6.SS0.SSS0.Px3.p1.1)\.
- \[23\]Y\. Lu, Q\. Liu, D\. Dai, X\. Xiao, H\. Lin, X\. Han, L\. Sun, and H\. Wu\(2022\)Unified structure generation for universal information extraction\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2022, Dublin, Ireland, May 22\-27, 2022,S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),pp\. 5755–5772\.External Links:[Link](https://doi.org/10.18653/v1/2022.acl-long.395),[Document](https://dx.doi.org/10.18653/V1/2022.ACL-LONG.395)Cited by:[§IV\-A](https://arxiv.org/html/2606.29407#S4.SS1.p1.1),[§IV\-C](https://arxiv.org/html/2606.29407#S4.SS3.p1.1)\.
- \[24\]Y\. Luan, L\. He, M\. Ostendorf, and H\. Hajishirzi\(2018\)Multi\-task identification of entities, relations, and coreference for scientific knowledge graph construction\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 \- November 4, 2018,E\. Riloff, D\. Chiang, J\. Hockenmaier, and J\. Tsujii \(Eds\.\),pp\. 3219–3232\.External Links:[Link](https://doi.org/10.18653/v1/d18-1360),[Document](https://dx.doi.org/10.18653/V1/D18-1360)Cited by:[§IV\-A](https://arxiv.org/html/2606.29407#S4.SS1.p1.1)\.
- \[25\]X\. Lyu, S\. Min, I\. Beltagy, L\. Zettlemoyer, and H\. Hajishirzi\(2023\)Z\-ICL: zero\-shot in\-context learning with pseudo\-demonstrations\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2023, Toronto, Canada, July 9\-14, 2023,A\. Rogers, J\. L\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),pp\. 2304–2317\.External Links:[Link](https://doi.org/10.18653/v1/2023.acl-long.129),[Document](https://dx.doi.org/10.18653/V1/2023.ACL-LONG.129)Cited by:[§I](https://arxiv.org/html/2606.29407#S1.p1.1)\.
- \[26\]X\. Ma, J\. Li, and M\. Zhang\(2023\)Chain of thought with explicit evidence reasoning for few\-shot relation extraction\.InFindings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6\-10, 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),pp\. 2334–2352\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.153)Cited by:[§I](https://arxiv.org/html/2606.29407#S1.p2.1)\.
- \[27\]T\. Mikolov, I\. Sutskever, K\. Chen, G\. S\. Corrado, and J\. Dean\(2013\)Distributed representations of words and phrases and their compositionality\.Advances in neural information processing systems26\.Cited by:[§VI](https://arxiv.org/html/2606.29407#S6.SS0.SSS0.Px3.p1.1)\.
- \[28\]S\. Min, X\. Lyu, A\. Holtzman, M\. Artetxe, M\. Lewis, H\. Hajishirzi, and L\. Zettlemoyer\(2022\)Rethinking the role of demonstrations: what makes in\-context learning work?\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7\-11, 2022,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),pp\. 11048–11064\.External Links:[Link](https://doi.org/10.18653/v1/2022.emnlp-main.759),[Document](https://dx.doi.org/10.18653/V1/2022.EMNLP-MAIN.759)Cited by:[§I](https://arxiv.org/html/2606.29407#S1.p1.1)\.
- \[29\]Y\. Mo, J\. Liu, J\. Yang, Q\. Wang, S\. Zhang, J\. Wang, and Z\. Li\(2024\)C\-icl: contrastive in\-context learning for information extraction\.arXiv preprint arXiv:2402\.11254\.Cited by:[§III\-C2](https://arxiv.org/html/2606.29407#S3.SS3.SSS2.p1.2),[§VI](https://arxiv.org/html/2606.29407#S6.SS0.SSS0.Px3.p1.1)\.
- \[30\]Y\. Mo, H\. Tang, J\. Liu, Q\. Wang, Z\. Xu, J\. Wang, W\. Wu, and Z\. Li\(2023\)Multi\-task transformer with relation\-attention and type\-attention for named entity recognition\.InIEEE International Conference on Acoustics, Speech and Signal Processing ICASSP 2023, Rhodes Island, Greece, June 4\-10, 2023,pp\. 1–5\.External Links:[Link](https://doi.org/10.1109/ICASSP49357.2023.10094905),[Document](https://dx.doi.org/10.1109/ICASSP49357.2023.10094905)Cited by:[§IV\-A](https://arxiv.org/html/2606.29407#S4.SS1.p2.1)\.
- \[31\]Y\. Mo, J\. Yang, J\. Liu, Q\. Wang, R\. Chen, J\. Wang, and Z\. Li\(2023\)MCL\-ner: cross\-lingual named entity recognition via multi\-view contrastive learning\.CoRRabs/2308\.09073\.External Links:[Link](https://doi.org/10.48550/arXiv.2308.09073),[Document](https://dx.doi.org/10.48550/ARXIV.2308.09073),2308\.09073Cited by:[§IV\-A](https://arxiv.org/html/2606.29407#S4.SS1.p2.1),[§VI](https://arxiv.org/html/2606.29407#S6.SS0.SSS0.Px1.p3.1)\.
- \[32\]S\. Riedel, L\. Yao, and A\. McCallum\(2010\)Modeling relations and their mentions without labeled text\.InMachine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2010, Barcelona, Spain, September 20\-24, 2010, Proceedings, Part III 21,pp\. 148–163\.Cited by:[§IV\-A](https://arxiv.org/html/2606.29407#S4.SS1.p1.1)\.
- \[33\]D\. Roth and W\. Yih\(2004\)A linear programming formulation for global inference in natural language tasks\.InProceedings of the Eighth Conference on Computational Natural Language Learning, CoNLL 2004, Held in cooperation with HLT\-NAACL 2004, Boston, Massachusetts, USA, May 6\-7, 2004,H\. T\. Ng and E\. Riloff \(Eds\.\),pp\. 1–8\.External Links:[Link](https://aclanthology.org/W04-2401/)Cited by:[§IV\-A](https://arxiv.org/html/2606.29407#S4.SS1.p1.1)\.
- \[34\]O\. Rubin, J\. Herzig, and J\. Berant\(2022\)Learning to retrieve prompts for in\-context learning\.InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10\-15, 2022,M\. Carpuat, M\. de Marneffe, and I\. V\. M\. Ruíz \(Eds\.\),pp\. 2655–2671\.External Links:[Link](https://doi.org/10.18653/v1/2022.naacl-main.191),[Document](https://dx.doi.org/10.18653/V1/2022.NAACL-MAIN.191)Cited by:[§VI](https://arxiv.org/html/2606.29407#S6.SS0.SSS0.Px2.p1.1)\.
- \[35\]O\. Sainz, I\. García\-Ferrero, R\. Agerri, O\. L\. de Lacalle, G\. Rigau, and E\. Agirre\(2023\)GoLLIE: annotation guidelines improve zero\-shot information\-extraction\.CoRRabs/2310\.03668\.External Links:[Link](https://doi.org/10.48550/arXiv.2310.03668),[Document](https://dx.doi.org/10.48550/ARXIV.2310.03668),2310\.03668Cited by:[§I](https://arxiv.org/html/2606.29407#S1.p2.1)\.
- \[36\]A\. Simhi, J\. Herzig, I\. Szpektor, and Y\. Belinkov\(2024\)Distinguishing ignorance from error in llm hallucinations\.arXiv preprint arXiv:2410\.22071\.Cited by:[§VI](https://arxiv.org/html/2606.29407#S6.SS0.SSS0.Px3.p1.1)\.
- \[37\]R\. Takanobu, T\. Zhang, J\. Liu, and M\. Huang\(2019\)A hierarchical framework for relation extraction with reinforcement learning\.InProceedings of the AAAI conference on artificial intelligence,Vol\.33,pp\. 7072–7079\.Cited by:[§IV\-A](https://arxiv.org/html/2606.29407#S4.SS1.p1.1)\.
- \[38\]B\. Tang, J\. Hu, X\. Wang, and Q\. Chen\(2018\)Recognizing continuous and discontinuous adverse drug reaction mentions from social media using lstm\-crf\.Wireless Communications and Mobile Computing2018\.Cited by:[§VI](https://arxiv.org/html/2606.29407#S6.SS0.SSS0.Px1.p1.1)\.
- \[39\]H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale,et al\.\(2023\)Llama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.Cited by:[§I](https://arxiv.org/html/2606.29407#S1.p1.1)\.
- \[40\]S\. Wadhwa, S\. Amir, and B\. C\. Wallace\(2023\)Revisiting relation extraction in the era of large language models\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2023, Toronto, Canada, July 9\-14, 2023,A\. Rogers, J\. L\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),pp\. 15566–15589\.External Links:[Link](https://doi.org/10.18653/v1/2023.acl-long.868),[Document](https://dx.doi.org/10.18653/V1/2023.ACL-LONG.868)Cited by:[§I](https://arxiv.org/html/2606.29407#S1.p2.1)\.
- \[41\]C\. Walker and L\. D\. Consortium\(2005\)ACE 2005 multilingual training corpus\.LDC corpora\.External Links:ISBN 9781585633760Cited by:[§IV\-A](https://arxiv.org/html/2606.29407#S4.SS1.p2.1)\.
- \[42\]Z\. Wan, F\. Cheng, Z\. Mao, Q\. Liu, H\. Song, J\. Li, and S\. Kurohashi\(2023\)GPT\-RE: in\-context learning for relation extraction using large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6\-10, 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),pp\. 3534–3547\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.214)Cited by:[§I](https://arxiv.org/html/2606.29407#S1.p2.1),[§VI](https://arxiv.org/html/2606.29407#S6.SS0.SSS0.Px2.p1.1)\.
- \[43\]C\. Wang, S\. Zhao, T\. Yan, S\. Song, W\. Ma, K\. Liu, and M\. Wang\(2025\)Hierarchical label\-enhanced contrastive learning for chinese ner\.IEEE Transactions on Neural Networks and Learning Systems\(\),pp\. 1–11\.External Links:[Document](https://dx.doi.org/10.1109/TNNLS.2025.3528416)Cited by:[§VI](https://arxiv.org/html/2606.29407#S6.SS0.SSS0.Px3.p1.1)\.
- \[44\]X\. Wang, W\. Zhou, C\. Zu, H\. Xia, T\. Chen, Y\. Zhang, R\. Zheng, J\. Ye, Q\. Zhang, T\. Gui,et al\.\(2023\)InstructUIE: multi\-task instruction tuning for unified information extraction\.arXiv preprint arXiv:2304\.08085\.Cited by:[§I](https://arxiv.org/html/2606.29407#S1.p2.1)\.
- \[45\]X\. Wang, S\. Li, and H\. Ji\(2023\)Code4Struct: code generation for few\-shot event structure prediction\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2023, Toronto, Canada, July 9\-14, 2023,A\. Rogers, J\. L\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),pp\. 3640–3663\.External Links:[Link](https://doi.org/10.18653/v1/2023.acl-long.202),[Document](https://dx.doi.org/10.18653/V1/2023.ACL-LONG.202)Cited by:[§I](https://arxiv.org/html/2606.29407#S1.p2.1)\.
- \[46\]X\. Wei, X\. Cui, N\. Cheng, X\. Wang, X\. Zhang, S\. Huang, P\. Xie, J\. Xu, Y\. Chen, M\. Zhang, Y\. Jiang, and W\. Han\(2023\)Zero\-shot information extraction via chatting with chatgpt\.CoRRabs/2302\.10205\.External Links:[Link](https://doi.org/10.48550/arXiv.2302.10205),[Document](https://dx.doi.org/10.48550/ARXIV.2302.10205),2302\.10205Cited by:[§VI](https://arxiv.org/html/2606.29407#S6.SS0.SSS0.Px1.p3.1)\.
- \[47\]X\. Wei, X\. Cui, N\. Cheng, X\. Wang, X\. Zhang, S\. Huang, P\. Xie, J\. Xu, Y\. Chen, M\. Zhang,et al\.\(2023\)Zero\-shot information extraction via chatting with chatgpt\.arXiv preprint arXiv:2302\.10205\.Cited by:[§I](https://arxiv.org/html/2606.29407#S1.p2.1)\.
- \[48\]D\. Xu, W\. Chen, W\. Peng, C\. Zhang, T\. Xu, X\. Zhao, X\. Wu, Y\. Zheng, and E\. Chen\(2023\)Large language models for generative information extraction: A survey\.CoRRabs/2312\.17617\.External Links:[Link](https://doi.org/10.48550/arXiv.2312.17617),[Document](https://dx.doi.org/10.48550/ARXIV.2312.17617),2312\.17617Cited by:[§I](https://arxiv.org/html/2606.29407#S1.p1.1)\.
- \[49\]T\. Yan, S\. Zhao, M\. Hu, M\. Wang, X\. Zhang, Z\. Luo, and M\. Wang\(2025\)HCL: a hierarchical contrastive learning framework for zero\-shot relation extraction\.IEEE Transactions on Neural Networks and Learning Systems36\(3\),pp\. 5694–5705\.External Links:[Document](https://dx.doi.org/10.1109/TNNLS.2024.3379527)Cited by:[§VI](https://arxiv.org/html/2606.29407#S6.SS0.SSS0.Px3.p1.1)\.
- \[50\]J\. Yang, S\. Huang, S\. Ma, Y\. Yin, L\. Dong, D\. Zhang, H\. Guo, Z\. Li, and F\. Wei\(2022\)CROP: zero\-shot cross\-lingual named entity recognition with multilingual labeled sequence translation\.InFindings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7\-11, 2022,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),pp\. 486–496\.External Links:[Link](https://doi.org/10.18653/v1/2022.findings-emnlp.34),[Document](https://dx.doi.org/10.18653/V1/2022.FINDINGS-EMNLP.34)Cited by:[§VI](https://arxiv.org/html/2606.29407#S6.SS0.SSS0.Px1.p3.1)\.
- \[51\]J\. Yang, J\. Wan, S\. Ma, H\. Huang, D\. Zhang, Y\. Yu, Z\. Li, and F\. Wei\(2021\)Learning to select relevant knowledge for neural machine translation\.InNatural Language Processing and Chinese Computing \- 10th CCF International Conference, NLPCC 2021, Qingdao, China, October 13\-17, 2021, Proceedings, Part I,L\. Wang, Y\. Feng, Y\. Hong, and R\. He \(Eds\.\),Lecture Notes in Computer Science, Vol\.13028,pp\. 79–91\.External Links:[Link](https://doi.org/10.1007/978-3-030-88480-2%5C_7),[Document](https://dx.doi.org/10.1007/978-3-030-88480-2%5F7)Cited by:[§VI](https://arxiv.org/html/2606.29407#S6.SS0.SSS0.Px2.p1.1)\.
- \[52\]H\. Zhao, M\. Andriushchenko, F\. Croce, and N\. Flammarion\(2024\)Is in\-context learning sufficient for instruction following in llms?\.arXiv preprint arXiv:2405\.19874\.Cited by:[§VI](https://arxiv.org/html/2606.29407#S6.SS0.SSS0.Px2.p2.1)\.
- \[53\]S\. Zhao, M\. Hu, Z\. Cai, and F\. Liu\(2023\)Dynamic modeling cross\-modal interactions in two\-phase prediction for entity\-relation extraction\.IEEE Transactions on Neural Networks and Learning Systems34\(3\),pp\. 1122–1131\.External Links:[Document](https://dx.doi.org/10.1109/TNNLS.2021.3104971)Cited by:[§VI](https://arxiv.org/html/2606.29407#S6.SS0.SSS0.Px1.p2.1)\.
- \[54\]L\. Zhong, J\. Wu, Q\. Li, H\. Peng, and X\. Wu\(2023\)A comprehensive survey on automatic knowledge graph construction\.CoRRabs/2302\.05019\.External Links:[Link](https://doi.org/10.48550/arXiv.2302.05019),[Document](https://dx.doi.org/10.48550/ARXIV.2302.05019),2302\.05019Cited by:[§I](https://arxiv.org/html/2606.29407#S1.p1.1)\.

## Appendix AImplementation Experiment

### A\-ADataset Statistics

To facilitate a thorough evaluation, we incorporate a diverse collection of datasets spanning both NER and RE tasks, comprising three widely\-used NER benchmarks and five representative RE benchmarks\. The detailed statistics of these datasets , including the number of entity and relation types, along with the instance distributions across the training, development, and test sets , are presented in Table[III](https://arxiv.org/html/2606.29407#A1.T3)\. This comprehensive overview not only illustrates the characteristics of each dataset but also underscores the robustness and generalizability of our evaluation framework\.

TABLE III:Statistics of NER and RE Datasets\.DatasetsEntityTypesRelationTypesTrainDevTestNERACE047/6202745812ACE057/72999711060NCBI1/5433924941RECoNLL0445922231288NYT103245619650005000NYT11/1262648149369ADE/13417427428SciERC671861275551

### A\-BImplementation Experiment Details

All experiments are conducted using the PyTorch deep learning framework on NVIDIA Tesla A100 GPUs\. To ensure optimal model performance, we carefully select the experimental settings and hyperparameters\. These include the maximum sequence length, number of beams for beam search, as well as the top\-p and temperature values that control the randomness during generation\. The parameters are detailed in Table[IV](https://arxiv.org/html/2606.29407#A1.T4)\.

TABLE IV:The main parameters of our methodLC\-ICLbased on Llama\.ParametersValuesMax Sequence Length8192Num\_beams1Do\_sampleTrueTop\_p0\.85Temperature0\.3

## Appendix BSupplementary Label Definition Explanation

Figure[7](https://arxiv.org/html/2606.29407#A2.F7)illustrates all label types and their corresponding definitions used for negative samples in our method\. For the Named Entity Recognition \(NER\) task, we categorize common sources of errors into five label types:Wrong entity boundary,Wrong entity types,Missing entities,Undefined entity types, andSpurious entities\. For the Relation Extraction \(RE\) task, we classify common error sources into six label types:Missing relation,Wrong entity/relation types,Wrong entity boundary,Undefined relation,Correct entities, incorrect relation, andReversed entities\. Each label type is immediately followed by an explanation of its definition\. When employing the Label\-Def method, as shown in experiment[4](https://arxiv.org/html/2606.29407#S5.F4), we provide the model with all label types and their definitions as prompts, depending on whether the current task is RE or NER\.

![Refer to caption](https://arxiv.org/html/2606.29407v1/x4.png)Figure 7:Label Categorization Illustration\. In the Relation Extraction \(RE\) task, labels are categorized into six classes, while in the Named Entity Recognition \(NER\) task, they are divided into five classes, each accompanied by corresponding label definitions\.
## Appendix CSupplementary Case Study

In this section, we present additional examples from the NER and RE tasks, as shown in Figure[8](https://arxiv.org/html/2606.29407#A3.F8)\. For each example, the left side displays the results using the LC\-ICL method with injected error labels, while the right side shows the results from the traditional ICL method\.

![Refer to caption](https://arxiv.org/html/2606.29407v1/x5.png)\(a\)Example 2
![Refer to caption](https://arxiv.org/html/2606.29407v1/x6.png)\(b\)Example 3

Figure 8:A supplementary case study on label\-guided contrastive in\-context learning is presented\. The left side shows the results of our method, while the right side presents those of the standard ICL approach\. The purple text highlights the annotated reasons for errors in negative samples\. Figure[8a](https://arxiv.org/html/2606.29407#A3.F8.sf1)illustrates the results of the Named Entity Recognition \(NER\) task, and Figure[8b](https://arxiv.org/html/2606.29407#A3.F8.sf2)shows the results of the Relation Extraction \(RE\) task\.In Example 2, when using only the traditional ICL method, the model fails to capture the relation \("Idle", "person"\)\. In contrast, our method provides negative samples that similarly omit three "person" entities—\["I", "person"\], \["some of the local people", "person"\], and \["the local people", "person"\]—and labels them asMissing entities\. By learning from these negative samples, the model is able to correctly recognize and avoid similar cases of entity omission, ultimately producing the correct output\.

In Example 3, the test sample is a relatively challenging sentence featuring three complex location entities: "The National Ignition Facility," "Lawrence Livermore National Laboratory," and "California\." Using the traditional ICL method, the model learns some basic patterns for location relations; however, in the presence of complex location structures, it becomes more prone to hallucinations, generating a large number of incorrect relations\. By incorporating negative samples, our method provides the model with error label information indicating missing correct relations as well as distracting location relations\. This additional supervision enables the model to better avoid hallucinations and significantly improves prediction accuracy\.