Schema-Constrained Document-Level Event Argument Extraction with Lightweight LLM Fine-Tuning

# Schema-Constrained Document-Level Event Argument Extraction with Lightweight LLM Fine-Tuning
Source: [https://arxiv.org/html/2607.16808](https://arxiv.org/html/2607.16808)
11institutetext:University of Naples Federico II, Naples, Italy
11email:roberto\.pietrantuono@unina\.it, p\.sattari@studenti\.unina\.it22institutetext:University of Salerno, Fisciano, Italy
22email:antguerriero@unisa\.it###### Abstract

Event Argument Extraction \(EAE\) converts documents into structured event records by identifying argument spans and assigning them schema\-defined roles\. Document\-level EAE is challenging due to long\-range dependencies between triggers and arguments, cross\-sentence context, and strict role constraints, which often lead to boundary errors, uncertainty in roles, and inconsistencies with restricted schemas\.

In this paper, we study whether mid\-sized open LLMs can perform schema\-constrained EAE reliably at the document level onMAVEN\-ARG\. Our approach combines \(i\) role\-set injection in prompts for schema compliance, \(ii\) parameter\-efficient supervised fine\-tuning \(LoRA\) using the same JSON\-only interface used at inference, and \(iii\) deterministic decoding with post\-processing that validates JSON, filters invalid roles, de\-duplicates arguments, and aligns spans to the document window\. Under the officialMAVEN\-ARGevaluator, fine\-tunedmid\-sized open modelsoutperform previously reported GPT baselines across mention, entity\-coreference, and event\-coreference evaluations; our best model \(Phi\-4, 14B\) reaches 42\.39% F1 at the event\-coreference level\. Code to reproduce experiments is publicly available at[https://github\.com/dessertlab/EAE/](https://github.com/dessertlab/EAE/)\.

## 1Introduction

Event Argument Extraction \(EAE\) extends event extraction beyond detecting triggers and event types to recovering who did what to whom, when, where, and how by identifyingargument spansand assigning them to schema\-defined roles\. EAE is a key step for converting unstructured documents into structured event records for downstream applications such as timeline construction, analytics, and information retrieval\. Figure[1](https://arxiv.org/html/2607.16808#S1.F1)provides a high\-level view of document\-level event understanding\. The sentence contains twoEvent Types \(*Military Operation*and*Loss*\), each anchored by aTriggerin the text \(e\.g\.,*invasion*for*Military Operation*and*lost*for*Loss*\)\. For each event, the goal of EAE is to extract anArgumentspan from the document and assign it anArgument Rolefrom the event schema \(e\.g\.,*Agent*,*Patient*,*Location*,*Content*\) as illustrated by the dashed links\. The figure also illustrates cross\-event relations \(e\.g\.,BEFOREandCAUSE\) between event types, motivating document\-level context\. In this work, we focus on schema\-constrained EAE: given a document, an event type, and a marked trigger, we predict role\-to\-argument spans as valid JSON that follows the role schema and can be evaluated by the official scorer\.

Prior work on EAE has progressed from rule\-based systems\[bui\-etal\-2013\-fast\]to traditional machine learning approaches\[Kalimoldayev2023MAPPINGTS,valenzuela\-escarcega\-etal\-2015\-domain\]and deep learning models – such as CNNs, RNNs, transformer architectures – to better capture context and semantic relations\[Li2021ACS\]\. More recently, large language models \(LLMs\) have been explored for event extraction via prompting and few\-shot learning, offering flexibility across domains and languages\[Rollo2024LeveragingLF\]\.

However, document\-level EAE remains challenging because correct outputs must satisfy two constraints simultaneously\. First, predictions must follow astrict role schema: for a given event type, only a fixed set of roles is valid, and models must avoid producing extra keys or mismatched roles\. Second, extracted arguments must bespan\-faithful: argument strings must correspond to substrings of the document, even when supporting evidence is distributed across multiple sentences or far from the trigger\. These constraints amplify common errors such as boundary mismatches, role confusion in long\-tailed inventories, and invalid or inconsistent structured outputs, especially when LLM generations drift into free\-form text\.

In this work, we investigate whether*mid\-sized open LLMs*can perform*schema\-constrained document\-level EAE*reliably on MAVEN\-ARG, a benchmark with a large ontology and coreference\-aware evaluation\. A central question we address is whether lightweight LLMs, when adapted with a strict schema\-conditioned interface, can achieve competitive document\-level EAE performance under the official MAVEN\-ARG evaluation protocol\. Our goal is not to propose a new model architecture, but to provide a*reproducible end\-to\-end pipeline*that produces structured outputs in the exact format required by the official evaluator\. The contribution is therefore deliberately empirical and systems oriented, since it offers a controlled single GPU comparison of mid\-sized open LLMs under one schema\-conditioned interface and the official scorer, rather than a new modeling paradigm\. The pipeline uses a JSON\-only instruction format, injects the valid role set for each event type into the prompt to constrain outputs, applies parameter\-efficient supervised fine\-tuning to adapt open models to the task, and performs deterministic validation to enforce schema consistency and span faithfulness before evaluation\. We summarize our contributions as follows:

![Refer to caption](https://arxiv.org/html/2607.16808v1/Figure/shot2.jpg)Figure 1:An example of the EAE problem\[wang\-etal\-2024\-maven\]- •End\-to\-end schema\-constrained EAE pipeline\. We formulate document level EAE as role\-to\-span generation per trigger under the officialMAVEN\-ARGJSON output format and evaluator requirements\.
- •Controlled comparison of open LLMs\.We benchmark several mid\-sized open LLM families under a consistent JSON\-only prompting interface, deterministic decoding, and identical evaluation and normalization\.
- •Practical schema compliance and output validity\. We use role\-set injection and deterministic output validation \(JSON extraction, schema filtering, de\-duplication, and span alignment\) to ensure predictions satisfy strict submission constraints\.
- •Empirical gains under official scoring\. Under the officialMAVEN\-ARGevaluator, fine\-tuned open models outperform previously reported few\-shot closed\-model baselines across mention, entity coreference, and event coreference level evaluations\. All code, prompts, and processed data needed to reproduce our experiments are publicly available at[https://github\.com/dessertlab/EAE/](https://github.com/dessertlab/EAE/)\.

## 2Related Work

### 2\.1LLMs and Event Extraction

#### 2\.1\.1Recent advancements using LLMs\.

Recent work highlights several challenges in event extraction, including the high cost of labeling data and the limited portability of models designed for narrow task formulations\. Event mining aims to understand what happened in text and categorize event\-related information\[Meng2024CEANCE\], but it has historically required substantial labeled supervision to reach strong performance\. More recently, large language models \(LLMs\) such as GPT\-style models, LLaMA, and Phi have been applied to event extraction with prompting and instruction\-following, enabling more flexible extraction and, in some settings, improved transfer across event types and domains\[Yang2024PromptbasedCE\]\. Overall, this trend reflects a shift from highly task\-specific architectures \(e\.g\., BERT\-based classifiers or BiLSTM pipelines\) toward more general\-purpose LLMs that can be adapted to different extraction settings with less bespoke modeling\[Meng2024CEANCE\]\.

A key difficulty, however, is that good performance still depends on carefully controlling model behavior and bridging the gap between open\-ended generation and the precise requirements of event extraction \(e\.g\., strict schemas and span\-faithful outputs\)\. Moreover, results can remain sensitive to task design choices such as prompt format, context selection, and output constraints\[sivarajkumar2023empiricalevaluationpromptingstrategies\]\.

#### 2\.1\.2Event Extraction methods\.

LLM\-based event extraction has been studied through several increasingly common approaches, including prompt engineering, few\-shot learning, zero\-shot learning, and retrieval\-augmented generation\. Prompt engineering focuses on designing instructions and templates that guide the model toward correct structured predictions in zero\-shot and few\-shot settings, often with minimal additional training\[sivarajkumar2023empiricalevaluationpromptingstrategies\]\. Few\-shot learning uses a small number of labeled examples and can be useful when data is sparse, where demonstration selection and template design play an important role\. Zero\-shot learning goes a step further by aiming to identify event types and roles without labeled examples, often by aligning contexts with label descriptions or external knowledge; it is also useful for cross\-lingual settings, where language\-agnostic prompts can support role and argument identification\[yue\-etal\-2023\-zero,zhang2022efficientzeroshoteventextraction\]\. Retrieval\-augmented generation can further improve robustness by retrieving relevant examples or auxiliary context at inference time, which can increase accuracy when the retrieved information is well\-aligned with the target schema and format requirements\.

Across these methods, a persistent challenge is that improvements often hinge on prompt design, the context provided to the model, and whether the model’s generation behavior aligns with the extraction template and schema constraints\[sivarajkumar2023empiricalevaluationpromptingstrategies\]\. In addition, comparisons across LLM families \(e\.g\., GPT\-style models, Gemini, and LLaMA\-2\) can be confounded by prompt, context\-length limits, and decoding settings, so cross\-model results should be interpreted cautiously without a controlled protocol\[geminiteam2025geminifamilyhighlycapable,touvron2023llama2openfoundation\]\.

##### Our approach\.

While prior work explores prompting, few/zero\-shot learning, and augmentation for event extraction, it remains unclear how well open LLMs perform on*document\-level, schema\-constrained*EAE when outputs must satisfy a strict JSON interface and are scored by the official MAVEN\-ARG evaluator\. Existing results are often reported under different prompts, context windows, and decoding settings, making comparisons difficult\. In contrast, we provide a unified pipeline – role\-set conditioned prompting, parameter\-efficient supervised fine\-tuning, deterministic decoding, and schema\-aware output validation – and benchmark multiplemid\-sized openLLMs under a consistent protocol\. Our best fine\-tuned open model \(Phi\-4, 14B\) achieves 42\.39 event\-coreference F1, significantly outperforms the strongest reported GPT baseline in event\-coreference F1 \(GPT\-4\) by \+12\.89 absolute points under the official MAVEN\-ARG evaluator\. This shows that that lightweight open models, could outperform few\-shot GPT prompting while enabling significantly low cost local inference on a single\-GPU with 4\-bit base weights\.

### 2\.2Datasets and Benchmarks

#### 2\.2\.1Key datasets for event extraction\.

ACE 2005\[walker2006ace2005multilingual\]is one of the most widely used datasets for event extraction and includes documents in English, Arabic, and Chinese, with annotated event types, subtypes, and argument roles\. It has been widely used for trigger identification and argument classification, and it is frequently used as a reference point in event extraction studies\[liu2021overvieweventextractionapplications,deng\-etal\-2022\-title2event\]\. DEIE is a large document\-level dataset that includes triggers, arguments, summaries, and relationships, and supports broader document\-level event information extraction\[ren\-etal\-2024\-deie\]\. Title2Event focuses on event extraction from news titles at scale\[deng\-etal\-2022\-title2event\]\.

In this work, MAVEN\-ARG is our main benchmark\. It provides a large schema with many event types and argument roles and includes expert\-written definitions as well as annotations that support argument and coreference\-aware evaluation\[wang\-etal\-2024\-maven\]\. The dataset and the official evaluation toolkit are publicly released by the MAVEN\-ARG benchmark authors and available through the official repository\.111[https://github\.com/THU\-KEG/MAVEN\-Argument](https://github.com/THU-KEG/MAVEN-Argument)Because MAVEN\-ARG evaluation requires strict structured outputs and coreference\-aware aggregation, it provides a stringent testbed for studying schema compliance and output validity in LLM\-based EAE\. Prior results on MAVEN\-ARG indicate that stronger closed LLMs \(e\.g\., GPT\-4\) can outperform weaker closed baselines \(e\.g\., GPT\-3\.5\) across mention\-, entity\-coreference\-, and event\-coreference\-based evaluation levels under the official scorer\[wang\-etal\-2024\-maven\]\. These characteristics make MAVEN\-ARG a challenging benchmark for document\-level argument extraction under strict schema and evaluation constraints\.

Benchmark results on ACE 2005 are also commonly used for comparison\. For example, ChatGPT has been evaluated for event extraction in dialogue\-oriented settings and for data generation/augmentation, with reported improvements including better coverage of long\-tail event types\[kan\-etal\-2024\-emancipating\]\. However, ACE 2005 contains more entity\-centered arguments and a more limited ontology than MAVEN\-ARG; as a result, it is less comprehensive for studying document\-level argument extraction with coreference\-aware evaluation\[kan\-etal\-2024\-emancipating\]\.

## 3Method

This paper evaluates LLMs for*document\-level*event argument extraction under*strict schema*and*structured\-output*constraints\. We follow theMAVEN\-ARGsetting, where each instance provides a document, a target event type, and a specific trigger mention\. Given these inputs, the system must extract argument spans from the document and assign each span to a role that is valid for the target event type\. Our method is motivated by two recurring failure modes observed in LLM\-based EAE\. The first isschema inconsistency: models may hallucinate roles that are invalid for the given event type or introduce extra keys that are not part of the allowed role set\. This behavior reduces accuracy and can also break the strict interface expected by the official evaluator\. To address it, we explicitly inject the event\-specific role set into every prompt and apply schema filtering during post\-processing to remove any out\-of\-schema keys\.

The second failure mode isunreliable structured outputs and span faithfulness\. Model generations may contain invalid JSON, inconsistent structures, duplicated values, or argument strings that do not exactly match the document text\. This is particularly damaging in document\-level EAE, where a system must produce consistent structured predictions for many triggers in long documents\. To mitigate these issues, we use a strict JSON\-only instruction format, deterministic decoding, and a post\-processing pipeline that validates JSON, normalizes special cases \(e\.g\.,"NA"\), removes duplicates, and enforces alignment with the schema constraints required by evaluation\. Overall, our approach is implemented as a unified pipeline with five stages: task formulation, role\-set conditioning, supervised fine\-tuning, deterministic inference, and schema\-aware post\-processing\. We describe each stage below\. Figure[2](https://arxiv.org/html/2607.16808#S3.F2)summarizes our end\-to\-end pipeline fromMAVEN\-ARGinputs to submission\-ready predictions\.

Train/ValidMAVEN\-ARGTask formulation\+ roleset conditioningLoRA SFT\+Save adapters \(unsloth\)TrainingTest instance\(doc, event type, trigger\)\+ roleset\-injected promptDeterministicinference\(greedy, EOS stop\)Schema\-awarepost\-processing\(parse/filter/dedupe\)Submission\-readyJSON predictionsInference

Figure 2:Overview of our training and inference pipeline for event argument extraction\.### 3\.1Task formulation and output constraints

We define EAE as a document\-level extraction problem constrained by an event schema\. The system is given a documentdd, an event typet∈𝒯t\\in\\mathcal\{T\}, and a trigger mentionmmwith character offsets indd\. The model must produce a single JSON object that maps each valid roler∈R​\(t\)r\\in R\(t\)to a list of argument strings extracted from the document text\. The role setR​\(t\)R\(t\)is determined by the dataset schema and is treated as a hard constraint at inference time\.

The output must contain exactly one JSON object\. Each key must be a role name fromR​\(t\)R\(t\)and each value must be a list of strings\. If no argument is found for a role, the model returns the value"NA", which is normalized to an empty list during post\-processing\.

### 3\.2Schema conditioning via roleset induction and prompt design

Roleset induction for schema control\.To enforce schema compliance, we construct a mapping from event type to its role set\. We derive this mapping by scanning gold annotations in the training and development splits and collecting the roles observed for each event type\. The resulting mapping is stored in alabel2role\.jsonfile and is used both to populate the role set in prompts and to filter outputs during post\-processing\.

Prompt template with roleset injection and trigger marking\.All models use a three\-part prompt \(Instruction/Input/Response\), with the role set injected immediately beforeResponse\. To bind extraction to the correct event reference, we explicitly mark the trigger mention in the text window by enclosing the trigger string in<< \>\>\. We use the same prompt format for both supervised fine\-tuning and inference; only theResponsecontent differs \(gold JSON during training and an empty response during inference\)\.

```
### Instruction:
Extract the list of arguments for the given event type from the
text window.
Use only the roles provided in the Roleset line; do not invent
new roles.
Return ONLY a JSON object where each key is a role and each value
is a list of strings.
If no arguments are found, return NA. Do not include the event
type. Do not add explanations.

### Input:
{"doc_id": "...", "mention_id": "...", "event_type": "...",
 "trigger_char_span": [s,e], "trigger_text": "...",
 "window_offset": [ws,we],
 "text_window": " ... << trigger >> ... "}
Roleset: [Role_1, Role_2, ...]

### Response:
{ ... JSON only ... }
```

Figure 3:Three\-part prompt template \(Instruction / Input / Response\) with role\-set injection\. The trigger is marked with<< \>\>; at inference the Response section is left emptyThis prompt design exposes the model to the allowed role keys for the current event type, reducing role drift and the generation of irrelevant or extra keys\. The strictResponseheader and the JSON\-only instruction also reduce free\-form continuations and simplify automated parsing\.

### 3\.3Supervised fine\-tuning data construction and LoRA adaptation

#### 3\.3\.1Building supervised instances per trigger mention\.

For supervised fine\-tuning, we create one training instance per trigger mention\. For each trigger, we build a text window from the document and highlight the trigger span inside the window\. We then collect the gold arguments grouped by role and serialize them into a stable JSON structure that serves as the goldResponse\. Arguments inMAVEN\-ARGcan refer either to non\-entity spans \(directly annotated offsets\) or to entities with multiple coreferent mentions\. For non\-entity arguments, we directly extract the text using the annotated offsets\. For entity\-linked arguments, we select a single surface form from the available entity mentions to include as the argument string\. To match the local trigger context, we select the entity mention whose midpoint is closest to the trigger midpoint\.

Entity surface selection by trigger proximity\.After collecting arguments for each role, we remove duplicates within each role while preserving order, and serialize the role\-to\-argument mapping into JSON\. This output becomes the gold response for the same training example in the fine\-tuning phase\.

Input:entity mentions\{\(si,ei,texti\)\}i=1n\\\{\(s\_\{i\},e\_\{i\},\\textit\{text\}\_\{i\}\)\\\}\_\{i=1\}^\{n\}, trigger span\(st,et\)\(s\_\{t\},e\_\{t\}\)\.Computemt=st\+et2m\_\{t\}=\\frac\{s\_\{t\}\+e\_\{t\}\}\{2\}andmi=si\+ei2m\_\{i\}=\\frac\{s\_\{i\}\+e\_\{i\}\}\{2\}\.Selectj=arg⁡mini∈\{1,…,n\}⁡\|mi−mt\|j=\\arg\\min\_\{i\\in\\\{1,\\dots,n\\\}\}\|m\_\{i\}\-m\_\{t\}\|\.Return:textj\\textit\{text\}\_\{j\}\.

Parameter\-efficient fine\-tuning with Unsloth and LoRA\.We fine\-tune open\-source models using theUnslothframework with parameter\-efficient LoRA adapters\. We apply LoRA to both attention projection layers and feed\-forward submodules in each transformer block, includingq\_proj,k\_proj,v\_proj,o\_proj,gate\_proj,up\_proj, anddown\_proj\. We enable gradient checkpointing for memory efficiency and load base model weights in 4\-bit precision to support single\-GPU training\.

A key aspect of our setup is*interface alignment*: fine\-tuning uses exactly the same strict prompt format used at inference time, including role\-set injection and JSON\-only outputs\. This reduces train–test mismatch and makes the model less likely to deviate from the required structured output format at test time\.

Deterministic inference and prompt preservation\.

During inference, we use greedy decoding \(temperature=0=0\) to obtain deterministic and repeatable outputs, enabling more controlled comparisons across different model backbones\. We configure the tokenizer with left truncation and left padding so that, for long text windows, theResponseheader remains close to the end of the input sequence\. Preserving this response anchor is important because the model must reliably start generating immediately after theResponseheader and produce JSON only\.

Inference prompts are constructed identically to training prompts, except that theResponsesection is left empty\. The role set is injected fromlabel2role\.jsonfor each event type, ensuring that the model always receives explicit schema constraints\.

Post\-processing, schema validation, and submission formatting\.After generation, predictions are post\-processed to ensure they are valid, schema\-consistent, and compatible with the benchmark submission format\. This step is necessary because even strong models may produce formatting errors or out\-of\-schema keys that would otherwise prevent evaluation\.

The post\-processing pipeline includes:

- •JSON extraction and validation\.We parse the output as JSON; we do not attempt to repair malformed generations, and any parsing failure is discarded and treated as an empty prediction\.
- •Normalization of missing values\.We normalize"NA"to an empty list to keep value types consistent\.
- •Event\-type attachment\.We attach the correspondingevent\_typeusing an auxiliary mapping to match the benchmark submission format\.
- •Schema filtering\.We remove any keys that are not in the injected role set\.
- •De\-duplication and merging\.Within each role, we remove duplicate argument strings; if multiple generations produce overlapping outputs for the same reference, we merge them\.
- •Window alignment\.We verify that each predicted span occurs within the provided text window; spans not found in the window are discarded\.

Source document schema \(one dataset record\)Expected inference output schema\{ "id":"364ed14fc610df6e25a2f446e2b2d2ab","title":"Expedition of the Thousand","document":"The Expedition of the Thousand \(Italian ‘‘Spedizione deiMille’’\) was an event of the Italian Risorgimento that took place in1860\. A corps of volunteers led by Giuseppe Garibaldi sailed fromQuarto, near Genoa \(now Quarto dei Mille\), \.\.\.\.","events": \[\{ "id":"EVENT\_451b7cde13d2b8c21426db027c51096f","type":"Motion","type\_id":46,"mention": \[\{ "id":"1c6807377cd111b86d2d3afc85bf2ca2","trigger\_word":"sailed","offset":\[185, 191\]\}\],"argument": \{"Agent": \[\{ "content":"a corps of volunteers led by GiuseppeGaribaldi","offset":\[137, 184\]\}\],"Location\_original": \[\{ "entity\_id":"ENTITY\_052c8614c4bb87b5321807d1980332f6"\}\],"Location\_final": \[\{ "entity\_id":"ENTITY\_185734cbbe4da16fb0782dfaa326529c"\}\]\}\]\}\{"id":"364ed14fc610df6e25a2f446e2b2d2ab","preds": \{"1c6807377cd111b86d2d3afc85bf2ca2": \{"event\_type":"Motion","Agent": \["a corps of volunteers led byGiuseppe Garibaldi"\],"Location\_original": \["Quarto dei Mille"\],"Location\_final": \["Marsala, Sicily"\]\},"187955a9227516bae5cf7198e30a3cd4": \{"event\_type":"Motion","Agent": \["a corps of volunteers led byGiuseppe Garibaldi"\],"Location\_original": \["Quarto, near Genoa"\],"Location\_final": \["Marsala, Sicily"\]\}\}\}Open LLMsInputExpected Output

Figure 4:Source document schema and expected inference output schema\. Red boxes highlight metadata and trigger fields \(e\.g\.id,trigger\_word,offset\); blue boxes highlight the schema argument roles \(e\.g\.Agent,Location\_original,Location\_final\)As shown in Figure[4](https://arxiv.org/html/2607.16808#S3.F4), post\-processing enforces schema consistency and structured\-output validity, reducing preventable errors such as malformed JSON, redundant or irrelevant keys, and duplicated values\. As a result, the system produces structured JSON outputs in the exact format required by the official evaluator\.

## 4Evaluation

This section describes the experimental setup used to evaluate LLMs for document level event argument extraction \(EAE\) on MAVEN\-ARG\. We summarize the dataset and its schema, explain how entity and event coreference affect evaluation, detail the official scoring and normalization rules, and present the compared models and baselines\. Our main research question is whether mid\-sized open LLMs, when adapted via supervised fine\-tuning and constrained by schema\-conditioned prompts, can match or surpass previously reported few\-shot GPT baselines on MAVEN\-ARG under the official evaluation protocol, and how performance changes across mention\-, entity\-coreference\-, and event\-coreference\-based evaluation levels\.

### 4\.1Dataset and dataset schema

In all experiments, we use MAVEN\-ARG\[wang\-etal\-2024\-maven\], a document\-level EAE dataset derived from MAVEN and built from English Wikipedia articles\. MAVEN\-ARG provides full argument annotations for both entity\-linked and non\-entity spans and is designed to support cross\-sentence extraction with full\-document context\. The dataset defines an extensive nested JSON schema with 162 event types and 612 argument roles and includes expert\-written definitions and human annotations aligned to character offsets\.

The training and development splits contain full document annotations\. Each document provides raw text, a list of events, and a list of entities\. Each event specifies an event type and contains one or more trigger mentions with offsets, along with an argument dictionary whose keys are roles\. Arguments are represented in two forms: \(i\) entity\-linked arguments that refer to an entity identifier associated with a within\-document coreference cluster, and \(ii\) non\-entity arguments recorded as free spans with surface text and character offsets\. The test split preserves blind evaluation: it provides event and entity references, but gold arguments and coreference links required for scoring are kept private, and evaluation is performed by submitting predictions in the required JSON format to the official online platform\. We follow the official split with 2,913 training documents, 710 development documents, and 857 test documents, containing 70,775, 16,996, and 19,736 events, respectively\[wang\-etal\-2024\-maven\]\.

### 4\.2Coreference annotations and evaluation metrics and levels

MAVEN\-ARG includes entity and event coreference: entities form mention clusters and events form trigger clusters referring to the same underlying event\[wang\-etal\-2024\-maven\], enabling evaluation beyond strict mention matching\.

We report results at the three levels supported by the official scorer\.Mention\-levelevaluation scores each trigger mention independently by comparing its predicted role\-to\-argument mapping to the gold mapping\.Entity\-coreferenceevaluation relaxes surface\-form matching: a prediction can be counted as correct if it matches any coreferent mention of the gold entity, reducing sensitivity to which mention string is produced\.Event\-coreferenceevaluation aggregates trigger mentions that belong to the same event cluster and evaluates arguments at the cluster level, rewarding systems that retrieve correct arguments consistently across multiple coreferent triggers\. Together, these levels reflect the document\-level nature of MAVEN\-ARG and measure robustness to coreference and cross\-sentence dependencies\.

### 4\.3Official metrics and normalization

All reported scores follow the official MAVEN\-ARG evaluation procedure\[wang\-etal\-2024\-maven\]\. We report Precision, Recall, F1, and Exact Match at mention, entity\-coreference, and event\-coreference levels\. Prior to scoring, both gold and predicted argument strings are normalized by lowercasing, removing punctuation, and collapsing extra whitespace\. Exact Match requires the normalized predicted string to exactly equal the normalized gold string\. In addition to Exact Match, the scorer computes bag\-of\-words similarity based on word overlap\.

At the mention level, scoring uses role\-wise bipartite matching between the predicted and gold argument lists\. For each role, the scorer constructs a similarity matrix between items in the gold and predicted lists and selects the best one\-to\-one matching to maximize the total similarity\. The scorer also applies a cardinality penalty when the numbers of predicted and gold arguments differ, penalizing over\- and under\-generation\. Scoring is schema\-aware: only roles valid for the gold event type can contribute positively, and out\-of\-schema role keys do not yield credit\. Entity\-coreference and event\-coreference scoring extend this procedure by allowing matches against coreferent entity mentions and by aggregating across coreferent trigger mentions, respectively\.

### 4\.4Models and baselines

We evaluate several mid\-sized open LLMs using supervised fine\-tuning with the schema\-conditioned prompting interface described in Section 3\. The open models include Llama\-3\.1 \(8B\), Mistral\-Nemo \(12B\), Qwen3 \(14B\), and Phi\-4 \(14B\)\. For all open models, we inject the event\-specific role set into the prompt, require JSON\-only outputs, decode deterministically, and apply the same validation and schema\-filtering post\-processing prior to evaluation\. At inference theResponsesection is left empty, the role set appears only in theRolesetline of the prompt, and we do not prefill the answer keys\.

As baselines, we report GPT\-3\.5 and GPT\-4 results previously reported on MAVEN\-ARG using two\-shot in\-context learning, both with and without adding event/role definitions to the prompt\[wang\-etal\-2024\-maven\]\. These baselines serve as reference points for closed\-model few\-shot prompting without supervised fine\-tuning on MAVEN\-ARG\. We do not additionally evaluate the open models under few\-shot prompting, since our scope is the fine\-tuned schema\-conditioned setting, and few\-shot evaluation of open models is left to future work\. Since we do not re\-run these systems, differences in prompt templates, context\-length limits, and decoding settings across studies may affect strict comparability, so we interpret comparisons to published closed\-model results as indicative rather than strictly controlled, and we do not claim a fully controlled head\-to\-head comparison against closed models\.

### 4\.5Compute environment and reproducibility protocol

Training and inference are performed on an NVIDIA Tesla T4 GPU \(16 GB VRAM\) with CUDA support\. To fit models within hardware constraints, base weights are loaded with 4\-bit quantization during fine\-tuning and inference\. Our implementation uses Python 3\.12 and PyTorch 2\.8\.0 \(CUDA\), along with standard libraries for training and inference, includingtransformers,trl,peft, andunsloth\.

We keep the testing protocol constant across all open models\. We construct the event\-type\-to\-role\-set mapping from the training and development splits, build supervised instances at the trigger\-mention level using the same strict Instruction/Input/Response format with role\-set injection, and decode deterministically\. Generated outputs are post\-processed via a JSON extraction pipeline, normalization, and schema filtering to produce predictions in the required format by the official evaluator\. Final test scores are obtained from the official MAVEN\-ARG online evaluation platform using the required submission format\.

### 4\.6Implementation details

We perform supervised fine\-tuning with LoRA using theunslothtraining stack\. Unless otherwise specified, we use a maximum input length of2048tokens and a maximum generation length of512new tokens at inference time\. For sequences longer than2048tokens, we rely on standard tokenizer truncation to the maximum sequence length\. We decode deterministically using greedy decoding withtemperature=0=0,top\-pp=1\.0, a repetition penalty of1\.05and stop at the end\-of\-turn token<\|𝚒𝚖​\_​𝚎𝚗𝚍\|\>\\mathtt\{<\|im\\\_end\|\>\}\(EOS\)\.

LoRA adapters use rankr=128r=128,α=32\\alpha=32, and dropout=0=0, applied toq\_proj,k\_proj,v\_proj,o\_proj,gate\_proj,up\_proj,down\_proj\. We train with AdamWadamw\_8bitat a learning rate2e\-4, batch size8with gradient accumulation4, for60optimization steps with linear scheduling, with warmup ratio5warmup steps and weight decay0\.01\. We use BF16 when supported \(otherwise FP16\), gradient checkpointing, a max grad norm of1\.0and load base weights in 4\-bit quantization on a single NVIDIA Tesla T4 GPU \(16 GB VRAM\)\. We note that this corresponds to a low\-data regime, since at batch size8with gradient accumulation4over60steps, fine\-tuning uses roughly1,920trigger\-level instances, on the order of3%of the70,775training events\. Further details regarding the implementation and experimental setup can be found in the accompanying GitHub repository\.

## 5Results and Discussion

In this section, we report results on the MAVEN\-ARG test set using the official evaluator and discuss the main performance patterns observed across models\. We compare mid\-sized open LLMs fine\-tuned with LoRA under the schema\-conditioned prompting interface \(Section 3\) against previously reported GPT\-3\.5 and GPT\-4 baselines on MAVEN\-ARG\. We evaluate at three official levels:mention,entity coreference, andevent coreference\. All metrics are micro\-averagedPrecision,Recall,F1, andExact Match \(EM\)\.

### 5\.1Overall test results

Table[1](https://arxiv.org/html/2607.16808#S5.T1)summarizes official test\-set results\. Open models are fine\-tuned with LoRA using schema\-conditioned prompts and JSON\-only outputs, while GPT results are reported from prior work as two\-shot prompting baselines, both with and without adding role definitions to the prompt\[wang\-etal\-2024\-maven\]\.

Table 1:Test\-set results on MAVEN\-ARG\. All numbers are percentages\.Mention Level

Entity Coref Level and Event Coref Level

Across all three evaluation levels, fine\-tuned open LLMs substantially outperform the reported few\-shot GPT baselines\. Among open models,Phi\-4 \(14B\)is the most robust overall, achieving the best event\-coreference F1 \(42\.39%\)\. Mistral\-Nemo and Qwen3 also perform strongly at the event\-coreference level, followed by Llama\-3\.1\. Notably, a mid\-sized open model \(Phi\-4, 14B\) achieves the best event\-coreference F1, outperforming the published GPT baselines reported for MAVEN\-ARG under the official evaluator\.

We observe that F1 scores are consistently higher than Exact Match, indicating that many predictions are*near matches*to the reference strings \(high token overlap\) but do not reproduce the gold span exactly\. Finally, note that the official scorer uses Hungarian \(bipartite\) matching with a cardinality penalty, so the reported micro\-averaged F1 is not guaranteed to equal the harmonic mean of the displayed global precision and recall and can exceed both\.

### 5\.2Comparison against GPT baselines

Figure[5](https://arxiv.org/html/2607.16808#S5.F5)highlights event\-coreference F1 gains over the strongest reported GPT baseline \(GPT\-4 \+ definition\)\. All fine\-tuned open LLMs show large improvements under the official evaluator, suggesting that supervised adaptation with a strict schema\-conditioned interface is more effective than few\-shot prompting alone for this benchmark\.

![Refer to caption](https://arxiv.org/html/2607.16808v1/Figure/results-plot4.png)Figure 5:F1 performance on MAVEN\-ARG: GPT baselines vs\. Open LLMs
### 5\.3Precision–recall profiles at the event\-coreference level

Models show different precision–recall trade\-offs at the event\-coreference level\. Phi\-4, Mistral\-Nemo, and Llama\-3\.1 are more precision\-oriented, indicating more conservative extraction \(fewer spurious arguments\)\. In contrast, Qwen3 is recall\-biased, extracting more candidate arguments and thus more false positives, consistent with its precision–recall gap in Table[1](https://arxiv.org/html/2607.16808#S5.T1)\.

### 5\.4Exact Match versus near matches

The gap between bag\-of\-words F1 and Exact Match indicates that span boundaries and surface\-form fidelity remain important sources of error\. At entity\- and event\-coreference levels, Llama\-3\.1, Mistral\-Nemo, and Phi\-4 show clear F1–EM gaps: many predictions are close to the reference \(earning token\-overlap credit\) but fail Exact Match due to boundary differences, modifiers, or small surface variations\. In contrast, Qwen3 shows a smaller F1–EM gap at the event\-coreference level; its main weakness is precision rather than boundary fidelity, consistent with the model producing additional plausible but non\-gold arguments\.

### 5\.5Qualitative case studies

To connect metrics to model behavior, we analyze the qualitative examples in Figure[6](https://arxiv.org/html/2607.16808#S5.F6)\. In theKillingexample \(trigger: “massacre”\), systems often identify correct roles but differ in span selection, sometimes extracting shorter spans or near matches that are not faithful to the gold text\. In theAchieveexample \(trigger: “achieved”\), models more consistently identifyAgentandGoal, and predictions tend to be closer to reference strings\. In theAimingexample \(trigger: “targeted”\), systems differ in role coverage: some models output many roles from the injected role set, while others only output roles they are confident about, illustrating that role coverage remains challenging even with the same text window and schema constraints\.

![Refer to caption](https://arxiv.org/html/2607.16808v1/Figure/qualitative7.jpg)Figure 6:Qualitative examples of event argument extraction
### 5\.6Qualitative error signatures from metric patterns

The metric profiles suggest two dominant error signatures\. First, Qwen3 combines high recall with strong EM at the event\-coreference level but lower precision, consistent with generating additional plausible arguments that are not present in the gold set\. Second, Phi\-4, Mistral\-Nemo, and Llama\-3\.1 exhibit larger F1–EM gaps, consistent with near\-correct predictions that differ from the gold by span boundaries, extra qualifiers, punctuation, or minor surface\-form variations\. Token\-overlap scoring gives partial credit for these near matches, while Exact Match penalizes them\.

### 5\.7Key takeaways

Overall,Phi\-4 \(14B\)achieves the best performance at the event\-coreference level under the official evaluator\. Fine\-tuned open LLMs substantially improve over the reported few\-shot GPT baselines, benefiting from supervised adaptation and strict schema\-conditioned structured outputs\. At the same time, models exhibit different precision–recall trade\-offs, and the remaining gap between F1 and Exact Match indicates that span boundary fidelity remains a major source of error even when token\-overlap scores are high\.

## 6Conclusion

This paper studies document\-level event argument extraction with LLMs under strict schema constraints and coreference\-aware evaluation\. OnMAVEN\-ARG, we compare previously reported GPT\-3\.5/GPT\-4 two\-shot baselines \(with/without role definitions\) to lightweight open LLMs adapted via supervised fine\-tuning\. Our pipeline uses role\-set conditioned prompting, deterministic decoding, and JSON\-validating post\-processing\. Under the official evaluator, fine\-tuned open models consistently outperform the reported GPT baselines at the mention, entity\-coreference, and event\-coreference levels, showing that mid\-sized models can match or exceed published closed\-model few\-shot results\. Results also confirm that coreference\-aware aggregation improves end\-to\-end performance, while the gap between bag\-of\-words F1 and Exact Match indicates that span\-boundary fidelity and surface\-form variation remain major sources of error\.

We envision three directions\. First, we will evaluate reasoning\-oriented LLMs under the same pipeline and metrics \(e\.g\., stronger Phi variants and DeepSeek\-R1\[Guo\_2025\]\) to test whether multi\-step reasoning improves span fidelity and role assignment\. Second, we will benchmark additional open backbones \(for instance,gpt\-oss\-20b\[openai2025gptoss120bgptoss20bmodel\]and newer LLaMA releases\) while controlling context length, decoding budget, and instruction\-tuning style\. Third, we will extend this approach to privacy\-sensitive domains such as healthcare, emphasizing local deployment and reproducible workflows aligned with FAIR principles\.

#### 6\.0\.1\\discintname

The authors have no competing interests to declare that are relevant to the content of this article\.

## References