DistilledGemma: Balanced Efficiency-Accuracy for Person-Place Relation Extraction from Multilingual Historical Articles

# DistilledGemma: Balanced Efficiency-Accuracy for Person-Place Relation Extraction from Multilingual Historical Articles
Source: [https://arxiv.org/html/2606.29130](https://arxiv.org/html/2606.29130)
\\conference

The Conference and Labs of the Evaluation Forum \(CLEF\) 2026

\[email=es\-YoussefHossamElDin2025@alexu\.edu\.eg\]\\fnmark\[1\]

\[email=es\-AhmedAbdelMaksoud2025@alexu\.edu\.eg\]\\fnmark\[1\]

\[email=nagwamakky@alexu\.edu\.eg\]

\[email=mtorki@alexu\.edu\.eg\]

\\fntext

\[1\]Equal contribution\.

###### Abstract

We presentDistilledGemma, an efficient and accurate system for theHIPE\-2026shared task on person\-place relation extraction from multilingual historical newspaper articles in English, German, and French\. Our approach adopts a three\-stage knowledge distillation pipeline designed to balance classification accuracy with computational efficiency\. In the first stage, we systematically explored prompt engineering strategies across eight large language models to identify the most effective reasoning architecture for this challenging task\. In the second stage, we applied supervised fine\-tuning \(SFT\) via QLoRA to aGemma 4 26B A4Bteacher model, leveraging its strong multilingual capabilities to generate silver\-standard chain\-of\-thought traces across the training corpus\. In the final stage, we performed response\-level distillation to transfer these learned reasoning patterns into a compactGemma 4 E2Bstudent model\. To address the class imbalance between positive and negative relation pairs and to enforce semantic consistency, we incorporated rule\-based post\-processing on all generated outputs\. Specifically, this step enforced entity\-type compatibility, relation\-direction constraints, contextual trigger patterns, and the logical constraint\. In the official evaluation, our teamWHEREAMIranked3rdon the standard test set with an accuracy profile mean score of0\.688, and2ndon the binary test set with a mean score of0\.8156\. Notably, by distilling knowledge from the 26B teacher to the 2\.3B student, we preserved strong reasoning capabilities while reducing the deployed model size to approximately2\.3B effective parameters; the LoRA adapters used during training were merged into the student for inference\. This configuration ranked2ndin the balanced efficiency\-accuracy profile across both the standard and binary test sets\. These results demonstrate that knowledge distillation provides a practical and scalable solution for historical document processing, achieving competitive performance without excessive computational cost\.

## 1Introduction

Historical newspapers constitute invaluable resources for studying past societies, migration patterns, and geopolitical events\. A critical step in unlocking this information isperson\-place relation extraction: determining whether a person mentioned in an article bears a geographic connection to a place also mentioned therein\.

The HIPE\-2026 shared task\[opitz2026clef,opitz\_overview\_2026,opitz\_extended\_2026\]formalizes this problem as a classification task over \(person, place\) pairs extracted from multilingual newspaper articles in English, German, and French\. Each pair must be labelled for two relations:at\(whether the person has*ever*been associated with the place;TRUE/PROBABLE/FALSE\) andisAt\(whether the person is at the place*within the immediate temporal horizon of the article*;TRUE/FALSE\) as shown in Figure[1](https://arxiv.org/html/2606.29130#S1.F1)and[2](https://arxiv.org/html/2606.29130#S1.F2)\.

![Refer to caption](https://arxiv.org/html/2606.29130v1/figures/at_isat.png)Figure 1:Illustration of the distinction betweenatandisAtrelations\.![Refer to caption](https://arxiv.org/html/2606.29130v1/figures/article_example.png)Figure 2:Illustration of the distinction betweenatandisAtrelations in HIPE\-2026\. Theatrelation captures historical or general associations, whileisAtdenotes the person’s location at the time described in the article\.This task presents several challenges:

- •Noisy OCR text: Historical documents contain frequent optical character recognition errors that degrade entity mention quality\.
- •Class imbalance: TheFALSElabel dominates both relations \(∼\{\\sim\}55% forat,∼\{\\sim\}78% forisAt\), biasing models toward negative predictions\.
- •Multilingual reasoning: Models must perform consistently across three typologically diverse languages with varying training data availability\.
- •Implicit evidence: Many person\-place connections are implied rather than explicitly stated, requiring world knowledge and contextual inference\.

Large language models \(LLMs\) have demonstrated impressive zero\-shot and few\-shot capabilities for relation extraction\[wadhwa2023revisiting,wan2023gpt\], but models with sufficient capacity \(e\.g\., 26B\+ parameters\) are impractical for deployment in digital humanities pipelines that must process millions of documents\. This motivates our central research question:*Can we distill the relation extraction capabilities of a large, fine\-tuned LLM into a model small enough for practical deployment without catastrophic performance loss?*

Our contributions are as follows:

1. 1\.Acomprehensive evaluationof eight LLM configurations spanning prompt engineering, zero\-shot inference, and supervised fine\-tuning for person\-place relation extraction across three languages\.
2. 2\.Athree\-stage distillation pipelinethat fine\-tunes a Gemma 4 26B A4B teacher via QLoRA, generates silver\-standard training data with chain\-of\-thought \(CoT\) reasoning, and trains a Gemma 4 E2B student on the teacher’s outputs\.
3. 3\.Empirical evidencethat knowledge distillation recovers∼\{\\sim\}88% of teacher performance, establishing a practical efficiency\-accuracy trade\-off for historical NLP applications\.

## 2Related Work

#### Encoder\-based relation extraction\.

Before the recent shift to generative LLMs, relation extraction was commonly framed as discriminative classification over marked entity pairs, using either hand\-engineered features\[zhang\-etal\-2017\-position\]or contextual encoders\[pathak2016context\]\. These lightweight baselines remain important for low\-resource and imbalanced settings because they condition directly on entity spans and require only a small task head\. Entity\-aware representation learning, including task\-agnostic relation representations from entity\-linked text\[baldini2019matching\], typed entity\-marker models\[zhou2022improved\], and PL\-Marker span\-pair packing\[ye2022packed\], provides strong baselines for sentence\-level RE\. Cross\-lingual encoders such as XLM\-R\[conneau2020xlmr\]and entity\-aware multilingual encoders such as mLUKE\[ri2022mluke\]are especially relevant for multilingual HIPE\-style data\. For longer article\-level evidence, document\-level RE benchmarks such as DocRED highlight the need to synthesize evidence across sentences and entity mentions\[yao2019docred\]\. Recent multilingual and digital\-humanities RE work has also used guided distant supervision to build German biographical relation data and evaluate multilingual transfer\[plum2024guided\]\. Low\-resource RE benchmarks show that class balancing, data augmentation, and self\-training are useful alternatives, but their gains can be inconsistent under long\-tailed label distributions\[xu2022towards\]\. These approaches are therefore strong lightweight baselines and complementary alternatives to our LLM distillation pipeline\.

#### Relation extraction with LLMs\.

Recent work has demonstrated that LLMs can perform competitive relation extraction through in\-context learning\[wan2023gpt\]and chain\-of\-thought prompting\[wei2022chain\], particularly when explicit reasoning steps are elicited before label prediction\. Wadhwa et al\.\[wadhwa2023revisiting\]showed that GPT\-class models match or exceed supervised baselines on standard benchmarks, albeit at significantly higher computational cost\.

#### Historical NLP\.

The HIPE shared task series\[hipe2022,ehrmann2023extended\]has driven substantial progress in named entity recognition and linking for historical texts, establishing standardized benchmarks across multiple languages and time periods\. These tasks highlight unique challenges including OCR noise, archaic language usage, and domain\-specific entity types that are absent from modern NLP benchmarks\. The HIPE\-2026 edition\[opitz\_overview\_2026\]extends the series to person\-place relation extraction, introducing new evaluation profiles that jointly consider accuracy and computational efficiency\.

#### Knowledge distillation and teacher alignment\.

Hinton et al\.\[hinton2015distilling\]introduced knowledge distillation \(KD\) as a model compression technique in which a compact student network learns to mimic a larger teacher’s output distribution\. For generative models, sequence\-level KD trains students on complete teacher outputs rather than only token\-level distributions\[kim2016sequence\]\. This paradigm has been widely adopted for language models: DistilBERT\[sanh2019distilbert\]compresses BERT while retaining 97% of its performance, and MiniLLM\[gu2024minillm\]extends KD to autoregressive LLMs with a policy\-level reverse KL objective\. On\-policy approaches such as GKD\[agarwal2024gkd\]further improve upon standard KD by training on the student’s own generations and allowing alternative KL divergences\. Closer to our setting, rationale\- and explanation\-trace distillation methods show that small models can benefit from LLM\-generated reasoning supervision\[hsieh2023distilling,mukherjee2023orca\]\. Preference\-based alignment methods such as DPO\[rafailov2023dpo\]provide another route by optimizing pairwise teacher or human preferences rather than directly imitating a single teacher response\. In low\-resource RE, self\-training and confidence calibration are also plausible alternatives; for example, PRiSM calibrates document\-level RE logits with relation\-aware scores when only a small amount of labeled data is available\[choi2023prism\]\. In our work, we adopt a*response\-level*distillation strategy in which the teacher’s evidence\-grounded explanations and label outputs serve as silver training data for the student, combining structured reasoning transfer with the simplicity of standard SFT and avoiding the extra preference construction or confidence\-threshold tuning required by these alternatives\.

#### Parameter\-efficient fine\-tuning\.

LoRA\[hu2022lora\]and its quantized variant QLoRA\[dettmers2023qlora\]enable fine\-tuning of billion\-parameter models on consumer hardware by training only low\-rank adapter matrices while keeping the base model weights frozen\. We leverage QLoRA throughout our pipeline for both teacher and student training, making the entire workflow feasible on a single GPU\.

## 3Approach

Our approach follows a three\-stage pipeline, illustrated in Figure[3](https://arxiv.org/html/2606.29130#S3.F3): \(1\) model selection through systematic experimentation, \(2\) teacher fine\-tuning and silver data generation, and \(3\) student distillation\.

![Refer to caption](https://arxiv.org/html/2606.29130v1/figures/pipeline.png)Figure 3:Overview of the DistilledGemma three\-stage pipeline\. First, multiple LLM configurations and prompting strategies are evaluated to select the strongest teacher model\. The teacher is then fine\-tuned and used to generate chain\-of\-thought silver annotations, which are subsequently used to distill knowledge into a compact student model\.### 3\.1Stage 1: Model Selection and Prompt Engineering

We evaluated a diverse set of LLM families in multiple configurations:

#### Prompt engineering\.

We designed five prompt variants ranging from a minimal zero\-shot baseline to structured chain\-of\-thought templates that guide the model through explicit evidence assessment before emitting labels\. The best\-performing prompt \(chain\-of\-thought\), shown in Listing[1](https://arxiv.org/html/2606.29130#LST1), was used as the inference prompt for all LLM candidates to predict theatandisAtrelations during base\-model selection\. It frames the model as a historian, defines the two relation labels, emphasizes OCR robustness and conservative inference, enforces the logical dependency between labels, and requires concise evidence\-grounded JSON outputs\.

Youareahistorianworkingonperson\-locationrelationsinmultilingualhistorical

Europeannewspapers\.

Readcarefully,expectOCRnoise,andbaseeveryjudgmentonlyonthedocumenttextand

metadatagivenhere\.

Task:

\-Classifyrelation"at"forthepersonandplace\.Allowedlabels:TRUE,PROBABLE,FALSE

\-Classifyrelation"isAt"forthepersonandplace\.Allowedlabels:TRUE,FALSE

Relationmeanings:

\-"at"=thearticlegivestextualevidencethatthepersonwasatthelocationatsome

pointbeforepublication\.

\-"isAt"=thearticlesupportsthatthepersonwasatthelocationinthearticle’s

immediatetemporalhorizon,meaningcurrent,ongoing,orveryrecentrelativetothe

publicationdate\.

Decisionguidance:

\-UseTRUEfor"at"onlywhenthetextgivesclearevidenceofpresence\.

\-UsePROBABLEfor"at"onlywhenthetextgivesindirect,partial,orweakevidencethat

stillpointstowardpresence\.

\-UseFALSEfor"at"whenthearticledoesnotsupportpresence\.

\-UseTRUEfor"isAt"onlywhenthearticleclearlyplacesthepersonthereinacurrent,

ongoing,orveryrecentframe\.

\-NeverusePROBABLEfor"isAt"\.ItmustbeTRUEorFALSE\.

\-If"at"isFALSE,then"isAt"mustalsobeFALSE\.

\-If"isAt"isTRUE,then"at"mustalsobeTRUE\.

Rules:

\-Donotuseexternalknowledge\.

\-Donotinfermorethanthetextwarrants\.

\-PreferFALSEwhenevidenceismissingortoouncertain\.

\-ReturnJSONonly\.

\-Usetheexactpersonandplacestringsgivenforthecurrentpair\.

\-BerobusttoOCRnoiseandline\-breakartifactsinthetextandintheentitymentions\.

\-Treatclearlyequivalentmentionsurfacesasthesameentityeveniftheydifferbyescapedlinebreaks,hyphenation,orminorOCRspellingnoise\.

\-Firstwrite‘at\_explanation‘,thendecide‘at‘\.

\-Thenwrite‘isAt\_explanation‘,thendecide‘isAt‘\.

\-Keepeachexplanationconcise,atmost100words\.

\-Eachexplanationmustmentiononlyevidencefromthearticle,notyourreasoningprocess\.

Documentlanguage:\{language\}

Publicationdate:\{publication\_date\}

Articletext:

\{article\_text\}

\{pair\_context\}

Listing 1:Prompt template used for LLM inference, teacher generation, and student fine\-tuning\.
#### Model families\.

We evaluated models from four families:Gemma\[gemma2team2024\]\(E2B, E4B, and 26B A4B\),Qwen\[qwen3\]\(2B to 9B\),Mistral\[mistral2024ministral\]\(3B\), andHY\-MT\[zheng2025hymt\]\(1\.8B\)\. Models were served via vLLM\[kwon2023vllm\]for efficient batch inference\.

#### Training strategies\.

Beyond zero\-shot prompting, we explored supervised fine\-tuning \(SFT\) using LoRA\[hu2022lora\]on the HIPE training data\. The SFT data was prepared by converting each labeled \(person, place\) pair into a chat\-format record with the prompt as the user message and the gold JSON as the assistant response\.

### 3\.2Stage 2: Teacher Fine\-Tuning

We evaluatedGemma 4 26B A4B\[gemma4modelcard2026\]using the same prompt as in the selection stage\. We selected it as the teacher model because it achieved the strongest zero\-shot macro\-averaged recall \(0\.7255 across languages\) and belongs to the same model family as the target student\.

The teacher was fine\-tuned using QLoRA\[dettmers2023qlora\]with the following configuration:

- •LoRA: rankr=32r\{=\}32,α=64\\alpha\{=\}64, dropout0\.10\.1, applied to all attention and MLP projections \(q/k/v/o/gate/up/down\)\.
- •Training: 3 epochs, batch size 1 with gradient accumulation 32 \(effective batch size 32\), cosine learning rate schedule withη=2×10−4\\eta\{=\}2\{\\times\}10^\{\-4\}and 5% warmup\.
- •Data: All labeled pairs from the newspaper and sandbox train splits across EN/DE/FR, with a 90/10 stratified train/dev split\.

After training, the LoRA adapter was merged into the base model for efficient inference\. The merged teacher was then used to generate silver\-standard labels over the entire training corpus, producing chain\-of\-thought explanations alongsideatandisAtpredictions for each pair\.

### 3\.3Stage 3: Student Distillation

The student model \(Gemma 4 E2B\) was trained on the teacher\-generated silver data using QLoRA with a lighter configuration:

- •LoRA: rankr=16r\{=\}16,α=32\\alpha\{=\}32, dropout0\.050\.05\.
- •Training: 6 epochs \(longer training compensates for the smaller model capacity\), effective batch size 16\.
- •Data: Teacher\-generated distilled JSONL, where the assistant messages contain the teacher’s chain\-of\-thought with its predicted outputs rather than gold labels\.

This*response\-level distillation*approach transfers not only the teacher’s label decisions but also its reasoning patterns, enabling the student to learn structured inference over noisy historical text\. The student learns to emulate the teacher’s chain\-of\-thought process, which we hypothesize improves generalization compared to training on gold labels alone, as the latter lack detailed explanatory reasoning for many relation pairs\.

For parameter accounting, we count only the deployable student model: the 26B teacher is used offline to create silver data, and the LoRA matrices are training\-time adapters that are merged into the 2\.3B base model for inference\. Consequently, the efficiency profiles and compression ratios report the student as a 2\.3B\-parameter deployed model rather than summing teacher and student parameters or treating the adapters as a separate multi\-billion\-parameter component\.

### 3\.4Post\-Processing

To address the imbalance between positive and negative relation pairs and ensure semantic consistency in model outputs, we applied rule\-based post\-processing to all generated predictions\. Specifically, this step enforced: \(1\) entity\-type compatibility between the predicted relation and the entity types involved, \(2\) relation\-direction constraints ensuring the correct mapping of person\-to\-place, \(3\) contextual trigger patterns that leverage surface\-level cues in the article, and \(4\) the logical constraint thatisAt=TRUE⇒at=TRUE\\text\{isAt\}\{=\}\\textsc\{TRUE\}\\Rightarrow\\text\{at\}\{=\}\\textsc\{TRUE\}\.

## 4Dataset

The HIPE\-2026 training corpus contains multilingual historical newspaper articles in German, English, and French with varying document lengths and entity distributions\. Table[1](https://arxiv.org/html/2606.29130#S4.T1)summarizes corpus\-level statistics, while Tables[2](https://arxiv.org/html/2606.29130#S4.T2)and[3](https://arxiv.org/html/2606.29130#S4.T3)report relation and entity characteristics\.

Table 1:Dataset overview by language\.LanguageDocumentsAvg\. WordsAvg\. CharactersGerman34744\.15000\.9English35325\.81802\.7French35585\.93635\.0Total104––Table 2:Relation label distribution across languages with overall statistics and class imbalance percentages\. The label distribution exhibits significant class imbalance: forat, FALSE accounts for∼\\sim55%, PROBABLE for∼\\sim10%, and TRUE for∼\\sim35%; forisAt, FALSE dominates at∼\\sim78% versus TRUE at∼\\sim22%\.LabelGermanEnglishFrenchOverall \(percentage\)at relationTRUE135125181441 \(35\.25%\)FALSE269152269690 \(55\.16%\)PROBABLE623028120 \(9\.59%\)isAt relationTRUE8983107279 \(22\.30%\)FALSE377224371972 \(77\.70%\)Table 3:Unique entity statistics\.LanguageUnique PersonsUnique LocationsGerman174217English120127French209237
## 5Experiments

### 5\.1Experimental Setup

All models were served using vLLM\[kwon2023vllm\]with temperature0\.00\.0and seed4242for output determinism\. We report macro\- and micro\-averaged recall following standard classification evaluation practice\[sokolova2009systematic\]\. The logical constraint \(isAt=TRUE⇒at=TRUE\\text\{isAt\}\{=\}\\textsc\{TRUE\}\\Rightarrow\\text\{at\}\{=\}\\textsc\{TRUE\}\) was enforced as a post\-processing step for all configurations\.

### 5\.2Results

Table[4](https://arxiv.org/html/2606.29130#S5.T4)presents the results across all eight model configurations\.

ModelENDEFRMacro AvgMicro AvgGemma 4 26B A4B0\.75160\.72390\.70090\.72550\.8417Qwen3\-4B0\.64520\.64430\.64930\.64630\.7626Gemma 4 E2B0\.66960\.63360\.61540\.63950\.7714Qwen3\.5 9B0\.56410\.62790\.62680\.60630\.7912Mistral3 3B0\.58460\.58520\.54610\.57200\.6986Gemma 4 E4B0\.54580\.58390\.58300\.57090\.7804Qwen3\.5 2B0\.50420\.48860\.46580\.48620\.5855HY\-MT1\.5 1\.8B0\.40430\.42190\.39060\.40560\.3649Table 4:Macro\-averaged recall per language and overall metrics on the HIPE\-2026 dataset\. Models are sorted by macro\-averaged recall\.Boldindicates best in column\.Several key findings emerge from these results:

#### Large models dominate\.

Gemma 4 26B A4B achieves the highest zero\-shot macro\-averaged recall \(0\.7255\), confirming that model scale provides substantial advantages for this task\. The model’s strong multilingual pretraining contributes to consistent performance across all three languages\.

#### Model scale is not the only factor\.

Interestingly, smaller models with superior architecture can outperform larger but less capable ones\. For instance, Qwen3\-4B \(macro 0\.6463\) and the base Gemma 4 E2B \(macro 0\.6395\) both outperform the larger Qwen3\.5 9B \(macro 0\.6063\) and Gemma 4 E4B \(macro 0\.5709\) in macro\-averaged recall, suggesting that model architecture and pretraining data composition play a significant role alongside parameter count\.

#### Small models are viable with distillation\.

As shown in Section[6\.1](https://arxiv.org/html/2606.29130#S6.SS1), the distilled Gemma 4 E2B student achieves competitive performance that surpasses several larger zero\-shot models, validating our distillation approach\.

## 6Analysis

### 6\.1Distillation Effectiveness

Table[5](https://arxiv.org/html/2606.29130#S6.T5)quantifies the efficiency\-accuracy trade\-off achieved through distillation\.

Deployed ParamsMacro AvgRecovery RateCompressionGemma 4 26B A4B \(teacher\)26B0\.7015——Gemma 4 E2B \(distilled\)2\.3B0\.6171∼\\sim88%∼\\sim11×\\timesGemma 4 E2B \(zero\-shot\)2\.3B0\.5329∼\\sim76%∼\\sim11×\\timesTable 5:Comparison of teacher vs\. distilled student performance on sandbox dev set\. Recovery rate is the ratio of student to teacher macro\-averaged recall\. Parameter counts report deployed base\-model size; LoRA adapters are merged for inference\.The distilled student achieves∼\\sim88% of the teacher’s macro\-averaged recall while being 11×\\timessmaller in deployed parameter count, confirming that teacher\-generated reasoning patterns transfer effectively to the smaller model\. These sandbox dev results are not directly comparable to the full test metrics reported in Table[4](https://arxiv.org/html/2606.29130#S5.T4)\.

### 6\.2Official Evaluation Results

In the official HIPE\-2026 evaluation, our system \(team12, “whereami”\) achieved competitive rankings across multiple profiles:

- •Standard accuracy profile: Ranked3rdamong all teams with a mean profile score of0\.688\.
- •Binary accuracy profile: Ranked2ndamong all teams with a mean profile score of0\.8156\.
- •Balanced efficiency\-accuracy profile: Ranked2ndamong all teams on both the standard and binary test sets, demonstrating an effective trade\-off between model size and classification performance\.

These results demonstrate that DistilledGemma provides a strong balance of accuracy and efficiency in several profiles while using substantially fewer parameters\.

### 6\.3Cross\-Lingual Analysis

Performance varies across languages, with English generally achieving the highest scores and French exhibiting the greatest variance\. German performance is notably consistent across model sizes, possibly because German historical newspapers in the HIPE dataset tend to feature cleaner OCR output and more explicit geographic references\.

The distilled student shows the largest improvement on German, suggesting that the teacher’s German reasoning patterns are particularly amenable to distillation\. French performance proves more challenging to distill, likely due to the more complex syntactic structures characteristic of French historical prose\.

### 6\.4Error Analysis

We identify three dominant error categories:

1. 1\.PROBABLE confusion: The three\-wayatclassification proves challenging, with models frequently conflating PROBABLE and TRUE\. This distinction requires nuanced assessment of evidence strength that smaller models struggle to capture\.
2. 2\.False negatives on isAt: The extreme class imbalance \(77\.7% FALSE\) biases all models toward negative predictions, particularly degrading recall on the minority TRUE class\.
3. 3\.OCR\-induced entity confusion: In documents with severe OCR degradation, entity mentions are corrupted, leading to misidentification of both persons and places, which cascades into downstream relation classification errors\.

### 6\.5Efficiency Considerations

The Gemma 4 E2B student model can be served on a single consumer GPU with<\{<\}8 GB VRAM, processing approximately 50 pairs per second via vLLM\. This deployment count excludes the 26B teacher, which is used only offline for silver\-data generation\. In contrast, the 26B teacher requires a high\-memory GPU\. For large\-scale historical document processing involving millions of articles, the 2\.3B student provides a practical solution with acceptable performance degradation, while the 26B teacher remains preferable when accuracy is paramount and computational resources are available\.

## 7Conclusion

We presented DistilledGemma, a knowledge distillation pipeline for person\-place relation extraction from multilingual historical newspaper texts\. Through systematic evaluation of eight LLM configurations, we demonstrated that supervised fine\-tuning of large models yields the strongest performance, and that response\-level distillation effectively transfers chain\-of\-thought reasoning patterns to compact models\.

Our key findings are: \(1\) QLoRA fine\-tuning enables the selected teacher to generate task\-specific silver data, and response\-level distillation transfers much of that performance to the student; \(2\) knowledge distillation from a 26B teacher to a 2\.3B student recovers∼\{\\sim\}88% of teacher performance at1/11\\nicefrac\{\{1\}\}\{\{11\}\}the parameter cost; and \(3\) chain\-of\-thought distillation, where the student learns the teacher’s reasoning process rather than labels alone, combined with rule\-based post\-processing, provides an effective pathway to competitive performance with limited computational resources\.

In the official HIPE\-2026 evaluation, DistilledGemma ranked 3rd in the standard accuracy profile and 2nd in both the binary accuracy and balanced efficiency\-accuracy profiles, demonstrating that strong performance on multilingual historical relation extraction can be achieved without excessive computational expenditure\.

Future work includes exploring on\-policy self\-distillation methods\[agarwal2024gkd\]that could further narrow the teacher\-student performance gap, investigating encoder\-based discriminative models as an alternative lightweight approach, and extending the framework to additional historical document domains\.

## References