REAP: Relation-Aware Elicitation and Parsing for Closed-Book Knowledge Base Construction from LLMs

# Relation-Aware Elicitation and Parsing for Closed-Book Knowledge Base Construction from LLMs
Source: [https://arxiv.org/html/2608.10963](https://arxiv.org/html/2608.10963)
Thanh\-Trung DoTuan\-Phong NguyenAffiliation:VNU University of Engineering and Technology, Hanoi, VietnamAffiliation:\{23020342, 24022472, tuanphong\}@vnu\.edu\.vn

###### Abstract

We present theReapsystem for the AKBC Shared Task 2026 on constructing knowledge bases from language models in a closed\-book setting, subject to a budget of at most 32B parameters and no model fine\-tuning\. Our system combines structured chain\-of\-thought reasoning, relation\-specific query strategies, and a reasoning\-based empty\-set gate to elicit parametric knowledge, followed by direct extraction into valid JSON arrays\. On the official test set, the system, built on Mistral\-Small\-24B\-Instruct\-2501, achieves a macro\-F1 score of0\.620\.62, with particularly strong results oncountryLandBordersCountry\(F1=0\.95F\_\{1\}=0\.95\),companyTradesAtStockExchange\(F1=0\.73F\_\{1\}=0\.73\), andhasArea\(F1=0\.77F\_\{1\}=0\.77\)\. Our code is publicly available\.111[https://github\.com/yammdd/AKBC\-Shared\-Task\-2026](https://github.com/yammdd/AKBC-Shared-Task-2026)

## 1Introduction

Large language models \(LLMs\) encode substantial factual knowledge in their parameters\([17](https://arxiv.org/html/2608.10963#bib.bib1)\)\. However, eliciting this knowledge for knowledge base construction \(KBC\) is considerably more challenging than answering isolated factual questions: a subject–relation pair\(s,r\)\(s,r\)may correspond to an empty \(∅\\emptyset\), single\-valued \(11\), or multi\-valued \(NN\) object set\. The AKBC Shared Task 2026\([8](https://arxiv.org/html/2608.10963#bib.bib5)\)formalizes this problem under a constrained setting: systems must operate in a closed\-book fashion, without retrieval\-augmented generation or external knowledge; use at most 32B parameters; and perform no model fine\-tuning\.

In this task, each input record specifies a subject entity and one of the six given relations \(see Table[1](https://arxiv.org/html/2608.10963#S1.T1)\), and the system must return the complete set of corresponding objects as a valid JSON array\. The relations cover geographic adjacency and area, a person’s city of death, venue capacity, award recipients, and the stock exchanges on which a company is traded\. Consequently, the required output may be empty, contain a single object, or contain a long list of objects, while quantitative answers must also be sufficiently precise\. Table[1](https://arxiv.org/html/2608.10963#S1.T1)summarizes the six target relations and the scope of their ground\-truth labels\.

Table 1:Definitions and ground\-truth scope of the six target relations\.Earlier factual probing methods use cloze\-style prompts or prompt ensembles to rank candidate objects\([17](https://arxiv.org/html/2608.10963#bib.bib1);[5](https://arxiv.org/html/2608.10963#bib.bib11)\), while the LM\-KBC challenge series additionally requires systems to materialize variable\-cardinality answer sets\([20](https://arxiv.org/html/2608.10963#bib.bib13);[10](https://arxiv.org/html/2608.10963#bib.bib20);[7](https://arxiv.org/html/2608.10963#bib.bib21);[9](https://arxiv.org/html/2608.10963#bib.bib10)\)\. A direct generative solution that asks the model to recall all objects and produce correctly formatted JSON in a single pass can therefore hallucinate objects, omit valid answers, or violate the output schema\.

To address this issue, we proposeReap\(Relation\-awareElicitationAndParsing\), a two\-stage pipeline that decouples factual elicitation from answer serialization\. Stage 1 prompts the LLM using strategies tailored to each relation, whereas Stage 2 uses deterministic JSON parsing when possible and LLM\-based extraction only as a fallback\. This separation allows the model to focus on recalling the answer set before the system enforces the required JSON format\.

Our main contributions are as follows:

1. 1\.Relation\-Aware Elicitation: We design specialized strategies for different relation types, including CoT reasoning with an empty\-set gate, a four\-direction geographical scan for land borders, and chronological multi\-pass querying for awards\.
2. 2\.Efficient Hybrid Parsing: Deterministic parsing handles approximately80%80\\%of the records, while LLM\-based extraction is reserved for complex outputs, substantially reducing the overall computational cost\.
3. 3\.Scalable Inference and Evaluation: Batched inference with vLLM on TPU hardware enables fast end\-to\-end processing\. The complete system achieves macro\-F10\.620\.62on the test set and highlights the limits of parametric knowledge for rare entities\.

## 2Related Work

#### Probing knowledge in language models\.

LAMA\([17](https://arxiv.org/html/2608.10963#bib.bib1)\)established cloze\-style probing but assumes exactly one object per relation, and what can be extracted depends heavily on the prompt\([5](https://arxiv.org/html/2608.10963#bib.bib11)\); see[2](https://arxiv.org/html/2608.10963#bib.bib12)for a survey\. The LM\-KBC challenge series\([20](https://arxiv.org/html/2608.10963#bib.bib13);[10](https://arxiv.org/html/2608.10963#bib.bib20);[7](https://arxiv.org/html/2608.10963#bib.bib21);[9](https://arxiv.org/html/2608.10963#bib.bib10)\)recast this as knowledge base construction with arbitrary\-cardinality object sets \(00,11, orNN\), the setting continued by the AKBC Shared Task 2026\([8](https://arxiv.org/html/2608.10963#bib.bib5)\)\. Prior systems pair LLM probing with entity mapping\([22](https://arxiv.org/html/2608.10963#bib.bib14)\); most recently, ReWiSe\([1](https://arxiv.org/html/2608.10963#bib.bib6)\), a top\-performing system at LM\-KBC 2025, combines chain\-of\-thought \(CoT\) reasoning with relation\-wise self\-consistency\. InReap, we adopt the idea of relation\-specific CoT and explicit empty\-set handling, but drop self\-consistency voting, which is costly and can be dominated by confident\-but\-incorrect reasoning chains\.

#### Parametric knowledge and its limits\.

Closed\-book QA shows that language models can answer factual questions without retrieval\([19](https://arxiv.org/html/2608.10963#bib.bib3)\), motivating the closed\-book setting we adopt\. This parametric knowledge is uneven, however: models recall popular facts well but struggle with long\-tail entities\([12](https://arxiv.org/html/2608.10963#bib.bib15);[15](https://arxiv.org/html/2608.10963#bib.bib16)\), which bounds any closed\-book system and matches our error analysis on rare subjects\.

#### Step\-by\-step reasoning and empty\-set gate\.

Chain\-of\-thought prompting\([21](https://arxiv.org/html/2608.10963#bib.bib2);[13](https://arxiv.org/html/2608.10963#bib.bib17)\)improves multi\-step reasoning and builds on the in\-context learning ability of instruction\-tuned LLMs\([3](https://arxiv.org/html/2608.10963#bib.bib18)\)\. Since a correct answer may itself be the empty set, inReap, we add a*reasoning\-based empty\-set gate*inspired by selective prediction\([11](https://arxiv.org/html/2608.10963#bib.bib4)\)\. Unlike calibrated methods that estimate whether a model knows an answer\([6](https://arxiv.org/html/2608.10963#bib.bib19)\), our gate is a prompt\-driven decision that returns an empty set as the final answer when the property does not exist or the evidence of it is insufficient\.

## 3Method

### 3\.1Architecture Overview

TheReapsystem consists of two sequential stages \(Figure[1](https://arxiv.org/html/2608.10963#S3.F1)\)\. In Stage 1, we use an LLM to*generate evidence*as free\-form text for multi\-valued relations or use CoT reasoning for single\-valued relations\. If the generation phase produces unparsable output, the system automatically retries or switches to a fallback prompt\. Stage 2*extracts*the evidence into JSON arrays using regular expressions; for multi\-valued relations that cannot be handled reliably by simple rules, the LLM performs the extraction itself\.

Figure 1:The architecture ofReap\.
### 3\.2Relation\-Specific Prompt Design

Stage 1 prompts are specialized per relation following three principles: \(i\)query decompositionto maximize recall on multi\-valued relations \(e\.g\., six decade\-scoped passes forawardWonByand a four\-direction geographical scan forcountryLandBordersCountry\); \(ii\)output\-format constraintsviaBORDERS:orFINAL\_ANSWER: \[…\]markers; and \(iii\)step\-by\-step reasoning with an empty\-set gatethat outputsFINAL\_ANSWER: \[\]for unanswerable entities \(e\.g\., living individuals or private companies\)\. Together, these relation\-tailored, multi\-step CoT procedures constitute the*structured reasoning*that Stage 1 performs before extraction\.

Few\-shot demonstrations are static exemplar pairs \(one positive, one empty\-set case\) fixed per relation to illustrate the reasoning chain and empty\-set triggering\. Crucially, all exemplars are drawn exclusively fromtrain\.jsonlor hand\-written triples, ensuring zero data leakage against the evaluation splits\.

#### awardWonBy: Chronological Multi\-Pass\.

For major awards with dozens or hundreds of recipients, a single query typically yields low recall\. We therefore use a multi\-pass strategy comprising one general prompt and five auxiliary prompts decomposed chronologically: from the award’s inception through the 1970s, the 1980s–1990s, the 2000s–2010s, 2020 onward, and a final pass targeting less prominent or non\-Western recipients\. All six generations use temperatureT=0\.7T=0\.7to diversify recall before the evidence is merged\.

#### countryLandBordersCountry: Four\-Direction Geographical Scan\.

The prompt instructs the model to act as a geography expert and inspect the subject’s eastern, western, southern, and northern boundaries\. This scan reduces omissions of small neighboring countries and adjacent detached territories\. The prompt includes land borders only, explicitly excludes maritime boundaries, and returnsBORDERS: NONEfor island countries\. If the response lacks theBORDERS:marker, the system automatically retries with a shorter fallback prompt\.

#### companyTradesAtStockExchange: Four\-Step Listing Check\.

The model follows a four\-step CoT procedure: \(1\) identify the company; \(2\) apply aStrict Public Check \(empty\-set gate\)to determine whether the entity is a private company, a nonprofit organization, or a subsidiary that is not separately listed, returningFINAL\_ANSWER: \[\]if any condition holds; \(3\) retrieve all listing exchanges using their full official names, such asNew York Stock ExchangeorSIX Swiss Exchange; and \(4\) conclude withFINAL\_ANSWER: \["Exchange Name"\]\.

#### hasArea: Four\-Step Entity Disambiguation and Unit Normalization\.

The model follows four CoT steps: \(1\) distinguish the target entity \(e\.g\., an island, lake, or region\) from the country containing it; \(2\) retrieve its area from parametric knowledge while preserving decimal precision; \(3\) convert the value tokm2\\text\{km\}^\{2\}; and \(4\) output an integer or decimal value asFINAL\_ANSWER: \["Value"\]\.

#### hasCapacity: Four\-Step Reasoning with Capacity\-Range Verification\.

The model follows four CoT steps: \(1\) determine the venue type and location; \(2\) distinguish a university or small local stadium from a larger national stadium with a similar name or in the same city; \(3\) check the estimate against a plausible capacity range \(1,0001\{,\}000–35,00035\{,\}000seats for small venues and35,00035\{,\}000–100,000100\{,\}000for large national stadiums\); and \(4\) return the capacity as an integer inFINAL\_ANSWER: \["Value"\]\.

#### personHasCityOfDeath: Five\-Step Reasoning with Empty\-Set Gate\.

The model performs five steps: \(1\) check the subject’s biographical status and record “ALIVE TODAY” if appropriate; \(2\) resolve name ambiguity; \(3\) retrieve the city or town of death; \(4\) isolate the place name; and \(5\) returnFINAL\_ANSWER: \[\]if the person is alive or the information is unavailable, orFINAL\_ANSWER: \["CityName"\]if the place of death is confirmed\.

### 3\.3Post\-processing

To ensure that extracted outputs adhere to the ground\-truth format, we apply the following automated post\-processing procedure after the generation phase:

1. 1\.Direct Parsing and Robust Array Extraction: The system directly parses the content of theFINAL\_ANSWER:line or recovers a truncated JSON array using balanced\-bracket scanning\.
2. 2\.Numeric Extraction and Normalization: For quantitative relations \(hasAreaandhasCapacity\), regular expressions extract integer or floating\-point values while removing thousands separators and measurement units\.
3. 3\.Title and Noise Filtering: ForawardWonBy, the system removes year prefixes \(e\.g\.,"1938: Albert Einstein"\), HTML tags, work\-title suffixes, and honorifics such asDr\.,Prof\.,Sir,Dame,Lord,Saint,St\., andMr\.to obtain normalized entity names\.
4. 4\.Parenthetical Filtering and Deduplication: The system removes trailing parenthetical qualifiers \(e\.g\.,"Guinea \(West Africa\)"→\\to"Guinea"\) and performs case\-insensitive deduplication while preserving the capitalization of the first occurrence\.

## 4Experiments

### 4\.1Experimental Setup

#### Data\.

The organizers provide training, validation, and test splits for six relations\. The validation and test splits each contain 475 records: 68 forcountryLandBordersCountryin validation \(67 in test\), 97 and 98 forhasCapacityrespectively, 10 forawardWonBy, and 100 for each of the other three relations\. Each record consists of a subject, a relation, and a set of objects, each accompanied by a list of aliases\. The data cover all three cardinality cases: empty sets \(e\.g\.,New Zealandfor land borders\), single\-valued relations, and multi\-valued relations \(e\.g\., Nobel Prize in Physics,229229recipients intrain\.jsonl, or AAAI Fellow, 350 people inval\.jsonl\)\.

#### Models\.

We evaluate three instruction\-tuned models—Gemma\-2\-9B\-it\([4](https://arxiv.org/html/2608.10963#bib.bib7)\), Llama\-3\.1\-8B\-Instruct\([14](https://arxiv.org/html/2608.10963#bib.bib8)\), and Mistral\-Small\-24B\-Instruct\-2501\([16](https://arxiv.org/html/2608.10963#bib.bib9)\)—and compare against the organizer’s Qwen3\.5\-9B baseline\([18](https://arxiv.org/html/2608.10963#bib.bib22)\)\. Gemma\-2\-9B\-it and Llama\-3\.1\-8B\-Instruct serve as comparison systems, while Mistral\-Small\-24B\-Instruct\-2501 \(hereafter Mistral\-24B\) is the primary model in our final system\.

#### Hardware and Runtime\.

Experiments are conducted on Kaggle using a TPU v5e\-8 with eight TPU v5e cores\. We serve the models with vLLM TPU Server and use bfloat16 precision\. With direct parsing and batched inference, one complete run over all six relations in the validation set \(475 records\) takes approximately*2–5 minutes*withbatch\_size=32\.

#### Metrics\.

We report macro/micro precision, recall, and F1 using the organizer\-providedevaluate\.pyscript\. For string\-valued relations, the evaluator normalizes predictions and matches aliases through maximum bipartite matching\. For quantitative relations, it applies a relative tolerance of5%5\\%\. Overall macro\-F1 is the mean of the*per\-record*F1 scores\.

### 4\.2Validation Results

We first evaluate all models in a*zero\-shot*setting, using direct queries without evidence generation or CoT reasoning\. As shown in Figure[2](https://arxiv.org/html/2608.10963#S4.F2), Llama\-3\.1\-8B, Gemma\-2\-9B, and Mistral\-24B achieve macro\-F1 scores of0\.380\.38,0\.420\.42, and0\.430\.43, respectively\. These relatively low and similar scores suggest that all three models struggle when required to recall facts and satisfy the output format simultaneously\. The limitation is particularly pronounced for multi\-valued relations that require long object lists, such asawardWonBy, and for quantitative relations requiring precise values\.

Llama\-3\.1\-8BGemma\-2\-9BMistral\-24B000\.10\.10\.20\.20\.30\.30\.40\.40\.50\.50\.60\.60\.70\.70\.380\.380\.420\.420\.430\.430\.480\.480\.510\.510\.650\.65Macro\-F1Zero\-shotReappipeline \(ours\)

Figure 2:Overall macro\-F1 of zero\-shot prompting and the two\-stage pipeline on the validation set\.#### Effectiveness of the Two\-Stage Pipeline\.

Figure[2](https://arxiv.org/html/2608.10963#S4.F2)shows that the two\-stage pipeline improves macro\-F1 for all three models\. Llama\-3\.1\-8B improves from0\.380\.38to0\.480\.48, Gemma\-2\-9B from0\.420\.42to0\.510\.51, and Mistral\-24B from0\.430\.43to0\.650\.65\. Mistral\-24B yields the largest absolute gain \(\+0\.22\+0\.22\) and is therefore selected as the primary model in our final system submitted to the AKBC Shared Task 2026\.

### 4\.3Test Results

Table[2](https://arxiv.org/html/2608.10963#S4.T2)reports the relation\-level and overall macro\-F1 scores of the primary system on the official test set\.

Table 2:Macro\-F1 of Mistral\-24B on the test set\.The relation\-level results in Figure[3](https://arxiv.org/html/2608.10963#S4.F3)show that the Mistral\-24B pipeline consistently outperforms all other configurations across all six relations on the F1 metric\. It achieves its highest scores oncountryLandBordersCountry\(0\.950\.95\) andhasArea\(0\.770\.77\)\. Overall, the Mistral pipeline leads with a macro\-F1 score of0\.620\.62, significantly outperforming the Llama \(0\.480\.48\) and Gemma \(0\.480\.48\) pipelines, as well as more than doubling the organizer’s baseline that is based on Qwen3\.5\-9B \(0\.300\.30\)\.

awardWonBycompanyTradescountryBordershasAreahasCapacitycityOfDeathOverall000\.20\.20\.40\.40\.60\.60\.80\.8110\.370\.370\.730\.730\.950\.950\.770\.770\.230\.230\.530\.530\.620\.62Macro\-F1Organizer’s baseline \(Qwen3\.5\-9B\)Llama\-3\.1\-8BGemma\-2\-9BMistral\-24B \(ours\)Figure 3:Relation\-level macro\-F1 of the organizer’s Qwen3\.5\-9B baseline and three models using the two\-stageReappipeline on the official test set of the AKBC Shared Task 2026\.

## 5Ablation Study

To quantify each component’s contribution, we perform a subtractive ablation study on the validation set using Mistral\-24B \(Table[3](https://arxiv.org/html/2608.10963#S5.T3)\)\.

Table 3:Ablation results on the validation set\.Removing*Task Decomposition*drops macro\-F1 from0\.650\.65to0\.590\.59, as single\-pass queries suffer severe recall loss on multi\-valued relations:awardWonBycollapses from0\.250\.25to0\.000\.00andcountryLandBordersCountryfalls from0\.980\.98to0\.630\.63\. Removing the*Empty\-Set Gate*lowers macro\-F1 to0\.570\.57due to hallucinated answers for unanswerable entities, severely hurting precision onpersonHasCityOfDeath\(0\.50→0\.360\.50\\to 0\.36\) andcompanyTradesAtStockExchange\(0\.76→0\.570\.76\\to 0\.57\)\. Finally, removing*CoT Reasoning*entirely causes the largest drop to0\.430\.43, heavily degradinghasArea\(0\.82→0\.550\.82\\to 0\.55\) andcountryLandBordersCountry\(0\.98→0\.270\.98\\to 0\.27\), confirming that our gains stem primarily from structured parametric reasoning rather than output parsing alone\.

## 6Discussion

#### Relation\-Aware Prompting versus Zero\-Shot Prompting\.

In the zero\-shot setting, all models exhibit limited and relatively similar performance\. Requiring a model to recall facts and produce valid JSON in a single pass is especially difficult for relations with large object sets, such asawardWonBy, and for quantitative relations that demand precise values, such ashasAreaandhasCapacity\. Applying the complete*two\-stage pipeline*ofReapwith relation\-specific prompts, CoT reasoning, a reasoning\-based empty\-set gate, and direct parsing improves all three models\. These results indicate that structured reasoning is important for reliably eliciting knowledge encoded in model parameters\.

#### Limits of Parametric Knowledge\.

Our results reveal a clear gap between the 9B model \(Gemma\-2\-9B\) and the 24B model \(Mistral\-24B\)\. In the zero\-shot setting, the models perform similarly, partly because both struggle with output formatting\. Under the full structured\-reasoning pipeline, however, Mistral\-24B reaches macro\-F10\.650\.65on the validation set, compared with0\.510\.51for Gemma\-2\-9B\. This result suggests that reasoning prompts can reduce formatting noise, while successful factual retrieval still depends on model capacity\. Mistral\-24B performs better on long\-tail entities such as small islands, club\-level stadiums, and less prominent awards\. Thus, in a closed\-book setting, a larger model may benefit more from such a pipeline\.

#### Run\-to\-run variance\.

Two minor sources of variation affect our runs\. First,awardWonByuses sampling \(T=0\.7T=0\.7, six decade\-scoped passes\) to diversify recall over long recipient lists, so its score fluctuates slightly given only 10 validation awards\. Second, even under greedy decoding \(T=0\.0T=0\.0\) the TPU v5e\-8 is not bit\-exact, because the accumulation order of bfloat16 values differs across its eight cores\. Both effects are small—overall macro\-F1 varies by about±0\.005\\pm 0\.005—and do not change our conclusions\.

## 7Conclusion

We presented theReapsystem for the AKBC Shared Task 2026 under a strict closed\-book setting with at most 32B parameters, no external knowledge, and no model fine\-tuning\. The system implements a unified two\-stage pipeline with Mistral\-Small\-24B\-Instruct\-2501, combining relation\-aware prompt engineering, a reasoning\-based empty\-set gate, and direct parsing\. It achieves a macro\-F1 of0\.650\.65on the validation set and a macro\-F1 of0\.620\.62on the official test set\. The results demonstrate that structured reasoning can effectively elicit factual knowledge encoded in model parameters for knowledge base construction\.

## Limitations

Despite its promising performance, our system has several limitations: \(i\) its parametric knowledge remains incomplete for very rare, long\-tail entities; \(ii\) predictions for quantitative relations sometimes fall outside the5%5\\%relative\-tolerance threshold because of retrieval or rounding errors; and \(iii\) stochastic sampling forawardWonBy, together with non\-deterministic distributed computation on TPU v5e\-8, causes slight variation across runs\. Among the six relations,hasCapacityis the most challenging \(test macro\-F10\.230\.23\): venues are frequently confused with larger namesakes in the same city, and even small numeric errors exceed the5%5\\%tolerance\.

## References

- Albert\-Roulhac and Zouaq \(2025\)E\. Albert\-Roulhac and A\. ZouaqReWiSe: Relation\-Wise Self\-consistency for LLM Probing\.InLM\-KBC @ ISWC 2025,External Links:[Link](https://ceur-ws.org/Vol-4041/paper8.pdf)Cited by:[§2](https://arxiv.org/html/2608.10963#S2.SS0.SSS0.Px1.p1.1)\.
- AlKhamissiet al\.\(2022\)B\. AlKhamissi, M\. Li, A\. Celikyilmaz, M\. Diab, and M\. GhazvininejadA review on language models as knowledge bases\.Note:[https://arxiv\.org/abs/2204\.06031](https://arxiv.org/abs/2204.06031)Cited by:[§2](https://arxiv.org/html/2608.10963#S2.SS0.SSS0.Px1.p1.1)\.
- Brownet al\.\(2020\)T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. AmodeiLanguage models are few\-shot learners\.InNeurIPS,External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html)Cited by:[§2](https://arxiv.org/html/2608.10963#S2.SS0.SSS0.Px3.p1.1)\.
- Gemma Team \(2024\)Gemma TeamGemma 2: improving open language models at a practical size\.Note:[https://arxiv\.org/abs/2408\.00118](https://arxiv.org/abs/2408.00118)Cited by:[§4\.1](https://arxiv.org/html/2608.10963#S4.SS1.SSS0.Px2.p1.1)\.
- Jianget al\.\(2020\)Z\. Jiang, F\. F\. Xu, J\. Araki, and G\. NeubigHow can we know what language models know?\.TACL\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00324)Cited by:[§1](https://arxiv.org/html/2608.10963#S1.p3.1),[§2](https://arxiv.org/html/2608.10963#S2.SS0.SSS0.Px1.p1.1)\.
- Kadavathet al\.\(2022\)S\. Kadavath, T\. Conerly, A\. Askell, T\. Henighan, D\. Drain, E\. Perez, N\. Schiefer, Z\. Hatfield\-Dodds, N\. DasSarma, E\. Tran\-Johnson, S\. Johnston, S\. El\-Showk, A\. Jones, N\. Elhage, T\. Hume, A\. Chen, Y\. Bai, S\. Bowman, S\. Fort, D\. Ganguli, D\. Hernandez, J\. Jacobson, J\. Kernion, S\. Kravec, L\. Lovitt, K\. Ndousse, C\. Olsson, S\. Ringer, D\. Amodei, T\. Brown, J\. Clark, N\. Joseph, B\. Mann, S\. McCandlish, C\. Olah, and J\. KaplanLanguage models \(mostly\) know what they know\.Note:[https://arxiv\.org/abs/2207\.05221](https://arxiv.org/abs/2207.05221)Cited by:[§2](https://arxiv.org/html/2608.10963#S2.SS0.SSS0.Px3.p1.1)\.
- Kaloet al\.\(2024\)J\. Kalo, T\. Nguyen, S\. Razniewski, and B\. ZhangPreface: LM\-KBC challenge 2024\.InISWC,External Links:[Link](https://ceur-ws.org/Vol-3853/paper0.pdf)Cited by:[§1](https://arxiv.org/html/2608.10963#S1.p3.1),[§2](https://arxiv.org/html/2608.10963#S2.SS0.SSS0.Px1.p1.1)\.
- Kaloet al\.\(2026\)J\. Kalo, T\. Nguyen, S\. Razniewski, and B\. ZhangAKBC shared task 2026: knowledge base construction from language models\.Note:[https://lm\-kbc\.github\.io/challenge2026/](https://lm-kbc.github.io/challenge2026/)Cited by:[§1](https://arxiv.org/html/2608.10963#S1.p1.1),[§2](https://arxiv.org/html/2608.10963#S2.SS0.SSS0.Px1.p1.1)\.
- Kaloet al\.\(2025\)J\. Kalo, S\. Razniewski, B\. Zhang, and T\. NguyenLM\-KBC 2025: 4th challenge on knowledge base construction from pre\-trained language models\.InISWC,External Links:[Link](https://ceur-ws.org/Vol-4041/paper7.pdf)Cited by:[§1](https://arxiv.org/html/2608.10963#S1.p3.1),[§2](https://arxiv.org/html/2608.10963#S2.SS0.SSS0.Px1.p1.1)\.
- Kaloet al\.\(2023\)J\. Kalo, S\. Singhania, S\. Razniewski, and J\. Z\. PanLM\-KBC 2023: 2nd challenge on knowledge base construction from pre\-trained language models\.InISWC,External Links:[Link](https://ceur-ws.org/Vol-3577/paper0.pdf)Cited by:[§1](https://arxiv.org/html/2608.10963#S1.p3.1),[§2](https://arxiv.org/html/2608.10963#S2.SS0.SSS0.Px1.p1.1)\.
- Kamathet al\.\(2020\)A\. Kamath, R\. Jia, and P\. LiangSelective question answering under domain shift\.InACL,External Links:[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.503)Cited by:[§2](https://arxiv.org/html/2608.10963#S2.SS0.SSS0.Px3.p1.1)\.
- Kandpalet al\.\(2023\)N\. Kandpal, H\. Deng, A\. Roberts, E\. Wallace, and C\. RaffelLarge language models struggle to learn long\-tail knowledge\.InICML,External Links:[Link](https://proceedings.mlr.press/v202/kandpal23a.html)Cited by:[§2](https://arxiv.org/html/2608.10963#S2.SS0.SSS0.Px2.p1.1)\.
- Kojimaet al\.\(2022\)T\. Kojima, S\. S\. Gu, M\. Reid, Y\. Matsuo, and Y\. IwasawaLarge language models are zero\-shot reasoners\.InNeurIPS,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/8bb0d291acd4acf06ef112099c16f326-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2608.10963#S2.SS0.SSS0.Px3.p1.1)\.
- Llama Team, Meta AI \(2024\)Llama Team, Meta AIThe llama 3 herd of models\.Note:[https://arxiv\.org/abs/2407\.21783](https://arxiv.org/abs/2407.21783)Cited by:[§4\.1](https://arxiv.org/html/2608.10963#S4.SS1.SSS0.Px2.p1.1)\.
- Mallenet al\.\(2023\)A\. Mallen, A\. Asai, V\. Zhong, R\. Das, D\. Khashabi, and H\. HajishirziWhen not to trust language models: investigating effectiveness of parametric and non\-parametric memories\.InACL,External Links:[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.546)Cited by:[§2](https://arxiv.org/html/2608.10963#S2.SS0.SSS0.Px2.p1.1)\.
- Mistral AI \(2025\)Mistral AIMistral\-small\-24b\-instruct\-2501\.Note:[https://huggingface\.co/mistralai/Mistral\-Small\-24B\-Instruct\-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501)Cited by:[§4\.1](https://arxiv.org/html/2608.10963#S4.SS1.SSS0.Px2.p1.1)\.
- Petroniet al\.\(2019\)F\. Petroni, T\. Rocktäschel, S\. Riedel, P\. Lewis, A\. Bakhtin, Y\. Wu, and A\. MillerLanguage models as knowledge bases?\.InEMNLP\-IJCNLP,External Links:[Document](https://dx.doi.org/10.18653/v1/D19-1250)Cited by:[§1](https://arxiv.org/html/2608.10963#S1.p1.1),[§1](https://arxiv.org/html/2608.10963#S1.p3.1),[§2](https://arxiv.org/html/2608.10963#S2.SS0.SSS0.Px1.p1.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.5: towards native multimodal agents\.Note:[https://qwen\.ai/blog?id=qwen3\.5](https://qwen.ai/blog?id=qwen3.5)Cited by:[§4\.1](https://arxiv.org/html/2608.10963#S4.SS1.SSS0.Px2.p1.1)\.
- Robertset al\.\(2020\)A\. Roberts, C\. Raffel, and N\. ShazeerHow much knowledge can you pack into the parameters of a language model?\.InEMNLP,External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.437)Cited by:[§2](https://arxiv.org/html/2608.10963#S2.SS0.SSS0.Px2.p1.1)\.
- Singhaniaet al\.\(2022\)S\. Singhania, T\. Nguyen, and S\. RazniewskiLM\-KBC: knowledge base construction from pre\-trained language models\.InISWC,External Links:[Link](https://ceur-ws.org/Vol-3274/paper1.pdf)Cited by:[§1](https://arxiv.org/html/2608.10963#S1.p3.1),[§2](https://arxiv.org/html/2608.10963#S2.SS0.SSS0.Px1.p1.1)\.
- Weiet al\.\(2022\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. Le, and D\. ZhouChain\-of\-thought prompting elicits reasoning in large language models\.InNeurIPS,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2608.10963#S2.SS0.SSS0.Px3.p1.1)\.
- Zhanget al\.\(2023\)B\. Zhang, I\. Reklos, N\. Jain, A\. Meroño\-Peñuela, and E\. SimperlUsing large language models for knowledge engineering \(LLMKE\): a case study on Wikidata\.InLM\-KBC @ ISWC 2023,External Links:[Link](https://arxiv.org/abs/2309.08491)Cited by:[§2](https://arxiv.org/html/2608.10963#S2.SS0.SSS0.Px1.p1.1)\.