OntoBook: Ontology-Grounded Synthetic Textbooks for Medical Encoder Pretraining
Summary
OntoBook converts medical ontology graphs into synthetic textbook prose using LLMs, then uses the resulting 1.3M French textbooks to pretrain ModernCamemBERT, achieving significant gains on medical coding benchmarks.
View Cached Full Text
Cached at: 07/22/26, 08:23 AM
# OntoBook: Ontology-Grounded Synthetic Textbooks for Medical Encoder Pretraining Source: [https://arxiv.org/abs/2607.18927](https://arxiv.org/abs/2607.18927) [View PDF](https://arxiv.org/pdf/2607.18927) > Abstract:We present OntoBook, a method that converts medical ontology structure into pretraining signal for encoder language models\. Our approach has three stages: random walks through ontology graphs capture hierarchical and causal relations between medical codes, a large language model reformulates these walks into fluent textbook\-style prose, and the resulting text is used to train ModernCamemBERT, a 149M\-parameter French encoder, with two objectives on the same data: masked language modeling and relation prediction between code pairs\. On three French medical coding benchmarks \(FRACCO, Cantemist\-FR, Distemist\-FR\), OntoBook achieves significant improvements over MLM\-only pretraining, with \+2\.5 micro\-F1 on FRACCO and \+8\.0 micro\-F1 on Distemist\. We find that alignment between objectives is necessary: misaligned training, where each task uses different data, causes a 30\-point degradation\. We release 1\.3 million LLM\-reformulated medical textbooks across three French ontologies \(CIM\-10, CCAM, ATC\) and pretrained model checkpoints\. ## Submission history From: Rian Touchent \[[view email](https://arxiv.org/show-email/32325f56/2607.18927)\] \[via CCSD proxy\] **\[v1\]**Tue, 21 Jul 2026 10:11:10 UTC \(35 KB\)
Similar Articles
Injecting Structured Biomedical Knowledge into Language Models: Continual Pretraining vs. GraphRAG
This paper compares two strategies for injecting structured biomedical knowledge from the UMLS Metathesaurus into language models: continual pretraining (embedding knowledge into model parameters) and GraphRAG (querying a knowledge graph at inference time). Results show improvements on biomedical QA benchmarks, with GraphRAG on LLaMA 3-8B yielding over 3 and 5 accuracy points on PubMedQA and BioASQ respectively without any retraining.
A Causal Language Modeling Detour Improves Encoder Continued Pretraining
This paper demonstrates that switching from Masked Language Modeling to Causal Language Modeling during encoder adaptation improves downstream performance on biomedical texts. The authors release ModernBERT-bio and ModernCamemBERT-bio as state-of-the-art biomedical encoders.
MedicalBench: Evaluating Large Language Models Toward Improved Medical Concept Extraction
MedicalBench is a new benchmark for evaluating large language models on medical concept extraction from electronic health records, focusing on implicit reasoning and evidence grounding. It includes 823 expert-annotated examples and shows that current models perform modestly, highlighting the difficulty of extracting implicitly stated medical concepts.
The Word and the Way: Strategies for Domain-Specific BERT Pre-Training in German Medical NLP
This paper introduces ChristBERT, a family of domain-specific RoBERTa-based language models for German clinical NLP, and evaluates three domain adaptation strategies (continued pre-training, pre-training from scratch, and vocabulary adaptation) on medical named entity recognition and text classification tasks, achieving state-of-the-art results.
OpenMedQ: Broad Open Pretraining for Medical Vision-Language Models
OpenMedQ is a fully-open medical vision-language model pretrained on 14 datasets (~3.35M samples), achieving state-of-the-art results on medical VQA and classification benchmarks.