Pretraining Language Models on Historical Text

# Pretraining Language Models on Historical Text
Source: [https://arxiv.org/html/2606.02991](https://arxiv.org/html/2606.02991)
Xiaoxi Luo1,2,Zachary Shinnick3,4,Niclas Griesshaber4,5,Yixuan Wang1,2, Junchi Yu4,Freda Shi1,2,Philip Torr4,Yao Lu6,, 1University of Waterloo,2Vector Institute,3AIML, Adelaide University, 4Department of Engineering Science, University of Oxford, 5Oxford Centre for Economic and Social History, University of Oxford 6Department of Computer Science, University College London \{x25luo,tyxw,fhs\}@uwaterloo\.cazachary\.shinnick@adelaide\.edu\.au niclas\.griesshaber@history\.ox\.ac\.uk\{junchi\.yu,philip\.torr\}@eng\.ox\.ac\.ukyao\.lu@cs\.ucl\.ac\.uk

###### Abstract

We introduceTypewriterLM, a 7\.24B History language model \(LM\) trained exclusively on English text predating 1913\. Developing History LMs requires addressing challenges in data quality and availability, preventing temporal leakage, desinging temporally consistent post\-training pipelines, and constructing reliable evaluations\. To address these issues, we constructTypewriterCorpus, a 54B\-token historical corpus collected from diverse archival and linguistically annotated sources with extensive data cleaning and leakage mitigation procedures\. Furthermore, we introduce*lexically grounded instructing tuning*, a post\-training framework that constraints responses to remain directly grounded in historical source documents\. Using this framework we construct two historical instruction tuning datasets:History\-LIMAandHistory\-SelfInstruct\. To evaluate capability and temporal consistency, we introduceHistory\-Event, a benchmark suite for evaluating competence, temporal grounding and data leakage\. We releaseTypewriterLMand all associated resources to support future research on historical language models\.111Models and datasets are available at[hf\.co/typewriter\-ai](https://huggingface.co/typewriter-ai)\.

Pretraining Language Models on Historical Text

Xiaoxi Luo1,2, Zachary Shinnick3,4, Niclas Griesshaber4,5, Yixuan Wang1,2,Junchi Yu4,Freda Shi1,2,Philip Torr4,Yao Lu6,††thanks:Corresponding author\.,1University of Waterloo,2Vector Institute,3AIML, Adelaide University,4Department of Engineering Science, University of Oxford,5Oxford Centre for Economic and Social History, University of Oxford6Department of Computer Science, University College London\{x25luo,tyxw,fhs\}@uwaterloo\.cazachary\.shinnick@adelaide\.edu\.auniclas\.griesshaber@history\.ox\.ac\.uk\{junchi\.yu,philip\.torr\}@eng\.ox\.ac\.ukyao\.lu@cs\.ucl\.ac\.uk

## 1Introduction

Modern foundation models\(Gemini Team,[2025](https://arxiv.org/html/2606.02991#bib.bib56); Yanget al\.,[2025](https://arxiv.org/html/2606.02991#bib.bib58); OpenAI,[2026](https://arxiv.org/html/2606.02991#bib.bib57); DeepSeek\-AI,[2026](https://arxiv.org/html/2606.02991#bib.bib22),inter alia\)achieve impressive performance across a wide range of natural language tasks\. However, when used to study historical settings, they are biased toward modern stylistic and lexical distributions\(Underwoodet al\.,[2025](https://arxiv.org/html/2606.02991#bib.bib35)\)\. Moreover, modern LMs have already been pre\-trained on the historical events and outcomes one might wish to analyze, so any apparent foresight could simply be memorized hindsight, which is known as lookahead bias\(Sarkar and Vafa,[2025](https://arxiv.org/html/2606.02991#bib.bib41)\)\.

QueryTypewriterLMResponseDo you think there will soon be a major war in Europe?“Certainly… It would not surprise me if this country were to go to war with France within ninety days\.”Do you know about the general theory of relativity by Albert Einstein?“…Professor A\. Einstein, of whom we have not yet learned to speak as a physicist…”What is a computer?“…an operator who multiplies or divides… a person who performs arithmetical operations…”Figure 1:ProbingTypewriterLMon “future” events\.The cleanest fix is to train language models under a strict knowledge cutoff that excludes information beyond a chosen date, an approach that is attracting increasing attention from both machine learning and the humanities\(Grigorian and Yaghoobian,[2025](https://arxiv.org/html/2606.02991#bib.bib30); Göttlichet al\.,[2025](https://arxiv.org/html/2606.02991#bib.bib7); Levineet al\.,[2026](https://arxiv.org/html/2606.02991#bib.bib8)\)\. While modern LLMs benefit from massive and diverse web\-collected corpora such as Common Crawl, only very limited historical data is available from OCR\-transcribed sources\. This is a typical data\-constrained pre\-training setup\(Muennighoffet al\.,[2023](https://arxiv.org/html/2606.02991#bib.bib33)\)\. For the post\-training stage, instruction tuning typically relies on QA pairs from human annotators or frontier models\. For History LMs, neither route is viable, as each would inject the modern information and perspectives that the cutoff was meant to exclude\. Finally, standard modern benchmarks are often temporally misaligned with History LMs, making it difficult to distinguish genuine reasoning limitations from unfamiliarity with modern language and contexts\.

To address these challenges, we curateTypewriterCorpus, a 54\-billion\-token pre\-training corpus collected from diverse sources and built with a strict data cleaning pipeline to mitigate temporal leakage\. For leakage\-free post\-training, we introduce lexically grounded instruction tuning, where responses come directly from historical source documents, and constructHistory\-LIMAandHistory\-SelfInstruct\. Using these resources, we trainTypewriterLM, a 7B\-parameter language model with a strict knowledge cutoff of 1913, in both base and instruction\-tuned variants\.

For evaluation, we consider both downstream capability and temporal consistency\. Our base model shows competitive performance on general benchmarks, and our instruction\-tuned models achieve performance comparable with other History LMs trained using modern LLM supervision\. To evaluate temporal consistency, we constructHistory\-Event, a benchmark of 2,344 significant historical events spanning 1700–2025\. Using perplexity\-based surprisingness evaluation, we find that History LMs become substantially more surprised by post\-cutoff events, whereas the modern baseline Llama\-3\.1\-8B remains comparatively flat across time\. This suggests that the intended historical cutoffs are meaningfully reflected in the models’ learned knowledge distributions\.

To summarize, our contributions are as follows:

1. 1\.We demonstrate that data\-constrained pre\-training can produce competitive, historically grounded language models, enabling future research in both the humanities and NLP\.
2. 2\.We control leakage at every stage of the pipeline, with strict cleaning forTypewriterCorpus, lexically grounded instruction tuning that sources every response from pre\-cutoff documents \(History\-LIMAandHistory\-SelfInstruct\), and a historically aligned evaluation suite \(History\-Event\) that verifies temporal integrity end\-to\-end\.
3. 3\.We openly releaseTypewriterLM\(in both base and instruction\-tuned versions\) along with the full pipeline to support transparent research in this emerging area\.

## 2Related Work

Research on using LLMs for temporal prediction tasks has focused on curating temporally bounded training datasets\. This aims to prevent lookahead bias, where models are exposed to information from after the cut\-off date during training\. Because this exposure to the test set biases a model’s forecasting ability, efforts to mitigate lookahead bias have so far been primarily focused on financial applications\. For example,Yanet al\.\([2026](https://arxiv.org/html/2606.02991#bib.bib23)\),Heet al\.\([2025](https://arxiv.org/html/2606.02991#bib.bib24)\), andKellyet al\.\([2026](https://arxiv.org/html/2606.02991#bib.bib60)\)introduce families of language models trained on annual cut\-off dates between 2000 and 2024\. These models are trained on approximately 100 billion tokens per year, with the largest containing 4 billion parameters\. We address the same underlying challenge with a much earlier, “historical” cut\-off date of 1913\. This introduces additional challenges, yet we curate a pretraining corpus of more than 50 billion tokens to train a 7B\-parameter model\.

Even for frontier models, the reported and effective cut\-off dates differ, emphasizing the challenges of curating training datasets from modern text\(Chenget al\.,[2024](https://arxiv.org/html/2606.02991#bib.bib25)\)\. As historical text corpora can be scraped from OCR\-transcribed sources available online, they remain susceptible to similar leakage issues\. One potential solution is to construct historical datasets directly from archival image scans\. For example,Dellet al\.\([2023](https://arxiv.org/html/2606.02991#bib.bib26)\)use tailored OCR and layout detection pipelines to build a large\-scale historical newspaper dataset\.Sarkar \([2024](https://arxiv.org/html/2606.02991#bib.bib27)\)then uses this dataset to train a family of BERT\-based 110M\-parameter models with cut\-off dates between 1900 and 1963\. By contrast, we construct our pretraining corpus using OCR text from institutional archives, as building text datasets directly from images remains difficult at scale due to the heterogeneity of historical documents, although recent advances in VLMs show promising progress toward a universal solution\(Greifet al\.,[2025](https://arxiv.org/html/2606.02991#bib.bib28); Griesshaber and Streb,[2025](https://arxiv.org/html/2606.02991#bib.bib29)\)\.

Recent efforts to scale History LMs to billions of parameters highlight the immense difficulty of constructing training datasets without leakage\.Grigorian and Yaghoobian \([2025](https://arxiv.org/html/2606.02991#bib.bib30)\)train a 1\.3B language model from 1800 to 1875 London text and report that OCR noise, such as “Digitized by Google” is still present in their outputs\.Levineet al\.\([2026](https://arxiv.org/html/2606.02991#bib.bib8)\)further scaled this line of work to 13 billion parameters using a pretraining dataset consisting of 260 billion tokens\. They report temporal leakage as their model knows who was US president in 1936 after their cut\-off date in 1930\. Other efforts include Ranke\-4B, a family of LLMs introduced by a team of economic historians inGöttlichet al\.\([2025](https://arxiv.org/html/2606.02991#bib.bib7)\)and trained on 80B tokens with cut\-off dates between 1913 and 1946; a language model trained on Victorian\-era British texts published between 1837 and 1899\(Venturella,[2026](https://arxiv.org/html/2606.02991#bib.bib45)\); and a model trained on a pre\-1900 text corpus\(michaelmla,[2026](https://arxiv.org/html/2606.02991#bib.bib46)\)\. With our data filtering approaches during pretraining and instruction tuning, we aim to further mitigate the risk of data leakage in this emerging line of research\.

## 3Historical Pretraining

### 3\.1Corpus Construction

We constructTypewriterCorpus, a historical English corpus spanning 1700–1913, where 1700 is the start of Late Modern English\(Barberet al\.,[2009](https://arxiv.org/html/2606.02991#bib.bib1)\), and 1913 is the year immediately preceding World War I\. The corpus combines large\-scale digitized books with curated historical datasets covering diverse domains, genres, and registers\.

The majority of the corpus is derived from Institutional Books\(Cargneluttiet al\.,[2025](https://arxiv.org/html/2606.02991#bib.bib3)\), a large collection of digitized books from Harvard Library’s collections spanning domains such as literature, science, law, and philosophy\. We further incorporate British Library Books\(BL books; British Library Labs,[2021](https://arxiv.org/html/2606.02991#bib.bib4)\), another large\-scale historical book collection digitized by the British Library\.

To increase linguistic and stylistic diversity, we additionally include several curated historical corpora\. Parliamentary discourse is represented by Hansard\(Brezina,[2024](https://arxiv.org/html/2606.02991#bib.bib5)\); scientific writing by the Royal Society Corpus v6\.0\(Fischeret al\.,[2020](https://arxiv.org/html/2606.02991#bib.bib2)\); legal and spoken language by Old Bailey\(Huberet al\.,[2016](https://arxiv.org/html/2606.02991#bib.bib10)\); and literary Late Modern English by The Corpus of Late Modern English Texts v3\.1\(CLMET; Dilleret al\.,[2011](https://arxiv.org/html/2606.02991#bib.bib9)\)\. We further include smaller corpora such as Corpus of English Dialogues\(CED; Kytö and Culpeper,[2006](https://arxiv.org/html/2606.02991#bib.bib12)\), The Lampeter Corpus of Early Modern English Tracts\(Lampeter; Siemund and Claridge,[1997](https://arxiv.org/html/2606.02991#bib.bib11)\), Corpus of Late Modern English Prose\(CLME Prose; Denison,[1994](https://arxiv.org/html/2606.02991#bib.bib51)\), and a pamphlet collection\(Bailyn,[1965](https://arxiv.org/html/2606.02991#bib.bib50)\)to broaden coverage of dialogue, pamphlets, and prose styles\.

To improve text quality, we apply several normalization and filtering procedures to remove OCR artifacts\. For example, OCR systems frequently insert spaces at visual column or line boundaries \(e\.g\., “im\- possible”\), and we apply a conservative split\-word joining pass\. We additionally remove spaces preceding punctuation marks and discard symbol\-heavy fragments\. More details about the construction ofTypewriterCorpuscan be found in Appendix[A\.1](https://arxiv.org/html/2606.02991#A1.SS1)\.

After data cleaning,TypewriterCorpuscontains approximately 54 billion tokens as counted by our custom BPE tokenizer \(§[3\.3](https://arxiv.org/html/2606.02991#S3.SS3)\)\. Table[1](https://arxiv.org/html/2606.02991#S3.T1)summarizes the corpus composition after filtering and cleaning\. Institutional Books dominates the corpus by volume \(97\.7%\), while the remaining sources provide important diversity in genre, register, and domain\. Figure[2](https://arxiv.org/html/2606.02991#S3.F2)illustrates the token distribution by decade, showing that the corpus is concentrated between 1800 and 1900\.

### 3\.2Leakage Mitigation

Temporal leakage presents a major challenge for history LMs, as archival documents frequently contain modern metadata or annotations\. We identify several common sources of leakage and apply rule\-based filtering to reduce contamination\.

#### Institutional provenance metadata\.

Some scanned books contain ownership stamps \(e\.g\., “Harvard College Library”\), institutional mottoes, catalog identifiers, and donor annotations added by modern libraries\. We remove such provenance metadata from Institutional Books and related archival sources\.

#### Web and HTML artifacts\.

URLs and HTML entities introduced during corpus processing are removed from all datasets\.

#### Publishing and editorial metadata\.

Title\-page imprint lines \(e\.g\., book price, publisher addresses\), attribution lines \(e\.g\., shorthand\-writer and editor credits in Old Bailey\), and similar editorial metadata are removed heuristically\.

Despite extensive filtering, completely eliminating temporal leakage from historical OCR corpora remains challenging\. We therefore additionally design leakage\-aware post\-training and evaluation protocols described in later sections\.

Table 1:Composition ofTypewriterCorpusafter filtering and cleaning\. Tokens are counted using our custom BPE tokenizer \(§[3\.3](https://arxiv.org/html/2606.02991#S3.SS3)\)\.![Refer to caption](https://arxiv.org/html/2606.02991v1/x1.png)Figure 2:Number of tokens by decade \(1700–1913\) in our pretraining corpus\.

### 3\.3Training Setup

To avoid modern vocabulary contamination, we train a custom byte\-pair encoding \(BPE\) tokenizer\(Sennrichet al\.,[2016](https://arxiv.org/html/2606.02991#bib.bib13)\)with a vocabulary size of 32,000 on the training corpus, using theo200k\_basepre\-tokenization strategy fromtiktoken\(OpenAI,[2023](https://arxiv.org/html/2606.02991#bib.bib17)\)\.

TypewriterLMis a 7\.24B\-parameter decoder\-only Transformer following the Llama 3 architecture\(Llama Team, AI @ Meta,[2024](https://arxiv.org/html/2606.02991#bib.bib14)\)\. The model uses grouped\-query attention\(Ainslieet al\.,[2023](https://arxiv.org/html/2606.02991#bib.bib16)\), RMSNorm\(Zhang and Sennrich,[2019](https://arxiv.org/html/2606.02991#bib.bib18)\), rotary position embeddings\(Suet al\.,[2024](https://arxiv.org/html/2606.02991#bib.bib19)\), and an 8,192\-token context length\. We train using AdamW\(Loshchilov and Hutter,[2019](https://arxiv.org/html/2606.02991#bib.bib20)\)with bfloat16 mixed precision on approximately 140B tokens \(2\.59 epochs over the corpus\)\. More training setup details are provided in Appendix[A\.2](https://arxiv.org/html/2606.02991#A1.SS2)\.

## 4Historical Instruction Tuning

Instruction tuning can substantially shift the behaviour of pretrained language models\(Weiet al\.,[2022](https://arxiv.org/html/2606.02991#bib.bib36); Ouyanget al\.,[2022](https://arxiv.org/html/2606.02991#bib.bib37)\)\. For historical language models, this presents a fundamental challenge: even when pretrained on temporally filtered corpora, standard supervised fine\-tuning reintroduces modern linguistic and semantic distributions through contemporary instruction data or synthetic responses generated by frontier LLMs\. Filtering explicit factual references or simply prompting a frontier LLM to imitate historical language is insufficient, since temporal leakage persists at the level of language distribution itself\. While prior historical language models primarily focus on temporally filtering pretraining corpora\(Grigorian and Yaghoobian,[2025](https://arxiv.org/html/2606.02991#bib.bib30); Göttlichet al\.,[2025](https://arxiv.org/html/2606.02991#bib.bib7)\),Levineet al\.\([2026](https://arxiv.org/html/2606.02991#bib.bib8)\)observe that modern post\-training pipelines, particularly reinforcement learning from AI feedback, inevitably introduce anachronistic linguistic and behavioural priors\. In contrast, we explicitly address leakage introduced during instruction tuning itself\. To this end, we construct the entire instruction\-tuning corpus under a strict lexical grounding constraint: responses are constructed directly from pre\-1913 source documents and may contain only lexical items appearing in the source passage, together with a small allowlist of function words222E\.g\., closed\-class grammatical tokens such as “the”, “this”, “thou”, auxiliary verbs, conjunctions, and pronouns required for fluent composition\.\. Our goal is not merely to prevent factual anachronisms, but to preserve temporally consistent lexical, stylistic, and semantic patterns throughout instruction tuning\.

### 4\.1Dataset Curation

#### Lexical Grounding Constraint\.

A responseyyis accepted only if every lexical token is derivable fromV​\(p\)∪AV\(p\)\\cup A, whereV​\(p\)V\(p\)denotes the set of words appearing in source textppandAAis a small allowlist of function words; numerals must also appear in the source text\. We enforce this constraint using a strict post\-generation verifier that discards any non\-compliant response\. Verification is based primarily on exact lexical matching, with controlled recovery operations including limited morphological normalization, dehyphenation, page\-break reconstruction, and fused\-word correction to reduce false rejections caused by OCR artefacts\. All recovery operations require the recovered form to remain derivable from the source text, preventing unsupported vocabulary while permitting limited lexical variation and OCR correction\.

#### History\-LIMA\.

Following the “less is more” philosophy ofZhouet al\.\([2023a](https://arxiv.org/html/2606.02991#bib.bib38)\), we construct a small high\-quality instruction\-tuning set consisting of 1,000 lexically grounded single\-turn examples\. Keeping the dataset small enables detailed human review while reducing drift away from the pretrained historical language distribution\. Candidate pairs are generated using the lexical grounding approach described above and ranked using an LLM judge evaluating response quality and standalone usability, after which the top\-scoring examples are retained\. Instructions are then generated by Claude Opus to align with the grounded responses and manually reviewed by human annotators for historical coherence and knowledge leakage\. We additionally construct a multi\-turn variant containing 1,030 total examples\. This extension includes 30 multi\-turn dialogues following the conversational setup introduced in LIMA\. Fifteen dialogues are curated from pre\-1913 dialogue\-oriented texts including catechisms and literary exchanges from authors such as Lucian and Dickens, while the remaining fifteen are derived from*Hansard*\(Brezina,[2024](https://arxiv.org/html/2606.02991#bib.bib5)\)parliamentary debates between 1853 and 1864, with assistant responses taken verbatim from the original speeches\.

#### History\-SelfInstruct\.

Inspired by Self\-Instruct\(Wanget al\.,[2023](https://arxiv.org/html/2606.02991#bib.bib39)\), we scale instruction\-tuning data by inverting the standard synthetic generation pipeline: responses remain fixed historical anchors, while only instructions are model\-generated\. The pipeline consists of three stages\.*\(i\) Seed construction\.*We use the 1,000 answer→\\rightarrowquestion examples fromHistory\-LIMAas the seed set, preserving the same grounded\-response design while treating instruction generation as the learned task\.*\(ii\) Question\-generator training\.*We then fine\-tune our pre\-1913 base model on this seed set using LoRA \(r=64r\{=\}64\) to obtain a generator that produces historically consistent instructions conditioned on arbitrary grounded responses\.*\(iii\) Filtered generation at scale\.*The generator produces candidate instructions for a large corpus of lexically grounded responses\. Generated pairs are filtered using an LLM\-judge to assess coherence, retaining 287,860 high\-quality instruction\-response pairs\. Crucially, only the instructions are self\-generated; all responses remain lexically grounded in historical source text\.

### 4\.2Training Details

We fine\-tune the pre\-1913 base modelTypewriterLMseparately onHistory\-LIMAandHistory\-SelfInstructusing parameter\-efficient LoRA fine\-tuning\(Huet al\.,[2022](https://arxiv.org/html/2606.02991#bib.bib40)\), yielding two instruction\-tuned variants\. FollowingZhouet al\.\([2023a](https://arxiv.org/html/2606.02991#bib.bib38)\), theTypewriterLM\(LIMA\) model is trained for 15 epochs, while theTypewriterLM\(SelfInstruct\) model is trained for a single epoch\. Our instruction\-tuning setup is intentionally lightweight to minimize drift away from the pretrained historical language distribution while enabling instruction\-following behaviours\. Full training details and hyperparameters are provided in Appendix[B](https://arxiv.org/html/2606.02991#A2)\.

## 5Evaluation

Evaluating History LMs requires addressing two complementary questions\. First, can they perform competitively on standard language understanding and reasoning benchmarks despite being trained under temporal and data constraints? Second, do they faithfully respect their intended historical cutoff?

To evaluate general capability, we testTypewriterLMon general benchmarks and compare it against several recent History LMs spanning different cutoff dates and model scales: Mr\. Chatterbox\(Venturella,[2026](https://arxiv.org/html/2606.02991#bib.bib45), 0\.34B, Victorian\-era English\), TimeCapsuleLLM\-v2\(Grigorian and Yaghoobian,[2025](https://arxiv.org/html/2606.02991#bib.bib30), 1\.22B, 1800–1875 English\), GPT\-1900\(michaelmla,[2026](https://arxiv.org/html/2606.02991#bib.bib46), 3\.29B, pre\-1900 English\), and Talkie\-1930\(Levineet al\.,[2026](https://arxiv.org/html/2606.02991#bib.bib8), 13B, pre\-1931 English\)\. We additionally include GPT2\-XL\(Radfordet al\.,[2019](https://arxiv.org/html/2606.02991#bib.bib47)\)for comparison\.

To evaluate cutoff faithfulness, we further construct a leakage\-aware evaluation suite measuring both temporal surprisingness and factual recall over historical events spanning 1700–2025 \(§[5\.1](https://arxiv.org/html/2606.02991#S5.SS1)\)\. Together, these evaluations assess both downstream capability and preservation of historical knowledge boundaries\.

Table 2:Accuracy \(%\) on ARC\-Easy \(ARC\-E\), ARC\-Challenge \(ARC\-C\), and HellaSwag\.### 5\.1Surprisingness, Recall, and Leakage

We evaluate ifTypewriterLMand other History LMs respect their knowledge cutoff date by using two metrics applied to the same benchmarking dataset\.

#### Dataset\.

We constructHistory\-Event, an evaluation dataset consisting of 2,344 historical events spanning the period 1700–2025, by scraping events from the Wikipedia century timelines\.333[https://en\.wikipedia\.org/wiki/Timeline\_of\_the\_20th\_century](https://en.wikipedia.org/wiki/Timeline_of_the_20th_century)and equivalent pages for the 18th, 19th, and 21st centuries\. See[C\.2](https://arxiv.org/html/2606.02991#A3.SS2)\.The scraped dataset contains the variablesevent\_yearandevent\_description\. The number of events is distributed unevenly across centuries \(Figure[4](https://arxiv.org/html/2606.02991#A3.F4)\)\.

#### BPB Surprisingness\.

Similar toLevineet al\.\([2026](https://arxiv.org/html/2606.02991#bib.bib8)\), which builds on ideas fromDuderstadt and Helm \([2026](https://arxiv.org/html/2606.02991#bib.bib59)\), we measure how “surprised” a model is by providing it with the description of a historical event and the year in which it took place in the form:

> What do you think about the following event:\[event\_description\]\.This took place in\[event\_year\]\.

The prefix serves as conditioning, while only the target span, i\.e\. the description and year phrase, is scored\. Bits\-per\-Byte \(BPB\) surprisingness is calculated as

BPB=NLLnatsln⁡2⋅\|target\|bytes\.\\text\{BPB\}=\\frac\{\\text\{NLL\}\_\{\\text\{nats\}\}\}\{\\ln 2\\cdot\|\\text\{target\}\|\_\{\\text\{bytes\}\}\}\.

#### Recall and Leakage\.

We test whether a model can recall an event inHistory\-Event\. Depending on whether the event occurred before or after the cutoff date, a correct answer either implies that the model is factually correct or that it suffers from data leakage\.444Even a correct pre\-cutoff answer could stem from leaked post\-cutoff sources \(e\.g\., a modern account of a historical event\)\. See §[6](https://arxiv.org/html/2606.02991#S6)\.For each of the 2,344 historical events, we create a question in the following format:

> “Do you know about the following event:\[event\_description\]?If so, explain what this event was and in what year did it take place?”

To prevent copying the date, we remove questions whose description contains any four\-digit year\. All remaining questions are first answered by Gemini\-3\.1\-Flash\-Lite\. We keep only events the model answers correctly, ensuring each question is well\-posed and yielding a gold reference answer\. At this point, 1,726 event questions remain\. All responses by the History LMs are then evaluated by Gemini\-3\.1\-Flash\-Lite, which receives theevent\_description, theevent\_year, its “gold” answer, and the answer by the History LM\.

We measure a model’s recall in a strict and relaxed setting\. Strict requires both conditions to hold: \(a\) the model correctly states theevent\_yearin which the event occurred, and \(b\) it contains correct information that goes beyond theevent\_description\. Relaxed requires only condition \(b\)\. Strict therefore provides a lower bound, while relaxed provides an upper bound on recall\.

![Refer to caption](https://arxiv.org/html/2606.02991v1/x2.png)Figure 3:Bits\-per\-Byte Surprisingness Scores
#### Results\.

Figure[3](https://arxiv.org/html/2606.02991#S5.F3)presents the surprisingness scores and Table[3](https://arxiv.org/html/2606.02991#S5.T3)reports recall performance\.

The BPB scores of the History LMs follow similar patterns, but begin to rise shortly before and after their respective cutoff dates, reaching a higher overall level in the post\-cutoff period\. The baseline, Llama 3\.1 8B Instruct\(Llama Team, AI @ Meta,[2024](https://arxiv.org/html/2606.02991#bib.bib14)\), remains flat and even slightly declines for more recent periods, likely due to greater exposure to modern than historical data during training\. Both patterns are expected, as the baseline was likely trained on all events, whereas the History LMs were not\.

The recall results confirm the BPB patterns, but also demonstrate that the BPB is not sufficient to detect data leakage \(Table[3](https://arxiv.org/html/2606.02991#S5.T3)\)\. We find leakage for the two largest models, Talkie\-1930 andTypewriterLM, with up to 0\.2% of post\-cutoff events leaking under the strict criterion and 0\.6% under the relaxed one\. Examples of the strict violations are in Appendix[C\.5](https://arxiv.org/html/2606.02991#A3.SS5)\.

Pre\-cutoff, the largest History LMs correctly recall a modest fraction of events inHist\-Event\. Mr\. Chatterbox and TimeCapsuleLM do not get a single question strictly right\. GPT\-1900 gets strictly right 2\.5% \(9\.1% relaxed\), andTypewriterLMbeat this at around the 6% mark \(around 24% relaxed\)\. Talkie\-1930 performs best, with 31\.1% strict and 51\.2% relaxed\.

The fact that we find data leakage on ourHistory\-Eventdataset suggests that all capable History LMs suffer from lookahead bias, which we expect to worsen as these models are further scaled\.

Correctness \(pre\-cutoff\)↑\\uparrowLeakage \(post\-cutoff\)↓\\downarrownnModelSizeCutoffStrictRelaxedStrictRelaxedprepostTalkie 1930 \(it\)13B193031\.151\.20\.10\.67031023GPT\-19003\.3B19002\.59\.10\.00\.24851241Mr\. Chatterbox340M18990\.00\.00\.00\.04791247TimeCapsuleLLM1\.2B18750\.00\.30\.00\.03841342TypewriterLM\(LIMA\)7\.24B19135\.122\.60\.00\.35631163TypewriterLM\(Sel\.In\.\)7\.24B19136\.225\.40\.00\.65631163Table 3:Factual historical correctness \(pre\-cutoff\) and data leakage \(post\-cutoff\) onHist\-Event\.

### 5\.2General Capability Benchmarks

#### Hellaswag\.

Standard HellaSwag\(Zellerset al\.,[2019](https://arxiv.org/html/2606.02991#bib.bib31)\)is a commonsense natural language inference benchmark written in modern English and derived from contemporary web text\. We evaluate all aforementioned History LMs and GPT2\-XL\(Radfordet al\.,[2019](https://arxiv.org/html/2606.02991#bib.bib47)\)on the full validation set \(10,042 examples\)\. As shown in Table[2](https://arxiv.org/html/2606.02991#S5.T2), performance on HellaSwag generally scales with model size among History LMs\. However, Talkie\-1930 requires substantially larger scale \(13B parameters\) to reach performance comparable to GPT2\-XL \(1\.5B\), suggesting that modern benchmarks may disadvantage History LMs, probably due to temporal and linguistic mismatch\.

#### ARC\.

The AI2 Reasoning Challenge\(ARC; Clarket al\.,[2018](https://arxiv.org/html/2606.02991#bib.bib21)\)is a grade\-school science multiple\-choice question answering benchmark\. We evaluate on both ARC\-Easy \(2,376 questions\) and ARC\-Challenge \(1,172 questions\), using the template"Question: \{\}\\nAnswer: "followed by each candidate answer\. As shown in Table[2](https://arxiv.org/html/2606.02991#S5.T2),TypewriterLMoutperforms all smaller History LMs on both ARC subsets, but remains below GPT2\-XL and Talkie\-1930\. Notably,TypewriterLMachieves performance on ARC\-Challenge comparable to GPT2\-XL despite its strict historical cutoff, indicating its nontrivial reasoning capability\.

Table 4:Pairwise AlpacaEval comparison againstGPT\-1900on 388 filtered prompts\.
#### AlpacaEval\.

Standard AlpacaEval prompts frequently require post\-1913 knowledge and are therefore unsuitable for evaluating historically grounded language models\. We therefore filter the 805 prompts from AlpacaEval 2\.0\(Duboiset al\.,[2024](https://arxiv.org/html/2606.02991#bib.bib42)\)using an LLM classifier, retaining only prompts that could plausibly be answered by a model trained exclusively on historic data\. The resulting benchmark contains 388 historically valid prompts\. We evaluate models using pairwise AlpacaEval comparisons judged by GPT\-4o\-mini following the AlpacaEval 2\.0 protocol, including position\-swapped comparisons to reduce positional bias\.

Despite being fine tuned under substantially stricter lexical grounding constraints, bothTypewriterLMvariants remain competitive againstGPT\-1900, achieving win rates above 43% \(Table[4](https://arxiv.org/html/2606.02991#S5.T4)\)\. Notably,GPT\-1900is instruction\-tuned using freely generated contemporary LLM responses, whereas our models constrain all assistant responses to lexically grounded historical source text\. These results suggest that historically grounded instruction tuning can retain strong instruction\-following behaviour without relying on unrestricted modern synthetic supervision\.

Table 5:Instruction\-following accuracy \(%\) under prompt\-based \(P\) and instruction\-based \(I\) settings using strict \(S\) and loose \(L\) matching\.
#### IFEval\.

We evaluate instruction\-following using IFEval\(Zhouet al\.,[2023b](https://arxiv.org/html/2606.02991#bib.bib48)\), a benchmark containing 541 prompts spanning 25 verifiable instruction types\. Following the standard protocol, we report prompt\-level and instruction\-level accuracy under both strict and loose matching criteria\. Despite substantially stricter lexical grounding constraints, Table[5](https://arxiv.org/html/2606.02991#S5.T5)showsTypewriterLMachieves competitive instruction\-following performance relative to existing History LMs, which rely on contemporary frontier LLMs for synthetic instruction tuning or reinforcement learning from AI feedback during post\-training\.

## 6Discussion: Challenges in History LM

#### Challenges in Evaluation

Modern benchmarks may fail to faithfully reflect the capabilities of History LMs due to two forms of temporal distribution shift: post\-cutoff*topics*and mismatched*language style*\. To disentangle these two effects, we construct two different versions of Hellaswag\. Starting from the original benchmark, we first apply keyword\-based filtering to remove examples requiring post\-1800 knowledge or cultural context, retaining 5,362 examples involving relatively timeless activities such as cooking, animal care, fishing, and family life\. We then rewrite the filtered examples into a pre\-1800 context using Claude Sonnet 4\.6 while preserving the original task’s domain and difficulty level\. For example, “baking cookies” may be rewritten as “baking biscuits at the hearth\.” This yields 2,048 multiple\-choice examples written in 1800s prose style\.

Table[6](https://arxiv.org/html/2606.02991#S6.T6)compares History LMs and GPT2\-XL on the original, filtered, and rewritten benchmarks\. For GPT\-1900 andTypewriterLM, topic filtering yields only small improvements, while rewriting produces substantially larger gains\. Talkie\-1930 improves consistently across both changes\. Modern LMs exhibit the opposite trend—topic filtering produces only a minor change in performance, while historical rewriting causes a substantially larger drop\. In contrast, modern LMs show minimal change after topic filtering but experience large performance drops under historical rewriting\. These results suggest that benchmark performance is strongly influenced by temporal mismatch in language style\.

Table 6:Performance on the original HellaSwag validation set, the topic\-filtered subset, and theHellaSwag\-1800benchmark\.
#### Challenges in Identifying Leakage\.

We useHistory\-Eventto successfully identify data leakage\. Although we apply extensive filtering during both corpus construction and post\-training, leakage remains difficult to eliminate entirely, as post\-cutoff information may still occur in prefaces, footnotes or additional sources\. Developing leakage\-free History LMs is important for applications requiring strict temporal fidelity, such as the Einstein test\(Perrigo,[2025](https://arxiv.org/html/2606.02991#bib.bib49)\)and historically grounded social science research\. This work represents a step toward a more leakage\-resistant History LM pipeline\.

## 7Future Directions

History LMs open up a range of research directions across NLP and related fields\.

History LMs could advance studies in reasoning–memorization interplay\(Razeghiet al\.,[2022](https://arxiv.org/html/2606.02991#bib.bib54); Xieet al\.,[2025](https://arxiv.org/html/2606.02991#bib.bib32),inter alia\)\. Since they are trained exclusively on pre\-cutoff corpora, their performance on modern reasoning benchmarks is less likely to arise from direct memorization of benchmark data\. In contrast, recent work has shown that modern LLMs can exhibit substantial performance degradation under relatively small perturbations or reformulations of popular reasoning benchmarks\(Orenet al\.,[2024](https://arxiv.org/html/2606.02991#bib.bib52); Zhanget al\.,[2024](https://arxiv.org/html/2606.02991#bib.bib53),inter alia\), suggesting potential benchmark contamination or shortcut memorization effects\.

They also provide a natural setting for studying temporal distribution shift\(Wileset al\.,[2022](https://arxiv.org/html/2606.02991#bib.bib34)\)\. Unlike conventional domain adaptation settings, the shift is historically grounded and affects multiple levels simultaneously, including vocabulary, language style, social values, and world knowledge\.

Historical corpora are finite and non\-renewable, making History LMs a useful testbed for studying scaling behavior under data\-limited pretraining setting\(Muennighoffet al\.,[2023](https://arxiv.org/html/2606.02991#bib.bib33)\)\.

From the perspective of the humanities and social sciences, they are a novel tool to study temporally grounded language, culture, and social values\. For example, training a series of models at successive temporal cutoffs would enable research in diachronic language change\.

## Limitations

We do not systematically study the effect of dataset composition or mixing ratios during pre\-training, despite the highly imbalanced proportions occupied by different corpora\. Moreover, training History LMs is inherently data\-constrained, and we leave the exploration of data\-efficient training strategies and historically grounded synthetic data generation to future work\.

Our model reflects one possible 1913 worldview due to its training cutoff and may generate content considered offensive by modern standards\. As History LMs scale and become more capable, they may pose societal risks\. Our future public releases will therefore include safety warnings and guardrails to mitigate harmful outputs\.

## References

- J\. Ainslie, J\. Lee\-Thorp, M\. de Jong, Y\. Zemlyanskiy, F\. Lebron, and S\. Sanghai \(2023\)GQA: training generalized multi\-query transformer models from multi\-head checkpoints\.InEMNLP,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),External Links:[Link](https://aclanthology.org/2023.emnlp-main.298/)Cited by:[§A\.2](https://arxiv.org/html/2606.02991#A1.SS2.p2.1),[§3\.3](https://arxiv.org/html/2606.02991#S3.SS3.p2.1)\.
- B\. Bailyn \(1965\)Pamphlets of the American Revolution\.OTA Core Collection\.Note:CorpusExternal Links:[Link](http://hdl.handle.net/20.500.14106/2021)Cited by:[§A\.1](https://arxiv.org/html/2606.02991#A1.SS1.SSS0.Px10.p1.1),[§3\.1](https://arxiv.org/html/2606.02991#S3.SS1.p3.1)\.
- C\. Barber, J\. C\. Beal, and P\. A\. Shaw \(2009\)The english language: a historical introduction\.2 edition,Cambridge Approaches to Linguistics,Cambridge University Press\.Cited by:[§3\.1](https://arxiv.org/html/2606.02991#S3.SS1.p1.1)\.
- V\. Brezina \(2024\)Cited by:[§A\.1](https://arxiv.org/html/2606.02991#A1.SS1.SSS0.Px3.p1.1),[§3\.1](https://arxiv.org/html/2606.02991#S3.SS1.p3.1),[§4\.1](https://arxiv.org/html/2606.02991#S4.SS1.SSS0.Px2.p1.1)\.
- British Library Labs \(2021\)Digitised books\. c\. 1510 – c\. 1900\.British Library\.Note:https://doi\.org/10\.23636/r7w6\-zy15Cited by:[§A\.1](https://arxiv.org/html/2606.02991#A1.SS1.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2606.02991#S3.SS1.p2.1)\.
- M\. Cargnelutti, C\. Brobston, J\. Hess, J\. Cushman, K\. Mukk, A\. Scourtas, K\. Courtney, G\. Leppert, A\. Watson, M\. Whitehead, and J\. Zittrain \(2025\)Institutional books 1\.0: a 242b token dataset from harvard library’s collections, refined for accuracy and usability\.External Links:2506\.08300,[Link](https://arxiv.org/abs/2506.08300)Cited by:[§A\.1](https://arxiv.org/html/2606.02991#A1.SS1.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2606.02991#S3.SS1.p2.1)\.
- J\. Cheng, M\. Marone, O\. Weller, D\. Lawrie, D\. Khashabi, and B\. V\. Durme \(2024\)Dated data: tracing knowledge cutoffs in large language models\.InCOLM,External Links:[Link](https://openreview.net/forum?id=wS7PxDjy6m)Cited by:[§2](https://arxiv.org/html/2606.02991#S2.p2.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv:1803\.05457v1\.Cited by:[§5\.2](https://arxiv.org/html/2606.02991#S5.SS2.SSS0.Px2.p1.1)\.
- DeepSeek\-AI \(2026\)DeepSeek\-v4: towards highly efficient million\-token context intelligence\.External Links:[Link](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf)Cited by:[§1](https://arxiv.org/html/2606.02991#S1.p1.1)\.
- M\. Dell, J\. Carlson, T\. Bryan, E\. Silcock, A\. Arora, Z\. Shen, L\. D’Amico\-Wong, Q\. Le, P\. Querubin, and L\. Heldring \(2023\)American stories: a large\-scale structured text dataset of historical U\.S\. newspapers\.NeurIPS\.Cited by:[§2](https://arxiv.org/html/2606.02991#S2.p2.1)\.
- D\. Denison \(1994\)A corpus of late Modern English prose\.InCorpora across the centuries: Proceedings of the First International Colloquium on English Diachronic Corpora, St Catharine’s College Cambridge, 25\-27 March 1993,M\. Kytö, M\. Rissanen, and S\. Wright \(Eds\.\),Language and Computers \- Studies in Practical Linguistics, Vol\.11,pp\. 7–16\.External Links:[Link](http://hdl.handle.net/20.500.14106/2077)Cited by:[§A\.1](https://arxiv.org/html/2606.02991#A1.SS1.SSS0.Px9.p1.1),[§3\.1](https://arxiv.org/html/2606.02991#S3.SS1.p3.1)\.
- H\. Diller, H\. De Smet, and J\. Tyrkkö \(2011\)A european database of descriptors of english electronic texts\.The European English Messenger19\(1\),pp\. 21–35\.Cited by:[§A\.1](https://arxiv.org/html/2606.02991#A1.SS1.SSS0.Px6.p1.1),[§3\.1](https://arxiv.org/html/2606.02991#S3.SS1.p3.1)\.
- Y\. Dubois, P\. Liang, and T\. Hashimoto \(2024\)Length\-controlled AlpacaEval: a simple debiasing of automatic evaluators\.InCOLM,External Links:[Link](https://openreview.net/forum?id=CybBmzWBX0)Cited by:[§5\.2](https://arxiv.org/html/2606.02991#S5.SS2.SSS0.Px3.p1.1)\.
- B\. Duderstadt and H\. Helm \(2026\)A model of the language process\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics,Note:Accessed: 2026\-05\-26External Links:[Link](https://www.calcifercomputing.com/tlm/A_Model_of_the_Language_Process.pdf)Cited by:[§5\.1](https://arxiv.org/html/2606.02991#S5.SS1.SSS0.Px2.p1.1)\.
- S\. Fischer, J\. Knappen, K\. Menzel, and E\. Teich \(2020\)The royal society corpus 6\.0: providing 300\+ years of scientific writing for humanistic study\.InProceedings of the Twelfth Language Resources and Evaluation Conference,N\. Calzolari, F\. Béchet, P\. Blache, K\. Choukri, C\. Cieri, T\. Declerck, S\. Goggi, H\. Isahara, B\. Maegaard, J\. Mariani, H\. Mazo, A\. Moreno, J\. Odijk, and S\. Piperidis \(Eds\.\),Marseille, France,pp\. 794–802\(eng\)\.External Links:[Link](https://aclanthology.org/2020.lrec-1.99/),ISBN 979\-10\-95546\-34\-4Cited by:[§A\.1](https://arxiv.org/html/2606.02991#A1.SS1.SSS0.Px4.p1.1),[§3\.1](https://arxiv.org/html/2606.02991#S3.SS1.p3.1)\.
- Gemini Team \(2025\)Gemini: a family of highly capable multimodal models\.External Links:2312\.11805,[Link](https://arxiv.org/abs/2312.11805)Cited by:[§1](https://arxiv.org/html/2606.02991#S1.p1.1)\.
- D\. Göttlich, D\. Loibner, G\. Jiang, and H\. Voth \(2025\)History llms\.Technical reportUniversity of Zurich and Cologne University\.External Links:[Link](https://github.com/DGoettlich/history-llms)Cited by:[§1](https://arxiv.org/html/2606.02991#S1.p2.1),[§2](https://arxiv.org/html/2606.02991#S2.p3.1),[§4](https://arxiv.org/html/2606.02991#S4.p1.1)\.
- G\. Greif, N\. Griesshaber, and R\. Greif \(2025\)Multimodal LLMs for OCR, OCR post\-correction, and named entity recognition in historical documents\.External Links:2504\.00414,[Link](https://arxiv.org/abs/2504.00414)Cited by:[§2](https://arxiv.org/html/2606.02991#S2.p2.1)\.
- N\. Griesshaber and J\. Streb \(2025\)Multimodal llms for historical dataset construction from archival image scans: german patents \(1877\-1918\)\.External Links:2512\.19675,[Link](https://arxiv.org/abs/2512.19675)Cited by:[§2](https://arxiv.org/html/2606.02991#S2.p2.1)\.
- H\. Grigorian and H\. Yaghoobian \(2025\)Historic london english \(1800–1875\)\.Hugging Face\.Note:[https://huggingface\.co/datasets/postgrammar/london\-llm\-1800](https://huggingface.co/datasets/postgrammar/london-llm-1800)Cited by:[§1](https://arxiv.org/html/2606.02991#S1.p2.1),[§2](https://arxiv.org/html/2606.02991#S2.p3.1),[§4](https://arxiv.org/html/2606.02991#S4.p1.1),[§5](https://arxiv.org/html/2606.02991#S5.p2.1)\.
- S\. He, L\. Lv, A\. Manela, and J\. Wu \(2025\)Chronologically consistent large language models\.External Links:2502\.21206,[Link](https://arxiv.org/abs/2502.21206)Cited by:[§2](https://arxiv.org/html/2606.02991#S2.p1.1)\.
- E\. J\. Hu, yelong shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.InICLR,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§4\.2](https://arxiv.org/html/2606.02991#S4.SS2.p1.1)\.
- M\. Huber, M\. Nissel, and K\. Puga \(2016\)Old Bailey Corpus 2\.0\.Note:hdl:11858/00\-246C\-0000\-0023\-8CFB\-2External Links:[Link](https://hdl.handle.net/11858/00-246C-0000-0023-8CFB-2)Cited by:[§A\.1](https://arxiv.org/html/2606.02991#A1.SS1.SSS0.Px5.p1.1),[§3\.1](https://arxiv.org/html/2606.02991#S3.SS1.p3.1)\.
- B\. T\. Kelly, S\. Malamud, J\. Schwab, and T\. A\. Xu \(2026\)Scaling point\-in\-time language models\.Working PaperTechnical Report35247,Working Paper Series,National Bureau of Economic Research\.External Links:[Document](https://dx.doi.org/10.3386/w35247),[Link](http://www.nber.org/papers/w35247)Cited by:[§2](https://arxiv.org/html/2606.02991#S2.p1.1)\.
- M\. Kytö and J\. Culpeper \(2006\)A Corpus of English Dialogues 1560–1760\.External Links:[Link](https://varieng.helsinki.fi/CoRD/corpora/CED/)Cited by:[§A\.1](https://arxiv.org/html/2606.02991#A1.SS1.SSS0.Px7.p1.1),[§3\.1](https://arxiv.org/html/2606.02991#S3.SS1.p3.1)\.
- N\. Levine, D\. Duvenaud, and A\. Radford \(2026\)Introducing talkie: a 13b vintage language model from 1930\.External Links:[Link](https://talkie-lm.com/introducing-talkie)Cited by:[§1](https://arxiv.org/html/2606.02991#S1.p2.1),[§2](https://arxiv.org/html/2606.02991#S2.p3.1),[§4](https://arxiv.org/html/2606.02991#S4.p1.1),[§5\.1](https://arxiv.org/html/2606.02991#S5.SS1.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2606.02991#S5.p2.1)\.
- Llama Team, AI @ Meta \(2024\)The llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§A\.2](https://arxiv.org/html/2606.02991#A1.SS2.p2.1),[§3\.3](https://arxiv.org/html/2606.02991#S3.SS3.p2.1),[§5\.1](https://arxiv.org/html/2606.02991#S5.SS1.SSS0.Px4.p2.1)\.
- I\. Loshchilov and F\. Hutter \(2019\)Decoupled weight decay regularization\.InICLR,External Links:[Link](https://openreview.net/forum?id=Bkg6RiCqY7)Cited by:[§A\.2](https://arxiv.org/html/2606.02991#A1.SS2.p3.6),[§3\.3](https://arxiv.org/html/2606.02991#S3.SS3.p2.1)\.
- michaelmla \(2026\)gpt1900\.Note:[https://github\.com/michaelhla/gpt1900](https://github.com/michaelhla/gpt1900)GitHub repositoryCited by:[§2](https://arxiv.org/html/2606.02991#S2.p3.1),[§5](https://arxiv.org/html/2606.02991#S5.p2.1)\.
- N\. Muennighoff, A\. M\. Rush, B\. Barak, T\. L\. Scao, N\. Tazi, A\. Piktus, S\. Pyysalo, T\. Wolf, and C\. Raffel \(2023\)Scaling data\-constrained language models\.InNeurIPS,External Links:[Link](https://openreview.net/forum?id=j5BuTrEj35)Cited by:[§1](https://arxiv.org/html/2606.02991#S1.p2.1),[§7](https://arxiv.org/html/2606.02991#S7.p4.1)\.
- OpenAI \(2023\)Tiktoken\.GitHub\.Note:[https://github\.com/openai/tiktoken](https://github.com/openai/tiktoken)Cited by:[§3\.3](https://arxiv.org/html/2606.02991#S3.SS3.p1.1)\.
- OpenAI \(2026\)Introducing GPT\-5\.5\.Note:[https://openai\.com/index/introducing\-gpt\-5\-5/](https://openai.com/index/introducing-gpt-5-5/)Official blog announcementCited by:[§1](https://arxiv.org/html/2606.02991#S1.p1.1)\.
- Y\. Oren, N\. Meister, N\. S\. Chatterji, F\. Ladhak, and T\. Hashimoto \(2024\)Proving test set contamination in black\-box language models\.InICLR,External Links:[Link](https://openreview.net/forum?id=KS8mIvetg2)Cited by:[§7](https://arxiv.org/html/2606.02991#S7.p2.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.NeurIPS35,pp\. 27730–27744\.Cited by:[§4](https://arxiv.org/html/2606.02991#S4.p1.1)\.
- B\. Perrigo \(2025\)Demis hassabis is preparing for ai’s endgame\.TIME\.External Links:[Link](https://time.com/7277608/demis-hassabis-interview-time100-2025/)Cited by:[§6](https://arxiv.org/html/2606.02991#S6.SS0.SSS0.Px2.p1.1)\.
- A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, I\. Sutskever,et al\.\(2019\)Language models are unsupervised multitask learners\.OpenAI blog\.Cited by:[§5\.2](https://arxiv.org/html/2606.02991#S5.SS2.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2606.02991#S5.p2.1)\.
- Y\. Razeghi, R\. L\. Logan IV, M\. Gardner, and S\. Singh \(2022\)Impact of pretraining term frequencies on few\-shot numerical reasoning\.InEMNLP,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),External Links:[Link](https://aclanthology.org/2022.findings-emnlp.59/)Cited by:[§7](https://arxiv.org/html/2606.02991#S7.p2.1)\.
- S\. K\. Sarkar and K\. Vafa \(2025\)Lookahead bias in pretrained language models\.InICML 2025 Workshop on Reliable and Responsible Foundation Models,Cited by:[§1](https://arxiv.org/html/2606.02991#S1.p1.1)\.
- S\. K\. Sarkar \(2024\)StoriesLM: a family of language models with time\-indexed training data\.Available at SSRN 4881024\.Cited by:[§2](https://arxiv.org/html/2606.02991#S2.p2.1)\.
- R\. Sennrich, B\. Haddow, and A\. Birch \(2016\)Neural machine translation of rare words with subword units\.InACL,K\. Erk and N\. A\. Smith \(Eds\.\),External Links:[Link](https://aclanthology.org/P16-1162/)Cited by:[§3\.3](https://arxiv.org/html/2606.02991#S3.SS3.p1.1)\.
- N\. Shazeer \(2020\)GLU variants improve transformer\.External Links:2002\.05202,[Link](https://arxiv.org/abs/2002.05202)Cited by:[§A\.2](https://arxiv.org/html/2606.02991#A1.SS2.p2.1)\.
- R\. Siemund and C\. Claridge \(1997\)The Lampeter Corpus of Early Modern English Tracts\.ICAME Journal21,pp\. 61–70\.Cited by:[§A\.1](https://arxiv.org/html/2606.02991#A1.SS1.SSS0.Px8.p1.1),[§3\.1](https://arxiv.org/html/2606.02991#S3.SS1.p3.1)\.
- J\. Su, M\. Ahmed, Y\. Lu, S\. Pan, W\. Bo, and Y\. Liu \(2024\)Roformer: enhanced transformer with rotary position embedding\.Neurocomputing568,pp\. 127063\.Cited by:[§A\.2](https://arxiv.org/html/2606.02991#A1.SS2.p2.1),[§3\.3](https://arxiv.org/html/2606.02991#S3.SS3.p2.1)\.
- T\. Underwood, L\. K\. Nelson, and M\. Wilkens \(2025\)Can language models represent the past without anachronism?\.arXiv preprint arXiv:2505\.00030\.External Links:[Link](https://arxiv.org/abs/2505.00030)Cited by:[§1](https://arxiv.org/html/2606.02991#S1.p1.1)\.
- T\. Venturella \(2026\)Mr\. ChatterBox\.Note:[https://huggingface\.co/spaces/tventurella/mr\_chatterbox](https://huggingface.co/spaces/tventurella/mr_chatterbox)Hugging Face SpaceCited by:[§2](https://arxiv.org/html/2606.02991#S2.p3.1),[§5](https://arxiv.org/html/2606.02991#S5.p2.1)\.
- Y\. Wang, Y\. Kordi, S\. Mishra, A\. Liu, N\. A\. Smith, D\. Khashabi, and H\. Hajishirzi \(2023\)Self\-instruct: aligning language models with self\-generated instructions\.InACL,Cited by:[§4\.1](https://arxiv.org/html/2606.02991#S4.SS1.SSS0.Px3.p1.2)\.
- J\. Wei, M\. Bosma, V\. Zhao, K\. Guu, A\. W\. Yu, B\. Lester, N\. Du, A\. M\. Dai, and Q\. V\. Le \(2022\)Finetuned language models are zero\-shot learners\.InICLR,External Links:[Link](https://openreview.net/forum?id=gEZrGCozdqR)Cited by:[§4](https://arxiv.org/html/2606.02991#S4.p1.1)\.
- O\. Wiles, S\. Gowal, F\. Stimberg, S\. Rebuffi, I\. Ktena, K\. D\. Dvijotham, and A\. T\. Cemgil \(2022\)A fine\-grained analysis on distribution shift\.InICLR,External Links:[Link](https://openreview.net/forum?id=Dl4LetuLdyK)Cited by:[§7](https://arxiv.org/html/2606.02991#S7.p3.1)\.
- C\. Xie, Y\. Huang, C\. Zhang, D\. Yu, X\. Chen, B\. Y\. Lin, B\. Li, B\. Ghazi, and R\. Kumar \(2025\)On memorization of large language models in logical reasoning\.InProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics,K\. Inui, S\. Sakti, H\. Wang, D\. F\. Wong, P\. Bhattacharyya, B\. Banerjee, A\. Ekbal, T\. Chakraborty, and D\. P\. Singh \(Eds\.\),External Links:[Link](https://aclanthology.org/2025.ijcnlp-long.148/)Cited by:[§7](https://arxiv.org/html/2606.02991#S7.p2.1)\.
- Y\. Yan, R\. Tang, Z\. Gao, W\. Jiang, and Y\. Lu \(2026\)DatedGPT: preventing lookahead bias in large language models with time\-aware pretraining\.External Links:2603\.11838,[Link](https://arxiv.org/abs/2603.11838)Cited by:[§2](https://arxiv.org/html/2606.02991#S2.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2606.02991#S1.p1.1)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)HellaSwag: can a machine really finish your sentence?\.InACL,Cited by:[§5\.2](https://arxiv.org/html/2606.02991#S5.SS2.SSS0.Px1.p1.1)\.
- B\. Zhang and R\. Sennrich \(2019\)Root mean square layer normalization\.InNeurIPS,H\. Wallach, H\. Larochelle, A\. Beygelzimer, F\. d'Alché\-Buc, E\. Fox, and R\. Garnett \(Eds\.\),Vol\.32,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2019/file/1e8a19426224ca89e83cef47f1e7f53b-Paper.pdf)Cited by:[§A\.2](https://arxiv.org/html/2606.02991#A1.SS2.p2.1),[§3\.3](https://arxiv.org/html/2606.02991#S3.SS3.p2.1)\.
- H\. Zhang, J\. Da, D\. Lee, V\. Robinson, C\. Wu, W\. Song, T\. Zhao, P\. Raja, C\. Zhuang, D\. Slack, Q\. Lyu, S\. Hendryx, R\. Kaplan, M\. Lunati, and S\. Yue \(2024\)A careful examination of large language model performance on grade school arithmetic\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 46819–46836\.External Links:[Document](https://dx.doi.org/10.52202/079017-1485),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/53384f2090c6a5cac952c598fd67992f-Paper-Datasets_and_Benchmarks_Track.pdf)Cited by:[§7](https://arxiv.org/html/2606.02991#S7.p2.1)\.
- C\. Zhou, P\. Liu, P\. Xu, S\. Iyer, J\. Sun, Y\. Mao, X\. Ma, A\. Efrat, P\. Yu, L\. Yu,et al\.\(2023a\)Lima: less is more for alignment\.Advances in Neural Information Processing Systems36,pp\. 55006–55021\.Cited by:[§4\.1](https://arxiv.org/html/2606.02991#S4.SS1.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2606.02991#S4.SS2.p1.1)\.
- J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. Hou \(2023b\)Instruction\-following evaluation for large language models\.arXiv preprint arXiv:2311\.07911\.Cited by:[§5\.2](https://arxiv.org/html/2606.02991#S5.SS2.SSS0.Px4.p1.1)\.

## Appendix APre\-training

### A\.1Pre\-training Datasets

Here we provide a more detailed introduction to the datasets comprisingtypewritercorpus, and dataset\-specific curation details\.

#### Institutional Books\.

It is our largest source consisting of digitized books from Harvard Library’s collections\(Cargneluttiet al\.,[2025](https://arxiv.org/html/2606.02991#bib.bib3)\), spanning 20 topics, e\.g\., literature, science, law, philosophy\. To guarantee data quality, we apply an OCR score filter, retaining only books with both original and post processed OCR scores higher than 92\. To avoid OCR fragments, We also discard any remaining paragraph shorter than 100 characters\. After filtering, this source contributes 52\.74B tokens, 97\.7% of the total corpus\.

#### British Library Books \(BL Books\)\.

This dataset consists of books digitized by the British Library, covering a wide range of subject areas, and the majority were published in the 18–19th Century\(British Library Labs,[2021](https://arxiv.org/html/2606.02991#bib.bib4)\)\. This dataset provides page\-level OCR score, and we only keep pages with OCR scores exceeding 80\. It contributes 914\.66 M tokens after filtering\.

#### Hansard\.

This dataset contains the official records of parliamentary proceedings and debates across the United Kingdom’s legislative bodies, known as Hansard\(Brezina,[2024](https://arxiv.org/html/2606.02991#bib.bib5)\)\. It represents a comprehensive collection of parliamentary discourse since 1802\. After data cleaning, it contributes 193\.05 M tokens\.

#### Royal Society Corpus v6\.0\.

It is a diachronic corpus of scientific English spanning more than 300 years of scientific writing \(1665–1996\)\. It contains primarily scientific articles, derived from publications of the Royal Society of London\(Fischeret al\.,[2020](https://arxiv.org/html/2606.02991#bib.bib2)\)\. It contributes 70\.38 M tokens\.

#### Old Bailey Corpus\.

This corpus captures speech\-related uses of Late Modern English in London’s Central Criminal Court\(Huberet al\.,[2016](https://arxiv.org/html/2606.02991#bib.bib10)\)\. It contributes 38\.51 M tokens\.

#### The Corpus of Late Modern English Texts v3\.1 \(CLMET\)\.

The corpus covers the period 1710–1920, covering five major genres: narrative fiction, narrative non\-fiction, drama, letters and treatise, and unclassified texts\(Dilleret al\.,[2011](https://arxiv.org/html/2606.02991#bib.bib9)\)\. After filtering to 1913, it contributes 31\.46 M tokens\.

#### Corpus of English Dialogues \(CED\)\.

This corpus contains dialogues from literary and didactic works from 1560 to 1760\(Kytö and Culpeper,[2006](https://arxiv.org/html/2606.02991#bib.bib12)\)\. After filtering, it contributes 0\.67 million tokens\.

#### The Lampeter Corpus of Early Modern English Tracts \(Lampeter\)\.

The Lampeter Corpus comprises political, economic, and religious pamphlets and tracts from 1640–1740\(Siemund and Claridge,[1997](https://arxiv.org/html/2606.02991#bib.bib11)\)\. After filtering, it contribute 0\.35 million tokens\.

#### Corpus of Late Modern English Prose \(CLME Prose\)\.

This corpus provides selected prose texts from the late Middle and early Modern English periods\(Denison,[1994](https://arxiv.org/html/2606.02991#bib.bib51)\)\. After filtering to the target window, it contributes 0\.12 million tokens\.

#### Pamphlets\.

This corpus contains curated set of five historical pamphlets\(Bailyn,[1965](https://arxiv.org/html/2606.02991#bib.bib50)\)in clean digitized form, and contributes 0\.08 million tokens\.

### A\.2Training details

Our tokenizer adds<bos\>before a document and appends<eos\>after it\.

Our base model is a 7\.24B\-parameter decoder\-only Transformer following the Llama 3 architecture\(Llama Team, AI @ Meta,[2024](https://arxiv.org/html/2606.02991#bib.bib14)\)\. It has 32 layers, hidden size 4096, and SwiGLU feed\-forward blocks with inner dimension 14336\(Shazeer,[2020](https://arxiv.org/html/2606.02991#bib.bib15)\)\. Attention uses 32 query heads with grouped\-query attention\(GQA; Ainslieet al\.,[2023](https://arxiv.org/html/2606.02991#bib.bib16)\)sharing 8 key/value heads, with head dimension 128\. We use RMSNorm\(Zhang and Sennrich,[2019](https://arxiv.org/html/2606.02991#bib.bib18)\), rotary position embeddings\(Suet al\.,[2024](https://arxiv.org/html/2606.02991#bib.bib19)\), and a context length of 8192 tokens\.

We optimize with AdamW\(Loshchilov and Hutter,[2019](https://arxiv.org/html/2606.02991#bib.bib20)\)\(β1=0\.9\\beta\_\{1\}\{=\}0\.9,β2=0\.95\\beta\_\{2\}\{=\}0\.95, weight decay 0\.1\) and gradient clipping at1\.01\.0\. The learning rate follows a cosine schedule with peak3×10−43\{\\times\}10^\{\-4\}, minimum3×10−53\{\\times\}10^\{\-5\}\. We use bfloat16 mixed precision for training\. Pre\-training uses approximately 140 B tokens, approximately2\.592\.59epochs over the training corpus\.

## Appendix BInstruction Tuning

More details about instruction tuning can be found in Table[7](https://arxiv.org/html/2606.02991#A2.T7)\.

Table 7:LoRA SFT hyperparameters\. Both adapters target all attention and MLP projection matrices and additionally fine\-tune the token embedding and LM head to accommodate ChatML special tokens\.
## Appendix CEvaluation Details

### C\.1Prompts

#### Judge: Strict Condition\.

\{internallinenumbers\*\}You score a model’s answer to a historical\-event question against ground truth\. The model was SHOWN the event description in the question, so simply restating or paraphrasing that description is NOT evidence of knowledge\. You are given the event description, the CORRECT YEAR \(from Wikipedia\), a REFERENCE answer, and the MODEL’s answer\.\{internallinenumbers\*\}Award score = 1 if and only if BOTH hold: \(a\) the answer states the correct year \(event\_year\); and \(b\) it provides correct, specific information about the event that GOES BEYOND the description \(e\.g\., the outcome, consequences, named participants, or the date\), consistent with the reference \-\- not a restatement of the description and not a plausible\-sounding guess\. Award score = 0 otherwise\.Return ONLY valid JSON with no markdown fences:"score": 0 or 1, "reasoning": "<one sentence\>"

#### Judged: Relaxed Condition\.

\{internallinenumbers\*\}You judge whether a model’s answer shows knowledge of a historical event BEYOND what it was told\. The model was SHOWN the event description in the question, so merely restating or paraphrasing that description is NOT evidence of knowledge\. You are given the event description, the correct year \(from Wikipedia\), a REFERENCE answer, and the MODEL’s answer\.\{internallinenumbers\*\}Award score = 1 if the answer contains at least some correct, specific information about the event that GOES BEYOND the description \(e\.g\., the date, outcome, consequences, or named participants not already stated in the description\), consistent with the reference\. Award score = 0 if the answer only restates or paraphrases the description, is vague, is wrong, or hallucinates\.Return ONLY valid JSON with no markdown fences:"score": 0 or 1, "reasoning": "<one sentence\>"

#### Judge: User Message \(both criteria\)\.

Event description \(shown to the model\): event\_descriptionCorrect year \(ground truth\): event\_yearReference answer: gold\_answerModel’s answer:model\_answer

### C\.2Dataset and Filtering

We scraped the 2,344 historical events from the following four Wikipedia pages, accessed in May 2026:

- •
- •
- •
- •

We filter them as follows:

1. 1\.Remove events whose description contains any four\-digit year:2,344→2,1482\{,\}344\\rightarrow 2\{,\}148\(−196\-196\)\.
2. 2\.Keep only events Gemini\-3\.1\-Flash\-Lite answers correctly:→𝟏,𝟕𝟐𝟔\\rightarrow\\mathbf\{1\{,\}726\}events retained\.

![Refer to caption](https://arxiv.org/html/2606.02991v1/fig/histevent_distribution.png)Figure 4:Number ofHist\-Eventevents per decade \(1700–2025\)\.
### C\.3Full BPB Statistics

The full BPB statistics is provided in Table[8](https://arxiv.org/html/2606.02991#A3.T8)\.

Table 8:Bits\-per\-byte before/after each model’s cutoff\. Ratio = post/pre\. Llama \(modern baseline\) is included for reference, but only includes 38 events post cutoff\.
### C\.4Models

An overview of the models we used is in Table[9](https://arxiv.org/html/2606.02991#A3.T9)\.

Table 9:Overview of the evaluated models
### C\.5Post\-Cutoff Leakage Cases

#### TypewriterLMep10 — Girl Scout Cookies, 1917 \(cutoff 1913\)\.

Q:*Do you know about the following event: ‘The first known sale of Girl Scout Cookies begins’? If so, explain what this event was and in what year did it take place?* A:“Youth’s Companion for November 29, 1917\. The sale of Girl Scouts cookies at ten cents a package opened with 200 dozen sold the first day\. \[…\]”

#### TypewriterLMep10 — Civil Rights Act, 1964 \(cutoff 1913\)\.

Q:*Do you know about the following event: ‘Civil Rights Act abolishes segregation in the USA’? If so, explain what this event was and in what year did it take place?* A:“The Civil Rights Act of 1964 abolished segregation in the USA and expanded freedom of choice by eliminating barriers to access to many of the social services and educational facilities previously prohibited\. \[…\]”

#### Talkie 1930 \(it\) — New Deal, 1933 \(cutoff 1930\)\.

Q:*Do you know about the following event: ‘New Deal begins in America’? If so, explain what this event was and in what year did it take place?* A:“In 1933, the New Deal began in the United States when Congress passed a bill establishing a national economic council to plan social and economic reconstruction\. \[…\]”