FrontierMat：面向材料研发前沿的语言模型基准测试

# Benchmarking Language Models at the Frontier of Materials R&D
Source: [https://arxiv.org/html/2610.41162](https://arxiv.org/html/2610.41162)
Jianguo HuangYushan XiaoYue LiBijun TangXuyu DongXingguo WangFeiyang XieJianhui GuDongfeng CaoHongda ChenYongji GongRuiqi LiShuxin LiXingji LiRuitao LvLi SongLei WangAffiliation:Nanyang Technological University, Singapore Peking University Shanghai Jiao Tong University Xi’an Jiaotong University Wuhan University of Technology Beihang University Eighth Research Institute of Nuclear Industry, CNNC Harbin Institute of Technology Tsinghua University University of Science and Technology of China Commercial Aircraft Corporation of China \(COMAC\) Tongji University Northwestern Polytechnical University Wanhua Chemical Group Co\., Ltd\. Correspondence:feng012@e\.ntu\.edu\.sg,boan@ntu\.edu\.sg,Z\.Liu@ntu\.edu\.sgYe ZhangXiaoxu ZhaoXitao ZhengBo AnZheng Liu

###### Abstract

To accelerate materials discovery, language models must be able to address problems at the research frontier\. Yet textbook\- and examination\-derived benchmarks primarily assess established knowledge, leaving performance on frontier research problems largely untested\. We introduce FrontierMat, a benchmark of 330 problems drawn from frontier academic and industrial research in fields including aerospace, semiconductors, embodied intelligence and nuclear fusion\. Every problem is validated by leading experimentalists and industry experts in its field\. Each problem presents a real study and asks the model to infer a withheld outcome or draw a conclusion from selected experimental evidence\. Problems are annotated by functional domain, material system and the scale of the governing mechanism; 130 require cross\-scale reasoning\. The strongest general\-purpose model scores only 53\.7 %, and on matched problems, cross\-scale problems score 8\.7 points below single\-scale ones\. Half of the problems are solved by at most one model, and models fail chiefly by endorsing what is plausible rather than what was observed\. A domain\-adaptation study with meta\-weighted continued pretraining and scale\-modular self\-distillation raises an open\-weight model from 36\.7 % to 55\.4 %, matching the strongest proprietary model, with improvements on public materials benchmarks and no decline in aggregate performance on general benchmarks\. FrontierMat provides a testbed for measuring progress toward language models that support frontier materials R&D\.

††footnotetext:∗Equal contribution\.††footnotetext:†Authors from Dongfeng Cao through Xitao Zheng contributed as expert reviewers from academia and industry and are listed alphabetically by surname\.## 1 Introduction

![Refer to caption](https://arxiv.org/html/2610.41162v1/FrontierMat_Leaderboard_v3_colm.png)Figure 1:FrontierMat leaderboard of six evaluated models\. Bars split each overall score by the scale of the governing mechanism\. Raw cross\-scale scores are higher because these problems are more often single\-answer; §[5\.2](https://arxiv.org/html/2610.41162#S5.SS2)compares matched problems\.Lighter airframes, faster chips, dexterous robots and viable fusion reactors require materials advances pursued by academic laboratories and industrial R&D teams\. Language models are increasingly involved in this work: tool\-augmented models already plan and run chemical experiments\([Boiko et al\., 2023](https://arxiv.org/html/2610.41162#bib.bib14);[Bran et al\., 2024](https://arxiv.org/html/2610.41162#bib.bib15)\), and their readiness for materials discovery is under active examination\([Miret and Krishnan, 2024](https://arxiv.org/html/2610.41162#bib.bib16)\)\. Their value depends on addressing problems at this frontier\. Such problems concern a specific material system under particular application constraints, often requiring researchers to connect defects and local structure to grains, fibres and pores, and ultimately to specimen or device performance\. Answering them requires experimental evidence\.

Current benchmarks provide limited evidence of this capability\. Materials benchmarks draw on examinations and textbooks \(MaScQA,[Zaki et al\., 2024](https://arxiv.org/html/2610.41162#bib.bib4); MaterialBENCH,[Yoshitake and others, 2024](https://arxiv.org/html/2610.41162#bib.bib5); MatSciBench,[Zhang and others, 2025](https://arxiv.org/html/2610.41162#bib.bib6)\) or use language models to generate problems from papers, textbooks and databases \(MSQA,[Cheung et al\., 2025](https://arxiv.org/html/2610.41162#bib.bib10); SciKnowEval,[Feng et al\., 2024](https://arxiv.org/html/2610.41162#bib.bib11)\)\. Beyond materials, SuperGPQA\([M\-A\-P Team, 2025](https://arxiv.org/html/2610.41162#bib.bib12)\)screens candidate problems with model responses, and ChemBench\([Mirza et al\., 2025](https://arxiv.org/html/2610.41162#bib.bib3)\)labels difficulty by expert judgement\. These benchmarks do not systematically combine problems from active R&D with annotations of the mechanisms linking atomic structure to component performance, a central requirement of materials design\.

We introduceFrontierMat \([Figure2](https://arxiv.org/html/2610.41162#S1.F2)\), a benchmark for evaluating language models on frontier materials R&D\. Its 330 problems follow three design principles:

- •Grounded in frontier research\.Problems draw on academic and industrial research in fields including aerospace, semiconductors, embodied intelligence and nuclear fusion\. Leading experimentalists and industry experts in each field validated every problem as relevant to their work\.
- •Research outcomes as answers\.Each problem asks for an experimental outcome or conclusion supported by a real study, requiring prediction from its setting or interpretation of evidence\.
- •Explicit domain and scale annotations\.Problems are labelled by functional domain, material system and the scale of the governing mechanism: micro, meso or macro\. The 130 cross\-scale problems also record the reasoning chain\.

Evaluating seven proprietary, open\-weight and domain\-adapted models, we find:

- •Performance remains limited\.The strongest general\-purpose model, GPT\-6\-Astra, scores 53\.7 % \([Figure1](https://arxiv.org/html/2610.41162#S1.F1)\), 129 problems are solved by no general\-purpose model, and half are solved by at most one \(§[4](https://arxiv.org/html/2610.41162#S4), §[5\.2](https://arxiv.org/html/2610.41162#S5.SS2)\)\.
- •Cross\-scale reasoning is difficult\.With answer format and construction channel held fixed, cross\-scale problems are the hardest, 8\.7 points below single\-scale problems \(§[5\.2](https://arxiv.org/html/2610.41162#S5.SS2)\)\.
- •Models select unsupported outcomes\.83 % of parsed, incorrect responses to multi\-answer problems endorse an outcome the study did not report \(§[5\.3](https://arxiv.org/html/2610.41162#S5.SS3)\)\.

We also use FrontierMat to guide domain adaptation of Qwen3\.5\-35B\-A3B through continued pretraining \(CPT\) and supervised fine\-tuning \(SFT\) with two scale\-aware components: CPT data weighted by its benefit to cross\-scale problems, and scale\-specific modules trained by self\-distillation \(§[6](https://arxiv.org/html/2610.41162#S6)\)\. Adaptation raises its FrontierMat score from 36\.7 % for the official post\-trained release to 55\.4 %, matching the strongest proprietary model, with gains at every scale and on public materials benchmarks and no decline in mean performance on the general benchmarks evaluated\.

#### Contributions\.

1. 1\.FrontierMat, 330 problems from frontier materials research with research outcomes as answers and domain and scale annotations \(§[3](https://arxiv.org/html/2610.41162#S3)\)\.
2. 2\.An evaluation of seven models analysing where difficulty lies and how models err \(§[4](https://arxiv.org/html/2610.41162#S4)–[5](https://arxiv.org/html/2610.41162#S5)\)\.
3. 3\.A scale\-aware adaptation recipe that adds 18\.7 points to an open\-weight model and transfers to materials benchmarks \(§[6](https://arxiv.org/html/2610.41162#S6)\)\.

aBenchmark constructionbInfer the outcome of a real study1Frontier researchAcademia \+ industry studies with outcomes2Construct and annotateSetting \+ candidate options Domain, system and scale3Verify against evidenceSource and expert checks330 problems⋅\\cdotexpert\-verifiedGIVEN TO THE MODELAerospace⋅\\cdotmechanicalBraided carbon/epoxy compositesGiven: study setting and related data\. Predict the target architecture’s strength\.Braid schematicTarget Six\-directional braid at40∘40^\{\\circ\} Model task Choose an outcome\.What did the experiment find?WITHHELD FOR SCORINGExperimental result≈\\approx328MPaSix\-directional braiding has the highest strength at40∘40^\{\\circ\}\.Source:Fig\. 9\(b\), Table 1 Reading resolution:±\\pm10 MPaCROSS\-SCALE REASONING130 cross\-scale problemsYarn architecturemeso\-scale structureLoad sharinglinking mechanismCompressive strengthmacro\-scale response

Figure 2:Overview of FrontierMat\.\(a\)A unified pipeline constructs and verifies problems from frontier academic and industrial research\.\(b\)An abridged example separates the study setting given to the model from the experimental result withheld for scoring\. The schematic and reasoning chain illustrate how yarn architecture connects to macroscopic strength; full problem details appear in[AppendixG](https://arxiv.org/html/2610.41162#A7)\.

## 2 Related Work

#### Problem sources\.

Materials benchmarks for language models primarily use educational material or problems generated from existing text\. MaScQA\([Zaki et al\., 2024](https://arxiv.org/html/2610.41162#bib.bib4)\)collects 650 problems from a graduate entrance examination, while MaterialBENCH\([Yoshitake and others, 2024](https://arxiv.org/html/2610.41162#bib.bib5)\)and MatSciBench\([Zhang and others, 2025](https://arxiv.org/html/2610.41162#bib.bib6)\)draw on university textbooks; MatSciBench uses a 31\-subfield taxonomy\. MSQA\([Cheung et al\., 2025](https://arxiv.org/html/2610.41162#bib.bib10)\)and SciKnowEval\([Feng et al\., 2024](https://arxiv.org/html/2610.41162#bib.bib11)\)use language models to generate problems from papers, textbooks and databases\. Beyond materials, ChemBench\([Mirza et al\., 2025](https://arxiv.org/html/2610.41162#bib.bib3)\)assesses chemical knowledge, reasoning and intuition, GPQA\([Rein et al\., 2024](https://arxiv.org/html/2610.41162#bib.bib1)\)collects expert\-written graduate\-level questions, and SuperGPQA\([M\-A\-P Team, 2025](https://arxiv.org/html/2610.41162#bib.bib12)\)and MMLU\-Pro\([Wang et al\., 2024](https://arxiv.org/html/2610.41162#bib.bib2)\)build harder multi\-discipline problem sets\. Other benchmarks target property prediction from composition or structure\([Dunn et al\., 2020](https://arxiv.org/html/2610.41162#bib.bib13);[Rubungo and others, 2024](https://arxiv.org/html/2610.41162#bib.bib18)\), the use of materials simulation tools\([Liu and others, 2025](https://arxiv.org/html/2610.41162#bib.bib19)\)or scientific images\([Alampara and others, 2025](https://arxiv.org/html/2610.41162#bib.bib17)\)\.

#### Answer targets and scale\.

The problem\-solving benchmarks above take their answers from textbooks, answer keys or source documents, and their tasks range from recall to calculation, reasoning and prediction\. FrontierMat centres on experimental outcomes and conclusions from active research, asking models to infer withheld results or interpret selected study evidence \([Table1](https://arxiv.org/html/2610.41162#S2.T1)\)\. Existing benchmarks organise problems by subfield or format but do not annotate the scale of the mechanism that determines the answer\.

#### Sources of difficulty\.

Benchmark difficulty may reflect the level of study, expert labels \(ChemBench\), tiers assigned from model behaviour \(MatSciBench uses the response lengths of six models\), or model\-based screening \(SuperGPQA\)\. In FrontierMat, difficulty comes from the research problems themselves\. Following work on construct validity\([Bean et al\., 2025](https://arxiv.org/html/2610.41162#bib.bib8)\), signal and noise in evaluation\([Heineman and others, 2025](https://arxiv.org/html/2610.41162#bib.bib9)\)and errors in benchmark labels\([Gema et al\., 2025](https://arxiv.org/html/2610.41162#bib.bib7)\), we test whether its verdicts are stable and its problems discriminate \(§[4\.3](https://arxiv.org/html/2610.41162#S4.SS3)\)\.

BenchmarkProblem sourceAnswer isFrontierresearchScalelabelsR&DpractitionersDifficulty from*Teaching material*MaScQA\([Zaki et al\., 2024](https://arxiv.org/html/2610.41162#bib.bib4)\)Entrance examinationTextbook knowledge✗✗✗Exam levelMaterialBENCH\([Yoshitake and others, 2024](https://arxiv.org/html/2610.41162#bib.bib5)\)Two textbooksTextbook knowledge✗✗✗Course levelMatSciBench\([Zhang and others, 2025](https://arxiv.org/html/2610.41162#bib.bib6)\)Ten textbooksTextbook knowledge✗✗✗Tiers by LLM response length*Generated from text*MSQA\([Cheung et al\., 2025](https://arxiv.org/html/2610.41162#bib.bib10)\)Papers \(2000–2021\)Methods and results in source papers✗✗✗–SciKnowEval\([Feng et al\., 2024](https://arxiv.org/html/2610.41162#bib.bib11)\)Papers, textbooks, databasesKnowledge, prediction and application tasks✗✗✗Five knowledge levels*Expert\-curated, multi\-domain*ChemBench\([Mirza et al\., 2025](https://arxiv.org/html/2610.41162#bib.bib3)\)Experts, exams, databasesKnowledge, reasoning, intuition✗✗✗Basic / advanced labelsSuperGPQA\([M\-A\-P Team, 2025](https://arxiv.org/html/2610.41162#bib.bib12)\)Expert\-collected resourcesGraduate\-level knowledge and reasoning✗✗✗Screening with LLMsFrontierMat \(ours\)Frontier academic and industrial researchWhat a study found✓✓✓Frontier research outcomesTable 1:Comparison of FrontierMat with related benchmarks\.Answer is: what the correct answer draws on; FrontierMat targets experimental outcomes and conclusions drawn from study evidence\.Frontier research: problems come from current research rather than established curricula\.Scale labels: each problem records the scale of the governing mechanism and whether it crosses scales\.R&D practitioners: researchers in active academic and industrial R&D contributed research problems and assessed their relevance to current work; ✗: not reported \(expert authoring or review alone does not count\)\.Difficulty from: how problem difficulty is established or controlled\.
#### Domain adaptation of LLMs for science\.

Continued pretraining on in\-domain text is an established route to specialisation\([Gururangan et al\., 2020](https://arxiv.org/html/2610.41162#bib.bib20)\)\. In materials science, MatSciBERT\([Gupta et al\., 2022](https://arxiv.org/html/2610.41162#bib.bib21)\)pretrains an encoder on materials text and HoneyBee\([Song et al\., 2023](https://arxiv.org/html/2610.41162#bib.bib22)\)applies progressive instruction tuning\. Continual instruction tuning can cause catastrophic forgetting of general abilities\([Luo et al\., 2023](https://arxiv.org/html/2610.41162#bib.bib23)\), and on MSQA several domain\-specialised models trail general\-purpose ones\([Cheung et al\., 2025](https://arxiv.org/html/2610.41162#bib.bib10)\)\. Our recipe builds on example reweighting by meta\-gradients\([Ren et al\., 2018](https://arxiv.org/html/2610.41162#bib.bib24)\), domain\-mixture optimisation\([Xie et al\., 2023](https://arxiv.org/html/2610.41162#bib.bib25)\), bottleneck adapters\([Houlsby et al\., 2019](https://arxiv.org/html/2610.41162#bib.bib28)\)and distillation from privileged context\([Lopez\-Paz et al\., 2016](https://arxiv.org/html/2610.41162#bib.bib26);[Snell et al\., 2022](https://arxiv.org/html/2610.41162#bib.bib27)\); organising them by scale is our extension\. It is evaluated on frontier problems rather than textbook tasks \(§[6](https://arxiv.org/html/2610.41162#S6)\)\.

## 3 The FrontierMat Benchmark

FrontierMat follows the three principles in §[1](https://arxiv.org/html/2610.41162#S1): it draws problems from frontier research, asks models to infer study outcomes, and annotates domain and scale\. We describe construction in §[3\.1](https://arxiv.org/html/2610.41162#S3.SS1)and coverage in §[3\.2](https://arxiv.org/html/2610.41162#S3.SS2)\.

### 3\.1 Construction

#### Mining frontier studies\.

We worked with academic laboratories and industrial R&D teams at the research frontier to identify problems from their recent work, including results not yet published\. We also drew on experimental studies in specialist journals across aerospace, semiconductors, embodied intelligence and nuclear fusion\. These sources enter a common construction pipeline: each study must provide a measured outcome or a conclusion supported by data, with sufficient context to formulate a problem\. Experts assess its relevance to current research, rather than publication date alone \([AppendixB](https://arxiv.org/html/2610.41162#A2)\)\.

#### Writing problems\.

Each problem presents a study setting and selected evidence, then asks the model to identify a target outcome or draw a conclusion from the evidence\. Some problems require quantitative derivation rather than prediction of an unreported measurement\. All use single\-answer or multi\-answer multiple\-choice formats\. Distractors differ from the supported result in direction, magnitude or mechanism\. Annotators with graduate training in materials science follow shared guidelines and record the source evidence for each answer, withholding the answer and its evidence locator from the model \([AppendixB](https://arxiv.org/html/2610.41162#A2)\)\.

#### Verification and expert review\.

Verification proceeds in two stages\. Two language models independently check each problem for consistency with its source\. Leading experimentalists and industry experts then review every problem in their field for relevance to their work, significance of the withheld outcome, and whether distractors represent plausible mistakes by a competent researcher\. Experts familiar with the originating work review problems drawn from partner research\. Problems that failed any criterion were revised or removed \([AppendixB](https://arxiv.org/html/2610.41162#A2)\)\. The problem set, options and answers were frozen before any model evaluation\.

### 3\.2 Domain and scale

We annotate every problem by domain and scale to characterise its scientific context and reasoning demands \([Figure3](https://arxiv.org/html/2610.41162#S3.F3)\)\.

Figure 3:Composition of all 330 problems in FrontierMat\.Left:problems by scale of the governing mechanism \(inner ring\) and, within each scale, by functional domain \(outer ring\)\.Right:problems by functional domain \(inner ring\) and, within each domain, by material system \(outer bars; percentages are shares within the domain\)\.#### Domain\.

The*functional domain*identifies the property under investigation:*mechanical*\(strength, stiffness, fatigue, damage\),*thermal*\(conduction, expansion, stability\),*electrical*,*electromagnetic*\(shielding, absorption, wave control\),*protective*\(resistance to ablation, oxidation, irradiation and impact\), and*multifunctional trade\-off*, which requires balancing two or more properties\. The*material system*identifies the material class: polymer\-matrix, filler–polymer, ceramic\-matrix, metal\-matrix and carbon/carbon composites, metamaterials, and others\. Mechanical problems form the largest domain \(40 %\)\. Matrix composites dominate this domain, while filler–polymer systems dominate electromagnetic and electrical problems\.

#### Scale\.

Scale denotes the structural and mechanistic level that determines the answer, rather than a strict length scale\.*Micro*concerns atoms, lattices, defects and local structure;*meso*concerns grains, phase domains, fibres or particles, pores and microstructure;*macro*concerns specimens, devices and bulk response\. A problem is*cross\-scale*when the answer requires linking levels, for example predicting a macroscopic property from micro\- or mesostructure; we also annotate the reasoning chain \(e\.g\.,*yarn architecture→\\rightarrowload sharing→\\rightarrowstructural strength*\)\. FrontierMat contains 200 single\-scale problems \(88 micro, 24 meso, 88 macro\) and 130 cross\-scale problems\. All domain–scale combinations except meso\-scale electromagnetic problems are represented \([Figure3](https://arxiv.org/html/2610.41162#S3.F3), left\), supporting analysis of performance across domains and scales \(§[5\.2](https://arxiv.org/html/2610.41162#S5.SS2)\)\. Domain distributions vary across scales: multifunctional trade\-offs are most numerous at the macro scale, and electrical problems are predominantly cross\-scale \(24 of 39\)\. Labelling rules are given in[AppendixA](https://arxiv.org/html/2610.41162#A1)\.

Full statistics appear in[Table5](https://arxiv.org/html/2610.41162#A1.T5)\([AppendixA](https://arxiv.org/html/2610.41162#A1)\)\. Coverage varies across functional domains and scales, so we report results by domain and scale without claiming uniform coverage of the field\.

Scale of the governing mechanismDomain groupModelReasoningAllMicroMesoMacroCrossStructuralFunctional\# problems330882488130159171Proprietary frontier modelsGPT\-6\-Astraxhigh53\.751\.147\.939\.266\.355\.152\.3GPT\-6\-Solxhigh51\.448\.350\.034\.165\.450\.052\.6Claude Opus 5\.5max47\.343\.235\.427\.365\.847\.846\.8Open\-weight frontier modelsDeepSeek V4 Prothinking47\.652\.327\.127\.861\.549\.146\.2GLM\-5\.3thinking45\.043\.831\.223\.962\.745\.344\.7Qwen3\.5\-35B\-A3B \(open\-weight, 3B active parameters\)Official post\-trainedthinking36\.730\.322\.023\.852\.435\.737\.6FrontierMat\-Model \(ours\)thinking55\.457\.535\.341\.667\.157\.353\.7gain from adaptation\+18\.7\+27\.2\+13\.3\+17\.8\+14\.7\+21\.6\+16\.1Table 2:Score \(%\) on FrontierMat \(scoring rule in §[4\.1](https://arxiv.org/html/2610.41162#S4.SS1)\)\.Reasoning: the highest reasoning setting each model offers\.Domain group: structural problems \(mechanical, protective\) and functional problems \(thermal, electrical, electromagnetic, multifunctional trade\-off\); per\-domain scores in[AppendixD](https://arxiv.org/html/2610.41162#A4)\. The two Qwen rows are means over 16 samples with identical prompts and decoding \(§[6](https://arxiv.org/html/2610.41162#S6)\); cross\-scale and single\-scale columns differ in answer format and construction channel \(§[5\.2](https://arxiv.org/html/2610.41162#S5.SS2)\)\. Bold: best per column; underlined: second best\.

## 4 Evaluating Language Models

### 4\.1 Setup

#### Models\.

We evaluate seven models: three proprietary frontier models \(GPT\-6\-Astra, GPT\-6\-Sol and Claude Opus 5\.5\), three open\-weight models \(DeepSeek V4 Pro, GLM\-5\.3 and the official post\-trained Qwen3\.5\-35B\-A3B\), and FrontierMat\-Model, our domain\-adapted Qwen3\.5\-35B\-A3B \(§[6](https://arxiv.org/html/2610.41162#S6)\)\. We enable reasoning mode where available\. FrontierMat\-Model and the official Qwen3\.5\-35B\-A3B use identical prompts, decoding settings and a 32k\-token output limit; their scores average 16 samples per problem\.[AppendixC](https://arxiv.org/html/2610.41162#A3)gives reasoning and decoding settings\.

#### Protocol and scoring\.

Evaluation is closed\-book: each model receives only the study setting, candidate options and required answer format\. We withhold answers, sources and other problems, and disable tools unless stated otherwise\. A response scores 1 if its predicted option set matches the gold set, 0\.5 if it is a non\-empty proper subset of the gold set, and 0 if it contains any wrong option; single\-answer problems are therefore scored 1 or 0\. Unparseable outputs score 0\. We report mean scores as percentages\.

#### Statistics\.

We report 95% confidence intervals from 10,000 bootstrap resamples of problems and compare models using paired permutation tests\. We report differences that are not significant atp<0\.05p<0\.05as unresolved\.

### 4\.2 Main results

#### Performance on frontier problems\.

Finding 1:Performance on frontier problems remains limited: the strongest general\-purpose model achieves barely half the available score on FrontierMat, leaving substantial room for improvement\.

GPT\-6\-Astra scores 53\.7 % \(95% CI 48\.9–58\.4;[Table2](https://arxiv.org/html/2610.41162#S3.T2)\), significantly ahead of the open\-weight DeepSeek V4 Pro \(47\.6 %\) and GLM\-5\.3 \(45\.0 %; paired permutation test,p<0\.01p<0\.01\)\. GPT\-6\-Sol \(51\.4 %\) does not differ significantly from GPT\-6\-Astra or DeepSeek V4 Pro, nor does Claude Opus 5\.5 \(47\.3 %\) differ from either open\-weight model\. Domain adaptation raises the official Qwen3\.5\-35B\-A3B from 36\.7 % to 55\.4 %, the highest overall score but within the CI of GPT\-6\-Astra\.

#### Performance across domains and scales\.

No model exceeds 67\.1% in any scale column of[Table2](https://arxiv.org/html/2610.41162#S3.T2), and scores by functional domain range from 32\.2% to 65\.2% \([AppendixD](https://arxiv.org/html/2610.41162#A4)\)\. FrontierMat\-Model leads on both structural \(57\.3%\) and functional problems \(53\.7%\)\. Because answer format and construction channel vary across scales, we compare matched problems in §[5\.2](https://arxiv.org/html/2610.41162#S5.SS2)\.

### 4\.3 Is FrontierMat a reliable instrument?

We assess ranking stability and whether individual problems distinguish stronger from weaker models using per\-problem scores \([AppendixE](https://arxiv.org/html/2610.41162#A5)\)\.

Finding 2:FrontierMat yields stable rankings among the five general\-purpose models analysed, and most informative problems have positive discrimination\. With 39 % of problems unsolved by any general\-purpose model, it offers headroom beyond saturated benchmarks\.

#### Rankings are stable\.

Resampling the problems 10,000 times tests ranking stability among five general\-purpose models: GPT\-6\-Astra ranks first in 97 % of resamples, and the two GPT\-6 models remain ahead of the other three in 94 %\.

#### Problems discriminate and leave headroom\.

For each of the 177 problems with non\-constant scores across six models, we correlate its scores with scores on the remaining problems\. 73 % have positive correlations \(median 0\.36\), indicating that stronger models generally perform better on these problems\. 129 problems remain unsolved by every general\-purpose model \(§[5\.2](https://arxiv.org/html/2610.41162#S5.SS2)\), indicating room for improvement beyond a near\-ceiling benchmark \(MaScQA: 94\.8 % versus 94\.9 % for the two Qwen models that FrontierMat separates by 18\.7 points\)\.

## 5 What FrontierMat Reveals

Aggregate scores do not explain which research problems models find difficult or why they fail\. Using FrontierMat’s annotations and the per\-problem responses of six models \([AppendixE](https://arxiv.org/html/2610.41162#A5)\), we ask where difficulty lies and how models fail\.

### 5\.1 Expert validation

The first test of a benchmark that claims to sample the frontier is whether the people working at that frontier recognise its problems\. Every FrontierMat problem was reviewed by leading experimentalists and industry experts in its field, who assessed whether it poses a question their own research or development work faces, whether the withheld outcome is the kind of result their work turns on, and whether the wrong options are mistakes a competent colleague could make\. Problems that failed any of these tests were revised or removed \(§[3\.1](https://arxiv.org/html/2610.41162#S3.SS1);[AppendixB](https://arxiv.org/html/2610.41162#A2)\)\. The problems that remain are, in the judgement of practitioners, questions from their own work rather than exercises about it\.

### 5\.2 Where does the difficulty lie?

Figure 4:Where and how models fail on FrontierMat\.\(a\)Responses to the 232 multi\-answer problems\.\(b\)Scores by scale on 202 problems with answer format and construction channel held fixed \(bars: mean of five general\-purpose models, dots: individual models\)\.\(c\)Problems by the number of general\-purpose models solving them\.Finding 3:With answer format and construction channel held fixed, cross\-scale problems are the most difficult\. Performance is also lowest in multi\-mechanism domains\.

#### Cross\-scale problems are the hardest\.

The aggregate scores in[Table2](https://arxiv.org/html/2610.41162#S3.T2)are higher for cross\-scale problems \(66\.3% versus 45\.5% for GPT\-6\-Astra\), but the groups differ in composition: cross\-scale problems are more often single\-answer, and one construction channel contributes only cross\-scale problems\. Controlling for both factors reverses this pattern\. Within the channel containing both scale types, the five general\-purpose models average 29\.5 % on micro, 32\.7 % on meso and 23\.0 % on macro problems, but only 17\.5 % on cross\-scale problems among the 202 multi\-answer problems \([Figure4](https://arxiv.org/html/2610.41162#S5.F4)b\), lowest for 4 of five models\. After weighting within\-format differences by cross\-scale problem counts \([AppendixE](https://arxiv.org/html/2610.41162#A5)\), the cross\-scale score is 8\.7 points below single\-scale problems \(stratified permutation test, one\-sidedp=0\.008p=0\.008\), with model\-specific gaps from 4\.6 points for GLM\-5\.3 to 11\.3 for DeepSeek V4 Pro\. On the same matched problems, scores are lowest for electrical problems \(19\.4 %\), which are mostly cross\-scale because composite conductivity depends on filler dispersion and interfaces, and multifunctional trade\-offs \(19\.8 %\), which require balancing competing properties\. These patterns point to difficulty linking scales and mechanisms\.

#### Half of the problems are solved by at most one model\.

Difficulty varies substantially across problems \([Figure4](https://arxiv.org/html/2610.41162#S5.F4)c\)\. 105 problems are answered exactly by all five general\-purpose models, while 164 \(50 %\) are solved by at most one of them and 129 by none, with few problems between these extremes; on 81 of the unsolved problems at least three models give the same wrong answer \(gold in[Figure4](https://arxiv.org/html/2610.41162#S5.F4)c\)\. They span every scale and domain \(17–56 % of each domain\)\.

### 5\.3 How do models fail?

Finding 4:Most errors include a plausible option unsupported by the study; the strongest general\-purpose model makes this error least often\.

#### Over\-endorsement dominates\.

Multi\-answer problems require models to distinguish reported findings from other plausible outcomes\. Of the general\-purpose models’ errors with a parsed answer on these problems, 83 % include at least one option the study did not support, and 49 % contain every correct option together with at least one unsupported option \([Figure4](https://arxiv.org/html/2610.41162#S5.F4)a\)\. Claude Opus 5\.5 selects 3\.03 options on average, compared with 2\.33 in the reference answer, and four of the five general\-purpose models select strict supersets of the correct answer on 35–49 % of these problems\. The corresponding rates are only 19 % for GPT\-6\-Astra and 5 % for FrontierMat\-Model\. GPT\-6\-Astra instead omits correct options on 30 % of problems\. Avoiding unsupported options distinguishes the strongest models\.

#### Inferring the reported outcome\.

Model explanations illustrate both response patterns \([AppendixH](https://arxiv.org/html/2610.41162#A8)\)\. On graphene\-reinforced Cu–Cr–Zr alloys, GPT\-6\-Astra computes the Hall–Petch increment the study reports but declines to infer the withheld bulk strength: “none of the proposed numerical comparisons is established”\. That contribution alone does not establish bulk strength, making evidence sufficiency relevant to interpreting the response\. On a carbon\-fibre/polyamide problem, all three proprietary models explain why a stronger interface need not give a stronger composite, yet only GPT\-6\-Sol selects the measured outcome\. On Cu\-doped spinel thermistors, the model instead selects an activation\-energy trend opposite to the measured one\.

## 6 Guiding Domain Adaptation with FrontierMat

#### Training\.

We adapt Qwen3\.5\-35B\-A3B \(3B of 35B parameters active\)\.*Continued pretraining*\(CPT\) uses 10\.94B tokens, with a reference mixture of 78\.5% materials literature \(2\.13M papers on composites and two\-dimensional materials\), 1\.5% synthesised text and 20% general replay\.*Supervised fine\-tuning*\(SFT\) uses 133K instruction samples \(23K materials\-specific\)\. No FrontierMat problem, option or answer entered training \([AppendixF](https://arxiv.org/html/2610.41162#A6)\)\. On top of this recipe, two components target the capability §[5\.2](https://arxiv.org/html/2610.41162#S5.SS2)found weakest: connecting evidence across scales\.

#### Meta\-weighted CPT\.

Natural sampling is dominated by well\-studied systems and single\-scale descriptions\.*Meta\-weighted CPT*keeps the next\-token objective but reweights materials data by their estimated utility for held\-out cross\-scale problems\([Ren et al\., 2018](https://arxiv.org/html/2610.41162#bib.bib24);[Xie et al\., 2023](https://arxiv.org/html/2610.41162#bib.bib25)\)\. Materials data are grouped into 99 buckets by material family, scale and source\. A stronger model reads a data block as context and answers 786 probe problems written from corpus papers; the block’s value is the gain in answer margin over a word\-shuffled copy of the same block, which controls for vocabulary and length\. Standardised bucket values set the sampling weights once before CPT, with the token budget unchanged \([AppendixF](https://arxiv.org/html/2610.41162#A6)\)\.

#### Scale\-modular self\-distillation\.

Evidence at different scales comes from different sources and may conflict, so joint training lets one scale override another\. After SFT,*scale\-modular self\-distillation*attaches three low\-rank residual modules\([Houlsby et al\., 2019](https://arxiv.org/html/2610.41162#bib.bib28)\), one each for micro, meso and macro, to the final hidden state, and a gated fusion module combines their outputs before the shared output head\. A frozen snapshot of the model reading the problem with the evidence of one scale teaches the corresponding module, and the snapshot reading all evidence teaches the fusion module\([Lopez\-Paz et al\., 2016](https://arxiv.org/html/2610.41162#bib.bib26);[Snell et al\., 2022](https://arxiv.org/html/2610.41162#bib.bib27)\)\. At inference the model sees only the problem, so each module must internalise the evidence it was taught with\.

#### Results\.

Adaptation raises Qwen3\.5\-35B\-A3B on FrontierMat from 36\.7 % \(official release\) to 55\.4 %, matching the strongest proprietary model and gaining at every scale and domain, from 13\.3 points on meso to 27\.2 points on micro problems, with 14\.7 points on cross\-scale problems \([Table2](https://arxiv.org/html/2610.41162#S3.T2),[AppendixD](https://arxiv.org/html/2610.41162#A4)\)\.

MaterialsGeneralModelMSQAMatSciB\.MaScQAKnow\.GPT\-4 \(CoT\)––62\.0⋆–Gemini\-2\.0\-Flash72\.9⋆–––Claude\-3\.7\-Sonnet70\.2–––DeepSeek\-R1–75\.2⋆––GPT\-5–75\.0––Qwen3\.5\-35B\-A3B65\.258\.494\.879\.0FrontierMat\-Model73\.872\.494\.979\.8Table 3:Public materials benchmarks and general ability \(%\)\. Upper rows are reported by the original papers under their own settings, with⋆for each paper’s best\. General Know\. averages six benchmarks \([Table12](https://arxiv.org/html/2610.41162#A6.T12)\)\.
#### Transfer and cost\.

Under matched settings, adaptation improves MSQA by 8\.6 points\([Cheung et al\., 2025](https://arxiv.org/html/2610.41162#bib.bib10)\)and MatSciBench by 14\.0 points\([Zhang and others, 2025](https://arxiv.org/html/2610.41162#bib.bib6)\)\([Table3](https://arxiv.org/html/2610.41162#S6.T3)\)\. Our scores exceed the original MSQA paper’s best and fall 2\.8 points below MatSciBench’s best, although those results use different settings\. MaScQA scores are nearly unchanged\. The six\-benchmark general mean rises from 79\.0 to 79\.8, despite a 2\.3\-point drop on one benchmark: aggregate performance holds, with some task\-level trade\-offs\.

Qwen3\.5\-0\.8BHeld\-outFrontierMatCPT \+ SFT \(natural mixture\)35\.918\.0\+ meta\-weighted CPT39\.4 \(\+3\.5†\)18\.3 \(\+0\.3\)\+ single\-module self\-distillation59\.020\.1\+ scale\-modular self\-distillation59\.4 \(\+23\.5†\)20\.5 \(\+2\.5†\)Table 4:Ablations on Qwen3\.5\-0\.8B\. Variants independently modify the natural\-mixture CPT \+ SFT baseline\. Held\-out problems \(513\) use exact match and FrontierMat its standard score\. Small figures are gains, and†marksp<0\.05p<0\.05\(paired bootstrap\)\.
#### Component ablations\.

On Qwen3\.5\-0\.8B with a 35B teacher \([Table4](https://arxiv.org/html/2610.41162#S6.T4),[SectionF\.4](https://arxiv.org/html/2610.41162#A6.SS4)\), meta\-weighted CPT improves the CPT \+ SFT baseline by 3\.5 points on held\-out problems and 0\.3 points on FrontierMat\. Scale\-modular self\-distillation adds 23\.5 and 2\.5 points, both significant, and scores above single\-module self\-distillation trained on the same distilled data\.

## 7 Conclusion

Materials R&D models must be evaluated at the research frontier\. On FrontierMat, the strongest general\-purpose model scores 53\.7 %, cross\-scale reasoning remains difficult, and models select plausible but unsupported outcomes\. Scale\-aware adaptation guided by FrontierMat brings an open\-weight model level with the strongest proprietary model while maintaining mean performance on general benchmarks\. We will extend FrontierMat to open\-ended answers and further material classes as the frontier moves\.

## Limitations

#### Size and coverage\.

FrontierMat contains 330 problems\. Differences of a few points between models and estimates for small groups, such as meso\-scale problems, are therefore subject to substantial uncertainty\. Application fields and functional domains are unevenly represented, with composites over\-represented among material systems\. It therefore offers an uneven sample of frontier materials R&D\. We report results by domain and scale to make this variation explicit\.

#### Reliance on source evidence\.

Answers are taken from source studies and may reflect errors in the original findings\. Some reported values are read from published plots, with the extraction resolution recorded for each problem\. Language models were used to cross\-check problems against their sources, which may favour problems that these models can readily parse\.

#### Answer format\.

Multiple\-choice options provide information that may be unavailable in practical R&D, and the number of correct options in multi\-answer problems is unevenly distributed\.

#### Adaptation study\.

The study in §[6](https://arxiv.org/html/2610.41162#S6)evaluates adaptation on one main model, with component ablations on a smaller model from the same family, and its literature corpus covers mainly composites and two\-dimensional materials\. The findings may not generalise to other architectures, data mixtures or material systems, and the general\-benchmark comparison in[Table12](https://arxiv.org/html/2610.41162#A6.T12)covers only six benchmarks and cannot establish performance across all capabilities required of a general assistant\.

## Ethical Considerations

#### Expert participation\.

The domain experts who nominated research directions and reviewed problems participated as research collaborators\. Group identities remain anonymous during review\.

#### Data and licensing\.

Problems paraphrase research settings and reproduce a small number of reported values\. Source evidence is recorded for every problem, and no figures or text are copied\. FrontierMat is intended for research evaluation\.

#### Intended use\.

FrontierMat measures performance on multiple\-choice problems derived from completed studies\. A high score does not certify a model for unsupervised use in materials R&D, where incorrect predictions can lead to costly experimental campaigns and compromise safety\-critical components\.

#### Use of AI assistants\.

Language models were used to cross\-check problems against their sources \(§[3\.1](https://arxiv.org/html/2610.41162#S3.SS1)\) and to assist with writing and editing text and code\. The authors reviewed all content and take responsibility for it\.

## References

- Alamparaet al\.\(2025\)N\. Alamparaet al\.Probing the limitations of multimodal language models for chemistry and materials research\.Nature Computational Science5,pp\. 952–961\.External Links:[Document](https://dx.doi.org/10.1038/s43588-025-00836-3)Cited by:[§2](https://arxiv.org/html/2610.41162#S2.SS0.SSS0.Px1.p1.1)\.
- Beanet al\.\(2025\)A\. M\. Bean, R\. O\. Kearns, A\. Romanou,et al\.Measuring what matters: construct validity in large language model benchmarks\.InAdvances in Neural Information Processing Systems \(Datasets and Benchmarks Track\),External Links:[Link](https://arxiv.org/abs/2511.04703)Cited by:[§2](https://arxiv.org/html/2610.41162#S2.SS0.SSS0.Px3.p1.1)\.
- Boikoet al\.\(2023\)D\. A\. Boiko, R\. MacKnight, B\. Kline, and G\. GomesAutonomous chemical research with large language models\.Nature624,pp\. 570–578\.External Links:[Document](https://dx.doi.org/10.1038/s41586-023-06792-0)Cited by:[§1](https://arxiv.org/html/2610.41162#S1.p1.1)\.
- Branet al\.\(2024\)A\. M\. Bran, S\. Cox, O\. Schilter, C\. Baldassari, A\. D\. White, and P\. SchwallerAugmenting large language models with chemistry tools\.Nature Machine Intelligence6,pp\. 525–535\.External Links:[Document](https://dx.doi.org/10.1038/s42256-024-00832-8)Cited by:[§1](https://arxiv.org/html/2610.41162#S1.p1.1)\.
- Cheunget al\.\(2025\)J\. J\. Cheung, S\. Shen, Y\. Zhuang, Y\. Li, R\. Ramprasad, and C\. ZhangMSQA: benchmarking LLMs on graduate\-level materials science reasoning and knowledge\.arXiv preprint arXiv:2505\.23982\.Cited by:[§1](https://arxiv.org/html/2610.41162#S1.p2.1),[§2](https://arxiv.org/html/2610.41162#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.41162#S2.SS0.SSS0.Px4.p1.1),[Table 1](https://arxiv.org/html/2610.41162#S2.T1.2.7.1.2.1.2.1),[§6](https://arxiv.org/html/2610.41162#S6.SS0.SSS0.Px5.p1.1)\.
- Dunnet al\.\(2020\)A\. Dunn, Q\. Wang, A\. Ganose, D\. Dopp, and A\. JainBenchmarking materials property prediction methods: the Matbench test set and Automatminer reference algorithm\.npj Computational Materials6,pp\. 138\.External Links:[Document](https://dx.doi.org/10.1038/s41524-020-00406-3)Cited by:[§2](https://arxiv.org/html/2610.41162#S2.SS0.SSS0.Px1.p1.1)\.
- Fenget al\.\(2024\)K\. Feng, X\. Shen, W\. Wang, X\. Zhuang, Y\. Tang, Q\. Zhang, and K\. DingSciKnowEval: a comprehensive dataset for evaluating scientific knowledge of large language models\.arXiv preprint arXiv:2406\.09098v4\.Cited by:[§1](https://arxiv.org/html/2610.41162#S1.p2.1),[§2](https://arxiv.org/html/2610.41162#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2610.41162#S2.T1.2.8.1.2.1.2.1)\.
- Gemaet al\.\(2025\)A\. P\. Gema, J\. O\. J\. Leang, G\. Hong, A\. Devoto, A\. C\. M\. Mancino, R\. Saxena, X\. He, Y\. Zhao, X\. Du, M\. R\. G\. Madani, C\. Barale, R\. McHardy, J\. Harris, J\. Kaddour, E\. van Krieken, and P\. MinerviniAre we done with MMLU?\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 5069–5096\.External Links:[Link](https://aclanthology.org/2025.naacl-long.262)Cited by:[§2](https://arxiv.org/html/2610.41162#S2.SS0.SSS0.Px3.p1.1)\.
- Guptaet al\.\(2022\)T\. Gupta, M\. Zaki, N\. M\. A\. Krishnan, and MausamMatSciBERT: a materials domain language model for text mining and information extraction\.npj Computational Materials8,pp\. 102\.External Links:[Document](https://dx.doi.org/10.1038/s41524-022-00784-w)Cited by:[§2](https://arxiv.org/html/2610.41162#S2.SS0.SSS0.Px4.p1.1)\.
- Gururanganet al\.\(2020\)S\. Gururangan, A\. Marasović, S\. Swayamdipta, K\. Lo, I\. Beltagy, D\. Downey, and N\. A\. SmithDon’t stop pretraining: adapt language models to domains and tasks\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 8342–8360\.Cited by:[§2](https://arxiv.org/html/2610.41162#S2.SS0.SSS0.Px4.p1.1)\.
- Heinemanet al\.\(2025\)D\. Heinemanet al\.Signal and noise: a framework for reducing uncertainty in language model evaluation\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/2508.13144)Cited by:[§2](https://arxiv.org/html/2610.41162#S2.SS0.SSS0.Px3.p1.1)\.
- Houlsbyet al\.\(2019\)N\. Houlsby, A\. Giurgiu, S\. Jastrzebski, B\. Morrone, Q\. de Laroussilhe, A\. Gesmundo, M\. Attariyan, and S\. GellyParameter\-efficient transfer learning for NLP\.InProceedings of the 36th International Conference on Machine Learning,pp\. 2790–2799\.Cited by:[§2](https://arxiv.org/html/2610.41162#S2.SS0.SSS0.Px4.p1.1),[§6](https://arxiv.org/html/2610.41162#S6.SS0.SSS0.Px3.p1.1)\.
- Liuet al\.\(2025\)S\. Liuet al\.MatTools: benchmarking large language models for materials science tools\.arXiv preprint arXiv:2505\.10852\.Cited by:[§2](https://arxiv.org/html/2610.41162#S2.SS0.SSS0.Px1.p1.1)\.
- Lopez\-Pazet al\.\(2016\)D\. Lopez\-Paz, L\. Bottou, B\. Schölkopf, and V\. VapnikUnifying distillation and privileged information\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2610.41162#S2.SS0.SSS0.Px4.p1.1),[§6](https://arxiv.org/html/2610.41162#S6.SS0.SSS0.Px3.p1.1)\.
- Luoet al\.\(2023\)Y\. Luo, Z\. Yang, F\. Meng, Y\. Li, J\. Zhou, and Y\. ZhangAn empirical study of catastrophic forgetting in large language models during continual fine\-tuning\.arXiv preprint arXiv:2308\.08747\.Cited by:[§2](https://arxiv.org/html/2610.41162#S2.SS0.SSS0.Px4.p1.1)\.
- M\-A\-P Team \(2025\)M\-A\-P TeamSuperGPQA: scaling LLM evaluation across 285 graduate disciplines\.arXiv preprint arXiv:2502\.14739\.Cited by:[§1](https://arxiv.org/html/2610.41162#S1.p2.1),[§2](https://arxiv.org/html/2610.41162#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2610.41162#S2.T1.2.11.1.2.1.2.1)\.
- Miret and Krishnan \(2024\)S\. Miret and N\. M\. A\. KrishnanAre LLMs ready for real\-world materials discovery?\.arXiv preprint arXiv:2402\.05200\.Cited by:[§1](https://arxiv.org/html/2610.41162#S1.p1.1)\.
- Mirzaet al\.\(2025\)A\. Mirza, N\. Alampara, S\. Kunchapu,et al\.A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists\.Nature Chemistry17,pp\. 1027–1034\.External Links:[Document](https://dx.doi.org/10.1038/s41557-025-01815-x)Cited by:[§1](https://arxiv.org/html/2610.41162#S1.p2.1),[§2](https://arxiv.org/html/2610.41162#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2610.41162#S2.T1.2.10.1.2.1.2.1)\.
- Reinet al\.\(2024\)D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. BowmanGPQA: a graduate\-level google\-proof Q&A benchmark\.InFirst Conference on Language Modeling,External Links:[Link](https://arxiv.org/abs/2311.12022)Cited by:[§2](https://arxiv.org/html/2610.41162#S2.SS0.SSS0.Px1.p1.1)\.
- Renet al\.\(2018\)M\. Ren, W\. Zeng, B\. Yang, and R\. UrtasunLearning to reweight examples for robust deep learning\.InProceedings of the 35th International Conference on Machine Learning,pp\. 4334–4343\.Cited by:[§2](https://arxiv.org/html/2610.41162#S2.SS0.SSS0.Px4.p1.1),[§6](https://arxiv.org/html/2610.41162#S6.SS0.SSS0.Px2.p1.1)\.
- Rubungoet al\.\(2024\)A\. N\. Rubungoet al\.LLM4Mat\-Bench: benchmarking large language models for materials property prediction\.arXiv preprint arXiv:2411\.00177\.Cited by:[§2](https://arxiv.org/html/2610.41162#S2.SS0.SSS0.Px1.p1.1)\.
- Snellet al\.\(2022\)C\. Snell, D\. Klein, and R\. ZhongLearning by distilling context\.arXiv preprint arXiv:2209\.15189\.Cited by:[§2](https://arxiv.org/html/2610.41162#S2.SS0.SSS0.Px4.p1.1),[§6](https://arxiv.org/html/2610.41162#S6.SS0.SSS0.Px3.p1.1)\.
- Songet al\.\(2023\)Y\. Song, S\. Miret, H\. Zhang, and B\. LiuHoneyBee: progressive instruction finetuning of large language models for materials science\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 5724–5739\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.380)Cited by:[§2](https://arxiv.org/html/2610.41162#S2.SS0.SSS0.Px4.p1.1)\.
- Wanget al\.\(2024\)Y\. Wang, X\. Ma, G\. Zhang,et al\.MMLU\-Pro: a more robust and challenging multi\-task language understanding benchmark\.InAdvances in Neural Information Processing Systems \(Datasets and Benchmarks Track\),External Links:[Link](https://arxiv.org/abs/2406.01574)Cited by:[§2](https://arxiv.org/html/2610.41162#S2.SS0.SSS0.Px1.p1.1)\.
- Xieet al\.\(2023\)S\. M\. Xie, H\. Pham, X\. Dong, N\. Du, H\. Liu, Y\. Lu, P\. Liang, Q\. V\. Le, T\. Ma, and A\. W\. YuDoReMi: optimizing data mixtures speeds up language model pretraining\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§2](https://arxiv.org/html/2610.41162#S2.SS0.SSS0.Px4.p1.1),[§6](https://arxiv.org/html/2610.41162#S6.SS0.SSS0.Px2.p1.1)\.
- Yoshitakeet al\.\(2024\)M\. Yoshitakeet al\.MaterialBENCH: evaluating college\-level materials science problem\-solving abilities of large language models\.arXiv preprint arXiv:2409\.03161\.Cited by:[§1](https://arxiv.org/html/2610.41162#S1.p2.1),[§2](https://arxiv.org/html/2610.41162#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2610.41162#S2.T1.2.4.1.2.1.2.1)\.
- Zakiet al\.\(2024\)M\. Zaki, Jayadeva, Mausam, and N\. M\. A\. KrishnanMaScQA: investigating materials science knowledge of large language models\.Digital Discovery3\(2\),pp\. 313–327\.External Links:[Document](https://dx.doi.org/10.1039/D3DD00188A)Cited by:[§1](https://arxiv.org/html/2610.41162#S1.p2.1),[§2](https://arxiv.org/html/2610.41162#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2610.41162#S2.T1.2.3.1.2.1.2.1)\.
- Zhanget al\.\(2025\)J\. Zhanget al\.MatSciBench: benchmarking the reasoning ability of large language models in materials science\.arXiv preprint arXiv:2510\.12171v2\.Cited by:[§1](https://arxiv.org/html/2610.41162#S1.p2.1),[§2](https://arxiv.org/html/2610.41162#S2.SS0.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2610.41162#S2.T1.2.5.1.2.1.2.1),[§6](https://arxiv.org/html/2610.41162#S6.SS0.SSS0.Px5.p1.1)\.

## Appendix AProblem Format and Labelling Rules

Items330Single\-answer / multi\-answer98 / 232SourcesPartner R&D groups \(academic / industry\)13 \(9 / 4\)Scale of the governing mechanismMicro / meso / macro88 / 24 / 88Cross\-scale130Table 5:Statistics of FrontierMat\. All items are multiple choice\. Scale is the level of the structure and mechanism that governs the answer \(§[3\.2](https://arxiv.org/html/2610.41162#S3.SS2)\)\.[Table5](https://arxiv.org/html/2610.41162#A1.T5)summarises FrontierMat\. Each problem contains the fields listed in[Table6](https://arxiv.org/html/2610.41162#A1.T6)\. Fields marked†\\daggerare hidden from models during evaluation\.

FieldContentidStable identifierpromptStudy setting, related results, options, answer formatanswer\_formatSingle\-answer or multi\-answergold†Letter, or alphabetically ordered letter setdomainFunctional domain \(§[3\.2](https://arxiv.org/html/2610.41162#S3.SS2)\)material\_systemClass of material \(§[3\.2](https://arxiv.org/html/2610.41162#S3.SS2)\)fieldApplication fieldscaleMicro, meso, macro or cross\-scalescale\_chain†For cross\-scale problems, the chain of mechanisms across scalesreference†Reported result: values with units, or the categorical outcomeevidence†Figure, table and page of the result in the sourcesource\_doi†DOI or URL, where availablesource\_yearPublication year, where availableTable 6:Schema of a problem record\.#### Answer annotation\.

- •The answer is a result reported by the source study: a measurement, a quantity computed directly from measured data, or a conclusion supported by the study’s data\. Simulation outputs may provide context and are labelled as such, but cannot serve as answers to problems about experimental behaviour\.
- •Values read from plots are marked as such, together with the reading resolution \(for example, about 10 MPa for the problem in[Figure2](https://arxiv.org/html/2610.41162#S1.F2)\); numerical options differ by more than this resolution\.
- •If a source’s summary statistic differs from its underlying values, the answer is recomputed from the values given in the prompt, and this decision is documented\.
- •Quantities use SI units or, where conventional in the field, the units used by the source\. Units are stated in the prompt\.

#### Scale annotation\.

The scale label identifies the structural or mechanistic level that determines the answer, rather than the specimen size\. A problem is*micro*if the answer depends on atoms, lattices, defects or local chemistry;*meso*if it depends on grains, phases, fibres or particles, pores or their arrangement;*macro*if it depends on the response of a specimen, component or device treated as a continuum\. A problem is*cross\-scale*if answering it requires a mechanism linking information in the prompt at one level to an outcome at another\. This chain is recorded as*structure at one scale→\\rightarrowmechanism→\\rightarrowproperty at another*, for example*clamping and temperature→\\rightarrowhole\-edge damage accumulation→\\rightarrowjoint fatigue life*\. When two labels are plausible, annotators select the level of the mechanism that distinguishes the correct option from the strongest distractor\.

#### Functional\-domain annotation\.

The functional\-domain label identifies the property addressed by the problem\. A problem is labelled*multifunctional trade\-off*only if choosing the correct option requires weighing two or more properties against each other, not merely because several properties are mentioned\.

## Appendix BConstruction and Expert Review

#### Literature screening\.

We retrieved 25,182 journal articles published in 2022–2026\. We first screened titles and abstracts for experimental materials studies in the four application fields, then reviewed the full texts of the 4,000 remaining articles\. Of these, we retained 2,186 as sources because they reported a specific experimental result with sufficient context to formulate a problem\.

#### Research collaboration\.

13 R&D groups participated \(9 academic and 4 industrial or national\-laboratory\)\. Their identities are withheld for anonymous review\. They contributed problems grounded in their latest research, including work not yet published, and identified research directions for literature\-based problem construction\. Their work spans structural composites, ceramic components, polymer and coating systems, semiconductor defects, low\-dimensional materials and electrocatalysts\. Across sources, eligibility depends on experimental evidence, sufficient context and relevance to current R&D\.

#### Problem writing\.

Annotators followed the same guidelines across sources: state the study setting and selected evidence, ask for a measured outcome or a conclusion derived from that evidence, write plausible distractors, and record the basis for the answer\. When the target is a withheld measurement, information that directly reveals it is removed\. When the target requires quantitative interpretation, the necessary measurements are retained\. Two language models checked each problem against its source \(§[3\.1](https://arxiv.org/html/2610.41162#S3.SS1)\)\.

#### Expert review\.

For each problem, experts received the answer, the reported result and its location in the source, and an explanation of each option\. They marked each problem Approve, Revise or Reject and provided free\-text comments\.

#### Source publication years\.

Published sources span 1998–2026\. Earlier studies are included for their relevance to problems identified by the participating researchers, rather than as evidence of recency\.

## Appendix CEvaluation Protocol

#### Prompting\.

Models receive each problem in the format shown in[AppendixG](https://arxiv.org/html/2610.41162#A7), in English, with no few\-shot examples and no tools\. Single\-answer prompts end with “End with exactly: Answer: <letter\>\.”; multi\-answer prompts ask for all correct letters in alphabetical order\.

#### Answer extraction and scoring\.

We parse the final “Answer:” line in each response, remove duplicate letters and sort the remaining letters\. The parsed set receives 1 point if it matches the gold set, 0\.5 points if it is a non\-empty proper subset of the gold set, and 0 otherwise, including responses with an incorrect option and empty or unparseable answers\.

#### Configurations\.

[Table7](https://arxiv.org/html/2610.41162#A3.T7)lists the reasoning and decoding settings of each model\.

ModelSettingGPT\-6\-Astrareasoning effort xhigh; median 71 s, 2\.2k output tokensGPT\-6\-Solreasoning effort xhigh; median 39 s, 1\.4k tokensClaude Opus 5\.5extended thinking \(max\); median 121 s, 5\.0k tokensDeepSeek V4 Proreasoning modeGLM\-5\.3reasoning modeQwen3\.5\-35B\-A3Bofficial post\-trained release; same prompts and decoding as FrontierMat\-Model; 16 samplesFrontierMat\-Modeldomain\-adapted Qwen3\.5\-35B\-A3B \(§[6](https://arxiv.org/html/2610.41162#S6)\); thinking mode,T=0\.6T=0\.6, top\-pp0\.95, at most 32k output tokens; 16 samplesTable 7:Model configurations\.

## Appendix DFull Results

[Table8](https://arxiv.org/html/2610.41162#A4.T8)reports each model’s score by functional domain\.

SubsetGPT\-6AstraGPT\-6SolClaudeOpus 5\.5DeepSeekV4 ProGLM\-5\.3Qwen3\.5\(official\)FrontierMat\-ModelAll53\.751\.447\.347\.645\.036\.755\.4Mechanical53\.948\.146\.648\.546\.236\.456\.6Thermal50\.750\.048\.051\.047\.034\.449\.8Multifunctional trade\-off44\.644\.136\.432\.233\.932\.248\.9Protective61\.159\.353\.751\.940\.732\.560\.6Electromagnetic65\.260\.952\.258\.754\.342\.362\.2Electrical58\.564\.157\.753\.852\.646\.961\.0

Table 8:Scores \(%\) by functional domain\.
## Appendix EAnalysis Details

### E\.1 Scale analysis \(§[5\.2](https://arxiv.org/html/2610.41162#S5.SS2)\)

All models are evaluated on the full set of 330 problems\. For the controlled scale comparison, however, all problems contributed by partner groups are cross\-scale, so that channel provides no within\-channel single\-scale comparison\. We therefore use the 265 problems from the channel containing both scale types\. The 202 multi\-answer problems within this set form the comparison in[Figure4](https://arxiv.org/html/2610.41162#S5.F4)b\. To estimate the overall scale gap, we use both answer formats within the 265\-problem set: we stratify by answer format \(single\-answer or multi\-answer\) and compute the difference in mean score between cross\-scale and single\-scale problems for each model and format\. The reported gap averages these differences over formats and models, weighted by the number of cross\-scale problems in each format\. We permute scale labels within each format 5,000 times\. The one\-sidedpp\-value is the proportion of permutations yielding a gap at least as negative as the observed gap\.

### E\.2 Per\-problem analyses \(§[4\.3](https://arxiv.org/html/2610.41162#S4.SS3), §[5](https://arxiv.org/html/2610.41162#S5)\)

#### Answers\.

Every analysis in §[4\.3](https://arxiv.org/html/2610.41162#S4.SS3)and §[5](https://arxiv.org/html/2610.41162#S5)uses one answer per model and problem: the majority answer across samples where a model was sampled more than once, and the single answer otherwise\. Scores computed from these answers therefore differ slightly from the means in[Table2](https://arxiv.org/html/2610.41162#S3.T2)\. The official Qwen3\.5\-35B\-A3B is excluded from these analyses\.

#### Response categories\.

On a multi\-answer problem with answer setGG, a responsePPis*exact*ifP=GP=G,*omits*ifP⊊GP\\subsetneq G,*adds*ifP⊋GP\\supsetneq G, and*other*otherwise \(it both adds and omits, or is disjoint fromGG\)\. An error contains an unsupported option ifP⊈GP\\not\\subseteq G\.

#### Reliability\.

For bootstrap rankings, we resample the 330 problems with replacement 10,000 times and rank the five general\-purpose models by total score\. For split\-half agreement, we draw 2,000 random partitions into halves of 165 problems and report the mean Kendallτ\\taubetween their rankings\. Discrimination is the Pearson correlation, across the six models with per\-problem answers, between a model’s score on a problem and its total score on the other problems; it is undefined, and omitted, for the 153 problems on which all six models receive the same score\.

## Appendix FDomain\-Adaptation Details

### F\.1 CPT Corpus

#### Sources\.

Materials text comes from two internal literature collections: composites \(about 2\.03M records, 1\.77M distinct papers after deduplication\) and two\-dimensional materials \(1\.40M records, 0\.36M distinct papers\)\. We add structured extractions \(English and Chinese\) and one\-sentence summaries generated by Qwen3\-32B \(0\.13B tokens\), each prefixed with its source paper title and not manually checked, plus replay data from 23 public corpora\.

#### Processing\.

For each paper, we use the best available text, prioritising publisher XML \(Elsevier, JATS\), then PDF, then abstract\. Re\-parsing previously unparsed PDFs with PyMuPDF recovered about 400K papers\. Cleaning removes reference lists, repeated headers and footers, page\-number and copyright lines, line\-break hyphenation and residual XML markup\. Gopher\-style heuristics on length, alphabetic ratio, duplicate lines and collapsed spacing remove low\-quality documents\. Exact deduplication uses the DOI or the normalised title and year, followed by cross\-collection MinHash\-LSH \(128 permutations, Jaccard threshold 0\.8\)\. A fastText classifier \(validation accuracy 0\.936\) labels documents as core \(p≥0\.8p\\geq 0\.8\), adjacent \(0\.3≤p<0\.80\.3\\leq p<0\.8\) or off\-domain \(p<0\.3p<0\.3\)\. We exclude documents outside English and Chinese, retracted papers and papers in clinical medicine, and hold out papers published in 2026 or later plus a random 1%\. The resulting corpus has 1\.58M trainable documents and 10\.60B tokens \(Qwen3\.5 tokenizer; 9\.15B core, 0\.25B adjacent, 1\.21B off\-domain\), of which 8\.34B come from the composites collection and 2\.26B from the two\-dimensional\-materials collection\. Documents are concatenated and cut into 4,096\-token blocks; long documents are split, not truncated\.

#### Buckets and packing\.

Core and adjacent literature is grouped by document type \(research papers versus reviews, books and chapters\) and by quality tier \(“hi” if the paper’s citation percentile within its publication year is at least 0\.5, otherwise “mid”\)\. Documents are ordered by a salted hash, joined with end\-of\-sequence separators and cut continuously into 4,096\-token blocks, so no text is truncated; block counts define natural sampling proportions\.[Table9](https://arxiv.org/html/2610.41162#A6.T9)reports pool sizes and the reference corpus mixture before materials data are reweighted\.

BucketPool \(B tok\.\)Reference share \(%\)Papers, hi4\.1140\.0Papers, mid5\.7431\.1Reviews, hi0\.525\.0Reviews, mid0\.242\.4Synthetic \(extractions, summaries\)0\.131\.5English web / encyclopaedia–6\.0 / 1\.4Chinese web / encyclopaedia–3\.0 / 0\.6Source code / code solutions–3\.0 / 1\.0Math web / math solutions–2\.75 / 1\.25STEM textbooks–1\.0

Table 9:CPT data pools and reference mixture for the 10\.94B\-token budget\. Materials buckets are reweighted once before CPT using the in\-context procedure below\. These proportions describe the corpus mixture, not the learned weights within materials data\. Papers are*hi*if their citation count is at or above the median for their publication year and*mid*otherwise \(including papers without citation data\); reviews, books and chapters form the review buckets\. Replay \(bottom, 20% in total\) draws on FineWeb\-Edu, Ultra\-FineWeb, FineWiki, Wikipedia, FineWeb2\-HQ, Stack\-Edu, StarCoderData, OpenCodeInstruct, CodeContests, OpenCodeReasoning, FineMath, InfiWebMath, OpenMathReasoning, NuminaMath\-1\.5, LibreTexts, Cosmopedia and Pressbooks\.
#### Replay share\.

In small\-model ablations, removing replay caused general capabilities, especially Chinese, mathematics and code, to deteriorate throughout training\. A 10% share was insufficient, whereas 15–25% had little effect on held\-out materials loss; we retain a 20% replay share\.

### F\.2 SFT Data

The final assistant turn in each sample contains either<think\>reasoning</think\>followed by the answer, or the answer with an empty reasoning block; earlier assistant turns retain only the answer\. The dataset contains 133,101 training and 690 validation samples \(304M tokens, 202M of them answer tokens\); 39% use reasoning and 39% are in Chinese\.[Table10](https://arxiv.org/html/2610.41162#A6.T10)lists the composition\.

PartSamplesContentGeneral capability99\.9Kmathematics, code and science reasoning \(OpenThoughts3, LIMO, s1K, Codeforces\-CoTs, Nemotron\-Science\); Chinese reasoning distillation; instruction following \(Nemotron\-IF, SmolTalk2\); dialogue \(Firefly, SmolTalk2\); structured outputMaterials tasks11\.0Kextraction from papers \(EN 3\.6K, ZH 3\.5K\), summarisation 3\.8KGeneral supplement9\.9Kconstrained instructions, rewriting, summarisation, Python, long contextMaterials questions9\.0Ksingle\-choice 3\.2K, multi\-select 4\.8K, calculation 1\.0KFormat\-aligned items3\.4Krecomposed, true/false, numeric\-variant and newly written items from an internal set disjoint from all evaluation setsTable 10:SFT data composition\.#### Materials questions\.

Qwen3\-235B\-A22B\-Thinking generates single\-choice, multi\-select and calculation problems from corpus papers, with a true/false judgment and supporting evidence for every option\. The same model then answers each problem without the paper\. If its answer differs from the gold answer, it reviews the problem with the paper; problems whose gold answer remains unconfirmed are discarded\. Reasoning samples are obtained by rejection sampling, retaining a closed\-book chain of thought only if its final answer exactly matches the gold answer; non\-reasoning samples are expert\-style explanations that assess each option and end with the answer letters\.

### F\.3 Scale\-Aware Training

#### Meta\-weighted CPT\.

The objective is to favour materials data that improve performance on held\-out cross\-scale problems\. A direct implementation would evaluate candidate bucket weights through virtual parameter updates and differentiate the cross\-scale loss through those updates\. We instead estimate data utility offline using a stronger model, then hold the resulting sampling weights fixed throughout CPT\.

#### In\-context approximation\.

We use in\-context learning as a proxy for the benefit of training on a data block, scoring buckets once before CPT\. A stronger model reads a data blockddas reference material and answers each probe problemjj; its marginΔj\\Delta\_\{j\}is the log\-probability of the correct answer minus that of the strongest distractor\. The value ofddis

v⁡\(d\)=1J​∑j\[Δj​\(d\)−Δj​\(d~\)\],v\(d\)=\\tfrac\{1\}\{J\}\\textstyle\\sum\_\{j\}\\big\[\\Delta\_\{j\}\(d\)\-\\Delta\_\{j\}\(\\tilde\{d\}\)\\big\],whered~\\tilde\{d\}isddwith its words shuffled, providing a control with the same words and length but disrupted structure\. The 786 probe problems are written from corpus papers with a domain mix matching FrontierMat’s and contain no FrontierMat problem\. Materials data are grouped into 99 buckets by material family, scale and source;zbz\_\{b\}is the standardised mean value of bucketbb, and materials data are resampled withpb∝pbnat​exp⁡\(λ​zb\)p\_\{b\}\\propto p\_\{b\}^\{\\text\{nat\}\}\\exp\(\\lambda z\_\{b\}\)\(λ=3\\lambda=3\), leaving the other data and the token budget unchanged\.

#### Scale\-modular self\-distillation\.

Lethth\_\{t\}be the final hidden state for the problemxxand answer prefixy<ty\_\{<t\}\. Each scale moduleAkA\_\{k\},k∈\{μ,m,M\}k\\in\\\{\\mu,m,M\\\}, is a low\-rank residual bottleneck,zk=ht\+Ak​\(ht\)z\_\{k\}=h\_\{t\}\+A\_\{k\}\(h\_\{t\}\); the fusion module computes gates from\[ht;zμ;zm;zM\]\[h\_\{t\};z\_\{\\mu\};z\_\{m\};z\_\{M\}\]and returnsh~t=ht\+∑kgk​\(zk−ht\)\\tilde\{h\}\_\{t\}=h\_\{t\}\+\\sum\_\{k\}g\_\{k\}\\,\(z\_\{k\}\-h\_\{t\}\), which the shared output head maps to the next\-token distribution\. Letpθ¯p\_\{\\bar\{\\theta\}\}be a frozen snapshot of the model andEkE\_\{k\}the evidence for scalekk\. Define the teacher distributions

qk,t\\displaystyle q\_\{k,t\}=pθ¯\(⋅∣x,Ek,y<t\),\\displaystyle=p\_\{\\bar\{\\theta\}\}\(\\cdot\\mid x,E\_\{k\},y\_\{<t\}\),qall,t\\displaystyle q\_\{\\mathrm\{all\},t\}=pθ¯\(⋅∣x,Eμ,Em,EM,y<t\)\.\\displaystyle=p\_\{\\bar\{\\theta\}\}\(\\cdot\\mid x,E\_\{\\mu\},E\_\{m\},E\_\{M\},y\_\{<t\}\)\.The distillation loss at tokenttis

ℒdist,t\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{dist\},t\}=∑kKL\(qk,t∥p\(⋅∣zk\)\)\\displaystyle=\\sum\_\{k\}\\mathrm\{KL\}\\big\(q\_\{k,t\}\\,\\\|\\,p\(\\cdot\\mid z\_\{k\}\)\\big\)\+KL\(qall,t∥p\(⋅∣h~t\)\)\.\\displaystyle\+\\mathrm\{KL\}\\big\(q\_\{\\mathrm\{all\},t\}\\,\\\|\\,p\(\\cdot\\mid\\tilde\{h\}\_\{t\}\)\\big\)\.Training minimises this loss together with cross\-entropy on the reference answer\. Micro evidence describes defects, dislocations and interaction mechanisms; meso evidence describes grains, precipitates, fibre architecture, porosity and phase distribution; macro evidence summarises property curves and test and service conditions\.

### F\.4 Component Ablations at 0\.8B

#### Setup\.

For the CPT comparison in[Table4](https://arxiv.org/html/2610.41162#S6.T4), we train Qwen3\.5\-0\.8B with the same 1\.2B\-token budget and SFT procedure, using either natural sampling proportions or the in\-context bucket weights above\. Distillation variants start from the natural\-mixture CPT \+ SFT model\.*Scale\-modular self\-distillation*is implemented with low\-rank adapters on the frozen SFT model\. Because a 0\.8B model reads long evidence unreliably, the evidence\-reading teacher in this ablation is a 35B model rather than a snapshot of the student; it answers 1,044 training problems with evidence of one scale, or with all scales plus the cross\-scale causal statements reported by the authors, extracted from about 1,600 source papers, and only correct responses, including their reasoning, are kept as targets\. A shared adapter trained on all distilled data is merged first; three scale adapters \(rank 16\) are then trained on the data of their scale, and a per\-layer gate over them plus a fusion adapter on the all\-evidence data\. The*single\-module self\-distillation*baseline trains a single rank\-64 adapter on exactly the same distilled data\.

#### Evaluation\.

Reasoning is enabled with temperature 0\.6\. Held\-out problems are generated as in[AppendixF](https://arxiv.org/html/2610.41162#A6)from papers excluded from all training, sampled four times with up to 12k output tokens and scored by exact match\. FrontierMat is sampled eight times with up to 32k output tokens, over three seeds for the CPT comparison and two for the distillation comparison, and scored with its standard rule \(§[4\.1](https://arxiv.org/html/2610.41162#S4.SS1)\)\. Differences use a paired bootstrap over problems\. The two evaluations differ in metric and output budget, so the magnitudes of their gains are not directly comparable\.[Table11](https://arxiv.org/html/2610.41162#A6.T11)lists all differences; meta\-weighted CPT also lowers the held\-out CPT materials loss from 1\.8549 to 1\.8529\.

ComparisonHeld\-outFrontierMatMeta\-weighted vs\. natural CPT\+3\.5 \[1\.1, 5\.9\]\+0\.3 \[−\-0\.6, 1\.1\]Scale\-modular vs\. CPT \+ SFT\+23\.5 \[20\.1, 26\.9\]\+2\.5 \[0\.5, 4\.6\]Scale\-modular vs\. single\-module\+0\.4 \[−\-2\.0, 2\.9\]\+0\.4 \[−\-1\.2, 2\.0\]Table 11:Differences \(points\) in the 0\.8B ablations, with 95% paired\-bootstrap intervals \(held\-out: exact match; FrontierMat: score\)\.

### F\.5 Transfer and General\-Benchmark Evaluation

#### Truncation\.

7\.6 % of the official model’s responses on multi\-answer problems reach the 32k\-token limit; scoring all of them as correct would add at most 7\.6 points, compared with a gap of 22\.2 points on those problems \(16\.1 % versus 38\.3 %\)\.

Both models in[Table3](https://arxiv.org/html/2610.41162#S6.T3)and[Table12](https://arxiv.org/html/2610.41162#A6.T12)are evaluated with the same harness, prompts and decoding settings, with reasoning mode enabled throughout\. Materials benchmarks are scored with each benchmark’s released protocol: the 1,757 binary questions of MSQA, the 1,025 text\-only questions of MatSciBench and all 650 questions of MaScQA, the subsets for which the original papers report the results quoted in[Table3](https://arxiv.org/html/2610.41162#S6.T3)\. General benchmarks are run with lm\-eval\-harness on vLLM\.

Qwen3\.5\-35B\-A3BFrontierMat\-Model*Public materials benchmarks*MSQA \(binary\)65\.273\.8MatSciBench \(text\)58\.472\.4MaScQA94\.894\.9*General benchmarks*MMLU84\.882\.5C\-Eval84\.685\.5GSM8K84\.685\.9ARC\-Challenge62\.566\.9WinoGrande74\.674\.7PIQA83\.083\.1Average \(6\)79\.079\.8Table 12:Scores \(%\) of the official post\-trained Qwen3\.5\-35B\-A3B and FrontierMat\-Model on public materials benchmarks and on six general benchmarks, under identical evaluation settings\.

### F\.6 Hyperparameters

CPTData10\.94B tokens \([Table9](https://arxiv.org/html/2610.41162#A6.T9)\)Learning rate10−510^\{\-5\}, cosine to10−610^\{\-6\}Batch, length256×\\times4,096Steps10,433Table 13:Hyperparameters for CPT\.

## Appendix GExample Problems

The following examples reproduce the problems shown to models, followed by their hidden annotations\.

Performance forecasting \(CWB7\-001, single\)Single\-choice: select the ONE option closest to the held\-out experimental outcome\. Numerical options are approximate outcome hypotheses, not additional measurements\.Forecast axial compressive strength for an untested architecture–angle combination of RTM T700 carbon/TDE\-86 epoxy\. Four\-directional braiding is the basic architecture\. Five\-directional adds longitudinal straight yarns; six\-directional additionally adds transverse yarns; seven\-directional additionally adds through\-thickness yarns\. Coupons are untrimmed; compression uses ASTM D6641 at 1 mm/min\. The following five\-specimen means are rounded graph readings\. Each row is \[directions, angle\_deg, fiber\_volume\_percent, strength\_MPa\]: \[4,20,57\.7,447\]; \[5,20,60\.4,507\]; \[6,20,56\.9,496\]; \[7,20,50\.3,313\]; \[4,40,55\.2,220\]; \[5,40,51\.0,290\]; \[7,40,56\.0,226\]\. Target: six\-directional, 40 degrees, fiber volume 55\.5%, thickness 3\.36 mm\. Estimate the held\-out measured mean; no deterministic constitutive law is stipulated\.A\. About 330 MPa; 6d is strongest at 40 degrees\. B\. About 250 MPa; 5d is strongest at 40 degrees\. C\. About 190 MPa; 5d is strongest at 40 degrees\. D\. About 420 MPa; 6d is strongest at 40 degrees\.Give brief reasoning\. End with exactly: Answer: <letter\>\.Gold:AHeadline model:BMeasured reference:\{"strength\_MPa": 328, "strongest\_at\_40": "6d"\} Evidence locator:Figure 9\(b\), Table 1Source:2022, en, expert\-authored

Data interpretation \(CWB7\-004, multi\)Multiple\-select: select ALL statements consistent with the stated experiment and its evidence\. More than one option is correct\.A matched T300/5405 laminate series underwent one common wet/hot aging program\. Unaged→\\rightarrowaged measurements are: plain tension: failure load 40\.39→\\rightarrow37\.30 kN, modulus 54\.10→\\rightarrow53\.40 GPa; open\-hole tension: 33\.50→\\rightarrow28\.94 kN, 56\.15→\\rightarrow54\.02 GPa; undamaged compression: 78\.05→\\rightarrow72\.93 kN, 55\.00→\\rightarrow53\.13 GPa; 15 J impact followed by compression: 59\.20→\\rightarrow42\.91 kN, 47\.96→\\rightarrow46\.20 GPa\. Within each pair geometry is unchanged\.A\. The impacted\-compression load retention is about 72\.5%\. B\. Impacted compression has the largest relative failure\-load loss among the four configurations\. C\. Open\-hole tension retains less failure load, proportionally, than impacted compression\. D\. The impacted\-compression modulus loss is about 27\.5%\.Give brief reasoning\. End with exactly: Answer: <letters in alphabetical order\>\.Gold:ABHeadline model:ABMeasured reference:\{"cai\_load\_retention\_pct": 72\.483, "most\_sensitive": "impacted\_compression", "stiffness\_proxy\_valid": "no"\} Evidence locator:Tables 1–2, scanned pp437–438Source:1998, zh, expert\-authored

Evidence judgment \(CWB7\-007, multi\)Multiple\-select: select ALL statements consistent with the stated experiment and its evidence\. More than one option is correct\.Through\-thickness Darcy tests use 17 plies of SW220C100b glass satin and 100 cSt silicone oil\. Fiber volume fractions span 46–54%; injection pressures span \.05–\.30 MPa\. A support plate has 58% open area; adding a 40\-mesh screen introduces an 82%\-open spreading layer\. The measured fabric imprint shows that the effective flowing area shrinks as pressure increases\. K is inferred from Q\*viscosity\*thickness/\(area\*pressure\_drop\)\. Predict the observed joint outcome using categorical fields: with\_screen\_K\_vs\_pressure and without\_screen\_K\_vs\_pressure at 46% fiber volume \(each approximately\_constant/increases/decreases\), K\_vs\_fiber\_fraction \(increases/decreases\), and screen\_effect\_on\_scatter \(reduces/increases\)\. Predictions refer to thickness\- and effective\-area\-corrected permeability, not uncorrected flow rate\.A\. Adding the screen increases between\-test scatter\. B\. With the screen, corrected permeability remains approximately constant as pressure increases at 46% fiber volume\. C\. Without the screen, corrected permeability increases as pressure increases at 46% fiber volume\. D\. Higher fiber volume fraction lowers corrected permeability in both fixture configurations\.Give brief reasoning\. End with exactly: Answer: <letters in alphabetical order\>\.Gold:BDHeadline model:BCDMeasured reference:\{"with\_screen\_K\_vs\_pressure": "approximately\_constant", "without\_screen\_K\_vs\_pressure": "decreases", "K\_vs\_fiber\_fraction": "decreases", "screen\_effect\_on\_scatter": "reduces"\} Evidence locator:Figures 8–12 and conclusions, PDF pp7–8Source:2020, zh, expert\-authored

## Appendix HCase\-Study Problems

We reproduce the three problems discussed in §[5\.3](https://arxiv.org/html/2610.41162#S5.SS3), as shown to models, with answers and supporting evidence\. Each problem records its source study\.

Case 1 \(FMB\-F125\): cross\-scale, micro→\\rightarrowmacro; metal\-matrix composite, mechanicalCeO2/GO\-Cu30Cr0\.2Zr electrical contact composites are fabricated by hydrothermal synthesis and spark plasma sintering at 950∘C under 50 MPa\. Electron backscatter diffraction gives average grain sizes of 1\.55 µm for Cu30Cr0\.2Zr and 0\.86 µm for 0\.5CeO2/GO\-Cu30Cr0\.2Zr\. The Hall–Petch coefficient for Cu is approximately 0\.071 MPa m1/2\. After tensile deformation, Williamson–Hall analysis yields dislocation densities of4\.14×10144\.14\\times 10^\{14\}m\-2for Cu30Cr0\.2Zr,8\.04×10148\.04\\times 10^\{14\}m\-2for 0\.3CeO2/GO\-Cu30Cr0\.2Zr, and9\.15×10149\.15\\times 10^\{14\}m\-2for 0\.5CeO2/GO\-Cu30Cr0\.2Zr\. The Burgers vector for Cu is 0\.256 nm and the shear modulus is 42\.1 GPa\. Uniaxial tensile testing at a constant deformation rate of 1 mm/min gives a tensile strength of 276 MPa for Cu30Cr0\.2Zr and 367 MPa for 0\.3CeO2/GO\-Cu30Cr0\.2Zr\. The tensile strength of 0\.5CeO2/GO\-Cu30Cr0\.2Zr is withheld from the data above\.Which of the following statements hold?A\. The tensile strength of the 0\.5 wt% composite is within 25 MPa of the 367 MPa measured for the 0\.3 wt% composite\. B\. The tensile strength of 0\.5CeO2/GO\-Cu30Cr0\.2Zr exceeds 390 MPa\. C\. The tensile strength increment from 0\.3 to 0\.5 wt% CeO2/GO is larger than the increment from 0 to 0\.3 wt% CeO2/GO\. D\. CeO2nanoparticles riveted on the graphene surface resist agglomeration caused by van der Waals forces, improving the dispersion of graphene in the Cu matrix\. E\. Load redistribution from the graphene reinforcement is identified as the most important strengthening mechanism in CeO2/GO\-Cu30Cr0\.2Zr composites\. F\. The conductivity improvement in CeO2/GO\-Cu30Cr0\.2Zr is partly attributed to the oxidation of graphene oxide during sintering, which partially recovers the intrinsic electrical conductivity of graphene\.Answer: B, D, E\.The withheld strength is 406 MPa, so B is correct and A and C are incorrect \(the increment falls from 91 to 39 MPa\)\. The study computes the Hall–Petch contribution as 19\.4 MPa and describes it as negligible, identifying load redistribution to graphene as the dominant mechanism \(E\)\. F reverses the actual transformation: XPS shows thermal reduction, not oxidation, of graphene oxide\. GPT\-6\-Astra selected D\(score 0\.5\), computing the Hall–Petch increment as 19\.5 MPa and declining to infer the strength; it rejected E as “unsupported”\.

Case 2 \(FMB\-F071\): interface→\\rightarrowbulk; polymer\-matrix composite, mechanicalShort carbon fiber \(10 wt%\) reinforced semi\-aromatic polyamide 6T \(PA6T\) composites were prepared using HF40S carbon fibers with different surface treatments\. Pristine carbon fibers were coated with PA6T oligomer from 1 wt% and 2 wt% ethanol suspensions and then polymerized at 320∘C for 10 min under vacuum, yielding CF\-1 and CF\-2 respectively\. A separate sample, CF\-1\-PRE, used the 1 wt% coating heated only to 200∘C without polymerization\.For single\-fiber microbond testing, an i\-CF specimen was prepared by applying PA6T oligomer matrix droplets directly on a pristine fiber and polymerizing at 320∘C for 10 min\. For bulk composite testing, the i\-CF composite was manufactured by in\-situ polymerization of the entire PA6T oligomer and carbon fiber system under vacuum at 320∘C, so that the whole composite underwent the polymerization process rather than only a surface coating\.Microbond testing gave the following interfacial shear strength \(IFSS\) values: pristine CF96±996\\pm 9MPa, CF\-1\-PRE93\.8±12\.493\.8\\pm 12\.4MPa, CF\-1107±13\.6107\\pm 13\.6MPa, CF\-2117\.6±10\.7117\.6\\pm 10\.7MPa, and i\-CF130±18130\\pm 18MPa\. AFM measurements showed the interphase thickness was approximately 350 nm for pristine CF composites and 800 nm for i\-CF composites\.Tensile strength of the bulk composites was: pristine CF184\.5±17\.9184\.5\\pm 17\.9MPa, CF\-1\-PRE214\.8±12\.2214\.8\\pm 12\.2MPa, CF\-1219\.1±10\.6219\.1\\pm 10\.6MPa, and CF\-2223\.1±9\.9223\.1\\pm 9\.9MPa\. The tensile strength of the i\-CF bulk composite is withheld from the data above\.Which of the following statements hold?A\. The tensile strength of the i\-CF composite is below 220 MPa\. B\. The high IFSS of i\-CF results from covalent C–C bonds grafted between the PA6T chains and the carbon fiber surface during in\-situ polymerization, producing adhesive failure at the fiber–matrix interface\. C\. The tensile strength of i\-CF composites is higher than that of CF\-2 composites, consistent with the interfacial shear strength ranking in which i\-CF exceeds CF\-2\. D\. The tensile strength of the i\-CF composite is within 10 MPa of the 223\.1 MPa tensile strength of CF\-2 composites\.Answer: A, D\.The withheld strength is 214\.2 MPa: below 220 \(A\) and 8\.9 MPa below CF\-2 \(D\), so the IFSS ranking does not hold for bulk strength \(C fails\)\. The study attributes the strong interface to physical entanglement and hydrogen bonding with cohesive failure in the matrix, not to covalent grafting \(B fails\), and attributes this discrepancy to voids introduced when the whole composite is polymerised under vacuum above 300∘C\. GPT\-6\-Sol selected A, D\(score 1\);GPT\-6\-Astra and Claude Opus 5\.5 selected A\(score 0\.5\), stating that D “requires the withheld i\-CF tensile measurement”\.

Case 3 \(FMB\-F253\): single\-scale, meso; ceramic, electricalNi0\.6\-xMn1\.8Co0\.6CuxO4\(0≤x≤0\.250\\leq x\\leq 0\.25\) spinel NTC thermistor ceramics are synthesised by solid\-state reaction and sintered at 1150∘C for 3 h\. All compositions reach a relative density above 97%\. X\-ray diffraction shows that the undoped ceramic \(x=0x=0\) is a pure cubic spinel, while the Cu\-doped ceramics develop a tetragonal spinel phase whose relative content reaches 35\.2% atx=0\.25x=0\.25\. X\-ray photoemission O 1s spectra of thex=0\.15x=0\.15andx=0\.25x=0\.25ceramics are deconvoluted into lattice\-oxygen and oxygen\-vacancy\-related components\. Room\-temperature resistivityρ25\\rho\_\{25\}falls from 271Ω\\Omegacm atx=0x=0to 20Ω\\Omegacm atx=0\.25x=0\.25\. AC impedance spectra of thex=0\.15x=0\.15andx=0\.25x=0\.25ceramics are described by an equivalent circuit that separates a grain resistanceRgR\_\{g\}from a grain\-boundary resistanceRg​bR\_\{gb\}, and the grain and grain\-boundary activation energiesEa​\(Rg\)E\_\{a\}\(R\_\{g\}\)andEa​\(Rg​b\)E\_\{a\}\(R\_\{gb\}\)are obtained by Arrhenius fitting of those fitted resistances\. SEM micrographs give average grain sizes of 3\.22 µm atx=0x=0, 3\.55 µm atx=0\.1x=0\.1, 3\.78 µm atx=0\.15x=0\.15and 14\.57 µm atx=0\.25x=0\.25\. The average grain size, in µm, atx=0\.2x=0\.2is withheld from the data above\.Which of the following statements hold?A\. The increase in tetragonal spinel phase fraction with Cu doping is primarily caused by Jahn–Teller distortion of Cu2\+ions at octahedral B sites\. B\. The average grain size is more than 4 µm smaller than the 14\.57 µm measured atx=0\.25x=0\.25\. C\. The relative contribution of the oxygen\-vacancy\-related O 1s component increases as Cu content rises fromx=0\.15x=0\.15tox=0\.25x=0\.25\. D\. The overall electrical resistivity of the ceramics is primarily governed by the grain resistance rather than the grain\-boundary resistance\. E\. Between thex=0\.15x=0\.15andx=0\.25x=0\.25compositions, the grain\-boundary activation energy decreases while the grain activation energy decreases\. F\. The average grain size exceeds 12 µm\.Answer: A, B, C\.The withheld grain size is 10\.22 µm \(B holds, F fails\)\. The study attributes the tetragonal phase to Jahn–Teller distortion of Cu2\+at B sites \(A\), reports an increase in the oxygen\-vacancy component from 35\.8% to 40\.1% \(C\), identifies grain\-boundary resistance as dominant \(D fails\), and reportsEa​\(Rg​b\)E\_\{a\}\(R\_\{gb\}\)falling from 0\.253 to 0\.229 eV whileEa​\(Rg\)E\_\{a\}\(R\_\{g\}\)rises from 0\.113 to 0\.195 eV \(E fails\)\. GPT\-6\-Astra selected A, B, C, E\(score 0\);GPT\-6\-Sol selected all six options\(score 0\)\.