Monkey King Bang: A Unified Scientific Multimodal Foundation Model

# Monkey King Bang: A Unified Scientific Multimodal Foundation Model
Source: [https://arxiv.org/html/2607.20557](https://arxiv.org/html/2607.20557)
###### Abstract

Scientific discovery is increasingly shifting from isolated disciplines to multi\-domain reasoning, and AI for science faces a similar transition\. Existing systems are either specialised for individual domains or unify scientific data mainly through text tokenisation and prompt\-based interfaces, limiting their ability to handle diverse scientific inputs, produce modality\-native outputs, and support joint understanding, reasoning, and generation across scientific domains\. We introduce MKB, a unified scientific multimodal model for both understanding and generation, built around a shared Transformer backbone and modality\-tailored encoders, adapters, and decoders\. MKB covers six scientific branches, including DNA, RNA, proteins, small molecules, earth science, and medical images, and supports native outputs such as biological sequences, molecular strings, meteorological fields, and segmentation masks\. Training follows a two\-stage modality\-then\-language curriculum: Stage 1 aligns modality\-specific components with the frozen backbone, and Stage 2 consolidates them with the language backbone using mixed scientific and general corpora\. Experiments show that MKB achieves competitive scientific understanding across biological and molecular benchmarks, produces high\-fidelity native outputs for weather forecasting, biological generation, and medical\-image segmentation, and largely retains the general capabilities of its Qwen3\-VL backbone\. These results demonstrate the feasibility of the proposed paradigm, suggesting that shared\-backbone models with modality\-tailored components can provide a promising foundation for future cross\-domain scientific multimodal exploration\. The model and code are publicly available athttps://github\.com/Shanghai\-Academy\-of\-AI\-For\-Science/MKBandhttps://huggingface\.co/sais\-org/MKB\.

## 1Introduction

Scientific discovery increasingly extends beyond the boundaries of individual disciplines\. Understanding how genomic variation contributes to disease, for example, often requires integrating evidence from genetics, transcriptomics, protein biology, and molecular pharmacology\. Similar interdependencies are pervasive across science, where relevant knowledge is distributed across heterogeneous representations, physical scales, and disciplinary domains\. Modern scientific discovery therefore increasingly depends on the coordinated analysis of multiple forms of scientific evidence\.

Artificial intelligence for science has followed a similar trajectory\. Early AI systems were largely developed as specialist models, each tailored to the data structure and prediction targets of an individual field, and such systems now define the state of the art within many individual scientific domains\. These specialist systems can be broadly grouped into two families\. The first targets scientific data whose structure resembles common visual or spatio\-temporal signals\. For medical images and atmospheric fields, segmentation and forecasting models adapt vision or spatio\-temporal architectures and learn domain\-specific structure from large modality\-native corpora, producing sharper segmentation masks and lower forecast errors than domain\-agnostic backbones\[[5](https://arxiv.org/html/2607.20557#bib.bib45),[30](https://arxiv.org/html/2607.20557#bib.bib42),[6](https://arxiv.org/html/2607.20557#bib.bib41),[4](https://arxiv.org/html/2607.20557#bib.bib38),[23](https://arxiv.org/html/2607.20557#bib.bib40),[17](https://arxiv.org/html/2607.20557#bib.bib39)\]\. The second targets scientific objects whose structure is difficult to capture through generic visual or textual representations\. Proteins, DNA, and RNA require specialised alphabets and long\-range sequence modelling, while molecules are more naturally represented as graphs or chemically meaningful tokenisations\. These requirements have led to strong specialist models, including protein language models\[[20](https://arxiv.org/html/2607.20557#bib.bib16),[18](https://arxiv.org/html/2607.20557#bib.bib30)\], nucleotide encoders\[[32](https://arxiv.org/html/2607.20557#bib.bib2),[7](https://arxiv.org/html/2607.20557#bib.bib60)\], and molecular graph networks\[[2](https://arxiv.org/html/2607.20557#bib.bib52)\]\. Their empirical strength arises from architectural priors and modality\-native supervision that a generic backbone would otherwise need to learn from scratch\. By construction, however, these systems typically remain confined to individual domains and do not support direct cross\-domain composition within a single model\.

Recent scientific generalists partly address cross\-domain modelling by connecting diverse scientific inputs to a shared language backbone, typically through text\-like serialisation or tool\-mediated prompts\[[27](https://arxiv.org/html/2607.20557#bib.bib59),[13](https://arxiv.org/html/2607.20557#bib.bib1),[33](https://arxiv.org/html/2607.20557#bib.bib58)\]\. Such approaches provide a unified conversational interface, but their predominantly text\-centric interaction mechanisms are not equally suitable for all scientific data\. For inputs, dense atmospheric fields, molecular structures, and biomedical images contain spatial, geometric, or numerical structure that may be difficult to preserve through serialisation\. For outputs, tasks such as weather forecasting and medical\-image segmentation require native dense predictions, including latitude–longitude fields and per\-pixel masks, while text\-centric models can only describe these outputs or delegate them to external predictors, rather than decoding them directly from shared hidden representations\. Even the largest current scientific multimodal LLMs, such as the trillion\-parameter Intern\-S1\-Pro\[[33](https://arxiv.org/html/2607.20557#bib.bib58)\], focus on textual scientific reasoning and do not natively emit continuous\-field forecasts or high\-resolution segmentation masks\.

Taken together, existing systems leave an important gap: a single shared\-backbone model that can directly encode heterogeneous scientific modalities in structurally appropriate forms, jointly contextualise information across them, and generate modality\-native outputs across multiple scientific domains\.

Motivated by this gap, we introduceMonkeyKingBang \(MKB\), a unified scientific multimodal foundation model for understanding and generating heterogeneous scientific data\. MKB covers six scientific branches: DNA, RNA, proteins, small molecules, Earth\-system data, and medical images\. It is built around a shared autoregressive Transformer backbone, with modality\-tailored encoders, adapters, and, where applicable, decoders\. For a given task, the relevant encoder–adapter pairs map scientific inputs into modality\-token sequences aligned with the hidden space of the shared backbone\. These modality blocks are arranged according to the task and jointly contextualised within a shared sequence\. For generation tasks, the target\-modality hidden states are selected from the contextualised sequence and passed to the corresponding domain\-specific decoder, which produces outputs in the required native form\. This design allows the modalities involved in a task to interact through a shared representation space while retaining important modality\-specific properties, including sequential dependencies, molecular geometry, continuous physical signals, and high\-dimensional spatial structure\.

However, jointly training heterogeneous modality\-specific pathways within a shared backbone is challenging, as differences in data structure, scale, and supervision can lead to unstable optimisation and interference between modalities\. To address this, training follows a two\-stage modality\-then\-language curriculum\. In the first stage, each modality\-specific pathway is aligned with the frozen shared backbone, enabling its encoder, adapter, and, when applicable, decoder to establish a stable interface with the backbone representation space\. In the second stage, the modality components and language backbone are jointly consolidated using mixed scientific and general\-purpose corpora\. This curriculum integrates heterogeneous scientific capabilities into a single checkpoint while limiting degradation of the backbone’s original language and vision capabilities\.

We evaluate MKB along three complementary dimensions: scientific understanding, scientific generation, and general capabilities\. Across biological sequence understanding benchmarks, MKB ranks within the top two on17/2017/20tasks, surpassing the LLM\-based state\-of\-the\-art systems \(16/2016/20\)\[[33](https://arxiv.org/html/2607.20557#bib.bib58)\]despite using roughly two orders of magnitude fewer parameters \(Figure[1](https://arxiv.org/html/2607.20557#S1.F1)\)\. On molecular benchmarks, it also surpasses specialist models on a subset of tasks\. For native scientific generation, it produces high\-fidelity outputs across biological sequences, molecular strings, meteorological fields, and medical\-image segmentation masks\. In particular, MKB achieves strong Earth\-system forecasting performance, including better medium\-range forecasts than HRES\[[10](https://arxiv.org/html/2607.20557#bib.bib69)\]on the evaluated settings, and obtains the best overall Dice performance across the evaluated medical\-imaging modalities\. After joint scientific multimodal training, it also largely retains the general\-purpose capabilities of its Qwen3\-VL backbone\[[24](https://arxiv.org/html/2607.20557#bib.bib51)\]\.

![Refer to caption](https://arxiv.org/html/2607.20557v1/fig/podium_stacked.png)Figure 1:Podium coverage on the Biology\-Instructions understanding suite\.For each biological category and the 20\-task overall, bars stack first\-place \(solid\) on second\-place \(hatched\) finishes, and the top\-of\-bar number is their sum\. MKB \(11B\) reaches17/2017/20top\-two, ahead of Intern\-S1\-Pro \(1T,16/2016/20\) and far above Biology\-Instructions \(8B,7/207/20\)\.Overall, these results demonstrate the feasibility of a shared\-backbone modelling paradigm in which biological sequences, molecular graphs, spatial image features, and continuous physical fields are encoded into a common representation space and decoded into modality\-native scientific outputs\. They also highlight remaining challenges in precise scalar regression, particularly for ADMET prediction, where stronger quantitative supervision and regression\-oriented modelling may be beneficial\.

## 2Architecture

![Refer to caption](https://arxiv.org/html/2607.20557v1/x1.png)

Figure 2:Overview of MKB\. MKB consists of general and domain\-specific encoders, adapters, and decoders attached to a shared autoregressive Transformer backbone\. Heterogeneous inputs are mapped into a unified multimodal token sequence for joint contextualization, after which target\-modality hidden states are dispatched to the corresponding decoder to generate outputs in their native forms\. Dashed pathways denote framework components that are not evaluated for native generation in this report\.### 2\.1Overview

We propose MKB, a unified scientific multimodal model that supports scientific understanding and modality\-native generation across heterogeneous scientific domains\. The model is built around a sharedQwen3\-VL\-8BTransformer backbone\[[24](https://arxiv.org/html/2607.20557#bib.bib51)\], with hidden dimension \(DLLMD\_\{\\mathrm\{LLM\}\}=4096\), together with structure\-aware representation pathways and modality\-specific generation components\. MKB covers six scientific branches: DNA, RNA, proteins, small molecules, Earth\-system data, and medical images, while retaining the general capabilities of Qwen3\-VL\.

Scientific inputs are represented according to their native structural properties before being aligned with the hidden space of the shared backbone\. Biological sequences and molecular graphs are encoded into variable\-length features and compressed into fixed\-length latent tokens through Perceiver\-style resamplers and modality\-specific projectors\. Earth\-system fields instead retain a dense latitude–longitude token grid to preserve their spatial organisation, while biomedical segmentation uses a dual\-path design that combines instruction\-conditioned Qwen3\-VL representations with dense SAM3 image features\. For each task, only the relevant modalities are composed into the shared multimodal sequence and jointly contextualised by the autoregressive Transformer\.

For understanding tasks, the shared language\-model head generates textual responses autoregressively\. For generation tasks, hidden states associated with the target modality are selected from the contextualised sequence and passed to the corresponding generation pathway, which produces outputs in their native form, including biological sequences, molecular strings, meteorological fields, and segmentation masks\. The following sections describe the unified representation framework, the modality\-specific representation pathways, the shared backbone and multimodal composition mechanism, and the modality\-native generation pathways\.

Table 1:Summary of the modality\-specific representation and generation pathways\. All branches support textual understanding through the shared language\-model head\. For sequence and graph modalities, encoder features are converted by a modality adapter, consisting of a Perceiver\-style resampler and an MLP projector, into the shared Transformer hidden sizeDLLM=4096D\_\{\\mathrm\{LLM\}\}=4096\. Dashes indicate that no modality\-native generation pathway is used in the reported experiments\.BranchNative inputRepresentation pathwayKey specificationBackbone\-facing representationNative\-output pathwayDNANucleotide sequenceEncoder \+ modality adapter1\-D convolutional Transformer; width512512;88blocks;K=64K=646464tokens inℝDLLM\\mathbb\{R\}^\{D\_\{\\mathrm\{LLM\}\}\}–RNANucleotide sequenceEncoder \+ modality adapter1\-D convolutional Transformer; width512512;88blocks;K=64K=646464tokens inℝDLLM\\mathbb\{R\}^\{D\_\{\\mathrm\{LLM\}\}\}Linear nucleotide head; RNA sequenceProteinAmino\-acid sequenceEncoder \+ modality adapterESM2\-150M; width640640;K=64K=646464tokens inℝDLLM\\mathbb\{R\}^\{D\_\{\\mathrm\{LLM\}\}\}–Small moleculeMolecular graphEncoder \+ modality adapterSuiren\-ConfAvg; width256256; local and fully connected graphs;K=64K=646464tokens inℝDLLM\\mathbb\{R\}^\{D\_\{\\mathrm\{LLM\}\}\}Cross\-attentive Transformer; 226\-token SMILES vocabularyEarth systemDense physical fieldStructure\-preserving spatial projectionSwin\-style encoder; width20482048;1212blocks;6×66\\times 6patchesDense120×240120\\times 240latitude–longitude token gridSwin\-style field decoder; 70\-channel meteorological fieldBiomedical imageImage and text instructionDual\-path semantic and spatial representationQwen3\-VL semantic pathway; SAM3 dense\-image pathway;2016×20162016\\times 2016inputSemantic conditioning tokens; dense SAM3 features retained for decoderSAM3 mask decoder;576×576576\\times 576segmentation mask

### 2\.2Unified Scientific Representation Framework

For each task, the participating scientific modalities are processed through modality\-specific representation pathways that preserve their native structural properties while aligning them with the hidden space of the shared Transformer\. Given a native input𝐗\(m\)\\mathbf\{X\}\_\{\(m\)\}from modalitymm, the corresponding encoder produces continuous representations

𝐇\(m\)∈ℝN\(m\)×d\(m\),\\mathbf\{H\}\_\{\(m\)\}\\in\\mathbb\{R\}^\{N\_\{\(m\)\}\\times d\_\{\(m\)\}\},\(1\)whereN\(m\)N\_\{\(m\)\}andd\(m\)d\_\{\(m\)\}denote the number and dimensionality of the encoder features, respectively\.

For sequence\-like and graph\-like modalities, including DNA, RNA, proteins, and molecules, the encoder output is passed to a modality\-specific adapter\. Each adapter consists of a Perceiver\-style resampler followed by a lightweight MLP projector, thereby converting variable\-length encoder features into a fixed number of backbone\-facing tokens\. Specifically, the resampler uses learnable latent queries𝐐\(m\)∈ℝK\(m\)×d\(m\)\\mathbf\{Q\}\_\{\(m\)\}\\in\\mathbb\{R\}^\{K\_\{\(m\)\}\\times d\_\{\(m\)\}\}to attend to the encoder features and produce

𝐙\(m\)=Resampler\(m\)⁡\(𝐇\(m\);𝐐\(m\)\)∈ℝK\(m\)×d\(m\)\.\\mathbf\{Z\}\_\{\(m\)\}=\\operatorname\{Resampler\}\_\{\(m\)\}\\left\(\\mathbf\{H\}\_\{\(m\)\};\\mathbf\{Q\}\_\{\(m\)\}\\right\)\\in\\mathbb\{R\}^\{K\_\{\(m\)\}\\times d\_\{\(m\)\}\}\.\(2\)Here,K\(m\)K\_\{\(m\)\}denotes the number of latent tokens, with all sequence and graph pathways usingK\(m\)=64K\_\{\(m\)\}=64in the current implementation\. The MLP projector then aligns the resampled representations with the hidden space of the shared Transformer:

𝐅\(m\)=MLP\(m\)⁡\(𝐙\(m\)\)∈ℝK\(m\)×DLLM\.\\mathbf\{F\}\_\{\(m\)\}=\\operatorname\{MLP\}\_\{\(m\)\}\\left\(\\mathbf\{Z\}\_\{\(m\)\}\\right\)\\in\\mathbb\{R\}^\{K\_\{\(m\)\}\\times D\_\{\\mathrm\{LLM\}\}\}\.\(3\)Together, the resampler and MLP form a compact modality adapter that controls the number of tokens introduced into the shared backbone while preserving the sequence\- or graph\-level information extracted by the native encoder\.

Dense scientific modalities instead use structure\-preserving representation pathways\. For Earth\-system data, the spatial encoder retains the complete latitude–longitude patch grid and projects the resulting weather tokens directly into the shared hidden space, avoiding the fixed\-length latent bottleneck used for sequence and graph modalities\. This preserves the two\-dimensional spatial organisation required by the positional encoding and field decoder\. Biomedical segmentation adopts a dual\-path design in which the Qwen3\-VL image–text pathway provides instruction\-conditioned semantic representations, while a parallel SAM3\[[5](https://arxiv.org/html/2607.20557#bib.bib45)\]vision pathway preserves high\-resolution spatial features for mask prediction\.

For each task, only the relevant input and target modalities are instantiated\. Their backbone\-facing representations are arranged within a shared multimodal token sequence and jointly contextualised by the autoregressive Transformer\. For understanding tasks, the contextualised representations support autoregressive textual prediction\. For generation tasks, the hidden states associated with the target modality are selected and passed to the corresponding modality\-specific decoder\. Where applicable, the Earth\-system and biomedical pathways additionally retain native dense features that support the reconstruction of spatially detailed outputs\.

### 2\.3Modality\-specific Input Representation Pathways

#### 2\.3\.1Biological Sequence Representation

Biological sequences are represented using sequence encoders that preserve local motifs and long\-range dependencies before mapping variable\-length inputs into the fixed\-length latent representation described above\. We use separate pathways for nucleotide and protein sequences because their alphabets, structural patterns, and available pretrained representations differ\.

##### Nucleotide sequences\.

For DNA and RNA, the inputs are character\-level nucleotide sequences,𝐗DNA∈\{A,T,G,C,N\}L,\\mathbf\{X\}\_\{\\mathrm\{DNA\}\}\\in\\left\\\{\\mathrm\{A\},\\mathrm\{T\},\\mathrm\{G\},\\mathrm\{C\},\\mathrm\{N\}\\right\\\}^\{L\},and𝐗RNA∈\{A,U,G,C,N\}L,\\mathbf\{X\}\_\{\\mathrm\{RNA\}\}\\in\\left\\\{\\mathrm\{A\},\\mathrm\{U\},\\mathrm\{G\},\\mathrm\{C\},\\mathrm\{N\}\\right\\\}^\{L\},, respectively\. Here,LLdenotes the variable sequence length andN\\mathrm\{N\}represents an ambiguous nucleotide\. DNA and RNA are tokenised at the character level using separate modality\-specific vocabularies,𝒱​DNA\\mathcal\{V\}\{\\mathrm\{DNA\}\}and𝒱​RNA\\mathcal\{V\}\{\\mathrm\{RNA\}\}\. For RNA inputs containing thymine,T\\mathrm\{T\}is normalised to uracil,U\\mathrm\{U\}, before tokenisation\. Both DNA and RNA sequences are truncated to at most 2048 tokens\.

DNA and RNA are encoded by independently parametrised one\-dimensional convolutional Transformers trained on their respective nucleotide corpora\. Each encoder combines a convolutional stem for capturing local motif\-level patterns with Transformer blocks for modelling longer\-range dependencies, producing contextualised per\-token representations𝐇DNA∈ℝB×LDNA′×dDNA,\\mathbf\{H\}\_\{\\mathrm\{DNA\}\}\\in\\mathbb\{R\}^\{B\\times L^\{\\prime\}\_\{\\mathrm\{DNA\}\}\\times d\_\{\\mathrm\{DNA\}\}\},and𝐇RNA∈ℝB×LRNA′×dRNA\\mathbf\{H\}\_\{\\mathrm\{RNA\}\}\\in\\mathbb\{R\}^\{B\\times L^\{\\prime\}\_\{\\mathrm\{RNA\}\}\\times d\_\{\\mathrm\{RNA\}\}\}, respectively\. The two encoders share the same architectural design but use modality\-specific vocabularies and independent parameters\. Their detailed configurations are summarised in Table[1](https://arxiv.org/html/2607.20557#S2.T1)\.

##### Protein sequences\.

For proteins, we use the pretrained ESM2 encoder\[[18](https://arxiv.org/html/2607.20557#bib.bib30)\]to extract residue\-level representations from an amino\-acid sequence𝐗prot∈ΣL\\mathbf\{X\}\_\{\\mathrm\{prot\}\}\\in\\Sigma^\{L\}, whereΣ\\Sigmadenotes the ESM2 amino\-acid vocabulary\. Sequences are truncated to at most10241024residues and processed by ESM2 to produce contextualised residue embeddings𝐇prot∈ℝB×Lprot′×dprot\.\\mathbf\{H\}\_\{\\mathrm\{prot\}\}\\in\\mathbb\{R\}^\{B\\times L^\{\\prime\}\_\{\\mathrm\{prot\}\}\\times d\_\{\\mathrm\{prot\}\}\}\.Unlike the nucleotide encoders, which are trained specifically for MKB, the protein pathway builds on the pretrained sequence representations learned by ESM2\.

The resulting DNA, RNA, and protein features are passed through their respective modality adapters, each consisting of a Perceiver\-style resampler followed by an MLP projector, producing6464backbone\-facing tokens for each participating biological modality\.

#### 2\.3\.2Molecular Graph Representation

Small molecules are represented as molecular graphs in order to preserve their native chemical connectivity\. Given a SMILES string𝐗mol\\mathbf\{X\}\_\{\\mathrm\{mol\}\}, we parse it into a two\-dimensional graph𝒢=\(𝒱,ℰ\),\\mathcal\{G\}=\(\\mathcal\{V\},\\mathcal\{E\}\),where𝒱\\mathcal\{V\}andℰ\\mathcal\{E\}denote the sets of atoms and chemical bonds, respectively\. Atom and bond attributes encode standard chemical properties, including atomic identity, charge, hybridisation, aromaticity, bond type, and stereochemistry\.

To capture both local chemical structure and longer\-range atom–atom dependencies, the molecular representation uses two complementary connectivity patterns: a sparse local graph𝐀loc\\mathbf\{A\}\_\{\\mathrm\{loc\}\}, containing the observed chemical bonds, and a fully connected graph𝐀full\\mathbf\{A\}\_\{\\mathrm\{full\}\}over all atom pairs\. The local graph preserves bonded interactions and functional\-group structure, while the fully connected graph enables information exchange between chemically distant atoms\.

Atom\-level representations are extracted using the pretrained Suiren\-ConfAvg graph foundation model\[[2](https://arxiv.org/html/2607.20557#bib.bib52)\]\. Given𝒢\\mathcal\{G\}, Suiren produces contextualised atom embeddings𝐇mol∈ℝB×Natom×dmol,\\mathbf\{H\}\_\{\\mathrm\{mol\}\}\\in\\mathbb\{R\}^\{B\\times N\_\{\\mathrm\{atom\}\}\\times d\_\{\\mathrm\{mol\}\}\},whereNatomN\_\{\\mathrm\{atom\}\}denotes the number of atoms after batch padding\. The resulting atom features are passed through the molecule\-specific modality adapter, consisting of a Perceiver\-style resampler followed by an MLP projector, producing6464backbone\-facing molecular tokens\.

#### 2\.3\.3Dense Physical\-field Representation

Earth\-system data are represented as dense physical fields in order to preserve their continuous spatial organisation\. The input is a global gridded atmospheric state𝐗w∈ℝB×Tin×C×H×W\\mathbf\{X\}\_\{\\mathrm\{w\}\}\\in\\mathbb\{R\}^\{B\\times T\_\{\\mathrm\{in\}\}\\times C\\times H\\times W\}, whereTin=1T\_\{\\mathrm\{in\}\}=1,C=70C=70, and\(H,W\)=\(721,1440\)\(H,W\)=\(721,1440\)correspond to the latitude–longitude dimensions of the0\.25∘0\.25^\{\\circ\}grid\. The input channels include upper\-air and surface meteorological variables, while static geographical fields and temporal metadata provide additional spatial and forecast conditioning\.

A Swin\-style shifted\-window Transformer encodes the input into a dense latitude–longitude patch grid\. Using a patch size of \(6×66\\times 6\), the encoder produces\(Hp,Wp\)=\(120,240\)\(H\_\{\\mathrm\{p\}\},W\_\{\\mathrm\{p\}\}\)=\(120,240\), yielding28,80028\{,\}800weather tokens\. The encoder configuration is summarised in Table[1](https://arxiv.org/html/2607.20557#S2.T1)\. Unlike sequence\-like modalities, these dense meteorological tokens are not compressed by a resampler\. Instead, they are projected to the shared Transformer hidden space through the cross\-modal merger and inserted into reserved atmosphere placeholder positions\. This preserves the two\-dimensional latitude–longitude structure required by the spatial positional encoding and the downstream field decoder\.

#### 2\.3\.4Dual\-path Biomedical Image Representation

Biomedical image segmentation requires both semantic understanding of the instruction and preservation of fine\-grained spatial details\. We therefore adopt a dual\-path representation design that processes the same medical image through complementary semantic and spatial pathways\.

Given a medical imageIIand a text instructionpp, the semantic pathway reuses the native Qwen3\-VL image–text encoding pipeline\. The image and instruction are encoded as a multimodal sequence, and the shared Transformer produces instruction\-conditioned hidden states𝐇sem∈ℝB×Lsem×DLLM\\mathbf\{H\}\_\{\\mathrm\{sem\}\}\\in\\mathbb\{R\}^\{B\\times L\_\{\\mathrm\{sem\}\}\\times D\_\{\\mathrm\{LLM\}\}\}\. These hidden states are projected into semantic conditioning tokens for the segmentation decoder\.

In parallel, the spatial pathway processes the same image with the SAM3 image processor and vision backbone\[[5](https://arxiv.org/html/2607.20557#bib.bib45)\], producing dense visual features that preserve local boundaries and fine\-grained spatial structure\. In our experiments, input images are resized to2016×20162016\\times 2016, and the segmentation pathway predicts masks at a spatial resolution of576×576576\\times 576\.

Together, these complementary pathways provide the segmentation decoder with instruction\-aware semantic context and dense spatial evidence, enabling accurate pixel\-level biomedical mask prediction\.

### 2\.4Shared Backbone and Multimodal Composition

The shared backbone of MKB is initialised from Qwen3\-VL\-8B\[[24](https://arxiv.org/html/2607.20557#bib.bib51)\], which provides a pretrained vision–language representation space and a decoder\-only causal Transformer for multimodal contextualisation\. The language Transformer contains3636layers with hidden sizeDLLM=4096D\_\{\\mathrm\{LLM\}\}=4096\. We retain its native attention, normalisation, feed\-forward, and multi\-axis rotary positional encoding mechanisms, as well as the original vision tower for processing general images and videos\.

For each task, text tokens, optional Qwen3\-VL image or video tokens, and the backbone\-facing representations of the participating scientific modalities are arranged into a single multimodal sequence,𝐄fused∈ℝB×Ltot×DLLM,\\mathbf\{E\}\_\{\\mathrm\{fused\}\}\\in\\mathbb\{R\}^\{B\\times L\_\{\\mathrm\{tot\}\}\\times D\_\{\\mathrm\{LLM\}\}\},whereLtotL\_\{\\mathrm\{tot\}\}depends on the modalities involved in the task\. Scientific representations are inserted at reserved modality\-specific positions delimited by dedicated control tokens\. Sequence and graph modalities contribute fixed\-length adapter tokens, whereas dense Earth\-system representations retain their latitude–longitude token layout\. The resulting sequence is processed jointly by the causal Transformer, enabling information exchange among the modalities participating in the current task\.

For textual understanding tasks, the contextualised hidden states are decoded autoregressively through the native language\-model head\. For modality\-native generation, control tokens and reserved target positions identify the required output pathway\. The hidden states associated with the target modality are selected from the contextualised sequence and passed to the corresponding generation component, such as the RNA head, molecular decoder, physical\-field decoder, or biomedical segmentation decoder\. Molecular generation uses a specialised SMILES vocabulary that remains separate from the language\-model vocabulary\.

Therefore, the shared backbone provides a common contextual representation space across heterogeneous scientific inputs, while modality\-specific generation pathways preserve the structural requirements of their respective outputs\. For dense Earth\-system and biomedical outputs, the shared hidden states additionally condition native spatial features retained by their structure\-preserving representation pathways\.

### 2\.5Modality\-native Output Generation Pathways

As summarised in Table[1](https://arxiv.org/html/2607.20557#S2.T1), MKB supports modality\-native generation for RNA sequences, small molecules, Earth\-system fields, and biomedical segmentation masks\. Given contextualised backbone states𝐇LLM∈ℝB×Ltot×DLLM\\mathbf\{H\}\_\{\\mathrm\{LLM\}\}\\in\\mathbb\{R\}^\{B\\times L\_\{\\mathrm\{tot\}\}\\times D\_\{\\mathrm\{LLM\}\}\}, the hidden states associated with the target modality are selected from the shared multimodal sequence and passed to the corresponding output pathway\. These output components differ according to the structural requirements of each target modality\.

##### RNA sequence generation\.

Because the shared backbone already provides autoregressive contextualisation across the target sequence, RNA generation uses a lightweight bias\-free linear head\. The head projects the target\-modality hidden states𝐇targetRNA∈ℝB×LRNA×DLLM\\mathbf\{H\}^\{\\mathrm\{RNA\}\}\_\{\\mathrm\{target\}\}\\in\\mathbb\{R\}^\{B\\times L\_\{\\mathrm\{RNA\}\}\\times D\_\{\\mathrm\{LLM\}\}\}directly into the RNA nucleotide vocabulary\. RNA tokens are generated autoregressively, and the head is trained using standard shifted cross\-entropy\.

##### Molecular generation\.

Molecular generation uses an independent autoregressive Transformer because the specialised whole\-atom SMILES vocabulary is decoupled from the language\-model vocabulary\. Hidden states gathered from the molecular conditioning positions,𝐇targetmol∈ℝB×Lcond×DLLM\\mathbf\{H\}^\{\\mathrm\{mol\}\}\_\{\\mathrm\{target\}\}\\in\\mathbb\{R\}^\{B\\times L\_\{\\mathrm\{cond\}\}\\times D\_\{\\mathrm\{LLM\}\}\}, are provided as cross\-attention context to a Flamingo\-style Transformer decoder\[[1](https://arxiv.org/html/2607.20557#bib.bib48)\]\. The decoder attends causally to the generated SMILES prefix while cross\-attending to the shared multimodal context, and predicts logits𝐘^mol∈ℝB×Lmol×Vmol\\widehat\{\\mathbf\{Y\}\}\_\{\\mathrm\{mol\}\}\\in\\mathbb\{R\}^\{B\\times L\_\{\\mathrm\{mol\}\}\\times V\_\{\\mathrm\{mol\}\}\}over a specialisedVmol=226V\_\{\\mathrm\{mol\}\}=226whole\-atom SMILES vocabulary\. It is trained with shifted cross\-entropy\. At inference time, the generated whole\-atom tokens are detokenised into a SMILES string\.

##### Dense physical\-field generation\.

The Earth\-system output pathway converts the contextualised weather\-token states back into a dense meteorological field\. A back\-projection module maps the selected weather states𝐇targetw∈ℝB×Lw×DLLM\\mathbf\{H\}^\{\\mathrm\{w\}\}\_\{\\mathrm\{target\}\}\\in\\mathbb\{R\}^\{B\\times L\_\{\\mathrm\{w\}\}\\times D\_\{\\mathrm\{LLM\}\}\}to the native weather\-token dimensiondwd\_\{\\mathrm\{w\}\}\. The projected features are fused with encoder skip features and refined by a Swin\-style shifted\-window Transformer decoder\. The resulting token grid is reshaped into a two\-dimensional feature map and upsampled to the original0\.25∘0\.25^\{\\circ\}resolution, yielding a one\-step prediction𝐘^earth\\widehat\{\\mathbf\{Y\}\}\_\{\\mathrm\{earth\}\}\. Multi\-step forecasts are generated by autoregressive rollout, where each predicted field is re\-fed as the next encoder input together with the updated lead\-time embedding\.

The Earth\-system decoder is trained with a latitude\-weighted Charbonnier loss, using cosine latitude weights to account for grid\-cell area and channel weights to balance variables with different dynamic ranges\. Specific humidity is down\-weighted withαhum=0\.3\\alpha\_\{\\mathrm\{hum\}\}=0\.3, near\-surface temperature and 10 m winds useαsur=0\.5\\alpha\_\{\\mathrm\{sur\}\}=0\.5, and upper\-air variables together with mean sea\-level pressure useαupper=1\.0\\alpha\_\{\\mathrm\{upper\}\}=1\.0\. During training, rollout supervision is applied only at selected checkpoints, including the verification step, which reduces memory consumption while still exposing the model to autoregressive error accumulation\. During inference, the same autoregressive rollout procedure is used to generate forecasts at user\-specified horizons\.

##### Biomedical mask generation\.

Biomedical segmentation combines the two complementary representations introduced in the dual\-path image pathway\. The contextualised Qwen3\-VL states provide instruction\-conditioned semantic information, while the SAM3 image branch supplies dense multi\-scale spatial features\. The SAM3 mask decoder\[[5](https://arxiv.org/html/2607.20557#bib.bib45)\]fuses these representations and predicts object\-level classification scores, bounding boxes, and segmentation masks\{\(ℓq,𝐛q,𝐌q\)\}q=1Q\\\{\(\\ell\_\{q\},\\mathbf\{b\}\_\{q\},\\mathbf\{M\}\_\{q\}\)\\\}\_\{q=1\}^\{Q\}, whereQQdenotes the number of object queries\.

The segmentation pathway is trained with standard SAM3\-style multi\-objective supervision\. Object\-query predictions are matched to ground\-truth instances through Hungarian assignment using classification, box, and mask\-overlap costs\. The matched predictions are optimized with sigmoid focal classification loss, L1 box loss, generalized IoU loss, sigmoid focal mask loss, and mask Dice loss\. The reported configuration further introduces an image\-level semantic\-mask Dice objective and an auxiliary meta\-object cross\-entropy objective\. At inference time, the highest\-confidence mask is upsampled to the target resolution as the final instruction\-guided segmentation output\.

## 3Data Construction

MKB is trained on six scientific modalities together with scientific text, general text, and vision–language data\. Because scientific supervision is scarce, noisy, and distributed across incompatible file formats, we treat data construction as a first\-class component of the system\. This section describes the coverage of scientific tasks and modalities, the unified instruction schema for native objects, the construction and leakage control of cross\-modal supervision, and the Stage\-2 consolidation mixture\.

### 3\.1Scientific Data Coverage

Forbiological sequences and molecules, we primarily adopt established benchmarks to maintain comparability with prior work\. DNA, RNA, protein, and RNA–protein tasks are primarily drawn from Biology\-Instructions\[[13](https://arxiv.org/html/2607.20557#bib.bib1)\], covering sequence understanding, regulatory prediction, protein\-property prediction, and biological interaction tasks\. Molecular data cover property prediction, ADMET assessment, molecule understanding, and text\-conditioned molecular generation, primarily using SMolInstruct\[[28](https://arxiv.org/html/2607.20557#bib.bib34)\], TDC\[[15](https://arxiv.org/html/2607.20557#bib.bib32)\], and MoleHB\[[2](https://arxiv.org/html/2607.20557#bib.bib52)\]\.Earth\-systemforecasting uses ERA5 reanalysis\[[14](https://arxiv.org/html/2607.20557#bib.bib35)\]at0\.25∘0\.25^\{\\circ\}resolution, andmedical\-image segmentationuses BiomedParse\[[30](https://arxiv.org/html/2607.20557#bib.bib42)\]across nine imaging modalities\. General multimodal capability is maintained with the native image–text, OCR, grounding, and instruction data inherited from the Qwen3\-VL\[[24](https://arxiv.org/html/2607.20557#bib.bib51)\]backbone\. Native objects are standardized, and invalid records are removed before training\.

### 3\.2Unified instruction schema

A central design choice is that native modality objects are*not tokenized as ordinary prompt text*\. Instead, native objects are stored separately from the dialogue and linked to it through modality references\. The dialogue refers to each object through a reserved modality placeholder, such as<protein\>\. At data\-loading time, each placeholder is expanded into the corresponding encoder segment \(Section[2](https://arxiv.org/html/2607.20557#S2)\), allowing the language model to condition on continuous modality representations rather than lossy textual serializations\. This schema keeps the interface consistent across understanding, generation, and cross\-modal samples\.

Each sample is formatted as an instruction dialogue: the system turn specifies the task and output contract, the human turn contains the placeholders and question, and the assistant turn contains the supervised answer\. Only assistant answer tokens are supervised by the text loss, while prompt tokens, modality placeholders, and decoder\-output positions are masked\. For native\-generation tasks, an output placeholder in the assistant response routes the corresponding hidden states to the target\-modality decoder rather than the language head\. Continuous targets are normalized using training\-set statistics only\.

### 3\.3Cross\-modal Data Construction

Ready\-made instruction data is dominated by single\-modality tasks, whereas many scientific questions require reasoning across coupled biological and chemical entities\. We therefore build cross\-modal datasets directly from authoritative, openly redistributable databases, as summarized in Table[2](https://arxiv.org/html/2607.20557#S3.T2)\. A source\-agnostic intermediate representation standardizes entities, relations, provenance, and evidence across databases\. Our pipeline is built around a Common Intermediate Representation \(CIR\) that standardizes heterogeneous database records into typed scientific entities, relations, and associated provenance and evidence\. By decoupling source\-specific parsing from downstream task construction, the CIR enables consistent entity reuse across datasets, simplifies the integration of new data sources, and supports systematic construction of cross\-modal tasks\.

##### Enzyme catalysis \(protein×\\timesmolecule\)\.

We construct enzyme\-conditioned molecular generation data by linking curated enzyme sequences to reaction substrates, products, and cofactors\[[3](https://arxiv.org/html/2607.20557#bib.bib54),[12](https://arxiv.org/html/2607.20557#bib.bib53)\]\. This construction yields two generation tasks: \(1\) substrate\-to\-product prediction, conditioned on an enzyme sequence and substrate set, and \(2\) cofactor prediction, conditioned on an enzyme sequence\. We retain directionally resolved, mass\-balanced reactions and reserve high\-confidence annotations for evaluation\.

##### Molecular interaction and binding \(protein/RNA×\\timesmolecule\)\.

We include drug–target binding\-affinity regression from DAVIS and KIBA\[[22](https://arxiv.org/html/2607.20557#bib.bib55)\]and BindingDB\[[9](https://arxiv.org/html/2607.20557#bib.bib56)\], with measurements split by affinity type andzz\-scored independently\. Together with RNA–protein interaction data from Biology\-Instructions\[[13](https://arxiv.org/html/2607.20557#bib.bib1)\], these datasets provide explicit supervision for cross\-modal reasoning over biological macromolecules and small molecules\.

Table 2:Cross\-modal supervision used in training and evaluation\. Grouped labels indicate generation \(gen\), affinity regression \(reg\), and binary interaction classification \(cls\)\.TypeDatasetModalitiesSourcegenenzyme\_substrate2product<protein\>\+\+<mol\>→\\rightarrow<mol\>UniProt\+Rhea\+ChEBIenzyme\_cofactor<protein\>→\\rightarrow<mol\>UniProt\+ChEBIregdavis\_dti<protein\>\+\+<mol\>→\\rightarrowscalarDeepDTA \(DAVIS\)kiba\_dti<protein\>\+\+<mol\>→\\rightarrowscalarDeepDTA \(KIBA\)bindingdb\_ki<protein\>\+\+<mol\>→\\rightarrowscalarBindingDBbindingdb\_ic50<protein\>\+\+<mol\>→\\rightarrowscalarBindingDBbindingdb\_ec50<protein\>\+\+<mol\>→\\rightarrowscalarBindingDBbindingdb\_kd<protein\>\+\+<mol\>→\\rightarrowscalarBindingDBclsrna\_protein\_rpi<rna\>\+\+<protein\>→\\rightarrow0/1Biology\-Instr\.

### 3\.4Entity\-level splitting and leakage control

Random record\-level splits systematically overestimate performance on biological data, because near\-duplicate sequences and scaffolds recur across the split boundary\. For natively constructed datasets, protein clusters and molecular scaffolds are assigned at the split level rather than the individual\-record level\. Exact duplicates are removed before split assignment\. We audit all splits for overlap under the specified protein\-cluster and molecular\-scaffold criteria\. For DAVIS and KIBA, we retain the official DeepDTA folds for comparability and cross\-deduplicate BindingDB against those test pairs\.

### 3\.5Stage\-2 Consolidation Mixture

The Stage\-2 consolidation mixture combines all scientific modalities with scientific\-text and general\-purpose corpora\. It includes scientific multiple\-choice and open\-ended QA in the style of MMLU/MMMU\[[25](https://arxiv.org/html/2607.20557#bib.bib49),[29](https://arxiv.org/html/2607.20557#bib.bib50)\], multimodal QA, general instruction\-following, mathematics, code, agent and GUI\-grounding data, general image–text and OCR data, and the scientific modality data described above\. To prevent high\-volume general corpora from dominating training, we down\-sample large sources relative to scientific data while keeping smaller scientific benchmarks at full size\.

## 4Training Strategy

MKB follows atwo\-stage modality\-then\-language curriculum\. Stage 1 aligns each scientific modality with the frozen Qwen3\-VL backbone\. For the sequence, molecular, and weather branches, the trainable components comprise the corresponding encoder, adapter, and applicable output decoder\. MedSeg follows a different dual\-path design: the Qwen3\-VL vision\-language pathway remains frozen, while the Qwen\-to\-SAM3 projection, all non\-text SAM3 modules, and the auxiliary prediction head are optimized\. This teaches each modality\-specific component to translate its native signal into representations consumable by the language model while preserving the pretrained language prior\. Stage 2 then loads andfreezesthe Stage\-1 modality\-specific components,unfreezesthe LLM backbone, and mixes all scientific modalities with large\-scale scientific\-text and general instruction corpora\. This consolidation stage improves scientific\-text benchmark performance while retaining the multimodal capabilities acquired in Stage 1\.

Table 3:Optimization settings and trainable components for the two\-stage training curriculum\. Here, bs denotes per\-device batch size and ga denotes gradient accumulation\. Bioseq\. \+ Mol\. comprises the RNA, protein, DNA, and molecule branches\.Modality / mixtureLRbs×\\timesgaTrainable componentsStage 1: Scientific\-interface alignmentBioseq\. \+ Mol\.5×10−55\\times 10^\{\-5\}16×116\\times 1Encoders \+ adapters; decoders as applicableWeather1×10−61\\times 10^\{\-6\}1×11\\times 1Encoder \+ adapter \+ decoderMedSeg2×10−62\\times 10^\{\-6\}1×21\\times 2SAM3 vision encoder \+ adapter \+ segmentation decoderStage 2: Shared\-backbone consolidationMixed modalities1×10−61\\times 10^\{\-6\}1×41\\times 4Qwen3\-VL LLM backbone \+lm\_head### 4\.1Stage 1: Independent Modality\-component Training

In Stage 1, each active scientific modality is trained against the frozen Qwen3\-VL language backbone\. The trainable modules include the corresponding domain encoder, adapter, and, when applicable, modality decoder, as summarised in Table[3](https://arxiv.org/html/2607.20557#S4.T3)\. RNA and protein are trained jointly in the Stage\-1 run used for the reported mixed model because RNA–protein interaction data require both encoders to be instantiated and supervised in the same forward graph\. DNA, molecule, weather, and medical segmentation modules are trained or loaded from their corresponding modality\-specific Stage\-1 checkpoints\. Protein–molecule binding and enzyme datasets are used as mixed corpus in Stage 2 rather than in Stage\-1\. Across Stage 1 runs, we use AdamW with weight decay0\.010\.01, a cosine learning\-rate schedule with linear warm\-up, gradient clipping at global norm1\.01\.0,bf16mixed precision, gradient checkpointing, and DeepSpeed ZeRO\-2\. The per\-modality learning rates, batch sizes, and trainable components are listed in Table[3](https://arxiv.org/html/2607.20557#S4.T3)\.

##### RNA and protein joint training\.

RNA and protein are trained jointly because the RNA–protein interaction dataset contains paired RNA and protein inputs\. This requires both modality encoders to be active in the same forward graph, while the frozen LLM provides the shared conditioning backbone\. To stabilise distributed training with RNA\-only, protein\-only, and paired batches, the sampler keeps each accumulation window single\-kind and synchronises the micro\-batch type across data\-parallel ranks\.

##### Weather rollout training\.

The weather component is trained with truncated autoregressive rollout\. At each iteration, the model is unrolled for several consecutive66\-hour steps by feeding its own predictions back as input, and gradients are back\-propagated only through the final step\. Lead\-time conditioning is sampled across short forecast horizons, with lead\-dependent loss scaling applied to the forecast loss\. We also maintain an exponential moving average of the weather weights and apply patch\-wise spatial dropout as augmentation\.

##### Medical\-segmentation training

MedSeg processes each image through two parallel visual pathways\. The frozen Qwen3\-VL vision\-language pathway produces instruction\-conditioned joint image\-text hidden states, which are mapped by a trainable two\-layer projection into the SAM3 text\-embedding space\. All non\-text components of SAM3, including its vision backbone, segmentation decoder, and prediction heads, are jointly optimized with the Qwen\-to\-SAM3 projection, while the native SAM3 text encoder remains frozen\. We further introduce a lightweight auxiliary meta\-object classification head to provide coarse semantic supervision\.

### 4\.2Stage 2: Joint Seven\-modality Consolidation

In Stage 2, all Stage 1 modality components, including encoders, adapters, and decoders, are loaded into a fresh base model and kept*frozen*\. We unfreeze only the Qwen3\-VL LLM backbone andlm\_head, and train them with all seven modalities activated using the consolidation mixture described in Section[3\.5](https://arxiv.org/html/2607.20557#S3.SS5)\. This stage aims to improve scientific\-text and general reasoning performance while preserving the modality\-specific skills acquired in Stage 1\.

##### Optimisation\.

As summarised in Table[3](https://arxiv.org/html/2607.20557#S4.T3), Stage 2 freezes all Stage\-1 modality components and optimises only the Qwen3\-VL LLM backbone andlm\_head\. We use AdamW with cosine scheduling,bf16mixed precision, gradient checkpointing, and DeepSpeed ZeRO\-2\. Because biological, weather, and MedSeg samples follow structurally different collators and loss paths, we use single\-modality micro\-batches and synchronise the micro\-batch type across data\-parallel ranks\.

### 4\.3Training objectives and supervision masks

The training mixture contains heterogeneous objectives, including text generation, biological sequence modelling, molecular generation, weather forecasting, and medical\-image segmentation\. We optimise a unified objective with a text loss and one modality\-specific loss for each active modality:

ℒ=ℒtext\+∑m∈ℳsm​ℒm,sm∈\{0,1\},\\mathcal\{L\}\\;=\\;\\mathcal\{L\}\_\{\\text\{text\}\}\\;\+\\;\\sum\_\{m\\in\\mathcal\{M\}\}s\_\{m\}\\,\\mathcal\{L\}\_\{m\},\\qquad s\_\{m\}\\in\\\{0,1\\\},\(4\)whereℳ=\{RNA,mol,weather,MedSeg\}\\mathcal\{M\}=\\\{\\text\{RNA\},\\text\{mol\},\\text\{weather\},\\text\{MedSeg\}\\\}, andsms\_\{m\}indicates whether modalitymmis active in the current optimisation step\. Active modality losses use equal outer weights, while structured losses such as weather forecasting and MedSeg retain the internal weights defined in their decoder sections\.

The text lossℒtext\\mathcal\{L\}\_\{\\text\{text\}\}is the standard causal language\-modelling cross\-entropy over assistant answer tokens, with prompt tokens, modality placeholders, and domain\-output positions masked usingignore\_index=−100=\-100\. For modality understanding tasks, encoders and adapters are trained through the task\-specific supervision signal, such as textual answer likelihood or dataset\-specific classification/regression losses\. In the reported configuration, sequence generation is supervised by two modality\-specific heads: a bias\-free RNA linear head over shared\-backbone hidden states and an independent molecular autoregressive decoder over a decoupled SMILES vocabulary\. Both heads read the LLM hidden states at their domain\-output positions, apply a one\-token causal shift, and compute cross\-entropy against domain\-native labels\. DNA and protein decoding heads are supported by the framework but are not activated in the reported experiments\.

## 5Experiment

### 5\.1Experimental Setup

#### 5\.1\.1Implementation Details

##### Default checkpoint and optimization\.

Unless otherwise stated, all main experiments use the default Stage\-2 unified checkpoint obtained after the two\-stage training pipeline described in Section[4](https://arxiv.org/html/2607.20557#S4)\. Optimization and distributed\-training settings follow the configuration reported in Section[4](https://arxiv.org/html/2607.20557#S4)\. The default evaluation setting uses a maximum context length of32,76832\{,\}768tokens\. Ablation studies, when reported, explicitly specify any deviation from this checkpoint, context length, or training configuration\.

##### Hardware and distributed training\.

All training runs are conducted on NVIDIA H200 GPUs with 141 GB of memory\. We use PyTorch together with DeepSpeed ZeRO Stage 2, which shards optimizer states and gradients across data\-parallel workers to reduce memory redundancy, and launch jobs withtorchrunover multiple 8\-GPU nodes\. The language backbone uses FlashAttention\-2 for efficient attention computation, while the native vision tower uses scaled\-dot\-product attention for numerical stability in the segmentation pathway\. The Stage\-1 single\-domain runs are distributed over 48 to 248 GPUs depending on the memory and sequence\-length requirements of each modality, and the Stage\-2 seven\-modality consolidation run is trained on 288 GPUs \(36 nodes\)\. For large multi\-node jobs, we reduce the ZeRO communication bucket sizes to improve inter\-node efficiency, and we disable communication\-computation overlap to ensure stable numerical behaviour for medical segmentation\.

##### Evaluation configuration\.

Table[4](https://arxiv.org/html/2607.20557#S5.T4)summarises the decoding and context\-length settings used for all reported benchmarks\. Values correspond to the CLI arguments actually passed by the invocation wrappers used to produce the reported numbers; per\-task overrides are indicated in the same row\. All inference runs usebf16mixed precision, share the same fixed random seed, and share the same trained Stage\-2 checkpoint\. We use two inference paths depending on the task\. Tasks that require the scientific modality encoders or decoders \(DNA, RNA, protein, and molecule understanding and generation, as well as weather and segmentation\) are run with the full modality\-augmented model\. Text\-only and image\-text benchmarks that exercise only the shared language backbone are served through vLLM withgpu\_mem\_util=0\.95for throughput\. Decoding hyperparameters for each group are listed in Table[4](https://arxiv.org/html/2607.20557#S5.T4)\.

Table 4:Evaluation configuration for MKB across benchmark groups\. “TT” is sampling temperature, “pp” and “kk” are top\-ppand top\-kktruncation, “pp” and “rp” are presence and repetition penalties\.GroupDecodingMax model lenScientific capability\. Default: greedy \(do\_sample=False\)\.Biology sequence group\(shared defaults\)32,768Molecule\(shared defaults\)32,768Cross\-modal\(shared defaults\)32,768Weather \(ERA5\)AR rollout, base lead∈\{6,12,18,24\}\\in\\\{6,12,18,24\\\}h→240\\rightarrow 240h—Medical SegSAM3\-head cls\+box\+mask \(no LM decoding\)—General capability\. Default:T=1\.0,p=1\.0,k=40,pp=2\.0,rp=1\.0T\{=\}1\.0,\\,p\{=\}1\.0,\\,k\{=\}40,\\,\\text\{pp\}\{=\}2\.0,\\,\\text\{rp\}\{=\}1\.0\.Default group\(shared defaults\)40,960IMO\-AnsBench\(shared defaults\)65,536RefCOCO\-avggreedy \(T=0T\{=\}0\)16,384

#### 5\.1\.2Benchmarks

Our benchmark covers six scientific domains and scientific text\. Unless otherwise stated, models are evaluated on held\-outtestsplits, with task\-specific evaluation settings summarized in Table[4](https://arxiv.org/html/2607.20557#S5.T4)\. Table[5](https://arxiv.org/html/2607.20557#S5.T5)summarises the biological and molecular benchmarks used in our evaluation, covering DNA, RNA, protein, cross\-modal, and molecule tasks across classification, regression, and generation settings\.

Table 5:Benchmark inventory for biological and molecular evaluation tasks\.DomainTaskDescriptionTask typeDNAEMPEpigenetic Marks PredictionBinary classificationPD300Promoter Detection 300Binary classificationCPDCore Promoter DetectionBinary classificationTB\-HTF Binding Sites Detection HumanBinary classificationTB\-MTF Binding Sites Detection MouseBinary classificationEAEnhancer Activity PredictionRegressionRNAncRNANon\-coding RNA Function ClassificationMulti\-class classificationAPAAPA Isoform PredictionRegressionMRLMean Ribosome Loading PredictionRegressionPRSProgrammable RNA SwitchesRegressionModifModification PredictionMulti\-class classificationCRI\-OnCRISPR On Target PredictionRegressionToeholdToehold Switch DesignGenerationProteinStaStability PredictionRegressionFluFluorescence PredictionRegressionTherThermostability PredictionRegressionECEnzyme Commission Number PredictionMulti\-label classificationSolSolubility PredictionBinary classificationCross\-modalAANAntibody\-Antigen NeutralizationBinary classificationRPIRNA\-Protein Interaction PredictionBinary classificationEPIEnhancer\-Promoter Interaction PredictionBinary classificationMoleculeClinToxClinTox ClassificationBinary classificationSIDERSIDER ClassificationBinary classificationBBBPBBBP ClassificationBinary classificationHIVHIV ClassificationBinary classificationESOLESOL RegressionRegressionLipophilicityLipophilicity RegressionRegressionADMETADMET ClassificationBinary classificationSmolDesignSMILES DesignGenerationMoleHBMoleHB RegressionRegression##### RNA and DNA\.

Most RNA and DNA benchmarks are drawn from the Biology\-Instructions suite\[[13](https://arxiv.org/html/2607.20557#bib.bib1)\]\. As summarised in Table[5](https://arxiv.org/html/2607.20557#S5.T5), they cover regulatory DNA tasks, RNA function and activity prediction, RNA–protein interaction, and RNA sequence generation across classification, regression, and generation settings\. Inputs are raw nucleotide sequences or sequence pairs processed by the modality\-specific DNA and RNA encoders described in Section[2\.3](https://arxiv.org/html/2607.20557#S2.SS3)\. Regression targets are evaluated on their raw physical scales, except for the binned enhancer\-activity variant, which discretises labels into100100equal\-frequency bins and maps predictions back to bin means\. We note that the siRNA Efficiency Prediction task from the Biology\-Instructions suite is excluded from Table[5](https://arxiv.org/html/2607.20557#S5.T5), as its underlying raw data is no longer available for download\.

##### Proteins\.

Protein benchmarks are also drawn from the Biology\-Instructions suite\[[13](https://arxiv.org/html/2607.20557#bib.bib1)\]\. As summarised in Table[5](https://arxiv.org/html/2607.20557#S5.T5), they cover protein\-property regression, solubility classification, antibody–antigen neutralization, and multi\-label enzyme\-function annotation\. Inputs are amino\-acid sequences, with paired sequences used for antibody–antigen neutralization, and outputs include scalar targets, binary labels, or EC\-code sets evaluated under the task\-provided metrics\.

##### Molecules\.

Molecule benchmarks cover ADMET classification, molecular property regression, MoleculeNet endpoints, and text\-to\-SMILES generation across classification, regression, and generation settings, as summarised in Table[5](https://arxiv.org/html/2607.20557#S5.T5)\. Prediction tasks use molecular graphs derived from SMILES strings, while the generation task maps text descriptions to SMILES\. The SMolInstruct tasks are based on instruction\-formatted MoleculeNet\[[26](https://arxiv.org/html/2607.20557#bib.bib31)\]and ChEBI\-20/MolT5\[[8](https://arxiv.org/html/2607.20557#bib.bib33)\]data\[[28](https://arxiv.org/html/2607.20557#bib.bib34)\], while ADMET and property\-regression tasks follow TDC\[[15](https://arxiv.org/html/2607.20557#bib.bib32)\]and MoleHB\[[2](https://arxiv.org/html/2607.20557#bib.bib52)\], respectively\.

##### Earth science\.

The Earth\-science benchmark uses ERA5 global reanalysis data\[[14](https://arxiv.org/html/2607.20557#bib.bib35)\]\. Given one0\.25∘0\.25^\{\\circ\}global weather snapshot, the task is to forecast future atmospheric and surface states\. We train from 2002 January to 2023 June and evaluate on a temporally disjoint hold\-out from 2023 July, with forecasts rolled out every66hours up to240240hours\.

##### Medical image segmentation\.

BiomedParse\[[30](https://arxiv.org/html/2607.20557#bib.bib42)\]is a text\-grounded medical\-image segmentation benchmark with image–prompt pairs and binary mask targets\. We evaluate at2016×20162016\\times 2016resolution on the pooled test set of102,855102\{,\}855pairs across nine imaging modalities: CT, MRI, X\-ray, Ultrasound, Dermoscopy, Endoscopy, Pathology, Fundus, and OCT\.

##### Scientific text\.

Scientific\-text capability is evaluated on two public, evaluation\-only benchmarks,MMLU\-Pro\[[25](https://arxiv.org/html/2607.20557#bib.bib49)\]andMMMU\-Pro\[[29](https://arxiv.org/html/2607.20557#bib.bib50)\]\.MMLU\-Prois a12,03212\{,\}032\-question text multiple\-choice benchmark across1414categories, with up to1010options per question, and is evaluated in a55\-shot chain\-of\-thought setting\.MMMU\-Prois a multimodal image–text multiple\-choice benchmark; we evaluate its standard \(1010options\) and vision subsets, each with1,7301\{,\}730questions, in a zero\-shot chain\-of\-thought setting\. In both cases, the task is to select a single answer option\.

#### 5\.1\.3Evaluation metrics

We report the primary metric used in each result table rather than listing all auxiliary evaluation statistics\. For classification tasks, we report Matthews correlation coefficient \(MCC\), accuracy, or AUROC, depending on the benchmark convention\. Regression tasks are evaluated withR2R^\{2\}or Spearman’sρ\\rho, and the protein EC multi\-label task is evaluated withFmaxF\_\{\\max\}\. Native RNA and molecule generation are evaluated with exact match \(EM\), recovery\-style sequence metrics, and fingerprint Tanimoto similarity \(FTS\), as appropriate to the output space\. Dense\-output tasks use domain\-standard metrics: Earth\-system forecasting is evaluated with latitude\-weighted RMSE and anomaly correlation coefficient \(ACC\), while biomedical segmentation is evaluated with Dice score\. General capability retention is reported using the official benchmark scores for each text, vision, reasoning, and coding evaluation\.

#### 5\.1\.4Competitors\.

We compare MKB against four families of baselines: general multimodal LLMs, scientific LLMs on biology understanding, specialist chemistry baselines, and modality\-specific specialists\. These baselines are chosen to position MKB along well\-defined axes rather than to construct an exhaustive leaderboard\.

##### General multimodal LLMs\.

For text and multimodal tasks, we compare MKB with two general\-purpose multimodal LLMs\.*Qwen3\-VL\-8B\-Instruct*\[[24](https://arxiv.org/html/2607.20557#bib.bib51)\]is the direct backbone of MKB \(Section[2](https://arxiv.org/html/2607.20557#S2)\)\. This same\-scale comparison tests whether integrating six scientific encoders and Stage\-2 consolidation preserves the base VLM’s general multimodal capability\.*Intern\-S1\-Pro*is a trillion\-parameter scientific mixture\-of\-experts model with an explicit “thinking” configuration for reasoning\-heavy tasks\. Its roughly100×100\\timeslarger parameter budget provides a high\-capacity reference for where a compact unified model can track a much larger reasoning\-optimised system and where the gap remains\.

##### Scientific LLMs on biology understanding\.

On the Biology\-Instructions understanding suite \(Table[6](https://arxiv.org/html/2607.20557#S5.T6)\), we compare MKB against two LLM\-based references at markedly different capacity:*Biology\-Instructions*\[[13](https://arxiv.org/html/2607.20557#bib.bib1)\]\(the released ChatMultiOmics checkpoint, a Llama\-3\.1\-8B\-Instruct fine\-tune that represents DNA, RNA, protein and multi\-molecular sequences directly as text tokens without modality\-specific encoders\) and*Intern\-S1\-Pro*\(the trillion\-parameter text\-token\-based scientific LLM reference introduced above\)\. The Biology\-Instructions comparison isolates the benefit of MKB’s dedicated biological encoders at a matched88B parameter scale, while Intern\-S1\-Pro bounds how far a much larger text\-token\-only scientific LLM can go on the same tasks\. We do not report per\-task specialist SOTAs on this suite, since MKB’s core comparison target here is other unified LLM\-based models rather than task\-tailored specialists trained end\-to\-end on a single benchmark\.

##### Specialist chemistry baselines\.

For molecule benchmarks, we compare MKB with standard chemistry\-specific baselines for each suite\. These baselines are dedicated molecular models rather than general\-purpose LLMs, and are selected to match the reference set commonly used by each benchmark family\.

ForMoleHB, we compare against*Mole\-BERT*,*UniMol\-v1*, and*UniMol\-v2*, together with the leaderboard SOTA,*Suiren*in its conformer\-averaged variant\[[2](https://arxiv.org/html/2607.20557#bib.bib52)\]\. Since Suiren is also the molecule encoder used by MKB \(Section[2](https://arxiv.org/html/2607.20557#S2), Table[1](https://arxiv.org/html/2607.20557#S2.T1)\), this comparison effectively tests the effect of MKB’s shared instruction backbone and routing on top of a fixed molecular representation\.

ForADMET, we compare against*ChemProp*, a widely used D\-MPNN framework for molecular property prediction, as well as*DeepAuto\-QSAR*and*Uni\-QSAR*, two QSAR\-competition\-tuned systems designed for ADMET\-style leaderboards\. We also report the specialist SOTA on the same suite\. These baselines are all per\-endpoint tuned, with task\-specific heads or hyperparameter searches, and thus represent the strongest dedicated\-model comparator for a unified instruction\-driven model\.

ForSMolInstruct\[[28](https://arxiv.org/html/2607.20557#bib.bib34)\]understanding tasks, we compare against*Uni\-Mol*\[[31](https://arxiv.org/html/2607.20557#bib.bib62)\], which is used by LlaSMol\[[28](https://arxiv.org/html/2607.20557#bib.bib34)\]as the task\-specific specialist for every MoleculeNet\-style property\-prediction endpoint\[[26](https://arxiv.org/html/2607.20557#bib.bib31)\], together with*LlaSMol*as an instruction\-tuned chemistry LLM at a comparable parameter budget\. For the SMolInstruct text\-to\-SMILES generation split, the task\-specific specialist is*MolT5*\[[8](https://arxiv.org/html/2607.20557#bib.bib33)\], again following LlaSMol’s compared\-model setup, we additionally include*ChemLLM*in this generation comparison\.

##### Weather baselines\.

For weather forecasting, we compare MKB against the operational*ECMWF HRES*\[[11](https://arxiv.org/html/2607.20557#bib.bib37)\]deterministic forecast, the physics\-based numerical weather prediction \(NWP\) system used by the European Centre for Medium\-Range Weather Forecasts, which is the standard baseline adopted by data\-driven medium\-range global forecast models such as Pangu\-Weather\[[4](https://arxiv.org/html/2607.20557#bib.bib38)\]and GraphCast\[[17](https://arxiv.org/html/2607.20557#bib.bib39)\]\. This places MKB against a production\-grade NWP system rather than another data\-driven model, and lets us report ML\-vs\-NWP gap and skill horizon on the same latitude\-weighted metrics\.

##### Medical segmentation baselines\.

For medical segmentation, we compare MKB with six segmentation networks: BiomedParse\[[30](https://arxiv.org/html/2607.20557#bib.bib42)\], SAM\[[16](https://arxiv.org/html/2607.20557#bib.bib43)\], MedSAM\[[19](https://arxiv.org/html/2607.20557#bib.bib44)\], SAM3\[[5](https://arxiv.org/html/2607.20557#bib.bib45)\], DINO\+SAM, and DINO\+MedSAM\[[21](https://arxiv.org/html/2607.20557#bib.bib47)\]\. These baselines cover text\-grounded biomedical segmentation, general and medical promptable segmentation, concept\-driven segmentation, and DINOv2\-conditioned mask decoding\.

### 5\.2Experimental Results

MKB is a unified scientific multimodal model designed to support both scientific understanding and generation across diverse scientific domains\. We therefore evaluate MKB along three complementary axes that collectively probe the quality and generality of its shared multimodal representation\. The first axis focuses on scientific understanding tasks, where paired text and scientific\-modality inputs, including DNA, RNA, proteins, and molecular graphs, are mapped to textual answers \(Section[5\.2\.1](https://arxiv.org/html/2607.20557#S5.SS2.SSS1)\)\. The second axis evaluates scientific\-modality generation, where text prompts are decoded into native\-format scientific outputs through modality\-specific decoder heads\. We compare MKB with corresponding specialist generative models \(Sections[5\.2\.2](https://arxiv.org/html/2607.20557#S5.SS2.SSS2)and[5\.2\.3](https://arxiv.org/html/2607.20557#S5.SS2.SSS3)\)\. The third axis focuses on text\-only and image\-text inputs with textual outputs, examining whether joint training across seven scientific and general modalities preserves the pretrained backbone’s general knowledge, reasoning, grounding, coding, and OCR capabilities \(Section[5\.2\.4](https://arxiv.org/html/2607.20557#S5.SS2.SSS4)\)\.

#### 5\.2\.1Scientific Understanding Results

##### Biological Sequence Understanding\.

Table[6](https://arxiv.org/html/2607.20557#S5.T6)evaluates DNA, RNA, protein, and paired\-sequence cross\-modality understanding across 20 benchmarks, comparing MKB against Biology\-Instructions and Intern\-S1\-Pro\.

Overall, MKB performs strongly on classification\-oriented biological tasks, especially those dominated by sequence\-level or motif\-like signals\. In DNA, it obtains the best results among the compared LLM\-based models on EMP, PD300, CPD, and TB\-M, and is nearly tied with Intern\-S1\-Pro on TB\-H\. In RNA, it achieves the strongest results on non\-coding RNA function classification and RNA modification prediction\. These results indicate that the modality\-specific encoders and adapters provide an effective component for biological sequence understanding, particularly when the target signal can be captured from local or medium\-range sequence patterns\.

Regression\-oriented tasks show more variable behaviour\. MKB is competitive on several protein regression benchmarks, such as stability prediction and CRI\-On prediction, but remains behind Intern\-S1\-Pro on thermostability and below the best compared model on several RNA regression tasks such as MRL and PRS\. A similar pattern appears in DNA enhancer\-activity prediction\. Since these tasks require precise scalar prediction rather than categorical decision boundaries, they may benefit from dedicated numeric regression heads or stronger task\-specific supervision, which are not the focus of the current configuration\.

On paired\-sequence cross\-modality tasks, MKB shows its strongest result on RNA–protein interaction prediction, reaching76\.4976\.49MCC and outperforming both Intern\-S1\-Pro and Biology\-Instructions\. It also substantially improves over Biology\-Instructions on antibody–antigen neutralization, although Intern\-S1\-Pro remains slightly higher\. In contrast, all compared LLM\-based models perform near chance on enhancer–promoter interaction prediction, suggesting that this task remains difficult for sequence\-only multimodal LLM pipelines and may require additional regulatory or structural priors\.

Aggregating across all biological understanding benchmarks, MKB achieves the best score on9/209/20tasks and matches or exceeds Intern\-S1\-Pro on10/2010/20, while outperforming Biology\-Instructions on16/2016/20tasks\. The last row of Table[6](https://arxiv.org/html/2607.20557#S5.T6)summarises this comparison as a single scalar: the “average” row reports the macro average over all 20 tasks with equal per\-task weight\. Under this metric, MKB reaches60\.1160\.11, outperforming Intern\-S1\-Pro by8\.148\.14points \(51\.9751\.97\) and the same\-scale Biology\-Instructions baseline by22\.6722\.67points \(37\.4437\.44\), despite Intern\-S1\-Pro being approximately two orders of magnitude larger in total parameters\. The main strength is concentrated in classification and interaction\-prediction settings, whereas the remaining gaps are concentrated in scalar regression and structure\-dependent pairwise tasks\. These results motivate future extensions with dedicated regression heads and additional biological supervision for tasks requiring quantitative or higher\-order structural reasoning\.

Table 6:Performance comparison across biological sequence and cross\-modality understanding tasks\.Underlinesdenote second\-best results, andbolddenotes best results\.DomainTaskMetricMKB \(11B\)Biology\-Instructions \(8B\)Intern\-S1\-Pro \(1T\)DNAEMPMCC71\.993\.6414\.02PD300MCC91\.1758\.1882\.65CPDMCC66\.3544\.5454\.60TB\-HMCC54\.0124\.4554\.11TB\-MMCC65\.9139\.9160\.80EAPCC52\.6453\.2855\.16RNAncRNAAcc91\.4663\.0934\.50APAR2R^\{2\}79\.8759\.0182\.95MRLR2R^\{2\}35\.5447\.6452\.41PRSR2R^\{2\}25\.9926\.5733\.97ModifAUC96\.0359\.0657\.77CRI\-OnSpearman’sρ\\rho28\.76\-0\.0215\.69ProteinStaSpearman’sρ\\rho70\.6360\.2560\.82FluSpearman’sρ\\rho70\.122\.5778\.14TherSpearman’sρ\\rho46\.3745\.0759\.56ECFmax68\.6519\.7972\.70SolAcc67\.2663\.0267\.60Cross\-modalAANMCC42\.961\.0644\.76RPIMCC76\.4974\.2658\.51EPIMCC\-0\.033\.37\-1\.30average60\.1137\.4451\.97
##### Molecular Understanding\.

Table[7](https://arxiv.org/html/2607.20557#S5.T7)and Table[8](https://arxiv.org/html/2607.20557#S5.T8)report MKB’s performance across three complementary chemistry benchmark families: MoleHB, ADMET, and SMolInstruct\.

On SMolInstruct, MKB is competitive across both classification and regression endpoints, achieving the best or tied\-best result on4/64/6tasks\. It obtains the strongest results on BBBP, SIDER, and ESOL, ties Uni\-Mol on HIV, and remains close on ClinTox and Lipophilicity\. These results indicate that the molecule encoder and shared language backbone can support instruction\-conditioned molecular property prediction across diverse endpoints\.

On the broader MoleHB and ADMET suites, MKB remains below the strongest chemistry\-specialist models\. It reaches a normalized average score of0\.84080\.8408on MoleHB, above UniMol\-v2 and Mole\-BERT but below Suiren, and obtains an ADMET average of0\.73040\.7304, below purpose\-built QSAR and ADMET systems\. This gap reflects the breadth–specialization trade\-off of a unified scientific model: MKB covers molecular tasks together with other scientific modalities, whereas the strongest chemistry baselines are optimized specifically for molecular property prediction\.

Table 7:Performance comparison on MoleHB and ADMET molecular understanding benchmarks\. For MoleHB, we report normalized average scores over all sub\-tasks due to the large number of fine\-grained tasks\. For ADMET, we report the direct average over all sub\-tasks\.Underlinesdenote second\-best results, andbolddenotes best results\.DatasetMetricMKBSuirenMole\-BERTUniMol\-v1UniMol\-v2MoleHBNormalized Avg\.0\.84080\.96930\.05970\.89800\.7139
DatasetMetricMKBSuirenDeepAuto\-QSARUni\-QSARChemPropADMETAvg\.0\.73040\.80460\.78580\.80840\.7624

Table 8:Performance comparison on SMolInstruct molecular understanding tasks\.Underlinesdenote second\-best results, andbolddenotes best results\.TaskMetricMKBUni\-MolLlaSMolBBBP classificationAcc96\.9585\.3074\.60ClinTox classification92\.3692\.4093\.10HIV classification97\.0097\.0096\.70SIDER classification71\.0070\.0070\.70ESOL regressionRMSE↓\\downarrow0\.5500\.8191\.150Lipophilicity regression0\.6280\.6121\.010

#### 5\.2\.2Scientific Generation Results

Beyond understanding, MKB supports native\-format scientific generation through modality\-specific decoders \(Section[2\.5](https://arxiv.org/html/2607.20557#S2.SS5)\)\. In this report, we evaluate two such capabilities: RNA sequence generation and molecule generation\.

##### RNA Generation\.

Within the biology suite, we evaluate native RNA generation on an internal toehold\-switch design split, where a trigger sequence and linker are mapped to a full switch RNA sequence by the RNA decoder\. Since no matched generalist or specialist baselines are available, we report standalone performance only\. MKB achieves near\-saturated scores, with BLEU99\.99699\.996and per\-position recovery99\.998%99\.998\\%, consistent with the largely deterministic structure of this design task, in which the switch stem is close to a reverse\-complement expansion of the trigger combined with a fixed linker/scaffold\. These results indicate that the shared\-backbone RNA decoder can recover the target switch sequence with near single\-nucleotide precision on this split\.

##### Molecule Generation\.

We evaluate native molecular generation on SMolInstruct SMILES design tasks\. It achieves a validity of89\.09%89\.09\\%, an exact\-match \(EM\) rate of22\.22%22\.22\\%, and a fingerprint Tanimoto similarity \(FTS\) of61\.9661\.96, compared with95\.3/31\.7/73\.295\.3/31\.7/73\.2for MolT5\[[8](https://arxiv.org/html/2607.20557#bib.bib33)\]and99\.7/19\.2/61\.799\.7/19\.2/61\.7for LlaSMol\. Although MKB has lower syntactic validity than LlaSMol, it achieves higher EM and FTS\. The remaining gap to MolT5 is most visible on exact reconstruction and fingerprint similarity, suggesting that future improvements should focus on both increasing SMILES validity and better constraining generation toward the target molecular structure\.

Table 9:Performance comparison on SMolInstruct molecule generation\. We report validity, exact match \(EM\), and fingerprint Tanimoto similarity \(FTS\)\.Underlinesdenote second best results,bolddenotes best results\.MetricMKBMolT5LlaSMolChemLLMValidity89\.0995\.399\.74\.3EM22\.2231\.719\.20\.9FTS61\.9673\.261\.714\.3
##### Cross\-modal molecule generation\.

The enzyme\-catalysis data introduced in Section[3\.3](https://arxiv.org/html/2607.20557#S3.SS3)provides a direct test of cross\-modal native generation, where protein context is used to condition molecular outputs\. We evaluate this capability on two enzyme\-centred tasks: cofactor prediction from enzyme sequence, and substrate\-to\-product prediction conditioned on enzyme sequence, substrate molecules, and a curated Rhea reaction identifier\. Since these internally constructed tasks have no matched generalist or specialist baselines, we report representative cases rather than model\-versus\-model scores, treating predictions as correct when they match the curated database answers under RDKit\-canonical SMILES or InChI\. Figure[3](https://arxiv.org/html/2607.20557#S5.F3)shows one example per task produced by the released checkpoint\. In the cofactor case the model recovers glutathione, the tripeptide cofactor annotated for the enzyme in UniProt, reproducing the reference structure exactly\. In the substrate\-to\-product case the model hydrolyses the ester bond of a monoacylglycerol substrate and returns free oleate and glycerol, which is exactly the transformation catalysed by the lipase\. These examples illustrate that the molecule decoder can generate molecular outputs conditioned on protein context within the shared multimodal framework, using the same unified checkpoint without task\-specific tuning\.

\(a\) Cofactor prediction\.System\.You are an enzyme cofactor annotation expert\. Given an enzyme protein sequence, output the complete set of UniProt\-annotated cofactors for this sequence as SMILES\. Cofactors may include metal ions; they are not limited to organic coenzymes\. Output only SMILES separated by’\.’, sorted lexicographically\.Input\. <protein\> Predict all UniProt\-annotated cofactors required by this enzyme sequence, including metal ions\. Output only the SMILES set\.where<protein\>is a 230\-residue enzyme \(MSTNSDLRVTLYTYFRSS…\)\.Model output\. \[NH3\+\]\[C@@H\]\(CCC\(=O\)N\[C@@H\]\(CS\)C\(=O\)NCC\(=O\)\[O\-\]\)C\(=O\)\[O\-\]Glutathione\. Matches the UniProt\-annotated cofactor exactly \(RDKit\-canonical identical\)\.

\(b\) Substrate\-to\-product prediction\.System\.You are an enzymatic reaction annotation expert\. Given an enzyme protein sequence, substrate molecule\(s\), and a Rhea reaction ID from a curated UniProt/Rhea annotation, predict the product molecule\(s\) for that specific reaction\. Output only the product SMILES; multiple species are separated by’\.’\.Input\. <protein\> <mol\> This enzyme is annotated with Rhea:38488\. For this specific Rhea reaction, predict the product molecule\(s\) from the substrate set\. Output only the product SMILES\.where<protein\>is a 1450\-residue lipase \(MESWPGVSLVGLLLLLL…\) and<mol\>is the substrate setCCCCCCCC/C=C\\CCCCCCCC\(=O\)OCC\(O\)CO\.O\(monoacylglycerol \+ water\)\.Model output\. CCCCCCCC/C=C\\CCCCCCCC\(=O\)\[O\-\]\.OCC\(O\)CO\.\[H\+\]Oleate \+ glycerol\. Matches the curated Rhea:38488 product exactly\.

Figure 3:Representative native molecular generation cases\. \(a\) Enzyme sequence to cofactor SMILES\. \(b\) Enzyme and substrate molecules to product SMILES\.

#### 5\.2\.3Dense Scientific Output Generation

![Refer to caption](https://arxiv.org/html/2607.20557v1/fig/weather_mkb.png)

Figure 4:MKB vs\. ECMWF HRES on global ERA5 forecasting\.\(a\)Latitude\-weighted RMSE and anomaly correlation coefficient \(ACC\) as a function of lead time \(up to1010days\) for500500\-hPa geopotential \(Z500\), 2\-metre temperature \(T2M\), and mean sea\-level pressure \(MSL\)\.\(b\)Absolute\-error maps at day1010for the same three variables\.##### Earth\-Science Forecasting\.

Figure[4](https://arxiv.org/html/2607.20557#S5.F4)reports medium\-range global forecasting results on the ERA5 hold\-out set at0\.25∘0\.25^\{\\circ\}resolution\. Forecasts are initialised on the 2023 July to 2024 June hold\-out at00000000/12001200UTC and rolled out autoregressively every66hours up to240240h\. We evaluate against ERA5 using latitude\-weighted RMSE and anomaly correlation coefficient \(ACC\), with ACC computed relative to the ERA5 1990–2019 climatology\. Following common practice in data\-driven medium\-range forecasting\[[4](https://arxiv.org/html/2607.20557#bib.bib38),[17](https://arxiv.org/html/2607.20557#bib.bib39),[23](https://arxiv.org/html/2607.20557#bib.bib40)\], we report three representative variables covering upper\-air dynamics, near\-surface impact, and large\-scale pressure:500500\-hPa geopotential \(Z500\), 2\-metre temperature \(T2M\), and mean sea\-level pressure \(MSL\)\.

Figure[4](https://arxiv.org/html/2607.20557#S5.F4)\(a\) shows that MKB generally outperforms HRES across Z500, T2M, and MSL, with larger gains at longer lead times\. For Z500, the two systems are similar through the early forecast range, but MKB maintains lower error and higher ACC after roughly day44\. By day1010, it reaches about680​m2/s2680~\\mathrm\{m^\{2\}/s^\{2\}\}RMSE and0\.640\.64ACC, compared with about800​m2/s2800~\\mathrm\{m^\{2\}/s^\{2\}\}and0\.550\.55for HRES\. Similar long\-range improvements appear for MSL, where MKB reaches about625​Pa625~\\mathrm\{Pa\}RMSE at day1010versus about740​Pa740~\\mathrm\{Pa\}for HRES, while T2M shows the most persistent advantage across the full1010\-day horizon, reaching day\-1010RMSE of about2\.5​K2\.5~\\mathrm\{K\}versus2\.9​K2\.9~\\mathrm\{K\}for HRES\. Across these variables, the slower medium\-range degradation of RMSE and ACC indicates stronger forecast skill from MKB at longer lead times\.

The day\-1010absolute\-error maps in Figure[4](https://arxiv.org/html/2607.20557#S5.F4)\(b\) show a consistent spatial pattern\. HRES has larger errors across mid\-latitude and polar regions for all three variables, and especially over northern\-hemisphere land areas for T2M\. MKB’s errors are lower in magnitude, with residual hotspots for Z500 concentrated over the Southern Ocean storm\-track band and the high\-latitude polar caps, and, for T2M, over northern\-hemisphere land in a spatially reduced version of the HRES pattern\.

##### Medical\-Image Segmentation\.

Table 10:Dice score comparison across medical imaging modalities\. We report the average Dice score \(%\) for each method\. The best result in each row is highlighted inbold, and the second\-best result isunderlined\.ModalityMKBBiomedParseDINO\+MedSAMDINO\+SAMMedSAMSAM3SAMAll91\.2090\.7315\.3715\.1083\.5535\.4071\.29CT93\.3692\.259\.5910\.3483\.8728\.9374\.10MRI85\.2985\.2513\.2812\.3975\.9053\.6468\.34OCT85\.3186\.636\.686\.9856\.268\.6955\.99X\-ray98\.0298\.2837\.2230\.6397\.7539\.9681\.35Dermoscopy98\.0897\.1181\.2878\.2997\.3551\.4788\.23Endoscopy97\.3996\.7725\.0124\.5497\.0538\.8292\.88Fundus91\.3391\.503\.192\.7388\.0618\.5857\.16Pathology87\.2981\.5725\.3824\.6943\.4426\.0842\.06Ultrasound90\.5491\.0317\.1222\.9189\.765\.2357\.47Table[10](https://arxiv.org/html/2607.20557#S5.T10)reports mean Dice on the BiomedParsetestsplits, aggregated over102,855102\{,\}855image–prompt pairs spanning nine imaging modalities\. On the pooled*All*split, MKB achieves the best average Dice score \(91\.2091\.20\), slightly ahead of BiomedParse \(90\.7390\.73\) and substantially above MedSAM, SAM, SAM3, and the DINO\-conditioned variants\. Across modalities, MKB obtains the best score on5/95/9subsets \(CT, MRI, Dermoscopy, Endoscopy, and Pathology\) and ranks second on the remaining four \(OCT, X\-ray, Fundus, and Ultrasound\), with only small gaps to the leading method on those subsets\. The largest margin appears on Pathology \(87\.2987\.29vs\.81\.5781\.57,\+5\.72\+5\.72Dice\), followed by CT \(93\.3693\.36vs\.92\.2592\.25,\+1\.11\+1\.11Dice\)\.

The comparison with SAM3 is particularly informative because MKB uses SAM3\-style dense visual features but adds instruction\-conditioned semantic guidance through the shared multimodal backbone\. The large gap between off\-the\-shelf SAM3 \(35\.4035\.40\) and MKB \(91\.2091\.20\) highlights the importance of semantic conditioning for text\-prompted biomedical segmentation\. The low scores of the DINO\-conditioned variants further suggest that generic visual features alone are insufficient for this setting\.

Overall, these results show that the dual\-path biomedical segmentation pathway can match or exceed dedicated segmentation specialists while remaining part of the same unified checkpoint used for the other scientific modalities\.

![Refer to caption](https://arxiv.org/html/2607.20557v1/fig/medseg_mkb_v2.png)Figure 5:Qualitative text\-prompted segmentation comparison between MKB and BiomedParse across the nine medical\-imaging modalities of Table[10](https://arxiv.org/html/2607.20557#S5.T10)\. Row 1: MKB’s predictions, Row 2: BiomedParse predictions, Row 3: ground\-truth masks; per\-instance Dice is overlaid on each non\-GT tile\. Targets, left to right, are:*COVID\-19 infection*\(chest CT, COVID\-19\-CT\),*right ventricle*\(cardiac MRI, ACDC\),*retinal edema*\(OCT, OCT\-CME\),*left lung*\(chest X\-ray, COVID\-QU\-Ex\),*skin lesion*\(dermoscopy, ISIC\),*colon polyp*\(endoscopy, NeoPolyp\),*optic disc*\(retinal fundus, G1020\),*glandular structure*\(H&E pathology, GlaS\), and*benign tumor*\(breast ultrasound, BreastUS\)\.Table 11:General capability comparison between MKB and its shared Transformer backbone, Qwen3\-VL\-8B\-Ins\.Bolddenotes the better result, with ties bolded for both models\.DatasetMetricMKBQwen3\-VL\-8B\-InsMMLU\-ProAccuracy73\.31%73\.36%MMMU\-Pro57\.60%57\.29%AIME\-202546\.67%43\.33%ScreenSpot V292\.30%92\.30%IMO\-Answer\-BenchAccuracy \(avg@8\)34\.94%34\.63%RefCOCO\-avgAcc@IoU≥\\geq0\.588\.00%88\.01%IFBenchStrict prompt\-level accuracy32\.33%32\.33%OCRBench V2 ENGOCRBench V2 score57\.40%57\.50%OCRBench V2 CHN63\.90%63\.80%SArena \(Icon\)SArena\-Icon score71\.49%74\.83%LCB V6pass@150\.43%50\.33%Figure[5](https://arxiv.org/html/2607.20557#S5.F5)compares MKB’s text\-prompted predictions with BiomedParse against the ground\-truth masks on one case per modality drawn from the pooled BiomedParse test split\. Reading each column top\-to\-bottom, we compare how closely the top\-row MKB mask and the middle\-row BiomedParse mask reproduce the bottom\-row annotation in shape, extent, and location\.

Reading the columns from left to right, the differences against the ground truth are most visible on four modalities\. On\[CT\], MKB tightly outlines the annotated COVID\-19 infection while BiomedParse bleeds into neighbouring parenchyma; on\[MRI\], MKB recovers the right\-ventricle blood pool while BiomedParse marks the surrounding myocardial wall instead; on\[Pathology\], MKB traces the glandular structures faithfully while BiomedParse fragments them; on\[Ultrasound\], MKB hits the correct lesion while BiomedParse mislocalises the tumour to an unrelated region of the frame\. On the remaining modalities \(\[OCT\],\[X\-Ray\],\[Dermoscopy\],\[Endoscopy\],\[Fundus\]\), both models produce masks that closely follow the annotation, with only minor boundary differences between the two\.

Overall, the qualitative comparison supports the quantitative trend of Table[10](https://arxiv.org/html/2607.20557#S5.T10): MKB matches or exceeds BiomedParse across the nine BiomedParse modalities, with the largest visible gains concentrated on targets whose extent requires instruction\-conditioned semantic reasoning rather than local texture cues alone\.

#### 5\.2\.4General Capability Results

Table[11](https://arxiv.org/html/2607.20557#S5.T11)compares MKB with its shared Transformer backbone, Qwen3\-VL\-8B\-Instruct, on general capability benchmarks\. Overall, MKB largely preserves the backbone’s general abilities after scientific multimodal training\. Across the 11 evaluated tasks, MKB matches or surpasses the backbone on 7 tasks and is slightly lower on the remaining 4\. Most differences are marginal and within seed\- or sampling\-level variation\. The only notable drop is observed on SArena\-Icon, where MKB scores 71\.49% compared with 74\.83% for Qwen3\-VL\-8B\-Instruct, possibly due to limited SVG\-oriented supervision in the current Stage\-2 mixture\. These results indicate that MKB retains the general\-purpose capabilities of its 8B VLM backbone while adding scientific multimodal understanding and generation abilities\.

## 6Conclusion

Existing scientific AI systems generally lack a unified capability to understand and generate heterogeneous scientific modalities, with most models remaining specialised to individual domains or limited to text\-centric interactions\. To address this gap, we introduce MKB, a unified scientific multimodal model that supports cross\-modal understanding, reasoning, and modality\-native generation across six scientific domains within a shared modelling framework\.

MKB combines modality\-tailored encoders, adapters, and decoders with a shared autoregressive backbone, allowing heterogeneous scientific inputs to be represented in structurally appropriate forms and jointly modelled within a shared representation space\. The results demonstrate the feasibility of this shared\-backbone paradigm, showing that a unified scientific model can remain competitive across diverse scientific tasks while largely preserving general\-purpose capabilities\.

Despite these promising results, performance remains less consistent on tasks requiring precise scalar prediction, particularly ADMET\-related endpoints\. This limitation suggests that unified scientific multimodal modelling remains an evolving direction, especially for tasks that require highly accurate quantitative prediction\.

## Author Contributions

Core Contributors Hesen Chen, Xinyu Su, Xiaomeng Yang, Yuetan Lin

Data Contributors Protein data: Zixiong Yang Molecule data: Junyi An, Fenglei Cao Scientific text: Yifeng Jiao RNA data: Yunqi Zhang, Yuan Cheng

Core Contributor Leads and Corresponding Authors Zhiyu Tan, Hao Li

Executive Sponsors Libo Wu, Yuan Qi

## References

- \[1\]J\. Alayrac, J\. Donahue, P\. Luc, A\. Miech, I\. Barr, Y\. Hasson, K\. Lenc, A\. Mensch, K\. Millican, M\. Reynolds,et al\.\(2022\)Flamingo: a visual language model for few\-shot learning\.Advances in neural information processing systems35,pp\. 23716–23736\.Cited by:[§2\.5](https://arxiv.org/html/2607.20557#S2.SS5.SSS0.Px2.p1.3)\.
- \[2\]J\. An, X\. Lu, Y\. Shi, L\. Xu, N\. Zhang, C\. Qu, Y\. Qi, and F\. Cao\(2026\)Suiren\-1\.0 technical report: a family of molecular foundation models\.arXiv preprint arXiv:2603\.21942\.Cited by:[§1](https://arxiv.org/html/2607.20557#S1.p2.1),[§2\.3\.2](https://arxiv.org/html/2607.20557#S2.SS3.SSS2.p3.4),[§3\.1](https://arxiv.org/html/2607.20557#S3.SS1.p1.1),[§5\.1\.2](https://arxiv.org/html/2607.20557#S5.SS1.SSS2.Px3.p1.1),[§5\.1\.4](https://arxiv.org/html/2607.20557#S5.SS1.SSS4.Px3.p2.1)\.
- \[3\]P\. Bansal, A\. Morgat, K\. B\. Axelsen, V\. Muthukrishnan, E\. Coudert, L\. Aimo, N\. Hyka\-Nouspikel, E\. Gasteiger, A\. Kerhornou, T\. B\. Neto, M\. Pozzato, M\. Blatter, A\. Ignatchenko, N\. Redaschi, and A\. Bridge\(2022\)Rhea, the reaction knowledgebase in 2022\.Nucleic Acids Research50\(D1\),pp\. D693–D700\.External Links:[Document](https://dx.doi.org/10.1093/nar/gkab1016)Cited by:[§3\.3](https://arxiv.org/html/2607.20557#S3.SS3.SSS0.Px1.p1.1)\.
- \[4\]K\. Bi, L\. Xie, H\. Zhang, X\. Chen, X\. Gu, and Q\. Tian\(2023\)Accurate medium\-range global weather forecasting with 3d neural networks\.Nature619\(7970\),pp\. 533–538\.Cited by:[§1](https://arxiv.org/html/2607.20557#S1.p2.1),[§5\.1\.4](https://arxiv.org/html/2607.20557#S5.SS1.SSS4.Px4.p1.1),[§5\.2\.3](https://arxiv.org/html/2607.20557#S5.SS2.SSS3.Px1.p1.6)\.
- \[5\]N\. Carion, L\. Gustafson, Y\. Hu, S\. Debnath, R\. Hu, D\. Suris, C\. Ryali, K\. V\. Alwala, H\. Khedr, A\. Huang,et al\.\(2025\)Sam 3: segment anything with concepts\.arXiv preprint arXiv:2511\.16719\.Cited by:[§1](https://arxiv.org/html/2607.20557#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.20557#S2.SS2.p3.1),[§2\.3\.4](https://arxiv.org/html/2607.20557#S2.SS3.SSS4.p3.2),[§2\.5](https://arxiv.org/html/2607.20557#S2.SS5.SSS0.Px4.p1.2),[§5\.1\.4](https://arxiv.org/html/2607.20557#S5.SS1.SSS4.Px5.p1.1)\.
- \[6\]L\. Chen, X\. Zhong, F\. Zhang, Y\. Cheng, Y\. Xu, Y\. Qi, and H\. Li\(2023\)FuXi: a cascade machine learning forecasting system for 15\-day global weather forecast\.npj climate and atmospheric science6\(1\),pp\. 190\.Cited by:[§1](https://arxiv.org/html/2607.20557#S1.p2.1)\.
- \[7\]H\. Dalla\-Torre, L\. Gonzalez, J\. Mendoza\-Revilla, N\. Lopez Carranza, A\. H\. Grzywaczewski, F\. Oteri, C\. Dallago, E\. Trop, B\. P\. De Almeida, H\. Sirelkhatim,et al\.\(2025\)Nucleotide transformer: building and evaluating robust foundation models for human genomics\.Nature Methods22\(2\),pp\. 287–297\.Cited by:[§1](https://arxiv.org/html/2607.20557#S1.p2.1)\.
- \[8\]C\. Edwards, T\. Lai, K\. Ros, G\. Honke, K\. Cho, and H\. Ji\(2022\)Translation between molecules and natural language\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Abu Dhabi, United Arab Emirates,pp\. 375–413\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.26)Cited by:[§5\.1\.2](https://arxiv.org/html/2607.20557#S5.SS1.SSS2.Px3.p1.1),[§5\.1\.4](https://arxiv.org/html/2607.20557#S5.SS1.SSS4.Px3.p4.1),[§5\.2\.2](https://arxiv.org/html/2607.20557#S5.SS2.SSS2.Px2.p1.5)\.
- \[9\]M\. K\. Gilson, T\. Liu, M\. Baitaluk, G\. Nicola, L\. Hwang, and J\. Chong\(2016\)BindingDB in 2015: a public database for medicinal chemistry, computational chemistry and systems pharmacology\.Nucleic Acids Research44\(D1\),pp\. D1045–D1053\.External Links:[Document](https://dx.doi.org/10.1093/nar/gkv1072)Cited by:[§3\.3](https://arxiv.org/html/2607.20557#S3.SS3.SSS0.Px2.p1.1)\.
- \[10\]T\. Haiden, M\. Janousek, F\. Vitart, Z\. Ben Bouallegue, L\. Ferranti, and F\. Prates\(2021\)Evaluation of ECMWF forecasts, including the 2021 upgrade\.Technical reportTechnical Report884,European Centre for Medium\-Range Weather Forecasts\.External Links:[Link](https://www.ecmwf.int/en/elibrary/81235-evaluation-ecmwf-forecasts-including-2021-upgrade)Cited by:[§1](https://arxiv.org/html/2607.20557#S1.p7.2)\.
- \[11\]T\. Haiden, M\. Janousek, F\. Vitart, Z\. B\. Bouallègue, L\. Ferranti, and F\. Prates\(2021\-09/2021\)Evaluation of ecmwf forecasts, including the 2021 upgrade\.ECMWF\(eng\)\.External Links:[Link](https://arxiv.org/html/2607.20557v1/&nbsp;),[Document](https://dx.doi.org/10.21957/90pgicjk4)Cited by:[§5\.1\.4](https://arxiv.org/html/2607.20557#S5.SS1.SSS4.Px4.p1.1)\.
- \[12\]J\. Hastings, G\. Owen, A\. Dekker, M\. Ennis, N\. Kale, V\. Muthukrishnan, S\. Turner, N\. Swainston, P\. Mendes, and C\. Steinbeck\(2016\)ChEBI in 2016: improved services and an expanding collection of metabolites\.Nucleic Acids Research44\(D1\),pp\. D1214–D1219\.External Links:[Document](https://dx.doi.org/10.1093/nar/gkv1031)Cited by:[§3\.3](https://arxiv.org/html/2607.20557#S3.SS3.SSS0.Px1.p1.1)\.
- \[13\]H\. He, Y\. Ren, Y\. Tang, Z\. Xu, J\. Li, M\. Yang, D\. Zhang, Y\. Dong, T\. Chen, S\. Zhang, Y\. Li, N\. Dong, W\. Ouyang, D\. Zhou, and P\. Ye\(2025\-11\)Biology\-instructions: a dataset and benchmark for multi\-omics sequence understanding capability of large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2025,Suzhou, China,pp\. 17984–18016\.Note:Preprint: arXiv:2412\.19191External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.978),[Link](https://aclanthology.org/2025.findings-emnlp.978/)Cited by:[§1](https://arxiv.org/html/2607.20557#S1.p3.1),[§3\.1](https://arxiv.org/html/2607.20557#S3.SS1.p1.1),[§3\.3](https://arxiv.org/html/2607.20557#S3.SS3.SSS0.Px2.p1.1),[§5\.1\.2](https://arxiv.org/html/2607.20557#S5.SS1.SSS2.Px1.p1.1),[§5\.1\.2](https://arxiv.org/html/2607.20557#S5.SS1.SSS2.Px2.p1.1),[§5\.1\.4](https://arxiv.org/html/2607.20557#S5.SS1.SSS4.Px2.p1.1)\.
- \[14\]H\. Hersbach, B\. Bell, P\. Berrisford, S\. Hirahara, A\. Horányi, J\. Muñoz\-Sabater, J\. Nicolas, C\. Peubey, R\. Radu, D\. Schepers, A\. Simmons, C\. Soci, S\. Abdalla, X\. Abellan, G\. Balsamo, P\. Bechtold, G\. Biavati, J\. Bidlot, M\. Bonavita, G\. De Chiara, P\. Dahlgren, D\. Dee, M\. Diamantakis, R\. Dragani, J\. Flemming, R\. Forbes, M\. Fuentes, A\. Geer, L\. Haimberger, S\. Healy, R\. J\. Hogan, E\. Hólm, M\. Janisková, S\. Keeley, P\. Laloyaux, P\. Lopez, C\. Lupu, G\. Radnoti, P\. de Rosnay, I\. Rozum, F\. Vamborg, S\. Villaume, and J\. Thépaut\(2020\)The ERA5 global reanalysis\.Quarterly Journal of the Royal Meteorological Society146\(730\),pp\. 1999–2049\.External Links:[Document](https://dx.doi.org/10.1002/qj.3803)Cited by:[§3\.1](https://arxiv.org/html/2607.20557#S3.SS1.p1.1),[§5\.1\.2](https://arxiv.org/html/2607.20557#S5.SS1.SSS2.Px4.p1.3)\.
- \[15\]K\. Huang, T\. Fu, W\. Gao, Y\. Zhao, Y\. Roohani, J\. Leskovec, C\. W\. Coley, C\. Xiao, J\. Sun, and M\. Zitnik\(2021\)Therapeutics data commons: machine learning datasets and tasks for drug discovery and development\.InAdvances in Neural Information Processing Systems \(NeurIPS\) Track on Datasets and Benchmarks,Cited by:[§3\.1](https://arxiv.org/html/2607.20557#S3.SS1.p1.1),[§5\.1\.2](https://arxiv.org/html/2607.20557#S5.SS1.SSS2.Px3.p1.1)\.
- \[16\]A\. Kirillov, E\. Mintun, N\. Ravi, H\. Mao, C\. Rolland, L\. Gustafson, T\. Xiao, S\. Whitehead, A\. C\. Berg, W\. Lo, P\. Dollár, and R\. Girshick\(2023\)Segment anything\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 3992–4003\.External Links:[Document](https://dx.doi.org/10.1109/ICCV51070.2023.00371)Cited by:[§5\.1\.4](https://arxiv.org/html/2607.20557#S5.SS1.SSS4.Px5.p1.1)\.
- \[17\]R\. Lam, A\. Sanchez\-Gonzalez, M\. Willson, P\. Wirnsberger, M\. Fortunato, F\. Alet, S\. Ravuri, T\. Ewalds, Z\. Eaton\-Rosen, W\. Hu,et al\.\(2023\)Learning skillful medium\-range global weather forecasting\.Science382\(6677\),pp\. 1416–1421\.Cited by:[§1](https://arxiv.org/html/2607.20557#S1.p2.1),[§5\.1\.4](https://arxiv.org/html/2607.20557#S5.SS1.SSS4.Px4.p1.1),[§5\.2\.3](https://arxiv.org/html/2607.20557#S5.SS2.SSS3.Px1.p1.6)\.
- \[18\]Z\. Lin, H\. Akin, R\. Rao, B\. Hie, Z\. Zhu, W\. Lu, N\. Smetanin, R\. Verkuil, O\. Kabeli, Y\. Shmueli,et al\.\(2023\)Evolutionary\-scale prediction of atomic\-level protein structure with a language model\.Science379\(6637\),pp\. 1123–1130\.Cited by:[§1](https://arxiv.org/html/2607.20557#S1.p2.1),[§2\.3\.1](https://arxiv.org/html/2607.20557#S2.SS3.SSS1.Px2.p1.4)\.
- \[19\]J\. Ma, Y\. He, F\. Li, L\. Han, C\. You, and B\. Wang\(2024\)Segment anything in medical images\.Nature communications15\(1\),pp\. 654\.Cited by:[§5\.1\.4](https://arxiv.org/html/2607.20557#S5.SS1.SSS4.Px5.p1.1)\.
- \[20\]J\. Meier, R\. Rao, R\. Verkuil, J\. Liu, T\. Sercu, and A\. Rives\(2021\)Language models enable zero\-shot prediction of the effects of mutations on protein function\.Advances in neural information processing systems34,pp\. 29287–29303\.Cited by:[§1](https://arxiv.org/html/2607.20557#S1.p2.1)\.
- \[21\]M\. Oquab, T\. Darcet, T\. Moutakanni, H\. Vo, M\. Szafraniec, V\. Khalidov, P\. Fernandez, D\. Haziza, F\. Massa, A\. El\-Nouby,et al\.\(2023\)Dinov2: learning robust visual features without supervision\.arXiv preprint arXiv:2304\.07193\.Cited by:[§5\.1\.4](https://arxiv.org/html/2607.20557#S5.SS1.SSS4.Px5.p1.1)\.
- \[22\]H\. Öztürk, A\. Özgür, and E\. Ozkirimli\(2018\)DeepDTA: deep drug–target binding affinity prediction\.Bioinformatics34\(17\),pp\. i821–i829\.External Links:[Document](https://dx.doi.org/10.1093/bioinformatics/bty593)Cited by:[§3\.3](https://arxiv.org/html/2607.20557#S3.SS3.SSS0.Px2.p1.1)\.
- \[23\]J\. Pathak, S\. Subramanian, P\. Harrington, S\. Raja, A\. Chattopadhyay, M\. Mardani, T\. Kurth, D\. Hall, Z\. Li, K\. Azizzadenesheli,et al\.\(2022\)FourCastNet: a global data\-driven high\-resolution weather model using adaptive Fourier neural operators\.arXiv preprint arXiv:2202\.11214\.Cited by:[§1](https://arxiv.org/html/2607.20557#S1.p2.1),[§5\.2\.3](https://arxiv.org/html/2607.20557#S5.SS2.SSS3.Px1.p1.6)\.
- \[24\]Qwen Team\(2025\)Qwen3\-VL technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[§1](https://arxiv.org/html/2607.20557#S1.p7.2),[§2\.1](https://arxiv.org/html/2607.20557#S2.SS1.p1.1),[§2\.4](https://arxiv.org/html/2607.20557#S2.SS4.p1.2),[§3\.1](https://arxiv.org/html/2607.20557#S3.SS1.p1.1),[§5\.1\.4](https://arxiv.org/html/2607.20557#S5.SS1.SSS4.Px1.p1.1)\.
- \[25\]Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang, T\. Li, M\. Ku, K\. Wang, A\. Zhuang, R\. Fan, X\. Yue, and W\. Chen\(2024\)MMLU\-Pro: a more robust and challenging multi\-task language understanding benchmark\.InAdvances in Neural Information Processing Systems 37 \(NeurIPS\) Track on Datasets and Benchmarks,Cited by:[§3\.5](https://arxiv.org/html/2607.20557#S3.SS5.p1.1),[§5\.1\.2](https://arxiv.org/html/2607.20557#S5.SS1.SSS2.Px6.p1.6)\.
- \[26\]Z\. Wu, B\. Ramsundar, E\. N\. Feinberg, J\. Gomes, C\. Geniesse, A\. S\. Pappu, K\. Leswing, and V\. Pande\(2018\)MoleculeNet: a benchmark for molecular machine learning\.Chemical Science9\(2\),pp\. 513–530\.External Links:[Document](https://dx.doi.org/10.1039/C7SC02664A)Cited by:[§5\.1\.2](https://arxiv.org/html/2607.20557#S5.SS1.SSS2.Px3.p1.1),[§5\.1\.4](https://arxiv.org/html/2607.20557#S5.SS1.SSS4.Px3.p4.1)\.
- \[27\]Y\. Xia, P\. Jin, S\. Xie, L\. He, C\. Cao, R\. Luo, G\. Liu, Y\. Wang, Z\. Liu, Y\. Chen,et al\.\(2025\)Nature language model: deciphering the language of nature for scientific discovery\.arXiv preprint arXiv:2502\.07527\.Cited by:[§1](https://arxiv.org/html/2607.20557#S1.p3.1)\.
- \[28\]B\. Yu, F\. N\. Baker, Z\. Chen, X\. Ning, and H\. Sun\(2024\)LlaSMol: advancing large language models for chemistry with a large\-scale, comprehensive, high\-quality instruction tuning dataset\.InConference on Language Modeling \(COLM\),Note:Introduces the SMolInstruct dataset; preprint arXiv:2402\.09391Cited by:[§3\.1](https://arxiv.org/html/2607.20557#S3.SS1.p1.1),[§5\.1\.2](https://arxiv.org/html/2607.20557#S5.SS1.SSS2.Px3.p1.1),[§5\.1\.4](https://arxiv.org/html/2607.20557#S5.SS1.SSS4.Px3.p4.1)\.
- \[29\]X\. Yue, T\. Zheng, Y\. Ni, Y\. Wang, K\. Zhang, S\. Tong, Y\. Sun, B\. Yu, G\. Zhang, H\. Sun, Y\. Su, W\. Chen, and G\. Neubig\(2025\)MMMU\-Pro: a more robust multi\-discipline multimodal understanding benchmark\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(ACL\),Note:Preprint: arXiv:2409\.02813Cited by:[§3\.5](https://arxiv.org/html/2607.20557#S3.SS5.p1.1),[§5\.1\.2](https://arxiv.org/html/2607.20557#S5.SS1.SSS2.Px6.p1.6)\.
- \[30\]T\. Zhao, Y\. Gu, J\. Yang, N\. Usuyama, H\. H\. Lee, S\. Kiblawi, T\. Naumann, J\. Gao, A\. Crabtree, J\. Abel, C\. Moung\-Wen, B\. Piening, C\. Bifulco, M\. Wei, H\. Poon, and S\. Wang\(2025\)A foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalities\.Nature Methods22\(1\),pp\. 166–176\.External Links:[Document](https://dx.doi.org/10.1038/s41592-024-02499-w)Cited by:[§1](https://arxiv.org/html/2607.20557#S1.p2.1),[§3\.1](https://arxiv.org/html/2607.20557#S3.SS1.p1.1),[§5\.1\.2](https://arxiv.org/html/2607.20557#S5.SS1.SSS2.Px5.p1.2),[§5\.1\.4](https://arxiv.org/html/2607.20557#S5.SS1.SSS4.Px5.p1.1)\.
- \[31\]G\. Zhou, Z\. Gao, Q\. Ding, H\. Zheng, H\. Xu, Z\. Wei, L\. Zhang, and G\. Ke\(2022\)Uni\-Mol: a universal 3d molecular representation learning framework\.ChemRxiv\.External Links:[Document](https://dx.doi.org/10.26434/chemrxiv-2022-jjm0j-v4)Cited by:[§5\.1\.4](https://arxiv.org/html/2607.20557#S5.SS1.SSS4.Px3.p4.1)\.
- \[32\]Z\. Zhou, Y\. Ji, W\. Li, P\. Dutta, R\. Davuluri, and H\. Liu\(2024\)DNABERT\-2: efficient foundation model and benchmark for multi\-species genome\.InThe Twelfth International Conference on Learning Representations \(ICLR\),External Links:2306\.15006,[Link](https://openreview.net/forum?id=oMLQB4EZE1)Cited by:[§1](https://arxiv.org/html/2607.20557#S1.p2.1)\.
- \[33\]Y\. Zou, D\. Zhu, L\. Zhu, T\. Zhu, Y\. Zhou, P\. Zhou, X\. Zhou, D\. Zhou, Z\. Zhou, Y\. Zhou,et al\.\(2026\)Intern\-s1\-pro: scientific multimodal foundation model at trillion scale\.arXiv preprint arXiv:2603\.25040\.Cited by:[§1](https://arxiv.org/html/2607.20557#S1.p3.1),[§1](https://arxiv.org/html/2607.20557#S1.p7.2)\.