MT-ProtBERT: Multi-task Learning ProtBERT for Intrinsically Disordered Proteins Classification with Scarce Data

# Multi-task Learning ProtBERT for Intrinsically Disordered Proteins Classification with Scarce Data
Source: [https://arxiv.org/html/2609.25334](https://arxiv.org/html/2609.25334)
## MT\-ProtBERT: Multi\-task Learning ProtBERT for Intrinsically Disordered Proteins Classification with Scarce DataThanks:Corresponding Author: Mohammad H\. Mahoor\.Thanks:Jian Sun is with the Department of Computer Science, Ritchie School of Engineering and Computer Science, University of Denver, 2155 E Wesley Ave, Denver, CO 80210, also with the Department of Surgery, David Geffen School of Medicine at the University of California, Los Angeles, 10833 Le Conte Ave, Los Angeles, CA 90095\. \(Email: JianSun@mednet\.ucla\.edu; Home Page: https://jiansfoggy\.github\.io/sj\-home\.github\.io//\)Thanks:Mohammad H\. Mahoor is with the Department of Computer Science, Ritchie School of Engineering and Computer Science, University of Denver, 2155 E Wesley Ave, Denver, CO 80210\. \(Email: mohammad\.mahoor@du\.edu; Home Page: http://mohammadmahoor\.com\)Thanks:Lilianna Houston is in the Department of Physics and Astronomy, at the University of Denver, CO 80210Thanks:Kingshuk Ghosh is in the Department of Physics and Astronomy, at the University of Denver, CO 80210 and also affiliated with the Molecular and Cellular Biophysics program at the University of Denver

###### Abstract

Intrinsically disordered proteins \(IDPs\) differ from folded proteins in that they are dynamic, lack a stable three\-dimensional conformation, and have low sequence similarity between similar proteins\. The conformational heterogeneity of IDPs — while beneficial for their diverse functions — limits the use of traditional experimental tools to determine their conformation\. The experimental difficulty, along with low sequence similarity, results indata scarcity, and makes it difficult to classify/detect IDPs that are similar or dissimilar, a task relevant to understand biology and evolution\. We address this challenge using Multi\-task ProtBERT \(MT\-ProtBERT\), a multi\-task extension of ProtBERT tailored for low\-data regimes\. MT\-ProtBERT integrates Dynamic Window Masking, a Multi\-Scale 1D Convolutional classifier \(MS\-Conv1D\), and auxiliary objectives that jointly optimize masked language modeling and biochemistry\-informed tasks\. We evaluate this framework on two tasks under limited data: \(i\) phosphorylation site prediction \(S/T/Y\) in short sequences and small datasets, and \(ii\) protein compaction prediction on two small datasets \(684 and 530 sequences\), including sequences comparable in length to typical disordered regions\. MT\-ProtBERT consistently outperforms PARROT, an RNN\-based IDP\-specific model, across all tasks\. These results demonstrate that combining self\-supervised and biochemistry\-informed tasks, and multi\-scale learning enables robust modeling of unstructured proteins under data scarcity\.

###### Index Terms:

Intrinsically Disordered Proteins \(IDPs\), Data Scarcity, Biochemistry\-informed Neural Network, Predicting Phosphorylation and Compaction in Peptides, Multi\-task Learning

## IIntroduction

Detecting functionally similar or dissimilar proteins is fundamental in biology and biotechnology\. Functional classification is typically done by aligning the structure or sequence of an unknown protein with that of a protein with known function\[[1](https://arxiv.org/html/2609.25334#bib.bib39)\]\. This approach has been largely successful for proteins that have a unique folded structure\. However, intrinsically disordered proteins \(IDPs\) are not amenable to this for two reasons\. First, IDPs do not possess a stable three\-dimensional structure\. Instead, IDPs exist in an ensemble, continuously interconverting between multiple conformations\. Second, functionally similar IDPs often share little sequence similarity\[[2](https://arxiv.org/html/2609.25334#bib.bib2),[3](https://arxiv.org/html/2609.25334#bib.bib3)\]\.

Besides the classification challenge, IDPs have an additional problem of data scarcity\. While conformational heterogeneity facilitates diverse biological functions\[[4](https://arxiv.org/html/2609.25334#bib.bib6),[5](https://arxiv.org/html/2609.25334#bib.bib7),[6](https://arxiv.org/html/2609.25334#bib.bib8),[7](https://arxiv.org/html/2609.25334#bib.bib9),[8](https://arxiv.org/html/2609.25334#bib.bib10),[9](https://arxiv.org/html/2609.25334#bib.bib11),[10](https://arxiv.org/html/2609.25334#bib.bib12)\], it also presents a challenge in experimentally determining IDP conformations\. Experimentally determined structures for IDPs are far fewer compared to their folded counterparts\. The lack of experimental measurements and low sequence homology, together results indata scarcity\. IDPs constitute 30–40% of the human proteome\[[11](https://arxiv.org/html/2609.25334#bib.bib4),[12](https://arxiv.org/html/2609.25334#bib.bib5)\], yet the PED database contains conformations of only about 400 IDPs from purely biochemical measurements \(or in combination with simulations\[[13](https://arxiv.org/html/2609.25334#bib.bib20)\]\), compared to 250,000 folded protein structures in the PDB\.

To mitigate this issue, coarse\-grained simulations generate large\-scale datasets\[[14](https://arxiv.org/html/2609.25334#bib.bib14),[15](https://arxiv.org/html/2609.25334#bib.bib15)\]which are subsequently used to train machine learning models\[[16](https://arxiv.org/html/2609.25334#bib.bib16),[17](https://arxiv.org/html/2609.25334#bib.bib17),[18](https://arxiv.org/html/2609.25334#bib.bib18),[19](https://arxiv.org/html/2609.25334#bib.bib40)\]\. These physics\-based models can predict IDP conformations and liquid\-liquid phase separation \(LLPS\) propensity\[[17](https://arxiv.org/html/2609.25334#bib.bib17),[20](https://arxiv.org/html/2609.25334#bib.bib19)\], an important function of IDPs\. Nonetheless, training directly from biochemical measurements remains challenging\. Additionally, the trained models do not necessarily help with functional classification other than LLPS\.

A major effort in the functional classification of IDPs so far has focused on unsupervised learning\. This has been supported by the emerging view that IDP sequence features correlate with conserved biophysical properties\. Statistical and physics\-based metrics—such as composition, patterning, and chain dimensions—have been used to predict functional similarity\[[21](https://arxiv.org/html/2609.25334#bib.bib21),[22](https://arxiv.org/html/2609.25334#bib.bib22),[15](https://arxiv.org/html/2609.25334#bib.bib15),[2](https://arxiv.org/html/2609.25334#bib.bib2),[23](https://arxiv.org/html/2609.25334#bib.bib23),[24](https://arxiv.org/html/2609.25334#bib.bib24),[25](https://arxiv.org/html/2609.25334#bib.bib25),[26](https://arxiv.org/html/2609.25334#bib.bib26)\]\.

Little attention has been paid to supervised classification when IDPs have known labels, particularly in small\-sample regimes \(hundreds of sequences\)\. In this paper we embrace the general problem of IDP classification in the small sample regime, by considering two binary classification tasks across four experiments on relatively small protein sequences and datasets to exacerbate thedata scarcityissue\. In the first experiment \(Experiment 1\), we consider whether a given sequence will undergo phosphorylation \(Task 1\)\. Briefly, phosphorylation is a biochemical process in which the amino acids S/T/Y are modified to a negatively charged group\. Phosphorylation can change the function of proteins and is biologically relevant\[[27](https://arxiv.org/html/2609.25334#bib.bib27),[28](https://arxiv.org/html/2609.25334#bib.bib28)\]\. However, not all S/T/Y amino acids are phosphorylated, thus, phosphorylation prediction is a binary task\. We consider two datasets PPA and PELM\[[29](https://arxiv.org/html/2609.25334#bib.bib70),[30](https://arxiv.org/html/2609.25334#bib.bib71)\]of peptides \(short sequences 19 amino acids long\) with a middle amino acid being S/T/Y, target site for phosphorylation\. The datasets are small, for example, PPA sequences \(with the central amino acid Y\) include only about 1,100 samples, evenly split between phosphorylated and non\-phosphorylated cases\.

In addition to predicting phosphorylation, we created an additional classification task \(Task 2\) of predicting protein compactness\. Ensemble average end\-to\-end distance is a measure of global dimension of IDPs\. This dimension is compared against a reference size \(of an ideal protein with no interaction\) to determine compact \(below the reference\) or expanded \(above the reference\) groups\. This sets another binary classification task for IDPs\. We test this task in Experiment 2 using a selected subset of sequences from the PELM set\. Additional experiments \(Experiments 3 and 4\) predicting compaction \(Task 2\) were created by selecting slightly larger sequences \(30 amino acids long\) to better mimic disordered proteins from a database different from PELM\[[31](https://arxiv.org/html/2609.25334#bib.bib29)\]and with sample size even smaller than Experiment 2\.

Existing supervised IDP models primarily use CNNs\[[31](https://arxiv.org/html/2609.25334#bib.bib29),[32](https://arxiv.org/html/2609.25334#bib.bib74),[33](https://arxiv.org/html/2609.25334#bib.bib77),[34](https://arxiv.org/html/2609.25334#bib.bib79)\], RNNs\[[35](https://arxiv.org/html/2609.25334#bib.bib31)\], LSTM\[[36](https://arxiv.org/html/2609.25334#bib.bib75)\], or CNN\-attention hybrids\[[37](https://arxiv.org/html/2609.25334#bib.bib76),[38](https://arxiv.org/html/2609.25334#bib.bib78)\]\. Phosphorylation prediction has not been explored using purely transformer\-based architectures\. In contrast, folded\-protein studies increasingly adopt large language models\[[39](https://arxiv.org/html/2609.25334#bib.bib13),[40](https://arxiv.org/html/2609.25334#bib.bib1)\]\. Inspired by the analogy between protein sequences and natural language, BERT and GNN models have been applied to protein tasks\[[41](https://arxiv.org/html/2609.25334#bib.bib32),[42](https://arxiv.org/html/2609.25334#bib.bib33),[43](https://arxiv.org/html/2609.25334#bib.bib34),[44](https://arxiv.org/html/2609.25334#bib.bib35),[45](https://arxiv.org/html/2609.25334#bib.bib41)\], yet BERT\-based approaches for IDPs remain limited\[[46](https://arxiv.org/html/2609.25334#bib.bib30)\]\. We therefore adopt ProtBERT\[[47](https://arxiv.org/html/2609.25334#bib.bib59)\]for IDP classification\.

To addressdata scarcity, we proposeMulti\-task ProtBERT \(MT\-ProtBERT\)with three components: \(i\) Dynamic Window Masking, which randomly masks contiguous residues of varying lengths for data augmentation; \(ii\) a Multi\-scale 1D Convolutional classifier \(MS\-Conv1D\) with parallel kernels to capture multi\-scale features; and \(iii\) Multi\-task Learning, integrating Masked Language Modeling \(MLM\) and S/T\-P motif prediction as auxiliary tasks\. MLM enhances structural representation, while S/T\-P prediction embeds phosphorylation\-related biochemical knowledge \(Task 1\)\.

These modules improve robustness and outperform the RNN\-based PARROT model\[[35](https://arxiv.org/html/2609.25334#bib.bib31)\]\.In summary, this work addresses binary classification of IDPs under data scarcity\.The main contributions are as follows:

- •We proposeMT\-ProtBERT, a multi\-task ProtBERT framework for improved classification of disordered protein sequences\.
- •We prove thatmulti\-view MLMoutperforms single\-view MLM and design aProtein\-aware Lossto further enhance MLM performance\.
- •We introduceDynamic Window Maskingfor sequence augmentation and aMulti\-scale 1D Convolutional classifier \(MS\-Conv1D\)to capture multi\-scale contextual features\.
- •We employMulti\-task Learning\(MLM \+ S/T\-P prediction\) to embed structural and biochemical priors, mitigating data scarcity\.
- •MT\-ProtBERT consistently outperforms PARROT, an RNN\-based model for IDPs\.

The remainder of this paper is organized as follows\. Section[II](https://arxiv.org/html/2609.25334#S2)reviews related work\. Section[III](https://arxiv.org/html/2609.25334#S3)presents the MT\-ProtBERT architecture\. Section[IV](https://arxiv.org/html/2609.25334#S4)describes datasets, experimental settings, and results\. Section[V](https://arxiv.org/html/2609.25334#S5)discusses implications and limitations\. Section[VI](https://arxiv.org/html/2609.25334#S6)concludes the paper\.

## IIRelated Works

This section reviews machine learning approaches for protein biophysical prediction and discusses strategies for handling data scarcity\.

### II\-AMachine learning models to predict biophysical properties of proteins

Protein property prediction depends ondata representation,embedding, andfeature extraction\.

Data Representation\.Researchers represent proteins as sequences, graphs, or secondary structures\. Graph\-based methods encode residue interactions and geometry\[[48](https://arxiv.org/html/2609.25334#bib.bib42),[39](https://arxiv.org/html/2609.25334#bib.bib13),[44](https://arxiv.org/html/2609.25334#bib.bib35)\], sometimes transforming secondary structures into graphs\[[43](https://arxiv.org/html/2609.25334#bib.bib34),[49](https://arxiv.org/html/2609.25334#bib.bib43),[39](https://arxiv.org/html/2609.25334#bib.bib13)\]\. Secondary structure provides geometric constraints; for example, Yuanet al\.\[[39](https://arxiv.org/html/2609.25334#bib.bib13)\]derived structures using AlphaFold2, and Chandraet al\.\[[42](https://arxiv.org/html/2609.25334#bib.bib33)\]embedded structural annotations into sequences\.

However, the disordered peptides studied here lack reliable secondary structure annotations\. We therefore adopt raw sequence representation and enhance it through masking, which increases diversity without adding structural assumptions\.

Embedding\.Standard NLP embeddings do not fully capture protein\-specific semantics\. Protein\-tailored embeddings such asdoc2vec,Word2Vec, andSeqVecaddress this gap\[[50](https://arxiv.org/html/2609.25334#bib.bib44),[51](https://arxiv.org/html/2609.25334#bib.bib45),[52](https://arxiv.org/html/2609.25334#bib.bib46),[53](https://arxiv.org/html/2609.25334#bib.bib47),[54](https://arxiv.org/html/2609.25334#bib.bib48)\]\. Embedding strategies include residue\-level andkk\-mer\-level representations\[[51](https://arxiv.org/html/2609.25334#bib.bib45),[52](https://arxiv.org/html/2609.25334#bib.bib46),[53](https://arxiv.org/html/2609.25334#bib.bib47)\]\. Other methods encode biochemical properties \(ESM, ESM1b, PAAC\)\[[50](https://arxiv.org/html/2609.25334#bib.bib44),[55](https://arxiv.org/html/2609.25334#bib.bib38)\]or structural context \(Protein Graphs, Distance Maps, DSSP, RaptorX\)\[[39](https://arxiv.org/html/2609.25334#bib.bib13),[56](https://arxiv.org/html/2609.25334#bib.bib37)\]\.

Our datasets lack structural annotations, and ProtBERT uses residue\-level tokenization\. We therefore embed amino acids individually to maintain task compatibility\.

Feature Extraction\.Protein sequences contain local motifs and long\-range dependencies\[[57](https://arxiv.org/html/2609.25334#bib.bib36),[56](https://arxiv.org/html/2609.25334#bib.bib37),[48](https://arxiv.org/html/2609.25334#bib.bib42)\]\. CNNs capture local patterns, while attention\-based Transformers model global context\[[48](https://arxiv.org/html/2609.25334#bib.bib42),[57](https://arxiv.org/html/2609.25334#bib.bib36),[41](https://arxiv.org/html/2609.25334#bib.bib32)\]\. RNN\-based models such as PARROT may not capture long\-range dependencies\[[35](https://arxiv.org/html/2609.25334#bib.bib31)\]\.

We employ ProtBERT for global modeling and integrate a multi\-scale Conv1D classifier to enrich local feature extraction\. This hybrid design balances contextual representation and structural sensitivity\.

Summary\.Rather than introducing new representations, we focus on extracting richer information from raw sequences while preserving interpretability and architectural simplicity\.

### II\-BData Scarcity

Data scarcity arises from limited labeled samples and short sequences \(19 residues in Experiments 1–2; 30 residues in Experiment 3\), which constrain model capacity\.

Common solutions include Transfer Learning\[[58](https://arxiv.org/html/2609.25334#bib.bib49),[59](https://arxiv.org/html/2609.25334#bib.bib50)\], Semi\-supervised Learning\[[60](https://arxiv.org/html/2609.25334#bib.bib51)\], Few\-shot Learning\[[61](https://arxiv.org/html/2609.25334#bib.bib52)\], Autoencoders\[[62](https://arxiv.org/html/2609.25334#bib.bib53)\], Bayesian Methods\[[63](https://arxiv.org/html/2609.25334#bib.bib54)\], Data Augmentation\[[61](https://arxiv.org/html/2609.25334#bib.bib52),[62](https://arxiv.org/html/2609.25334#bib.bib53)\], Self\-Supervised Learning\[[64](https://arxiv.org/html/2609.25334#bib.bib55)\], Physics\-informed Modeling\[[65](https://arxiv.org/html/2609.25334#bib.bib56)\], Multi\-task Learning\[[65](https://arxiv.org/html/2609.25334#bib.bib56),[66](https://arxiv.org/html/2609.25334#bib.bib57)\], and Multi\-scale Learning\[[67](https://arxiv.org/html/2609.25334#bib.bib58)\]\.

However, Transfer learning requires large dataset to prepare pretraining model\. Semi\-supervised and autoencoder approaches risk noisy pseudo\-labels or posterior collapse\. Few\-shot and Bayesian methods increase computational cost\.

Data augmentation, physics\-informed modeling, self\-supervised learning, multi\-task learning, and multi\-scale learning better align with our setting\. Data augmentation broadens the training distribution; physics\-informed constraints inject domain knowledge; self\-supervision extracts intrinsic sequence patterns; multi\-task learning provides auxiliary guidance; and multi\-scale learning captures features at different resolutions\.

Guided by these principles, we design two auxiliary tasks:Masked Token Prediction \(MLM\)andS/T\-P Prediction\. MLM introduces self\-supervision through masking, while S/T\-P prediction encodes a phosphorylation\-related motif at the sequence center\. For downstream classification, we integrate a multi\-scale Conv1D classifier to enhance global and local representation learning\.

This combination of self\-supervised, multi\-task, and multi\-scale learning forms our strategy to address data scarcity in IDP classification\.

## IIIMethodology

### III\-AMulti\-task ProtBERT

![Refer to caption](https://arxiv.org/html/2609.25334v1/Figures/S3_1_MT-ProtBERT.png)Fig\. 1:Overall architecture of MT\-ProtBERT\. The input sequence is augmented twice via Dynamic Window Masking to generatet1t\_\{1\}andt2t\_\{2\}, which are encoded by ProtBERT into embeddingsf1f\_\{1\}andf2f\_\{2\}\. Three heads are applied: two MS\-Conv1D classifiers and one MLM head\. MS\-Conv1D1and MS\-Conv1D2usef1f\_\{1\}to predict the protein label \(yy, Task 1/2\) and the S/T\-P feature, respectively, while the MLM head reconstructs masked amino acidsA​A1AA\_\{1\}andA​A2AA\_\{2\}fromf1f\_\{1\}andf2f\_\{2\}\. These joint objectives enable MT\-ProtBERT to learn sequence semantics and physical priors under data scarcity\.Figure[1](https://arxiv.org/html/2609.25334#S3.F1)shows MT\-ProtBERT’s pipeline\. We apply Dynamic Window Masking twice to each input sequence to produce two masked views, encode both with a shared ProtBERT backbone, and obtain embeddingsf1f\_\{1\}andf2f\_\{2\}\. We attach three heads: two MS\-Conv1D classifiers for the main protein label and an auxiliary S/T\-P or SCD label, and one MLM head that predicts masked tokens from bothf1f\_\{1\}andf2f\_\{2\}\. Performing two MLM predictions per forward pass strengthens token\-level representation learning\. Below we describe each component\.

### III\-BDynamic Window Masking

Dynamic Window Masking produces context\-aware augmented sequences\. For a sequence of lengthLL, we pick a random start positionPPand maskkkconsecutive residues wherek∈\{1,2,3\}k\\in\\\{1,2,3\\\}with probabilities\{0\.6,0\.3,0\.1\}\\\{0\.6,0\.3,0\.1\\\}\. The expected window size equals

E⁡\[k\]=0\.6×1\+0\.3×2\+0\.1×3=1\.5\.\\displaystyle E\[k\]=0\.6\\times 1\+0\.3\\times 2\+0\.1\\times 3=1\.5\.\(1\)We mask multiple non\-overlapping regions to reach a total masking ratio of 20%\. This ratio balances information retention and diversity for short sequences \(e\.g\.,L=19L=19\)\. The number of masked regions follows

NM​a​s​k=⌈20%×L⌉/E⁡\[k\]≈⌈20%×L⌉/1\.5\.\\displaystyle\\begin\{split\}N\_\{Mask\}&=\\lceil 20\\%\\times L\\rceil/E\[k\]\\\\ &\\approx\\lceil 20\\%\\times L\\rceil/1\.5\.\\end\{split\}\(2\)ForL=19L=19,NM​a​s​k≈2\.67N\_\{Mask\}\\approx 2\.67, so we generate 2–3 masked regions\. We repeat this masking twice to producet1t\_\{1\}andt2t\_\{2\}, providing multiple contextual views that increase training diversity, lower gradient variance, and speed convergence\.

### III\-CProtBERT Backbone

ProtBERT\[[47](https://arxiv.org/html/2609.25334#bib.bib59)\]uses amino acids as tokens and pre\-trains on UniRef\[[68](https://arxiv.org/html/2609.25334#bib.bib60)\]\. It generalizes well to protein tasks \(e\.g\., secondary structure, localization\)\. We use ProtBERT as a shared encoder fort1t\_\{1\}andt2t\_\{2\}and obtain contextual embeddingsf1f\_\{1\}andf2f\_\{2\}of size\[batch size,L,768\]\[\\text\{batch size\},L,768\]\. These embeddings capture sequence semantics for downstream heads\.

### III\-DMulti\-scale 1D Convolutional Classifier \(MS\-Conv1D\)

![Refer to caption](https://arxiv.org/html/2609.25334v1/Figures/S3_4_MS-Conv1D.png)Fig\. 2:Architecture of the Multi\-scale 1D Convolutional Classifier \(MS\-Conv1D\)\. Given an input embeddingf1f\_\{1\}of size \[batch size, embedding length = 768\], three parallel 1D convolutional layers with kernel sizes of 2, 3, and 4 are used to extract local and global context features at multiple scales\. Each branch outputs a feature of size \[batch size, 128\], which are concatenated into \[batch size, 128\*3 = 384\] and passed through a fully connected layer to produce final prediction scoresyyor S/TP of size \[batch size, 2\]\. This design enhances feature richness and improves robustness under limited data\.Figure[2](https://arxiv.org/html/2609.25334#S3.F2)depicts MS\-Conv1D\. We apply three parallel 1D convolutions with kernel sizes 2, 3, and 4 to ProtBERT embeddings\. Each branch captures features at a different receptive field; we concatenate their outputs and feed them to a fully connected layer to predict either the main label or an auxiliary label \(S/T\-P or SCD\)\. This multi\-scale, multi\-branch design captures complementary local contexts and enhances global structural modeling, similar to MC\-ViViT\[[69](https://arxiv.org/html/2609.25334#bib.bib61)\]and SSL\-V3\[[70](https://arxiv.org/html/2609.25334#bib.bib62)\]\.

### III\-EMulti\-task Learning

We train with auxiliary tasks to reinforce the main classification objective\. Experiments 1–3 use MLM \+ S/T\-P; Experiment 4 uses MLM \+ SCD\.

Masked Language Modeling \(MLM\)\.The MLM head contains two fully connected layers: a projection fromf1f\_\{1\}to vocabulary space \(fM​L​M​1f\_\{MLM1\}\) and a decoder producing logitsfM​L​M​2∈\[batch size,L,V\]f\_\{MLM2\}\\in\[\\text\{batch size\},L,V\], whereV=30V=30\(20 amino acids \+ special tokens\)\. We predict masked amino acidsA​A1AA\_\{1\}fromf1f\_\{1\}andA​A2AA\_\{2\}fromf2f\_\{2\}\. Multi\-view MLM backpropagates twice per pass and enforces token\-level consistency\. Eq\.[3](https://arxiv.org/html/2609.25334#S3.E3)shows multi\-view MLM lossℒmulti\\mathcal\{L\}\_\{\\text\{multi\}\}does not exceed single\-view lossℒsingle\\mathcal\{L\}\_\{\\text\{single\}\}\(proof in Appendix[B](https://arxiv.org/html/2609.25334#A2)\)\.

ℒsingle=𝔼M∼ℳ​𝔼X∼DWM​\(S,M\)\[−log⁡pθ​\(A​Am\|hθ​\(X\)\)\]ℒmulti=𝔼M∼ℳ​𝔼X1,⋯,XK​∼i​i​d​DWM​\(S,M\)\[−log⁡\(1K​∑i=1Kpθ​\(A​Am\|hθ​\(Xi\)\)\)\]ℒmulti≤ℒsingle\\displaystyle\\begin\{split\}\\mathcal\{L\}\_\{\\text\{single\}\}&=\\mathbb\{E\}\_\{M\\sim\\mathcal\{M\}\}\\mathbb\{E\}\_\{X\\sim\\text\{DWM\}\(S,M\)\}\\\\ &\\\>\[\-\\log p\_\{\\theta\}\(AA\_\{m\}\|h\_\{\\theta\}\(X\)\)\]\\\\ \\mathcal\{L\}\_\{\\text\{multi\}\}&=\\mathbb\{E\}\_\{M\\sim\\mathcal\{M\}\}\\mathbb\{E\}\_\{X\_\{1\},\\dotsm,X\_\{K\}\\overset\{iid\}\{\\sim\}\\text\{DWM\}\(S,M\)\}\\\\ &\\\>\[\-\\log\(\\frac\{1\}\{K\}\\displaystyle\\sum^\{K\}\_\{i=1\}p\_\{\\theta\}\(AA\_\{m\}\|h\_\{\\theta\}\(X\_\{i\}\)\)\)\]\\\\ \\mathcal\{L\}\_\{\\text\{multi\}\}&\\leq\\mathcal\{L\}\_\{\\text\{single\}\}\\end\{split\}\(3\)
S/T\-P Prediction\.We predict whether the central residue forms \(S or T\)–P\. The MS\-Conv1D processesf1f\_\{1\}and outputsfSTP∈\[batch size,2\]f\_\{\\text\{STP\}\}\\in\[\\text\{batch size\},2\]\. This auxiliary task injects phosphorylation\-related biochemical priors and helps the model learn motif\-level structure\. We keep S/T\-P prediction in experiments where it does not directly apply \(e\.g\., central Y\) to preserve a consistent training scheme\.

SCD\-based Binary Classification\.SCD correlates with IDP compaction\[[71](https://arxiv.org/html/2609.25334#bib.bib63),[72](https://arxiv.org/html/2609.25334#bib.bib64),[73](https://arxiv.org/html/2609.25334#bib.bib65),[74](https://arxiv.org/html/2609.25334#bib.bib66),[75](https://arxiv.org/html/2609.25334#bib.bib67),[76](https://arxiv.org/html/2609.25334#bib.bib68)\]\. We binarize SCD at threshold−1\.06\-1\.06: set SCD=1 if SCD\>−1\.06\>\-1\.06, else 0\. Experiment 4 replaces S/T\-P with SCD prediction while keeping other settings identical, testing robustness with both relevant and irrelevant auxiliary tasks\.

### III\-FLoss Function

#### III\-F1Protein\-aware Loss Function for MLM

Each amino acid belongs to a chemical equivalence group following classification introduced by Pappu\[[77](https://arxiv.org/html/2609.25334#bib.bib69)\]\(Table[I](https://arxiv.org/html/2609.25334#S3.T1)\)\. We define a soft accuracy factorq⁡\(a​a\|xi\)q\(aa\|x\_\{i\}\)that rewards exact matches and allows partial credit for chemically equivalent substitutions\. Letϵ\\epsiloncontrol the soft weight; we setϵ=0\.2\\epsilon=0\.2\. Formally:

TABLE I:Chemical Equivalence Category for 20 AAs\.Polar residues \(μ\\mathbf\{\\mu\}\)S,T,N,Q,C,HHydrophobic residues \(h\)I,L,M,VBasic residues \(\+\)R,KAcidic residues \(–\)E,DAromatic residues \(π\\mathbf\{\\pi\}\)F,W,YAlanine \(A\)AProline \(P\)PGlycine \(G\)G

TABLE II:Symbols appeared in the Protein\-aware loss function for MLM\.q⁡\(a​a\|xi\)\\displaystyle q\(aa\|x\_\{i\}\)=\{1−ϵa​a=xiϵ\|G⁡\(xi\)\|−1a​a∈G⁡\(xi\),a​a∉xi0a​a∉G⁡\(xi\)\\displaystyle=\\begin\{cases\}1\-\\epsilon&aa=x\_\{i\}\\\\ \\frac\{\\epsilon\}\{\|G\(x\_\{i\}\)\|\-1\}&aa\\in G\(x\_\{i\}\),aa\\notin x\_\{i\}\\\\ 0&aa\\notin G\(x\_\{i\}\)\\end\{cases\}\(4\)ℒi\\displaystyle\\mathcal\{L\}\_\{i\}=−∑a​a∈A​Aq\(aa\|xi\)logpθ\(aa\|x~\)\\displaystyle=\-\\displaystyle\\sum\\limits\_\{aa\\in AA\}q\(aa\|x\_\{i\}\)\\log p\_\{\\theta\}\(aa\|\\tilde\{x\}\)\(5\)ℒmlm\\displaystyle\\mathcal\{L\}\_\{\\text\{mlm\}\}=1\|ℳ\|​∑i∈ℳℒi\\displaystyle=\\frac\{1\}\{\|\\mathcal\{M\}\|\}\\displaystyle\\sum\\limits\_\{i\\in\\mathcal\{M\}\}\\mathcal\{L\}\_\{i\}\(6\)For example, ifxi=Dx\_\{i\}=\\text\{D\}andG⁡\(xi\)=\{D,E\}G\(x\_\{i\}\)=\\\{\\text\{D\},\\text\{E\}\\\}, thenq⁡\(D\|D\)=0\.8q\(\\text\{D\}\|\\text\{D\}\)=0\.8andq⁡\(E\|D\)=0\.2q\(\\text\{E\}\|\\text\{D\}\)=0\.2\. We embedq⁡\(a​a\|xi\)q\(aa\|x\_\{i\}\)into the cross\-entropy loss to form the Protein\-aware Loss for MLM\.

#### III\-F2Overall Loss

We combine MLM, auxiliary, and main losses\. Letℒmlm1\\mathcal\{L\}\_\{\\text\{mlm1\}\},ℒmlm2\\mathcal\{L\}\_\{\\text\{mlm2\}\}denote MLM losses fort1t\_\{1\}andt2t\_\{2\},ℒstp\\mathcal\{L\}\_\{\\text\{stp\}\}denote the auxiliary loss, andℒmain\\mathcal\{L\}\_\{\\text\{main\}\}denote the main classification loss\. We define:

ℒMLM=0\.5×\(ℒmlm1\+ℒmlm2\)ℒ=ℒM​L​M\+0\.1×\(ℒstp\+ℒmain\)\.\\begin\{split\}\\mathcal\{L\}\_\{\\text\{MLM\}\}&=0\.5\\times\(\\mathcal\{L\}\_\{\\text\{mlm1\}\}\+\\mathcal\{L\}\_\{\\text\{mlm2\}\}\)\\\\ \\mathcal\{L\}&=\\mathcal\{L\}\_\{MLM\}\+0\.1\\times\(\\mathcal\{L\}\_\{\\text\{stp\}\}\+\\mathcal\{L\}\_\{\\text\{main\}\}\)\.\\end\{split\}\(7\)We scale auxiliary and main losses by 0\.1 to align magnitudes\. We emphasize MLM during early training to learn structural and semantic regularities; after epoch 25, MLM primarily stabilizes token\-level representations\.

Combining Dynamic Window Masking, ProtBERT, MS\-Conv1D, and multi\-task objectives lets MT\-ProtBERT learn physicochemical and structural patterns from limited data and improve IDP classification\.

## IVExperiment

This section presents the datasets, implementation details, evaluation metrics, experimental results, and analyses used to assess the effectiveness of the proposed MT\-ProtBERT model\.

### IV\-ADataset

We evaluate MT\-ProtBERT on two benchmark phosphorylation datasets:PPAandPELM\.PPAoriginates from the PhosphAt database and contains phosphorylation sites exclusively from Arabidopsis thaliana\[[29](https://arxiv.org/html/2609.25334#bib.bib70)\]\.PELM, created by Dinkelet al\., is derived from the Phospho\.ELM database and includes experimentally verified phosphorylation sites in various animal species such as Homo sapiens and Mus musculus\[[30](https://arxiv.org/html/2609.25334#bib.bib71)\]\.

TABLE III:Example samples from PPA and PELM datasets\. Each row corresponds to a 19\-length protein sequence with its central amino acid \(AA\) and label indicating phosphorylation \(Active\) or no\-phosphorylation \(Non\-active\)\. These examples illustrate the short\-sequence and balanced\-label characteristics of both datasets\.Table[III](https://arxiv.org/html/2609.25334#S4.T3)lists representative sequence samples from both datasets\. Each sequence is 19 amino acids long, centered on a target residue \(S, T, or Y\)\. The label indicates whether the site is phosphorylation\-active \(Active\) or non\-active \(Non\-active\)\.

Both PPA and PELM contain balanced positive and negative classes \(28,001 samples each\), as summarized in Table[IV](https://arxiv.org/html/2609.25334#S4.T4)\. However, subsets with central residue T or Y contain far fewer samples than those with S, resulting in a pronounced data scarcity problem\. For example, PPA with central Y has only 1,156 samples\. This is a motivation to choose the dataset to test models in the low sample regime\. This scarcity, combined with short sequence lengths, poses a challenge for deep learning models, which motivates our multi\-task and multi\-scale learning strategy\.

TABLE IV:Statistical summary of PPA and PELM datasets\. AA denotes the central amino acid type in each sequence\. While both datasets are label\-balanced overall, subsets with AA=T or Y contain far fewer samples, revealing significant data scarcity that challenges model training\.To evaluate MT\-ProtBERT on a different task and small datasets, we design Experiments 2–4 for protein compaction prediction\. We quantify compaction using the ratiox=Re​e2/Re​e,F​R​C2x=R\_\{ee\}^\{2\}/R\_\{ee,FRC\}^\{2\}, whereRe​e=⟨re​e2⟩1/2R\_\{ee\}=\\langle r\_\{ee\}^\{2\}\\rangle^\{1/2\}is the ensemble\-averaged end\-to\-end distance andRe​e,F​R​CR\_\{ee,FRC\}corresponds to a Flory Random Coil \(FRC\), an ideal non\-interacting reference chain\.x\>1x\>1indicates expansion andx<1x<1indicates collapse, yielding a binary classification\. A previously developed physics\-based machine learning model – verified against experimental data – was used to predictxxdirectly from sequence\[[16](https://arxiv.org/html/2609.25334#bib.bib16)\]\.

PELM\-684 \(Experiment 2\)\.We construct a balanced subset from PELM\. Sequences with0\.90<x<0\.980\.90<x<0\.98form the compact class, excluding values near 1 to avoid ambiguous cases\. We then sample sequences with1\.02<x<1\.101\.02<x<1\.10to match this distribution as the expanded class\. The final PELM\-684 dataset contains 684 sequences evenly split between compact and expanded classes\.

AD\-530 \(Experiments 3 and 4\)\.We derive a second balanced dataset from 30\-residue sequences generated by Erijman et al\.\[[31](https://arxiv.org/html/2609.25334#bib.bib29)\], originally designed to study activation domains in yeast\. These sequence lengths better reflect intrinsically disordered regions\. Sequences withx<0\.99x<0\.99define the compact class, and sequences withx\>1\.01x\>1\.01define the expanded class\. After matching class distributions, the AD\-530 dataset contains 530 sequences with equal class sizes\. Experiment 3 evaluates compaction prediction on this unseen dataset\. Experiment 4 uses the same AD\-530 set but replaces the S/T\-P auxiliary task with SCD\-based binary classification \(Section[III\-D](https://arxiv.org/html/2609.25334#S3.SS4)\)\.

### IV\-BImplementation Details

All models were implemented in PyTorch\. The input amino acid sequences were first tokenized and embedded using the ProtBERT tokenizer\. We used an 9:1 split for training and testing, with no overlap between subsets\. Training was conducted on an NVIDIA RTX 3090 Ti GPU using the AdamW optimizer with an initial learning rate of1×10−61\\times 10^\{\-6\}and batch size of 32\. The model was trained for 30 epochs with a cyclic learning rate scheduler \(triangular2 mode\)\. The backbone parameters of ProtBERT were fine\-tuned jointly with the multi\-task heads\.

Hyper\-parameters for Dynamic Window Masking were empirically set to mask 20% of residues per sequence, with a window length expectation ofE⁡\[k\]=1\.5E\[k\]=1\.5\. All other configurations followed the same setup across both datasets\.

### IV\-CMetrics

We evaluate model performance using six commonly adopted metrics for binary classification in bioinformatics:Accuracy,F1 Score,AUPRC\(Area Under the Precision–Recall Curve\),Sensitivity,Specificity,MCC\(Matthews Correlation Coefficient\),PLL\(Pseudo\-Log\-Likelihood\),MRR\(Mean Reciprocal Rank\)\. Among them, MCC provides a balanced evaluation even under class imbalance, and AUPRC is particularly informative for tasks with limited positive samples\. PLL measures the average log\-likelihood of the ground\-truth amino acids over all masked positions, ranging\(−∞,0\]\(\-\\infty,0\]\. MRR emphasizes how early the correct amino acid appears in the ranked list, ranging\(0,1\]\(0,1\]\. Higher PLL and MRR indicate better MLM performance\.

### IV\-DProtein Phosphorylation Prediction

TABLE V:Comparison of phosphorylation prediction results between MT\-ProtBERT and existing methods on PPA and PELM datasets\. Our model achieves superior performance across most subsets, particularly on data\-scarce groups such as PPA \(AA=Y and AA = T\), demonstrating its robustness and effectiveness under limited data\.Table[V](https://arxiv.org/html/2609.25334#S4.T5)compares MT\-ProtBERT with PARROT and MusiteDeep on PPA and PELM\. For subsets with central residue S, MT\-ProtBERT achieves accuracy comparable to or slightly higher than existing methods\. On smaller subsets, performance gains become substantial\. For PPA with central residue T \(2,722 sequences\) and Y \(1,156 sequences\), MT\-ProtBERT improves mean accuracy by at least 6\.6% and 9\.6% over PARROT and MusiteDeep, respectively\. It also achieves the highest MCC across all subsets\.

MT\-ProtBERT generally outperforms competing methods on most metrics with the exception of AUPRC \(higher in PARROT on PPA\) and Sensitivity \(higher in MusiteDeep on PELM\)\. However, MusiteDeep’s high Sensitivity accompanies low Specificity, indicating biased predictions\. Similarly, PARROT’s higher AUPRC often coincides with reduced Sensitivity and/or Specificity\. These results show that integrating Dynamic Window Masking, MS\-Conv1D, and multi\-task learning improves performance while maintaining balanced metrics under data scarcity\.

Several other phosphorylation predictors exist\[[80](https://arxiv.org/html/2609.25334#bib.bib73),[32](https://arxiv.org/html/2609.25334#bib.bib74),[36](https://arxiv.org/html/2609.25334#bib.bib75),[37](https://arxiv.org/html/2609.25334#bib.bib76),[33](https://arxiv.org/html/2609.25334#bib.bib77),[38](https://arxiv.org/html/2609.25334#bib.bib78),[81](https://arxiv.org/html/2609.25334#bib.bib81)\]\. However, most use different datasets, sequence lengths, or incomplete evaluation metrics, preventing direct comparison\. CapsNet PTM\[[32](https://arxiv.org/html/2609.25334#bib.bib74)\], PhosIDN\[[37](https://arxiv.org/html/2609.25334#bib.bib76)\], PhosTransfer\[[33](https://arxiv.org/html/2609.25334#bib.bib77)\], and EMBER\[[38](https://arxiv.org/html/2609.25334#bib.bib78)\]employ datasets distinct from PPA/PELM and different input lengths, and some do not report accuracy or MCC\. TransPhos\[[80](https://arxiv.org/html/2609.25334#bib.bib73)\], DeepPPSIte\[[36](https://arxiv.org/html/2609.25334#bib.bib75)\], and Limited Protein DNN\[[34](https://arxiv.org/html/2609.25334#bib.bib79)\]use PPA and PELM, but TransPhos and DeepPPSIte rely on large datasets, while Limited Protein DNN targets relatively small samples with sequences of only 9 residues—much shorter than typical disordered regions and the 19\-residue sequences used in this study\.

### IV\-EProtein Compaction Prediction

To evaluate generalization beyond phosphorylation, we introduced protein compaction prediction as an additional task\. Experiment 2 uses the small PELM\-684 dataset \(Section[IV\-A](https://arxiv.org/html/2609.25334#S4.SS1)\)\. Experiments 3 and 4 use the AD\-530 dataset, which contains 30\-residue sequences outside PPA and PELM\. Experiment 3 tests performance on sequences of different length comparable to intrinsically disordered regions\. We compare MT\-ProtBERT with PARROT and its optimized variant \(Table[VI](https://arxiv.org/html/2609.25334#S4.T6)\)\. In Experiments 2 and 3, MT\-ProtBERT consistently surpasses PARROT in distinguishing compact and expanded sequences, improving accuracy by up to 13% and substantially increasing MCC\.

In Experiment 4, we replace S/T\-P with a different auxiliary task\. We convert sequence charge decoration \(SCD\) into a binary classification task using a threshold \(see section 3\.5\)\. With SCD supervision, MT\-ProtBERT achieves performance comparable to Experiment 3 and remains superior to PARROT\.

These results show that MT\-ProtBERT generalizes beyond phosphorylation and does not depend on S/T\-P as the only auxiliary task\. This also shows the framework can be readily used to other biophysically grounded objectives\.

TABLE VI:Results of protein compaction prediction from different experiments and comparison between MT\-ProtBERT and PARROT\. MT\-ProtBERT consistently achieves the highest mean Accuracy and MCC\.
### IV\-FMLM, S/T\-P, and SCD Prediction

TABLE VII:Results of the auxiliary tasks besides main Task \(1 or 2\): Masked Language Modeling \(MLM\) and the binary task \(S/T\-P or SCD classification\) Prediction\. MT\-ProtBERT achieves stable soft Top\-1 and Top\-5 accuracies in MLM, near\-perfect performance in S/T\-P task, and stable SCD Prediction, confirming that the model effectively learns sequence structure and biochemical properties while enhancing main\-task performance\.Masked Language Modeling \(MLM\)\.Table[VII](https://arxiv.org/html/2609.25334#S4.T7)reports MLM performance\. MT\-ProtBERT achieves higher Top\-1 and Top\-5 accuracy on PELM than on PPA, reflecting the effect of sample size\. Top\-1 accuracy ranges from 46\.58%–52\.51% on PELM and 40\.49%–49\.69% on PPA\. All PLL values remain below \-2, and MRR stays between 45% and 58\.5%, indicating stable training\.

Within PPA, accuracy and MRR follow S\>\>T\>\>Y, consistent with subset size; PLL shows the same order in both PPA and PELM\. Larger datasets therefore improve masked residue recovery and strengthen structural priors\. Top\-5 accuracy substantially exceeds Top\-1, showing that the model often ranks the correct residue among its top candidates even when it misses the top prediction\.

In compaction prediction \(Experiment 2\), MLM achieves 50\.45% soft Top\-1 accuracy\. When the dataset shrinks from 684 to 530 samples \(Experiments 3 and 4\), Top\-1 accuracy drops to 28\.86% and 28\.66%\. Thus, task type, dataset distribution, and sample size directly influence auxiliary\-task performance\.

S/T\-P Prediction\.S/T\-P prediction encodes a biologically relevant phosphorylation motif \(e\.g\., CDK\-recognized S/T\-P\)\. MT\-ProtBERT achieves near\-perfect accuracy in identifying whether the central residue is S/T and the adjacent residue is P, confirming that the model captures biochemical motifs\. For compaction prediction \(Experiments 2–3\), S/T\-P accuracy reaches 100%, indicating that this auxiliary task is relatively simple\.

SCD\-based Binary Prediction\.Experiment 4 replaces S/T\-P with SCD\-based binary classification\. The auxiliary accuracy reaches 72\.07%, lower than S/T\-P in Experiments 2–3\. However, Experiment 4 improves compaction prediction accuracy by 3% over Experiment 3\. Because SCD directly relates to charge patterning and compaction, this auxiliary task aligns more closely with the main objective and yields stronger synergy\.

### IV\-GAblation Study

![Refer to caption](https://arxiv.org/html/2609.25334v1/Figures/S4_7_tsk1_exp1_abl_plot.png)Fig\. 3:Ablation study results on Task 1 in PPA and PELM datasets \(Experiment 1\)\. Each bar chart compares model variants by Accuracy and MCC, demonstrating the effectiveness of the proposed Dynamic Window Masking, MS\-Conv1D, and multi\-task learning\. MT\-ProtBERT consistently achieves the best balance between accuracy and correlation, confirming the contribution of each module\.We evaluated each module by selectively removing or modifying components\. Figures[3](https://arxiv.org/html/2609.25334#S4.F3)–[6](https://arxiv.org/html/2609.25334#S4.F6)and Tables[VIII](https://arxiv.org/html/2609.25334#A1.T8)–[XI](https://arxiv.org/html/2609.25334#A1.T11)\(Appendix[A](https://arxiv.org/html/2609.25334#A1)\) summarize the results\.

Multi\-position Dynamic Window Masking\.

Multi\-position masking places non\-overlapping masks at several locations instead of a single site, exposing the model to diverse local contexts\. In Task 1 \(Experiment 1\), it improves accuracy and MCC by up to 3\.5% and 7\.6%, respectively, on small subsets such as PPA \(AA=Y\), with similar trends across other subsets\. In Task 2 \(Experiments 2–4\), multi\-position masking consistently outperforms single\-position masking on small datasets \(see Figures[4](https://arxiv.org/html/2609.25334#S4.F4),[5](https://arxiv.org/html/2609.25334#S4.F5)and[6](https://arxiv.org/html/2609.25334#S4.F6)\)\. Multiple masked views, therefore, enhance robustness under limited data\.

![Refer to caption](https://arxiv.org/html/2609.25334v1/Figures/S4_7_tsk2_exp2_abl_plot.png)Fig\. 4:Ablation study results for Experiment 2\. Each bar chart compares model variants by Accuracy and MCC, demonstrating the effectiveness of the proposed Dynamic Window Masking, MS\-Conv1D, and multi\-task learning\. MT\-ProtBERT consistently achieves the best balance between accuracy and correlation, confirming the contribution of each module\.MS\-Conv1D\.

Replacing MS\-Conv1D with single\-scale convolution degrades performance across subsets\. In Task 1, MCC drops by at least 1\.1% for PPA \(AA=T\) and PELM \(AA=Y\)\. Gains in MCC generally exceed those in accuracy, and Task 2 shows the same pattern \(see Figures[4](https://arxiv.org/html/2609.25334#S4.F4),[5](https://arxiv.org/html/2609.25334#S4.F5), and[6](https://arxiv.org/html/2609.25334#S4.F6)\)\. Multi\-scale convolution captures complementary spatial dependencies and supports hierarchical protein feature learning\.

MLM Objective\.Adding MLM consistently improves both accuracy and MCC, with larger gains in small datasets\. For example, in Task 1 \(PELM, AA=Y\), MLM increases accuracy by 0\.29%\. MCC gains reach 5\.3% \(PPA, AA=Y\) and 2\.9% \(PELM, AA=T\)\. In Experiment 2, MLM improves accuracy by 2\.7% and MCC by 7\.0%\. These results show that MLM strengthens contextual encoding and token\-level semantic consistency\.

![Refer to caption](https://arxiv.org/html/2609.25334v1/Figures/S4_7_tsk2_exp3_abl_plot.png)Fig\. 5:Ablation study results for Experiment 3\. Each bar chart compares model variants by Accuracy and MCC, demonstrating the effectiveness of the proposed Dynamic Window Masking, MS\-Conv1D, and multi\-task learning\. MT\-ProtBERT consistently achieves the best balance between accuracy and correlation, confirming the contribution of each module\.S/T\-P or SCD Prediction\.

Adding S/T\-P prediction yields modest but consistent gains, especially in MCC for small subsets\. In Task 1 \(PPA and PELM, AA=Y\), MCC increases by 7\.5% and 0\.2%, while accuracy improves by 0\.3–4\.4%\. In Experiments 2–3, accuracy improves by 1\.2–2\.6% and MCC by 1\.9–5\.4%\. This auxiliary task guides the model toward biochemically meaningful representations\.

In Experiment 4, SCD prediction produces larger gains—6\.4% in accuracy and 13\.8% in MCC—indicating stronger relevance to protein compaction than S/T\-P prediction \(Figure[6](https://arxiv.org/html/2609.25334#S4.F6)\)\.

Multi\-task Learning\.

Combining MLM and S/T\-P under the multi\-task framework yields clear gains\. In Experiment 1, full MT\-ProtBERT improves accuracy by 0\.4–4% and MCC by 0\.5–7% compared with removing both MLM and S/T\-P, except for PELM \(AA=S\), where performance remains similar\. In protein compaction prediction \(Experiments 2–4\), multi\-task learning improves accuracy by 4–7% and MCC by 8\.5–15\.8%\. These results confirm that structural \(MLM\) and biochemical \(S/T\-P\) priors act synergistically\.

Multi\-view MLM\.

Multi\-view MLM consistently improves accuracy and MCC across experiments, except for PELM \(AA=S\), where performance remains near\-optimal\. Overall, multi\-view MLM plays a critical role in MT\-ProtBERT\.

Protein\-aware Loss Function\.

Protein\-aware Loss enables best or near\-best performance in Task 1, including nearly 2% accuracy improvement on PPA \(AA=Y\)\. In Task 2, gains exceed 4% in Experiments 3 and 4\. These results justify integrating Protein\-aware Loss into MT\-ProtBERT\.

![Refer to caption](https://arxiv.org/html/2609.25334v1/Figures/S4_7_tsk2_exp4_abl_plot.png)Fig\. 6:Ablation study results for Experiment 4 \(Task 2 with binary SCD prediction for the third dataset\)\. Each bar chart compares model variants by Accuracy and MCC, demonstrating the effectiveness of the proposed Dynamic Window Masking, MS\-Conv1D, and multi\-task learning\. MT\-ProtBERT consistently achieves the best balance between accuracy and correlation, confirming the contribution of each module\.
### IV\-HData Analysis and Summary

The ablation study shows that Dynamic Window Masking, MS\-Conv1D, and Multi\-task Learning consistently improve accuracy and MCC, especially on small datasets\. ProtBERT performs well on large subsets \(e\.g\., PELM, central residue S, 31K samples\) but degrades sharply in low\-data settings such as PPA \(central residue Y\)\.

In contrast, MT\-ProtBERT maintains stable and strong performance across all subsets, demonstrating robustness to limited data and short sequences\. It achieves a favorable balance between accuracy and model complexity, making it well\-suited for phosphorylation and compaction prediction under realistic small\-sample constraints\.

## VDiscussion

This study proposes MT\-ProtBERT, a multi\-task framework for predicting phosphorylation \(Task 1\) and compaction \(Task 2\) of IDPs when data is scarce\. By integrating self\-supervised and protein\-informed objectives, MT\-ProtBERT consistently outperforms state\-of\-the\-art baselines such as PARROT\. On the most data\-scarce subset \(PPA, central residue Y, 1,156 samples\), it achieves 65\.57% accuracy and 31\.42% MCC in Task 1, exceeding other models by nearly or over 10%\. Strong performance is also observed in Task 2\. These results indicate that the proposed modules enable ProtBERT to learn more generalizable and physically meaningful sequence representations for diverse IDP properties\.

### V\-AMulti\-task Learning Benefits

The performance gains stem from synergistic auxiliary tasks\. Masked Language Modeling \(MLM\) captures sequence structure and contextual dependencies, while multi\-view MLM strengthens supervision and feature extraction\. The S/T\-P prediction task embeds biochemical priors by emphasizing phosphorylation motifs \(e\.g\., S/T followed by P\)\. Together, these complementary signals regularize the backbone, reduce overfitting, and enhance robustness in low\-data regimes\.

However, improvements over ProtBERT diminish on large datasets \(e\.g\., PELM, AA=S\), where sufficient supervision allows full task specialization\. Auxiliary constraints mainly benefit small\-sample settings; thus, MT\-ProtBERT is best viewed as a robustness\-enhancing architecture for data\-scarce scenarios rather than a universal replacement in high\-resource cases\.

### V\-BRole of Dynamic Window Masking and MS\-Conv1D

Dynamic Window Masking augments training by masking multiple random regions, increasing contextual diversity without external data synthesis\. The Multi\-scale 1D Convolutional \(MS\-Conv1D\) classifier extracts hierarchical features across different receptive fields, capturing both local motifs and long\-range dependencies consistent with protein structural hierarchy\.

### V\-CImpact of Data Volume and Distribution

Model performance strongly correlates with sample size\. ProtBERT performs best on large subsets \(e\.g\., PELM, central residue S, 31K samples, 77\.45% accuracy\) but degrades markedly in small datasets \(e\.g\., PPA, Y\)\. MT\-ProtBERT mitigates this drop by leveraging auxiliary objectives to extract additional structural information, compensating for limited labeled data\.

### V\-DImpact of Protein\-aware Loss function

The Protein\-aware Loss incorporates chemical equivalence categories \(Table[I](https://arxiv.org/html/2609.25334#S3.T1)\), embedding biochemical knowledge into optimization\. Compared with the “No Soft Acc” setting, it yields up to 10% accuracy improvements \(Figures[3](https://arxiv.org/html/2609.25334#S4.F3),[4](https://arxiv.org/html/2609.25334#S4.F4),[5](https://arxiv.org/html/2609.25334#S4.F5), and[6](https://arxiv.org/html/2609.25334#S4.F6)\), leading to more biologically consistent predictions\.

### V\-EModel Efficiency and Feasibility

MT\-ProtBERT maintains architectural simplicity and computational efficiency while outperforming prior methods\. It generalizes across datasets without external handcrafted features or large\-scale fine\-tuning \(Table[V](https://arxiv.org/html/2609.25334#S4.T5)\), demonstrating the adaptability of BERT\-based models in IDP research and providing a template for other molecular tasks with limited data\.

### V\-FAddressing Data Scarcity through Self\-supervision

Data scarcity is alleviated through intrinsic self\-supervision \(MLM\) and biochemical auxiliary tasks \(S/T\-P prediction\)\. Rather than relying solely on annotations, the model exploits sequence structure and residue relationships as implicit supervision\. Experiment 4 further supports this strategy by incorporating a physics\-based metric \(SCD\) as an auxiliary task for compaction prediction, improving generalization in small datasets\.

### V\-GLimitations and Future Directions

Three limitations remain\. First, MLM Top\-1 accuracy is relatively low due to short sequence length \(19 residues\), limiting effective masking and contextual reconstruction\. Second, S/T\-P prediction rapidly saturates \(100% accuracy\), indicating coarse supervision\. Future work could incorporate richer physical descriptors such as hydrophobicity, solvent\-accessible surface area, torsional angles, or conformation\-related metrics \(e\.g\., SCD\[[71](https://arxiv.org/html/2609.25334#bib.bib63)\], SCDM, SHDM\[[74](https://arxiv.org/html/2609.25334#bib.bib66)\]\)\. Contrastive learning or Conditional Variational Autoencoders \(CVAE\) may further enhance representation diversity\. Third, current experiments focus on equal\-length sequences \(≤\\leq30 residues\), whereas functionally similar IDPs may vary in length\. Extending MT\-ProtBERT to handle variable and longer sequences is therefore essential\.

### V\-HSummary

MT\-ProtBERT offers an effective and interpretable framework for IDP classification when data is scarce\. By integrating self\-supervised, multi\-scale, and biology\-informed learning, it bridges sequence modeling and biochemical priors, providing methodological insights for data\-efficient deep learning in molecular biophysics\.

## VIConclusion

In this work, we developedMT\-ProtBERT, a multi\-task learning framework for disordered protein classification in the limit of small sample size\. The model integrates several complementary modules—including Dynamic Window Masking, ProtBERT backbone, MS\-Conv1D, and two auxiliary heads for MLM and another sequence\-based \(S/T\-P or SCD\) classification task\. Through this design, MT\-ProtBERT effectively captures both global biochemical patterns and local structural motifs, achieving strong generalization even in short and sparse protein sequences such as PPA and PELM\.

### VI\-AFuture Work

Building upon MT\-ProtBERT, future research will explore several new directions\. First, we plan to design aSelf\-Adaptive ProtBERTcapable of dynamically adjusting its learning behavior based on data sufficiency—performing robustly across both data\-abundant and data\-scarce regimes\. For abundant samples, the Self\-Adaptive model should only use the main task, while for the data scarce case, it would add auxiliary tasks\. Second, to better learn sequence structure, we aim to replace MLM task byConditional Variational Autoencoder \(CVAE\), reconstructing token embeddings from ProtBERT outputs\. Third, we will investigate contrastive learning strategies to alleviate posterior collapse in CVAE\-based architectures and to formally establish the mathematical link between latent regularization and representation stability\[[21](https://arxiv.org/html/2609.25334#bib.bib21),[22](https://arxiv.org/html/2609.25334#bib.bib22),[15](https://arxiv.org/html/2609.25334#bib.bib15),[2](https://arxiv.org/html/2609.25334#bib.bib2),[23](https://arxiv.org/html/2609.25334#bib.bib23),[24](https://arxiv.org/html/2609.25334#bib.bib24),[25](https://arxiv.org/html/2609.25334#bib.bib25),[74](https://arxiv.org/html/2609.25334#bib.bib66),[26](https://arxiv.org/html/2609.25334#bib.bib26)\]\.

### VI\-BBroader Impact

Overall, MT\-ProtBERT demonstrates that coupling multi\-task learning, and self\-supervised objectives offers a scalable path forward for datasets with small sample size, typical in modeling IDPs\. Beyond phosphorylation site prediction, the principles established here can generalize to a wide range of molecular sequence modeling tasks—such as protein–protein interaction, detailed modeling of IDP conformation ensemble, and mutation impact analysis—where labeled data are often limited\. By combining interpretability, efficiency, and adaptability, MT\-ProtBERT contributes to the broader goal of developing robust, data\-efficient deep learning systems for computational biology and protein science\.

## Acknowledgments

We acknowledge support from NIH R01GM138901\.

## References

- \[1\]P\. Radivojac, W\. T\. Clark, T\. R\. Oron, A\. M\. Schnoes, T\. Wittkop, A\. Sokolov, K\. Graim, C\. Funk, K\. Verspoor, A\. Ben\-Hur,et al\.\(2013\)A large\-scale evaluation of computational protein function prediction\.Nature methods10\(3\),pp\. 221–227\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p1.1)\.
- \[2\]T\. Zarin, C\. N\. Tsai, A\. N\. N\. Ba, and A\. M\. Moses\(2017\)Selection maintains signaling function of a highly diverged intrinsically disordered region\.Proc\. Natl\. Acad\. Sci\.114\(8\),pp\. E1450–E1459\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p1.1),[§I](https://arxiv.org/html/2609.25334#S1.p4.1),[§VI\-A](https://arxiv.org/html/2609.25334#S6.SS1.p1.1)\.
- \[3\]J\. Lange, L\. S\. Wyrwicz, and G\. Vriend\(2016\)KMAD: knowledge\-based multiple sequence alignment for intrinsically disordered proteins\.Bioinformatics32,pp\. 932–936\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p1.1)\.
- \[4\]T\. Mittag, L\. E\. Kay, and J\. D\. Forman\-Kay\(2010\)Protein dynamics and conformational disorder in molecular recognition\.J\. Molecular Recognition23\(2\),pp\. 105\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p2.1)\.
- \[5\]P\. E\. Wright and H\. J\. Dyson\(2015\)Intrinsically disordered proteins in cellular signaling and regulation\.Nat Rev Mol Cell Biol16\(1\),pp\. 18–29\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p2.1)\.
- \[6\]H\.J\. Dyson and P\.E\. Wright\(2016\)Role of intrinsic protein disorder in the function and interactions of the transcriptional coactivators creb\-binding protein \(cbp\) and p300\.Journal of Biological Chemistry13\(\),pp\. 6714–6722\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p2.1)\.
- \[7\]S\. Brodsky, T\. Jana, and N\. Barkai\(2021\)Order through disorder: the role of intrinsically disordered regions in transcription factor binding specificity\.Curr\. Opin\. Struct\. Biol\.\(\),pp\. 110–115\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p2.1)\.
- \[8\]H\. Xie, S\. Vucetic, L\. M\. Iakoucheva, C\. J\. Oldfield, A\. Dunker, V\. Uversky, and Z\. Obradovic\(2007\)Functional anthology of intrinsic disorder\. 1\. biological processes and functions of proteins with long disordered regions\.J\. Proteome Res\.6\(5\),pp\. 1882–98\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p2.1)\.
- \[9\]P\. Tompa\(2012\)Intrinsically disordered proteins: a 10\-year recap\.Trends in Biochemical Sciences37\(12\),pp\. 509–516\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p2.1)\.
- \[10\]C\. P\. Brangwynne, C\. R\. Eckmann, D\. Courson, A\. Rybarska, C\. Hoege, J\. Gharakhani, F\. Jullicher, and A\. A\. Hyman\(2009\)Germline p granules are liquid droplets that localize by controlled dissolution/condensation\.Science324\(5935\),pp\. 1729–32\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p2.1)\.
- \[11\]A\. K\. Dunker, C\. J\. Brown, J\. D\. Lawson, L\. M\. Iakoucheva, and Z\. Obradovic\(2002\)Intrinsic disorder and protein function\.Biochemistry41\(\),pp\. 6573\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p2.1)\.
- \[12\]R\. Van Der Lee, M\. Buljan, B\. Lang, R\. J\. Weatheritt, G\. W\. Daughdrill, A\. K\. Dunker, M\. Fuxreiter, J\. Gough, J\. Gsponer, D\. T\. Jones,et al\.\(2014\)Classification of intrinsically disordered regions and proteins\.Chemical reviews114\(13\),pp\. 6589–6631\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p2.1)\.
- \[13\]et\. al\. Lazar\(2021\)PED in 2021: a major update of the protein ensemble database for intrinsically disordered proteins\.Nucleic Acids Research49\(\),pp\. D404–D411\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p2.1)\.
- \[14\]W\. Zheng, G\. Dignon, M\. Brown, Y\. C\. Kim, and J\. Mittal\(2020\)Hydropathy patterning complements charge patterning to describe conformational preferences of disordered proteins\.J\. Phys\. Chem\. Lett\.11\(9\),pp\. 3408–3415\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p3.1)\.
- \[15\]G\. Tessei, A\. Trolle, A\. Jonsson, J\. Betz, F\.E\. Knudsen, F\. Pesce, K\. Johansson, and K\. Lindorff\-Larsen\(2024\)Conformational ensembles of the human intrinsically disordered proteome\.Nature626\(\),pp\. 897–904\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p3.1),[§I](https://arxiv.org/html/2609.25334#S1.p4.1),[§VI\-A](https://arxiv.org/html/2609.25334#S6.SS1.p1.1)\.
- \[16\]L\. Houston, M\. Phillips, A\. Torres, K\. Gaalswyk, and K\. Ghosh\(2024\)Physics\-based machine learning trains hamiltonians and decodes the sequence–conformation relation in the disordered proteome\.Journal of Chemical Theory and Computation20\(22\),pp\. 10266–10274\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p3.1),[§IV\-A](https://arxiv.org/html/2609.25334#S4.SS1.p4.1)\.
- \[17\]G\. Tesei, T\.K\. Schulze, C\. R\., and L\. K\.\(2021\)Accurate model of liquid–liquid phase behavior of intrinsically disordered proteins from optimization of single\-chain properties\.Proceedings of the National Academy of Sciences118\(44\),pp\. e2111696118\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p3.1)\.
- \[18\]J\.A\. Joseph, A\. Reinhardt, A\. Aguirre, P\.Y\. Chew, K\.O\. Russell, J\.R\. Espinosa, A\. Garaizar, and R\. Collepardo\-Guevara\(2021\)Physics\-driven coarse\-grained model for biomolecular phase separation with near\-quantitative accuracy\.Nature computational science1\(\),pp\. 732–743\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p3.1)\.
- \[19\]B\. Novak, J\. M\. Lotthammer, R\. J\. Emenecker, and A\. S\. Holehouse\(2026\)Accurate predictions of disordered protein ensembles with starling\.Nature,pp\. 1–11\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p3.1)\.
- \[20\]S\. Bulow, G\. Tessei, F\.K\. Zaidi, T\. Mittag, and K\. Lindorff\-Larsen\(2025\)Prediction of phase separation propensities of disordered proteins from sequence\.Proc\. Natl\. Acad\. Sci\.122\(13\),pp\. e2417920122\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p3.1)\.
- \[21\]M\. Cohan, K\. Ruff, and R\. Pappu\(2019\)Information theoretic measures for quantifying sequence\-ensemble relationships of intrinsically disordered proteins\.Protein Engineering, Design, and Selection\(\),pp\. 10\.1093/protein/gzz014\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p4.1),[§VI\-A](https://arxiv.org/html/2609.25334#S6.SS1.p1.1)\.
- \[22\]K\.M\. Ruff\(2020\)Predicting conformational properties of intrinsically disordered proteins from sequence\.Methods in Molecular Biology2142\(\),pp\. 347–389\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p4.1),[§VI\-A](https://arxiv.org/html/2609.25334#S6.SS1.p1.1)\.
- \[23\]K\. P\. Sherry, R\. K\. Das, R\. V\. Pappu, and D\. Barrick\(2017\)Control of transcriptional activity by design of charge patterning in the intrinsically disordered ram region of the notch receptor\.Proc\. Natl\. Acad\. Sci\.114\(44\),pp\. E9243–9252\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p4.1),[§VI\-A](https://arxiv.org/html/2609.25334#S6.SS1.p1.1)\.
- \[24\]T\. Zarin, B\. Strome, B\. A\. N\. Nguyen, S\. Alberti, J\. Forman\-Kay, and A\. Moses\(2019\)Proteome\-wide signatures of function in highly diverged intrinsically disordered regions\.Elife8\(\),pp\. 46883\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p4.1),[§VI\-A](https://arxiv.org/html/2609.25334#S6.SS1.p1.1)\.
- \[25\]J\. Huihui and K\. Ghosh\(2021\)Intrachain interaction topology can identify functionally similar intrinsically disordered proteins\.Biophysical J\.120\(10\),pp\. 1860–1868\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p4.1),[§VI\-A](https://arxiv.org/html/2609.25334#S6.SS1.p1.1)\.
- \[26\]A\. Haider, K\. Gaalswyk, L\. Houston, N\. J\. Ose, S\. B\. Ozkan, and K\. Ghosh\(2025\)Statistical physics\-based approaches to model the function and complexation of disordered proteins\.The Journal of Physical Chemistry B\.External Links:[Document](https://dx.doi.org/10.1021/acs.jpcb.5c05422)Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p4.1),[§VI\-A](https://arxiv.org/html/2609.25334#S6.SS1.p1.1)\.
- \[27\]A\. Bah, R\. M\. Vernon, Z\. Siddiqui, M\. Krzeminski, R\. Muhandiram, C\. Zhao, N\. Sonenberg, L\. E\. Kay, and J\. D\. Forman\-Kay\(2015\)Folding of an intrinsically disordered protein by phosphorylation as a regulatory switch\.Nature519\(\),pp\. 106–109\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p5.1)\.
- \[28\]T\. H\. Kim, B\. Tsang, R\. M\. Vernon, N\. Sonenberg, L\. E\. Kay, and J\. D\. Forman\-Kay\(2019\)Phospho\-dependent phase separation of fmrp and caprin1 recapitulates regulation of translation and deadenylation\.Science365\(\),pp\. 825–829\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p5.1)\.
- \[29\]P\. Durek, R\. Schmidt, J\. L\. Heazlewood, A\. Jones, D\. MacLean, A\. Nagel, B\. Kersten, and W\. X\. Schulze\(2010\)PhosPhAt: the arabidopsis thaliana phosphorylation site database\. an update\.Nucleic acids research38\(suppl\_1\),pp\. D828–D834\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p5.1),[§IV\-A](https://arxiv.org/html/2609.25334#S4.SS1.p1.1)\.
- \[30\]H\. Dinkel, C\. Chica, A\. Via, C\. M\. Gould, L\. J\. Jensen, T\. J\. Gibson, and F\. Diella\(2010\)Phospho\. elm: a database of phosphorylation sites—update 2011\.Nucleic acids research39\(suppl\_1\),pp\. D261–D267\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p5.1),[§IV\-A](https://arxiv.org/html/2609.25334#S4.SS1.p1.1)\.
- \[31\]A\. Erijman, L\. Kozlowski, S\. Sohrabi\-Jahromi, J\. Fishburn, L\. Warfield, J\. Schreiber, W\. S\. Noble, J\. Söding, and S\. Hahn\(2020\)A high\-throughput screen for transcription activation domains reveals their sequence features and permits prediction by deep learning\.Molecular Cell78\(5\),pp\. 890–902\.e6\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p6.1),[§I](https://arxiv.org/html/2609.25334#S1.p7.1),[§IV\-A](https://arxiv.org/html/2609.25334#S4.SS1.p6.1)\.
- \[32\]D\. Wang, Y\. Liang, and D\. Xu\(2019\)Capsule network for protein post\-translational modification site prediction\.Bioinformatics35\(14\),pp\. 2386–2394\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p7.1),[§IV\-D](https://arxiv.org/html/2609.25334#S4.SS4.p3.1)\.
- \[33\]Y\. Xu, C\. Wilson, A\. Leier, T\. T\. Marquez\-Lago, J\. Whisstock, and J\. Song\(2020\)PhosTransfer: a deep transfer learning framework for kinase\-specific phosphorylation site prediction in hierarchy\.InPacific\-Asia Conference on Knowledge Discovery and Data Mining,pp\. 384–395\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p7.1),[§IV\-D](https://arxiv.org/html/2609.25334#S4.SS4.p3.1)\.
- \[34\]F\. R\. Lumbanraja, B\. Mahesworo, T\. W\. Cenggoro, A\. Budiarto, and B\. Pardamean\(2019\)An evaluation of deep neural network performance on limited protein phosphorylation site prediction data\.Procedia Computer Science157,pp\. 25–30\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p7.1),[§IV\-D](https://arxiv.org/html/2609.25334#S4.SS4.p3.1)\.
- \[35\]D\. Griffith and A\. Holehouse\(2021\)PARROT is a flexible recurrent neural network framework for analysis of large protein datasets\.eLife\(\),pp\. e70576\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p7.1),[§I](https://arxiv.org/html/2609.25334#S1.p9.1),[§II\-A](https://arxiv.org/html/2609.25334#S2.SS1.p6.1)\.
- \[36\]S\. Ahmed, M\. Kabir, M\. Arif, Z\. U\. Khan, and D\. Yu\(2021\)DeepPPSite: a deep learning\-based model for analysis and prediction of phosphorylation sites using efficient sequence information\.Analytical biochemistry612,pp\. 113955\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p7.1),[§IV\-D](https://arxiv.org/html/2609.25334#S4.SS4.p3.1)\.
- \[37\]H\. Yang, M\. Wang, X\. Liu, X\. Zhao, and A\. Li\(2021\)PhosIDN: an integrated deep neural network for improving protein phosphorylation site prediction by combining sequence and protein–protein interaction information\.Bioinformatics37\(24\),pp\. 4668–4676\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p7.1),[§IV\-D](https://arxiv.org/html/2609.25334#S4.SS4.p3.1)\.
- \[38\]K\. E\. Kirchoff and S\. M\. Gomez\(2022\)EMBER: multi\-label prediction of kinase\-substrate phosphorylation events through deep learning\.Bioinformatics38\(8\),pp\. 2119–2126\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p7.1),[§IV\-D](https://arxiv.org/html/2609.25334#S4.SS4.p3.1)\.
- \[39\]Q\. Yuan, S\. Chen, J\. Rao, S\. Zheng, H\. Zhao, and Y\. Yang\(2022\)AlphaFold2\-aware protein–dna binding site prediction using graph transformer\.Briefings in bioinformatics23\(2\),pp\. bbab564\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p7.1),[§II\-A](https://arxiv.org/html/2609.25334#S2.SS1.p2.1),[§II\-A](https://arxiv.org/html/2609.25334#S2.SS1.p4.1)\.
- \[40\]Z\. Lin, H\. Akin, R\. Rao, B\. Hie, Z\. Zhu, W\. Lu, N\. Smetanin, R\. Verkuil, O\. Kabeli, Y\. Shmueli,et al\.\(2023\)Evolutionary\-scale prediction of atomic\-level protein structure with a language model\.Science379\(6637\),pp\. 1123–1130\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p7.1)\.
- \[41\]N\. Strodthoff, P\. Wagner, M\. Wenzel, and W\. Samek\(2020\)UDSMProt: universal deep sequence models for protein classification\.Bioinformatics36\(8\),pp\. 2401–2409\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p7.1),[§II\-A](https://arxiv.org/html/2609.25334#S2.SS1.p6.1)\.
- \[42\]A\. Chandra, L\. Tünnermann, T\. Löfstedt, and R\. Gratz\(2023\)Transformer\-based deep learning for predicting protein properties in the life sciences\.Elife12,pp\. e82819\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p7.1),[§II\-A](https://arxiv.org/html/2609.25334#S2.SS1.p2.1)\.
- \[43\]M\. Réau, N\. Renaud, L\. C\. Xue, and A\. M\. Bonvin\(2023\)DeepRank\-gnn: a graph neural network framework to learn patterns in protein–protein interfaces\.Bioinformatics39\(1\),pp\. btac759\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p7.1),[§II\-A](https://arxiv.org/html/2609.25334#S2.SS1.p2.1)\.
- \[44\]S\. Dong and S\. Wang\(2022\)Assembled graph neural network using graph transformer with edges for protein model quality assessment\.Journal of Molecular Graphics and Modelling110,pp\. 108053\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p7.1),[§II\-A](https://arxiv.org/html/2609.25334#S2.SS1.p2.1)\.
- \[45\]N\. Huynh, I\. C\. Kazan, J\. Lu, B\. Kolbaba\-Kartchner, J\. H\. Mills, and S\. B\. Ozkan\(2025\)A protein dynamics–based deep learning model enhances predictions of fitness and epistasis\.Proceedings of the National Academy of Sciences122\(42\),pp\. e2502444122\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p7.1)\.
- \[46\]P\. Mollaei, D\. Sadasivam, C\. Guntuboina, and A\.B\. Farimani\(2024\)IDP\-bert: predicting properties of intrinsically disordered proteins using large language models\.Journal of Physical Chemistry B128\(49\),pp\. 12030–12037\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p7.1)\.
- \[47\]A\. Elnaggar, M\. Heinzinger, C\. Dallago, G\. Rehawi, Y\. Wang, L\. Jones, T\. Gibbs, T\. Feher, C\. Angerer, M\. Steinegger,et al\.\(2021\)ProtTrans: towards cracking the language of life’s code through self\-supervised learning\.IEEE Transactions on Pattern Analysis and Machine Intelligence44,pp\. 7112–7127\.Cited by:[§I](https://arxiv.org/html/2609.25334#S1.p7.1),[§III\-C](https://arxiv.org/html/2609.25334#S3.SS3.p1.1)\.
- \[48\]Z\. Shi and B\. Li\(2022\)Graph neural networks and attention\-based cnn\-lstm for protein classification\.arXiv preprint arXiv:2204\.09486\.Cited by:[§II\-A](https://arxiv.org/html/2609.25334#S2.SS1.p2.1),[§II\-A](https://arxiv.org/html/2609.25334#S2.SS1.p6.1)\.
- \[49\]M\. Gao, D\. Zhang, Y\. Chen, Y\. Zhang, Z\. Wang, X\. Wang, S\. Li, Y\. Guo, G\. I\. Webb, A\. T\. Nguyen,et al\.\(2024\)GraphormerDTI: a graph transformer\-based approach for drug\-target interaction prediction\.Computers in Biology and Medicine,pp\. 108339\.Cited by:[§II\-A](https://arxiv.org/html/2609.25334#S2.SS1.p2.1)\.
- \[50\]K\. K\. Yang, Z\. Wu, C\. N\. Bedbrook, and F\. H\. Arnold\(2018\)Learned protein embeddings for machine learning\.Bioinformatics34\(15\),pp\. 2642–2648\.Cited by:[§II\-A](https://arxiv.org/html/2609.25334#S2.SS1.p4.1)\.
- \[51\]Q\. Ho, D\. Phan, Y\. Ou,et al\.\(2019\)Using word embedding technique to efficiently represent protein sequences for identifying substrate specificities of transporters\.Analytical biochemistry577,pp\. 73–81\.Cited by:[§II\-A](https://arxiv.org/html/2609.25334#S2.SS1.p4.1)\.
- \[52\]M\. Hamid and I\. Friedberg\(2019\)Identifying antimicrobial peptides using word embedding with deep recurrent neural networks\.Bioinformatics35\(12\),pp\. 2009–2016\.Cited by:[§II\-A](https://arxiv.org/html/2609.25334#S2.SS1.p4.1)\.
- \[53\]D\. Ofer, N\. Brandes, and M\. Linial\(2021\)The language of proteins: nlp, machine learning & protein sequences\.Computational and Structural Biotechnology Journal19,pp\. 1750–1758\.Cited by:[§II\-A](https://arxiv.org/html/2609.25334#S2.SS1.p4.1)\.
- \[54\]C\. Tran, S\. Khadkikar, and A\. Porollo\(2023\)Survey of protein sequence embedding models\.International Journal of Molecular Sciences24\(4\),pp\. 3775\.Cited by:[§II\-A](https://arxiv.org/html/2609.25334#S2.SS1.p4.1)\.
- \[55\]C\. Zhao, T\. Liu, and Z\. Wang\(2022\)PANDA2: protein function prediction using graph neural networks\.NAR genomics and bioinformatics4\(1\),pp\. lqac004\.Cited by:[§II\-A](https://arxiv.org/html/2609.25334#S2.SS1.p4.1)\.
- \[56\]B\. Lai and J\. Xu\(2022\)Accurate protein function prediction via graph attention networks with predicted structure information\.Briefings in Bioinformatics23\(1\),pp\. bbab502\.Cited by:[§II\-A](https://arxiv.org/html/2609.25334#S2.SS1.p4.1),[§II\-A](https://arxiv.org/html/2609.25334#S2.SS1.p6.1)\.
- \[57\]N\. Brandes, D\. Ofer, Y\. Peleg, N\. Rappoport, and M\. Linial\(2022\)ProteinBERT: a universal deep\-learning model of protein sequence and function\.Bioinformatics38\(8\),pp\. 2102–2110\.Cited by:[§II\-A](https://arxiv.org/html/2609.25334#S2.SS1.p6.1)\.
- \[58\]M\. Nawar, M\. Shomer, S\. Faddel, and H\. Gong\(2023\)Transfer learning in deep learning models for building load forecasting: case of limited data\.InSoutheastCon 2023,pp\. 532–538\.Cited by:[§II\-B](https://arxiv.org/html/2609.25334#S2.SS2.p2.1)\.
- \[59\]S\. Khanal, S\. Tirupathi, M\. Dzaferagic, and T\. B\. Pedersen\(2025\)Addressing data scarcity and distribution shifts in communication networks using pre\-trained transformers and transfer learning\.In1st Workshop on AI for Urban Planning, AAAI 2025,pp\. 0–0\.Cited by:[§II\-B](https://arxiv.org/html/2609.25334#S2.SS2.p2.1)\.
- \[60\]G\. Li, Z\. Yu, K\. Yang, M\. Lin, and C\. P\. Chen\(2024\)Exploring feature selection with limited labels: a comprehensive survey of semi\-supervised and unsupervised approaches\.IEEE Transactions on Knowledge and Data Engineering36\(11\),pp\. 6124–6144\.Cited by:[§II\-B](https://arxiv.org/html/2609.25334#S2.SS2.p2.1)\.
- \[61\]J\. Xu and H\. Le\(2022\)Generating representative samples for few\-shot classification\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 9003–9013\.Cited by:[§II\-B](https://arxiv.org/html/2609.25334#S2.SS2.p2.1)\.
- \[62\]C\. Ren, D\. He, and J\. Huang\(2025\)De novo functional protein sequence generation: overcoming data scarcity through regeneration and large models\.arXiv preprint arXiv:2503\.21123\.Cited by:[§II\-B](https://arxiv.org/html/2609.25334#S2.SS2.p2.1)\.
- \[63\]F\. Bargagna, L\. A\. De Santi, N\. Martini, D\. Genovesi, B\. Favilli, G\. Vergaro, M\. Emdin, A\. Giorgetti, V\. Positano, and M\. F\. Santarelli\(2023\)Bayesian convolutional neural networks in medical imaging classification: a promising solution for deep learning limits in data scarcity scenarios\.Journal of Digital Imaging36\(6\),pp\. 2567–2577\.Cited by:[§II\-B](https://arxiv.org/html/2609.25334#S2.SS2.p2.1)\.
- \[64\]A\. X\. Lu, H\. Zhang, M\. Ghassemi, and A\. Moses\(2020\)Self\-supervised contrastive learning of protein representations by mutual information maximization\.BioRxiv,pp\. 2020–09\.Cited by:[§II\-B](https://arxiv.org/html/2609.25334#S2.SS2.p2.1)\.
- \[65\]Y\. You and Y\. Shen\(2022\)Cross\-modality and self\-supervised protein embedding for compound–protein affinity and contact prediction\.Bioinformatics38\(Supplement\_2\),pp\. ii68–ii74\.Cited by:[§II\-B](https://arxiv.org/html/2609.25334#S2.SS2.p2.1)\.
- \[66\]R\. Schäfer, T\. Nicke, H\. Höfener, A\. Lange, D\. Merhof, F\. Feuerhake, V\. Schulz, J\. Lotz, and F\. Kiessling\(2024\)Overcoming data scarcity in biomedical imaging with a foundational multi\-task model\.Nature Computational Science4\(7\),pp\. 495–509\.Cited by:[§II\-B](https://arxiv.org/html/2609.25334#S2.SS2.p2.1)\.
- \[67\]N\. A\. Samee, G\. Atteia, S\. Meshoul, M\. A\. Al\-antari, and Y\. M\. Kadah\(2022\)Deep learning cascaded feature selection framework for breast cancer classification: hybrid cnn with univariate\-based approach\.Mathematics10\(19\),pp\. 3631\.Cited by:[§II\-B](https://arxiv.org/html/2609.25334#S2.SS2.p2.1)\.
- \[68\]B\. E\. Suzek, H\. Huang, P\. McGarvey, R\. Mazumder, and C\. H\. Wu\(2007\)UniRef: comprehensive and non\-redundant uniprot reference clusters\.Bioinformatics23\(10\),pp\. 1282–1288\.Cited by:[§III\-C](https://arxiv.org/html/2609.25334#S3.SS3.p1.1)\.
- \[69\]J\. Sun, H\. H\. Dodge, and M\. H\. Mahoor\(2024\)MC\-vivit: multi\-branch classifier\-vivit to detect mild cognitive impairment in older adults using facial videos\.Expert systems with applications238,pp\. 121929\.Cited by:[§III\-D](https://arxiv.org/html/2609.25334#S3.SS4.p1.1)\.
- \[70\]J\. Sun and M\. Mahoor\(2025\)Video quality assessment\-jointed video classification\.Available at SSRN 5174457\.Cited by:[§III\-D](https://arxiv.org/html/2609.25334#S3.SS4.p1.1)\.
- \[71\]L\. Sawle and K\. Ghosh\(2015\)A theoretical method to compute sequence dependent configurational properties in charged polymers and proteins\.J\. Chem\. Phys\.143\(8\),pp\. 085101\.Cited by:[§III\-E](https://arxiv.org/html/2609.25334#S3.SS5.p4.1),[§V\-G](https://arxiv.org/html/2609.25334#S5.SS7.p1.1)\.
- \[72\]T\. Firman and K\. Ghosh\(2018\)Sequence charge decoration dictates coil\-globule transition in intrinsically disordered proteins\.J\. Chem\. Phys\.148\(12\),pp\. 123305\.Cited by:[§III\-E](https://arxiv.org/html/2609.25334#S3.SS5.p4.1)\.
- \[73\]J\. Huihui and K\. Ghosh\(2020\)An analytical theory to describe sequence\-specific inter\-residue distance profiles for polyampholytes and intrinsically disordered proteins\.J\. Chem\. Phys\.52\(\),pp\. 161102\.Cited by:[§III\-E](https://arxiv.org/html/2609.25334#S3.SS5.p4.1)\.
- \[74\]K\. Ghosh, J\. Huihui, M\. Phillips, and A\. Haider\(2022\)Rules of physical mathematics govern intrinsically disordered proteins\.Annu\. Rev\. Biophys\.51\(\),pp\. 355–376\.Cited by:[§III\-E](https://arxiv.org/html/2609.25334#S3.SS5.p4.1),[§V\-G](https://arxiv.org/html/2609.25334#S5.SS7.p1.1),[§VI\-A](https://arxiv.org/html/2609.25334#S6.SS1.p1.1)\.
- \[75\]M\. Phillips, M\. Muthukumar, and K\. Ghosh\(2024\)Beyond monopole electrostatics in regulating conformations of intrinsically disordered proteins\.PNAS Nexus3\(9\),pp\. 367\.Cited by:[§III\-E](https://arxiv.org/html/2609.25334#S3.SS5.p4.1)\.
- \[76\]A\. Firouzbakht, A\. Haider, K\. Gaalswyk, S\. Alaeen, K\. Ghosh, and M\. Gruebele\(2024\)HYPK: a marginally disordered protein sensitive to charge decoration\.Proc\. Natl\. Acad\. Sci\.121\(8\),pp\. e2316408121\.Cited by:[§III\-E](https://arxiv.org/html/2609.25334#S3.SS5.p4.1)\.
- \[77\]M\. C\. Cohan, M\. K\. Shinn, J\. M\. Lalmansingh, and R\. V\. Pappu\(2022\)Uncovering non\-random binary patterns within sequences of intrinsically disordered proteins\.Journal of molecular biology434\(2\),pp\. 167373\.Cited by:[§III\-F1](https://arxiv.org/html/2609.25334#S3.SS6.SSS1.p1.1)\.
- \[78\]D\. Griffith and A\. S\. Holehouse\(2021\)PARROT is a flexible recurrent neural network framework for analysis of large protein datasets\.Elife10,pp\. e70576\.Cited by:[TABLE V](https://arxiv.org/html/2609.25334#S4.T5.1.1.3.2),[TABLE V](https://arxiv.org/html/2609.25334#S4.T5.1.1.6.2),[TABLE V](https://arxiv.org/html/2609.25334#S4.T5.1.1.9.2)\.
- \[79\]D\. Wang, S\. Zeng, C\. Xu, W\. Qiu, Y\. Liang, T\. Joshi, and D\. Xu\(2017\)MusiteDeep: a deep\-learning framework for general and kinase\-specific phosphorylation site prediction\.Bioinformatics33\(24\),pp\. 3909–3916\.Cited by:[TABLE V](https://arxiv.org/html/2609.25334#S4.T5.1.1.10.1),[TABLE V](https://arxiv.org/html/2609.25334#S4.T5.1.1.4.1),[TABLE V](https://arxiv.org/html/2609.25334#S4.T5.1.1.7.1)\.
- \[80\]X\. Wang, Z\. Zhang, C\. Zhang, X\. Meng, X\. Shi, and P\. Qu\(2022\)Transphos: a deep\-learning model for general phosphorylation site prediction based on transformer\-encoder architecture\.International Journal of Molecular Sciences23\(8\),pp\. 4263\.Cited by:[§IV\-D](https://arxiv.org/html/2609.25334#S4.SS4.p3.1)\.
- \[81\]A\. C\. da Silva Junior, A\. M\. Shimaoka, L\. R\. Lopes, J\. H\. Coelho Campos, P\. B\. Paiva, and H\. Pequeno Monteiro\(2025\)Deep learning techniques applied to phosphorylation site prediction: a systematic review\.SN Computer Science6\(4\),pp\. 351\.Cited by:[§IV\-D](https://arxiv.org/html/2609.25334#S4.SS4.p3.1)\.

## Appendix AAblation Study Results

TABLE VIII:Detailed ablation study results comparing model variants on PPA and PELM datasets\. MT\-ProtBERT consistently achieves the highest mean Accuracy and MCC across all subsets, validating the necessity of each proposed module\.*MT\-ProtBERT \(Mask=1\)*indicates single\-position Dynamic Window Masking;*ProtBERT Only*removes all proposed modules except the backbone and a fully connected output layer;*No \(MLM\+STP\)*disables multi\-task learning;*No STP*and*No MLM*individually remove each auxiliary task;*NoA​A2AA\_\{2\}*removes the second MLM task;*No Soft Acc*switches soft Top\-1 accuracy to hard one and deactivates Protein \- aware Loss Function but use standard cross entropy loss;*No MS\-Conv1D*replaces the multi\-scale classifier with a single fully connected layer; and combinations such as*No \(MS\-Conv1D\+MLM\)*or*No \(MS\-Conv1D\+STP\)*disable both modules simultaneously\. Results confirm that Dynamic Window Masking, MS\-Conv1D, and multi\-task learning jointly drive the performance gain\.TABLE IX:Detailed ablation study results comparing model variants for Experiment 2\. MT\-ProtBERT consistently achieves the highest mean Accuracy and MCC across all subsets, validating the necessity of each proposed module\. Results confirm that Dynamic Window Masking, MS\-Conv1D, and multi\-task learning jointly drive the performance gain\.TABLE X:Detailed ablation study results comparing model variants for Experiment 3\. MT\-ProtBERT consistently achieves the highest mean Accuracy and MCC across all subsets, validating the necessity of each proposed module\. Results confirm that Dynamic Window Masking, MS\-Conv1D, and multi\-task learning jointly drive the performance gain\.TABLE XI:Detailed ablation study results comparing model variants for Experiment 4\. MT\-ProtBERT consistently achieves the highest mean Accuracy and MCC across all subsets, validating the necessity of each proposed module\. Results confirm that Dynamic Window Masking, MS\-Conv1D, and multi\-task learning jointly drive the performance gain\.Tables[VIII](https://arxiv.org/html/2609.25334#A1.T8),[IX](https://arxiv.org/html/2609.25334#A1.T9),[X](https://arxiv.org/html/2609.25334#A1.T10), and[XI](https://arxiv.org/html/2609.25334#A1.T11)present the detailed results from the ablation studies conducted on both PPA/PELM and new datasets\. These results comprehensively verify the contribution of each proposed component within the MT\-ProtBERT architecture\. Specifically, the integration ofDynamic Window Masking,MS\-Conv1D,Multi\-view MLM,ProtBERT\-aware Loss function, andMulti\-task Learningconsistently leads to superior performance across most metrics, including Accuracy, F1 Score, and MCC\.

The experiments reveal several key observations:

- •Dynamic Window Maskingprovides multiple contextual views of short protein sequences, enhancing the model’s capacity to learn both local residue\-level and global structural information\.
- •MS\-Conv1Dintroduces multi\-scale convolutional filters that enrich token embeddings and improve feature generalization, particularly beneficial for data\-scarce subsets such as PPA with AA=Y and new dataset\.
- •Multi\-view MLM\(use bothA​A1AA\_\{1\}andA​A2AA\_\{2\}\) provides extra supervision\.
- •Multi\-task Learning\(combining MLM and S/TP prediction\) injects diverse supervision signals that help stabilize the ProtBERT encoder, reduce overfitting, and guide the model toward learning physics\-aware and semantically consistent representations\.
- •ProtBERT\-aware Loss functionembeds biology knowledge into the model\.

Overall, the ablation results clearly demonstrate that the full MT\-ProtBERT configuration achieves the most balanced and robust performance across different amino acid types and datasets\. This validates the necessity of each component in addressing bothprotein compaction and size predictionand the underlyingdata scarcity challenge\.

## Appendix BThe Effectiveness of Multi\-view MLM

MT\-ProtBERT predicts both AA1 and AA2 in the MLM task, which constitutes a multi\-view approach\. To prove the advantage of training MLM in a multi\-view setting, we provide the following proof\.

TABLE XII:Symbols appeared in the proving the advantage of doing multi\-view MLM\.First, Table[XII](https://arxiv.org/html/2609.25334#A2.T12)shows the definition of all relevant symbols\.

The optimization objective of single\-view MLM is Eq\.[8](https://arxiv.org/html/2609.25334#A2.E8), while that of multi\-view MLM is Eq\.[9](https://arxiv.org/html/2609.25334#A2.E9)\. The goal is to proveℒmulti\\mathcal\{L\}\_\{\\text\{multi\}\}≤\\leqℒsingle\\mathcal\{L\}\_\{\\text\{single\}\}\.

ℒsingle=𝔼M∼ℳ​𝔼X∼DWM​\(S,M\)\[−log⁡pθ​\(A​Am\|hθ​\(X\)\)\]\\displaystyle\\begin\{split\}\\mathcal\{L\}\_\{\\text\{single\}\}&=\\mathbb\{E\}\_\{M\\sim\\mathcal\{M\}\}\\mathbb\{E\}\_\{X\\sim\\text\{DWM\}\(S,M\)\}\\\\ &\\\>\[\-\\log p\_\{\\theta\}\(AA\_\{m\}\|h\_\{\\theta\}\(X\)\)\]\\end\{split\}\(8\)ℒmulti=𝔼M∼ℳ​𝔼X1,⋯,XK​∼i​i​d​DWM​\(S,M\)\[−log⁡\(1K​∑i=1Kpθ​\(A​Am\|hθ​\(Xi\)\)\)\]\\displaystyle\\begin\{split\}\\mathcal\{L\}\_\{\\text\{multi\}\}&=\\mathbb\{E\}\_\{M\\sim\\mathcal\{M\}\}\\mathbb\{E\}\_\{X\_\{1\},\\dotsm,X\_\{K\}\\overset\{iid\}\{\\sim\}\\text\{DWM\}\(S,M\)\}\\\\ &\\\>\[\-\\log\(\\frac\{1\}\{K\}\\displaystyle\\sum^\{K\}\_\{i=1\}p\_\{\\theta\}\(AA\_\{m\}\|h\_\{\\theta\}\(X\_\{i\}\)\)\)\]\\end\{split\}\(9\)
We apply Jensen’s inequality to Eq\.[9](https://arxiv.org/html/2609.25334#A2.E9)to obtain

E​q\.[9](https://arxiv.org/html/2609.25334#A2.E9)≤𝔼M∼ℳ​𝔼X1,⋯,XK​∼i​i​d​DWM​\(S,M\)\[1K∑Ki=1−log\(pθ\(AAm\|hθ\(Xi\)\)\)\]\\displaystyle\\begin\{split\}Eq\.~\\ref\{eq: SApp\_2\_E\_multi\-view\}&\\leq\\mathbb\{E\}\_\{M\\sim\\mathcal\{M\}\}\\mathbb\{E\}\_\{X\_\{1\},\\dotsm,X\_\{K\}\\overset\{iid\}\{\\sim\}\\text\{DWM\}\(S,M\)\}\\\\ &\\quad\[\\frac\{1\}\{K\}\\displaystyle\\sum^\{K\}\_\{i=1\}\-\\log\(p\_\{\\theta\}\(AA\_\{m\}\|h\_\{\\theta\}\(X\_\{i\}\)\)\)\]\\end\{split\}\(10\)
=𝔼M∼ℳ​1K​∑i=1K𝔼X1,⋯,XK​∼i​i​d​DWM​\(S,M\)\[−log⁡\(pθ​\(A​Am\|hθ​\(Xi\)\)\)\]\\displaystyle\\begin\{split\}&=\\mathbb\{E\}\_\{M\\sim\\mathcal\{M\}\}\\frac\{1\}\{K\}\\displaystyle\\sum^\{K\}\_\{i=1\}\\mathbb\{E\}\_\{X\_\{1\},\\dotsm,X\_\{K\}\\overset\{iid\}\{\\sim\}\\text\{DWM\}\(S,M\)\}\\\\ &\\\>\[\-\\log\(p\_\{\\theta\}\(AA\_\{m\}\|h\_\{\\theta\}\(X\_\{i\}\)\)\)\]\\end\{split\}\(11\)
Next, it is important to prove that

𝔼X1,⋯,XK​∼i​i​d​DWM​\(S,M\)​\[−log⁡\(pθ​\(A​Am\|hθ​\(Xi\)\)\)\]=𝔼Xi∼DWM​\(S,M\)​\[−log⁡\(pθ​\(A​Am\|hθ​\(Xi\)\)\)\]\\displaystyle\\begin\{split\}\\mathbb\{E\}\_\{X\_\{1\},\\dotsm,X\_\{K\}\\overset\{iid\}\{\\sim\}\\text\{DWM\}\(S,M\)\}\[\-\\log\(p\_\{\\theta\}\(AA\_\{m\}\|h\_\{\\theta\}\(X\_\{i\}\)\)\)\]\\\\ =\\mathbb\{E\}\_\{X\_\{i\}\\sim\\text\{DWM\}\(S,M\)\}\[\-\\log\(p\_\{\\theta\}\(AA\_\{m\}\|h\_\{\\theta\}\(X\_\{i\}\)\)\)\]\\\\ \\end\{split\}\(12\)
Given that\(X1,⋯,XK\)\(X\_\{1\},\\dotsm,X\_\{K\}\)∼i​i​d\\overset\{iid\}\{\\sim\}DWM​\(S,M\)\\text\{DWM\}\(S,M\), we havep⁡\(X1,⋯,XK\)p\(X\_\{1\},\\dotsm,X\_\{K\}\)==Πj=1K​p​\(Xj\)\\displaystyle\\Pi^\{K\}\_\{j=1\}p\(X\_\{j\}\)\.

Therefore,

E​q\.[12](https://arxiv.org/html/2609.25334#A2.E12)=∑X1∈χ⋯​∑XK∈χ\[−log⁡\(pθ​\(A​Am\|hθ​\(Xi\)\)\)\]p⁡\(X1,⋯,XK\)\\displaystyle\\begin\{split\}Eq\.~\\ref\{eq: SApp\_4\_reduce\_dimension\}&=\\displaystyle\\sum\_\{X\_\{1\}\\in\\chi\}\\dotsm\\displaystyle\\sum\_\{X\_\{K\}\\in\\chi\}\[\-\\log\(p\_\{\\theta\}\(AA\_\{m\}\|h\_\{\\theta\}\(X\_\{i\}\)\)\)\]\\\\ &\\quad p\(X\_\{1\},\\dotsm,X\_\{K\}\)\\\\ \\end\{split\}\(13\)
=∑X1∈χ⋯​∑XK∈χ\[−log⁡\(pθ​\(A​Am\|hθ​\(Xi\)\)\)\]Πj=1K​p​\(Xj\)=∑Xi∈χ\[−log⁡\(pθ​\(A​Am\|hθ​\(Xi\)\)\)\]​p​\(Xi\)Πj≠i​∑Xjp⁡\(Xj\)=∑Xi∈χ\[−log⁡\(pθ​\(A​Am\|hθ​\(Xi\)\)\)\]p⁡\(Xi\)​Πj≠i​1=∑Xi∈χ\[−log⁡\(pθ​\(A​Am\|hθ​\(Xi\)\)\)\]​p​\(Xi\)=𝔼Xi∼DWM​\(S,M\)​\[−log⁡\(pθ​\(A​Am\|hθ​\(Xi\)\)\)\]\\displaystyle\\begin\{split\}&=\\displaystyle\\sum\_\{X\_\{1\}\\in\\chi\}\\dotsm\\displaystyle\\sum\_\{X\_\{K\}\\in\\chi\}\[\-\\log\(p\_\{\\theta\}\(AA\_\{m\}\|h\_\{\\theta\}\(X\_\{i\}\)\)\)\]\\\\ &\\quad\\displaystyle\\Pi^\{K\}\_\{j=1\}p\(X\_\{j\}\)\\\\ &=\\displaystyle\\sum\_\{X\_\{i\}\\in\\chi\}\[\-\\log\(p\_\{\\theta\}\(AA\_\{m\}\|h\_\{\\theta\}\(X\_\{i\}\)\)\)\]p\(X\_\{i\}\)\\\\ &\\quad\\displaystyle\\Pi\_\{j\\neq i\}\\displaystyle\\sum\_\{X\_\{j\}\}p\(X\_\{j\}\)\\\\ &=\\displaystyle\\sum\_\{X\_\{i\}\\in\\chi\}\[\-\\log\(p\_\{\\theta\}\(AA\_\{m\}\|h\_\{\\theta\}\(X\_\{i\}\)\)\)\]\\\\ &\\quad p\(X\_\{i\}\)\\displaystyle\\Pi\_\{j\\neq i\}1\\\\ &=\\displaystyle\\sum\_\{X\_\{i\}\\in\\chi\}\[\-\\log\(p\_\{\\theta\}\(AA\_\{m\}\|h\_\{\\theta\}\(X\_\{i\}\)\)\)\]p\(X\_\{i\}\)\\\\ &=\\mathbb\{E\}\_\{X\_\{i\}\\sim\\text\{DWM\}\(S,M\)\}\[\-\\log\(p\_\{\\theta\}\(AA\_\{m\}\|h\_\{\\theta\}\(X\_\{i\}\)\)\)\]\\end\{split\}\(14\)
Now, Eq\.[14](https://arxiv.org/html/2609.25334#A2.E14)proves Eq\.[12](https://arxiv.org/html/2609.25334#A2.E12)\. We insert Eq\.[14](https://arxiv.org/html/2609.25334#A2.E14)into Eq\.[11](https://arxiv.org/html/2609.25334#A2.E11)\.

E​q\.[11](https://arxiv.org/html/2609.25334#A2.E11)=𝔼M∼ℳ​1K​∑i=1K𝔼Xi∼DWM​\(S,M\)\[−log⁡\(pθ​\(A​Am\|hθ​\(Xi\)\)\)\]=𝔼M∼ℳ​1K​∑i=1Kℒsingle=𝔼M∼ℳ​ℒsingle=ℒsingle\\displaystyle\\begin\{split\}Eq\.~\\ref\{eq: SApp\_3\_Jensen\_Ineq\}&=\\mathbb\{E\}\_\{M\\sim\\mathcal\{M\}\}\\frac\{1\}\{K\}\\displaystyle\\sum^\{K\}\_\{i=1\}\\mathbb\{E\}\_\{X\_\{i\}\\sim\\text\{DWM\}\(S,M\)\}\\\\ &\\quad\[\-\\log\(p\_\{\\theta\}\(AA\_\{m\}\|h\_\{\\theta\}\(X\_\{i\}\)\)\)\]\\\\ &=\\mathbb\{E\}\_\{M\\sim\\mathcal\{M\}\}\\frac\{1\}\{K\}\\displaystyle\\sum^\{K\}\_\{i=1\}\\mathcal\{L\}\_\{\\text\{single\}\}\\\\ &=\\mathbb\{E\}\_\{M\\sim\\mathcal\{M\}\}\\mathcal\{L\}\_\{\\text\{single\}\}\\\\ &=\\mathcal\{L\}\_\{\\text\{single\}\}\\end\{split\}\(15\)
Hence, we proveℒmulti≤ℒsingle\\mathcal\{L\}\_\{\\text\{multi\}\}\\leq\\mathcal\{L\}\_\{\\text\{single\}\}\.

Whenℒmulti\\mathcal\{L\}\_\{\\text\{multi\}\}==ℒsingle\\mathcal\{L\}\_\{\\text\{single\}\}?

The prediction probabilities of all views are identical to each other\.

pθ​\(A​Am\|hθ​\(X1\)\)p\_\{\\theta\}\(AA\_\{m\}\|h\_\{\\theta\}\(X\_\{1\}\)\)==⋯\\dotsm==pθ​\(A​Am\|hθ​\(XK\)\)p\_\{\\theta\}\(AA\_\{m\}\|h\_\{\\theta\}\(X\_\{K\}\)\)

Whenℒmulti\\mathcal\{L\}\_\{\\text\{multi\}\}<<ℒsingle\\mathcal\{L\}\_\{\\text\{single\}\}?

The prediction probabilities of at least two views are unequal\.

pθ​\(A​Am\|hθ​\(X1\)\)p\_\{\\theta\}\(AA\_\{m\}\|h\_\{\\theta\}\(X\_\{1\}\)\)≠\\neq⋯\\dotsm≠\\neqpθ​\(A​Am\|hθ​\(XK\)\)p\_\{\\theta\}\(AA\_\{m\}\|h\_\{\\theta\}\(X\_\{K\}\)\)

## Biography Section

![[Uncaptioned image]](https://arxiv.org/html/2609.25334v1/Figures/App_Bio_Jian.png)Jian Sunis a postdoctoral scholar at the University of California, Los Angeles, focusing on medical AI\. Jian received the Ph\.D\. in Computer Science at the University of Denver with an emphasis on data\-related issues in machine learning in 2026, received the M\.S\. in Statistics from the George Washington University in 2017, received B\.S\. in Mathematics and Applied Mathematics from Shandong Agricultural University in 2014\. Jian’s research interests include Computer Vision, NLP, and Speech Analysis\.![[Uncaptioned image]](https://arxiv.org/html/2609.25334v1/Figures/Kings.jpg)Kingshuk Ghoshis a Professor in the Department of Physics and Astronomy at the University of Denver working on theoretical biophysics\. Ghosh received his Ph\.D\. in Physics from the University of Massachusetts, Amherst in 2003 and worked as a Post\-doctoral scholar at the University of California, San Francisco\. Ghosh groups works on building physics based models of disordered proteins, gene networks and other complex problems in Biophysics\.![[Uncaptioned image]](https://arxiv.org/html/2609.25334v1/Figures/lhouston_headshot.jpg)Lilianna Houstonis a PhD candidate in the University of Denver’s Physics and Astronomy Department\. She works in Dr\. Kingshuk Ghosh’s lab on physics and machine learning models to better understand disordered proteins\. She received a B\.S\. in Astrophysics from Tufts University in 2021\.![[Uncaptioned image]](https://arxiv.org/html/2609.25334v1/Figures/mahoor.jpeg)Mohammad H\. Mahoorreceived an MS in Biomedical Engineering from Sharif University of Technology in 1998 and a Ph\.D\. in Electrical and Computer Engineering from the University of Miami in 2007\. Currently a professor of Computer Science at the University of Denver, his research focuses on computer vision, deep machine learning, affective computing, and human\-robot interaction, particularly with humanoid robots for children with autism and older adults with depression and dementia\.