CAMEL：面向超轻量级视网膜基础模型的跨架构掩码嵌入学习

# CAMEL: Cross-Architecture Masked Embedding Learning for Ultra-Lightweight Retinal Foundation Models
Source: [https://arxiv.org/html/2610.40954](https://arxiv.org/html/2610.40954)
DOI:[XXXXXXX\.XXXXXXX](https://doi.org/XXXXXXX.XXXXXXX)Conference:The 42nd ACM/SIGAPP Symposium on Applied Computing; April 5–9, 2027; Gwangju, South KoreaISBN:979\-X\-XXXX\-XXXX\-X/27/04CCS:Computing methodologies Machine learningCCS:Computing methodologies Transfer learningCCS:Applied computing Health informaticsAndre Guarnier De Mitri[https://orcid.org/0009-0006-2197-3460](https://orcid.org/0009-0006-2197-3460),Giovani Decico Lucafoemail:[giovanilucafo@usp\.br](mailto:[email protected])Affiliation:ICMC, University of São Paulo,São Carlos,São Paulo,Brazil,Miller Matheus Lima Anacleto Rochaemail:[anacletomiller@usp\.br](mailto:[email protected])Affiliation:ICMC, University of São Paulo,São Carlos,São Paulo,BrazilandDiego Furtado Silvaemail:[diegofsilva@usp\.br](mailto:[email protected])Affiliation:ICMC, University of São Paulo,São Carlos,São Paulo,Brazil

© none

###### Abstract\.

Recent progress in deep learning has substantially advanced automated grading of ophthalmologic pathologies, supported by large annotated datasets and improved representation learning\. However, a significant gap remains between state\-of\-the\-art performance and clinical scalability\. In particular, foundation models have reshaped image\-based medical screening and diagnosis\. These large\-scale architectures trained on extensive datasets learn rich, generalizable feature representations that can be adapted to a wide range of downstream tasks\. In ophthalmology, RETFound, one of the most relevant foundation models for retinal images, performs strongly across classification, segmentation, and risk prediction tasks\. Despite their effectiveness, foundation models are often computationally prohibitive, hindering their deployment in real\-world clinical environments where hardware constraints and latency are critical\. To bridge this gap, we introduce CAMEL \(Cross\-Architecture Masked Embedding Learning\), a novel cross\-architecture distillation framework that uses an asymmetric masking strategy during embedding learning to effectively transfer knowledge from massive teacher foundation models to compact, lightweight student models\. Our experimental evaluation across four datasets shows that CAMEL preserves accuracy while drastically reducing computational complexity, enabling lightweight yet highly accurate models suitable for resource\-limited clinical settings\.

###### Keywords:

Diabetic retinopathy, retinal fundus imaging, mobile screening, knowledge distillation, lightweight models

## 1\.Introduction

Recent progress in deep learning has substantially advanced automated grading of ophthalmologic pathologies, supported by large annotated datasets and improved representation learning\([Sevgi et al\., 2024](https://arxiv.org/html/2610.40954#bib.bib1)\)\. However, a significant gap remains between state\-of\-the\-art performance and clinical scalability\([Ting et al\., 2019](https://arxiv.org/html/2610.40954#bib.bib2)\)\. In particular, foundation models have reshaped the medical imaging landscape\. These large\-scale architectures trained on extensive datasets learn rich, generalizable feature representations that easily adapt to a wide range of downstream tasks\([Bommasani and others, 2022](https://arxiv.org/html/2610.40954#bib.bib23)\)\. In ophthalmology, RETFound\([Zhou et al\., 2023](https://arxiv.org/html/2610.40954#bib.bib5)\), a prominent foundation model trained on vast retinal image repositories, has demonstrated exceptional performance across classification, segmentation, and risk prediction tasks\.

Despite their effectiveness, foundation models are computationally restrictive, severely hindering their deployment in real\-world clinical environments where hardware constraints, energy consumption, and latency are critical\. Model distillation\([Hinton et al\., 2015](https://arxiv.org/html/2610.40954#bib.bib21)\)has emerged as a promising strategy for transferring knowledge from a massive teacher model to a compact student model\. This approach is particularly valuable in medical applications, enabling real\-time screening on diverse instruments, including mobile devices\([Nakayama et al\., 2024](https://arxiv.org/html/2610.40954#bib.bib18)\)\. However, conventional distillation approaches often struggle to bridge structurally different architectures\.

To address these limitations, we introduce CAMEL \(Cross\-Architecture Masked Embedding Learning\), a novel feature\-distillation framework that transfers representations across heterogeneous architectures using masked distillation\. To demonstrate the versatility of CAMEL, we evaluate its performance across a broad spectrum of lightweight student backbones, including Convolutional Neural Networks \(MobileNet\([Howard et al\., 2019](https://arxiv.org/html/2610.40954#bib.bib16)\),EfficientNet\([Tan and Le, 2019](https://arxiv.org/html/2610.40954#bib.bib15)\), and a standardU\-Net\([Ronneberger et al\., 2015](https://arxiv.org/html/2610.40954#bib.bib17)\)encoder\), a Vision Transformer \(TinyViT\([Wu et al\., 2022](https://arxiv.org/html/2610.40954#bib.bib14)\)\), and a State\-Space Model \(Vision Mamba \- Vim\([Zhu et al\., 2024](https://arxiv.org/html/2610.40954#bib.bib6)\)\)\.

Figure 1\.Comparison of parameter count, computational cost, and AUROC between RETFound, a large Transformer\-based foundation model, and a series of lightweight models for the dataset IDRID\. The suffix “D” represents models distilled from RETFound\.To assess model performance, we evaluate four publicly available benchmark datasets for Diabetic Retinopathy \(DR\) grading: IDRiD\([Porwal et al\., 2018](https://arxiv.org/html/2610.40954#bib.bib10)\), APTOS 2019\([Kassani et al\., 2019](https://arxiv.org/html/2610.40954#bib.bib11)\), mBRSET\([Nakayama et al\., 2024](https://arxiv.org/html/2610.40954#bib.bib18)\), and MESSIDOR\-2\([Decencière et al\., 2014](https://arxiv.org/html/2610.40954#bib.bib13)\)\. Including mBRSET is particularly meaningful, as its images are captured via mobile devices, reflecting the unconstrained, low\-resource clinical conditions our lightweight models target\. Together, these benchmarks present significant heterogeneity in image quality, illumination, hardware, and patient demographics, serving as a rigorous testbed for cross\-domain generalization\.

As illustrated in Fig\.[1](https://arxiv.org/html/2610.40954#S1.F1), our empirical results show that distilled student models consistently outperform their counterparts trained from scratch across all evaluated architectures\. Furthermore, we show that masked feature distillation\([Yang et al\., 2022](https://arxiv.org/html/2610.40954#bib.bib20)\)provides greater robustness under domain shifts than unmasked distillation\. By benchmarking a wide range of architectures within CAMEL, we offer a comprehensive spectrum of operational choices, enabling practitioners to select the optimal trade\-off between computational cost and diagnostic precision according to their deployment constraints\. Finally, our code and implementation details are publicly available111[https://anonymous\.4open\.science/r/CAMELL/](https://anonymous.4open.science/r/CAMELL/)\.

In summary, our main contributions are threefold:

- •We propose the Cross\-Architecture Masked Embedding Learning \(CAMEL\) framework, a novel feature\-distillation framework that leverages asymmetric masked embedding alignment to transfer knowledge from large\-scale foundation models \(such as RETFound\) into compact, lightweight student backbones across heterogeneous architectural paradigms\.
- •We comprehensively evaluate CAMEL across diverse vision backbones, demonstrating that distilled models usually outperform their counterparts trained from scratch and providing a flexible spectrum of accuracy\-efficiency trade\-offs\.
- •We make our complete implementation codebase, model configurations, and pre\-processing pipeline publicly available to foster further research into resource\-constrained medical imaging deployment\.

## 2\.Related Work

The transition from high\-performance research models to practical clinical tools requires navigating several technical bottlenecks\. This section reviews three complementary research directions that motivate our approach: general self\-supervised learning and foundation models, their adaptation to ophthalmology, and RETFound, a specific, recent, yet widely used foundation model for retinal images\.

### 2\.1\.Foundation Models and Self\-Supervised Learning

Foundation models have fundamentally influenced modern machine learning by acting as versatile “base” systems that learn general\-purpose features\. As defined by Bommasani et al\.\([Bommasani and others, 2022](https://arxiv.org/html/2610.40954#bib.bib23)\), these models are trained on massive datasets and can be adapted to a wide variety of downstream tasks\. Their success is largely driven by self\-supervised learning \(SSL\), which enables networks to learn deep semantic representations without exhaustive human labeling\.

Among SSL methods, DINO\([Caron et al\., 2021](https://arxiv.org/html/2610.40954#bib.bib9)\)demonstrated that Vision Transformers \(ViTs\) could learn object\-centric representations through self\-supervision by having a “student” network predict the outputs of a “teacher” model\. This teacher\-student paradigm is central to our proposal; however, while DINO typically employs homogeneous architectures, we explore a heterogeneous distillation strategy, transferring knowledge from a massive Transformer\-based teacher to a State Space Model\-based student or Convolutional Neural Networks \(CNNs\)\.

### 2\.2\.Foundation Models in Ophthalmology

Inspired by their success on natural images, foundation models have been specialized for several application domains, including retinal imaging \(e\.g\., Figure[2](https://arxiv.org/html/2610.40954#S2.F2)\), where they are designed to handle the distinctive textures and structures of the eye\. The most significant milestone in this domain is RETFound, introduced by Zhou et al\.\([Zhou et al\., 2023](https://arxiv.org/html/2610.40954#bib.bib5)\)\. Pretraining a large Vision Transformer \(ViT\) on 1\.6 million unlabeled fundus images using Masked Autoencoder \(MAE\), RETFound established a new state\-of\-the\-art for ophthalmic tasks, acting as a robust feature extractor from retinal images\. However, this power comes at a high computational price: 303M parameters and 119\.29 GFLOPs per forward pass\.

Other ophthalmic foundation models explore different modalities and training strategies\. RetCLIP\([Du et al\., 2024](https://arxiv.org/html/2610.40954#bib.bib8)\)aligns retinal images with clinical text reports through contrastive language, enabling zero\-shot inference and cross\-modal retrieval\. EyeFounder\([Shi et al\., 2024](https://arxiv.org/html/2610.40954#bib.bib7)\)further extends foundation models to broader ocular datasets and three\-dimensional imaging modalities\. Despite their expanded scope, such models remain computationally expensive, motivating the need for efficient distilled alternatives\.

![Refer to caption](https://arxiv.org/html/2610.40954v1/dataset_dr.png)Figure 2\.Classification task, IDRiD sample images illustrating diabetic retinopathy severity levels \(DR0–DR4\)\. Diabetic retinopathy is a diabetes complication that progressively damages retinal blood vessels and can lead to vision loss

## 3\.CAMEL: Cross\-Architecture Masked Embedding Learning

The primary challenge in deploying models like RETFound is the Transformer bottleneck\. Although Transformers dominate current benchmarks\([Zhou et al\., 2023](https://arxiv.org/html/2610.40954#bib.bib5);[Du et al\., 2024](https://arxiv.org/html/2610.40954#bib.bib8);[Shi et al\., 2024](https://arxiv.org/html/2610.40954#bib.bib7)\), their quadratic complexity\([Heidari et al\., 2024](https://arxiv.org/html/2610.40954#bib.bib22)\)in sequence length limits scalability for high\-resolution medical images and their use in resource\-constrained environments\. To address this, we move from heavy, attention\-based foundation models to highly efficient, lightweight architectures\.

One strategy to achieve this computational reduction is transferring the representation capabilities of the large foundation model into diverse lightweight student architectures\. Rather than being restricted to homogeneous distillation, our framework supports a wide spectrum of compact backbones, from structurally similar, lightweight Vision Transformers \(such as TinyViT\([Wu et al\., 2022](https://arxiv.org/html/2610.40954#bib.bib14)\)\) to heterogeneous paradigms including Convolutional Neural Networks \(CNNs\) and selective State\-Space Models \(SSMs, such as Vision Mamba \- Vim\([Zhu et al\., 2024](https://arxiv.org/html/2610.40954#bib.bib6)\)\)\. This flexible cross\-architecture alignment enables student models to inherit the teacher’s spatial and semantic representations while leveraging their respective architectural efficiencies for linear\-time inference or reduced operational footprints\.

To capture RETFound’s representational strengths in a computationally efficient architecture, we propose distilling the knowledge comprised in its feature embeddings into a lightweight model\. As illustrated in Figure[3](https://arxiv.org/html/2610.40954#S3.F3), our approach uses a teacher\-student paradigm in which the RETFound teacher remains frozen throughout the process\. This ensures that the teacher acts as a static source of high\-quality retinal features, while the lightweight student is optimized to minimize the discrepancy between their respective embedding spaces\.

![Refer to caption](https://arxiv.org/html/2610.40954v1/dist.png)Figure 3\.CAMEL framework: distillation process for training the student backbone using RETFound as the teacher model\. In this configuration, the ViM student processes a masked version of the input image, whereas the teacher receives the full, unmasked image; an MLP projection layer approximates the student’s and teacher’s feature embeddings\.To evaluate feature reconstruction under varying spatial constraints, we explore both unmasked and masked distillation strategies\. In the unmasked setting, both the student and teacher process identical inputs, using complete fundus images\. Conversely, in the masked configuration, we apply an MAE\-style random masking strategy\([He et al\., 2022](https://arxiv.org/html/2610.40954#bib.bib19)\)to the student’s input\. Specifically, we randomly mask 75% of the image patches using a uniform distribution without replacement, while the frozen RETFound teacher receives the full, unmasked image\. This forces the student model to learn rich spatial dependencies and infer missing structural details by aligning its projected representations with the complete teacher embeddings\. Initial empirical evaluations on both the IDRiD and APTOS datasets demonstrated that this masked distillation strategy yields superior feature representations for lightweight student backbones\. Consequently, we selected the masked approach as our standard distillation protocol for comparing models\.

The distillation objective is to align the student’s internal representations with the teacher’s feature geometry\. This is achieved by minimizing the cosine distance between the projected student features and the teacher’s embeddings, defined by the following loss function:

\(1\)ℒdistill=1B​∑i=1B\(1−𝐬iproj⋅𝐭i\|𝐬iproj\|​\|𝐭i\|\),\\mathcal\{L\}\_\{\\text\{distill\}\}=\\frac\{1\}\{B\}\\sum\_\{i=1\}^\{B\}\\left\(1\-\\frac\{\\mathbf\{s\}\_\{i\}^\{\\text\{proj\}\}\\cdot\\mathbf\{t\}\_\{i\}\}\{\|\\mathbf\{s\}\_\{i\}^\{\\text\{proj\}\}\|\|\\mathbf\{t\}\_\{i\}\|\}\\right\),
where𝐬iproj\\mathbf\{s\}\_\{i\}^\{\\text\{proj\}\}and𝐭i\\mathbf\{t\}\_\{i\}denote the projected student features and the corresponding teacher features for theii\-th sample, respectively, in a batch of sizeBB\.

After the distillation phase, the student backbone is finetuned to better adapt the distilled retinal knowledge\. We then perform a supervised downstream finetuning phase to adapt the model for Diabetic Retinopathy \(DR\) grading\. For this purpose, we attach a classification head inspired by the DINO framework\([Caron et al\., 2021](https://arxiv.org/html/2610.40954#bib.bib9)\), consisting of two fully connected layers with Batch Normalization, GELU activations, and dropout, followed by a final linear layer that produces the class logits\. This two\-stage pipeline allows CAMEL to achieve foundation\-level diagnostic accuracy while remaining light enough for real\-time inference on mobile devices\.

The remainder of this section details the backbone architecture illustrated in Fig\.[3](https://arxiv.org/html/2610.40954#S3.F3)\.

### 3\.1\.Projection layer

To align the representation spaces of the teacher and student models, we introduce a lightweight projection module that maps the student embedding dimension to the teacher’s feature dimension\. The projector is implemented as a two\-layer MLP with a GELU activation, consisting of a first linear layer with 128 neurons \(expanding the 64\-dimensional student embedding\) followed by a second linear layer with 1024 neurons corresponding to the teacher embedding dimension\. This component contains 468K parameters, adding negligible computational overhead while proving essential for stable distillation\.

### 3\.2\.Optimizer

Based on the optimization strategy used in RETFound, we adopt the AdamW optimizer for both the distillation stage and the subsequent downstream task\. As our experiments confirm \(cf\. Section[5](https://arxiv.org/html/2610.40954#S5)\), this choice provides stable convergence and effective regularization, which are particularly important for models derived from sequence\-processing architectures such as Vim\. Training stability is further enhanced by a learning rate schedule that combines a warmup phase with cosine annealing: the learning rate increases linearly over the first 10 epochs and then decays smoothly toward a small final value as training progresses\.

Across all experiments, we maintain a consistent configuration of hyperparameters\. The base learning rate is set to5×10−45\\times 10^\{\-4\}, gradually reduced to a final value of1×10−61\\times 10^\{\-6\}by the end of training\. Weight decay is fixed at1×10−41\\times 10^\{\-4\}to provide additional regularization throughout optimization\.

### 3\.3\.Metrics and Callbacks

During the distillation phase, the objective is to align the features between the RETFound teacher and the lightweight student\. Thus, we monitor the distillation loss, computed as the cosine distance between the projected student features and the teacher features\.

To ensure stable convergence and avoid overfitting to noisy feature alignments, we employ two key callbacks\. The first one is Early Stopping, triggered when the validation distillation loss stops improving for 15 consecutive epochs\. We also employ Model Checkpointing, saving only the checkpoint with the lowest validation distillation loss\.

After distillation, a second training phase is performed where the distilled backbone and a lightweight classification head are trained using supervised labels\. In this stage, different metrics are tracked, such as accuracy, macro\-F1, AUROC, and AUPR, together with early stopping and checkpointing based on the validation F1 score, ensuring optimal downstream classification performance\.

### 3\.4\.Image Preprocessing and Data Augmentation

In order to improve feature extraction and mitigate inherent variations in fundus image quality \(e\.g\., uneven illumination and camera artifacts\), we apply a preprocessing pipeline prior to network ingestion\.

First, extraneous black backgrounds are removed using contour\-based bounding box cropping\. We then isolate the green color channel, which empirically yields the highest contrast for critical retinal structures such as blood vessels and lesions\. To correct uneven illumination across the fundus, we apply Ben Graham’s luminance normalization method, combining the original green channel with a Gaussian\-blurred version to equalize lighting\. This is followed by Contrast Limited Adaptive Histogram Equalization \(CLAHE\)\([Pizer et al\., 1987](https://arxiv.org/html/2610.40954#bib.bib3)\)to enhance local structural details\. Finally, a central circular mask is applied to eliminate artificial corner artifacts, and the enhanced grayscale image is converted back to a 3\-channel RGB format for compatibility with the pretrained feature spaces\.

During the downstream fine\-tuning phase, we employ an augmentation strategy to improve generalization\. This regime begins with standard geometric variations, including random horizontal and vertical flips and random rotations up to45∘45^\{\\circ\}\. To make the model anatomically robust against variations in eye positioning and lens geometry, we incorporate elastic deformations \(α=50\.0\\alpha=50\.0\)\. Finally, we apply RandAugment\([Cubuk et al\., 2020](https://arxiv.org/html/2610.40954#bib.bib4)\)to optimize the augmentation policy space, then apply standard ImageNet normalization before passing the tensors to the network\.

## 4\.Experimental Setup

We evaluate the CAMEL framework using four retinal fundus datasets: IDRiD, APTOS, mBRSET, and MESSIDOR\-2\. These datasets vary substantially in size, image quality, acquisition hardware, and patient demographics, providing an environment to assess the model’s adaptability and performance consistency\.

To build a comprehensive benchmark, we compare it against a variety of vision models\. Our lightweight convolutional baselines include MobileNet\([Howard et al\., 2019](https://arxiv.org/html/2610.40954#bib.bib16)\), EfficientNet\-B0\([Tan and Le, 2019](https://arxiv.org/html/2610.40954#bib.bib15)\), and a standard U\-Net encoder\([Ronneberger et al\., 2015](https://arxiv.org/html/2610.40954#bib.bib17)\)\. For Vision Transformers, we employ TinyViT\-5M\-224\([Wu et al\., 2022](https://arxiv.org/html/2610.40954#bib.bib14)\)and, as a State Space Model, we use Vim\([Zhu et al\., 2024](https://arxiv.org/html/2610.40954#bib.bib6)\)\. To assess the impact of our approach, we train each student backbone from scratch as a baseline and compare it directly with its distilled counterpart\. This paired configuration allows us to isolate the performance gains from knowledge distillation within each evaluated architectural family\. Finally, we benchmark all compact models against the full\-scale RETFound teacher to establish a speculative performance upper bound and measure the diagnostic capacity retained post\-distillation\. Table[1](https://arxiv.org/html/2610.40954#S4.T1)details the computational complexity and parameter counts for all evaluated architectures, as well as estimated energy and carbon dioxide usage\.

Table 1\.Computational complexity, inference efficiency, and environmental footprint across evaluated base architectures using a NVIDIA RTX A5000 GPU\. Latency is measured per image at batch size 1, while throughput represents images processed per second at batch size 32\. Energy and CO2metrics are scaled by10−610^\{\-6\}for readability\. Results are aggregated across all runs \(mean±\\pmstd\)\.ModelGFLOPsMobileNet1\.520\.1111\.18±\\pm0\.0211611\.9±\\pm82\.50\.95±\\pm1\.950\.09±\\pm0\.19EfficientNet4\.010\.7691\.83±\\pm0\.042649\.6±\\pm3\.28\.50±\\pm2\.100\.84±\\pm0\.21UNet4\.856\.5610\.71±\\pm0\.002348\.2±\\pm30\.98\.60±\\pm1\.850\.84±\\pm0\.18Vim5\.020\.47617\.46±\\pm0\.751691\.2±\\pm77\.818\.15±\\pm1\.701\.78±\\pm0\.17TinyViT5\.072\.5101\.92±\\pm0\.031695\.6±\\pm7\.414\.40±\\pm2\.201\.41±\\pm0\.21Foundation TeacherRETFound303\.31119\.29214\.79±\\pm0\.1293\.2±\\pm0\.9278\.50±\\pm2\.9027\.38±\\pm0\.28

The Indian Diabetic Retinopathy Image Dataset \(IDRiD\)\([Porwal et al\., 2018](https://arxiv.org/html/2610.40954#bib.bib10)\)serves as our primary benchmark for both feature distillation and downstream classification\. After standardizing the preprocessing and split construction, we obtained training and validation sets composed of 413 and 103 images, respectively\. Despite its relatively small size, IDRiD’s high\-quality annotations make it an essential baseline for evaluating the precision of distilled ophthalmic features\.

To examine scalability, we further train and evaluate the models on the APTOS 2019 Blindness Detection dataset\([Kassani et al\., 2019](https://arxiv.org/html/2610.40954#bib.bib11)\)\. This dataset contains 3,662 labeled retinal images, from which we allocated 2,930 for training and 366 for validation\. Its larger scale and distinct distributional characteristics relative to IDRiD let us assess how well models adapt to diverse clinical imaging conditions and varying grading protocols across populations\.

We employ the mBRSET\([Nakayama et al\., 2024](https://arxiv.org/html/2610.40954#bib.bib18)\)dataset to assess the robustness under realistic acquisition conditions\. mBRSET consists of fundus images captured with handheld portable cameras in community screening settings, exhibiting significant variations in illumination, focus, and field\-of\-view compared to the high\-end desktop cameras used in IDRiD and APTOS\. Notably, we first evaluate mBRSET without additional fine\-tuning to assess the viability of masked and unmasked variants\. Afterward, this dataset is fine\-tuned for in\-domain testing of all architectures\. We acknowledge that this dataset is highly imbalanced, with a predominant majority of healthy \(DR0\) samples and only 1\.68% representing severe \(DR3\) cases, posing a significant challenge for robust automated grading\.

Furthermore, we incorporate the MESSIDOR\-2 dataset\([Decencière et al\., 2014](https://arxiv.org/html/2610.40954#bib.bib13)\), a widely recognized standard for diabetic retinopathy assessment\. It comprises 1,748 macula\-centered fundus images collected across 874 patient examinations\. Including MESSIDOR\-2 allows us to validate the framework’s learning capacity on an established international benchmark\.

To comprehensively evaluate the effectiveness of each approach, we structured our experimental protocol to directly train and evaluate the model independently on all four datasets\. In this approach, both feature distillation and downstream classification training are conducted using the respective training split of each individual source dataset, then evaluated on the evaluation split\. This enables us to establish robust in\-domain performance metrics across diverse clinical settings, ranging from high\-end desktop imaging to unconstrained portable community screenings\. To ensure the reliability of our findings, all experiments are conducted across five random seeds, and statistical significance is determined using a non\-parametric Friedman test followed by a Nemenyi post\-hoc analysis to compare the performance of the lightweight models\.

## 5\.Results & Discussion

In this section, we first analyze the effectiveness of the masked and unmasked variants of the CAMEL framework by testing them on Vim models and comparing them with the RETFound teacher for in\-domain and cross\-domain evaluation, as well as a Zero\-Shot test on mBRSET\. Once we establish the best alternative across datasets, we use it in all subsequent experiments for UNet, EfficientNet, Mobilenet, and TinyVIT\. We trained and evaluated all models on IDRiD, APTOS, mBRSET, and MESSIDOR\.

### 5\.1\.Distillation Masking Analysis

To investigate how distillation strategy shapes representation learning, Figure[4](https://arxiv.org/html/2610.40954#S5.F4)illustrates the trajectory of the cosine distance loss between the RETFound teacher and student configurations throughout training\. For this analysis, we select the Vision Mamba \(Vim\) backbone as our representative student, as its linear\-time state\-space mechanisms offer a compelling non\-Transformer paradigm to probe how distinct inductive biases reconstruct teacher representations under spatial constraints\. While the unmasked student aligns more closely with the teacher’s representation, the masked variant maintains a slightly higher yet remarkably stable distance\. This behavior indicates that asymmetric masking forces the student model to learn a distinct, structurally robust feature space\. From a practical standpoint, this suggests that masking the input acts as a regularizer once the model no longer closely mimics the teacher’s representation\.

![Refer to caption](https://arxiv.org/html/2610.40954v1/distances.png)Figure 4\.Average loss during the distillation of the RETFound teacher and the masked/unmasked Vim\-based students\.To determine the best configuration, Table[2](https://arxiv.org/html/2610.40954#S5.T2)consolidates the in\-domain and cross\-dataset generalization results across the IDRiD, APTOS, and mBRSET datasets\. In\-domain evaluations demonstrate that while RETFound achieves the highest overall performance, both student variants remain highly competitive\. However, the cross\-dataset evaluations reveal a critical distinction\. Under severe domain shifts \(e\.g\., training on APTOS and evaluating on IDRiD\), the masked student demonstrates superior generalization capability, notably outperforming its unmasked counterpart in accuracy and ranking metrics\.

Table 2\.Masked vs\. unmasked variants across validation and cross\-dataset evaluations \(mean±\\pmstd\)\. The best results in each column per dataset arebolded, and the second best areunderlined\.
### 5\.2\.Benchmark Evaluations

With the masked student established as our primary approach, we present an expanded evaluation against the wider range of competitive baseline architectures\. We name each distilled version using the CAMEL framework by the backbone name followed by the identifier “D”: TinyVit\-D, Vim\-D, MobileNet\-D, EfficientNet\-D, and UNet\-D\. Table[3](https://arxiv.org/html/2610.40954#S5.T3)summarizes performance across IDRiD, APTOS, mBRSET, and MESSIDOR, while Figure[5](https://arxiv.org/html/2610.40954#S5.F5)visually shows these results\.

Figure 5\.Mean validation AUROC by model and dataset\. Bar height shows the mean AUROC, and error bars show the standard deviation\. Each dot is the result of a single model run\.On the IDRiD dataset, the CAMEL framework’s impact is notably apparent\. TinyVit\-D achieves the best overall results across F1 \(0\.471±0\.0190\.471\\pm 0\.019\) and AUPR \(0\.477±0\.0370\.477\\pm 0\.037\), improving over the baseline TinyVit\. Similarly, UNet\-D achieves the highest AUROC \(0\.782±0\.0890\.782\\pm 0\.089\) on the dataset, substantially improving upon the standard UNet\. While RETFound shows the advantages of large\-scale foundation pretraining with a slightly better accuracy, the CAMEL\-based ViT shows that our knowledge distillation successfully imbues lightweight architectures with diagnostic capabilities, surpassing the foundation model on all other metrics\.

On the APTOS dataset, which generally yields higher baseline scores due to more uniform image quality, the CAMEL framework’s advantage becomes even more pronounced\. Every single distilled variant, MobileNet\-D, EfficientNet\-D, UNet\-D, and TinyVit\-D yields strict improvements over its respective baseline in nearly all metrics\. Notably, TinyVit\-D reaches the highest Accuracy \(0\.849±0\.0080\.849\\pm 0\.008\) and F1 \(0\.736±0\.0130\.736\\pm 0\.013\), even surpassing the RETFound teacher\. Vim\-D also shows surprisingly large gains over the base Vim architecture, raising AUROC from0\.7870\.787to0\.8670\.867, showing that the distilled backbone retains strong diagnostic discriminability if compared to training from scratch\.

On the highly imbalanced mBRSET dataset, real\-world handheld device image acquisition, EfficientNet\-D shows a remarkable AUROC jump to0\.874±0\.0080\.874\\pm 0\.008compared to the standard EfficientNet’s0\.754±0\.1160\.754\\pm 0\.116\. While the standard UNet baseline performs exceptionally well on this specific distribution, TinyVit\-D continues to show consistent, broad upgrades over TinyVit across all metrics\. Furthermore, Vim\-D generalizes well in this challenging environment, outperforming the raw Vim baseline in F1 and showing robustness comparable to standard convolutional networks, yet within a much more structurally efficient framework\.

Finally, on the MESSIDOR dataset, TinyVit\-D and EfficientNet\-D capture the highest scores across F1, AUROC, and AUPR, improving upon their standard versions\. Furthermore, Vim\-D achieves an accuracy peak \(0\.849±0\.0030\.849\\pm 0\.003\), outperforming heavier conventional architectures like EfficientNet and MobileNet\.

This systematic uplift across diverse architectures confirms that the spatial mapping mechanisms and structural regularization enforced by the CAMEL framework’s masking and distillation process may be well\-suited to identifying specific lesion structures characteristic of diabetic retinopathy, regardless of the base model topology\.

Table 3\.Comprehensive Benchmark Performance on validation across IDRiD, APTOS, mBRSET, and MESSIDOR datasets \(mean±\\pmstd\)\. The best results in each column per dataset arebolded, and the second best areunderlined\.The core justification for CAMEL is that its masking mechanism acts as a strong regularizer while balancing classification power with strict constraints on parameters, FLOPs, and inference time\. While massive models like RETFound or heavier CNN architectures maximize diagnostic scores, they are fundamentally unsuited for mobile, resource\-limited deployments\. By asymmetrically distilling RETFound’s robust capabilities into a compact masked model, CAMEL minimizes the classification decay often associated with lightweight models\. The statistical parity observed \(Figure[6](https://arxiv.org/html/2610.40954#S5.F6)\) further shows that computational reduction does not cause a statistically meaningful loss of diagnostic power, positioning CAMEL as a highly capable, edge\-deployable alternative for diabetic retinopathy screening\.

Figure 6\.Critical difference \(CD\) diagram for AUROC\. Models connected by a horizontal bar are not significantly different according to the Friedman test with Nemenyi post\-hoc correction \(α=0\.05\\alpha=0\.05\)\.

## 6\.Conclusion

In this work, we addressed a critical barrier to deploying ophthalmic machine learning models in decentralized clinical settings: the prohibitive computational cost of Transformer\-based foundation models\. To bridge this gap, we introduced CAMEL\. This generalizable feature\-distillation framework transfers the rich ophthalmic representations of massive foundation models \(such as the 303M\-parameter RETFound\) into compact student backbones across heterogeneous architectural paradigms\. Our findings strongly advocate for knowledge distillation as an essential strategy in this scenario, confirming that the representational power required for accurate retinal screening can be condensed into lightweight networks suitable for edge and mobile deployment without compromising diagnostic precision\.

Across extensive benchmarking on four diverse datasets \(IDRiD, APTOS, mBRSET, and MESSIDOR\-2\), CAMEL\-distilled models consistently outperformed their counterparts trained from scratch\. Crucially, specific distilled student variants, such as TinyViT\-D and EfficientNet\-D, not only closed the performance gap with the teacher but also surpassed RETFound across key diagnostic metrics, while reducing parameter counts by up to75×75\\timesand floating operations by up to155×155\\times\. By evaluating a broad spectrum of CNN, ViT, and State\-Space Model backbones, CAMEL establishes an effective Pareto frontier of efficiency\-versus\-accuracy trade\-offs\. This enables practitioners and deployment engineers to select the optimal operational point tailored to their specific hardware, latency, and clinical constraints\. Furthermore, our analysis confirms that asymmetric masked feature distillation provides crucial structural regularization, significantly boosting robustness under severe domain and acquisition shifts\.

Future work will extend this framework to optimize cross\-architecture feature alignment further\. We plan to investigate hybrid distillation strategies that combine feature\-level embedding alignment with logit\-based loss and masked reconstruction objectives to make accurate task\-specific models\. Additionally, we aim to scale the distillation process using larger, multi\-center repositories, such as PAPILA\([Kovalyk et al\., 2022](https://arxiv.org/html/2610.40954#bib.bib12)\)and MESSIDOR\([Decencière et al\., 2014](https://arxiv.org/html/2610.40954#bib.bib13)\), to assess robustness across broader clinical demographics\. Beyond diabetic retinopathy grading, we will evaluate the CAMEL framework’s generality across a wider spectrum of ophthalmic tasks, including glaucoma detection, retinal vessel segmentation, and microaneurysm identification\.

## References

- Bommasaniet al\.\(2022\)R\. Bommasaniet al\.On the opportunities and risks of foundation models\.arXiv preprint arXiv:2108\.07258abs/2108\.07258\(1\),pp\. 1–212\.External Links:[Link](https://arxiv.org/abs/2108.07258)Cited by:[§1](https://arxiv.org/html/2610.40954#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.40954#S2.SS1.p1.1)\.
- Caronet al\.\(2021\)M\. Caron, H\. Touvron, I\. Misra, H\. Jégou, J\. Mairal, P\. Bojanowski, and A\. JoulinEmerging properties in self\-supervised vision transformers\.CoRRabs/2104\.14294,pp\. 9630–9640\.Cited by:[§2\.1](https://arxiv.org/html/2610.40954#S2.SS1.p2.1),[§3](https://arxiv.org/html/2610.40954#S3.p8.1)\.
- Cubuket al\.\(2020\)E\. D\. Cubuk, B\. Zoph, J\. Shlens, and Q\. V\. LeRandaugment: practical automated data augmentation with a reduced search space\.In2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops \(CVPRW\),Vol\.,pp\. 3008–3017\.External Links:[Document](https://dx.doi.org/10.1109/CVPRW50498.2020.00359)Cited by:[§3\.4](https://arxiv.org/html/2610.40954#S3.SS4.p3.1)\.
- Decencièreet al\.\(2014\)E\. Decencière, X\. Zhang, G\. Cazuguel, B\. Lay, B\. Cochener, C\. Trone, P\. Gain, J\. Ordóñez\-Varela, P\. Massin, A\. Erginay,et al\.Feedback on a publicly distributed image database: the messidor database\.Image Analysis & Stereology33\(3\),pp\. 231–234\.Cited by:[§1](https://arxiv.org/html/2610.40954#S1.p4.1),[§4](https://arxiv.org/html/2610.40954#S4.p6.1),[§6](https://arxiv.org/html/2610.40954#S6.p3.1)\.
- Duet al\.\(2024\)J\. Du, J\. Guo, W\. Zhang, S\. Yang, H\. Liu, H\. Li, and N\. WangRET\-clip: a retinal image foundation model pre\-trained with clinical diagnostic reports\.InMedical Image Computing and Computer Assisted Intervention – MICCAI 2024,M\. G\. Linguraru, Q\. Dou, A\. Feragen, S\. Giannarou, B\. Glocker, K\. Lekadir, and J\. A\. Schnabel \(Eds\.\),Cham,pp\. 709–719\.External Links:ISBN 978\-3\-031\-72390\-2Cited by:[§2\.2](https://arxiv.org/html/2610.40954#S2.SS2.p2.1),[§3](https://arxiv.org/html/2610.40954#S3.p1.1)\.
- Heet al\.\(2022\)K\. He, X\. Chen, S\. Xie, Y\. Li, P\. Dollár, and R\. GirshickMasked autoencoders are scalable vision learners\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),New York, NY, USA,pp\. 15979–15988\.Cited by:[§3](https://arxiv.org/html/2610.40954#S3.p4.1)\.
- Heidariet al\.\(2024\)M\. Heidari, R\. Azad, S\. G\. Kolahi, R\. Arimond, L\. Niggemeier, A\. Sulaiman, A\. Bozorgpour, E\. K\. Aghdam, A\. Kazerouni, I\. Hacihaliloglu, and D\. MerhofEnhancing efficiency in vision transformer networks: design techniques and insights\.External Links:2403\.19882,[Link](https://arxiv.org/abs/2403.19882)Cited by:[§3](https://arxiv.org/html/2610.40954#S3.p1.1)\.
- Hintonet al\.\(2015\)G\. Hinton, O\. Vinyals, and J\. DeanDistilling the knowledge in a neural network\.CoRRabs/1503\.02531\.External Links:[Link](https://arxiv.org/abs/1503.02531)Cited by:[§1](https://arxiv.org/html/2610.40954#S1.p2.1)\.
- Howardet al\.\(2019\)A\. Howard, M\. Sandler, G\. Chu, L\. Chen, B\. Chen, M\. Tan, W\. Wang, Y\. Zhu, R\. Pang, V\. Vasudevan, Q\. V\. Le, and H\. AdamSearching for mobilenetv3\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),New York, NY, USA,pp\. 1314–1324\.Cited by:[§1](https://arxiv.org/html/2610.40954#S1.p3.1),[§4](https://arxiv.org/html/2610.40954#S4.p2.1)\.
- Kassaniet al\.\(2019\)S\. H\. Kassani, P\. H\. Kassani, R\. Khazaeinezhad, M\. J\. Wesolowski, K\. A\. Schneider, and R\. DetersDiabetic retinopathy classification using a modified xception architecture\.In2019 IEEE International Symposium on Signal Processing and Information Technology \(ISSPIT\),New York, NY, USA,pp\. 1–6\.External Links:[Document](https://dx.doi.org/10.1109/ISSPIT47144.2019.9001846)Cited by:[§1](https://arxiv.org/html/2610.40954#S1.p4.1),[§4](https://arxiv.org/html/2610.40954#S4.p4.1)\.
- Kovalyket al\.\(2022\)O\. Kovalyk, J\. Morales\-Sánchez, R\. Verdú\-Monedero, I\. Sellés\-Navarro, A\. Palazón\-Cabanes, and J\. Sancho\-GómezPAPILA: Dataset with fundus images and clinical data of both eyes of the same patient for glaucoma assessment\.Scientific Data9\(1\),pp\. 291\.External Links:ISSN 2052\-4463,[Link](https://doi.org/10.1038/s41597-022-01388-1),[Document](https://dx.doi.org/10.1038/s41597-022-01388-1)Cited by:[§6](https://arxiv.org/html/2610.40954#S6.p3.1)\.
- Nakayamaet al\.\(2024\)L\. F\. Nakayama, L\. Zago Ribeiro, D\. Restrepo, N\. Santos Barboza, R\. Dias Fiterman, M\. l\. Vieira Sousa, A\. D\. A\. Pereira, C\. Regatieri, F\. K\. Malerbi, and R\. AndrademBRSET, a Mobile Brazilian Retinal Dataset\.Note:Version 1\.0PhysioNetExternal Links:[Document](https://dx.doi.org/10.13026/qxpd-1y65),[Link](https://doi.org/10.13026/qxpd-1y65)Cited by:[§1](https://arxiv.org/html/2610.40954#S1.p2.1),[§1](https://arxiv.org/html/2610.40954#S1.p4.1),[§4](https://arxiv.org/html/2610.40954#S4.p5.1)\.
- Pizeret al\.\(1987\)S\. M\. Pizer, E\. P\. Amburn, J\. D\. Austin, R\. Cromartie, A\. Geselowitz, T\. Greer, B\. ter Haar Romeny, J\. B\. Zimmerman, and K\. ZuiderveldAdaptive histogram equalization and its variations\.Computer Vision, Graphics, and Image Processing39\(3\),pp\. 355–368\.External Links:ISSN 0734\-189X,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/S0734-189X%2887%2980186-X),[Link](https://www.sciencedirect.com/science/article/pii/S0734189X8780186X)Cited by:[§3\.4](https://arxiv.org/html/2610.40954#S3.SS4.p2.1)\.
- Porwalet al\.\(2018\)P\. Porwal, S\. Pachade, R\. Kamble, M\. Kokare, G\. Deshmukh, V\. Sahasrabuddhe, and F\. MeriaudeauIndian diabetic retinopathy image dataset \(idrid\): a database for diabetic retinopathy screening research\.Data3\(3\),pp\. 25\.Cited by:[§1](https://arxiv.org/html/2610.40954#S1.p4.1),[§4](https://arxiv.org/html/2610.40954#S4.p3.1)\.
- Ronnebergeret al\.\(2015\)O\. Ronneberger, P\. Fischer, and T\. BroxU\-net: convolutional networks for biomedical image segmentation\.InInternational Conference on Medical Image Computing and Computer\-Assisted Intervention \(MICCAI\),Cham, Switzerland,pp\. 234–241\.Cited by:[§1](https://arxiv.org/html/2610.40954#S1.p3.1),[§4](https://arxiv.org/html/2610.40954#S4.p2.1)\.
- Sevgiet al\.\(2024\)M\. Sevgi, E\. Ruffell, F\. Antaki, M\. A\. Chia, and P\. A\. KeaneFoundation models in ophthalmology: opportunities and challenges\.Current opinion in ophthalmology36\(1\),pp\. 90\.Cited by:[§1](https://arxiv.org/html/2610.40954#S1.p1.1)\.
- Shiet al\.\(2024\)D\. Shi, W\. Zhang, X\. Chen, Y\. Liu, J\. Yang, S\. Huang, Y\. C\. Tham, Y\. Zheng, and M\. HeEyeFound: a multimodal generalist foundation model for ophthalmic imaging\.External Links:2405\.11338,[Link](https://arxiv.org/abs/2405.11338)Cited by:[§2\.2](https://arxiv.org/html/2610.40954#S2.SS2.p2.1),[§3](https://arxiv.org/html/2610.40954#S3.p1.1)\.
- Tan and Le \(2019\)M\. Tan and Q\. V\. LeEfficientNet: rethinking model scaling for convolutional neural networks\.InProceedings of the 36th International Conference on Machine Learning \(ICML\),Long Beach, CA, USA,pp\. 6105–6114\.Cited by:[§1](https://arxiv.org/html/2610.40954#S1.p3.1),[§4](https://arxiv.org/html/2610.40954#S4.p2.1)\.
- Tinget al\.\(2019\)D\. S\.W\. Ting, L\. Peng, A\. V\. Varadarajan, P\. A\. Keane, P\. M\. Burlina, M\. F\. Chiang, L\. Schmetterer, L\. R\. Pasquale, N\. M\. Bressler, D\. R\. Webster, M\. Abramoff, and T\. Y\. WongDeep learning in ophthalmology: the technical and clinical considerations\.Progress in Retinal and Eye Research72,pp\. 100759\.External Links:ISSN 1350\-9462,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.preteyeres.2019.04.003),[Link](https://www.sciencedirect.com/science/article/pii/S1350946218300909)Cited by:[§1](https://arxiv.org/html/2610.40954#S1.p1.1)\.
- Wuet al\.\(2022\)K\. Wu, J\. Zhang, H\. Peng, M\. Liu, B\. Xiao, J\. Fu, and L\. YuanTinyViT: fast pretraining distillation for small vision transformers\.External Links:2207\.10666,[Link](https://arxiv.org/abs/2207.10666)Cited by:[§1](https://arxiv.org/html/2610.40954#S1.p3.1),[§3](https://arxiv.org/html/2610.40954#S3.p2.1),[§4](https://arxiv.org/html/2610.40954#S4.p2.1)\.
- Yanget al\.\(2022\)Z\. Yang, Z\. Li, M\. Shao, D\. Shi, Z\. Yuan, and C\. YuanMasked generative distillation\.External Links:2205\.01529,[Link](https://arxiv.org/abs/2205.01529)Cited by:[§1](https://arxiv.org/html/2610.40954#S1.p5.1)\.
- Zhouet al\.\(2023\)Y\. Zhou, M\. A\. Chia, S\. K\. Wagner, M\. S\. Ayhan, D\. J\. Williamson, R\. R\. Struyven, T\. Liu, M\. Xu, M\. G\. Lozano, P\. Woodward\-Court, Y\. Kihara, UK Biobank Eye & Vision Consortium, N\. Allen, J\. E\. J\. Gallacher, T\. Littlejohns, T\. Aslam, P\. Bishop, G\. Black, P\. Sergouniotis, D\. Atan, A\. D\. Dick, C\. Williams, S\. Barman, J\. H\. Barrett, S\. Mackie, T\. Braithwaite, R\. O\. Carare, S\. Ennis, J\. Gibson, A\. J\. Lotery, J\. Self, U\. Chakravarthy, R\. E\. Hogg, E\. Paterson, J\. Woodside, T\. Peto, G\. Mckay, B\. Mcguinness, P\. J\. Foster, K\. Balaskas, A\. P\. Khawaja, N\. Pontikos, J\. S\. Rahi, G\. Lascaratos, P\. J\. Patel, M\. Chan, S\. Y\. L\. Chua, A\. Day, P\. Desai, C\. Egan, M\. Fruttiger, D\. F\. Garway\-Heath, A\. Hardcastle, S\. P\. T\. Khaw, T\. Moore, S\. Sivaprasad, N\. Strouthidis, D\. Thomas, A\. Tufail, A\. C\. Viswanathan, B\. Dhillon, T\. Macgillivray, C\. Sudlow, V\. Vitart, A\. Doney, E\. Trucco, J\. A\. Guggeinheim, J\. E\. Morgan, C\. J\. Hammond, K\. Williams, P\. Hysi, S\. P\. Harding, Y\. Zheng, R\. Luben, P\. Luthert, Z\. Sun, M\. McKibbin, E\. O’Sullivan, R\. Oram, M\. Weedon, C\. G\. Owen, A\. R\. Rudnicka, N\. Sattar, D\. Steel, I\. Stratton, R\. Tapp, M\. M\. Yates, A\. Petzold, S\. Madhusudhan, A\. Altmann, A\. Y\. Lee, E\. J\. Topol, A\. K\. Denniston, D\. C\. Alexander, and P\. A\. KeaneA foundation model for generalizable disease detection from retinal images \(RetFound\)\.Nature622\(7981\),pp\. 156–163\(en\)\.Note:Publisher: Springer Science and Business Media LLCExternal Links:ISSN 0028\-0836, 1476\-4687,[Link](https://www.nature.com/articles/s41586-023-06555-x),[Document](https://dx.doi.org/10.1038/s41586-023-06555-x)Cited by:[§1](https://arxiv.org/html/2610.40954#S1.p1.1),[§2\.2](https://arxiv.org/html/2610.40954#S2.SS2.p1.1),[§3](https://arxiv.org/html/2610.40954#S3.p1.1)\.
- Zhuet al\.\(2024\)L\. Zhu, B\. Liao, Q\. Zhang, X\. Wang, W\. Liu, and X\. WangVision mamba: efficient visual representation learning with bidirectional state space model\.External Links:2401\.09417,[Link](https://arxiv.org/abs/2401.09417)Cited by:[§1](https://arxiv.org/html/2610.40954#S1.p3.1),[§3](https://arxiv.org/html/2610.40954#S3.p2.1),[§4](https://arxiv.org/html/2610.40954#S4.p2.1)\.