MADJEPA:用于医学异常检测的增强型多阶段联合嵌入预测架构

arXiv cs.LG 论文

摘要

MADJEPA 论文提出一种用于医学异常检测的多阶段增强型联合嵌入预测架构(JEPA),通过三阶段训练流程在六个医学图像数据集上取得竞争性结果并开源代码。

arXiv:2611.25194v1 Announce Type: cross Abstract: Anomaly detection in medical images is a complex task, requiring the development of a robust model using only healthy patient data. Without access to labeled data, modern approaches typically employ self-supervised learning to obtain a model of normality. Since world models promise to build strong internal representations of their environment, we consider them as suitable candidates for the medical anomaly detection environment, namely medical investigations of healthy patients. To this end, we propose MADJEPA, a medical anomaly detector based on an enhanced multi-stage joint-embedding predictive architecture (JEPA). We design a multi-stage training pipeline to produce a robust detector: (1) image-based JEPA (I-JEPA) training aims to learn a strong representation of normal training data; (2) target encoder distillation aims to modulate over-generalization of I-JEPA representations to abnormal data; (3) pseudo-anomaly classification aims to calibrate anomaly predictions by learning a tight boundary around the normal data manifold. To demonstrate our design, we carry out comprehensive experiments on six datasets covering various medical image domains (e.g.~MRI, OCT, X-ray, histopathology), comparing our MADJEPA with numerous state-of-the-art methods. The results show that our carefully designed world model, MADJEPA, attains competitive results, ranking first on three datasets. Our ablation studies further confirm the utility of each training stage, indicating that I-JEPA alone is not sufficient to obtain competitive results in medical anomaly detection. We publicly release our code at https://github.com/Sabina085/madjepa.
查看原文
查看缓存全文

缓存时间: 2026/11/20 09:18

# MADJEPA: Enhanced Multi-Stage Joint-Embedding Predictive Architecture for Medical Anomaly Detection
Source: [https://arxiv.org/html/2611.25194](https://arxiv.org/html/2611.25194)
Ioana\-Sabina StoianFlorinel\-Alin CroitoruAffiliation:​University of Bucharest, RomaniaVlad HondruAffiliation:​University of Bucharest, RomaniaAndreea Iuliana IonescuAffiliation:​“Carol Davila” University of Medicine and Pharmacy, RomaniaAffiliation:Colţea Clinical Hospital, RomaniaAndrei PăunAffiliation:Romanian Academy Research Institute for Artificial Intelligence, RomaniaAffiliation:​University of Bucharest, RomaniaRadu Tudor IonescuAffiliation:​University of Bucharest, RomaniaAffiliation:Corresponding author: raducu\.ionescu@gmail\.com\.

###### Abstract

Anomaly detection in medical images is a complex task, requiring the development of a robust model using only healthy patient data\. Without access to labeled data, modern approaches typically employ self\-supervised learning to obtain a model of normality\. Since world models promise to build strong internal representations of their environment, we consider them as suitable candidates for the medical anomaly detection environment, namely medical investigations of healthy patients\. To this end, we propose MADJEPA, amedicalanomalydetector based on an enhanced multi\-stagejoint\-embeddingpredictivearchitecture \(JEPA\)\. We design a multi\-stage training pipeline to produce a robust detector: \(1\) image\-based JEPA \(I\-JEPA\) training aims to learn a strong representation of normal training data; \(2\) target encoder distillation aims to modulate over\-generalization of I\-JEPA representations to abnormal data; \(3\) pseudo\-anomaly classification aims to calibrate anomaly predictions by learning a tight boundary around the normal data manifold\. To demonstrate our design, we carry out comprehensive experiments on six datasets covering various medical image domains \(e\.g\. MRI, OCT, X\-ray, histopathology\), comparing our MADJEPA with numerous state\-of\-the\-art methods\. The results show that our carefully designed world model, MADJEPA, attains competitive results, ranking first on three datasets\. Our ablation studies further confirm the utility of each training stage, indicating that I\-JEPA alone is not sufficient to obtain competitive results in medical anomaly detection\. We publicly release our code at[https://github\.com/Sabina085/madjepa](https://github.com/Sabina085/madjepa)\.

## 1Introduction

Medical anomaly detection \(MAD\) aims to identify features that deviate from a highly diverse and partially characterized distribution of normal anatomy\. The task poses additional challenges compared with the usual supervised recognition setting, where the objective is just to discriminate between a predefined set of pathological categories\. In medical imaging, this open\-set formulation\[[59](https://arxiv.org/html/2611.25194#bib.bib9),[11](https://arxiv.org/html/2611.25194#bib.bib10)\]is particularly relevant, as pathological samples are often rare and expensive to annotate\[[7](https://arxiv.org/html/2611.25194#bib.bib11),[34](https://arxiv.org/html/2611.25194#bib.bib14),[16](https://arxiv.org/html/2611.25194#bib.bib16)\]\. In contrast, hospitals and imaging repositories contain large collections of unlabeled examinations, the vast majority of which exhibit normal anatomy or contain only weakly characterized findings\[[7](https://arxiv.org/html/2611.25194#bib.bib11),[5](https://arxiv.org/html/2611.25194#bib.bib15),[16](https://arxiv.org/html/2611.25194#bib.bib16)\]\. These observations motivate unsupervised MAD: rather than attempting to enumerate every possible pathology, the model learns a representation of normal anatomy and identifies samples whose representations deviate from it\[[51](https://arxiv.org/html/2611.25194#bib.bib12),[50](https://arxiv.org/html/2611.25194#bib.bib13),[7](https://arxiv.org/html/2611.25194#bib.bib11)\]\.

The central problem is therefore to determine what constitutes a useful representation of normality\. This question is more subtle in medical imaging than in natural\-image anomaly detection\. Normal anatomical structures exhibit substantial variation across patients, imaging modalities, acquisition protocols, anatomical regions, and disease prevalence\. At the same time, abnormalities may manifest at very different levels of abstraction,*e\.g*\.a lesion can differ from its surroundings through local texture, alter the spatial configuration of an organ, or disrupt anatomical context\. Consequently, a representation that is sensitive to low\-level appearance may miss contextual abnormalities, while an overly invariant representation may discard precisely the subtle local differences required for detection\. The five medical domains from BMAD\[[5](https://arxiv.org/html/2611.25194#bib.bib15)\]provide a concrete illustration of how anomalies broadly vary, and demonstrate how methods that perform well in one domain do not necessarily transfer to another\.

![Refer to caption](https://arxiv.org/html/2611.25194v1/ijepa_pipeline.png)Figure 1:Overview of MADJEPA\. Our pipeline is divided into three training stages:\\raisebox\{\-0\.85pt\}\{1\}⃝self\-supervised training via I\-JEPA;\\raisebox\{\-0\.85pt\}\{2\}⃝target encoder distillation to a randomly\-initialized student;\\raisebox\{\-0\.85pt\}\{3\}⃝A: prompt\-based pseudo\-anomaly generation, and B: normal vs\. pseudo\-anomaly classification\. Together, these training stages build a strong and calibrated representation of normal data\. Best viewed in color\.To address the above challenges, we tap into world models\[[2](https://arxiv.org/html/2611.25194#bib.bib1),[4](https://arxiv.org/html/2611.25194#bib.bib3),[20](https://arxiv.org/html/2611.25194#bib.bib2),[29](https://arxiv.org/html/2611.25194#bib.bib4)\], specifically I\-JEPA \(Image\-based Joint\-Embedding Predictive Architecture\)\[[2](https://arxiv.org/html/2611.25194#bib.bib1)\], which aim to learn robust semantic representations instead of surface details learned by reconstructing raw input pixels\. By predicting target features directly in an abstract latent space, a world model can filter out high\-frequency noise, background variations, and irrelevant image artifacts, thus focusing on the true structural priors of healthy tissue\. Because the network is trained as a predictive world model purely on normal anatomical structures, unseen anomalous patterns \(*e\.g*\.tumors, lesions, or out\-of\-distribution structures\) break the learned latent space dependencies, generating an elevated latent prediction error that serves as a natural proxy for unsupervised MAD\.

I\-JEPA learns self\-supervised representations via spatial context\-block prediction without relying on handcrafted data augmentations \(such as patch interpolation\[[55](https://arxiv.org/html/2611.25194#bib.bib5)\], color jittering\[[56](https://arxiv.org/html/2611.25194#bib.bib6)\]or synthetic intensity alterations\[[23](https://arxiv.org/html/2611.25194#bib.bib7)\]\) that can distort subtle pathological features or introduce artificial domain biases\. While this increases the faithfulness of the representation for normal training data, it can lead to a poorly calibrated representation when normal training data is insufficient to comprehensively represent the full range of healthy anatomical structures\. As a result, the model can either label unrecognized normal tissue as abnormal, or may over\-generalize to represent subtle anomalies\. To this end, we propose MADJEPA, amedicalanomalydetector based on an enhanced multi\-stagejoint\-embeddingpredictivearchitecture \(JEPA\)\. As illustrated in Figure[1](https://arxiv.org/html/2611.25194#S1.F1), we design a three\-stage training pipeline to produce a robust medical anomaly detector\. In stage\\raisebox\{\-0\.85pt\}\{1\}⃝, we perform self\-supervised training via I\-JEPA to learn a strong representation of normal training data\. In stage\\raisebox\{\-0\.85pt\}\{2\}⃝, we distill the target encoder into a lighter randomly\-initialized student, further harnessing the discrepancy between latent space predictions in stages\\raisebox\{\-0\.85pt\}\{1\}⃝and\\raisebox\{\-0\.85pt\}\{2\}⃝to modulate over\-generalization of I\-JEPA to abnormal data\. In stage\\raisebox\{\-0\.85pt\}\{3\}⃝, we generate pseudo\-anomalies and learn to discriminate normal tissue from pseudo\-anomalies, enabling the calibration of anomaly predictions by learning a tight boundary around the normal data manifold\.

We conduct comprehensive experiments on six datasets covering various medical image domains \(e\.g\. MRI, OCT, X\-ray, histopathology\), which form the recently proposed BMAD benchmark\[[5](https://arxiv.org/html/2611.25194#bib.bib15)\]\. By comparing our MADJEPA with numerous state\-of\-the\-art methods\[[50](https://arxiv.org/html/2611.25194#bib.bib13),[1](https://arxiv.org/html/2611.25194#bib.bib51),[63](https://arxiv.org/html/2611.25194#bib.bib52),[17](https://arxiv.org/html/2611.25194#bib.bib53),[48](https://arxiv.org/html/2611.25194#bib.bib54),[40](https://arxiv.org/html/2611.25194#bib.bib17),[41](https://arxiv.org/html/2611.25194#bib.bib55),[49](https://arxiv.org/html/2611.25194#bib.bib56),[22](https://arxiv.org/html/2611.25194#bib.bib18),[62](https://arxiv.org/html/2611.25194#bib.bib57),[21](https://arxiv.org/html/2611.25194#bib.bib59),[46](https://arxiv.org/html/2611.25194#bib.bib19),[39](https://arxiv.org/html/2611.25194#bib.bib60),[28](https://arxiv.org/html/2611.25194#bib.bib61),[47](https://arxiv.org/html/2611.25194#bib.bib62),[36](https://arxiv.org/html/2611.25194#bib.bib63),[19](https://arxiv.org/html/2611.25194#bib.bib30)\], we find that our carefully designed multi\-stage world model, MADJEPA, attains competitive results, ranking first on three datasets, while being the only method surpassing the80%80\\%threshold on all six datasets\. Furthermore, our ablation studies confirm the utility of each training stage, indicating that I\-JEPA alone is not sufficient to obtain competitive results on the MAD task\.

In summary, our contribution is threefold:

- •An enhanced multi\-stage joint\-embedding predictive architecture, where multiple training stages are integrated to obtain a robust and calibrated representation for MAD\.
- •Comprehensive experiments on six datasets comprising multiple medical imaging domains, where the proposed MADJEPA exhibits competitive results, often ranking first or second when compared with state\-of\-the\-art methods\.
- •Ablation studies on the proposed training stages, confirming that each stage plays a significant role in boosting performance\.

## 2Related Work

Prior work on medical anomaly detection can be grouped into four research directions: reconstruction\-based methods that capture deviations from learned normal anatomy\[[69](https://arxiv.org/html/2611.25194#bib.bib42),[65](https://arxiv.org/html/2611.25194#bib.bib43),[7](https://arxiv.org/html/2611.25194#bib.bib11),[15](https://arxiv.org/html/2611.25194#bib.bib40),[58](https://arxiv.org/html/2611.25194#bib.bib41),[44](https://arxiv.org/html/2611.25194#bib.bib44),[8](https://arxiv.org/html/2611.25194#bib.bib45),[6](https://arxiv.org/html/2611.25194#bib.bib46),[12](https://arxiv.org/html/2611.25194#bib.bib47),[10](https://arxiv.org/html/2611.25194#bib.bib29)\], foundation or vision\-language models, some of which enable few\-shot and zero\-shot detection\[[66](https://arxiv.org/html/2611.25194#bib.bib48),[31](https://arxiv.org/html/2611.25194#bib.bib31),[42](https://arxiv.org/html/2611.25194#bib.bib34),[32](https://arxiv.org/html/2611.25194#bib.bib49),[68](https://arxiv.org/html/2611.25194#bib.bib36),[27](https://arxiv.org/html/2611.25194#bib.bib50)\], unified learning frameworks that generalize across domains and modalities\[[68](https://arxiv.org/html/2611.25194#bib.bib36),[27](https://arxiv.org/html/2611.25194#bib.bib50)\], and multi\-task learning strategies for building shared, generalizable representations\[[33](https://arxiv.org/html/2611.25194#bib.bib38),[56](https://arxiv.org/html/2611.25194#bib.bib6),[14](https://arxiv.org/html/2611.25194#bib.bib39)\]\.

Medical anomaly detection has traditionally been dominated by reconstruction\-based methods\[[69](https://arxiv.org/html/2611.25194#bib.bib42),[65](https://arxiv.org/html/2611.25194#bib.bib43),[7](https://arxiv.org/html/2611.25194#bib.bib11),[15](https://arxiv.org/html/2611.25194#bib.bib40),[58](https://arxiv.org/html/2611.25194#bib.bib41)\], which learn only from healthy samples and identify anomalies from reconstruction errors,*i\.e*\.treating deviations in the reconstructed output as anomaly cues\. More recently, leveraging diffusion models, which showcase strong generative capabilities\[[18](https://arxiv.org/html/2611.25194#bib.bib58)\], the healthy counterparts of anomalous samples are synthesized, the difference between the two being analyzed to detect anomalous regions\[[44](https://arxiv.org/html/2611.25194#bib.bib44),[8](https://arxiv.org/html/2611.25194#bib.bib45),[6](https://arxiv.org/html/2611.25194#bib.bib46),[12](https://arxiv.org/html/2611.25194#bib.bib47),[10](https://arxiv.org/html/2611.25194#bib.bib29)\]\. Most notably, Behrendt*et al*\.\[[9](https://arxiv.org/html/2611.25194#bib.bib28)\]improved the reconstruction robustness by estimating the anomaly likelihood from multiple pseudo\-healthy samples, then computing the Mahalanobis distance at the pixel\-level\. In order to preserve the healthy anatomical structure, Beizaee*et al*\.\[[10](https://arxiv.org/html/2611.25194#bib.bib29)\]proposed masked latent diffusion to specifically carry the diffusion process only on the anomalous regions\.

With the emergence of generic vision\-language models \(VLMs\), particularly CLIP\[[45](https://arxiv.org/html/2611.25194#bib.bib64)\], researchers have started to employ such models for anomaly detection, even in few\-shot and zero\-shot regimes\[[66](https://arxiv.org/html/2611.25194#bib.bib48),[31](https://arxiv.org/html/2611.25194#bib.bib31),[64](https://arxiv.org/html/2611.25194#bib.bib32),[52](https://arxiv.org/html/2611.25194#bib.bib33),[67](https://arxiv.org/html/2611.25194#bib.bib35),[42](https://arxiv.org/html/2611.25194#bib.bib34),[32](https://arxiv.org/html/2611.25194#bib.bib49)\]\. Among these, Huang*et al*\.\[[31](https://arxiv.org/html/2611.25194#bib.bib31)\]inserted residual adapters at multiple CLIP encoder stages and combined zero\-shot text\-based scoring with few\-shot memory\-bank comparison\. Zhang*et al*\.\[[64](https://arxiv.org/html/2611.25194#bib.bib32)\]proposed MediCLIP, fine\-tuning CLIP on synthetic anomalous samples that mimic clinically plausible patterns, using adapted features from multiple levels and learnable prompt tokens, obtaining substantial gains using as much as1%1\\%of the training set\. Shiri*et al*\.\[[52](https://arxiv.org/html/2611.25194#bib.bib33)\]introduced MadCLIP, a dual\-branch adapter trained contrastively with SigLIP loss to explicitly separate normal and abnormal feature subspaces\. Ma*et al*\.\[[42](https://arxiv.org/html/2611.25194#bib.bib34)\]proposed AA\-CLIP, a two\-stage sequential adaptation of both text and visual encoders\. It builds anomaly\-aware text anchors that capture distinct normal and abnormal semantics, before aligning patch\-level image features with these to facilitate the localization of anomalies\. These ideas have also been extended beyond binary anomaly detection to fine\-grained disease classification, as demonstrated by UltraAD for ultrasound imaging\[[67](https://arxiv.org/html/2611.25194#bib.bib35)\]\.

More broadly, recent work has sought to improve cross\-domain generalization by training unified models across heterogeneous anomaly detection benchmarks\[[68](https://arxiv.org/html/2611.25194#bib.bib36),[27](https://arxiv.org/html/2611.25194#bib.bib50)\]or transferring learned anomaly representations to entirely unseen imaging modalities\[[43](https://arxiv.org/html/2611.25194#bib.bib37)\]\. Recently, Dalmonte*et al*\.\[[19](https://arxiv.org/html/2611.25194#bib.bib30)\]demonstrated that a frozen vision foundation model combined with lightweight learnable bottlenecks can achieve strong unsupervised anomaly detection, and thus, the encoders do not require any training\.

Another research direction relies on self\-supervised learning, where models acquire semantic representations through proxy tasks without requiring pixel\-level anomaly annotations\. Existing methods combine objectives such as rotation prediction, jigsaw solving, recolorization\[[33](https://arxiv.org/html/2611.25194#bib.bib38)\], contrastive learning, or synthetic anomaly generation\[[56](https://arxiv.org/html/2611.25194#bib.bib6),[14](https://arxiv.org/html/2611.25194#bib.bib39)\]\. However, these proxy tasks are typically optimized independently or combined only through score fusion at inference time, limiting their ability to learn complementary representations jointly\. Similar to our method, Le*et al*\.\[[38](https://arxiv.org/html/2611.25194#bib.bib65)\]employed I\-JEPA on medical images\. However, their focus is on segmentation rather than detection, with a limited evaluation on chest X\-rays\. In contrast, we integrate I\-JEPA in a multi\-stage training pipeline, which brings significant gains over I\-JEPA alone\.

## 3Method

As illustrated in Figure[1](https://arxiv.org/html/2611.25194#S1.F1), the proposed medical anomaly detector is based on an enhanced multi\-stage joint\-embedding predictive architecture\. The training process is specifically divided into three stages, and the underlying architecture is based on a vision transformer \(ViT\)\[[24](https://arxiv.org/html/2611.25194#bib.bib8)\]\. MADJEPA predicts both image\-level and pixel\-level anomaly scores based on discrepancies among latent representations\.

Architecture\.In stage\\raisebox\{\-0\.85pt\}\{1\}⃝, we employ a ViT\-S architecture for two of the three I\-JEPA components, namely the context encoder and the target encoder, and a custom ViT\-based architecture for the predictor\. ViT consists of several transformer encoder layers, each comprising a multi\-head self\-attention module and a position\-wise feed\-forward network with GeLU activation, connected via skip\-connections and pre\-layer normalization\. The context encoder and the target encoder employ Rotary Positional Embeddings \(RoPE\)\[[54](https://arxiv.org/html/2611.25194#bib.bib67)\], while the predictor leverages sinusoidal positional embeddings\[[57](https://arxiv.org/html/2611.25194#bib.bib68)\]\. The input image is resized to a fixed resolutionH×WH\\times W, and partitioned intoK=\(H/P\)×\(W/P\)K=\(H/P\)\\times\(W/P\)non\-overlapping patches, each patch being projected into add\-dimensional vector via a patch encoder, implemented as a 2D convolutional layer\. Both the target encoder and the context encoder are pretrained on the LVD\-1689M\[[53](https://arxiv.org/html/2611.25194#bib.bib66)\]dataset using DINOv3\[[53](https://arxiv.org/html/2611.25194#bib.bib66)\]\. The context encoder shares the same architectural configuration as the target encoder and is initialized with the same weights\. The predictor model is pretrained on the ImageNet\-22K dataset following the I\-JEPA framework\[[2](https://arxiv.org/html/2611.25194#bib.bib1)\]\. For the student model introduced in stage\\raisebox\{\-0\.85pt\}\{2\}⃝, we employ a lighter variant of the ViT architecture, comprising a reduced number of transformer encoder layers, with randomly initialized weights\. In stage\\raisebox\{\-0\.85pt\}\{3\}⃝, the pseudo\-anomaly classifier leverages an MLP head consisting of two dense layers with GeLU activation\.

Stage 1: I\-JEPA training\.Many self\-supervised methods learn image representations by leveraging handcrafted augmentations\. However, such artificial perturbations may be detrimental to the unsupervised anomaly detection task, as they can enable the model to be less sensitive to small abnormal shifts, ultimately resulting in a failure to detect subtle anomalies\. At the same time, an effective method must remain robust to slight variations within the normal space, requiring its representations to encode semantic context and structural relationships from the normal data\. I\-JEPA\[[2](https://arxiv.org/html/2611.25194#bib.bib1)\]represents a strong candidate in this regard, as it addresses both limitations\. More specifically, without relying on synthetic augmentations, I\-JEPA learns semantically rich representations through latent space feature prediction\. By constructing prediction targets directly in the latent space, I\-JEPA learns representations that are highly semantic and less sensitive to pixel\-level variations, making them potentially more effective for distinguishing normal from abnormal patterns\. To this end, our first training stage is based on I\-JEPA\[[2](https://arxiv.org/html/2611.25194#bib.bib1)\]\. A predictor𝐏\\mathbf\{P\}is trained to reconstruct the target encoder’s latent space representations of masked image regions, relying only on the encoding of a limited visible context\. First, the target encoder𝐓\\mathbf\{T\}processes the complete \(unmasked\) image, generating representations in the latent space for each patch, denoted as𝒮𝐓=\{𝐭i\}i=1,K¯\\mathcal\{S\}\_\{\\mathbf\{T\}\}=\\\{\\mathbf\{t\}\_\{i\}\\\}\_\{i=\\overline\{1,K\}\}, where𝐭i∈ℝd\\mathbf\{t\}\_\{i\}\\in\\mathbb\{R\}^\{d\}is the representation of the patch at indexii\. Letℬ=⋃j=1Mℬj\\mathcal\{B\}=\\bigcup\_\{j=1\}^\{M\}\{\\mathcal\{B\}\_\{j\}\}be the union of patch index sets of theMMrandomly sampled target blocks\. Second, a partially masked image is passed to the context encoder in order to generate rich semantic representations for the patches contained in the visible context\. Subsequently, the predictor takes as input the encoded context and a mask token for each patch that needs to be predicted, and outputs per\-patch representations𝒮𝐏=\{𝐩i\}i∈ℬ\\mathcal\{S\}\_\{\\mathbf\{P\}\}=\\\{\\mathbf\{p\}\_\{i\}\\\}\_\{i\\in\\mathcal\{B\}\}, where𝐩i∈ℝd\\mathbf\{p\}\_\{i\}\\in\\mathbb\{R\}^\{d\}is the predictor representation of the patch at indexii\. The training objective enforces similarity between the latent representations generated by the predictor \(\{𝐩i\}i∈ℬj\\\{\\mathbf\{p\}\_\{i\}\\\}\_\{i\\in\\mathcal\{B\}\_\{j\}\}\) and the target encoder \(\{𝐭i\}i∈ℬj\\\{\\mathbf\{t\}\_\{i\}\\\}\_\{i\\in\\mathcal\{B\}\_\{j\}\}\) for theMMrandomly sampled target blocks\{ℬj\}j=1,M¯\\\{\\mathcal\{B\}\_\{j\}\\\}\_\{j=\\overline\{1,M\}\}\. To this end, we employ a smoothL1L\_\{1\}loss\[[26](https://arxiv.org/html/2611.25194#bib.bib69)\]between the two representations as the objective function\. For an input image𝐗\\mathbf\{X\}, the objective is formally defined as:

ℒJEPA\(𝐗\)=1M⋅\|ℬ\|⋅d∑j=1M∑i∈ℬj∑k=1dli,k,with:\\mathcal\{L\}\_\{\\text\{JEPA\}\}\(\\mathbf\{X\}\)=\\frac\{1\}\{M\\cdot\|\\mathcal\{B\}\|\\cdot d\}\\sum\_\{j=1\}^\{M\}\\sum\_\{i\\in\\mathcal\{B\}\_\{j\}\}\\sum\_\{k=1\}^\{d\}l\_\{i,k\},\\text\{with:\}\(1\)li,k=\{0\.5⋅\(𝐩i,k−𝐭i,k\)2,\|𝐩i,k−𝐭i,k\|<1\|𝐩i,k−𝐭i,k\|−0\.5,otherwise\.l\_\{i,k\}=\\begin\{cases\}0\.5\\cdot\(\\mathbf\{p\}\_\{i,k\}\-\{\\mathbf\{t\}\_\{i,k\}\}\)^\{2\},&\|\{\\mathbf\{p\}\_\{i,k\}\}\-\{\\mathbf\{t\}\_\{i,k\}\}\|<1\\\\ \|\{\\mathbf\{p\}\_\{i,k\}\}\-\{\\mathbf\{t\}\_\{i,k\}\}\|\-0\.5,&\\text\{otherwise\}\.\\end\{cases\}\(2\)We train the entire system via full fine\-tuning\. The context encoder and the predictor are optimized via gradient descent, while the target encoder is updated using an exponential moving average \(EMA\) of the context encoder weights\.

Stage 2: Target encoder distillation\.This stage leverages a lightweight ViT student which is trained to perform knowledge distillation \(KD\) from the I\-JEPA target encoder obtained in stage\\raisebox\{\-0\.85pt\}\{1\}⃝\. The integration of a distilled target encoder model into our anomaly detection pipeline is primarily motivated by the need to mitigate over\-generalization\. Since the target encoder was pretrained on a large\-scale dataset with natural images and subsequently fine\-tuned on medical data \(via stage\\raisebox\{\-0\.85pt\}\{1\}⃝\), it may exhibit a tendency to over\-generalize on anomalous regions\. To address this limitation, we introduce a student model, a lightweight ViT trained from scratch, to replicate the output of the target encoder on normal training images\. Given that the student’s weights are randomly initialized and its training is restricted exclusively to normal samples, it possesses no prior knowledge of how anomalies are represented in the latent space\. We hypothesize that, when presented with an anomalous region, the distilled student model will diverge in behavior from the more capable target encoder\. This behavioral divergence is expected to manifest as an increased discrepancy in the latent space representations, thereby serving as a reliable indicator of potentially anomalous regions\.

The student model𝐒\\mathbf\{S\}is trained to replicate the latent representations of normal samples generated by the teacher \(target encoder\)𝐓\\mathbf\{T\}\. Both models receive an unmasked image as input and produce, for each patch, an embedding vector that aggregates the semantic information contained within the corresponding image region\. The objective of the student is to produce patch\-level latent representations𝒮𝐒=\{𝐬i\}i=1,K¯\\mathcal\{S\}\_\{\\mathbf\{S\}\}=\\\{\\mathbf\{s\}\_\{i\}\\\}\_\{i=\\overline\{1,K\}\}that closely approximate those of the teacher𝒮𝐓=\{𝐭i\}i=1,K¯\\mathcal\{S\}\_\{\\mathbf\{T\}\}=\\\{\\mathbf\{t\}\_\{i\}\\\}\_\{i=\\overline\{1,K\}\}, where𝐬i∈ℝd\\mathbf\{s\}\_\{i\}\\in\\mathbb\{R\}^\{d\}is the student representation of the patch at indexii, and𝐭i∈ℝd\\mathbf\{t\}\_\{i\}\\in\\mathbb\{R\}^\{d\}is defined in stage\\raisebox\{\-0\.85pt\}\{1\}⃝\. To this end, we perform target encoder distillation using a mean squared error objective:

ℒKD​\(𝐗\)=1K​∑i=1K‖𝐬i−𝐭i‖22\.\\displaystyle\\mathcal\{L\}\_\{\\text\{KD\}\}\(\\mathbf\{X\}\)=\\frac\{1\}\{K\}\\sum\_\{i=1\}^\{K\}\\\|\\mathbf\{s\}\_\{i\}\-\\mathbf\{t\}\_\{i\}\\\|^\{2\}\_\{2\}\.\(3\)
Stage 3: Pseudo\-anomaly generation and classification\.This stage comprises two sequential phases\. In phase A, pseudo\-abnormal images are generated from the normal training images\. In phase B, a classification head is trained on top of the frozen I\-JEPA target encoder backbone to distinguish between normal and pseudo\-abnormal samples\.

*Phase A:*We generate pseudo\-abnormal samples by leveraging Qwen\-Image\[[61](https://arxiv.org/html/2611.25194#bib.bib20)\]\. Specifically, we modify a healthy sample using a prompt and a mask, via inpainting\. The prompts are specific to the medical domain of the real image that is perturbed\. We generate the inpainting masks using a random number of overlapping ellipses of varying sizes and locations\. To ensure the inpainting process targets actual tissue in images with extensive background regions, we constrain the ellipse centers within organ boundaries\. Ultimately, this randomized overlapping approach naturally yields final masks with highly diverse and complex shapes\. Examples of generated images are in the supplementary\.

*Phase B:*We train a classifier to discriminate between normal samples and pseudo\-abnormal samples synthesized in phase A\. The introduction of this classification stage is motivated by the fact that the appearance of anomalous regions is highly specific to both the medical domain and the acquisition modality, while the boundary around the normal data manifold remains too loose, and may fail to capture subtle anomalies that require a fine distinction from normal samples\. To tighten the detection boundary around the normal data manifold, we introduce a separate prediction task in which a binary classifier is trained to separate normal and pseudo\-abnormal samples\.

We employ the target encoder trained in stage\\raisebox\{\-0\.85pt\}\{1\}⃝as a frozen feature extractor and train a classifier on top of its pooled latent representation\. Specifically, the patch\-level representations𝒮𝐓=\{𝐭i\}i=1,K¯\\mathcal\{S\}\_\{\\mathbf\{T\}\}=\\\{\\mathbf\{t\}\_\{i\}\\\}\_\{i=\\overline\{1,K\}\}produced by the target encoder are aggregated via mean pooling into a compact image embedding vector:𝐭^=1K​∑i=1K𝐭i∈ℝd\\hat\{\\mathbf\{t\}\}=\\frac\{1\}\{K\}\\sum\_\{i=1\}^\{K\}\\mathbf\{t\}\_\{i\}\\in\\mathbb\{R\}^\{d\}\. This global representation is passed as input to an MLP classifier consisting of two fully\-connected layers with intermediate GeLU activation and a final sigmoid non\-linearity:y^=σ​\(MLP​\(𝐭^\)\)\\hat\{y\}=\\sigma\\left\(\\text\{MLP\}\(\\hat\{\\mathbf\{t\}\}\)\\right\)\. The classification head predicts a probability that indicates if a given input image contains a synthetic anomaly\. The model is trained on both normal and pseudo\-abnormal images using the binary cross\-entropy loss:

ℒcls​\(𝐗,y\)=−\(y⋅log⁡y^\+\(1−y\)⋅log⁡\(1−y^\)\)\.\\displaystyle\\mathcal\{L\}\_\{\\text\{cls\}\}\(\\mathbf\{X\},\\ y\)=\-\(y\\cdot\\log\\hat\{y\}\+\(1\-y\)\\cdot\\log\(1\-\\hat\{y\}\)\)\.\(4\)
Inference\.Our anomaly detection pipeline builds upon the trained system described in the three preceding stages, whose outputs are combined to produce the final image\-level and pixel\-level anomaly scores\.

In stage\\raisebox\{\-0\.85pt\}\{1\}⃝, a context block and a target block are randomly sampled, using the sampling procedure employed during training\. The entire target block is masked \(including the overlapping region between context and target blocks\), and the remaining visible patches are encoded via the context encoder\. Subsequently, the predictor processes the sequence of context embeddings and produces latent representations for the masked target patches\. Concurrently, the target encoder processes the full image to generate embedding vectors for all constituent patches\. For each patch within the sampled target block, a distance metric \(*e\.g*\.L1L\_\{1\}orL2L\_\{2\}\) is computed between the corresponding latent representations produced by the predictor and the target encoder, respectively\. This procedure is repeated acrossNNiterations with a new context block and a different target block resampled at each step\. As a result, multiple distance values are obtained per patch\. The resulting per\-patch values are aggregated into an anomaly mapAJEPAA\_\{\\text\{JEPA\}\}of size\(H/P\)×\(W/P\)\(H/P\)\\times\(W/P\)\. The map is resized to the input image resolution using bilinear interpolation, smoothed via a Gaussian blurring filter, and scaled to the\[0,1\]\[0,1\]range\. Regions exhibiting larger aggregated distances will appear as high\-intensity pixel groups, suggesting a higher likelihood of anomalous content, whereas regions with smaller distances are indicative of normal tissue\. The image\-level AUC score, in turn, is obtained by aggregating the values of the anomaly map via a pooling function \(*e\.g*\.max or mean\), producing a single anomaly score per image, denoted asaJEPAa\_\{\\text\{JEPA\}\}\.

In stage\\raisebox\{\-0\.85pt\}\{2\}⃝, the input image is provided to both the teacher and the student models, each producing an embedding vector for each patch\. A distance metric \(*e\.g*\.L1L\_\{1\}orL2L\_\{2\}\) is then computed between the corresponding patch\-level representations of the two models\. Following the same procedure outlined in stage\\raisebox\{\-0\.85pt\}\{1\}⃝, the resulting per\-patch distance values are used to construct a second anomaly mapAKDA\_\{\\text\{KD\}\}, from which an image\-level anomaly scoreaKDa\_\{\\text\{KD\}\}is derived\.

In stage\\raisebox\{\-0\.85pt\}\{3\}⃝, the full image is passed to the pseudo\-anomaly classifier, which outputs the probability that the image represents a synthetic anomaly sample \(denotedaclsa\_\{\\text\{cls\}\}\)\.

The three image\-level scores are subsequently combined via a weighted mean using coefficientsλ1\\lambda\_\{1\},λ2\\lambda\_\{2\}andλ3\\lambda\_\{3\}in order to produce the final anomaly score:

afinal=λ1⋅aJEPA\+λ2⋅aKD\+λ3⋅acls,a\_\{\\text\{final\}\}=\\lambda\_\{1\}\\cdot a\_\{\\text\{JEPA\}\}\+\\lambda\_\{2\}\\cdot a\_\{\\text\{KD\}\}\+\\lambda\_\{3\}\\cdot a\_\{\\text\{cls\}\},\(5\)where∑i=13λi=1\\sum\_\{i=1\}^\{3\}\\lambda\_\{i\}=1,λi∈\[0,1\]\\lambda\_\{i\}\\in\[0,1\]\.

Similarly, the anomaly maps generated in stage\\raisebox\{\-0\.85pt\}\{1\}⃝and\\raisebox\{\-0\.85pt\}\{2\}⃝are aggregated through a weighted combination\. The corresponding coefficients \(λ1\\lambda\_\{1\}andλ2\\lambda\_\{2\}\), introduced previously, are normalized by their sum prior to aggregation, ensuring that the resulting weights sum to11:

Afinal\\displaystyle A\_\{\\text\{final\}\}=λ1λ1\+λ2⋅AJEPA\+λ2λ1\+λ2⋅AKD\.\\displaystyle=\\frac\{\\lambda\_\{1\}\}\{\\lambda\_\{1\}\+\\lambda\_\{2\}\}\\cdot A\_\{\\text\{JEPA\}\}\+\\frac\{\\lambda\_\{2\}\}\{\\lambda\_\{1\}\+\\lambda\_\{2\}\}\\cdot A\_\{\\text\{KD\}\}\.\(6\)

## 4Experiments

MethodBraTS2021BTCV\+LiTSRESCOCT2017RSNACAMELYON16Image AUROCPixel AUROCImage AUROCPixel AUROCImage AUROCPixel AUROCImage AUROCImage AUROCImage AUROCf\-AnoGAN\[[50](https://arxiv.org/html/2611.25194#bib.bib13)\]77\.3±0\.1877\.3\\pm 0\.18NA58\.5±0\.2158\.5\\pm 0\.21NA77\.4±0\.8577\.4\\pm 0\.85NA73\.4±1\.8573\.4\\pm 1\.8555\.2±0\.0955\.2\\pm 0\.0969\.5±1\.9869\.5\\pm 1\.98GANomaly\[[1](https://arxiv.org/html/2611.25194#bib.bib51)\]74\.8±1\.9374\.8\\pm 1\.93NA54\.6±3\.0654\.6\\pm 3\.06NA52\.6±3\.9552\.6\\pm 3\.95NA70\.5±9\.9870\.5\\pm 9\.9862\.9±0\.6562\.9\\pm 0\.6554\.4±2\.5754\.4\\pm 2\.57DRAEM\[[63](https://arxiv.org/html/2611.25194#bib.bib52)\]62\.4±9\.0362\.4\\pm 9\.0382\.3±4\.0782\.3\\pm 4\.0770\.0±3\.8670\.0\\pm 3\.8687\.5±3\.2387\.5\\pm 3\.2383\.2±8\.2183\.2\\pm 8\.2186\.8±3\.1486\.8\\pm 3\.1488\.0±8\.3688\.0\\pm 8\.3667\.7±1\.7267\.7\\pm 1\.7252\.4±0\.7752\.4\\pm 0\.77UTRAD\[[17](https://arxiv.org/html/2611.25194#bib.bib53)\]82\.9±2\.3282\.9\\pm 2\.3292\.6±0\.6792\.6\\pm 0\.6755\.8±5\.6655\.8\\pm 5\.6687\.9±1\.3287\.9\\pm 1\.3289\.4±1\.9289\.4\\pm 1\.9294\.5±1\.2494\.5\\pm 1\.2496\.8±0\.5696\.8\\pm 0\.5675\.6±1\.2475\.6\\pm 1\.2470\.0±4\.6470\.0\\pm 4\.64DeepSVDD\[[48](https://arxiv.org/html/2611.25194#bib.bib54)\]87\.0±0\.6687\.0\\pm 0\.66NA54\.0±1\.8454\.0\\pm 1\.84NA74\.1±1\.2974\.1\\pm 1\.29NA76\.8±1\.3776\.8\\pm 1\.3764\.5±3\.1764\.5\\pm 3\.1761\.0±1\.8261\.0\\pm 1\.82CutPaste\[[40](https://arxiv.org/html/2611.25194#bib.bib17)\]78\.8±0\.6778\.8\\pm 0\.67NA59\.3±4\.8659\.3\\pm 4\.86NA90\.2±0\.6190\.2\\pm 0\.61NA96\.8±0\.6296\.8\\pm 0\.6282\.6±1\.2282\.6\\pm 1\.2275\.2±0\.4175\.2\\pm 0\.41SimpleNet\[[41](https://arxiv.org/html/2611.25194#bib.bib55)\]82\.5±3\.3482\.5\\pm 3\.3494\.8±1\.0494\.8\\pm 1\.0472\.3±2\.68¯\\underline\{72\.3\\pm 2\.68\}97\.5±0\.56¯\\underline\{97\.5\\pm 0\.56\}76\.2±7\.4676\.2\\pm 7\.4677\.1±4\.7677\.1\\pm 4\.7694\.7±2\.1794\.7\\pm 2\.1769\.1±1\.2769\.1\\pm 1\.2762\.4±3\.7162\.4\\pm 3\.71MKD\[[49](https://arxiv.org/html/2611.25194#bib.bib56)\]81\.5±0\.3681\.5\\pm 0\.3689\.4±0\.2489\.4\\pm 0\.2460\.7±1\.1960\.7\\pm 1\.1996\.1±0\.2796\.1\\pm 0\.2789\.0±0\.2589\.0\\pm 0\.2586\.7±0\.6586\.7\\pm 0\.6596\.7±0\.2696\.7\\pm 0\.2682\.0±0\.1282\.0\\pm 0\.1277\.5±0\.2777\.5\\pm 0\.27RD4AD\[[22](https://arxiv.org/html/2611.25194#bib.bib18)\]89\.5±0\.9189\.5\\pm 0\.9196\.5±0\.1796\.5\\pm 0\.1760\.4±1\.1760\.4\\pm 1\.1796\.0±1\.1996\.0\\pm 1\.1987\.8±0\.8787\.8\\pm 0\.8796\.2±0\.15¯\\underline\{96\.2\\pm 0\.15\}97\.3±0\.7997\.3\\pm 0\.7967\.6±1\.1167\.6\\pm 1\.1166\.8±0\.7166\.8\\pm 0\.71STFPM\[[62](https://arxiv.org/html/2611.25194#bib.bib57)\]83\.0±0\.6783\.0\\pm 0\.6795\.6±0\.1295\.6\\pm 0\.1261\.8±1\.5861\.8\\pm 1\.5891\.2±5\.5291\.2\\pm 5\.5284\.8±0\.5084\.8\\pm 0\.5094\.7±0\.5794\.7\\pm 0\.5796\.8±0\.2396\.8\\pm 0\.2372\.9±1\.9672\.9\\pm 1\.9666\.4±1\.0166\.4\\pm 1\.01PaDiM\[[21](https://arxiv.org/html/2611.25194#bib.bib59)\]79\.0±0\.3879\.0\\pm 0\.3894\.4±1\.0394\.4\\pm 1\.0350\.8±0\.6150\.8\\pm 0\.6190\.9±0\.8490\.9\\pm 0\.8475\.9±0\.5475\.9\\pm 0\.5491\.4±0\.4291\.4\\pm 0\.4291\.8±0\.9691\.8\\pm 0\.9677\.5±1\.8777\.5\\pm 1\.8767\.3±0\.3267\.3\\pm 0\.32PatchCore\[[46](https://arxiv.org/html/2611.25194#bib.bib19)\]91\.7±0\.3691\.7\\pm 0\.3697\.0±0\.0497\.0\\pm 0\.0460\.3±0\.7660\.3\\pm 0\.7696\.4±0\.1996\.4\\pm 0\.1991\.6±0\.1091\.6\\pm 0\.1096\.5±0\.10\\mathbb\{96\.5\\pm 0\.10\}98\.6±0\.0398\.6\\pm 0\.0376\.1±0\.6776\.1\\pm 0\.6769\.3±0\.2169\.3\\pm 0\.21CFA\[[39](https://arxiv.org/html/2611.25194#bib.bib60)\]84\.4±0\.8784\.4\\pm 0\.8796\.3±0\.1496\.3\\pm 0\.1462\.0±1\.0862\.0\\pm 1\.0897\.2±0\.1497\.2\\pm 0\.1469\.9±0\.2669\.9\\pm 0\.2691\.1±0\.8791\.1\\pm 0\.8779\.5±0\.5679\.5\\pm 0\.5666\.8±0\.2366\.8\\pm 0\.2365\.6±0\.1065\.6\\pm 0\.10CFLOW\[[28](https://arxiv.org/html/2611.25194#bib.bib61)\]74\.8±5\.3274\.8\\pm 5\.3293\.8±0\.6793\.8\\pm 0\.6750\.8±4\.4750\.8\\pm 4\.4792\.4±1\.1692\.4\\pm 1\.1675\.0±5\.8175\.0\\pm 5\.8193\.8±0\.5793\.8\\pm 0\.5785\.4±2\.1185\.4\\pm 2\.1171\.5±1\.4971\.5\\pm 1\.4955\.7±1\.9755\.7\\pm 1\.97CS\-Flow\[[47](https://arxiv.org/html/2611.25194#bib.bib62)\]90\.9±0\.8390\.9\\pm 0\.83NA59\.4±0\.5459\.4\\pm 0\.54NA87\.3±0\.5887\.3\\pm 0\.58NA98\.5±0\.2898\.5\\pm 0\.2883\.2±0\.4683\.2\\pm 0\.4668\.4±0\.4268\.4\\pm 0\.42P\-VQ\[[36](https://arxiv.org/html/2611.25194#bib.bib63)\]94\.3±0\.23\\mathbb\{94\.3\\pm 0\.23\}98\.4±0\.02\\mathbb\{98\.4\\pm 0\.02\}60\.6±0\.6260\.6\\pm 0\.6296\.4±0\.0696\.4\\pm 0\.0689\.0±0\.4889\.0\\pm 0\.4892\.8±0\.3592\.8\\pm 0\.3599\.3±0\.22\\mathbb\{99\.3\\pm 0\.22\}79\.2±0\.0479\.2\\pm 0\.04NAQFAE\[[19](https://arxiv.org/html/2611.25194#bib.bib30)\]94\.3±0\.18\\mathbb\{94\.3\\pm 0\.18\}NA65\.5±1\.9665\.5\\pm 1\.96NA91\.8±0\.55\\mathbb\{91\.8\\pm 0\.55\}NANA83\.8±0\.46¯\\underline\{83\.8\\pm 0\.46\}NAMADJEPA \(Ours\)94\.1±0\.06¯\\underline\{94\.1\\pm 0\.06\}97\.3±0\.07¯\\underline\{97\.3\\pm 0\.07\}84\.1±0\.20\\mathbb\{84\.1\\pm 0\.20\}99\.5±0\.15\\mathbb\{99\.5\\pm 0\.15\}88\.4±0\.1688\.4\\pm 0\.1694\.9±0\.2294\.9\\pm 0\.2296\.9±0\.2696\.9\\pm 0\.2684\.4±0\.28\\mathbb\{84\.4\\pm 0\.28\}82\.0±0\.01\\mathbb\{82\.0\\pm 0\.01\}Table 1:Anomaly detection performance of our method MADJEPA on the complete BMAD benchmark\[[5](https://arxiv.org/html/2611.25194#bib.bib15)\]\. We compare our method against a comprehensive set of state\-of\-the\-art anomaly detection techniques from the literature, achieving the best performance on three challenging datasets \(BTCV\+LiTS, RSNA, CAMELYON16\), and second\-best on one dataset \(BraTS2021\)\. MADJEPA is the only method surpassing the80%80\\%threshold on all datasets and metrics\. The best score on each dataset is highlighted inblue boldand the second\-best score is highlighted inorange underline\. We report mean image\-level and pixel\-level AUROC scores \(where pixel\-level annotations are available\), along with the corresponding standard deviations, computed across five different seeds\. Best viewed in color\.Datasets\.We evaluate MADJEPA on all six datasets from BMAD\[[5](https://arxiv.org/html/2611.25194#bib.bib15)\], a comprehensive benchmark for medical anomaly detection\. These datasets cover five distinct medical domains: brain magnetic resonance imaging \(BraTS2021\[[3](https://arxiv.org/html/2611.25194#bib.bib21)\]\), liver computed tomography \(BTCV\[[37](https://arxiv.org/html/2611.25194#bib.bib24)\]\+ LiTS\[[13](https://arxiv.org/html/2611.25194#bib.bib25)\]\), retinal optical coherence tomography \(RESC\[[30](https://arxiv.org/html/2611.25194#bib.bib22)\], OCT2017\[[35](https://arxiv.org/html/2611.25194#bib.bib23)\]\), Chest X\-ray \(RSNA\[[60](https://arxiv.org/html/2611.25194#bib.bib26)\]\) and digital histopathology \(CAMELYON16\[[25](https://arxiv.org/html/2611.25194#bib.bib27)\]\)\. Pixel\-level annotations are available for three of the six datasets \(BraTS2021, BTCV\+LiTS and RESC\), whereas image\-level annotations are provided across all six\. More details about the six datasets are provided in supplementary\. In stage\\raisebox\{\-0\.85pt\}\{3\}⃝, a pseudo\-abnormal image is generated for each normal sample in the training set, doubling the amount of training data\.

Architecture details\.In stage\\raisebox\{\-0\.85pt\}\{1\}⃝, the context and target encoders are DINOv3\[[53](https://arxiv.org/html/2611.25194#bib.bib66)\]ViT\-S/16 backbones with1212transformer layers,66attention heads, and a hidden size of384384\. The predictor uses a separate pretrained ViT with1616layers,1616heads, and a hidden size of384384\. Following the official I\-JEPA implementation\[[2](https://arxiv.org/html/2611.25194#bib.bib1)\], each image is sampled with one context block covering8585\-100%100\\%of the image, and four target blocks with a scale of1515\-20%20\\%\. In stage\\raisebox\{\-0\.85pt\}\{2\}⃝, the student is a lightweight ViT\-S/16 with66transformer layers,66attention heads, and a hidden size of384384\. Its weights are randomly initialized, with the target encoder copied from stage\\raisebox\{\-0\.85pt\}\{1\}⃝and kept frozen\. In stage\\raisebox\{\-0\.85pt\}\{3\}⃝, patch\-level representations from the frozen target encoder are mean\-pooled and projected to a128128\-dimensional representation using a linear layer with GeLU activation, followed by a second linear layer for the final pseudo\-anomaly recognition\.

Evaluation measures\.We report the standard evaluation metrics for BMAD, namely the image\-level and pixel\-level AUROC values\.

Hyperparameters\.For all training stages, we tune the hyperparameters of MADJEPA on the corresponding official validation splits\. All baselines receive similar attention regarding hyperparameter tuning\.

In our experiments, we use a patch size of 16, and resize all images to224×224224\\times 224pixels\. For stage\\raisebox\{\-0\.85pt\}\{1\}⃝, we follow the training procedure described in I\-JEPA\[[2](https://arxiv.org/html/2611.25194#bib.bib1)\], sampling four target blocks per input image and subsequently removing from the context all patches contained within the sampled boxes\. The model is fully fine\-tuned for2020\-6060epochs using AdamW with linear warm\-up followed by cosine annealing\. The learning rate is warmed up for33\-1010epochs from5×10−55\\times 10^\{\-5\}to10−310^\{\-3\}, then decayed to10−610^\{\-6\}\. We use a batch size of3232, random cropping with minimum crop scales of0\.30\.3\-0\.80\.8, and a target\-encoder momentum schedule from0\.9960\.996to1\.01\.0\. For stage\\raisebox\{\-0\.85pt\}\{2\}⃝, the student is trained for4040\-100100epochs with the same optimizer and scheduling strategy, using learning rates from10−610^\{\-6\}to a peak of10−310^\{\-3\}and a final value of10−610^\{\-6\}\. Batch sizes range from6464to512512and weight decay from0\.010\.01to0\.050\.05\. In stage\\raisebox\{\-0\.85pt\}\{3\}⃝, only the classifier is optimized with AdamW using learning rates between10−410^\{\-4\}and3×10−33\\times 10^\{\-3\}\. The learning rate is halved after three epochs without validation AUROC improvement\.

As far as the inference hyperparameters are concerned, we setN=30N=30across all datasets, and compare mean and maximum aggregation at both patch and image levels\.L1L\_\{1\}andL2L\_\{2\}distances are evaluated for patch\-level discrepancies\. All other hyperparameters, such as the task weights,λ1\\lambda\_\{1\},λ2\\lambda\_\{2\}, andλ3\\lambda\_\{3\}, as well as the Gaussian smoothing parameters, are specified in the supplementary\.

![Refer to caption](https://arxiv.org/html/2611.25194v1/fig_anomaly_localization.png)Figure 2:Qualitative visualizations of anomaly maps for the three benchmarks that have ground\-truth segmentation maps: BTCV\+LiTs, BraTS2021 and RESC\. We employ the exact examples selected by Bao et al\.\[[5](https://arxiv.org/html/2611.25194#bib.bib15)\]from the BMAD original paper\. Best viewed in color\.Main results\.In Table[1](https://arxiv.org/html/2611.25194#S4.T1), we compare MADJEPA against a wide range of state\-of\-the\-art anomaly detection methods on the complete BMAD benchmark\. For each experiment, we report the mean and standard deviation of performance across five runs, varying the seed\. MADJEPA achieves state\-of\-the\-art performance on three datasets \(BTCV\+LiTS, RSNA and CAMELYON16\), the second\-best score on BraTS2021, and competitive results on the two retinal OCT datasets, RESC and OCT2017, where it outperforms the majority of baselines\.

On the most challenging dataset, namely liver CT \(BTCV\+LiTS\), our method substantially surpasses \(\+11\.8%\+11\.8\\%\) the previous state\-of\-the\-art established by SimpleNet\[[41](https://arxiv.org/html/2611.25194#bib.bib55)\], achieving an image\-level AUROC of84\.1±0\.2084\.1\\pm 0\.20\. We observe another significant improvement for MADJEPA in the second\-most difficult dataset \(CAMELYON16\), where it attains an image\-level AUROC of82\.0±0\.0182\.0\\pm 0\.01, surpassing MKD\[[49](https://arxiv.org/html/2611.25194#bib.bib56)\]\(77\.5±0\.2777\.5\\pm 0\.27\), the strongest baseline, by4\.5%4\.5\\%\. We further note that two recent methods, P\-VQ\[[36](https://arxiv.org/html/2611.25194#bib.bib63)\]and QFAE\[[19](https://arxiv.org/html/2611.25194#bib.bib30)\], are particularly good on certain datasets,*e\.g*\.BraTS2021, while lagging far behind on others,*e\.g*\.BTCV\+LiTS, and even missing results on some challenging datasets,*e\.g*\.CAMELYON16\. Although MADJEPA is not always a top\-performer,*its results are more consistent across the board, being the only method surpassing the80%80\\%threshold on all six datasets, in terms of both image\-level and pixel\-level AUROC\.*

Qualitative results\.At the pixel level, the relative ranking of our method is consistent with the trends observed at the image level\. The strong results prove the effectiveness and versatility of MADJEPA in detecting anomalies across diverse medical imaging datasets\. In Figure[2](https://arxiv.org/html/2611.25194#S4.F2), we present some estimated anomaly maps using the proposed method\. For visualization purposes, the anomaly maps are displayed as heatmaps, with high\-intensity red regions denoting elevated anomaly likelihood and dark blue regions indicating normal tissue\.

On the BTCV\+LiTS dataset, DRAEM and RD4AD tend to underestimate lesion extent, producing smaller and more restrictive responses, whereas CFA, CFLOW, and UTRAD tend to overestimate it, generating activations that extend into the surrounding normal hepatic parenchyma\. PatchCore, STFPM, MKD, PaDiM, and SimpleNet show an intermediate behavior\. However, in the third liver example, MADJEPA correctly identifies the lesion, but also generates two additional foci of activation with no corresponding ground\-truth abnormalities, representing false\-positive responses\. Even in the presence of these isolated false\-positive activations, MADJEPA provides, overall, the most balanced visual correspondence with the ground truth, avoiding both the systematic underestimation and the marked spatial overextension observed with the competing methods\. On the BraTS2021 dataset, DRAEM and RD4AD tend to underestimate the pathological region, whereas CFA, CFLOW, and UTRAD show the most evident overestimation, with anomaly responses extending into brain regions without corresponding pathological findings\. PatchCore, STFPM, MKD, PaDiM, and SimpleNet exhibit varying degrees of deviation from the ground truth\. Compared with these methods, MADJEPA shows better spatial correspondence with the ground truth, concentrating its activation within the pathological regions, while limiting nonspecific extension into the surrounding normal brain parenchyma\. On the RESC dataset, DRAEM and RD4AD again generate the most restrictive responses, suggesting underestimation of anomaly extent, whereas CFLOW, UTRAD, and SimpleNet produce more extensive activations, consistent with overestimation\. PatchCore, STFPM, MKD, CFA, and PaDiM show an intermediate behavior\. MADJEPA does not perfectly reproduce the ground\-truth boundaries at the retinal level\. However, the regions of highest activation remain predominantly associated with the pathological changes\.

Overall, MADJEPA provides a favorable visual balance between the two major error patterns observed among competing methods \(underestimation of lesion extent and overestimation through activation of normal tissue\), while maintaining this behavior across three distinct anatomical and imaging contexts\.

DatasetTraining StagesAUROC\\raisebox\{\-0\.85pt\}\{1\}⃝\\raisebox\{\-0\.85pt\}\{2\}⃝\\raisebox\{\-0\.85pt\}\{3\}⃝Image\-levelPixel\-levelBraTS2021✓✗✗89\.389\.396\.996\.9✗✓✗82\.382\.395\.595\.5✗✗✓90\.190\.1\-✓✓✓94\.194\.197\.397\.3BTCV \+ LiTS✓✗✗83\.783\.799\.699\.6✗✓✗63\.063\.099\.499\.4✗✗✓71\.071\.0\-✓✓✓84\.284\.299\.699\.6RESC✓✗✗65\.065\.086\.486\.4✗✓✗85\.585\.595\.695\.6✗✗✓82\.182\.1\-✓✓✓88\.388\.395\.195\.1OCT2017✓✗✗71\.171\.1\-✗✓✗89\.389\.3\-✗✗✓87\.387\.3\-✓✓✓96\.896\.8\-RSNA✓✗✗81\.481\.4\-✗✓✗83\.583\.5\-✗✗✓69\.469\.4\-✓✓✓84\.684\.6\-CAMELYON16✓✗✗72\.572\.5\-✗✓✗80\.780\.7\-✗✗✓68\.668\.6\-✓✓✓82\.082\.0\-Table 2:Ablation study on the impact of each MADJEPA stage on the final model performance, measured in terms of both image\-level AUROC and pixel\-level AUROC \(where possible\)\.Ablation studies\. We first examine the contribution of each individual MADJEPA stage to the overall system performance\. This ablation study is conducted across all six datasets, with results presented in Table[2](https://arxiv.org/html/2611.25194#S4.T2)\. Across all datasets, aggregating the anomaly scores from the three stages yields higher overall performance than any individual stage alone\. The most notable example is the OCT2017 dataset, where none of the individual stages exceed90%90\\%AUROC in isolation, while aggregating the scores from the three stages allows the system to attain96\.8%96\.8\\%AUROC\.

A further observation is that the individual stage achieving the highest performance varies across datasets\. Specifically, the I\-JEPA stage attains the highest standalone performance on BTCV\+LiTS, the distillation stage achieves the best individual results on RESC, OCT2017, RSNA and CAMELYON16, while the pseudo\-anomaly classifier performs the best on BraTS2021\.

An additional noteworthy finding is that, on the three datasets where MADJEPA achieves state\-of\-the\-art performance, namely BTCV\+LiTS, RSNA and CAMELYON16, our method preserves the same or a similar ranking even when relying on the anomaly scores of a single stage alone\. Specifically, the I\-JEPA stage attains state\-of\-the\-art performance on BTCV\+LiTS with83\.7%83\.7\\%AUROC, the distilled model reaches state\-of\-the\-art performance on CAMELYON with80\.7%80\.7\\%AUROC, and the distilled model attains near state\-of\-the\-art performance on RSNA with83\.5%83\.5\\%AUROC\.

Furthermore, we analyze the impact of the number of sampled target blocks \(NN\)\. During inference, we sequentially sample multiple target blocks and aggregate the per\-patch distances between the target encoder and the predictor representations\. For an anomalous image, sampling a larger number of blocks increases the likelihood of capturing patches belonging to the anomalous region within multiple target boxes\. However, this comes at the cost of increased inference time\. To characterize this trade\-off, we examine how model performance varies with the number of sampled target blocks on two of our datasets, BraTS2021 and BTCV\+LiTS, illustrated in Figure[3](https://arxiv.org/html/2611.25194#S4.F3)\. On both datasets, we observe that performance increases only marginally beyond3030sampled blocks, such that the resulting increase in latency is not justified by the modest performance gain\. We therefore set the number of target blocksNNto3030in all experiments\.

Figure 3:Ablation on the number of sampled target blocks \(NN\) on BraTS2021 \(top\) and BTCV\+LiTS \(bottom\)\. The performance of MADJEPA increases with the number of target blocks\. However, on both datasets, beyond3030sampled blocks, the performance gain is marginal\.
## 5Conclusion

In this work, we introduced MADJEPA, a multi\-stage joint\-embedding predictive architecture for unsupervised medical anomaly detection\. Our ablation results showed that I\-JEPA alone is insufficient for anomaly detection, as its representations can either over\-penalize unseen healthy structures or become too invariant to subtle pathologies\. This is mostly due to the open\-set nature of medical anomaly detection\. MADJEPA addresses this issue with three complementary training stages: firstly, I\-JEPA establishes a semantic representation of the normal anatomy, secondly, teacher\-student distillation then introduces a discrepancy signal that curbs over generalization to abnormal structures, and thirdly, pseudo\-anomaly training calibrates the decision boundary around the normal manifold\. Together, these stages turn a general\-purpose predictive world model into a representation specifically calibrated for medical anomaly detection\.

Across all six BMAD datasets, spanning MRI, CT, OCT, X\-ray, and histopathology, MADJEPA performs competitively against state\-of\-the\-art methods, being the only method surpassing the80%80\\%threshold on all six datasets\. More broadly, these results suggest that predictive world models stand as a promising foundation for medical anomaly detection\. We hope this motivates further work on such models as an alternative to pixel\-level reconstruction and conventional self\-supervised objectives\.

## Acknowledgment

This work was supported by a grant of the Ministry of Research, Innovation and Digitization, CCCDI \- UEFISCDI, project number PN\-IV\-P7\-7\.1\-PED\-2024\-1856, within PNCDI IV\.

## References

- \[1\]S\. Akcay, A\. Atapour\-Abarghouei, and T\. P\. Breckon\(2019\)GANomaly: Semi\-supervised Anomaly Detection via Adversarial Training\.InProceedings of ACCV,pp\. 622–637\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p5.1),[Table 1](https://arxiv.org/html/2611.25194#S4.T1.3.4.1)\.
- \[2\]M\. Assran, Q\. Duval, I\. Misra, P\. Bojanowski, P\. Vincent, M\. Rabbat, Y\. LeCun, and N\. Ballas\(2023\)Self\-supervised learning from images with a joint\-embedding predictive architecture\.InProceedings of CVPR,pp\. 15619–15629\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p3.1),[§3](https://arxiv.org/html/2611.25194#S3.p2.1),[§3](https://arxiv.org/html/2611.25194#S3.p3.1),[§4](https://arxiv.org/html/2611.25194#S4.p2.1),[§4](https://arxiv.org/html/2611.25194#S4.p5.1)\.
- \[3\]U\. Baid, S\. Ghodasara, S\. Mohan, M\. Bilello, E\. Calabrese, E\. Colak, K\. Farahani, J\. Kalpathy\-Cramer, F\. C\. Kitamura, S\. Pati,et al\.\(2021\)The RSNA\-ASNR\-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification\.arXiv preprint arXiv:2107\.02314\.Cited by:[§4](https://arxiv.org/html/2611.25194#S4.p1.1)\.
- \[4\]R\. Balestriero and Y\. LeCun\(2025\)LeJEPA: Provable and scalable self\-supervised learning without the heuristics\.arXiv preprint arXiv:2511\.08544\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p3.1)\.
- \[5\]J\. Bao, H\. Sun, H\. Deng, Y\. He, Z\. Zhang, and X\. Li\(2024\)BMAD: Benchmarks for Medical Anomaly Detection\.InProceedings of CVPRW,pp\. 4042–4053\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p1.1),[§1](https://arxiv.org/html/2611.25194#S1.p2.1),[§1](https://arxiv.org/html/2611.25194#S1.p5.1),[§S2](https://arxiv.org/html/2611.25194#S2a.p1.1),[Figure 2](https://arxiv.org/html/2611.25194#S4.F2),[Figure 2](https://arxiv.org/html/2611.25194#S4.F2.4),[Table 1](https://arxiv.org/html/2611.25194#S4.T1),[Table 1](https://arxiv.org/html/2611.25194#S4.T1.7),[§4](https://arxiv.org/html/2611.25194#S4.p1.1)\.
- \[6\]M\. Baugh, H\. Reynaud, S\. N\. Marimont, S\. Cechnicka, J\. P\. Müller, G\. Tarroni, and B\. Kainz\(2025\)Image\-Conditioned Diffusion Models for Medical Anomaly Detection\.InProceedings of UNSURE,pp\. 117–127\.Cited by:[§2](https://arxiv.org/html/2611.25194#S2.p1.1),[§2](https://arxiv.org/html/2611.25194#S2.p2.1)\.
- \[7\]C\. Baur, S\. Denner, B\. Wiestler, N\. Navab, and S\. Albarqouni\(2021\)Autoencoders for Unsupervised Anomaly Segmentation in Brain MR Images: A Comparative Study\.Medical Image Analysis69,pp\. 101952\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p1.1),[§2](https://arxiv.org/html/2611.25194#S2.p1.1),[§2](https://arxiv.org/html/2611.25194#S2.p2.1)\.
- \[8\]F\. Behrendt, D\. Bhattacharya, J\. Kruger, R\. Opfer, and A\. Schlaefer\(2023\)Patched Diffusion Models for Unsupervised Anomaly Detection in Brain MRI\.InProceedings of MIDL,pp\. 1019–1032\.Cited by:[§2](https://arxiv.org/html/2611.25194#S2.p1.1),[§2](https://arxiv.org/html/2611.25194#S2.p2.1)\.
- \[9\]F\. Behrendt, D\. Bhattacharya, R\. Mieling, L\. Maack, J\. Krüger, R\. Opfer, and A\. Schlaefer\(2024\)Leveraging the Mahalanobis Distance to enhance Unsupervised Brain MRI Anomaly Detection\.InProceedings of MICCAI,pp\. 394–404\.Cited by:[§2](https://arxiv.org/html/2611.25194#S2.p2.1)\.
- \[10\]F\. Beizaee, G\. Lodygensky, C\. Desrosiers, and J\. Dolz\(2025\)MAD\-AD: Masked diffusion for unsupervised brain anomaly detection\.InProceedings of IPMI,pp\. 139–153\.Cited by:[§2](https://arxiv.org/html/2611.25194#S2.p1.1),[§2](https://arxiv.org/html/2611.25194#S2.p2.1)\.
- \[11\]C\. I\. Bercea, B\. Wiestler, D\. Rueckert, and J\. A\. Schnabel\(2023\)Generalizing Unsupervised Anomaly Detection: Towards Unbiased Pathology Screening\.InProceedings of MIDL,pp\. 39–52\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p1.1)\.
- \[12\]Y\. Bi, L\. Huang, R\. Clarenbach, R\. Ghotbi, A\. Karlas, N\. Navab, and Z\. Jiang\(2025\)Synomaly noise and multi\-stage diffusion: A novel approach for unsupervised anomaly detection in medical images\.Medical Image Analysis105,pp\. 103737\.Cited by:[§2](https://arxiv.org/html/2611.25194#S2.p1.1),[§2](https://arxiv.org/html/2611.25194#S2.p2.1)\.
- \[13\]P\. Bilic, P\. Christ, H\. B\. Li, E\. Vorontsov, A\. Ben\-Cohen, G\. Kaissis, A\. Szeskin, C\. Jacobs, G\. E\. H\. Mamani, G\. Chartrand,et al\.\(2023\)The Liver Tumor Segmentation Benchmark \(LiTS\)\.Medical image analysis84,pp\. 102680\.Cited by:[§4](https://arxiv.org/html/2611.25194#S4.p1.1)\.
- \[14\]B\. Bozorgtabar, D\. Mahapatra, and J\. Thiran\(2023\)AMAE: Adaptation of Pre\-trained Masked Autoencoder for Dual\-Distribution Anomaly Detection in Chest X\-Rays\.InProceedings of MICCAI,pp\. 195–205\.Cited by:[§2](https://arxiv.org/html/2611.25194#S2.p1.1),[§2](https://arxiv.org/html/2611.25194#S2.p5.1)\.
- \[15\]Y\. Cai, H\. Chen, and K\. Cheng\(2024\)Rethinking Autoencoders for Medical Anomaly Detection from A Theoretical Perspective\.InProceedings of MICCAI,pp\. 544–554\.Cited by:[§2](https://arxiv.org/html/2611.25194#S2.p1.1),[§2](https://arxiv.org/html/2611.25194#S2.p2.1)\.
- \[16\]Y\. Cai, W\. Zhang, H\. Chen, and K\. Cheng\(2025\)MedIAnomaly: A Comparative Study of Anomaly Detection in Medical Images\.Medical Image Analysis102,pp\. 103500\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p1.1)\.
- \[17\]L\. Chen, Z\. You, N\. Zhang, J\. Xi, and X\. Le\(2022\)UTRAD: Anomaly detection and localization with U\-Transformer\.Neural Networks147,pp\. 53–62\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p5.1),[Table 1](https://arxiv.org/html/2611.25194#S4.T1.3.6.1)\.
- \[18\]F\. Croitoru, V\. Hondru, R\. T\. Ionescu, and M\. Shah\(2023\)Diffusion models in vision: a survey\.IEEE Transactions on Pattern Analysis and Machine Intelligence45\(9\),pp\. 10850–10869\.Cited by:[§2](https://arxiv.org/html/2611.25194#S2.p2.1)\.
- \[19\]F\. Dalmonte, E\. Bayar, E\. Akbas, and M\. Georgescu\(2026\)Q\-Former Autoencoder: A modern framework for medical anomaly detection\.InProceedings of WACV,pp\. 7985–7995\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p5.1),[§2](https://arxiv.org/html/2611.25194#S2.p4.1),[Table 1](https://arxiv.org/html/2611.25194#S4.T1.3.19.1),[§4](https://arxiv.org/html/2611.25194#S4.p8.1)\.
- \[20\]A\. Dawid and Y\. LeCun\(2024\)Introduction to latent variable energy\-based models: a path toward autonomous machine intelligence\.Journal of Statistical Mechanics: Theory and Experiment2024\(10\),pp\. 104011\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p3.1)\.
- \[21\]T\. Defard, A\. Setkov, A\. Loesch, and R\. Audigier\(2021\)PaDiM: A Patch Distribution Modeling Framework for Anomaly Detection and Localization\.InProceedings of ICPR,pp\. 475–489\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p5.1),[Table 1](https://arxiv.org/html/2611.25194#S4.T1.3.13.1)\.
- \[22\]H\. Deng and X\. Li\(2022\)Anomaly Detection via Reverse Distillation From One\-Class Embedding\.InProceedings of CVPR,pp\. 9737–9746\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p5.1),[Table 1](https://arxiv.org/html/2611.25194#S4.T1.3.11.1)\.
- \[23\]H\. Deng and X\. Li\(2022\)Self\-supervised anomaly detection with random\-shape pseudo\-outliers\.InProceedings of EMBC,pp\. 4768–4772\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p4.1)\.
- \[24\]A\. Dosovitskiy, L\. Beyer, A\. Kolesnikov,et al\.\(2021\)An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale\.InProceedings of ICLR,Cited by:[§3](https://arxiv.org/html/2611.25194#S3.p1.1)\.
- \[25\]B\. Ehteshami Bejnordi, M\. Veta, P\. Johannes van Diest, B\. Van Ginneken, N\. Karssemeijer, G\. Litjens, J\. A\.W\.M\. Van Der Laak, C\. Consortium, M\. Hermsen, Q\. F\. Manson,et al\.\(2017\)Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer\.The Journal of the American Medical Association318\(22\),pp\. 2199–2210\.Cited by:[§4](https://arxiv.org/html/2611.25194#S4.p1.1)\.
- \[26\]R\. Girshick\(2015\)Fast R\-CNN\.InProceedings of ICCV,pp\. 1440–1448\.Cited by:[§3](https://arxiv.org/html/2611.25194#S3.p3.1)\.
- \[27\]Z\. Gu, B\. Zhu, G\. Zhu, Y\. Chen, M\. Tang, and J\. Wang\(2025\)UniVAD: A Training\-Free Unified Model for Few\-Shot Visual Anomaly Detection\.InProceedings of CVPR,pp\. 15194–15203\.Cited by:[§2](https://arxiv.org/html/2611.25194#S2.p1.1),[§2](https://arxiv.org/html/2611.25194#S2.p4.1)\.
- \[28\]D\. Gudovskiy, S\. Ishizaka, and K\. Kozuka\(2022\)CFLOW\-AD: Real\-Time Unsupervised Anomaly Detection with Localization via Conditional Normalizing Flows\.InProceedings of WACV,pp\. 1819–1828\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p5.1),[Table 1](https://arxiv.org/html/2611.25194#S4.T1.3.16.1)\.
- \[29\]D\. Ha and J\. Schmidhuber\(2018\)Recurrent world models facilitate policy evolution\.InProceedings of NeurIPS,pp\. 2455–2467\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p3.1)\.
- \[30\]J\. Hu, Y\. Chen, and Z\. Yi\(2019\)Automated segmentation of macular edema in OCT using deep neural networks\.Medical Image Analysis55,pp\. 216–227\.Cited by:[§4](https://arxiv.org/html/2611.25194#S4.p1.1)\.
- \[31\]C\. Huang, A\. Jiang, J\. Feng, Y\. Zhang, X\. Wang, and Y\. Wang\(2024\)Adapting visual\-language models for generalizable anomaly detection in medical images\.InProceedings of CVPR,pp\. 11375–11385\.Cited by:[§2](https://arxiv.org/html/2611.25194#S2.p1.1),[§2](https://arxiv.org/html/2611.25194#S2.p3.1)\.
- \[32\]J\. Jeong, Y\. Zou, T\. Kim, D\. Zhang, A\. Ravichandran, and O\. Dabeer\(2023\)WinCLIP: Zero\-/Few\-Shot Anomaly Classification and Segmentation\.InProceedings of CVPR,pp\. 19606–19616\.Cited by:[§2](https://arxiv.org/html/2611.25194#S2.p1.1),[§2](https://arxiv.org/html/2611.25194#S2.p3.1)\.
- \[33\]L\. Jézéquel, N\. Vu, J\. Beaudet, and A\. Histace\(2023\)Efficient Anomaly Detection Using Self\-Supervised Multi\-Cue Tasks\.IEEE Transactions on Image Processing32\(\),pp\. 807–821\.Cited by:[§2](https://arxiv.org/html/2611.25194#S2.p1.1),[§2](https://arxiv.org/html/2611.25194#S2.p5.1)\.
- \[34\]A\. Kascenas, P\. Sanchez, P\. Schrempf, C\. Wang, W\. Clackett, S\. S\. Mikhael, J\. P\. Voisey, K\. Goatman, A\. Weir, N\. Pugeault, S\. A\. Tsaftaris, and A\. Q\. O’Neil\(2023\)The Role of Noise in Denoising Models for Anomaly Detection in Medical Images\.Medical Image Analysis90,pp\. 102963\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p1.1)\.
- \[35\]D\. S\. Kermany, M\. Goldbaum, W\. Cai, C\. C\.S\. Valentim, H\. Liang, S\. L\. Baxter, A\. McKeown, G\. Yang, X\. Wu, F\. Yan,et al\.\(2018\)Identifying medical diagnoses and treatable diseases by image\-based deep learning\.Cell172\(5\),pp\. 1122–1131\.Cited by:[§4](https://arxiv.org/html/2611.25194#S4.p1.1)\.
- \[36\]T\. Kim, Y\. Lee, I\. Jeong, S\. Ham, and S\. S\. Woo\(2024\)Patch\-wise vector quantization for unsupervised medical anomaly detection\.Pattern Recognition Letters184,pp\. 205–211\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p5.1),[Table 1](https://arxiv.org/html/2611.25194#S4.T1.3.18.1),[§4](https://arxiv.org/html/2611.25194#S4.p8.1)\.
- \[37\]B\. Landman, Z\. Xu, J\. Igelsias, M\. Styner, T\. Langerak, and A\. Klein\(2015\)Multi\-Atlas Labeling Beyond the Cranial Vault–Workshop and Challenge\.InProceedings of BCV,pp\. 12\.Cited by:[§4](https://arxiv.org/html/2611.25194#S4.p1.1)\.
- \[38\]K\. D\. D\. Le, N\. N\. Son, G\. L\. Nguyen, and H\. X\. Huynh\(2026\)Seg\-JEPA: Joint\-Embedding Predictive Architecture for Self\-Supervised Medical Image Segmentation\.InProceedings of ACIIDS,pp\. 86–101\.Cited by:[§2](https://arxiv.org/html/2611.25194#S2.p5.1)\.
- \[39\]S\. Lee, S\. Lee, and B\. Song\(2022\)CFA: Coupled\-Hypersphere\-Based Feature Adaptation for Target\-Oriented Anomaly Localization\.IEEE Access10,pp\. 78446–78454\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p5.1),[Table 1](https://arxiv.org/html/2611.25194#S4.T1.3.15.1)\.
- \[40\]C\. Li, K\. Sohn, J\. Yoon, and T\. Pfister\(2021\)CutPaste: Self\-Supervised Learning for Anomaly Detection and Localization\.InProceedings of CVPR,pp\. 9664–9674\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p5.1),[Table 1](https://arxiv.org/html/2611.25194#S4.T1.3.8.1)\.
- \[41\]Z\. Liu, Y\. Zhou, Y\. Xu, and Z\. Wang\(2023\)SimpleNet: A Simple Network for Image Anomaly Detection and Localization\.InProceedings of CVPR,pp\. 20402–20411\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p5.1),[Table 1](https://arxiv.org/html/2611.25194#S4.T1.3.9.1),[§4](https://arxiv.org/html/2611.25194#S4.p8.1)\.
- \[42\]W\. Ma, X\. Zhang, Q\. Yao, F\. Tang, C\. Wu, Y\. Li, R\. Yan, Z\. Jiang, and S\. K\. Zhou\(2025\)AA\-CLIP: Enhancing zero\-shot anomaly detection via anomaly\-aware CLIP\.InProceedings of CVPR,pp\. 4744–4754\.Cited by:[§2](https://arxiv.org/html/2611.25194#S2.p1.1),[§2](https://arxiv.org/html/2611.25194#S2.p3.1)\.
- \[43\]K\. Mao, P\. Wei, Y\. Lian, Y\. Wang, and N\. Zheng\(2025\)Beyond single\-modal boundary: Cross\-modal anomaly detection through visual prototype and harmonization\.InProceedings of CVPR,pp\. 9964–9973\.Cited by:[§2](https://arxiv.org/html/2611.25194#S2.p4.1)\.
- \[44\]W\. H\. L\. Pinaya, M\. S\. Graham, R\. Gray, P\. F\. da Costa, P\. Tudosiu, P\. Wright, Y\. H\. Mah, A\. D\. MacKinnon, J\. T\. Teo, R\. Jager, D\. Werring, G\. Rees, P\. Nachev, S\. Ourselin, and M\. J\. Cardoso\(2022\)Fast Unsupervised Brain Anomaly Detection and Segmentation with Diffusion Models\.InProceedings of MICCAI,pp\. 705–714\.Cited by:[§2](https://arxiv.org/html/2611.25194#S2.p1.1),[§2](https://arxiv.org/html/2611.25194#S2.p2.1)\.
- \[45\]A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark,et al\.\(2021\)Learning transferable visual models from natural language supervision\.InProceedings of ICML,pp\. 8748–8763\.Cited by:[§2](https://arxiv.org/html/2611.25194#S2.p3.1)\.
- \[46\]K\. Roth, L\. Pemula, J\. Zepeda, B\. Schölkopf, T\. Brox, and P\. Gehler\(2022\)Towards Total Recall in Industrial Anomaly Detection\.InProceedings of CVPR,pp\. 14318–14328\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p5.1),[Table 1](https://arxiv.org/html/2611.25194#S4.T1.3.14.1)\.
- \[47\]M\. Rudolph, T\. Wehrbein, B\. Rosenhahn, and B\. Wandt\(2022\)Fully Convolutional Cross\-Scale\-Flows for Image\-based Defect Detection\.InProceedings of WACV,pp\. 1829–1838\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p5.1),[Table 1](https://arxiv.org/html/2611.25194#S4.T1.3.17.1)\.
- \[48\]L\. Ruff, R\. Vandermeulen, N\. Goernitz, L\. Deecke, S\. A\. Siddiqui, A\. Binder, E\. Müller, and M\. Kloft\(2018\)Deep One\-Class Classification\.InProceedings of ICML,pp\. 4393–4402\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p5.1),[Table 1](https://arxiv.org/html/2611.25194#S4.T1.3.7.1)\.
- \[49\]M\. Salehi, N\. Sadjadi, S\. Baselizadeh, M\. H\. Rohban, and H\. R\. Rabiee\(2021\)Multiresolution Knowledge Distillation for Anomaly Detection\.InProceedings of CVPR,pp\. 14897–14907\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p5.1),[Table 1](https://arxiv.org/html/2611.25194#S4.T1.3.10.1),[§4](https://arxiv.org/html/2611.25194#S4.p8.1)\.
- \[50\]T\. Schlegl, P\. Seeböck, S\. M\. Waldstein, G\. Langs, and U\. Schmidt\-Erfurth\(2019\)f\-AnoGAN: Fast Unsupervised Anomaly Detection with Generative Adversarial Networks\.Medical Image Analysis54,pp\. 30–44\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p1.1),[§1](https://arxiv.org/html/2611.25194#S1.p5.1),[Table 1](https://arxiv.org/html/2611.25194#S4.T1.3.3.1)\.
- \[51\]T\. Schlegl, P\. Seeböck, S\. M\. Waldstein, U\. Schmidt\-Erfurth, and G\. Langs\(2017\)Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery\.InProceedings of IPMI,pp\. 146–157\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p1.1)\.
- \[52\]M\. Shiri, C\. Beyan, and V\. Murino\(2025\)MadCLIP: Few\-shot Medical Anomaly Detection with CLIP\.InProceedings of MICCAI,pp\. 416–426\.Cited by:[§2](https://arxiv.org/html/2611.25194#S2.p3.1)\.
- \[53\]O\. Siméoni, H\. V\. Vo, M\. Seitzer, F\. Baldassarre, M\. Oquab, C\. Jose, V\. Khalidov, M\. Szafraniec, S\. E\. Yi, M\. Ramamonjisoa,et al\.\(2026\)DINOv3\.Transactions on Machine Learning Research\.Cited by:[§3](https://arxiv.org/html/2611.25194#S3.p2.1),[§4](https://arxiv.org/html/2611.25194#S4.p2.1)\.
- \[54\]J\. Su, M\. Ahmed, Y\. Lu, S\. Pan, W\. Bo, and Y\. Liu\(2024\)RoFormer: Enhanced transformer with rotary position embedding\.Neurocomputing568,pp\. 127063\.Cited by:[§3](https://arxiv.org/html/2611.25194#S3.p2.1)\.
- \[55\]J\. Tan, B\. Hou, J\. Batten, H\. Qiu, and B\. Kainz\(2022\)Detecting outliers with foreign patch interpolation\.Machine Learning for Biomedical Imaging1,pp\. 1–27\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p4.1)\.
- \[56\]Y\. Tian, F\. Liu, G\. Pang, Y\. Chen, Y\. Liu, J\. W\. Verjans, R\. Singh, and G\. Carneiro\(2023\)Self\-supervised pseudo multi\-class pre\-training for unsupervised anomaly detection and segmentation in medical images\.Medical Image Analysis90,pp\. 102930\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p4.1),[§2](https://arxiv.org/html/2611.25194#S2.p1.1),[§2](https://arxiv.org/html/2611.25194#S2.p5.1)\.
- \[57\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.InProceedings of NeurIPS,Vol\.30\.Cited by:[§3](https://arxiv.org/html/2611.25194#S3.p2.1)\.
- \[58\]S\. Walluscheck, A\. Gerken, I\. Galinovic, K\. Villringer, J\. B\. Fiebach, J\. Klein, and S\. Heldmann\(2024\)Generative adversarial network\-based reconstruction of healthy anatomy for anomaly detection in brain CT scans\.Journal of Medical Imaging11\(4\),pp\. 044508\.Cited by:[§2](https://arxiv.org/html/2611.25194#S2.p1.1),[§2](https://arxiv.org/html/2611.25194#S2.p2.1)\.
- \[59\]M\. Wang, T\. Lin, L\. Wang,et al\.\(2023\)Uncertainty\-inspired open set learning for retinal anomaly identification\.Nature Communications14,pp\. 6757\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p1.1)\.
- \[60\]X\. Wang, Y\. Peng, L\. Lu, Z\. Lu, M\. Bagheri, and R\. M\. Summers\(2017\)ChestX\-Ray8: Hospital\-Scale Chest X\-Ray Database and Benchmarks on Weakly\-Supervised Classification and Localization of Common Thorax Diseases\.InProceedings of CVPR,pp\. 3462–3471\.Cited by:[§4](https://arxiv.org/html/2611.25194#S4.p1.1)\.
- \[61\]C\. Wu, J\. Li, J\. Zhou, J\. Lin, K\. Gao, K\. Yan, S\. Yin, S\. Bai, X\. Xu, Y\. Chen,et al\.\(2025\)Qwen\-Image Technical Report\.arXiv preprint arXiv:2508\.02324\.Cited by:[Figure S5](https://arxiv.org/html/2611.25194#S1.F5),[Figure S5](https://arxiv.org/html/2611.25194#S1.F5.5),[§S2](https://arxiv.org/html/2611.25194#S2a.p2.1),[§3](https://arxiv.org/html/2611.25194#S3.p7.1)\.
- \[62\]S\. Yamada and K\. Hotta\(2022\)Reconstruction Student with Attention for Student\-Teacher Pyramid Matching\.arXiv preprint arXiv:2111\.15376\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p5.1),[Table 1](https://arxiv.org/html/2611.25194#S4.T1.3.12.1)\.
- \[63\]V\. Zavrtanik, M\. Kristan, and D\. Skocaj\(2021\)DRAEM \- A Discriminatively Trained Reconstruction Embedding for Surface Anomaly Detection\.InProceedings of ICCV,pp\. 8330–8339\.Cited by:[§1](https://arxiv.org/html/2611.25194#S1.p5.1),[Table 1](https://arxiv.org/html/2611.25194#S4.T1.3.5.1)\.
- \[64\]X\. Zhang, M\. Xu, D\. Qiu, R\. Yan, N\. Lang, and X\. Zhou\(2024\)MediCLIP: Adapting CLIP for Few\-shot Medical Image Anomaly Detection\.InProceedings of MICCAI,pp\. 458–468\.Cited by:[§2](https://arxiv.org/html/2611.25194#S2.p3.1)\.
- \[65\]K\. Zhou, J\. Li, W\. Luo, Z\. Li, J\. Yang, H\. Fu, J\. Cheng, J\. Liu, and S\. Gao\(2022\)Proxy\-Bridged Image Reconstruction Network for Anomaly Detection in Medical Images\.IEEE Transactions on Medical Imaging41\(3\),pp\. 582–594\.Cited by:[§2](https://arxiv.org/html/2611.25194#S2.p1.1),[§2](https://arxiv.org/html/2611.25194#S2.p2.1)\.
- \[66\]Q\. Zhou, G\. Pang, Y\. Tian, S\. He, and J\. Chen\(2024\)AnomalyCLIP: Object\-agnostic Prompt Learning for Zero\-shot Anomaly Detection\.InProceedings of ICLR,pp\. 49705–49737\.Cited by:[§2](https://arxiv.org/html/2611.25194#S2.p1.1),[§2](https://arxiv.org/html/2611.25194#S2.p3.1)\.
- \[67\]Y\. Zhou, Y\. Bi, W\. Tong, W\. Wang, N\. Navab, and Z\. Jiang\(2025\)UltraAD: Fine\-Grained Ultrasound Anomaly Classification via Few\-Shot CLIP Adaptation\.InProceedings of MICCAI,pp\. 625–635\.Cited by:[§2](https://arxiv.org/html/2611.25194#S2.p3.1)\.
- \[68\]J\. Zhu and G\. Pang\(2024\)Toward generalist anomaly detection via in\-context residual learning with few\-shot sample prompts\.InProceedings of CVPR,pp\. 17826–17836\.Cited by:[§2](https://arxiv.org/html/2611.25194#S2.p1.1),[§2](https://arxiv.org/html/2611.25194#S2.p4.1)\.
- \[69\]D\. Zimmerer, F\. Isensee, J\. Petersen, S\. Kohl, and K\. Maier\-Hein\(2019\)Unsupervised Anomaly Localization Using Variational Auto\-Encoders\.InProceedings of MICCAI,pp\. 289–297\.Cited by:[§2](https://arxiv.org/html/2611.25194#S2.p1.1),[§2](https://arxiv.org/html/2611.25194#S2.p2.1)\.

## Supplementary Material

## S1Additional Ablations

Impact of patch\-level pooling\. In Table[S1](https://arxiv.org/html/2611.25194#S1.T1), we investigate the effect of the patch\-level pooling strategy used when building the anomaly maps\. We evaluate it on the BraTS2021 and CAMELYON16 test splits\. During stage\\raisebox\{\-0\.85pt\}\{1\}⃝inference, multiple blocks are sampled and patches contained within more than one target block are consequently assigned multiple distance values\. Therefore, we experiment with both mean and maximum pooling functions for each dataset\. On BraTS2021, max pooling achieves a marginally higher score than our selected configuration \(shown in bold\) on the test set \(94\.4%94\.4\\%vs\.94\.1%94\.1\\%\)\. However, this selection was made based on validation set performance, and the test set outcome was not observed at selection time\. On CAMELYON16, mean pooling, the selected strategy, yields a better result than max pooling on the test set \(82\.0%82\.0\\%vs\.81\.9%81\.9\\%\), consistent with the selection made on the validation split\. In both cases, the differences are rather marginal\.

DatasetPatch\-levelpoolingImage\-levelAUROCPixel\-levelAUROCBraTS2021mean94\.197\.3max94\.497\.4CAMELYON16mean82\.0\-max81\.9\-Table S1:Ablation study on patch\-level pooling\. We report image\-level and pixel\-level AUROC on BraTS2021, and image\-level AUROC on CAMELYON16\. The configuration used to produce our final results, selected on the official validation split, is indicated in bold\.Impact of global pooling\.In Table[S2](https://arxiv.org/html/2611.25194#S1.T2), we examine the influence of the global \(image\-level\) pooling of the anomaly maps, on BraTS2021 and CAMELYON16 test sets\. Specifically, after constructing an anomaly map, we aggregate the pixel values into a unique score employing one of the two pooling functions: mean or maximum\. On BraTS2021, max pooling over the anomaly map yields the best result \(94\.1%94\.1\\%vs\.91\.4%91\.4\\%\), with a non\-negligible margin over mean pooling\. On CAMELYON16, this effect is even more pronounced, with mean pooling substantially outperforming maximum pooling \(82\.0%82\.0\\%vs\.78\.3%78\.3\\%\)\. On BraTS2021, since this aggregation choice affects only the image\-level score and does not alter the underlying anomaly maps, the pixel\-level AUROC remains unaffected across the two aggregation strategies\.

DatasetImage\-levelpoolingImage\-levelAUROCPixel\-levelAUROCBraTS2021mean91\.497\.3max94\.197\.3CAMELYON16mean82\.0\-max78\.3\-Table S2:Ablation study on image\-level pooling\. We report image\-level and pixel\-level AUROC on BraTS2021, and image\-level AUROC on CAMELYON16\. The configuration used to produce our final results, selected on the official validation split, is indicated in bold\.Impact of the distance metric between patch embeddings\.In Table[S3](https://arxiv.org/html/2611.25194#S1.T3), we explore the impact of the distance metric used for measuring the discrepancies between patch representations\. Choosing theL2L\_\{2\}distance consistently yields the best results on both BraTS2021 and CAMELYON16, in terms of image\-level AUROC\. For BraTS2021, the findings remain consistent on the pixel\-level metric\.

DatasetDistancemeasureImage\-levelAUROCPixel\-levelAUROCBraTS2021L1L\_\{1\}94\.196\.9𝐋𝟐\\mathbf\{L\_\{2\}\}94\.197\.3CAMELYON16L1L\_\{1\}81\.9\-𝐋𝟐\\mathbf\{L\_\{2\}\}82\.0\-Table S3:Ablation study on the distance between patch\-level representations\. We report image\-level and pixel\-level AUROC on BraTS2021, and image\-level AUROC on CAMELYON16\. The configuration used to produce our final results, selected on the official validation split, is indicated in bold\.Figure S1:Ablation study on the Gaussian blurring factorσJEPA\\sigma\_\{\\text\{JEPA\}\}for the anomaly maps generated in stage\\raisebox\{\-0\.85pt\}\{1\}⃝, evaluated on the BraTS2021 test set\. We report the image\-level AUROC \(top\) and pixel\-level AUROC \(bottom\)\. The performance of MADJEPA increases with the value ofσJEPA\\sigma\_\{\\text\{JEPA\}\}\. However, on the validation set, gains beyondσJEPA\>4\\sigma\_\{\\text\{JEPA\}\}\>4are marginal\. Consequently, we employσJEPA=4\\sigma\_\{\\text\{JEPA\}\}=4for the final results on BraTS2021\.Figure S2:Ablation study on the Gaussian blurring factorσKD\\sigma\_\{\\text\{KD\}\}for the anomaly maps generated in stage\\raisebox\{\-0\.85pt\}\{2\}⃝, evaluated on the BraTS2021 test set\. We report the image\-level AUROC \(top\) and pixel\-level AUROC \(bottom\)\. The performance of MADJEPA is only marginally affected by the variation ofσKD\\sigma\_\{\\text\{KD\}\}\. Given these diminishing gains, we setσKD=σJEPA=4\\sigma\_\{\\text\{KD\}\}=\\sigma\_\{\\text\{JEPA\}\}=4for all the experiments performed on BraTS2021\.Figure S3:Ablation study on the anomaly score weightsλ1\\lambda\_\{1\}andλ2\\lambda\_\{2\}\. We report image\-level AUROC \(top\) and pixel\-level AUROC \(bottom\) on BraTS2021 test set\. We fixλ3=0\\lambda\_\{3\}=0and, since we impose the weights to sum to11,λ1=1−λ2\\lambda\_\{1\}=1\-\\lambda\_\{2\}\. The findings show that the best performance is obtained whenλ1=λ2=0\.5\\lambda\_\{1\}=\\lambda\_\{2\}=0\.5, which is consistent with the selection we made on the official validation set\. Specifically, for all the experiments performed on BraTS2021, we employ equal weights for stage\\raisebox\{\-0\.85pt\}\{1\}⃝and stage\\raisebox\{\-0\.85pt\}\{2\}⃝anomaly scores\.Figure S4:Ablation study on the anomaly score weightλ3\\lambda\_\{3\}\. We report image\-level AUROC \(top\) and pixel\-level AUROC \(bottom\) on BraTS2021\. For this ablation, we fixλ1=λ2\\lambda\_\{1\}=\\lambda\_\{2\}\. Settingλ3=0\.1\\lambda\_\{3\}=0\.1orλ3=0\.2\\lambda\_\{3\}=0\.2yields the highest image\-level AUROC values on the test set\. The classification head does not influence the anomaly maps, so changingλ3\\lambda\_\{3\}does not modify the pixel\-level AUROC\.Impact of the Gaussian blurring factors\.As illustrated in Figure[S1](https://arxiv.org/html/2611.25194#S1.F1a), the performance of MADJEPA is considerably influenced by the value ofσJEPA\\sigma\_\{\\text\{JEPA\}\}\. In terms of image\-level AUROC, the performance increases from93\.3%93\.3\\%without Gaussian blurring \(σJEPA=0\\sigma\_\{\\text\{JEPA\}\}=0\) to94\.4%94\.4\\%whenσJEPA=5\\sigma\_\{\\text\{JEPA\}\}=5\. Regarding the pixel\-level AUROC, the gains are less pronounced, increasing from97\.0%97\.0\\%\(σJEPA=0\\sigma\_\{\\text\{JEPA\}\}=0\) to97\.4%97\.4\\%\(σJEPA=5\\sigma\_\{\\text\{JEPA\}\}=5\)\. These results highlight that the performance of MADJEPA increases withσJEPA\\sigma\_\{\\text\{JEPA\}\}\. However, on the validation set, gains beyondσJEPA\>4\\sigma\_\{\\text\{JEPA\}\}\>4are marginal\. Therefore, we setσJEPA=4\\sigma\_\{\\text\{JEPA\}\}=4for the final results reported on BraTS2021\.

Conversely, in Figure[S2](https://arxiv.org/html/2611.25194#S1.F2), we notice that the performance of MADJEPA is marginally influenced by the value ofσKD\\sigma\_\{\\text\{KD\}\}\. Considering that the gains are reduced, for convenience, we setσKD\\sigma\_\{\\text\{KD\}\}to be equal toσJEPA\\sigma\_\{\\text\{JEPA\}\}\. Consequently, we employedσKD=4\\sigma\_\{\\text\{KD\}\}=4for all the experiments we conducted on BraTS2021\.

DatasetMedical domainTrain setValidation setTest setNormalAnomalyImage sizeNormalAnomalyImage sizeNormalAnomalyImage sizeBraTS2021Brain MRI75000240×\\times2403944240×\\times2406403075240×\\times240BTCV\+LiTSLiver CT15420512×\\times5129373512×\\times512833660512×\\times512RESCRetinal OCT429701024×\\times51245701024×\\times51210417641024×\\times512OCT2017Retinal OCT263150496\-512×\\times384\-1536824496×\\times512/768242726496×\\times512\-1536RSNAChest X\-ray800001024×\\times10247014201024×\\times1024781164131024×\\times1024CAMELYON16Digital Histopathology50880256×\\times256117119256×\\times2561003994256×\\times256Table S4:Overview of the six medical imaging datasets included in BMAD benchmark\.![Refer to caption](https://arxiv.org/html/2611.25194v1/anomalies_qwen.png)Figure S5:Examples of synthetic pseudo\-anomalies generated with Qwen\-Image\[[61](https://arxiv.org/html/2611.25194#bib.bib20)\]\. These are used in training stage\\raisebox\{\-0\.85pt\}\{3\}⃝of MADJEPA\. Best viewed in color\.Impact of anomaly score weights\.We next examine the influence of the anomaly score weights, namelyλ1\\lambda\_\{1\},λ2\\lambda\_\{2\}andλ3\\lambda\_\{3\}, on the BraTS2021 dataset\. We first fixλ3=0\\lambda\_\{3\}=0and search for the optimal weighting betweenλ1\\lambda\_\{1\}andλ2\\lambda\_\{2\}\. As shown in Figure[S3](https://arxiv.org/html/2611.25194#S1.F3), the settingλ1=λ2=0\.5\\lambda\_\{1\}=\\lambda\_\{2\}=0\.5yields the best performance\. This observation aligns with the coefficients selected on the official validation split, where we setλ1\\lambda\_\{1\}=λ2=0\.45\\lambda\_\{2\}=0\.45for the final results on BraTS2021\.

Second, we fix the ratio between the first two coefficients, as identified in the previous step \(in our case,λ1=λ2\\lambda\_\{1\}=\\lambda\_\{2\}\), and we search for the optimal value of the third coefficient,λ3\\lambda\_\{3\}\. As illustrated in Figure[S4](https://arxiv.org/html/2611.25194#S1.F4), settingλ3=0\.1\\lambda\_\{3\}=0\.1orλ3=0\.2\\lambda\_\{3\}=0\.2yields the largest image\-level AUROC\. In terms of pixel\-level AUROC, the choice ofλ3\\lambda\_\{3\}does not have any impact\. This result is consistent with the anomaly score weights selected on the validation set, namely:λ1=0\.45\\lambda\_\{1\}=0\.45,λ2=0\.45\\lambda\_\{2\}=0\.45,λ3=0\.1\\lambda\_\{3\}=0\.1\. Interestingly, by further examining Figure[S4](https://arxiv.org/html/2611.25194#S1.F4), we observe that for any value ofλ3\\lambda\_\{3\}, the AUROC score does not decrease under90%90\\%\. This highlights the robustness of our method to the choice of hyperparameters\. While the values of the three coefficients matter, the performance remains competitive even when the values of the weights are potentially suboptimal\.

## S2Data Details

BMAD splits\.In Table[S4](https://arxiv.org/html/2611.25194#S1.T4), we provide a detailed overview of the official training, validation, and test splits of the BMAD benchmark\[[5](https://arxiv.org/html/2611.25194#bib.bib15)\]for all six datasets, including the distribution of samples across each split\.

Synthetic pseudo\-anomalies\.In Figure[S5](https://arxiv.org/html/2611.25194#S1.F5), we showcase some synthetic samples used in our pseudo\-anomaly classifier\. Starting from healthy samples, we apply inpainting inside the regions marked by the red contours, using Qwen\-Image\[[61](https://arxiv.org/html/2611.25194#bib.bib20)\]\.

DatasetHyperparameterValueBraTS2021Number of sampled target boxes \(NN\)30Patch\-level aggregation strategymeanImage\-level aggregation strategymaxDistance for patch\-level representationsL2L\_\{2\}JEPA weight \(λ1\\lambda\_\{1\}\)0\.45KD weight \(λ2\\lambda\_\{2\}\)0\.45Classification weight \(λ3\\lambda\_\{3\}\)0\.10σJEPA\\sigma\_\{\\text\{JEPA\}\}for Gaussian blur4σKD\\sigma\_\{\\text\{KD\}\}for Gaussian blur4BTCV\+LiTSNumber of sampled target boxes \(NN\)30Patch\-level aggregation strategymeanImage\-level aggregation strategymaxDistance for patch\-level representationsL2L\_\{2\}JEPA weight \(λ1\\lambda\_\{1\}\)0\.74KD weight \(λ2\\lambda\_\{2\}\)0\.24Classification weight \(λ3\\lambda\_\{3\}\)0\.02σJEPA\\sigma\_\{\\text\{JEPA\}\}for Gaussian blur4σKD\\sigma\_\{\\text\{KD\}\}for Gaussian blur4RESCNumber of sampled target boxes \(NN\)30Patch\-level aggregation strategymaxImage\-level aggregation strategymaxDistance for patch\-level representationsL2L\_\{2\}JEPA weight \(λ1\\lambda\_\{1\}\)0\.23KD weight \(λ2\\lambda\_\{2\}\)0\.52Classification weight \(λ3\\lambda\_\{3\}\)0\.25σJEPA\\sigma\_\{\\text\{JEPA\}\}for Gaussian blur4σKD\\sigma\_\{\\text\{KD\}\}for Gaussian blur2OCT2017Number of sampled target boxes \(NN\)30Patch\-level aggregation strategymaxImage\-level aggregation strategymaxDistance for patch\-level representationsL1L\_\{1\}JEPA weight \(λ1\\lambda\_\{1\}\)0\.45KD weight \(λ2\\lambda\_\{2\}\)0\.45Classification weight \(λ3\\lambda\_\{3\}\)0\.10σJEPA\\sigma\_\{\\text\{JEPA\}\}for Gaussian blur1σKD\\sigma\_\{\\text\{KD\}\}for Gaussian blur1RSNANumber of sampled target boxes \(NN\)30Patch\-level aggregation strategymaxImage\-level aggregation strategymeanDistance for patch\-level representationsL2L\_\{2\}JEPA weight \(λ1\\lambda\_\{1\}\)0\.29KD weight \(λ2\\lambda\_\{2\}\)0\.69Classification weight \(λ3\\lambda\_\{3\}\)0\.02σJEPA\\sigma\_\{\\text\{JEPA\}\}for Gaussian blur4σKD\\sigma\_\{\\text\{KD\}\}for Gaussian blur4CAMELYON16Number of sampled target boxes \(NN\)30Patch\-level aggregation strategymeanImage\-level aggregation strategymeanDistance for patch\-level representationsL2L\_\{2\}JEPA weight \(λ1\\lambda\_\{1\}\)0\.10KD weight \(λ2\\lambda\_\{2\}\)0\.80Classification weight \(λ3\\lambda\_\{3\}\)0\.10σJEPA\\sigma\_\{\\text\{JEPA\}\}for Gaussian blur4σKD\\sigma\_\{\\text\{KD\}\}for Gaussian blur4Table S5:Inference hyperparameters for our anomaly detection pipeline\. The values of the hyperparameters are selected using grid search on the official validation splits\.
## S3Hyperparameter Configurations

In Table[S5](https://arxiv.org/html/2611.25194#S2.T5), we provide a comprehensive list of all hyperparameters used by MADJEPA, such as the task weights or the Gaussian smoothing factorσ\\sigma\.

## S4Training and inference details

Compute environment\.All experiments are performed across two workstations, each equipped with an NVIDIA GeForce RTX 3090 GPU \(24GB VRAM\), on a system leveraging 64GB of RAM, and CPUs with 24 and 12 cores, respectively\. Each individual experiment ran on a single GPU, without model or data parallelism\.

Training time for each stage\.In Table[S6](https://arxiv.org/html/2611.25194#S4.T6), we report the training time \(in hours\) per stage for each dataset\. On most datasets, the total training time, accumulated over the three stages, is under five hours\. The only exception is OCT2017, the largest dataset from BMAD, which requires approximately seven hours of training\. On average, the most demanding stage is represented by the target encoder distillation\. This stage usually requires more training epochs due to the student model, which is randomly initialized and trained from scratch\.

Inference time details\.In stage\\raisebox\{\-0\.85pt\}\{1\}⃝, the inference time is monotonically increasing with the number of sampled blocks \(NN\)\. ForN=30N=30, the average inference time per image is0\.88580\.8858seconds, with a standard deviation of0\.00710\.0071seconds\.

DatasetStage\\raisebox\{\-0\.85pt\}\{1\}⃝Stage\\raisebox\{\-0\.85pt\}\{2\}⃝Stage\\raisebox\{\-0\.85pt\}\{3\}⃝BraTS2021<1<1h≈2\\approx 2h<1<1hBTCV\+LiTS<1<1h<1<1h<1<1hRESC<1<1h≈1\\approx 1h<1<1hOCT2017≈2\\approx 2h≈4\\approx 4h≈1\\approx 1hRSNA<1<1h≈3\\approx 3h<1<1hCAMELYON16<1<1h≈1\\approx 1h<1<1hTable S6:Training time \(in hours\) for each stage, reported per dataset\.

相似文章

注解版JEPA

Hacker News Top

联合嵌入预测架构(JEPA)用于自监督学习的逐步注解实现与解释,涵盖I-JEPA、V-JEPA和LeJEPA。

AC-JEPA:用于视觉到符号转换的锚定式跨模态联合嵌入预测架构

arXiv cs.AI

论文提出 AC-JEPA(锚定式跨模态联合嵌入预测架构),一种可与监督微调(SFT)兼容的即插即用模块,通过双向跨模态预测被掩码的中间表示来提升从视觉到符号序列的翻译能力。在光学化学结构识别(OCSR)任务上,AC-JEPA 在 WildMol Simple 上达到 76.07% 的准确率(SFT 基线为 67.40%),并在使用含噪声伪标签数据进行自训练时提升至 80.23%,而直接进行 SFT 反而导致性能下降。