Token-Based Dual-view Fusion and Adaptation of Large Vision Models for Breast Cancer Classification

Hugging Face Daily Papers Papers
breast-cancer mammography vision-transformer multi-view-learning token-fusion prompt-adaptation medical-imaging

Summary

This paper proposes a token-centric dual-view learning framework that unifies prompt-based adaptation and cross-view fusion within a frozen vision transformer to improve breast cancer classification from mammography images, achieving consistent improvements on VinDr-Mammo and CMMD datasets.

Accurate breast cancer classification from mammography requires effective integration of complementary information from craniocaudal (CC) and mediolateral oblique (MLO) views, which provide a more complete characterization of breast abnormalities. However, existing multi-view learning approaches typically rely on feature-level aggregation or single-stage cross-attention, which can entangle view-specific and shared representations and restrict interaction to limited network depths. To address these limitations, we propose a token-centric dual-view learning framework that unifies prompt-based adaptation and cross-view fusion within a frozen vision transformer backbone. The framework reformulates inter-view interaction as structured token-level communication, where dedicated fusion tokens explicitly encode bidirectional information exchange between CC and MLO views via cross-attention, serving as intermediate carriers of cross-view dependencies rather than relying on direct feature fusion. Unlike conventional methods that apply fusion at a single layer, fusion modules are inserted at multiple transformer depths, enabling progressive and repeated interaction across the encoder hierarchy. Fusion tokens are reintegrated into the token sequence and refined by subsequent transformer layers, facilitating hierarchical propagation of complementary information while preserving view-specific structure. Experiments on VinDr-Mammo and CMMD datasets demonstrate consistent improvements over linear probing, prompt-only adaptation, and conventional fusion baselines. On the VinDr-Mammo BI-RADS classification task, the framework achieves 50.40% F1-score and 0.8090 AUC, including a 0.10 AUC improvement over a dual-view fusion baseline in the binary setting. Ablation studies further validate the effectiveness of token-based fusion and multi-depth interaction design.
Original Article
View Cached Full Text

Cached at: 07/09/26, 07:41 PM

Paper page - Token-Based Dual-view Fusion and Adaptation of Large Vision Models for Breast Cancer Classification

Source: https://huggingface.co/papers/2607.06309

Abstract

A token-centric dual-view learning framework unifies prompt-based adaptation and cross-view fusion in a frozen vision transformer to improve breast cancer classification from mammography images.

Accurate breast cancer classification from mammography requires effective integration of complementary information from craniocaudal (CC) and mediolateral oblique (MLO) views, which provide a more complete characterization of breast abnormalities. However, existing multi-view learning approaches typically rely on feature-level aggregation or single-stagecross-attention, which can entangle view-specific and shared representations and restrict interaction to limited network depths. To address these limitations, we propose atoken-centric dual-view learningframework that unifiesprompt-based adaptationandcross-view fusionwithin a frozenvision transformerbackbone. The framework reformulates inter-view interaction as structured token-level communication, where dedicatedfusion tokensexplicitly encode bidirectional information exchange between CC and MLO views viacross-attention, serving as intermediate carriers of cross-view dependencies rather than relying on direct feature fusion. Unlike conventional methods that apply fusion at a single layer, fusion modules are inserted at multiple transformer depths, enabling progressive and repeated interaction across the encoder hierarchy.Fusion tokensare reintegrated into the token sequence and refined by subsequenttransformer layers, facilitatinghierarchical propagationof complementary information while preserving view-specific structure. Experiments on VinDr-Mammo and CMMD datasets demonstrate consistent improvements over linear probing, prompt-only adaptation, and conventional fusion baselines. On the VinDr-Mammo BI-RADS classification task, the framework achieves 50.40% F1-score and 0.8090 AUC, including a 0.10 AUC improvement over a dual-view fusion baseline in the binary setting. Ablation studies further validate the effectiveness of token-based fusion andmulti-depth interactiondesign.

View arXiv pageView PDFGitHub0Add to collection

Get this paper in your agent:

hf papers read 2607\.06309

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.06309 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.06309 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.06309 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding

Hugging Face Daily Papers

ClinFusion is a vision-centric multimodal large language model for holistic medical understanding that unifies 2D and 3D medical image analysis using a cascaded vision encoder. It achieves state-of-the-art results on 20 out of 24 benchmarks and outperforms proprietary models like GPT-5.2 and Gemini-3-Flash on 13 out of 16 benchmarks.

VFA: Empowering Multilingual MLLMs via Vision-Free Adaptation

arXiv cs.CL

This paper proposes Vision-Free Adaptation (VFA), a framework that enhances multilingual capabilities in multimodal large language models by merging multilingual and vision-aligned task vectors without visual data, demonstrating improved performance and data efficiency.

From 2D Grids to 1D Tokens: Reforming Shared Representations for Multimodal Image Fusion

Hugging Face Daily Papers

This paper introduces a multimodal image fusion method that uses a 1D token interface from a pretrained image tokenizer to enhance global appearance coherence while preserving local details through selective token editing (STE). Experiments on four benchmarks show state-of-the-art performance in both global coherence and local fidelity.