AuralSAM2: Enabling SAM2 Hear Through Pyramid Audio-Visual Feature Prompting

Hugging Face Daily Papers Papers
audio-visual segmentation sam2 feature-prompting cross-modal multimodal ai-research

Summary

AuralSAM2 integrates audio into SAM2 via an AuralFuser module that generates sparse and dense prompts from audio-visual features, enhancing cross-modal segmentation while maintaining interactive efficiency.

Segment Anything Model 2 (SAM2) exhibits strong generalisation for promptable segmentation in video clips; however, its integration with the audio modality remains underexplored. Existing approaches either convert audio into visual prompts (e.g., boxes) via foundation models, or inject adapters into the image encoder for audio-visual fusion. Yet both directions fall short in human-in-the-loop scenarios due to limited prompt accuracy and increased inference overhead. In particular, these adapter-based methods often suffer from audio prompt dilution, where the signal gradually weakens as it propagates through the network. In this work, we propose AuralSAM2, which integrates audio into SAM2 while largely preserving its promptable segmentation capability. Its core module, AuralFuser, fuses audio and visual features to generate sparse and dense prompts. Guided by audio and built upon SAM2's feature pyramid, these prompts propagate auditory cues across visual layers, reinforcing cross-modal influence. To further align modalities, we introduce an audio-guided contrastive loss that emphasises auditory relevance in dominant visual features. Our method achieves notable accuracy gains on public benchmarks with only minimal impact on the interactive efficiency of promptable segmentation. Our code is available at https://github.com/yyliu01/AuralSAM2.
Original Article
View Cached Full Text

Cached at: 05/18/26, 06:27 PM

Paper page - AuralSAM2: Enabling SAM2 Hear Through Pyramid Audio-Visual Feature Prompting

Source: https://huggingface.co/papers/2506.01015

Abstract

AuralSAM2 integrates audio into SAM2 through an AuralFuser module that generates sparse and dense prompts, enhancing cross-modal influence while maintaining interactive segmentation efficiency.

Segment Anything Model 2(SAM2) exhibits strong generalisation forpromptable segmentationin video clips; however, its integration with the audio modality remains underexplored. Existing approaches either convert audio into visual prompts (e.g., boxes) via foundation models, or inject adapters into the image encoder foraudio-visual fusion. Yet both directions fall short in human-in-the-loop scenarios due to limited prompt accuracy and increased inference overhead. In particular, these adapter-based methods often suffer from audio prompt dilution, where the signal gradually weakens as it propagates through the network. In this work, we propose AuralSAM2, which integrates audio into SAM2 while largely preserving itspromptable segmentationcapability. Its core module,AuralFuser, fuses audio and visual features to generate sparse and dense prompts. Guided by audio and built upon SAM2’sfeature pyramid, these prompts propagate auditory cues across visual layers, reinforcingcross-modal influence. To further align modalities, we introduce anaudio-guided contrastive lossthat emphasises auditory relevance in dominant visual features. Our method achieves notable accuracy gains on public benchmarks with only minimal impact on the interactive efficiency ofpromptable segmentation. Our code is available at https://github.com/yyliu01/AuralSAM2.

View arXiv pageView PDFGitHub9Add to collection

Get this paper in your agent:

hf papers read 2506\.01015

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### yyliu01/AuralSAM2 Updated3 days ago

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2506.01015 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2506.01015 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

When Vision Speaks for Sound

Hugging Face Daily Papers

This paper identifies that video-capable multimodal LLMs often appear to understand audio but actually rely on visual cues, a failure mode termed the audio-visual Clever Hans effect. It introduces Thud, an intervention-driven probing framework to diagnose this issue, and proposes an alignment recipe that improves audio-visual consistency by 28 percentage points.

SAM 3: Segment Anything with Concepts

Papers with Code Trending

SAM 3 introduces a unified model for promptable concept segmentation and tracking, achieving state-of-the-art performance with a decoupled recognition and localization architecture and a scalable data engine.

StepAudio 2.5 Technical Report

Hugging Face Daily Papers

StepAudio 2.5 is a unified audio-language model that achieves state-of-the-art results across ASR, TTS, and real-time spoken interaction by leveraging task-tailored reinforcement learning from human feedback to optimize shared representations.