Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners
Summary
NAPE is a self-supervised audio learning framework that uses causal Transformers to predict next spectrogram patch embeddings, achieving state-of-the-art performance on multiple audio and speech benchmarks with a minimalist design.
View Cached Full Text
Cached at: 08/21/26, 08:08 AM
Paper page - Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners
Source: https://huggingface.co/papers/2608.19863
Abstract
NAPE uses causal Transformers to predict successive spectrogram patch embeddings for self-supervised audio learning without auxiliary components.
Self-supervised learning(SSL) has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance. A markedly different pre-training philosophy underpins the most influential progress in language modeling and, more recently, in visual representation learning: rather than train encoders as static feature extractors, models are trained to predict the next element, a discrete token or a continuous embedding, from the preceding context.Autoregressive predictionthereby provides a unified pre-training interface that transfers across modalities, compelling the model to learn the underlying data distribution. We ask whether such a simple causal paradigm can yield strong audio learners, given that audio’s temporal structure makesautoregressive predictionofpatch embeddingsa natural fit. We introduce NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised framework in which acausal Transformerpredicts each next patch embedding of alog-mel spectrogramfrom the previous ones, usingcausal maskingandstop-gradientas its sole training signal. The design is intentionally minimalist, avoiding reconstruction decoders, acoustic tokenizers, student-teacher setups, and auxiliary regularization losses. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results. NAPE also produces structured attention patterns without explicit supervision.
View arXiv pageView PDFProject pageGitHubAdd to collection
Get this paper in your agent:
hf papers read 2608\.19863
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.19863 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.19863 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.19863 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
@NielsRogge: NEPA has now been added here: Check the evals at the bottom to compare to other models
NEPA is a new method for visual self-supervised learning and generative pretraining that predicts the next embedding autoregressively, and has been added to a benchmark for evaluation.
Introducing next-generation audio models in the API
OpenAI introduced next-generation audio models for the API, including improved speech-to-text (gpt-4o-transcribe, gpt-4o-mini-transcribe) and customizable text-to-speech models that enable developers to build more intelligent and expressive voice agents with enhanced accuracy across challenging scenarios.
AudioMosaic: Contrastive Masked Audio Representation Learning
AudioMosaic introduces a contrastive learning-based audio encoder that uses structured time-frequency masking on spectrogram patches for efficient large-batch training, achieving state-of-the-art performance on audio benchmarks and improving audio-language models.
Curriculum-Based Noise Adaptation for Phoneme-to-Text Reconstruction in Visual Speech Recognition
This paper proposes progressive error curriculum training (PECT) to improve phoneme-to-text reconstruction robustness in visual speech recognition by gradually adapting to realistic phoneme prediction errors, achieving reduced word error rates on LRS2 and LRS3 benchmarks.
Unified Audio Intelligence Without Regressing on Text Intelligence
This paper introduces Audex, a unified audio-text LLM from NVIDIA that achieves state-of-the-art performance across multiple audio and speech tasks while preserving strong text reasoning capabilities without regression.