YuE: Scaling Open Foundation Models for Long-Form Music Generation

Papers with Code Trending 03/11/25, 05:26 PM Papers

music-generation foundation-model long-form lyrics-to-song open-source pre-training generative-ai

Summary

YuE is a family of open foundation models that generates long-form music with aligned lyrics and coherent structure, using innovative techniques like track-decoupled next-token prediction and structural progressive conditioning.

We tackle the task of long-form music generation--particularly the challenging lyrics-to-song problem--by introducing YuE, a family of open foundation models based on the LLaMA2 architecture. Specifically, YuE scales to trillions of tokens and generates up to five minutes of music while maintaining lyrical alignment, coherent musical structure, and engaging vocal melodies with appropriate accompaniment. It achieves this through (1) track-decoupled next-token prediction to overcome dense mixture signals, (2) structural progressive conditioning for long-context lyrical alignment, and (3) a multitask, multiphase pre-training recipe to converge and generalize. In addition, we redesign the in-context learning technique for music generation, enabling versatile style transfer (e.g., converting Japanese city pop into an English rap while preserving the original accompaniment) and bidirectional generation. Through extensive evaluation, we demonstrate that YuE matches or even surpasses some of the proprietary systems in musicality and vocal agility. In addition, fine-tuning YuE enables additional controls and enhanced support for tail languages. Furthermore, beyond generation, we show that YuE's learned representations can perform well on music understanding tasks, where the results of YuE match or exceed state-of-the-art methods on the MARBLE benchmark. Keywords: lyrics2song, song generation, long-form, foundation model, music generation

Original Article

View Cached Full Text

Cached at: 09/11/26, 02:15 AM

Paper page - YuE: Scaling Open Foundation Models for Long-Form Music Generation

Source: https://huggingface.co/papers/2503.08638

Abstract

YuE, a family of open foundation models based on LLaMA2, can generate long-form music with aligned lyrics, coherent structure, and appropriate accompaniment using innovative techniques in next-token prediction, conditioning, and pre-training.

We tackle the task of long-formmusic generation--particularly the challenging lyrics-to-song problem--by introducing YuE, a family of open foundation models based on the LLaMA2 architecture. Specifically, YuE scales to trillions of tokens and generates up to five minutes of music while maintaining lyrical alignment, coherent musical structure, and engaging vocal melodies with appropriate accompaniment. It achieves this through (1)track-decoupled next-token predictionto overcome dense mixture signals, (2)structural progressive conditioningfor long-context lyrical alignment, and (3) amultitask,multiphase pre-trainingrecipe to converge and generalize. In addition, we redesign thein-context learningtechnique formusic generation, enabling versatilestyle transfer(e.g., converting Japanese city pop into an English rap while preserving the original accompaniment) and bidirectional generation. Through extensive evaluation, we demonstrate that YuE matches or even surpasses some of the proprietary systems in musicality and vocal agility. In addition, fine-tuning YuE enables additional controls and enhanced support for tail languages. Furthermore, beyond generation, we show that YuE’s learned representations can perform well onmusic understandingtasks, where the results of YuE match or exceed state-of-the-art methods on the MARBLE benchmark. Keywords: lyrics2song, song generation, long-form, foundation model,music generation

View arXiv page View PDF Project page GitHub6.57k Add to collection

Get this paper in your agent:

hf papers read 2503\.08638

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash