Improving Text-to-Music Generation with Human Preference Rewards
Summary
This paper presents a text-to-music generation system that leverages reward conditioning, expert iteration, and preference tuning to improve audio quality within a 120M-parameter model, submitted to the ATTM Grand Challenge at ICME 2026.
View Cached Full Text
Cached at: 06/23/26, 09:41 AM
Paper page - Improving Text-to-Music Generation with Human Preference Rewards
Source: https://huggingface.co/papers/2606.21670
Abstract
A text-to-music generation system uses reward conditioning, expert iteration, and preference tuning to improve audio quality while maintaining efficiency within a 120M-parameter model framework.
We describe our entry to the efficiency track of the Academic Text-to-Music (ATTM) Grand Challenge at ICME 2026. Beyond the challenge protocol’sFAD-CLAPandCLAP score, we add a learned human-preference reward fromTuneJury, atwin pairwise rankertrained over open music-preference datasets. The reward serves both as a training-time conditioning signal and as a sample-selection criterion. The pipeline combines five engineering decisions on a 120M-parameterFluxAudio-Sbackbone, four at training time and one at inference: (i)training-time reward conditioningthat doubles as aninference-time CFGaxis, (ii) a sweep over fivescore-conditioning architectures, where training and inference use different variants, (iii)expert iterationon the top decile, (iv) a short preference-tuning pass (CRPO) foraudio-text alignment, and (v) inference post-processing viajoint CFG,source separation, andloudness normalization. Per-stage decomposition on 100 Song Describer prompts showstraining-time reward conditioningas a functional conditioning axis,expert iterationas the dominant contributor, the preference-tuning pass adding only noise-level gain, and the inference-time score scalar already saturated by the end of the chain.
View arXiv pageView PDFProject pageGitHub1Add to collection
Get this paper in your agent:
hf papers read 2606\.21670
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.21670 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.21670 in a dataset README.md to link it from this page.
Spaces citing this paper1
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
TuneJury: An Open Metric for Improving Music Generation Preference Alignment
TuneJury is an open-source pairwise reward model for text-to-music generation that provides calibrated preference scoring and generalizes across multiple downstream applications.
VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generation
VA-Judger is the first reward model for joint video-audio generation that uses human preference feedback to evaluate holistic quality, including a dataset and benchmark, and demonstrates significant improvements in human preference rates when applied to models like LTX-2.
Fine-tuning GPT-2 from human preferences
OpenAI demonstrates fine-tuning GPT-2 (774M parameters) using human preference feedback for text continuation and summarization tasks, requiring 5k labels for stylistic tasks and 60k for summarization, with models achieving 86-88% human preference rates though revealing labeler heuristic exploitation.
I trained a game music generator
I trained a 1.2B DiT model for instrumental game music generation, using Stable Audio's VAE and aiming to cover diverse styles. The project is open-source with a WebUI and samples available on HuggingFace.
Taste-aware music retrieval from audio embeddings
This paper introduces a benchmark for predicting taste qualities (sweet, bitter, etc.) from audio embeddings. It evaluates 10 pretrained audio encoders, achieving 0.134 RMSE, outperforming previous state-of-the-art and enabling taste-based music retrieval.