VIBE - AI background-music generator for videos; generates music that tries to match the video’s mood, timing, genre, tempo, and key.

• 10s synced instrumental tracks
• MiniCPM4-0.5B
• SongBloom VAE (48 kHz audio)
• Automated background scoring for social media, advertising, and prototyping

https://huggingface.co/aryanvibhosale/vibe…

aryanvibhosale/vibe · Hugging Face

Source: https://huggingface.co/aryanvibhosale/vibe

VIBE: Video Instruction-aligned Background Music Generation

🎉 Accepted to Findings of EMNLP 2026

https://huggingface.co/aryanvibhosale/vibe#descriptionDescription

VIBE generates background music for a video that follows an explicit text instruction, not just “music that fits this video”, but music that fitsanddoes what the user asked. It extends a tokenizer-free, continuous-latent music model with reinforcement learning against a multimodal LLM judge and rule-based verifiable rewards, so the generated music aligns with both the video and the instruction.

Rather than converting audio to discrete tokens, VIBE models music in a continuous latent space: amultimodal semantic LMproduces planning latents, aRITE(Residual Integration Transformer Encoder) stack refines them, and a local diffusion transformer, theLocDiT, denoises under conditional flow matching before a 48 kHz stereo VAE decodes to audio. Video conditioning enters through a CLIP-based encoder, andsemantic routingmerges the video and instruction signals into a single conditioning stream.

Built onVoxCPM(Apache-2.0), adapted from speech to music.

• **Instruction-Aligned Music Generation:**conditions on a videoanda free-form text instruction, so tempo, key, mood, and instrumentation can be requested explicitly rather than inferred from the video alone.
• **Reinforcement Learning from a Multimodal Judge:**GRPO againstQwen2.5-Omni-7B, which watches the video and listens to the generated music, scoring musicality, text-music alignment, and video-music alignment. Scores come from constrained digit-logit decoding, so there are no JSON parse failures.
• **Hard Verifiable Rewards:**rule-based tempo/BPM and musical-key agreement, measured on the generated audio with librosa/essentia and scored against targets parsed from the instruction. Deterministic, CPU-only, no learned parameters, and immune to reward hacking.
• **Continuous Latents, No Audio Tokenizer:**48 kHz stereo output through a music VAE, avoiding the quality ceiling and codebook artifacts of discrete tokenization.

https://huggingface.co/aryanvibhosale/vibe#model-architectureModel Architecture

**VIBE Architecture.**Video frames, text prompt, and previously generated patch embeddings are passed to the Multimodal Semantic LM. Layer-wise hidden states are linearly combined via learnable per-DiT-layer coefficients to form Conditioning Connectors, which are routed to every LocDiT layer alongside residual-integrated embeddings to generate each music patch.

Semantic LMMiniCPM4-0.5B architecture: 24 layers, hidden 1024, 16 heads, vocab 73 448RITE8 layers, hidden 1024LocDiT8 layers, conditional flow matching, Euler solverSemantic routing8 LocDiT blocks × learned softmax over 24 LM layersLatentspatch size 4, feature dim 64, FSQ latent dim 256Video encoderCLIP ViT-B/32, 8 sampled frames, projection dim 512 (frozen, weights embedded)Audio decoderSongBloom / Stable Audio VAE (frozen, external)Output48 kHz stereoParameters953.5 M across 873 tensors (3.51 GB)Precisionbf16 at inference; generator stored fp32, CLIP encoder bf16 VIBE is a custom architecture. It isnotloadable withtransformers\.AutoModel; use the inference scripts in theVIBE repository.

https://huggingface.co/aryanvibhosale/vibe#trainingTraining

Overview of our training curriculum.

Training data:JamendoMaxCaps(text-to-music pretraining),MusicBench(text-to-music SFT),CMI-Pref(captions for text-to-music preference optimisation), andHarmonySet(video-to-music). Evaluation is onReelBench.

https://huggingface.co/aryanvibhosale/vibe#provenanceProvenance

This is theStage-5 RL policy. The RL LoRA adapter (r=8, α=16, onq\_projandv\_projof both the LM and the LocDiT) has already beenfolded into the base weightsusingW \+= \(α/r\) · B·A, so it loads as a plain model with no LoRA machinery.

⚠️**Do not pass\-\-lora\_weights\_pathagainst this checkpoint.**The adapter is already merged; supplying it again would apply it a second time.

https://huggingface.co/aryanvibhosale/vibe#usageUsage

This repository holdsweights only, no code. Clone the code repository, then point\-\-ckpt\_dirat a download of this model.

git clone https://github.com/aryanvibhosale/vibe.git && cd vibe
conda env create -f environment.yml && conda activate vibe

**Requirestransformers\>=5\.0\.0.**The video encoder relies onCLIPModel\.get\_image\_features\(\)returning a pooled-output object, which is v5 behaviour. On transformers 4.x, video-to-music fails withAttributeError: 'Tensor' object has no attribute 'pooler\_output'. If environment creation fails on thetransformerspin, install it separately:pip install "transformers\>=5\.0\.0".

Download the weights and the external music VAE:

hf download aryanvibhosale/vibe --local-dir ./vibe_ckpt

hf download rsxdalv/SongBloom \
    --include "autoencoder_music_dsp1920.ckpt" "stable_audio_1920_vae.json" \
    --local-dir ./music_vae_cache

**Video-to-music.**Video plus instruction in, music out:

python scripts/infer_v2m.py \
    --ckpt_dir      ./vibe_ckpt \
    --audiovae_path ./music_vae_cache \
    --text          "An ambient electronic track at 125 BPM in B Minor." \
    --video_path    /path/to/input_video.mp4 \
    --output        v2m_out.wav

**Text-to-music.**Instruction only, no video. Usescripts/infer\_ttm\.pywith the same\-\-ckpt\_dirand\-\-audiovae\_path.

Tune quality with\-\-cfg\_value(default 2.0) and\-\-inference\_timesteps(default 10 for video-to-music, 20 for text-to-music).

Only thefirst 10 secondsof\-\-video\_pathare used; 8 frames are sampled uniformly from that window. Longer clips are truncated.

https://huggingface.co/aryanvibhosale/vibe#files-in-this-repositoryFiles in this repository

Filemodel\.safetensors3.51 GB, 873 tensors, 953.5 M parametersconfig\.jsonmodel configurationtokenizer\.json,tokenizer\_config\.json,special\_tokens\_map\.json``LlamaTokenizerFast``static/model card images, not used by the model

https://huggingface.co/aryanvibhosale/vibe#required-external-componentsRequired external components

ComponentWherePassed asSongBloom / Stable Audio music VAErsxdalv/SongBloom(Apache-2.0)\-\-audiovae\_path``openai/clip\-vit\-base\-patch32preprocessor configHugging Face, fetched at init (a few kB)n/a \-\-audiovae\_pathis adirectoryholdingautoencoder\_music\_dsp1920\.ckptandstable\_audio\_1920\_vae\.json. It is deliberately not bundled here: the audio VAE is never read from the checkpoint directory, and its weights carry their own licence.

The CLIPweights are embedded inmodel\.safetensors. Only the small image preprocessor config is fetched from the Hub at init, so a populatedHF\_HOMEcache is enough for offline use. You donotneed to download MiniCPM4-0.5B: the semantic LM weights are inmodel\.safetensorsand the tokenizer ships here.

https://huggingface.co/aryanvibhosale/vibe#limitationsLimitations

• **Instruction adherence is strongest for tempo.**Automatic musical-key detection is unreliable across the board, including on real reference audio, so key agreement should not be read as a headline metric.
• **RL trades distributional metrics for adherence and diversity.**The reinforcement-learned policy is not tuned to minimise FAD or maximise ImageBind similarity, and should not be ranked by them or framed as “closest to ground truth”.
• Generated audio may be unexpected or contain artifacts, particularly for instructions far from the training distribution. Released for research purposes; we do not recommend production use without further testing.
• **Short-form instrumental music.**Trained for background music clips, not full arrangements or vocals. Output is instrumental; theenlanguage tag refers to the text instruction, not to the audio.
• **Respect the rights in your inputs.**Generating music conditioned on video you do not have rights to, or producing music that imitates a specific artist’s protected work, may infringe. We recommend clearly marking generated audio as AI-generated.

https://huggingface.co/aryanvibhosale/vibe#licenseLicense

Released under theApache-2.0license.

These weights derive fromMiniCPM4-0.5B(Apache-2.0) and embed a frozenCLIP ViT-B/32(MIT) video encoder. TheQwen2.5-Omnijudge andCMI-RM(CMI-RewardBench) shaped training as reward signals only and are not distilled into these weights; neither is redistributed here. CMI-RM’s published weights are released for non-commercial use; consult the upstream repository for its current terms. Full third-party attribution is in theNOTICEfile. The audio VAE is not included; seeRequired external components.

https://huggingface.co/aryanvibhosale/vibe#acknowledgementsAcknowledgements

This work builds on open-source research and code. With thanks to:VoxCPM·MiniCPM-4·CosyVoice·DAC·Qwen2.5-Omni·MuQ / MuQ-MuLan·CMI-RewardBench