Mothersuperior/yue2-mothersuperior-realaudio-tokenizer-v4 · Hugging Face

Source: https://huggingface.co/Mothersuperior/yue2-mothersuperior-realaudio-tokenizer-v4

https://huggingface.co/Mothersuperior/yue2-mothersuperior-realaudio-tokenizer-v4#yue2-mothersuperior-realaudio-tokenizer-v4yue2-mothersuperior-realaudio-tokenizer-v4

Real-audio tooling forYuE2-3B: theaudio → semantic-token encoderYuE2 doesn’t ship, plus aNAR-branch LoRAso the decoder renders real-production latents. Together they let you tokenize your own recordings, LoRA-tune YuE2’s AR on an artist, and generate new songs or covers.

https://huggingface.co/Mothersuperior/yue2-mothersuperior-realaudio-tokenizer-v4#filesFiles

filewhattokenizer\_head\_joint\_v4\.pt/\.safetensors/\.bf16\.safetensorsMERT-v2-FullSong layer-20 features (per-track instance-normalised, 25 Hz) → 32,768 YuE2 semantic codes. 8-layer transformer, d=512, 512-frame windows. Held-out exact match on YuE2’s own songs: 16.1% top-1 (near-miss codes render almost identically; ear tests of NAR round-trips sit around 95%).nar\_lora\_joint\_v4\.pt/\.safetensors/\.bf16\.safetensors/\_comfyui\.safetensorsrank-32 LoRA onnar\_self\_attn\.\{q,k,v,o\}\_proj+nar\_mlp\.\{gate,up,down\}\_proj(28 layers) + fullvae2llm/llm2vae. Trained jointly with the head on real audio.scripts/the training loop and inference scripts (below).scripts/ckpt\_io\.pyloads either format.

https://huggingface.co/Mothersuperior/yue2-mothersuperior-realaudio-tokenizer-v4#safetensors-layoutSafetensors layout

Same weights as the\.ptfiles, bit-exact in fp32 (the\.bf16variants are half the size; head top-1 agreement with fp32 is 98.6%). The previously published scripts accept either extension viascripts/ckpt\_io\.load\_ckpt\(path\), which returns the same dict the\.ptfiles hold.

• Head: the plainstate\_dictof the 8-layer encoder (inp\.\*,pos,enc\.layers\.\{0\.\.7\}\.\*,norm\.\*,head\.\*), 103 tensors.
• NAR LoRA:layers\.\{0\.\.27\}\.nar\_self\_attn\.\{q,k,v,o\}\_proj\.lora\_A``\[32, in\]and\.lora\_B``\[out, 32\], pluslayers\.\{i\}\.nar\_mlp\.\{gate,up,down\}\_proj\.lora\_\{A,B\}. Apply asW \+= lora\_B @ lora\_A(scale 1.0, no alpha) to the matchingmodel\.layers\[i\]Linear of YuE2-3B.vae2llm\.\{weight,bias\}andllm2vae\.\{weight,bias\}arefull replacement weightsfor those two Linear layers, not LoRA deltas. Rank and the delta rule are also in the file metadata.

https://huggingface.co/Mothersuperior/yue2-mothersuperior-realaudio-tokenizer-v4#comfyuiComfyUI

nar\_lora\_joint\_v4\_comfyui\.safetensorsis the same NAR LoRA in ComfyUI’s native YuE2 layout: load it with the standardLoraLoaderon theMODELoutput of the YuE2 checkpoint loader (Comfy\-Org/YuE2,yue2\_3b\_bf16\.safetensors), strength 1.0; the CLIP input is untouched by this file. ComfyUI stores the NAR branch asdiffusion\_model\.model\.layers\.\{i\}\.self\_attn\.qkv\_proj/o\_projandmlp\.gate\_up\_proj/down\_proj(q,k,v and gate,up fused), so the separate q/k/v and gate/up LoRAs are packed as one block-diagonal LoRA per fused matrix (lora\_up/lora\_down, no alpha = scale 1.0;up @ downequals the stacked deltas exactly). The fullvae2llm/llm2vaereplacements ship as\.diff/\.diff\_bagainst the m-a-p base.

Trained on 4,765 YuE2 self-generated songs, then adapted to real audio. If your material sounds off, rerunjoint\.pyon your own audio (step 3 below).

https://huggingface.co/Mothersuperior/yue2-mothersuperior-realaudio-tokenizer-v4#requirementsRequirements

Python 3.12 venv withyue2\-infer(commit 92a73cc7), torch 2.10 + cu128, torchaudio 2.10, transformers, soundfile, scipy, safetensors, demucs;HF\_HOMEwithm\-a\-p/YuE2\-3B,m\-a\-p/YuE2\-Vae,m\-a\-p/MERT\-v2\-FullSong. A 24 GB GPU is enough for every stage (14–18 GB measured with gradient checkpointing).

You also need the minted regularizer pack:regularizer/minted\_regularizer\_pack\.ptfromMothersuperior/yue2-minted-corpus(~100 MB, 4,732 YuE2-generated songs as\{name, src, style, lyrics, codec\}). The AR trainer draws 50% of its songs from it so a small artist set cannot collapse YuE2’s token grammar; theminted\_valitems are the held-out check whose loss should stay flat. The full corpus (audio + tokens + latents) is in the same dataset if you want to retrain the head.

Paths are hard-coded to our pod layout(/workspace/tok/full,/workspace/real/\.\.\.,/workspace/yue2\-corpus/tracks,/workspace/real/ar/dataset\.pt). Recreate that layout or edit the constants at the top of each script.

https://huggingface.co/Mothersuperior/yue2-mothersuperior-realaudio-tokenizer-v4#train-an-artist-lora-folder-of-songs-%E2%86%92-loraTrain an artist LoRA (folder of songs → LoRA)

Per song you need<name\>\.flac,<name\>\.lyrics\.txt(fulllyrics with\[verse\]/\[chorus\]/\[bridge\]/\.\.\.tags — truncated lyrics ruin structure), and<name\>\.txt= a style caption starting with your trigger phrase (e.g.xyzq, in the style of xyzq\. <description of the sound\>).

1. python prep\_real\.py— MERT features, VAE latents and the prompt prefix per song.
2. python cursor\_prep\.py— Demucs vocal stem → MMS forced alignment of the lyrics → lyric-cursor targets (automatic).
3. (optional, recommended for a new artist/era)HOLD\_TRACK=<one song name\> python joint\.py joint\_mine 3000 1 1 tokenizer\_head\_joint\_v4\.pt nar\_lora\_joint\_v4\.pt— adapts head + NAR to your audio. Otherwise use the v3 files as-is.
4. python ar\_prep\.py <head\.pt\>— tokenizes your songs and merges the regularizer pack intodataset\.pt(ar\_prep\.pyexpects the pack’s records; point it at the downloaded file).
5. SCHED\_STEPS=3000 CK\_FROM=600 CK\_EVERY=200 python ar\_lora\_cursor\.py my\_lora 1600 64 0\.5 none 1e\-4 0\.08— rank-64 AR LoRA, 50/50 artist vs minted, lyric-cursor weight 0.08, checkpoints at 600/800/1000/1200/1400/1600.Do not train longer: past ~1,500 steps the model memorises the songs.
6. LADDER\_STYLE\_TRACK=<song\> LADDER\_LYRICS=<lyrics\.txt\> bash ladder\.sh my\_lora nar\_lora\_joint\_v4\.pt— renders one fixed prompt from every checkpoint (optionalFINALS=<file\>with linestag style\_track lyrics seed). Pick by ear (ours: step 800).

https://huggingface.co/Mothersuperior/yue2-mothersuperior-realaudio-tokenizer-v4#decoderDecoder

Consider decoding withMothersuperior/YuE2-Vae-merge-0.666: a weight merge of YuE2-Vae (0.666) and YuE2-Vae-legacy (0.334). The two releases share one encoder and only differ in the decoder; the merge sits between the clean default decoder and the more musical legacy decoder. Drop-in:YuE2Pipeline\.from\_pretrained\("m\-a\-p/YuE2\-3B", vae="Mothersuperior/YuE2\-Vae\-merge\-0\.666"\), or pointar\_generate\.pyat it.

https://huggingface.co/Mothersuperior/yue2-mothersuperior-realaudio-tokenizer-v4#inferenceInference

# new song: style caption with your trigger + lyrics, score-free
python ar_generate.py my_lora/step-800.pt nar_lora_joint_v4.pt out_tag <style_track> lyrics.txt 12
# cover: transcribe any recording with SheetSage2 (--melody-only), then
ABC_FILE=score.abc COT=melody STRIP_TEMPO_KEY=1 python ar_generate.py my_lora/step-800.pt nar_lora_joint_v4.pt cover_tag <style_track> lyrics.txt 21
# LoRA strength: AR_SCALE=0.77 ...   stock model control: pass `none` for either LoRA

ar\_generate\.pyfolds both LoRAs into the base weights and runs YuE2’s own pipeline, so the stock sampler, CFG and VAE apply unchanged.

https://huggingface.co/Mothersuperior/yue2-mothersuperior-realaudio-tokenizer-v4#other-scriptsOther scripts

extract\_full\.py+train\_v2\.py(retrain the head on the minted corpus),nar\_lora\.py(NAR LoRA alone),teacher\_train\.py(head fine-tune with the NAR as teacher),ar\_lora\.py(AR LoRA without the cursor),build\_reg\_pack\.py(rebuild the regularizer pack from a corpus).

Weights derive from YuE2-3B (CC BY-NC 4.0): non-commercial use only. Scripts are provided as-is.

https://huggingface.co/Mothersuperior/yue2-mothersuperior-realaudio-tokenizer-v4#v5-head-2026-09-16-tokenizer_head_v5_30ksafetensors–bf16v5 head (2026-09-16):tokenizer\_head\_v5\_30k\.safetensors(+\.bf16)

Same architecture and input as the v4 head, retrained on the full 14,547-track minted corpus with a better schedule: optimi AdamW (lr 5e-4, betas 0.9/0.99, weight decay 0.01), 30,000 steps, cosine. Held-out token accuracy on minted tracks:top-1 18.9%, top-5 45.0%, top-1-or-neighbour 34.1%(v4-era head: 15.9 / 39.2 / 30.9). Drop-in for the v4 head inckpt\_io\.pyand in the ComfyUI FS_Audio Dataset Builder (models/fs\_audio/). For the matching decoder LoRA see the v5 joint pair below.

https://huggingface.co/Mothersuperior/yue2-mothersuperior-realaudio-tokenizer-v4#v5-joint-pair-2026-09-16-tokenizer_head_joint_v5–nar_lora_joint_v5–bf16–_comfyuiv5 joint pair (2026-09-16):tokenizer\_head\_joint\_v5+nar\_lora\_joint\_v5(+\.bf16, +\_comfyui)

The v5 head co-trained for 3,000 steps with a rank-32 decoder LoRA on**128 real tracks from three artists (rock, metal, bass music)**plus 4,000 minted anchor tracks, using the decoder’s flow loss on the real latents as the teacher (previous joint rounds used a single artist’s catalog). Held-out real-audio decoder loss 1.056 ->0.981; minted top-1 unchanged (~18.9%, so the head stays universal); token repeat rate on real audio 9.3% ->3.0%. Use the pair together: head for tokenizing real audio, LoRA on the decoder when rendering those tokens.nar\_lora\_joint\_v5\_comfyui\.safetensorsis the ComfyUI-native layout (load with a LoRA loader, model strength 1.0).

https://huggingface.co/Mothersuperior/yue2-mothersuperior-realaudio-tokenizer-v4#v8–v9-joint-pairs-2026-09-16-training-against-the-audio-not-just-the-latentsv8 / v9 joint pairs (2026-09-16): training against the audio, not just the latents

tokenizer\_head\_joint\_v8+nar\_lora\_joint\_v8andtokenizer\_head\_joint\_v9+nar\_lora\_joint\_v9(each fp32,\.bf16, and\_comfyuifor the LoRA). The v5 pair stays available above; these are additions.

What changed.Every joint round so far trained the head and decoder LoRA against a loss in VAE-latent space (flow matching). That proxy was not tracking what we heard: v1 -> v3 -> v5 cut the latent loss from 1.159 to 0.981 while audio-domain distances to the original recordings barely moved. Starting with v6 the trainer adds anaudio-domain loss: at each real-audio step the decoder’s clean-latent estimate is decoded through the frozen (differentiable) YuE2 VAE and compared with the same seconds of theoriginal recordingusing log-mel L1, multi-resolution STFT (spectral convergence + log-magnitude, windows 512/1024/2048) and a stereo-width term, applied at noise levels <= 0.4. Gradients reach the decoder LoRA directly and the head through the straight-through token choice. Data and everything else are identical to v5 (128 real tracks from three artists: rock, metal, bass music; 4,000 minted anchor tracks; 3,000 steps; init =tokenizer\_head\_v5\_30k+nar\_lora\_joint\_v4). The idea of penalising the rendered audio against the real production came from Kytra listening to the v5 reconstructions.

Weight sweep, measured on the three held-out tracks (whole songs) against the originals. LTAS = long-term average spectrum distance (the “production fingerprint”; this is the number that tracked the listening results). Stereo = L/R correlation (originals: 0.84 / 0.71 / 0.87).

pairaudio-loss weightlatent lossminted top-1LTAS dB rock / metal / bassmel L1 dB rock / metal / bassstereo rock / metal / bassv50 (latent only)0.98118.9%1.56 / 2.07 / 2.274.36 / 4.59 / 4.440.77 / 0.69 / 0.90v60.30.98118.8%1.40 / 1.94 / 2.094.18 / 4.55 / 4.360.78 / 0.76 / 0.92v71.00.98418.7%1.44 / 1.28 / 1.924.50 / 4.54 / 4.390.63 / 0.63 / 0.84v8****2.00.98319.0%1.28 / 1.75 / 1.844.28 / 4.49 / 4.510.76 / 0.70 / 0.92v9****4.00.98418.7%1.23 / 1.49 / 1.924.29 / 4.48 / 4.440.74 / 0.65 / 0.88 Listening (Kytra): v7 clearly better than v6, v8 clearly better than v7, v9 preferred overall. Minted top-1 stays flat across the sweep, so the head remains universal; the latent loss moves by 0.003, so the audio term is not fighting the latent objective. Mel L1 didnotfollow the listening results, LTAS did. Token-choice errors (occasional out-of-tune notes) are unchanged by this loss; that is a head-accuracy problem.

**Use:**tokenize real audio with the v9 (or v8) head and load the matchingnar\_lora\_joint\_v9\_comfyui/\_v8\_comfyuion the decoder at model strength 1.0. Trainer:scripts/joint\_v6\.py, the recovered original audio-loss variant ofscripts/joint\.py(envAUX\_W,AUX\_TMAX,AUX\_FR). Read thesetup and historical-reproduction notesfirst: this original script uses\.ptcheckpoints and the old pod paths; unlike the other published scripts, it has not been adapted tockpt\_io. The required neighbor tables are included underassets/.