akhaliq/nemotron-3.5-transcribe-mini-64e · Hugging Face

Source: https://huggingface.co/akhaliq/nemotron-3.5-transcribe-mini-64e

https://huggingface.co/akhaliq/nemotron-3.5-transcribe-mini-64e#nemotron-35-transcribe-mini-reap-prunedNemotron-3.5-Transcribe-Mini (REAP-pruned)

The first pruned derivative of NVIDIA’sNemotron-3.5-Transcribe-30B-A4B— the #1 open-weight speech LLM — shrunk with zero retraining. From 67.5GB to 10GB, and faster than the original.

Routed MoE experts were pruned one-shot withREAP(Router-weighted Expert Activation Pruning,arXiv:2510.13999), calibrated on multilingual speech transcripts (FLEURS, 8 languages). No recovery training, no distillation — the audio encoder, speaker encoder, and CTC timestamp head are untouched.

🎙️Demo:akhaliq/nemotron-transcribe-mini

https://huggingface.co/akhaliq/nemotron-3.5-transcribe-mini-64e#wer-vs-the-full-30b-teacher-300-clips-fleurs-endees–librispeech-test-clean-greedy-decodingWER vs the full 30B teacher (300 clips: FLEURS en/de/es + LibriSpeech test-clean, greedy decoding)

VariantExpertsSizefleurs-enfleurs-defleurs-eslibrispeechMeanTeacher12867.5GB5.79%4.59%4.37%1.73%4.12%****96e96~53GB5.61%4.65%4.53%1.81%4.15%(+0.7% rel)64e64~38GB6.73%5.67%5.17%2.75%5.08%(+23% rel)64e-W4A166410.0GB6.29%6.35%5.70%2.28%5.16%(+25% rel) **96e is within noise of the teacher with 25% of its routed experts removed.**The 10GB 4-bit variant costs just +0.08 WER over its BF16 version — quantization is essentially free on top of pruning. A quarter of the experts were dead weight.

https://huggingface.co/akhaliq/nemotron-3.5-transcribe-mini-64e#speed-h200-vllm-greedy-fleurs-clipsSpeed (H200, vLLM, greedy, FLEURS clips)

ModelMean latencySingle-stream RTFx~90s clipBurst RTFx ×8Teacher 67.5GB0.24s39.30.49s / 107.1142.164e BF16 38GB0.24s39.60.47s / 112.2147.864e-W4A16 10GB0.19s50.40.50s / 103.8171.3 The 10GB model is the fastest configuration: +27% single-stream throughput and +21% burst vs the teacher, while being 6.7× smaller.(RTFx = seconds of audio processed per second. Raw data:eval/speed\_results\.json.)

https://huggingface.co/akhaliq/nemotron-3.5-transcribe-mini-64e#transformers-usage-%F0%9F%A4%97-verifiedTransformers usage 🤗 (verified)

The fused checkpoints load in-process with Hugging Face Transformers, exactly like the teacher:

pip install git+https://github.com/huggingface/transformers librosa av

from transformers import AutoModelForMultimodalLM, AutoProcessor

# download the fused variant first (subdir load)
#   hf download akhaliq/nemotron-3.5-transcribe-mini-64e --include "fused-96e/*"
model_dir = "~/.cache/huggingface/hub/models--akhaliq--nemotron-3.5-transcribe-mini-64e/snapshots/*/fused-96e"

processor = AutoProcessor.from_pretrained(model_dir)
model = AutoModelForMultimodalLM.from_pretrained(model_dir, device_map="auto")

# custom instructions are supported — this is an instruction-following speech LLM
inputs = processor.apply_transcription_request(
    "audio.wav",
    prompt="Transcribe this audio with punctuation and capitalization. "
           "Put speaker tags <spk:0> <spk:1> on the transcription.",
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=1024)
print(processor.decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True))

Verified onfused\-64eend-to-end. For fast Mamba kernels, also installmamba\-ssmandcausal\-conv1d(otherwise transformers uses slower reference implementations).

The standalone quantized text backbone loads the standard way (text-only — no audio encoder):

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "akhaliq/nemotron-3.5-transcribe-mini-64e",
    subfolder="backbone-64e-w4a16-g64", device_map="auto")
tok = AutoTokenizer.from_pretrained(
    "akhaliq/nemotron-3.5-transcribe-mini-64e", subfolder="backbone-64e-w4a16-g64")

https://huggingface.co/akhaliq/nemotron-3.5-transcribe-mini-64e#serving-vllm-same-stack-as-the-teacherServing (vLLM, same stack as the teacher)

pip install vllm==0.32.0
pip install "nemo-toolkit[asr-only,core,common-only] @ git+https://github.com/NVIDIA-NeMo/Speech.git"
hf download akhaliq/nemotron-3.5-transcribe-mini-64e --include "fused-64e-w4a16-g64/*"

export VLLM_PLUGINS=nemo_speechlm
vllm serve ~/.cache/huggingface/hub/models--akhaliq--nemotron-3.5-transcribe-mini-64e/snapshots/*/fused-64e-w4a16-g64 \
  --served-model-name nemotron-mini \
  --trust-remote-code --dtype bfloat16 \
  --gpu-memory-utilization 0.9 --max-model-len 8192

The 10GB W4A16 fused checkpoint serves via vLLM’s compressed-tensors Marlin kernels — it fits a12GB consumer GPU. Transcribe with the OpenAI chat API (audio as base64 data URI), exactly as in theteacher’s model card.

https://huggingface.co/akhaliq/nemotron-3.5-transcribe-mini-64e#layoutLayout

• fused\-64e\-w4a16\-g64/—10GB servable 4-bit speech model(W4A16 backbone + BF16 audio encoder), WER + speed verified
• fused\-96e/— servable 96-expert checkpoint (BF16, near-lossless)
• fused\-64e/— servable 64-expert checkpoint (BF16)
• backbone\-64e\-w4a16\-g64/— 10.0GB W4A16 INT4 quantized 64e backbone (text-only)
• backbone\-96e/,backbone\-64e/— standalone HF NemotronH text backbones
• eval/— raw transcripts and speed data from every eval run
• scripts/— the exact eval/release/quantization scripts used

https://huggingface.co/akhaliq/nemotron-3.5-transcribe-mini-64e#capabilities-preservedCapabilities preserved

40 languages, recordings up to 1 hour, speech translation, speaker-attributed transcription, word timestamps (CTC head), contextual biasing, instruction-controlled transcription style, audio Q&A — all inherited unchanged; only the LLM’s routed experts were pruned.

https://huggingface.co/akhaliq/nemotron-3.5-transcribe-mini-64e#methodMethod

1. Extract the NemotronH backbone from the fused NeMo checkpoint.
2. REAP saliency per expert (router weight × activation norm) over ~2,500 multilingual FLEURS transcripts, one calibration pass, layer-wise.
3. Drop the lowest-saliency experts per MoE layer (128→96 / 128→64), slice router rows + correction bias, keep shared experts, MTP head, and all dense layers.
4. Reassemble the fused SALM checkpoint with the original audio encoder.
5. Model-free W4A16 quantization of the pruned backbone — zero calibration, router/embeddings/lm\_head/Mamba scalars kept at full precision.

https://huggingface.co/akhaliq/nemotron-3.5-transcribe-mini-64e#limitationsLimitations

• 64e loses ~1 WER point vs the teacher; 96e is the recommended default for datacenter GPUs, 64e-W4A16 for consumer GPUs.
• Evaluated on short clips (≤30s) plus one ~90s clip; long-form behavior is expected to track the teacher but is not yet measured at full 1-hour length.
• License: OpenMDW-1.1 (inherited from the teacher).