Diffuspace/Clover-35B-A3B-Preview · Hugging Face

Source: https://huggingface.co/Diffuspace/Clover-35B-A3B-Preview

Clover-35B-A3B-Previewis the first model released byDiffuSpace, delivering state-of-the-art performance among open-source diffusion language models at release on the benchmarks reported in ourlaunch blog. Clover combines a 35B-parameter mixture-of-experts backbone with approximately 3B active parameters per token, block diffusion generation, and a128K context window.

Blog·Code & inference backends·Hugging Face

https://huggingface.co/Diffuspace/Clover-35B-A3B-Preview#performancePerformance

See thebenchmark discussionfor evaluation results, settings, and comparisons.

https://huggingface.co/Diffuspace/Clover-35B-A3B-Preview#architecture-and-decodingArchitecture and decoding

Clover is a text-only model derived fromQwen3.6-35B-A3B, retaining its hybrid Gated DeltaNet (GDN) / full-attention MoE backbone with 40 layers, 256 routed experts, top-8 token routing, and one shared expert per layer.

• **Bidirectional block attention:**additional learned GDN parameters, including right-convolution weights and softmax controls, let tokens attend within a noisy block while preserving a causal prefix.
• **128K context:**up to 131,072 tokens for the prompt and generated response together.
• **Block diffusion with LowConfidence decoding:**this Hugging Face implementation starts each block with masked tokens and resolves confident positions in parallel. Positions with confidence above the threshold are filled; if none qualify, the most confident position is filled to ensure progress. Once resolved, the block becomes part of the prefix for the next block. Defaults areblock\_size=32andthreshold=0\.95.

Forself-speculative decodingwith our SGLang and llama.cpp implementations, see theofficial code repository.

https://huggingface.co/Diffuspace/Clover-35B-A3B-Preview#quick-startQuick start

Install PyTorch with CUDA support, then the Hugging Face dependencies. The example below uses Transformers 5.19.0 and PyTorch 2.10 or later.

pip install "transformers==5.19.0" accelerate safetensors

Use a GPU with enough memory for the BF16 weights (approximately 69 GB) plus inference state.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Diffuspace/Clover-35B-A3B-Preview"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="cuda:0",
).eval()

messages = [{"role": "user", "content": "What is diffusion?"}]
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True,
)
inputs = tokenizer(prompt, add_special_tokens=False, return_tensors="pt").to("cuda:0")
output = model.generate(
    **inputs,
    max_new_tokens=24576,
    block_size=32,
    threshold=0.95,
    do_sample=False,
)
completion = output[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(completion, skip_special_tokens=True))

You can also setmodel\_idto a local model directory. Generation defaults are loaded automatically fromgeneration\_config\.json; keyword arguments togenerate\(\)override them. Requests exceeding the 128K context limit are rejected, including space needed to resolve the final full block.

https://huggingface.co/Diffuspace/Clover-35B-A3B-Preview#generation-controlsGeneration controls

We observe that very long generations can sometimes become repetitive. The model includes repetition controls and gradual biases toward ending the response or thinking phase.The defaults are already configured, so no extra arguments are needed for normal use. They also apply during greedy decoding.

ParameterDefaultPurposepresence\_penalty``0\.9Discourages tokens that have already appeared in the generated response.frequency\_penalty``0\.0Penalizes tokens in proportion to their occurrence count; disabled by default.repetition\_penalty``1\.05Reduces the preference for previously generated tokens;1\.0disables it.dllm\_length\_penalty``10\.0Gradually increases the preference for end-of-response tokens.dllm\_think\_end\_bias``10\.0Gradually encourages</think\>to close the thinking phase. Length control begins atdllm\_length\_penalty\_start=8192response tokens and reaches its full strength overdllm\_length\_penalty\_window=8192more tokens. Thinking control usesdllm\_think\_end\_start=4096anddllm\_think\_end\_window=4096. These are soft controls;max\_new\_tokenssets the hard output limit. Repetition controls operate on generated response history, excluding the prompt.

https://huggingface.co/Diffuspace/Clover-35B-A3B-Preview#expert-selectionExpert selection

During block denoising, Clover can restrict routing to a shared subset of experts for each block. This reduces the number of distinct experts used across the block. The relevant model configuration settings are:

SettingDefaultPurposemoe\_block\_coreset\_enabled``trueEnables block-level expert selection during denoising.moe\_block\_coreset\_top\_p``0\.9Sets the cumulative routing-mass threshold used to select the expert subset. The selector aggregates the routing scores of each token’s top-8 candidate experts across the block, ranks experts by their aggregate score, and keeps enough to cover the requested fraction of that mass. A value of0\.9means90% of the aggregated routing mass, not 90% of all experts. Lower values generally select fewer experts, with a possible quality tradeoff. This setting is separate from token-samplingtop\_pand does not reduce the size of the stored or loaded model weights.

https://huggingface.co/Diffuspace/Clover-35B-A3B-Preview#serving-and-evaluationServing and evaluation

The Hugging Face implementation provides a standalone PyTorch inference path. Small numerical differences between its operations and SGLang’s kernels can lead to different generated text, even with the same decoding settings.

**We recommend our official SGLang implementation for serving and evaluation.**Setup instructions and supported decoding modes are available in theofficial repository.

https://huggingface.co/Diffuspace/Clover-35B-A3B-Preview#upstream-attribution-and-modificationsUpstream attribution and modifications

Clover is derived from Qwen3.6-35B-A3B. Upstream model assets are Copyright 2026 Alibaba Cloud and distributed under Apache-2.0; the accompanyingLICENSEpreserves the upstream license and copyright statement.

The model implementation derives from the Hugging Face Transformers Qwen3.5-MoE implementation used for the Qwen3.6 architecture, Copyright 2025 The Qwen Team and The HuggingFace Inc. team. Clover adds text-only block diffusion inference, bidirectional GDN parameters, and generation controls. SeeATTRIBUTIONS.mdfor file-level provenance and modification notices.