Motif-Technologies/Motif-3-Beta · Hugging Face

Source: https://huggingface.co/Motif-Technologies/Motif-3-Beta

⚠️Preview / beta checkpoint — not the final release.This repository hosts an intermediate checkpoint ofMotif-3. Thefinal checkpoint will be released soon.

Motif-3is a large-scale Mixture-of-Experts (MoE) language model built from the ground up byMotif Technologiesfollowing a fully in-house, proprietary design — not a re-parameterization of existing open-source architectures.

https://huggingface.co/Motif-Technologies/Motif-3-Beta#highlightsHighlights

• 🧠~314B total parameters / ~13B activeper token (sparse MoE)
• 📏256K context length(262,144 tokens), natively long-context
• ⚡ Sparse routing: 384 experts with 8 activated per token, plus 1 shared expert
• 🌐Multilingual, general-purpose

https://huggingface.co/Motif-Technologies/Motif-3-Beta#model-detailsModel details

Model typeMixture-of-Experts causal language modelTotal parameters~314BActive parameters~13B / tokenHidden size4096Layers53Routed experts384 (top-8)Shared experts1Context length262,144 (256K)Vocabulary220,160Tensor typebfloat16

https://huggingface.co/Motif-Technologies/Motif-3-Beta#architectureArchitecture

Motif-3 is a fully in-house design and introduces several custom components:

• Grouped Differential Latent Attention (GDLA)
• Grouped PolyNorm activation, applied per expert
• Modified mHC

https://huggingface.co/Motif-Technologies/Motif-3-Beta#benchmarksBenchmarks

Artificial Analysis Intelligence Index (AAII): 44

SeeArtificial Analysisfor details.

https://huggingface.co/Motif-Technologies/Motif-3-Beta#usageUsage

A dedicatedvLLM serving guide is coming soon.

The model ships with custom modeling code, so load it withtrust\_remote\_code=True:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Motif-Technologies/Motif-3-Beta"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [{"role": "user", "content": "Hello!"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

outputs = model.generate(inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

https://huggingface.co/Motif-Technologies/Motif-3-Beta#accessAccess

This model isopenly available— anyone can download the weights, no access request required.

https://huggingface.co/Motif-Technologies/Motif-3-Beta#licenseLicense

Permission is granted to use, modify, and redistribute this software for personal, educational, and non-commercial research purposes only.

Commercial use is prohibited without prior written permission from Motif Technologies.

© Motif Technologies. All rights reserved.