StarpowerTechnology/arXiv-WVY-43M · Hugging Face

Source: https://huggingface.co/StarpowerTechnology/arXiv-WVY-43M

https://huggingface.co/StarpowerTechnology/arXiv-WVY-43M#arxiv-wvy-43marXiv-WVY-43M

arXiv-WVY-43Mis a 43.5M parameter tiny language model developed by Starpower Technology intended for autonomous research. This is the prototype experiment. We are still creating finetuning dataset

Kaggle [https://www.kaggle.com/code/starpowertechnology/arxiv-wvy-43m-demo]

The model uses a compact DeepSeek-V3-style architecture and was trained from scratch on arXiv titles and abstracts.

This is abase language model, not an instruction-tuned or chat-tuned model.

https://huggingface.co/StarpowerTechnology/arXiv-WVY-43M#model-detailsModel Details

PropertyValueParameters43,489,608Vocabulary24,000Hidden size384Transformer layers6MTP layers1Attention heads6MLP intermediate size1,024Routed experts8Experts selected per token2Shared experts1Q LoRA rank128KV LoRA rank96QK RoPE head dim16QK non-RoPE head dim48Value head dim64RoPE theta10,000YaRN factor4.0Original max position1,024

https://huggingface.co/StarpowerTechnology/arXiv-WVY-43M#architectureArchitecture

WVY-43M uses a compact DeepSeek-V3-style causal language model architecture containing:

• Multi-head latent attention
• Mixture-of-Experts layers
• 8 routed experts with Top-2 routing
• 1 shared expert
• Q/KV low-rank projections
• Rotary positional embeddings
• YaRN RoPE scaling
• Multi-Token Prediction layer

The architecture is intentionally kept small for research into compact language models, training from scratch, experimentation, and low-compute deployment.

https://huggingface.co/StarpowerTechnology/arXiv-WVY-43M#trainingTraining

The model was pretrained from random initialization.

**Training corpus:**arXiv paper titles and abstracts

**Observed training tokens:**approximately 726 million

**Checkpoint:**step 5,600

The training corpus gives the model significant exposure to scientific and technical language, particularly terminology appearing in academic research.

https://huggingface.co/StarpowerTechnology/arXiv-WVY-43M#using-the-model-on-kaggleUsing the Model on Kaggle

Enable Internet access for the Kaggle notebook so the model can be downloaded from Hugging Face.

https://huggingface.co/StarpowerTechnology/arXiv-WVY-43M#1-install-dependencies1. Install dependencies

!pip install -q -U transformers huggingface_hub tokenizers accelerate

https://huggingface.co/StarpowerTechnology/arXiv-WVY-43M#2-download-and-load-wvy-43m2. Download and load WVY-43M

import os
import torch

from huggingface_hub import hf_hub_download
from transformers import (
    AutoConfig,
    AutoModelForCausalLM,
    PreTrainedTokenizerFast,
)

REPO_ID = "StarpowerTechnology/arXiv-WVY-43M"

# ---------------------------------------------------------
# Download tokenizer and weights
# ---------------------------------------------------------

tokenizer_path = hf_hub_download(
    repo_id=REPO_ID,
    filename="tokenizer.json"
)

weights_path = hf_hub_download(
    repo_id=REPO_ID,
    filename="model.pt"
)

# ---------------------------------------------------------
# Tokenizer
# ---------------------------------------------------------

tokenizer = PreTrainedTokenizerFast(
    tokenizer_file=tokenizer_path
)

# ---------------------------------------------------------
# Load the custom DeepSeek-V3 configuration from Hugging Face
# ---------------------------------------------------------

config = AutoConfig.from_pretrained(
    REPO_ID,
    trust_remote_code=True
)

# Build the model architecture without looking for
# pytorch_model.bin / safetensors because WVY uses model.pt.
model = AutoModelForCausalLM.from_config(
    config,
    trust_remote_code=True
)

# ---------------------------------------------------------
# Load WVY-43M weights
# ---------------------------------------------------------

checkpoint = torch.load(
    weights_path,
    map_location="cpu",
    weights_only=False
)

# Support common checkpoint formats.
if isinstance(checkpoint, dict):
    for key in ["state_dict", "model_state_dict", "model"]:
        if key in checkpoint and isinstance(checkpoint[key], dict):
            checkpoint = checkpoint[key]
            break

# Remove DataParallel prefix if present.
if (
    isinstance(checkpoint, dict)
    and len(checkpoint) > 0
    and all(k.startswith("module.") for k in checkpoint)
):
    checkpoint = {
        k[len("module."):]: v
        for k, v in checkpoint.items()
    }

model.load_state_dict(checkpoint, strict=True)

# ---------------------------------------------------------
# Device
# ---------------------------------------------------------

device = torch.device(
    "cuda" if torch.cuda.is_available() else "cpu"
)

model = model.to(device)
model.eval()

print("Model:", REPO_ID)
print("Device:", device)

params = sum(p.numel() for p in model.parameters())

print(f"Parameters: {params:,}")

https://huggingface.co/StarpowerTechnology/arXiv-WVY-43M#generate-textGenerate Text

Because arXiv-WVY-43M is a base causal language model, prompts can be passed directly as text.

prompt = "Quantum entanglement is"

inputs = tokenizer(
    prompt,
    return_tensors="pt",
    add_special_tokens=False
)

input_ids = inputs["input_ids"]

# WVY-43M configuration:
# BOS = 0
# EOS = 1

bos_id = config.bos_token_id

bos = torch.full(
    (input_ids.shape[0], 1),
    bos_id,
    dtype=torch.long
)

input_ids = torch.cat(
    [bos, input_ids],
    dim=1
).to(device)

attention_mask = torch.ones_like(input_ids)

with torch.no_grad():
    output = model.generate(
        input_ids=input_ids,
        attention_mask=attention_mask,
        max_new_tokens=150,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        eos_token_id=config.eos_token_id,
        pad_token_id=config.eos_token_id,
    )

generated_tokens = output[
    0,
    input_ids.shape[1]:
]

text = tokenizer.decode(
    generated_tokens,
    skip_special_tokens=True
)

print(text)

https://huggingface.co/StarpowerTechnology/arXiv-WVY-43M#simple-generation-functionSimple Generation Function

def generate(
    prompt,
    max_new_tokens=150,
    temperature=0.7,
    top_p=0.9
):
    encoded = tokenizer(
        prompt,
        return_tensors="pt",
        add_special_tokens=False
    )

    input_ids = encoded["input_ids"]

    bos = torch.full(
        (input_ids.shape[0], 1),
        config.bos_token_id,
        dtype=torch.long
    )

    input_ids = torch.cat(
        [bos, input_ids],
        dim=1
    ).to(device)

    attention_mask = torch.ones_like(input_ids)

    with torch.no_grad():
        output = model.generate(
            input_ids=input_ids,
            attention_mask=attention_mask,
            max_new_tokens=max_new_tokens,
            do_sample=True,
            temperature=temperature,
            top_p=top_p,
            eos_token_id=config.eos_token_id,
            pad_token_id=config.eos_token_id,
        )

    generated = output[
        0,
        input_ids.shape[1]:
    ]

    return tokenizer.decode(
        generated,
        skip_special_tokens=True
    )

print(
    generate(
        "The relationship between gravity and spacetime is"
    )
)

https://huggingface.co/StarpowerTechnology/arXiv-WVY-43M#intended-useIntended Use

arXiv-WVY-43M is intended for research involving:

• Small language models
• Language-model pretraining
• Scientific text generation
• Physics and technology language modeling
• Mixture-of-Experts architectures
• Low-parameter language-model experimentation
• Fine-tuning and continued pretraining
• Educational experiments with models trained from scratch

https://huggingface.co/StarpowerTechnology/arXiv-WVY-43M#limitationsLimitations

WVY-43M contains approximately 43.5 million parameters and should be evaluated as a small experimental language model rather than as a replacement for modern large language models.

The released checkpoint is a base pretrained model and has not been instruction-tuned for assistant-style conversations.

No formal benchmark results are currently included in this model card.

https://huggingface.co/StarpowerTechnology/arXiv-WVY-43M#licenseLicense

MIT

https://huggingface.co/StarpowerTechnology/arXiv-WVY-43M#developerDeveloper

Starpower Technology

Hugging Face organization:StarpowerTechnology