I Trained a 117M parameters Silia model on an H100 in 5 hours.

About a month ago I posted my very first paper about my custom Silia architecture here https://www.reddit.com/r/LocalLLaMA/s/J19Qi4NXeJ With the help of Ok-Internal9317 who decided to sponsor the paper with compute I was able to train a 117M parameters model. You can checkout the model here Hugging Face https://huggingface.co/Srijan-Srivastava/Strawberry-s1 GitHub https://github.com/SrijanSriv211/Silia/ How to Generate? Example prompt: Which animal has more poison - the salamander that sticks out its bone or the frog with the sharp head thing, and how do they both make their enemies hurt? Use uv for inferencing. Install torch, numpy, regex and colorama. uv run inference.py -i 117M_fp32/final.bin -e cl16k.bin -T "Which animal has more poison - the salamander that sticks out its bone or the frog with the sharp head thing, and how do they both make their enemies hurt?" Generated output: ``` I dont understand why they all work together. 1. Query Decomposition "pouring your animal's survival" → food safety concern "all the animals" → dual danger threshold "potential danger" → threshold question, not just threshold ● High confidence: food safety, environmental impact, food safety User seeking: food safety considerations, policy implications, environmental interactions 2. Domain Contextualization Food safety (food safety): food safety, food safety, health needs "all the animals" → food safety threshold, environmental impact "all the animals" → food safety threshold, food safety concerns "did it work together" → safety safety requirements, environmental conditions Key domains needed: - food safety physiology (flight, feeding, food safety) - environmental risk stratification - environmental factors - environmental interactions 3. Information State Assessment ● High confidence ``` Silia research paper Hugging Face https://huggingface.co/Srijan-Srivastava/Strawberry-s1/blob/main/Silia%3A%20Tiny%20Scale%20Is%20All%20I%20Can%20Spare%20To%20Play%20With%20Transformer.pdf Zenodo https://zenodo.org/records/20631957 More stuff. The model was trained on an H100 for 5 hours using https://huggingface.co/datasets/codelion/synth-100M dataset with ~82M (81,920,000) tokens in total, with a batch size of 8 and context length of 1024. Since it's a 117M parameters model and trained only on 82M tokens it is severely under-trained, especially considering it was trained with Muon optimizer enabled but the learning rate was fairly low (or at least that's what I feel). So yeah this model is very under-trained and it could've achieved even better loss. I haven't run it on any benchmarks yet. Also, I didn't get the chance to train a 117M nanoGPT model but since the model is under trained and also on lower learning rate I'd say it'd perform worse than nanoGPT. However, I was able to train a 11.5M parameters Silia & nanoGPT model on the same synth-100M dataset for 20k steps and Silia's loss final val loss was 3.2123 while nanoGPT's final val loss was 3.1945. Though one difference was that Silia required a slightly higher learning rate that was max 3e-3 & min 3e-4 (with cosine decay) while nanoGPT required 2e-3 max & 2e-4 min. On same 2e-3 -> 2e-4 lr as nanoGPT, Silia performed worse with the final val loss of 3.2857. As a quick recap the architecture diagram looks like this: Input tokens | [Token Embedding] | [Silia Block xN:] |--- Multi-Headed Attention | |--- Rotary Positional Embeddings | |--- QK Norm | |--- Scaled Dot Product Attention |--- Silu activation function |--- Multi-Headed Attention |--- Attention Residuals [Output Projection (weight-tied)] | Next token logits Thank you :)