pranavupadhyaya52/Wiki-SmartBotLM-Instruct · Hugging Face

Source: https://huggingface.co/pranavupadhyaya52/Wiki-SmartBotLM-Instruct

https://huggingface.co/pranavupadhyaya52/Wiki-SmartBotLM-Instruct#wiki-smartbotlm-instructWiki-SmartBotLM-Instruct

A 270M parameter decoder-only Small Language Model (SLM) pretrained on English Wikipedia and instruction-tuned for general-purpose conversational AI.

https://huggingface.co/pranavupadhyaya52/Wiki-SmartBotLM-Instruct#overviewOverview

Wiki-SmartBotLM-Instructis a compact autoregressive language model developed entirely from scratch as an independent research project. The model was designed to explore the complete lifecycle of modern language model development—from pretraining through instruction tuning—using open datasets and modern transformer architectures.

Unlike models that rely on continued pretraining of existing foundation models, Wiki-SmartBotLM-Instruct was initialized with random weights and trained using a multi-stage pipeline.

The primary goals of this project were:

• Understand large language model training from first principles.
• Build a compact yet capable language model.
• Explore modern transformer architectures.
• Demonstrate the effectiveness of instruction tuning on a pretrained base model.

https://huggingface.co/pranavupadhyaya52/Wiki-SmartBotLM-Instruct#model-detailsModel Details

PropertyValueModel NameWiki-SmartBotLM-InstructArchitectureDecoder-only TransformerParameters270 MillionContext Length8192 TokensTraining FrameworkPyTorchPrecisionBF16 / FP16LicenseApache-2.0 (or your chosen license)

https://huggingface.co/pranavupadhyaya52/Wiki-SmartBotLM-Instruct#architectureArchitecture

Wiki-SmartBotLM-Instruct incorporates several modern transformer improvements including:

• Rotary Positional Embeddings (RoPE)
• RMSNorm
• SwiGLU Feed Forward Networks
• Grouped Query Attention (GQA)
• Flash Attention
• Causal Self Attention
• Mixed Precision Training

https://huggingface.co/pranavupadhyaya52/Wiki-SmartBotLM-Instruct#training-pipelineTraining Pipeline

The model was trained in two stages.

https://huggingface.co/pranavupadhyaya52/Wiki-SmartBotLM-Instruct#stage-1–pretrainingStage 1 — Pretraining

The model was pretrained on the English Wikipedia corpus using next-token prediction.

Objectives:

• Learn English grammar
• Learn factual knowledge
• Learn encyclopedic writing style
• Learn semantic relationships
• Learn long-context language modeling

https://huggingface.co/pranavupadhyaya52/Wiki-SmartBotLM-Instruct#stage-2–supervised-fine-tuning-sftStage 2 — Supervised Fine-Tuning (SFT)

Following pretraining, the model was instruction tuned using an open instruction-following dataset.

The objective of this stage was to improve:

• Instruction following
• Conversational ability
• Question answering
• Technical explanations
• General helpfulness

https://huggingface.co/pranavupadhyaya52/Wiki-SmartBotLM-Instruct#training-dataTraining Data

https://huggingface.co/pranavupadhyaya52/Wiki-SmartBotLM-Instruct#pretrainingPretraining

• English Wikipedia (Hugging Face)

https://huggingface.co/pranavupadhyaya52/Wiki-SmartBotLM-Instruct#supervised-fine-tuningSupervised Fine-Tuning

• Alpaca Instruction Dataset

https://huggingface.co/pranavupadhyaya52/Wiki-SmartBotLM-Instruct#capabilitiesCapabilities

Wiki-SmartBotLM-Instruct can perform a variety of natural language tasks including:

• General Question Answering
• Technical Explanations
• Programming Assistance
• Educational Content Generation
• Text Summarization
• Brainstorming
• Long-form Text Generation
• Basic Reasoning
• Instruction Following

https://huggingface.co/pranavupadhyaya52/Wiki-SmartBotLM-Instruct#benchmarksBenchmarks

The model was evaluated on the following text-based benchmarks:

• MMLU
• ARC Challenge
• HellaSwag
• Winogrande
• BoolQ

BenchmarkScoreMMLUTBDARC ChallengeTBDHellaSwagTBDWinograndeTBDBoolQTBD

https://huggingface.co/pranavupadhyaya52/Wiki-SmartBotLM-Instruct#exampleExample

Prompt

Explain what artificial intelligence is.

Response

Artificial intelligence (AI) is a branch of computer science focused on building systems capable of performing tasks that typically require human intelligence. These tasks include reasoning, learning, planning, perception, natural language understanding, and decision making. Modern AI encompasses techniques such as machine learning, deep learning, computer vision, and reinforcement learning, and is widely used in applications ranging from recommendation systems to autonomous vehicles.

https://huggingface.co/pranavupadhyaya52/Wiki-SmartBotLM-Instruct#intended-usesIntended Uses

Wiki-SmartBotLM-Instruct is intended for:

• Educational use
• AI research
• NLP experimentation
• Local inference
• Learning transformer architectures
• Small-scale deployment
• Prototype conversational assistants

https://huggingface.co/pranavupadhyaya52/Wiki-SmartBotLM-Instruct#limitationsLimitations

As a compact language model, Wiki-SmartBotLM-Instruct has several limitations.

• May hallucinate factual information.
• Limited reasoning compared to larger frontier models.
• Knowledge is limited to the pretraining corpus.
• May struggle with highly specialized or domain-specific questions.
• Performance may degrade on very long conversations.

https://huggingface.co/pranavupadhyaya52/Wiki-SmartBotLM-Instruct#ethical-considerationsEthical Considerations

Wiki-SmartBotLM-Instruct is intended for research and educational purposes.

Although instruction tuning improves response quality, users should independently verify information before relying on generated content in high-stakes domains such as healthcare, finance, or legal advice.

https://huggingface.co/pranavupadhyaya52/Wiki-SmartBotLM-Instruct#future-workFuture Work

Future versions of Wiki-SmartBotLM aim to include:

• Improved factual grounding
• Larger instruction datasets
• Enhanced reasoning performance
• Better multilingual support
• Retrieval-Augmented Generation (RAG)
• Tool calling support
• Longer context windows

https://huggingface.co/pranavupadhyaya52/Wiki-SmartBotLM-Instruct#citationCitation

If you use Wiki-SmartBotLM-Instruct in your research, please cite:

@misc{wikismartbotlm2026,
  title={Wiki-SmartBotLM-Instruct: A 270M Parameter Decoder-Only Small Language Model},
  author={Pranav Upadhyaya},
  year={2026},
  howpublished={Hugging Face Model Repository}
}

https://huggingface.co/pranavupadhyaya52/Wiki-SmartBotLM-Instruct#acknowledgementsAcknowledgements

This project was made possible by the open-source AI community.

Special thanks to:

• Hugging Face
• PyTorch
• English Wikipedia contributors
• Alpaca dataset contributors
• The open-source transformer research community

https://huggingface.co/pranavupadhyaya52/Wiki-SmartBotLM-Instruct#about-the-projectAbout the Project

Wiki-SmartBotLM-Instruct was developed as an independent research initiative to better understand modern language model development. From random initialization to pretraining and instruction tuning, every stage of the model was built to explore the practical engineering challenges of creating compact, efficient, and accessible language models.

Feedback, issues, and contributions are always welcome.