How to Fine-Tune a Small Model with Your Own Data

Author: Rahul (@sairahul1)

You don’t need a 70B model. You don’t need a $100,000 compute budget. You don’t need a machine learning team.

A 1.5B parameter model, fine-tuned with 200 to 500 good samples, can outperform frontier models with generic prompts on your specific task.

This is the complete guide: data collection, cleaning, dataset construction, QLoRA training, Google Colab, evaluation, deployment. Every step includes runnable code.

1. Correcting Misconceptions About Fine-Tuning

Most people think fine-tuning makes a model smarter. It doesn’t.

Fine-tuning makes a model consistently better at one narrow task. That’s its entire value proposition.

The guide uses this example: input a startup briefing (like: “We help independent clinics reduce no-shows through automated reminders, scheduling workflows, and patient follow-ups. We’re raising our seed round to scale product and sales”), output a fundraising pitch copy—clean, evidence-based, no made-up metrics, no rigid slide deck structure.

A 1.5B model doing this reliably is more valuable than a 70B model doing it inconsistently.

2. Fine-Tuning Is Not Your First Step

Before training anything, establish three baselines:

• Baseline 1: Strong prompting alone
• Baseline 2: Prompting with retrieval (RAG)
• Baseline 3: The fine-tuned model

What fine-tuning improves: tone, structure, consistency, domain vocabulary, instruction following.

What fine-tuning cannot reliably provide: current market data, competitor intelligence, real-time facts. If your model needs to know current market size or funding trends, use retrieval, not training.

The usable architecture is a chain like this: Startup Briefing → Optional Retrieval (market data, evidence) → Small Fine-Tuned Model → Plain Text Output.

Only start fine-tuning after your prompting baseline fails on a specific point.

3. Step One: Choose a Small Base Model

Start between 0.5B and 3B parameters. Recommended starting points: Qwen/Qwen2.5-1.5B-Instruct (the one chosen for the guide), Qwen/Qwen3-0.6B, HuggingFaceTB/SmolLM2-1.7B-Instruct.

1.5B is small enough to run QLoRA on a T4, yet large enough to write useful business copy.

Don’t start with 7B or 14B. Larger models mean: higher VRAM requirements, longer training times, higher serving costs, more operational complexity, and a higher risk of overfitting on small datasets.

Start with a small model. Only upgrade if it fails evaluation.

4. Step Two: Confirm Data Rights Before Scrape

This is the step everyone skips, and the most dangerous one.

Publicly available does not equal authorized for training.

Before downloading any file, record this for each data source: source_id, source_url, owner, rights_status (must be approved, not assumed), permission_scope, commercial_use_permitted, redistribution_permitted, rights_reviewed_at, and a citation to private proof.

Put in a public GitHub repo: source metadata, file hashes, processing code, dataset schema, evaluation code.

Keep private: original PDFs, OCR outputs, human gold-standard transcripts, permission emails, user-submitted data.

Get this right first. The rest is just engineering.

5. Steps Three and Four: Reproducible Download, Text Extraction from PDFs and OCR

Never crawl blindly. Use a clear, approved URL allowlist. For each file, capture: hash, timestamp, manifest entry. Skip duplicates by content hash, don’t silently re-download. The code includes a validation check—confirm the response body actually starts with %PDF-, throw an error if not.

Pitch decks are notoriously hard documents: large blocks of text, tiny footnotes, charts, tables, rotated text, numbers embedded in images, multi-column layouts. Use a cascade strategy:

1. Try native PDF text extraction first → keep if sufficient
2. If insufficient, use layout-aware OCR (Docling, PaddleOCR) → keep if results are good
3. If still not working, use a second OCR engine or manual review

Key rule: Never silently discard pages where OCR returned nothing. Empty results are your hardest samples; they show where the pipeline breaks.

Another rule: Machine draft is not ground truth, never conflate the two. The recording structure specifically includes a gold_text field, left empty until manual review.

6. Step Five: Clean Text Without Destroying It

Cleaning means removing extraction noise, not making it “prettier.”

Safe operations only: Unicode NFKC normalization, remove null bytes, replace non-breaking spaces with normal spaces, collapse consecutive spaces to one, collapse three or more consecutive newlines to two.

Never automatically correct numbers.

7. Step Seven: Split by Company, Not by Page (This Mistake Invalidates Evaluation)

If the same company appears in both the training and validation sets: the model will memorize that company’s specific phrasing, looking like it generalized. It hasn’t.

Split by company, and keep a split manifest recording split_version, dataset_hash, train/validation/test company ID lists, creation time.

After looking at the results, never alter the test set. Once you’ve evaluated with it, it’s burnt—it becomes part of your training signal, no longer a true evaluation.

8. Step Nine: Train with QLoRA

QLoRA loads the base model in 4-bit precision and trains only a small adapter. VRAM usage is much lower than full fine-tuning, with nearly identical results.

Key hyperparameters and why they’re set this way:

• r=16 → LoRA adapter rank, start here
• lora_alpha=32 → Adapter scaling (typically double the rank)
• lora_dropout=0.05 → Regularization
• learning_rate=2e-4 → A good starting point for LoRA
• num_train_epochs=2 → Start low; overfitting is real
• gradient_accumulation_steps=8 → Simulate a larger batch
• packing=True → Better GPU utilization for short samples

Don’t increase epochs just because training loss is dropping. The model can memorize your samples while performing worse on unseen companies. Lower training loss does not equal better generalization.

9. Step Twelve: Evaluation and What “A Number is a Hallucination” Means

A practical check in evaluation: take the set of numbers in the generated text, subtract the set of numbers from the source evidence. Any number in the result that doesn’t appear in the source evidence is a hallucination.

The guide provides a sample scorecard:

• Prompting baseline: Usefulness 7.1/10, Unsupported claims 8%, Numeric errors 3%, Avg latency 2.4s
• Fine-tuned model: Usefulness 8.0/10, Unsupported claims 2%, Numeric errors 1%, Avg latency 1.5s

You can only claim improvement if the test set was isolated before you started evaluating. If you peeked at the test set, these results are meaningless.

10. Step Thirteen: Deploy the Adapter

Two paths, depending on traffic:

• High throughput: Use vLLM. Launch the base model as a service, enable --enable-lora, mount the adapter as a LoRA module. Before deployment, check the current version’s LoRA documentation, as options can change.
• Low traffic: Use FastAPI with Transformers.

Never expose a raw model service directly to the public internet. A production structure should be: Client → HTTPS with authentication → VPS with rate limiting → Request queue → GPU model service → Plain text response. The VPS handles TLS, auth, rate limiting, queueing; the GPU only handles inference.

11. Step Fourteen: Monitor Quality Post-Deployment, and When to Retrain

Start tracking quality metrics from day one. Don’t automatically collect user data for retraining—use explicit opt-in, and even with consent, strip sensitive parts before storage. Production training data must always be separate from evaluation data.

Retraining triggers should not be every user submission, but a confluence of: a sufficient number of reviewed samples, a clear, documented failure mode, a stable evaluation set, a recorded dataset version, and a rollback plan.

A reasonable cadence:

• v0: Prompting baseline
• v1: 200 reviewed samples
• v2: 1,000 reviewed samples
• v3: Larger dataset plus retrieval

For each version, record: version number, base model and revision, dataset hash, training sample count, GPU type, total GPU hours, all evaluation metrics, deployment date, rollback artifact, known failure modes.

Do not deploy without a clearly documented rollback path.

12. Where This Guide’s Real Value Lies

It’s not the QLoRA code snippet—that’s everywhere. The value is that it places two typically skipped items at the very beginning and very end of the process:

At the beginning: Data rights. The point that “publicly available doesn’t equal authorized for training,” plus “public repos only hold metadata and hashes; originals and permission emails stay private,” is a part almost never covered in Chinese articles on fine-tuning. It doesn’t take much time, but it’s the gate between “playing around” and “deliverable to a client.”

At the end: Evaluation discipline. Splitting by company (not by page), burning the test set after looking at it, requiring every number in the output to be traceable to source evidence, not deploying without a rollback path. These sound like process pedantry, but they determine whether your 8.0/10 score is real or just self-deception.

Links

Original article (Rahul): https://x.com/sairahul1/article/2100882424343265527

Qwen2.5-1.5B-Instruct: https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct

#FineTuning #QLoRA #LocalModels