LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques
Summary
LoRA Speedrun is a public wall-clock leaderboard for fine-tuning techniques, measuring how fast LoRA adapters can achieve target accuracy on GSM8K and SQuAD tasks using Qwen2.5-1.5B and SmolLM2-1.7B on a single L40S GPU, with automated verification.
View Cached Full Text
Cached at: 07/20/26, 09:46 AM
Saivineeth147/lora-speedrun
Source: https://github.com/Saivineeth147/lora-speedrun
LoRA Speedrun 🏁
How fast can you LoRA-fine-tune Qwen2.5-1.5B to ≥ 57% on GSM8K — on a single L40S?
This is modded-nanogpt for fine-tuning: a frozen task, frozen hardware, and a public leaderboard of wall-clock records. Every record is independently re-run 3× with fresh seeds on identical hardware before it counts.
Attempting and verifying are free: official timing runs on a Modal L40S sandbox, and Modal’s free monthly compute credits cover full runs — so anyone can compete, and anyone can re-verify any record with one command.
Leaderboard
Track 1 — GSM8K · Qwen2.5-1.5B · target ≥ 57.0% · 1× L40S
Current record: 6m 05s by @Saivineeth147 — Sequence packing + completion-only loss masking, 2 epochs. Same LoRA config as #0; ~2x faster at higher accuracy.
| # | Date | Author | Train time | GSM8K/EM | Δ | Technique |
|---|---|---|---|---|---|---|
| 0 | 2026-07-18 | @Saivineeth147 | 11m 57s | 59.4% | — | Baseline: plain LoRA r=16 on all linear layers, 3 epochs, cosine LR. No tricks. (report) |
| 1 | 2026-07-18 | @Saivineeth147 | 6m 05s | 61.1% | −49% | Sequence packing + completion-only loss masking, 2 epochs. Same LoRA config as #0; ~2x faster at higher accuracy. (report) |
Track 2 — SQuAD v1.1 · SmolLM2-1.7B · target ≥ 75.5% · 1× L40S
Current record: 11m 08s by @Saivineeth147 — Track 2 baseline: plain LoRA r=16 on SQuAD, first 20k examples, 1 epoch, full-sequence loss. No tricks.
| # | Date | Author | Train time | GSM8K/EM | Δ | Technique |
|---|---|---|---|---|---|---|
| 0 | 2026-07-20 | @Saivineeth147 | 11m 08s | 77.5% | — | Track 2 baseline: plain LoRA r=16 on SQuAD, first 20k examples, 1 epoch, full-sequence loss. No tricks. (report) |
Full history with verification reports: records/RECORDS.md
The tracks
Two frozen tracks, deliberately different model families and task types — so a technique only proves general by winning on both. Same hardware, caps, and verification everywhere.
| Track 1 | Track 2 | |
|---|---|---|
| Base model | Qwen/Qwen2.5-1.5B | HuggingFaceTB/SmolLM2-1.7B |
| Task | GSM8K math → ≥ 57.0% exact-match | SQuAD v1.1 QA → ≥ 75.5% EM |
| Training data | GSM8K train split only | SQuAD train split only |
| Metric | Training wall-clock. Lower wins. | same |
| Hardware | 1× L40S (48 GB), Modal sandbox | same |
| Constraint | adapter-only, ≤ 30M trainable params | same |
Machine-readable specs: spec.yaml · spec-t2.yaml. Full rules: TASK.md.
You control everything else: LoRA rank and placement, quantization, learning-rate schedules, sequence packing, data subset selection and ordering, custom kernels, when to stop. Train on 1,000 well-chosen examples for 90 seconds if you can make it clear the bar.
Why this exists
LoRA/QLoRA is how most real-world fine-tuning actually happens, and the technique space exploded — DoRA, rsLoRA, PiSSA, LoRA+, NEFTune, Unsloth kernels, rank-adaptive methods — but there’s no adversarial, apples-to-apples arena where these ideas race each other in public. Papers report numbers on different models, data, and hardware; nothing is comparable.
The nanoGPT speedrun fixed this for pretraining and produced real science (Muon came out of it). This repo does the same for parameter-efficient fine-tuning: one frozen task, one GPU, wall-clock time, receipts required.
Quickstart
Official-hardware run (free). Make a Modal account, then:
git clone https://github.com/Saivineeth147/lora-speedrun && cd lora-speedrun
pip install modal pyyaml && modal setup # one-time browser auth
python harness/modal_verify.py --prefetch # one-time: cache model + data in a volume
# one timed, evaluated attempt of the baseline on the exact spec hardware:
python harness/modal_verify.py --submission submissions/000-baseline --runs 1
# full record-style verification (3 fresh seeds, all must pass):
python harness/modal_verify.py --submission submissions/000-baseline --runs 3
Local iteration (optional). Any 24 GB+ card runs the baseline for fast experimenting —
bash scripts/setup_gpu.sh, then python harness/run_submission.py submissions/000-baseline --runs 1.
Local times aren’t official; the leaderboard clock is the Modal L40S.
Then copy submissions/TEMPLATE/, make it faster, and open a PR. See CONTRIBUTING.md.
How records get verified
A real verification, replayed from the logs (seed 463953844, time-compressed): train → integrity + adapter audit → eval → 3-seed verdict.
- You open a PR with your training script, config, notes, and self-reported numbers.
- CI statically validates it, and an automated Claude security screen reviews the diff (exfiltration attempts, network use, harness tampering, test-set contact) and posts its findings publicly.
- A maintainer reviews the code, then comments
/verify— which re-runs your submission 3× with fresh seeds in a network-blocked Modal sandbox on the spec L40S. All 3 runs must clear the target; official time is the mean. - The harness audits the adapter param count and re-verifies model/data content hashes (anti-tampering), and the verification report is posted on the PR and committed to records/verifications/ with the accept/reject reasoning.
Full protocol, rubric, and threat model: JUDGING.md · SECURITY.md.
Ideas nobody has claimed yet
Taken so far: sequence packing + completion-only masking (record #1).
1-epoch aggressive-LR schedules · data pruning (train on the hardest 2k examples?) ·
block-diagonal/varlen packing attention · QLoRA NF4 vs bf16 tradeoff · rsLoRA / DoRA /
PiSSA init · LoRA+ (asymmetric LR for A/B) · NEFTune noise · curriculum ordering ·
rank/placement search (MLP-only vs attention-only) · torch.compile · Unsloth kernels ·
Liger kernels · fused cross-entropy · smarter warmup for short runs
Claim one, beat 6m 05s, get your name on the board.
FAQ
Won’t techniques overfit to one model + one task? That’s exactly why there are two tracks with different model families and task types — and more will follow the same freeze-and-calibrate protocol. A trick that only wins on one track is a record, but the techniques worth trusting are the ones that transfer. The track system makes that an empirical question instead of an argument.
Why Qwen2.5-1.5B? Isn’t it pretrained on math? Probably, like every modern base model. It doesn’t matter: the target is an anchor, not a claim about mathematical discovery. The race is the interesting part — same reason nanoGPT speedrunning targets an arbitrary val loss. (Track 2 uses a different family, SmolLM2, partly for this reason.)
Why wall-clock instead of FLOPs or steps? Because wall-clock is what you pay for, and it forces kernels, data loading, and algorithms to compete in the same currency. Same rule as modded-nanogpt.
Why an L40S on Modal instead of a 4090 or H100? Three reasons. It’s one consistent datacenter SKU, so times are actually comparable (rented consumer cards vary host-to-host). It’s free to use via Modal’s monthly credits, so competing and re-verifying costs nothing. And submissions are strangers’ code — Modal sandboxes run them network-blocked and secretless. (The L40S is the same AD102 silicon as the 4090, so consumer-GPU tricks transfer.)
Can I train on other data / distill from a bigger model? No. GSM8K train split only, no teacher models, no synthetic data. See TASK.md for the full banned list.
Multiple GPUs? No. One L40S. That’s the point.
License
MIT. Records, reports, and write-ups are as public as the code.
Similar Articles
@TanejaPriyal: i wanted to understand LoRA beyond “adapters are cheaper than full fine-tuning.” so, i wrote a two-part series and ran …
The author benchmarks serving 1,000 LoRA adapters on one GPU using vLLM, finding that active adapter count and traffic shape are the real bottlenecks, and provides recommendations for tuning max_loras.
ReLoRA: Knowledge-Reusing Adaptation for Fast Rollout of Evolving LLM Services
ReLoRA is a knowledge-reusing adaptation framework that efficiently restores service-ready LoRA adapters for evolving LLM services, reducing time-to-readiness by up to 8.9× and improving accuracy by up to 4.6% through adaptive initialization and scheduled regularization.
Beyond LoRA: Is Sparsity-Induced Adaptation Better?
This paper proposes sparsity-induced adaptations to LoRA, including Cheap LoRA (cLA) and a chained circulant variant (c³LA), and provides theoretical generalization bounds along with empirical evaluations showing up to 10% training time reduction and 15% peak GPU memory savings while maintaining competitive performance.
PEFT of SLM for Telecommunications Customer Support: A Comparative Study of LoRA Configurations with Energy Consumption Analysis
Presents a systematic study of parameter-efficient fine-tuning using LoRA on Qwen2.5-3B for telecommunications customer support, comparing 16 LoRA configurations with both traditional metrics and energy consumption analysis. Finds divergence between quantitative and qualitative performance.
Hybrid-LoRA: Bridging Full Fine-Tuning and Low-Rank Adaptation for Post-Training
Hybrid-LoRA proposes a framework that selectively applies full fine-tuning to a small subset of modules while using LoRA for the rest, achieving performance near full fine-tuning with significantly lower computational cost. Experiments show improvements of up to 5.65% over existing parameter-efficient baselines.