A 0.8B model just beat a 2B model on ARC-Challenge (42.15%): Closed-form weight surgery beat multi-GPU SFT with 0 backprop (Independently verified on NVIDIA L4)
DynamicTune transfers reasoning ability from a larger teacher into an 0.8B model via closed-form linear algebra in ~12 minutes with zero backpropagation, achieving 42.15% acc_norm on ARC-Challenge and beating a stock 2B model (41.10%). The FP16 checkpoint, GGUF file, and GitHub repo are open source and independently benchmarked on an NVIDIA L4.
A few days ago we shared the idea behind DynamicTune: transferring the trajectory flow from a larger teacher model directly into a smaller student via closed-form linear algebra in ~12 minutes on consumer hardware. Zero backpropagation, zero training tokens, zero gradient descent. To eliminate local bias, we uploaded the unquantized FP16 checkpoint to Hugging Face, and TPN Bench (TaoFu Protocol) independently evaluated it on a datacenter NVIDIA L4 GPU using the official lm_eval 0.4.12 framework (coordinator run ce494664-d077-4ff1-8741-15cedabc434c). Huge thanks to TPN Bench for the cloud GPU compute! Here are the independent numbers on full ARC-Challenge (1,172 items, zero-shot, greedy temp 0): * Stock Qwen3.5-0.8B Base (unquantized BF16): 37.50% acc_norm (34.60% acc) * 3-epoch SFT distillation (Mythos-0.8B, 25k Claude pairs, multi-GPU DDP): 38.10% acc_norm (35.80% acc) * SFT + Model Soup Merge: 37.00% acc_norm (catastrophic forgetting) * Stock Qwen3.5-2B Base (2.5x larger model, Q8): 41.10% acc_norm (37.80% acc) * DynamicTune 0.8B Base (Ours, 4-anchor closed-form surgery): 42.15% acc_norm (40.19% acc) WHY THIS IS COMPLETELY INSANE: A 0.8B model physically beat a 2.5x larger 2B model: In LLM scaling, parameter count is supposed to be king. An 800M model is not supposed to beat an uncompressed 2B model on ARC-Challenge (42.15% vs 41.10%). By extracting trajectory dynamics from 4B and pulling them back into the student SwiGLU blocks, higher-order reasoning is compressed directly into edge weights. Zero backpropagation beat 25,000 SFT instruction pairs: A recently published project (kmamine/merge-corrected-sft-distillation-Qwen-Mythos-0.8B) trained Qwen3.5-0.8B across 3 epochs on 25,000 Claude reasoning pairs on a multi-GPU cluster, reaching 38.10% before overfitting. DynamicTune reached 42.15% with zero gradient descent, zero loss functions, and zero training tokens. Ironclad 3.23-sigma statistical significance: A delta of +4.65% across 1,172 questions with stderr +-1.44% gives a Z-score of 3.23sigma (p < 0.001). This is not prompt tuning noise or random variance. 12 minutes on consumer hardware vs datacenter verification: The weight surgery was solved locally in ~12 minutes on an 8GB AMD RX 580 using layer-streaming (loading each layer in FP16, computing closed-form SVD deltas, and dumping to RAM). But the benchmark was conducted 100% in the cloud on datacenter NVIDIA L4 hardware via TPN Bench. WHY PAST ATTEMPTS FAILED: THE SPECTRAL ENTROPY BARRIER If you blindly apply weight deltas across all 24 layers of the student, the model collapses (+64.78% NLL explosion). When we scanned all 24 layers calculating the normalized spectral entropy H (from 0.0 to 1.0) of the representation residuals: * Layer 0 (H = 0.71): Clean semantic grounding. High receptivity to trajectory alignment. * Layers 1-22 (H between 0.90 and 0.96): Chaotic superposition knots. In an 800M model with only 1024 dimensions, polysemantic features are crammed into dense superposition. Forcing linear updates here causes catastrophic interference. * Layer 23 (H = 0.93): Pre-unembed boundary where features unpack toward vocabulary logits. By restricting surgery to 4 sparse anchor blocks (layers 0, 7, 15, and 23) and using damped Levenberg-Marquardt Tikhonov pseudoinverse + adaptive spectral rank truncation, we protect the fragile superposition knots while imparting corrective trajectory velocity. REPRODUCIBILITY & WEIGHTS Everything is 100% open source and available to test right now: * GitHub Repository: https://github.com/dsadawq3/DynamicTune * Base Model (Safetensors): https://huggingface.co/F-Labs/Qwen3.5-0.8B-DynamicTune-Base * GGUF Checkpoint (FP16): https://huggingface.co/F-Labs/Qwen3.5-0.8B-DynamicTune-Base-GGUF (Qwen3.5-0.8B-DynamicTune-Base-F16.gguf, SHA256: d77cf505108271d72f28298f20c2d158e7aeaf50cc22987db05a9a8973e08709) To run inference locally with standard llama.cpp: llama-cli -m Qwen3.5-0.8B-DynamicTune-Base-F16.gguf -p "Question: How does DNA replication initiate?\nAnswer:" -c 2048 -n 128 Special thanks to TPN Bench (TaoFu Protocol) for providing the independent datacenter NVIDIA L4 evaluation resources. Clone the repo, run your own benchmarks, and test it yourself.
Researchers propose a backprop-free, closed-form trajectory-matching method to transfer 4B model capabilities into a 0.8B student, finding that restricting weight surgery to 4 anchor layers avoids NLL explosions, yielding 10.8% lower held-out NLL across 30 benchmarks while running entirely on an 8GB RX 580.
Overmind launched a platform that turns production agent traces into training data to fine-tune and serve small open-weight specialist models the customer owns, claiming its tuned 9B model beats frontier flags like GPT-5.6 Luna on biomedical, legal, and engineering extraction tasks (e.g. 91.6% vs 82.0% F1 on contract clause extraction), with platform code open-sourced under AGPL-3.0.
LottoLabs announces LiquidAI's LFM2.5-8B-A1B-GGUF model, an 8B parameter model trained on a massive token count and optimized for fast inference on limited GPU hardware, with support for llama.cpp, Ollama, vLLM, and more.
Liquid AI released LFM2.5-2.6B, a 2.69B parameter model with 128K context and tool calling, optimized for multi-step agent workflows and capable of running at 30 tok/s on a phone with a 1.67GB Q4_K_M GGUF, though coding and knowledge-heavy tasks remain weak compared to larger models.
GPT-6-Astra-Max achieved 62.7% on the ARC-AGI-3 benchmark within six months, and with a memory adapter, the benchmark is saturated, though costs remain high, indicating a trend toward cheaper AI models.