Model grafting: turning Qwen3.5-4B into a causal encoder-decoder after the fact

Reddit r/LocalLLaMA 09/22/26, 01:43 AM Models

model-grafting qwen3-5b causal-encoder-decoder speedup self-distillation ai-efficiency inference-optimization

Summary

Model Grafting technique modifies Qwen3.5-4B into a causal encoder-decoder, creating variants with up to 3.7x speedup in prompt processing and minimal accuracy loss.

Recently, the new DeepSeek-V4.1-Flash architecture showed how a causal encoder-decoder can work, but it was trained from scratch. Model Grafting does it to an existing model: cut at some depth, let the lower layers read the prompt, and use the upper layers get for encoder's residual stream as prefix KV via identity-init adapters, then heal with self-distillation from the unmodified parent. Decoding part stays the same, this method was described in this blog post https://latentnode.pages.dev/articles/model-grafting I applied the same recipe to Qwen3.5-4B to create two graft variants - https://huggingface.co/LocalLLaMA/Qwen3.5-4B-graft8 and https://huggingface.co/LocalLLaMA/Qwen3.5-4B-graft16 The graft8 variant shows speedup of ~3.7x at 128K prompt with some loss in accuracy. The graft16 is much more closer to the parent model while showing 2.0x speedup in prompt processing with minimum loss in accuracy.

Original Article

Similar Articles

@rasbt: Crazy model! It actually uses the old Qwen2.5-Coder-3B stack and got really great performance with their post-training …

X AI KOLs Following

A 3B parameter model using the Qwen2.5-Coder-3B stack achieves coding benchmark scores comparable to Claude Opus 4.5, with detailed post-training techniques including synthetic data, filtering, two-stage SFT, and a novel RL method (MGPO).

Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decoding

Hugging Face Daily Papers

Graft is a training-free framework that enhances speculative decoding by combining pruning and retrieval to improve acceptance rates and inference speed, achieving up to 5.41x speedup on short-context benchmarks and up to 21.8% improvement over EAGLE-3 on Qwen3-235B.

Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model

Simon Willison's Blog

Qwen releases Qwen3.6-27B, a 27B dense model claiming flagship-level coding performance surpassing the larger Qwen3.5-397B-A17B MoE, with impressive SVG generation demos.

Qwengram-0.8B: I transferred Qwen3.8 Flash-Next’s n-gram memory into Qwen3.5-0.8B — 5.05% lower validation perplexity

Reddit r/LocalLLaMA

The author transferred pretrained PLE n-gram memory from Qwen3.8-Flash-Next to Qwen3.5-0.8B, achieving a 5.05% reduction in validation perplexity without fine-tuning the backbone. Key findings include the effectiveness of dynamic gating and reader training for memory injection.

Jackrong/Qwopus-GLM-18B-Merged-GGUF

Hugging Face Models Trending

Jackrong released Qwopus-GLM-18B-Merged-GGUF, a 64-layer frankenmerge combining two Qwen3.5-9B finetunes into an ~18B parameter model, healed with 1000-step LoRA fine-tuning to fix layer boundary issues. The model achieves 90.9% on capability benchmarks while using less than half the VRAM of Qwen 3.6-35B MoE.