Tag
This paper compares three strategies to improve speculative decoding efficiency for non-English languages, finding that task-specific distillation improves acceptance rates but generalizes poorly, while n-gram draft models offer consistent speed-ups despite lower acceptance rates.
Draft-OPD introduces on-policy distillation with target-assisted rollouts and error replay to overcome the offline-to-inference mismatch in training draft models for speculative decoding, achieving over 5x lossless acceleration and improving upon EAGLE-3 and DFlash by 23% and 13% respectively.
ConFu introduces a novel speculative decoding framework that enables draft models to anticipate future generation directions through contemplate tokens and soft prompts, achieving 8-20% improvements in token acceptance rates and generation speed over EAGLE-3 across multiple LLM models.