hybrid-sequence-models

This paper investigates the discrepancy between log-likelihood and generation-based evaluation in distilled hybrid sequence models, showing that perplexity alone can be misleading. It introduces GenDistill, a pipeline for distilling Transformers into Hybrid-KDA models, achieving up to 75% KV cache reduction and 2-4x faster time-to-first-token while retaining 86-90% of teacher accuracy.