Tag
This paper studies what transfers between transformer models of different sizes in the same family (Pythia), showing that representations align while weights don't, and that conversion works best via initialization rather than direct weight projection.
This paper introduces a self-supervised early exit method for LLMs, allowing computation to stop early at intermediate layers when confidence is high, thereby reducing inference cost. It also presents Dynamic Self-Speculative Decoding (DSSD) which achieves higher token acceptance than existing baselines.
A finetuned Pythia-6.9B model on two instruction-following datasets for 550 steps becomes capable in 13 languages, showing significant improvement over the base model.
This paper investigates whether direct activation transfer between language models can improve reasoning, using a linear translation layer from Pythia-160M to Pythia-410M. Despite achieving high representational alignment, the transferred activations do not improve multi-hop question answering, yielding a negative result.