Tag
LayerRoute introduces a parameter-efficient method for adaptive transformer layer-skipping that combines per-layer routing with joint LoRA fine-tuning, achieving verified speedups and quality improvements in LLM inference.
This paper presents a rigor-matched audit comparing periodic-step layer-skipping methods like ConfLayers and SWIFT for efficient LLM inference, and analyzes trained routing alternatives, finding SWIFT superior in accuracy and true inference speed.
CausalGate introduces a method that uses causal interventions to measure the importance of transformer sub-layers and distills this into static scalar gates for efficient inference without runtime overhead, outperforming existing pruning and routing methods.
This paper introduces Program-of-Layers (PoLar), a method that allows LLMs to dynamically skip or loop pretrained layers per input, improving accuracy and efficiency over fixed-depth inference.