Library of Layers (LoL): Adaptively Ordering Transformer Layers at Runtime
Summary
The paper introduces Library of Layers (LoL), a transformer architecture that dynamically reorders its layers at inference time, unifying parameter reuse and sparsity tradeoffs across dense, depth-recurrent, and MoE baselines while enabling interpretable routing across reasoning, multilingual, and safety tasks.
View Cached Full Text
Cached at: 11/03/26, 07:34 AM
Paper page - Library of Layers (LoL): Adaptively Ordering Transformer Layers at Runtime
Source: https://huggingface.co/papers/2611.03526
Abstract
Library of Layers dynamically orders transformer blocks during inference to adapt computation to tasks, enabling parameter reuse, sparsity, and interpretable routing across reasoning, multilingual, and safety settings.
Computer programmers chain together the same reusable computational primitives, for example, conditional statements or loops, but in different orders to implement different algorithms. In contrast, existing language model architectures always execute their layers in the same order. We introduceLibrary of Layers(LoL), an architecture that dynamically orders its layers during inference to solve the problem at hand. LoL has a single hyperparameter, the size of its global library oftransformer blocks, that allows it to transition seamlessly betweenparameter reuseandsparsity, generalizing the parameter-efficiency and FLOPs-efficiency tradeoffs of dense,depth-recurrent, andmixture of expertsarchitectures in one unified family. We show that LoL achieves competitive performance with all three above baselines when pretrained on text data. On synthetic sudoku and maze data, we observe that the model employs distinct layer orderings to implement different solution strategies, and we can intervene onroutingto force a model to apply a specific solution strategy to a problem instance. In a multilingual natural language setting, we find that LoL chooses different layer order patterns based on language and script. Similarly, we train LoL on safety data and fit probes to itsrouter outputs, finding that these probes distinguish safe from unsafe text with high AUROC. LoL provides a path toward a future in which models dynamically compose their own computational graph, enabling more efficient and controllable reasoning.
View arXiv pageView PDFGitHubAdd to collection
Get this paper in your agent:
hf papers read 2611\.03526
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2611.03526 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2611.03526 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2611.03526 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
LayerRoute: Adaptive Layer-Skipping with LoRA-Preserved Quality for Efficient LLM Inference
LayerRoute introduces a parameter-efficient method for adaptive transformer layer-skipping that combines per-layer routing with joint LoRA fine-tuning, achieving verified speedups and quality improvements in LLM inference.
Skip a Layer or Loop It? Learning Program-of-Layers in LLMs
This paper introduces PoLar, a framework that learns input-specific execution programs for frozen transformer layers, allowing layers to be skipped, kept, or repeated. It improves accuracy and reduces inference overhead compared to fixed-depth methods.
LayerRoute: Input-Conditioned Adaptive Layer Skipping via LoRA Fine-Tuning for Agentic Language Models
LayerRoute is a lightweight adapter that selectively skips transformer blocks during inference based on input type, achieving compute savings while maintaining or improving model quality through gated routing and LoRA adaptation. It achieves a 12.91% skip differential on agentic language models.
Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models
Proposes Memory-Efficient Looped Transformer (MELT), a novel recurrent LLM architecture that decouples reasoning depth from memory consumption by sharing a single KV cache across loops and using chunk-wise training with interpolated transition and attention-aligned distillation.
Aletheia: Gradient-Guided Layer Selection for Efficient LoRA Fine-Tuning Across Architectures
Aletheia introduces a gradient-guided layer selection method for efficient LoRA fine-tuning that identifies task-relevant transformer layers via lightweight gradient probes and applies adapters selectively, achieving 15-28% training speedup across 14 models while maintaining downstream performance on MMLU, GSM8K, and HumanEval benchmarks.