Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training
Summary
A study finds that reinforcement learning gains in transformers are concentrated in a single middle layer, and training just that layer can match or surpass full-parameter RL training across multiple models and tasks.
View Cached Full Text
Cached at: 07/09/26, 07:52 AM
Paper page - Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training
Source: https://huggingface.co/papers/2607.01232
Abstract
Reinforcement learning adaptation in transformer models shows highly concentrated improvements in specific middle layers rather than uniform parameter updates across all layers.
Reinforcement learning(RL) has become a central component of post-training large language models (LLMs), yet little is understood about how RL adaptation is distributed acrosstransformer layers. Existing approaches typically update all model parameters uniformly, implicitly assuming that every layer contributes similarly to the gains obtained during RL post-training. In this work, we challenge this assumption through a systematic layer-wise study of RL training. Surprisingly, we find that training a single transformer layer can recover most of the gains achieved byfull-parameter RL training, and in some cases even surpass it. To quantify this phenomenon, we introduce the quantitylayer contribution, which measures the fraction of full RL improvement recovered by training a layer in isolation. Across seven models spanning two model families (Qwen3, Qwen2.5), three RL algorithms (GRPO,GiGPO,Dr. GRPO), and multiple task domains includingmathematical reasoning,code generation, andagentic decision-making, we observe a remarkably stable pattern: RL gains are highly concentrated in a small subset of, and in many cases even a single,transformer layers. More strikingly, the same structural pattern consistently emerges: high-contribution layers concentrate in the middle of the transformer stack, while layers near the input and output ends contribute substantially less. The resulting layer rankings remain strongly correlated across datasets, tasks, model families, and RL algorithms.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2607\.01232
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2607.01232 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2607.01232 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2607.01232 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Is One Layer Enough? A Single Transformer Layer Matches Full-Parameter RL Train
This paper systematically studies layer-wise contribution in RL post-training for LLMs, finding that training a single middle transformer layer can recover or even surpass full-parameter RL gains, with consistent patterns across models and tasks.
Training transformers where every layer W = V·Uᵀ from initialization reveals a corpus-determined optimal rank - looking for arXiv endorser (cs.LG) [D]
This paper proposes Native Factorized Weights for transformers, where every linear layer is trained as a product of two low-rank matrices from initialization. Experiments show a corpus-determined optimal rank that minimizes validation loss and a generalization band, outperforming dense baselines with fewer parameters.
@a1zhang: Transformers struggle to generalize to tasks they were not explicitly trained on. Instead, we propose in 2026 that it i…
The article proposes that Transformers can generalize to new tasks through a well-designed harness that induces composition, without needing intrinsic model generalization. It shows RLMs can generalize from short tasks to 8-32x longer tasks and across domains.
@lateinteraction: The "harness" is starting to blur with the neural architecture, in terms of who carries the inductive biases that unloc…
Researchers propose that the harness (training setup) should carry inductive biases for generalization, showing that training RLMs is far superior to vanilla Transformers for scaling and generalization to harder tasks.
Agentic Transformers Provably Learn to Search via Reinforcement Learning
This paper theoretically studies how transformer-based policies acquire search capabilities from reinforcement learning training dynamics in a stochastic tree environment. It shows that a two-head transformer can implement depth-first search and that this mechanism emerges naturally from sparse reward signals under a depth-wise curriculum.