强化学习缩放定律的实证研究

# An Empirical Study of Reinforcement Learning Scaling Laws
Source: [https://arxiv.org/html/2610.33420](https://arxiv.org/html/2610.33420)
###### Abstract

The discovery of scaling laws enables prediction ofthe performance,the optimal resource allocation strategy, andthe best hyperparameter configurationof a large\-scale training job based on inexpensive experiments at smaller scales\. Guided by these empirical scaling laws, the deep learning community has achieved great success in training large \(self\-\)supervised models on Internet\-scale datasets\. In contrast, despite its potential and wide application, reinforcement learning \(RL\) has yet to find a guiding principle for predictable scaling\. In this work, we conduct an empirical study of RL scaling laws of an actor\-critic algorithm in two procedurally generated environments, Starpilot and Craftax\. We discover empirical scaling laws with strong predictive performance in both environments\. The discovered scaling laws accurately predict the aforementioned three quantities for larger\-scale training jobs up to two orders of magnitude in total training compute\. In particular, we employ a variant of Elo scores for the single\-agent RL paradigm as an alternative performance metric than the commonly used mean episode return\. Our results show that the Elo score is a much more predictable metric than the mean episode return, making it a better choice for fitting scaling laws\. By applying our discovered scaling laws to increase training compute, we establish a new state of the art in the challenging Craftax environment\. Beneath the surface of the global scaling trend, we also observe a non\-monotonic scaling pattern of the compute\-optimal model size in Craftax\. We connect this observation to a pathology known as Ray interference\.

## 1Introduction

The discovery of empirical scaling laws has revolutionized deep learning, in particular for generative modeling, by transforming model development from a process of heuristic trial\-and\-error into apredictableengineering discipline\([Kaplan et al\., 2020](https://arxiv.org/html/2610.33420#bib.bib23);[Hoffmann et al\., 2022](https://arxiv.org/html/2610.33420#bib.bib20);[Bi et al\., 2024](https://arxiv.org/html/2610.33420#bib.bib3)\)\. Before launching a large\-scale training job with a compute budget ofCCFLOPs, three critical questions need to be answered\([Muennighoff et al\., 2023](https://arxiv.org/html/2610.33420#bib.bib27)\): \(i\)\(Performance\)What is the best achievable performance? \(ii\)\(Allocation\)What is the optimal way of allocating the compute budget among axes such as the model size and the dataset size? \(iii\)\(Hyperparameter\)What are the optimal hyperparameters for the training job? Scaling laws provide a tool for researchers to answer these questions by conducting inexpensive experiments at smaller scales and extrapolating to larger scales\.

In contrast to the success of scaling in \(self\-\)supervised learning, reinforcement learning \(RL\) has yet to find its own guiding principles for predictive scaling\. While scaling has powered landmark achievements from game\-playing agents\([Silver et al\., 2017](https://arxiv.org/html/2610.33420#bib.bib33);[Vinyals et al\., 2019](https://arxiv.org/html/2610.33420#bib.bib35);[Berner et al\., 2019](https://arxiv.org/html/2610.33420#bib.bib2)\), to fine\-tuning large language models\([Guo et al\., 2025](https://arxiv.org/html/2610.33420#bib.bib11);[Khatri et al\., 2026](https://arxiv.org/html/2610.33420#bib.bib24)\), the foundational scaling dynamics of RL remain poorly understood\. Unlike \(self\-\)supervised learning, where the data distribution is stationary, RL agents learn from experiences that continuously change as their own policy evolves\. Learning in RL is thus inherently non\-stationary\. This active coupling between policy improvement and data generation implies that RL may be governed by scaling laws distinct from those of passive, supervised regimes\. Moreover, unlike the cross entropy loss which has a clear information\-theoretical grounding, rewards in RL are often designed in an ad\-hoc manner based on heuristics or domain knowledge\. Consequently, the common RL performance metric, mean episode return, may not vary smoothly\([Hilton et al\., 2023](https://arxiv.org/html/2610.33420#bib.bib18)\)\. Due to these differences, it is unclear if the same conclusion, and predictive power, from the \(self\-\)supervised learning scaling laws apply to RL\.

Existing work on RL scaling laws primarily answer only a subset of the aforementioned three questions while also lacking validation of the predictive performance of the fitted laws\([Jones, 2021](https://arxiv.org/html/2610.33420#bib.bib22);[Hilton et al\., 2023](https://arxiv.org/html/2610.33420#bib.bib18);[Neumann and Gros, 2023](https://arxiv.org/html/2610.33420#bib.bib28)\)\. We argue that the most critical value of scaling laws is in their predictive power and thus validating the predictions is a centerpiece of scaling law studies\. In this paper, we present an empirical study of scaling laws in RL\. We study all three questions with validation of the predictive performance from small\-scale experiments to large\-scale experiments in each case\. We conduct experiments in two procedurally generated environments, Starpilot\([Cobbe et al\., 2020](https://arxiv.org/html/2610.33420#bib.bib6)\)and Craftax\([Matthews et al\., 2024](https://arxiv.org/html/2610.33420#bib.bib29)\)\. We fit scaling laws for the optimal hyperparameters, model and data sizes, and performance as a function of total training FLOPsCC\. We validate the predictive power of the fitted laws by testing their prediction accuracy at one or two orders of magnitude larger training FLOPs\. To overcome the non\-smoothness of the mean episode return, we employ a variant of the Elo score\([Elo, 1978](https://arxiv.org/html/2610.33420#bib.bib8)\)for the single\-agent RL paradigm and showed that it extrapolates effectively whereas the mean episode return does not, making it a critical choice for scaling studies\. Our analysis in Craftax reveals that, despite the empirical accuracy of the fitted laws, the compute\-optimal frontier does not follow a global log\-linear law throughout training\. We connect this observation to a phenomenon known as Ray interference\([Schaul et al\., 2019](https://arxiv.org/html/2610.33420#bib.bib32)\)\. Based on this observation, we argue against the existence of scaling laws for arbitrary RL environments\.

Our main contributions are:

- •We establish a two\-stage empirical framework for studying RL scaling laws\. Our experimental results show that optimal model size and key hyperparameters \(e\.g\., batch size and learning rate\) follow predictable power\-law relationships with respect to training compute\.
- •We address the non\-smoothness of mean episode return and show that a single\-agent Elo metric scales log\-linearly as a function of training compute, enabling reliable performance extrapolation up to two orders of magnitude\.
- •Guided by our scaling laws, we train an actor\-critic agent to1​e​191\\text\{e\}\{19\}FLOPs and set a new state\-of\-the\-art performance of54\.0%54\.0\\%on the challenging Craftax benchmark\.
- •We identify a local inverse scaling pattern in Craftax and connect it to Ray interference, showing how global trends can mask non\-monotonic local dynamics\.

## 2Related Work

In the past decade, empirical power laws were discovered in multiple deep learning application domains including image classification\([Hestness et al\., 2017](https://arxiv.org/html/2610.33420#bib.bib17);[Rosenfeld et al\., 2020](https://arxiv.org/html/2610.33420#bib.bib31)\), image generation\([Henighan et al\., 2020](https://arxiv.org/html/2610.33420#bib.bib16)\), and language modeling\([Kaplan et al\., 2020](https://arxiv.org/html/2610.33420#bib.bib23);[Hoffmann et al\., 2022](https://arxiv.org/html/2610.33420#bib.bib20)\)\.[Kaplan et al\. \(2020\)](https://arxiv.org/html/2610.33420#bib.bib23)observed that the cross entropy loss of an autoregressive language model scales as a power law with the model size, the dataset size, and the amount of compute used for training across several orders of magnitude\. Their work provides a method to answer theperformancequestion\. Based on their fitted scaling laws,[Kaplan et al\. \(2020\)](https://arxiv.org/html/2610.33420#bib.bib23)recommended scaling model size more aggressively than dataset size as training compute increases\. Following[Kaplan et al\. \(2020\)](https://arxiv.org/html/2610.33420#bib.bib23),[Hoffmann et al\. \(2022\)](https://arxiv.org/html/2610.33420#bib.bib20)conducted a study on the optimal model size for a certain training compute budget, directly addressing theallocationquestion\. Contradictory to[Kaplan et al\. \(2020\)](https://arxiv.org/html/2610.33420#bib.bib23), their work showed that the model size and dataset size should grow at an equal pace while scaling up training compute\. In both works, the power laws were fitted to explain the observed data\. Their predictions were not validated at larger scales\. Moreover, many experimental details such as hyperparameter choices are missing or poorly studied, leaving thehyperparameterquestion unanswered\. To address these limitations, DeepSeek\([Bi et al\., 2024](https://arxiv.org/html/2610.33420#bib.bib3)\)studied all three questions of performance, allocation, and hyperparameter and tested the predictive accuracy of the fitted scaling laws at larger scales\. Our work builds upon these works\. We employed their general methodology to study all three questions and tested the predictive accuracy of the corresponding scaling laws\. Meanwhile, our work is different from these works as we focus on RL instead of language modeling\.

There are few existing works on RL scaling laws\. The most closely related work is by[Hilton et al\. \(2023\)](https://arxiv.org/html/2610.33420#bib.bib18), who studied scaling laws of single\-agent RL\. They identified the difficulty of predicting the mean episode return at larger training compute due to its non\-smoothness\. To address this challenge, they proposed a new metric calledintrinsic performancewhich measures the performance of an RL agent by the minimum amount of training compute needed to achieve parity with it\. Although by definition intrinsic performance is not suited for answering the performance question, it served as a tool to answer the allocation question\.[Fu et al\. \(2026\)](https://arxiv.org/html/2610.33420#bib.bib10)studied scaling laws for value\-based RL algorithms with a focus on the allocation question and the hyperparameter question\.[Jones \(2021\)](https://arxiv.org/html/2610.33420#bib.bib22)and[Neumann and Gros \(2023\)](https://arxiv.org/html/2610.33420#bib.bib28)studied scaling laws of the AlphaZero agent\([Silver et al\., 2017](https://arxiv.org/html/2610.33420#bib.bib33)\)in the context of two\-player zero\-sum games\. Both works found that the Elo\([Elo, 1978](https://arxiv.org/html/2610.33420#bib.bib8)\)of an agent scales predictably with the amount of training compute\.[Jones \(2021\)](https://arxiv.org/html/2610.33420#bib.bib22)further showed that the Elo scaling law itself scales with the size of the game\. Like[Hilton et al\. \(2023\)](https://arxiv.org/html/2610.33420#bib.bib18), we focus on the single\-agent RL setup\. Instead of using intrinsic performance, we employ an Elo variant inspired by[Jones \(2021\)](https://arxiv.org/html/2610.33420#bib.bib22);[Neumann and Gros \(2023\)](https://arxiv.org/html/2610.33420#bib.bib28)as the performance metric which allows us to answer the performance question\. In contrast to prior works, which only studied a subset of the three questions, to the best of our knowledge our work is the first to study all three questions in the context of RL, rendering a more complete picture of RL scaling laws\.

Recently, the success of reasoning models\([Jaech et al\., 2024](https://arxiv.org/html/2610.33420#bib.bib21);[Comanici et al\., 2025](https://arxiv.org/html/2610.33420#bib.bib7);[Guo et al\., 2025](https://arxiv.org/html/2610.33420#bib.bib11)\)reinvigorated interest in RL scaling laws in the context of large language model \(LLM\) finetuning\. Examples include[Khatri et al\. \(2026\)](https://arxiv.org/html/2610.33420#bib.bib24),[Cheng et al\. \(2026\)](https://arxiv.org/html/2610.33420#bib.bib5), and[Tan et al\. \(2026\)](https://arxiv.org/html/2610.33420#bib.bib34)\. These works focus on a specific application domain of RL and concern domain\-specific quantities such as the number of parallel rollouts per prompt and the pass rate on a specific dataset\. In contrast, our work focuses on the more general sequential\-decision making setup with minimum assumptions about the environment and the reward structures\.

## 3Method

Our methodology implements a two\-stage framework adapted from the paradigm proposed by[Bi et al\. \(2024\)](https://arxiv.org/html/2610.33420#bib.bib3)\. First, we characterize hyperparameter scaling dynamics by conducting a systematic grid search across small\-to\-medium compute budgets to establish empirical power laws that parameterize optimal configurations as functions of training compute\. Second, we determine the joint scaling behavior of model size and performance using the isoFLOPs profile approach\([Hoffmann et al\., 2022](https://arxiv.org/html/2610.33420#bib.bib20)\), dynamically configuring the hyperparameters in this stage using the optimal values prescribed by our first\-stage scaling laws\. To validate this framework, we first conduct a rigorous scientific evaluation of predictive performance at a10×10\\timescompute frontier by independently comparing each scaling law’s prediction against the empirical ground truth\. We then showcase the practical utility of these scaling laws at a100×100\\timescompute scale, mirroring real\-world constraints where validating prediction accuracy is computationally prohibitive\. Rather than evaluating the laws themselves, this step demonstrates their use in practice: we execute a large\-scale training run configured by our hyperparameter and model size prescriptions, comparing its realized performance to the performance scaling law’s prediction to verify that the high\-budget training job meets expectation\.

### 3\.1Hyperparameter scaling laws

We used the A2C algorithm\([Mnih et al\., 2016](https://arxiv.org/html/2610.33420#bib.bib30)\)across all experiments in this work\. In addition to batch size and learning rate, the entropy regularization strength is also an important hyperparameter in A2C\. We did a preliminary grid search over combinations of these three hyperparameters at a low compute regime\. The optimal entropy regularization strength is consistent across different compute budgets\. Following this observation, we decided to focus on studying the scaling dynamics of batch size and learning rate\. Specifically, we fixed the rollout length of each trajectory and varied batch size by changing the number of parallel environment instances\.

To characterize the scaling dynamics of hyperparameters, we first need to find their optimal values for a set of different training budgets at a small scale\. For each training FLOPsCC, we conduct a grid search on batch size and learning rate across a few different model sizes to identify their optimal values\. Due to randomness in the training process, the empirical performance is a noisy estimate of the true performance\. Thus the empirical optimal hyperparameter configuration is often not the true optimum\. RL performance evaluation is known to suffer from higher variance than supervised learning\([Henderson et al\., 2018](https://arxiv.org/html/2610.33420#bib.bib14)\), which makes the identification problem even harder\. To reduce the variance of the empirical performance measurement, we conduct multiple independent jobs for each configuration with different random seeds\. We then aggregate the episode returns across random seeds by their interquartile mean \(IQM\)\. Moreover, to take into account the randomness of the empirical results, we do bootstrapping by resampling the random seeds for the experiments\. For each resampling, we pick the batch size and learning rate that yields the highest IQM of the episode return\. We call this configuration acompetitiveconfiguration\. Through bootstrapping, we obtain a set of competitive configurations for each training computeCC\. Instead of fitting the empirically optimal configurations, we fit power laws to all competitive configurations in the following forms:

B∗​\(C\)\\displaystyle B^\{\*\}\(C\)=aB​CdB\\displaystyle=a\_\{B\}C^\{d\_\{B\}\}\(1\)α∗​\(C\)\\displaystyle\\alpha^\{\*\}\(C\)=aα​Cdα\\displaystyle=a\_\{\\alpha\}C^\{d\_\{\\alpha\}\}\(2\)B∗​\(C\)B^\{\*\}\(C\)andα∗​\(C\)\\alpha^\{\*\}\(C\)represent the average competitive batch size and learning rate at training FLOPsCCrespectively\.aBa\_\{B\},dBd\_\{B\},aαa\_\{\\alpha\}, anddαd\_\{\\alpha\}are parameters fitted from the data\.

[Bi et al\. \(2024\)](https://arxiv.org/html/2610.33420#bib.bib3)employ a method that shares the same spirit as ours to combat the noise in optimal hyperparameter identification\. They define “near\-optimal” hyperparameters as the ones that yield a loss within0\.25%0\.25\\%of the empirical minimum\. Similar to our method, they then fit the scaling laws to the near\-optimal hyperparameters\. We do not use the same method because unlike cross entropy the episode return need not vary smoothly through training\. It is hard to pick a threshold for the “near\-optimal” definition\. Instead, we rely on bootstrapping to address randomness in experiments\.

### 3\.2Allocation scaling laws

To understand the optimal allocation of training compute, we first need to calculate the total FLOPsCCof a training run\. In Transformer\-based language models, this is often approximated byC=6​N​DC=6NDwhereNNdenotes the total number of parameters in the model andDDdenotes the total number of tokens consumed during training\([Kaplan et al\., 2020](https://arxiv.org/html/2610.33420#bib.bib23);[Hoffmann et al\., 2022](https://arxiv.org/html/2610.33420#bib.bib20)\)\. This approximation is based on dense layers\. However, we use convolutional networks which do not follow the same calculation\. Therefore, similar to the FLOPs\-per\-token notion used by[Bi et al\. \(2024\)](https://arxiv.org/html/2610.33420#bib.bib3), we use FLOPs\-per\-frame \(FPF\), denoted byMM, as a measurement of the model size\. Here a frame refers to the observation of a single step in the environment\. FPF is defined as the number of FLOPs spent on each observation during training\. The total training compute is thusC=M​DC=MDwhereDDdenotes the number of frames the agent is trained for\. Note that for a model\-based RL agent we may need to take additional factors such as planning steps into account\. We focus on model\-free RL agents in this work and leave the study of model\-based agents to future work\.

With the clear factoring of training FLOPs, we use the isoFLOPs profile approach\([Hoffmann et al\., 2022](https://arxiv.org/html/2610.33420#bib.bib20)\)to study the optimal compute allocation between model size and data size\. We need to find the optimal allocation at a set of different training FLOPs at small scale\. For each training FLOPsCC, we train models of different sizes using hyperparameters predicted by their corresponding scaling laws\. To combat variance, we run multiple independent jobs for each model size with different random seeds and aggregate the results by IQM\. Then we fit a parabola to approximate the mapping from FPF to performance\. We extract the optimum of the parabola to identify the optimal FPFM∗M^\{\*\}for training FLOPsCC\. Lastly, we fit a power law in the following form:

M∗​\(C\)=aM​CdM\\displaystyle M^\{\*\}\(C\)=a\_\{M\}C^\{d\_\{M\}\}\(3\)aMa\_\{M\}anddMd\_\{M\}are parameters fitted from data\. We do not need to fit a separate law for the optimal data sizeD∗D^\{\*\}as it can be derived fromM∗M^\{\*\}\.

### 3\.3Performance scaling laws in Elo

Before predicting the performance as a function of the training compute, we must first determine how performance is measured\. As observed by[Hilton et al\. \(2023\)](https://arxiv.org/html/2610.33420#bib.bib18), the compute\-optimal frontier need not vary smoothly when the performance is measured in mean episode return\. The fundamental reason is that the magnitude of rewards \(and thus returns\) does not track learning progress continuously\. To address this problem, we employ a variant of the Elo score\([Elo, 1978](https://arxiv.org/html/2610.33420#bib.bib8)\)as a performance measurement of asingle\-agentRL training system\. Elo is typically used to measure the relative playing strength of different players within a league in two\-player games\. In this experiment, the league consists of the final agent checkpoint of each training run\. We compute the Elo for each agent in our single\-agent environment by creating simulated two\-player games\. We reserve a fixed set of environment instances for evaluation\. Every agent is evaluated on this evaluation set\. We then set up a simulated tournament between every pair of agents\. For each pair, we treat each evaluation environment as an independent competition where the agent that achieved a higher return wins\. If both agents achieve the same score, it is a tie and counts0\.50\.5win for each agent\. By aggregating over the entire evaluation set, we compute a win rate of one agent against the other\. We do this for every pair of agents in the league and obtain a win rate matrix where cell\(i,j\)\(i,j\)denotes the win rate of agentiiagainst agentjj\. By fitting a Bradley\-Terry model\([Bradley and Terry, 1952](https://arxiv.org/html/2610.33420#bib.bib4)\)to this win rate matrix, we obtain an Elo score for each agent\. Note that Elo is a relative metric and thus the solution is not unique\. To anchor it, we subtract every agent’s Elo by the Elo of the random policy post hoc\.

For the performance scaling laws, we use the same data from the allocation experiment\. We solve the parabola to find the optimal performance for each training computeCCand fitted a scaling law in Elo\. Since the Elo score itself is already in log\-scale, instead of a power law, we fit a log\-linear law in the form below wheredElod\_\{\\text\{Elo\}\}andaEloa\_\{\\text\{Elo\}\}are parameters fitted from data:

Elo∗​\(C\)=dElo​log10​C\+aElo\\displaystyle\\text\{Elo\}^\{\*\}\(C\)=d\_\{\\text\{Elo\}\}\\log\_\{10\}\{C\}\+a\_\{\\text\{Elo\}\}\(4\)

## 4Experiment Results

EnvironmentaBa\_\{B\}dBd\_\{B\}aαa\_\{\\alpha\}dαd\_\{\\alpha\}aMa\_\{M\}dMd\_\{M\}aEloa\_\{\\text\{Elo\}\}dElod\_\{\\text\{Elo\}\}Starpilot7\.14​e−47\.14\\text\{e\}\{\-4\}0\.390\.394\.08​e−34\.08\\text\{e\}\{\-3\}−0\.023\-0\.0235\.35\.30\.440\.44−3186\-3186240\.97240\.97Craftax0\.280\.280\.260\.261\.57​e−21\.57\\text\{e\}\{\-2\}−0\.082\-0\.0820\.530\.530\.430\.43−1987\-1987223\.86223\.86Table 1:Summary of all fitted coefficients and exponents in the experiments presented in Section[4](https://arxiv.org/html/2610.33420#S4)\.In this section we present experiment results in Starpilot\([Cobbe et al\., 2020](https://arxiv.org/html/2610.33420#bib.bib6)\)and Craftax\([Matthews et al\., 2024](https://arxiv.org/html/2610.33420#bib.bib29)\)\. In the following sections we present the fitted scaling laws for hyperparameters, compute\-optimal model sizes, and optimal performance in detail\. Table[1](https://arxiv.org/html/2610.33420#S4.T1)summarizes the parameters of all fitted laws\.

### 4\.1Hyperparameters

In this experiment, we selected77training budgets between1​e​141\\text\{e\}\{14\}FLOPs and1​e​171\\text\{e\}\{17\}FLOPs\. For each budget, we conducted a grid search of77different learning rates and77different batch sizes with44models of different sizes\. We used the bootstrapping method described in Section[3](https://arxiv.org/html/2610.33420#S3)to identify a set of10001000competitive batch sizes and learning rates at each training budget\. Then we fitted power laws for the learning rate and the batch size using data between1​e​141\\text\{e\}\{14\}FLOPs and1​e​161\\text\{e\}\{16\}FLOPs and evaluated their predictions at1​e​171\\text\{e\}\{17\}FLOPs\. In Craftax,1​e​141\\text\{e\}\{14\}FLOPs was excluded from the power law fitting because we observed a violation of monotonic model size scaling from1​e​141\\text\{e\}\{14\}to3​e​143\\text\{e\}\{14\}, for which we provide a deeper analysis in Section[4\.4](https://arxiv.org/html/2610.33420#S4.SS4)\.

Figure[1](https://arxiv.org/html/2610.33420#S4.F1)visualizes the distributions of the competitive hyperparameters and the fitted power laws\. In both Starpilot and Craftax, the optimal batch size increases as a function of the training FLOPs\. The optimal learning rate roughly remains constant with a small negative exponent in Starpilot and a small positive exponent in Craftax\. Note that the learning rate scaling trend contradicts the common wisdom that it should decrease as training compute increases\. Nonetheless we will see that empirically its predictions are accurate\. The heatmaps in Figure[2](https://arxiv.org/html/2610.33420#S4.F2)visualize where the predicted batch sizes and learning rates are located in the performance contour at1​e​171\\text\{e\}\{17\}training FLOPs\. In both environments, the predictions are located near the center of the high performance region\. In Starpilot the optimal configuration yielded a mean episode return of32\.332\.3while the predicted configuration yielded32\.0232\.02\. In Craftax, the optimal mean episode return is64\.7664\.76while the predicted configuration yielded63\.4663\.46\. Overall, the predicted hyperparameters yield performance close to the empirical optimum, validating the predictive performance of the scaling laws\.

For the next isoFLOPs profile experiment, we used the scaling laws fitted on all data between1​e​141\\text\{e\}\{14\}FLOPs \(3​e​143\\text\{e\}\{14\}for Craftax\) and1​e​171\\text\{e\}\{17\}FLOPs\. The parameters of the fitted laws are shown in Table[1](https://arxiv.org/html/2610.33420#S4.T1)\.

![Refer to caption](https://arxiv.org/html/2610.33420v1/figures/starpilot_batch_size.png)

![Refer to caption](https://arxiv.org/html/2610.33420v1/figures/starpilot_learning_rate.png)

![Refer to caption](https://arxiv.org/html/2610.33420v1/figures/craftax_batch_size.png)

![Refer to caption](https://arxiv.org/html/2610.33420v1/figures/craftax_learning_rate.png)

Figure 1:The violin plots show the distributions of the competitive batch sizes and learning rates respectively\. The dashed lines visualize the fitted power laws\. Red stars mark predicted values\.![Refer to caption](https://arxiv.org/html/2610.33420v1/figures/starpilot_hyper_eval.png)

![Refer to caption](https://arxiv.org/html/2610.33420v1/figures/craftax_hyper_eval.png)

Figure 2:Visualization of performance contour of different hyperparameter configurations at1​e​171\\text\{e\}\{17\}training FLOPs\. The number in each cell is the mean episode return on the evaluation set, aggregated by IQM across55seeds\. The red stars denote the predicted configurations by the scaling laws\.
### 4\.2Allocation

We used the isoFLOPs profile approach\([Hoffmann et al\., 2022](https://arxiv.org/html/2610.33420#bib.bib20)\)to answer the allocation question\. We selected66training FLOPs budgets:1​e​151\\text\{e\}\{15\},3​e​153\\text\{e\}\{15\},1​e​161\\text\{e\}\{16\},3​e​163\\text\{e\}\{16\},1​e​171\\text\{e\}\{17\}, and1​e​181\\text\{e\}\{18\}\. For each training budget ofCCFLOPs, we trained multiple different model sizes\. We kept the depth of the neural network fixed and varied the width to scale the model size\. The batch sizes and the learning rates were predicted by the scaling laws fitted in the previous section\. To obtain robust results, we ran1010independent trials with different random seeds for each configuration\. We kept the final agent of each training job and formed a league of them together with a uniform random policy to compute an Elo score for each agent, using the method described in Section[3](https://arxiv.org/html/2610.33420#S3)\. For each configuration, we aggregated the Elo scores across all random seeds by IQM\. We fitted a parabola to the relationship betweenlog10⁡M\\log\_\{10\}\{M\}and the Elo score by weighted least squares where each data point is weighted by the inverse of its variance across random seeds\. Then we extracted its optima to find the corresponding optimal FPF and optimal performance\.

The left column in Figure[3](https://arxiv.org/html/2610.33420#S4.F3)shows the data points and the fitted parabolas and their optima\. We fitted a power law in the form of Eq\.[3](https://arxiv.org/html/2610.33420#S3.E3)using data between1​e​151\\text\{e\}\{15\}and1​e​171\\text\{e\}\{17\}FLOPs and evaluated its prediction at1​e​181\\text\{e\}\{18\}FLOPs\. The right column in Figure[3](https://arxiv.org/html/2610.33420#S4.F3)visualizes the fitted laws\. The coefficients and exponents of the fitted laws can be found in Table[1](https://arxiv.org/html/2610.33420#S4.T1)\. The predicted and the empirically optimal FPFs are summarized in Table[2](https://arxiv.org/html/2610.33420#S4.T2)\. In Starpilot, the power law fits the data well\. Although the prediction overestimates the optimal FPF by31%31\\%, the performance of the predicted model size is close to optimal \(see the yellow circles and stars at1​e​181\\text\{e\}\{18\}FLOPs in Figure[4](https://arxiv.org/html/2610.33420#S4.F4)\)\. In Craftax, the fit is less clean\. But nevertheless, the FPF prediction is extremely accurate with a relative error of only3%3\\%\. Overall the fitted scaling laws demonstrate strong prediction accuracy in both environments, rendering them reliable for informing the model size decisions for large scale training runs\.

EnvironmentpredictedFPFoptimalFPFrelativeerrorElo ofpredicted FPFoptimal Elowin rateerrorStarpilot4\.85​e​84\.85\\text\{e\}83\.69​e​83\.69\\text\{e\}8\+31%\+31\\%1102\.61102\.61110\.71110\.7−1\.17%\-1\.17\\%Craftax2\.73​e​72\.73\\text\{e\}72\.65​e​72\.65\\text\{e\}7\+3%\+3\\%2062\.82062\.82087\.52087\.5−3\.55%\-3\.55\\%Table 2:Data for evaluating the prediction accuracy of the optimal model size\. We compare the predicted FPF and its realized Elo to the empirical optimum\.![Refer to caption](https://arxiv.org/html/2610.33420v1/figures/starpilot_parabola.png)

![Refer to caption](https://arxiv.org/html/2610.33420v1/figures/starpilot_fpf.png)

![Refer to caption](https://arxiv.org/html/2610.33420v1/figures/craftax_parabola.png)

![Refer to caption](https://arxiv.org/html/2610.33420v1/figures/craftax_fpf.png)

Figure 3:Visualization of data from the isoFLOPs profile experiment in Section[4\.2](https://arxiv.org/html/2610.33420#S4.SS2)\. The left column visualizes the final performance of each training job and the fitted parabolas\. The x\-axis denotes the model size measured in FPF\. The y\-axis denotes the performance measured in Elo\. Each parabola corresponds to one training FLOPs\. The right column visualizes the derived optimal FPF and the fitted power law\. The x\-axis is the training FLOPs and the y\-axis is the optimal FPF for each training budget\. The blue dots are used for fitting the power laws and the yellow dots are used for evaluation\.
### 4\.3Performance

We used the same data from the previous section to fit a law betweenCCand the Elo score in the form of Eq\.[4](https://arxiv.org/html/2610.33420#S3.E4)\. The parameters of the fitted laws are in Table[1](https://arxiv.org/html/2610.33420#S4.T1)\. The left column in Figure[4](https://arxiv.org/html/2610.33420#S4.F4)visualizes the fitted laws\. In both Starpilot and Craftax, the laws fit the data in\[1​e​15,1​e​17\]\[1\\text\{e\}\{15\},1\\text\{e\}\{17\}\]well and predict the Elo at1​e​181\\text\{e\}\{18\}accurately, resulting in a\+5\.84%\+5\.84\\%and−6\.37%\-6\.37\\%error in win rates respectively\.

We also compare Elo to mean episode return as the performance metric\. We fitted similar log\-linear laws for the mean episode return, which are shown in the right column of Figure[4](https://arxiv.org/html/2610.33420#S4.F4)\. In Starpilot, the mean episode return law fits the data well but extrapolates poorly in comparison to the Elo law\. In Craftax, the mean episode return law does not even fit the data well and the prediction error at1​e​181\\text\{e\}\{18\}FLOPs is huge\. In both environments, Elo extrapolates significantly better than mean episode return, proving itself as the better performance metric for fitting predictive scaling laws\.

In each environment, we also conducted a hero run which trained a model with the size predicted by the FPF scaling law for1​e​191\\text\{e\}\{19\}FLOPs\. Each run consists of1010independent trials with different random seeds\. The performance results are summarized in Table[3](https://arxiv.org/html/2610.33420#S4.T3)and Figure[4](https://arxiv.org/html/2610.33420#S4.F4)\. We compare the predicted Elo to the Elo actually achieved\. The deviations are\+80\.2\+80\.2and\+80\.9\+80\.9for Starpilot and Craftax respectively, which translate to\+11\.34%\+11\.34\\%and\+11\.44%\+11\.44\\%deviation in the win rate\. We want to highlight the result in Craftax\. Through a compute\-optimal allocation of training resources, we achieve a new state\-of\-the\-art performance of54\.0%54\.0\\%, more than doubling the previous best published baseline of20\.6%20\.6\\%\([Hamon, 2024](https://arxiv.org/html/2610.33420#bib.bib12)\)\. Note that our hero run consumed significantly more data than the baseline \(135135billion framesvs44billion frames\)\. This is intentional as the focus of this work is on compute optimality rather than data efficiency\.

Environmentpredicted@1​e​181\\text\{e\}\{18\}optimal@1​e​181\\text\{e\}\{18\}win rate error@1​e​181\\text\{e\}\{18\}predicted@1​e​191\\text\{e\}\{19\}actual@1​e​191\\text\{e\}\{19\}win rate error@1​e​191\\text\{e\}\{19\}Starpilot1151\.51151\.51110\.71110\.7\+5\.84%\+5\.84\\%1392\.51392\.51312\.31312\.3\+11\.34%\+11\.34\\%Craftax2043\.02043\.02087\.52087\.5−6\.37%\-6\.37\\%2266\.82266\.82185\.92185\.9\+11\.44%\+11\.44\\%Table 3:Data for evaluating the accuracy of the Elo predictions\. At1​e​181\\text\{e\}\{18\}FLOPs, we compare the predicted Elo to the empirically optimal Elo\. At1​e​191\\text\{e\}\{19\}FLOPs, we compare the predicted Elo to the Elo realized by the predicted optimal configuration\.![Refer to caption](https://arxiv.org/html/2610.33420v1/figures/starpilot_elo.png)

![Refer to caption](https://arxiv.org/html/2610.33420v1/figures/starpilot_mean_episode_return.png)

![Refer to caption](https://arxiv.org/html/2610.33420v1/figures/craftax_elo.png)

![Refer to caption](https://arxiv.org/html/2610.33420v1/figures/craftax_mean_episode_return.png)

Figure 4:Comparison of different performance metrics\. The left column is Elo and the right column is mean episode return\. Dots denote empirical optimum\. Stars denote the performance of the configuration predicted by the scaling laws\. Blue dots are used for fitting the scaling laws\. Yellow dots and stars are used for evaluation\.
### 4\.4Inverse scaling analysis

In Craftax, we observed a double\-crossover between3​e​133\\text\{e\}\{13\}and1​e​151\\text\{e\}\{15\}FLOPs, demonstrating a local inverse scaling pattern where a smaller model outperforms a larger model as the training budget increases\. The left plot in Figure[5](https://arxiv.org/html/2610.33420#S4.F5)shows this phenomenon\. The bigger model outperforms the smaller model at1​e​141\\text\{e\}\{14\}FLOPs but underperforms at3​e​143\\text\{e\}\{14\}FLOPs\. Eventually at1​e​151\\text\{e\}\{15\}the bigger model performs better again\. This observation contradicts the common belief that the optimal model size grows monotonically as we scale up training compute\.

We conjecture that this phenomenon stems from a specific design of the game dynamics and the pathological learning dynamics it incurs\. In Craftax, the agent must defeat at least88enemies on each level to unlock the door to the next level\. This rule applies to every level except the first, on which the door to the second level \(the “dungeon”\) is always open from the beginning of the episode, rendering battling optional\. However, defeating enemies is relatively easier than entering the dungeon because the door is always located at the center of a “mountain”, surrounded by stones\. Consequently, the agent learns to defeat enemies before switching to entering the dungeon\. During the transition, it stops defeating enemies because it is unnecessary though rewarding\. In other words, as the experience accumulates, the agent learns to master the more rewarding skill of entering the dungeon while forgetting the less rewarding skill of battling\. These learning dynamics differ drastically from the supervised learning setting where the model needs to fit a growing set of data, which naturally demands more capacity\. In this case, however, the agent can reallocate the model capacity from battling to path finding without growing the model size\. The smaller model then has an advantage over the larger model because it sees more data under the same compute budget\.

Our conjecture is supported by the achievements the agent unlocked during training, shown in the right plot of Figure[5](https://arxiv.org/html/2610.33420#S4.F5)\. Up to1​e​141\\text\{e\}\{14\}FLOPs, the agent has little success in entering the dungeon regardless of its model size\. Progress mainly comes from better battling skills\. However, at3​e​143\\text\{e\}\{14\}FLOPs, the agent learns to enter the dungeon while avoiding battles at the same time\. Instead of growing its skill set, which demands more model capacity, the agent switches from one skill to another, giving the smaller model advantages\.

This pathology of one reward source suppressing the learning signal from another reward source during the early phase of training resembles Ray interference\([Schaul et al\., 2019](https://arxiv.org/html/2610.33420#bib.bib32)\)\. It is a product of the interaction between the environment dynamics and the learning dynamics\. Therefore, a different learning algorithm than actor\-critic may demonstrate a different scaling behavior\. We leave the comparison of the scaling properties of different algorithms to future work\.

![Refer to caption](https://arxiv.org/html/2610.33420v1/figures/crossover.png)

![Refer to caption](https://arxiv.org/html/2610.33420v1/figures/achievements.png)

Figure 5:Visualization of the inverse scaling pattern in Craftax\. The left plot shows the best performance of two different model sizes across different training FLOPs\. The right plot shows the success rate of the enemy defeat achievements and the entering dungeon achievement\. Error bars show the standard error across55random seeds\.

## 5Conclusion

In this work, we presented a systematic empirical study of scaling laws in RL, investigating hyperparameter configuration, compute allocation, and performance prediction across the Starpilot and Craftax environments\. We established that optimal hyperparameters follow power\-law relationships with respect to total training compute\. Furthermore, we also characterized the optimal allocation between model size and data size as functions of training FLOPs\. To address the non\-smoothness of mean episode return, we employed a single\-agent variant of the Elo score\. We demonstrated that the Elo score scales predictably, enabling robust performance extrapolation up to two orders of magnitude in training compute\. Guided by these scaling laws, we scaled up a simple actor\-critic algorithm to1​e​191\\text\{e\}\{19\}training FLOPs and achieved a new state\-of\-the\-art performance of54\.0%54\.0\\%in the challenging Craftax benchmark\. However, our analysis also uncovered a local inverse scaling pathology in Craftax, which we connected to a pathology known as Ray interference, illustrating that global scaling trends can obscure non\-monotonic local dynamics\.

Despite these contributions, several limitations remain to be addressed in future work\. First, our empirical study was restricted to a single model\-free actor\-critic algorithm \(A2C\); further research is required to verify if our findings generalize to other RL algorithms such as value\-based or model\-based algorithms\. Second, while the Elo score offers predictable mathematical scaling, it is fundamentally a relative metric within a closed league of evaluated agents, lacking direct semantic grounding to the underlying task\. Consequently, in domains where rewards carry critical semantic meanings, Elo\-based scaling laws may be less effective at providing informative or absolute performance predictions\. Finally, our discovery of Ray interference in Craftax indicates that RL scaling is highly sensitive to the specific alignment of environmental dynamics and reward structures, suggesting that a single, universal scaling law may not apply across all arbitrary RL environments\. Designing algorithms resilient to Ray interference remains a critical avenue for future research\.

### AI use statement

In this work, we used generative AI tools forimplement methods, design or provide feedback on research methodology or experiments, clean and reformat dataset, interpret results\. We have not used generative AI tools for generate synthetic data sets, help develop theoretical models or conceptual framework, propose or refine hypotheses and formulate mathematical claims, provide critical ingredients for proving mathematical claims, assist in the writing of proofs, assist with translation, support qualitative and thematic data analysis\. are not applicable to this work\. Additionally, we used generative AI tools foredit a research paper to improve readability\. We have reviewed all AI\-assisted work\. We reviewed the AI\-generated code for experiment and data analysis\. We proofread the text revised by AI tools\. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI\.

#### Acknowledgments

We thank Dan A\. Calian and Satinder Singh for their valuable feedback on an early version of this manuscript\.

## References

- Baet al\.\(2016\)J\. L\. Ba, J\. R\. Kiros, and G\. E\. HintonLayer normalization\.arXiv preprint arXiv:1607\.06450\.Cited by:[§A\.1](https://arxiv.org/html/2610.33420#A1.SS1.p1.1)\.
- Berneret al\.\(2019\)C\. Berner, G\. Brockman, B\. Chan, V\. Cheung, C\. Dennison, D\. Farhi, Q\. Fischer, S\. Hashme, C\. Hesse, R\. JĂłzefowicz,et al\.Dota 2 with large scale deep reinforcement learning\.arXiv preprint arXiv:1912\.06680\.Cited by:[§1](https://arxiv.org/html/2610.33420#S1.p2.1)\.
- Biet al\.\(2024\)X\. Bi, D\. Chen, G\. Chen, S\. Chen, D\. Dai, C\. Deng, H\. Ding, K\. Dong, Q\. Du, Z\. Fu,et al\.Deepseek llm: scaling open\-source language models with longtermism\.arXiv preprint arXiv:2401\.02954\.Cited by:[§1](https://arxiv.org/html/2610.33420#S1.p1.1),[§2](https://arxiv.org/html/2610.33420#S2.p1.1),[§3\.1](https://arxiv.org/html/2610.33420#S3.SS1.p3.1),[§3\.2](https://arxiv.org/html/2610.33420#S3.SS2.p1.1),[§3](https://arxiv.org/html/2610.33420#S3.p1.1)\.
- Bradley and Terry \(1952\)R\. A\. Bradley and M\. E\. TerryRank analysis of incomplete block designs: i\. the method of paired comparisons\.Biometrika39\(3/4\),pp\. 324–345\.Cited by:[§3\.3](https://arxiv.org/html/2610.33420#S3.SS3.p1.1)\.
- Chenget al\.\(2026\)Z\. Cheng, Y\. Xie, Y\. Qu, A\. Setlur, S\. Hao, V\. Pimpalkhute, T\. Liang, F\. Yao, Z\. Liu, E\. Xing,et al\.IsoCompute playbook: optimally scaling sampling compute for llm rl\.arXiv preprint arXiv:2603\.12151\.Cited by:[§2](https://arxiv.org/html/2610.33420#S2.p3.1)\.
- Cobbeet al\.\(2020\)K\. Cobbe, C\. Hesse, J\. Hilton, and J\. SchulmanLeveraging procedural generation to benchmark reinforcement learning\.InInternational conference on machine learning,pp\. 2048–2056\.Cited by:[§1](https://arxiv.org/html/2610.33420#S1.p3.1),[§4](https://arxiv.org/html/2610.33420#S4.p1.1)\.
- Comaniciet al\.\(2025\)G\. Comanici, E\. Bieber, M\. Schaekermann, I\. Pasupat, N\. Sachdeva, I\. Dhillon, M\. Blistein, O\. Ram, D\. Zhang, E\. Rosen,et al\.Gemini 2\.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.arXiv preprint arXiv:2507\.06261\.Cited by:[§2](https://arxiv.org/html/2610.33420#S2.p3.1)\.
- Elo \(1978\)A\. E\. EloThe rating of chessplayers, past and present\.Arco Pub\.,New York\.External Links:ISBN 0668047216Cited by:[§1](https://arxiv.org/html/2610.33420#S1.p3.1),[§2](https://arxiv.org/html/2610.33420#S2.p2.1),[§3\.3](https://arxiv.org/html/2610.33420#S3.SS3.p1.1)\.
- Espeholtet al\.\(2018\)L\. Espeholt, H\. Soyer, R\. Munos, K\. Simonyan, V\. Mnih, T\. Ward, Y\. Doron, V\. Firoiu, T\. Harley, I\. Dunning,et al\.Impala: scalable distributed deep\-rl with importance weighted actor\-learner architectures\.InInternational conference on machine learning,pp\. 1407–1416\.Cited by:[§A\.1](https://arxiv.org/html/2610.33420#A1.SS1.p1.1),[§A\.2](https://arxiv.org/html/2610.33420#A1.SS2.p2.1)\.
- Fuet al\.\(2026\)P\. Fu, O\. Rybkin, Z\. P\. Zhou, M\. Nauman, P\. Abbeel, S\. Levine, and A\. KumarCompute\-optimal scaling for value\-based deep rl\.Advances in Neural Information Processing Systems38,pp\. 160363–160402\.Cited by:[§2](https://arxiv.org/html/2610.33420#S2.p2.1)\.
- Guoet al\.\(2025\)D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§1](https://arxiv.org/html/2610.33420#S1.p2.1),[§2](https://arxiv.org/html/2610.33420#S2.p3.1)\.
- Hamon \(2024\)G\. HamontransformerXL\_PPO\_JAX\.Note:External Links:[Link](https://inria.hal.science/hal-04659863)Cited by:[§4\.3](https://arxiv.org/html/2610.33420#S4.SS3.p3.1)\.
- Heet al\.\(2016\)K\. He, X\. Zhang, S\. Ren, and J\. SunDeep residual learning for image recognition\.InProceedings of the IEEE conference on computer vision and pattern recognition,pp\. 770–778\.Cited by:[§A\.1](https://arxiv.org/html/2610.33420#A1.SS1.p1.1)\.
- Hendersonet al\.\(2018\)P\. Henderson, R\. Islam, P\. Bachman, J\. Pineau, D\. Precup, and D\. MegerDeep reinforcement learning that matters\.InProceedings of the AAAI conference on artificial intelligence,Vol\.32\.Cited by:[§3\.1](https://arxiv.org/html/2610.33420#S3.SS1.p2.1)\.
- Hendrycks and Gimpel \(2016\)D\. Hendrycks and K\. GimpelGaussian error linear units \(gelus\)\.arXiv: Learning\.External Links:[Link](https://arxiv.org/abs/1606.08415)Cited by:[§A\.1](https://arxiv.org/html/2610.33420#A1.SS1.p2.1)\.
- Henighanet al\.\(2020\)T\. Henighan, J\. Kaplan, M\. Katz, M\. Chen, C\. Hesse, J\. Jackson, H\. Jun, T\. B\. Brown, P\. Dhariwal, S\. Gray,et al\.Scaling laws for autoregressive generative modeling\.arXiv preprint arXiv:2010\.14701\.Cited by:[§2](https://arxiv.org/html/2610.33420#S2.p1.1)\.
- Hestnesset al\.\(2017\)J\. Hestness, S\. Narang, N\. Ardalani, G\. Diamos, H\. Jun, H\. Kianinejad, M\. M\. A\. Patwary, Y\. Yang, and Y\. ZhouDeep learning scaling is predictable, empirically\.arXiv preprint arXiv:1712\.00409\.Cited by:[§2](https://arxiv.org/html/2610.33420#S2.p1.1)\.
- Hiltonet al\.\(2023\)J\. Hilton, J\. Tang, and J\. SchulmanScaling laws for single\-agent reinforcement learning\.arXiv preprint arXiv:2301\.13442\.Cited by:[§1](https://arxiv.org/html/2610.33420#S1.p2.1),[§1](https://arxiv.org/html/2610.33420#S1.p3.1),[§2](https://arxiv.org/html/2610.33420#S2.p2.1),[§3\.3](https://arxiv.org/html/2610.33420#S3.SS3.p1.1)\.
- Hochreiter and Schmidhuber \(1997\)S\. Hochreiter and J\. SchmidhuberLong short\-term memory\.Neural computation9\(8\),pp\. 1735–1780\.Cited by:[§A\.1](https://arxiv.org/html/2610.33420#A1.SS1.p2.1)\.
- Hoffmannet al\.\(2022\)J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. d\. L\. Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark,et al\.Training compute\-optimal large language models\.arXiv preprint arXiv:2203\.15556\.Cited by:[§1](https://arxiv.org/html/2610.33420#S1.p1.1),[§2](https://arxiv.org/html/2610.33420#S2.p1.1),[§3\.2](https://arxiv.org/html/2610.33420#S3.SS2.p1.1),[§3\.2](https://arxiv.org/html/2610.33420#S3.SS2.p2.1),[§3](https://arxiv.org/html/2610.33420#S3.p1.1),[§4\.2](https://arxiv.org/html/2610.33420#S4.SS2.p1.1)\.
- Jaechet al\.\(2024\)A\. Jaech, A\. Kalai, A\. Lerer, A\. Richardson, A\. El\-Kishky, A\. Low, A\. Helyar, A\. Madry, A\. Beutel, A\. Carney,et al\.Openai o1 system card\.arXiv preprint arXiv:2412\.16720\.Cited by:[§2](https://arxiv.org/html/2610.33420#S2.p3.1)\.
- Jones \(2021\)A\. L\. JonesScaling scaling laws with board games\.arXiv preprint arXiv:2104\.03113\.Cited by:[§1](https://arxiv.org/html/2610.33420#S1.p3.1),[§2](https://arxiv.org/html/2610.33420#S2.p2.1)\.
- Kaplanet al\.\(2020\)J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. AmodeiScaling laws for neural language models\.arXiv preprint arXiv:2001\.08361\.Cited by:[§1](https://arxiv.org/html/2610.33420#S1.p1.1),[§2](https://arxiv.org/html/2610.33420#S2.p1.1),[§3\.2](https://arxiv.org/html/2610.33420#S3.SS2.p1.1)\.
- Khatriet al\.\(2026\)D\. Khatri, L\. Madaan, R\. Tiwari, R\. Bansal, V\. S\. S\. S\. Duvvuri, M\. Zaheer, I\. Dhillon, D\. Brandfonbrener, and R\. AgarwalThe art of scaling reinforcement learning compute for llms\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 72438–72467\.Cited by:[§1](https://arxiv.org/html/2610.33420#S1.p2.1),[§2](https://arxiv.org/html/2610.33420#S2.p3.1)\.
- Kingma and Ba \(2015\)D\. P\. Kingma and J\. BaAdam: a method for stochastic optimization\.\.InICLR \(Poster\),Y\. Bengio and Y\. LeCun \(Eds\.\),External Links:[Link](http://dblp.uni-trier.de/db/conf/iclr/iclr2015.html#KingmaB14)Cited by:[§A\.2](https://arxiv.org/html/2610.33420#A1.SS2.p1.1),[§A\.2](https://arxiv.org/html/2610.33420#A1.SS2.p2.1)\.
- Malladiet al\.\(2022\)S\. Malladi, K\. Lyu, A\. Panigrahi, and S\. AroraOn the sdes and scaling rules for adaptive gradient algorithms\.Advances in Neural Information Processing Systems35,pp\. 7697–7711\.Cited by:[Appendix B](https://arxiv.org/html/2610.33420#A2.p1.1)\.
- Matthewset al\.\(2024\)M\. Matthews, M\. Beukman, B\. Ellis, M\. Samvelyan, M\. T\. Jackson, S\. Coward, and J\. N\. FoersterCraftax: a lightning\-fast benchmark for open\-ended reinforcement learning\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 35104–35137\.External Links:[Link](https://proceedings.mlr.press/v235/matthews24a.html)Cited by:[§1](https://arxiv.org/html/2610.33420#S1.p3.1),[§4](https://arxiv.org/html/2610.33420#S4.p1.1)\.
- Mnihet al\.\(2016\)V\. Mnih, A\. P\. Badia, M\. Mirza, A\. Graves, T\. Lillicrap, T\. Harley, D\. Silver, and K\. KavukcuogluAsynchronous methods for deep reinforcement learning\.InProceedings of The 33rd International Conference on Machine Learning,M\. F\. Balcan and K\. Q\. Weinberger \(Eds\.\),Proceedings of Machine Learning Research, Vol\.48,New York, New York, USA,pp\. 1928–1937\.External Links:[Link](https://proceedings.mlr.press/v48/mniha16.html)Cited by:[Appendix A](https://arxiv.org/html/2610.33420#A1.p1.1),[§3\.1](https://arxiv.org/html/2610.33420#S3.SS1.p1.1)\.
- Muennighoffet al\.\(2023\)N\. Muennighoff, A\. Rush, B\. Barak, T\. Le Scao, N\. Tazi, A\. Piktus, S\. Pyysalo, T\. Wolf, and C\. A\. RaffelScaling data\-constrained language models\.Advances in Neural Information Processing Systems36,pp\. 50358–50376\.Cited by:[§1](https://arxiv.org/html/2610.33420#S1.p1.1)\.
- Neumann and Gros \(2023\)O\. Neumann and C\. GrosScaling laws for a multi\-agent reinforcement learning model\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=ZrEbzL9eQ3W)Cited by:[§1](https://arxiv.org/html/2610.33420#S1.p3.1),[§2](https://arxiv.org/html/2610.33420#S2.p2.1)\.
- Rosenfeldet al\.\(2020\)J\. S\. Rosenfeld, A\. Rosenfeld, Y\. Belinkov, and N\. ShavitA constructive prediction of the generalization error across scales\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=ryenvpEKDr)Cited by:[§2](https://arxiv.org/html/2610.33420#S2.p1.1)\.
- Schaulet al\.\(2019\)T\. Schaul, D\. Borsa, J\. Modayil, and R\. PascanuRay interference: a source of plateaus in deep reinforcement learning\.arXiv preprint arXiv:1904\.11455\.Cited by:[§1](https://arxiv.org/html/2610.33420#S1.p3.1),[§4\.4](https://arxiv.org/html/2610.33420#S4.SS4.p4.1)\.
- Silveret al\.\(2017\)D\. Silver, J\. Schrittwieser, K\. Simonyan, I\. Antonoglou, A\. Huang, A\. Guez, T\. Hubert, L\. Baker, M\. Lai, A\. Bolton,et al\.Mastering the game of go without human knowledge\.nature550\(7676\),pp\. 354–359\.Cited by:[§1](https://arxiv.org/html/2610.33420#S1.p2.1),[§2](https://arxiv.org/html/2610.33420#S2.p2.1)\.
- Tanet al\.\(2026\)Z\. Tan, H\. Geng, X\. Yu, M\. Zhang, G\. Wan, Y\. Zhou, Q\. He, X\. Xue, H\. Zhou, Y\. Fan,et al\.Scaling behaviors of llm reinforcement learning post\-training: an empirical study in mathematical reasoning\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 31300–31319\.Cited by:[§2](https://arxiv.org/html/2610.33420#S2.p3.1)\.
- Vinyalset al\.\(2019\)O\. Vinyals, I\. Babuschkin, W\. M\. Czarnecki, M\. Mathieu, A\. Dudzik, J\. Chung, D\. H\. Choi, R\. Powell, T\. Ewalds, P\. Georgiev,et al\.Grandmaster level in starcraft ii using multi\-agent reinforcement learning\.nature575\(7782\),pp\. 350–354\.Cited by:[§1](https://arxiv.org/html/2610.33420#S1.p2.1)\.

## Appendix ATraining Algorithm Details

In both Starpilot and Craftax, we train agents using the A2C\([Mnih et al\., 2016](https://arxiv.org/html/2610.33420#bib.bib30)\)algorithm\.

### A\.1Neural Network Architectures\.

In Starpilot experiments, we use the IMPALA\([Espeholt et al\., 2018](https://arxiv.org/html/2610.33420#bib.bib9)\)residual convolutional neural network architecture\([He et al\., 2016](https://arxiv.org/html/2610.33420#bib.bib13)\)with a shared torso between the policy and the value function\. Layer normalization\([Ba et al\., 2016](https://arxiv.org/html/2610.33420#bib.bib1)\)is applied between every layer and its following activation function\. We vary the model size by the number of convolutional channels of the first layer\. The width of subsequent layers scale proportionally\.

In Craftax we use the symbolic observations\. We apply a1×11\\times 1convolutional layer withEEchannels to the map grid\. We then flatten the grid and apply a linear layer with dimensionH=16​EH=16Eto compress the map to a compact vector representation\. The inventory is encoded to a vector of the same dimensionalityHHby a linear layer\. We fuse the map and the inventory by adding their vectors together\. Note that we apply layer normalization to their vectors independentlybeforefusion\. The fused vector is fed into an LSTM\([Hochreiter and Schmidhuber, 1997](https://arxiv.org/html/2610.33420#bib.bib19)\)cell withHHhidden units to handle partial observability\. Lastly, we apply linear layers to the output of the LSTM cell to map the state representation to the policy logits and the value function\. We used GELU\([Hendrycks and Gimpel, 2016](https://arxiv.org/html/2610.33420#bib.bib15)\)activation\. Similarly, we apply layer normalization before every activation layer\. We control the model size by varying the number of channels in the1×11\\times 1convolutional layerEE\.

### A\.2Hyperparameters

In Starpilot experiments, we use a discount factor of0\.9990\.999\. We set the rollout length to a constant1616\. The weight for the value function loss is0\.50\.5and the entropy regularization strength is0\.010\.01\. We use the Adam\([Kingma and Ba, 2015](https://arxiv.org/html/2610.33420#bib.bib25)\)optimizer withβ1=0\.9\\beta\_\{1\}=0\.9,β2=0\.999\\beta\_\{2\}=0\.999, andϵ=1​e−8\\epsilon=1\\text\{e\}\{\-8\}\. The learning rate is linearly decayed to00through training\. We clip the gradient by global norm to0\.50\.5\.

In Craftax, we use a discount factor of0\.9990\.999\. We set the rollout length to a constant1616\. We break each batch of rollouts into44minibatches and thus apply44gradient updates per batch\. We do minibatching to improve the overall throughput of the training system because we can employ more parallel environments\. The batch size in Section[4](https://arxiv.org/html/2610.33420#S4)is the number of observations in each minibatch\. We apply VTrace\([Espeholt et al\., 2018](https://arxiv.org/html/2610.33420#bib.bib9)\)to handle off\-policyness caused by minibatch\. The weight for the value function loss is0\.50\.5and the entropy regularization strength is0\.0010\.001\. We use the Adam\([Kingma and Ba, 2015](https://arxiv.org/html/2610.33420#bib.bib25)\)optimizer withβ1=0\.9\\beta\_\{1\}=0\.9,β2=0\.99\\beta\_\{2\}=0\.99, andϵ=1​e−8\\epsilon=1\\text\{e\}\{\-8\}\. The learning rate is linearly decayed to00through training\. We clip the gradient by global norm to0\.50\.5\.

In both environments, we reserved10241024instances for evaluation\.

## Appendix BExperiment Configurations

Table[4](https://arxiv.org/html/2610.33420#A2.T4)and Table[5](https://arxiv.org/html/2610.33420#A2.T5)summarize the configuration of the grid search in the hyperparameter scaling law experiments for Starpilot and Craftax respectively\. Table[6](https://arxiv.org/html/2610.33420#A2.T6)and Table[7](https://arxiv.org/html/2610.33420#A2.T7)summarize the configuration for the isoFLOPs profile experiments in Starpilot and Craftax respectively\. Note that in the isoFLOPs profile experiments we rounded the batch size to the closest power of22and scaled the paired learning rate by the square\-root rule\([Malladi et al\., 2022](https://arxiv.org/html/2610.33420#bib.bib26)\)\.

FLOPsModel widthBatch sizeLearning rate1​e​141\\text\{e\}\{14\}\{1,2,3,4\}\\\{1,2,3,4\\\}\{26,27,28,29,210,211,212\}\\\{2^\{6\},2^\{7\},2^\{8\},2^\{9\},2^\{10\},2^\{11\},2^\{12\}\\\}\{2−11,2−10\.5,2−10,2−9\.5,2−9,2−8\.5,2−8\}\\\{2^\{\-11\},2^\{\-10\.5\},2^\{\-10\},2^\{\-9\.5\},2^\{\-9\},2^\{\-8\.5\},2^\{\-8\}\\\}3​e​143\\text\{e\}\{14\}\{2,3,4,6\}\\\{2,3,4,6\\\}\{26,27,28,29,210,211,212\}\\\{2^\{6\},2^\{7\},2^\{8\},2^\{9\},2^\{10\},2^\{11\},2^\{12\}\\\}\{2−11,2−10\.5,2−10,2−9\.5,2−9,2−8\.5,2−8\}\\\{2^\{\-11\},2^\{\-10\.5\},2^\{\-10\},2^\{\-9\.5\},2^\{\-9\},2^\{\-8\.5\},2^\{\-8\}\\\}1​e​151\\text\{e\}\{15\}\{3,4,6,8\}\\\{3,4,6,8\\\}\{27,28,29,210,211,212,213\}\\\{2^\{7\},2^\{8\},2^\{9\},2^\{10\},2^\{11\},2^\{12\},2^\{13\}\\\}\{2−11,2−10\.5,2−10,2−9\.5,2−9,2−8\.5,2−8\}\\\{2^\{\-11\},2^\{\-10\.5\},2^\{\-10\},2^\{\-9\.5\},2^\{\-9\},2^\{\-8\.5\},2^\{\-8\}\\\}3​e​153\\text\{e\}\{15\}\{4,6,8,12\}\\\{4,6,8,12\\\}\{27,28,29,210,211,212,213\}\\\{2^\{7\},2^\{8\},2^\{9\},2^\{10\},2^\{11\},2^\{12\},2^\{13\}\\\}\{2−11,2−10\.5,2−10,2−9\.5,2−9,2−8\.5,2−8\}\\\{2^\{\-11\},2^\{\-10\.5\},2^\{\-10\},2^\{\-9\.5\},2^\{\-9\},2^\{\-8\.5\},2^\{\-8\}\\\}1​e​161\\text\{e\}\{16\}\{6,8,12,16\}\\\{6,8,12,16\\\}\{27,28,29,210,211,212,213\}\\\{2^\{7\},2^\{8\},2^\{9\},2^\{10\},2^\{11\},2^\{12\},2^\{13\}\\\}\{2−11,2−10\.5,2−10,2−9\.5,2−9,2−8\.5,2−8\}\\\{2^\{\-11\},2^\{\-10\.5\},2^\{\-10\},2^\{\-9\.5\},2^\{\-9\},2^\{\-8\.5\},2^\{\-8\}\\\}3​e​163\\text\{e\}\{16\}\{8,12,16,24\}\\\{8,12,16,24\\\}\{28,29,210,211,212,213,214\}\\\{2^\{8\},2^\{9\},2^\{10\},2^\{11\},2^\{12\},2^\{13\},2^\{14\}\\\}\{2−11,2−10\.5,2−10,2−9\.5,2−9,2−8\.5,2−8\}\\\{2^\{\-11\},2^\{\-10\.5\},2^\{\-10\},2^\{\-9\.5\},2^\{\-9\},2^\{\-8\.5\},2^\{\-8\}\\\}1​e​171\\text\{e\}\{17\}\{8,12,16,24\}\\\{8,12,16,24\\\}\{29,210,211,212,213,214,215\}\\\{2^\{9\},2^\{10\},2^\{11\},2^\{12\},2^\{13\},2^\{14\},2^\{15\}\\\}\{2−11,2−10\.5,2−10,2−9\.5,2−9,2−8\.5,2−8\}\\\{2^\{\-11\},2^\{\-10\.5\},2^\{\-10\},2^\{\-9\.5\},2^\{\-9\},2^\{\-8\.5\},2^\{\-8\}\\\}Table 4:Grid search configurations for fitting hyperparameter scaling laws in Starpilot\.FLOPsModel widthBatch sizeLearning rate3​e​143\\text\{e\}\{14\}\{3,4,6,8\}\\\{3,4,6,8\\\}\{27,28,29,210,211,212,213\}\\\{2^\{7\},2^\{8\},2^\{9\},2^\{10\},2^\{11\},2^\{12\},2^\{13\}\\\}\{2−12,2−11\.5,2−11,2−10\.5,2−10,2−9\.5,2−9\}\\\{2^\{\-12\},2^\{\-11\.5\},2^\{\-11\},2^\{\-10\.5\},2^\{\-10\},2^\{\-9\.5\},2^\{\-9\}\\\}1​e​151\\text\{e\}\{15\}\{4,6,8,12\}\\\{4,6,8,12\\\}\{28,29,210,211,212,213,214\}\\\{2^\{8\},2^\{9\},2^\{10\},2^\{11\},2^\{12\},2^\{13\},2^\{14\}\\\}\{2−12,2−11\.5,2−11,2−10\.5,2−10,2−9\.5,2−9\}\\\{2^\{\-12\},2^\{\-11\.5\},2^\{\-11\},2^\{\-10\.5\},2^\{\-10\},2^\{\-9\.5\},2^\{\-9\}\\\}3​e​153\\text\{e\}\{15\}\{6,8,12,16\}\\\{6,8,12,16\\\}\{28,29,210,211,212,213,214\}\\\{2^\{8\},2^\{9\},2^\{10\},2^\{11\},2^\{12\},2^\{13\},2^\{14\}\\\}\{2−12,2−11\.5,2−11,2−10\.5,2−10,2−9\.5,2−9\}\\\{2^\{\-12\},2^\{\-11\.5\},2^\{\-11\},2^\{\-10\.5\},2^\{\-10\},2^\{\-9\.5\},2^\{\-9\}\\\}1​e​161\\text\{e\}\{16\}\{6,8,12,16\}\\\{6,8,12,16\\\}\{29,210,211,212,213,214,215\}\\\{2^\{9\},2^\{10\},2^\{11\},2^\{12\},2^\{13\},2^\{14\},2^\{15\}\\\}\{2−12,2−11\.5,2−11,2−10\.5,2−10,2−9\.5,2−9\}\\\{2^\{\-12\},2^\{\-11\.5\},2^\{\-11\},2^\{\-10\.5\},2^\{\-10\},2^\{\-9\.5\},2^\{\-9\}\\\}3​e​163\\text\{e\}\{16\}\{8,12,16,24\}\\\{8,12,16,24\\\}\{29,210,211,212,213,214,215\}\\\{2^\{9\},2^\{10\},2^\{11\},2^\{12\},2^\{13\},2^\{14\},2^\{15\}\\\}\{2−12,2−11\.5,2−11,2−10\.5,2−10,2−9\.5,2−9\}\\\{2^\{\-12\},2^\{\-11\.5\},2^\{\-11\},2^\{\-10\.5\},2^\{\-10\},2^\{\-9\.5\},2^\{\-9\}\\\}1​e​171\\text\{e\}\{17\}\{12,16,24,32\}\\\{12,16,24,32\\\}\{210,211,212,213,214,215,216\}\\\{2^\{10\},2^\{11\},2^\{12\},2^\{13\},2^\{14\},2^\{15\},2^\{16\}\\\}\{2−12,2−11\.5,2−11,2−10\.5,2−10,2−9\.5,2−9\}\\\{2^\{\-12\},2^\{\-11\.5\},2^\{\-11\},2^\{\-10\.5\},2^\{\-10\},2^\{\-9\.5\},2^\{\-9\}\\\}Table 5:Grid search configurations for fitting hyperparameter scaling laws in Craftax\.FLOPsModel widthsBatch sizeLearning rate1​e​151\\text\{e\}\{15\}\{4,5,6,7\}\\\{4,5,6,7\\\}292^\{9\}2−9\.122^\{\-9\.12\}3​e​153\\text\{e\}\{15\}\{5,6,7,8,9,10\}\\\{5,6,7,8,9,10\\\}2102^\{10\}2−8\.972^\{\-8\.97\}1​e​161\\text\{e\}\{16\}\{6,7,8,9,10,11,12,14\}\\\{6,7,8,9,10,11,12,14\\\}2102^\{10\}2−9\.352^\{\-9\.35\}3​e​163\\text\{e\}\{16\}\{8,10,12,14,16,18,20\}\\\{8,10,12,14,16,18,20\\\}2112^\{11\}2−9\.192^\{\-9\.19\}1​e​171\\text\{e\}\{17\}\{10,12,14,16,18,20,24\}\\\{10,12,14,16,18,20,24\\\}2122^\{12\}2−9\.072^\{\-9\.07\}1​e​181\\text\{e\}\{18\}\{16,20,24,26,28,34,40\}\\\{16,20,24,26,28,34,40\\\}2132^\{13\}2−9\.302^\{\-9\.30\}1​e​191\\text\{e\}\{19\}\{43\}\\\{43\\\}2142^\{14\}2−9\.532^\{\-9\.53\}Table 6:Configurations for the isoFLOPs profile experiment in Starpilot\.FLOPsModel widthsBatch sizeLearning rate1​e​151\\text\{e\}\{15\}\{5,6,7,8,9,10,11,12\}\\\{5,6,7,8,9,10,11,12\\\}2112^\{11\}2−10\.172^\{\-10\.17\}3​e​153\\text\{e\}\{15\}\{6,7,8,9,10,11,12,14,16\}\\\{6,7,8,9,10,11,12,14,16\\\}2112^\{11\}2−10\.512^\{\-10\.51\}1​e​161\\text\{e\}\{16\}\{7,8,9,10,11,12,13,14,16,18,20\}\\\{7,8,9,10,11,12,13,14,16,18,20\\\}2122^\{12\}2−10\.382^\{\-10\.38\}3​e​163\\text\{e\}\{16\}\{10,12,14,16,18,20,24,28\}\\\{10,12,14,16,18,20,24,28\\\}2122^\{12\}2−10\.712^\{\-10\.71\}1​e​171\\text\{e\}\{17\}\{16,18,20,22,24,26,28,30,32,34\}\\\{16,18,20,22,24,26,28,30,32,34\\\}2132^\{13\}2−10\.582^\{\-10\.58\}1​e​181\\text\{e\}\{18\}\{24,28,30,32,34,35,36,40,44,48\}\\\{24,28,30,32,34,35,36,40,44,48\\\}2142^\{14\}2−10\.792^\{\-10\.79\}1​e​191\\text\{e\}\{19\}\{58\}\\\{58\\\}2152^\{15\}2−11\.002^\{\-11\.00\}Table 7:Configurations of the isoFLOPs profile experiment in Craftax\.