Gauging, Measuring, and Controlling Critic Complexity in Actor-Critic Reinforcement Learning

# Gauging, Measuring, and Controlling Critic Complexity in Actor-Critic Reinforcement Learning
Source: [https://arxiv.org/html/2607.00452](https://arxiv.org/html/2607.00452)
###### Abstract

Actor\-critic methods depend on learned critics, but critic quality is often evaluated only indirectly through return, temporal\-difference error, or value loss\. Critic complexity is introduced as an additional diagnostic and intervention dimension for actor\-critic reinforcement learning\. The analysis uses spectral effective\-rank entropy, a rank\-like summary of the singular\-value distributions of critic weight matrices, to assess critic model complexity\. Across TD3 and PPO experiments, critic complexity is tracked together with return and Monte Carlo value\-estimation bias\. The results show that critic complexity is measurable throughout training and is systematically associated with training behavior, while also making clear that the relationship is heterogeneous across algorithms, tasks, and hyperparameters\. A direct complexity\-control intervention is then evaluated by adding a spectral\-entropy penalty to the critic loss\. This intervention reliably changes the targeted spectral quantity, demonstrating that critic complexity can be controlled rather than only observed\. Return effects are treated as task\-dependent evidence rather than as a general performance claim, because overall complexity\-control results vary\.

## 1Introduction

Actor\-critic reinforcement learning depends on learned critics to estimate values and guide policy improvement\. This makes the critic a central source of both progress and failure\. If the critic overestimates certain states or actions, propagates bootstrapping errors, or learns an unnecessarily irregular value function, the actor may optimize against a distorted objective\.

Much of actor\-critic research is therefore concerned with improving critics\. Double Q\-learning reduces maximization bias by separating action selection from value evaluation\[[1](https://arxiv.org/html/2607.00452#bib.bib1),[2](https://arxiv.org/html/2607.00452#bib.bib2)\], while TD3 extends this idea to continuous control through clipped double critics, delayed policy updates, and target\-policy smoothing\[[4](https://arxiv.org/html/2607.00452#bib.bib4)\]\. PPO provides a useful on\-policy contrast, where the critic is trained under a different update regime\[[3](https://arxiv.org/html/2607.00452#bib.bib3)\]\. These methods improve critic reliability indirectly through targets, architectures, or optimization procedures\.

A complementary route is the critic training process itself: whether critic complexity can be measured, related to performance, and controlled during training\. The guiding hypothesis is Occam\-style: among critics that capture the relevant value structure, a simpler critic may be more reliable, matching broader neural\-network generalization arguments that use norm\-based and spectral quantities as complexity proxies\[[6](https://arxiv.org/html/2607.00452#bib.bib6),[7](https://arxiv.org/html/2607.00452#bib.bib7)\]\. This does not imply that lower complexity is always better\. A critic that is too simple may underfit, while a critic that is too complex may overestimate, become unstable, or represent sharp value artifacts\.

To make this hypothesis testable, critic complexity is measured using spectral effective\-rank entropy\. This metric summarizes how diffusely a critic layer uses its singular directions: high entropy means many directions contribute, while low entropy means the spectrum is concentrated in fewer dominant directions\[[9](https://arxiv.org/html/2607.00452#bib.bib9)\]\. This quantity is then tracked during TD3 and PPO training, compared with return and Monte Carlo estimates of value\-estimation bias, and directly controlled in separate experiments by adding a spectral\-entropy penalty to the critic loss\.

The main conclusion is deliberately narrow\. Critic spectral complexity is measurable and controllable, and it is related to actor\-critic performance in structured but task\-dependent ways\. Spectral\-entropy regularization reliably reduces critic rank\-like complexity and improves TD3/HalfCheetah\-v4 performance in the tested setting, but the return benefit does not transfer cleanly across all tasks\.

Concrete contributions:

- •Defines critic complexity as an explicit evaluation dimension for actor\-critic reinforcement learning\.
- •Uses spectral effective\-rank entropy as a computable rank\-like measure of critic complexity\.
- •Provides observational evidence that critic complexity is systematically related to performance and bias, but not through a simple monotonic rule\.
- •Introduces a spectral\-entropy regularizer for directly controlling critic complexity during critic training\.
- •Shows that the regularizer reliably reduces critic rank\-like complexity and can improve TD3/HalfCheetah\-v4 performance, while cross\-task results remain mixed\.

## 2Related Work

Three strands of prior work are most relevant\. First, value\-estimation bias is a standard concern in reinforcement learning\. Double Q\-learning and Deep Double Q\-learning reduce maximization bias by separating action selection from value evaluation\[[1](https://arxiv.org/html/2607.00452#bib.bib1),[2](https://arxiv.org/html/2607.00452#bib.bib2)\], and TD3 adapts this motivation to continuous control with clipped double critics, delayed policy updates, and target\-policy smoothing\[[4](https://arxiv.org/html/2607.00452#bib.bib4)\]\. PPO is included as an on\-policy contrast\[[3](https://arxiv.org/html/2607.00452#bib.bib3)\], while SAC is relevant mainly as a policy\-entropy baseline: unlike SAC’s maximum\-entropy objective\[[5](https://arxiv.org/html/2607.00452#bib.bib5)\], the intervention regularizes the critic’s weight spectrum rather than the policy\.

Second, neural\-network generalization work has used norm\-based and spectral quantities as complexity proxies\[[6](https://arxiv.org/html/2607.00452#bib.bib6),[7](https://arxiv.org/html/2607.00452#bib.bib7)\]\. Spectral normalization shows that singular\-value control can also be a practical training tool\[[8](https://arxiv.org/html/2607.00452#bib.bib8)\]\. Effective rank is a continuous alternative to exact matrix rank and has been studied as a measure of effective dimensionality\[[9](https://arxiv.org/html/2607.00452#bib.bib9)\]\.

Third, recent work connects complexity control to grokking and reasoning, mostly in transformer or supervised compositional settings\. Liu et al\. relate grokking to weight norm and loss\-landscape mismatch\[[10](https://arxiv.org/html/2607.00452#bib.bib10)\], while Zhang et al\. show that initialization scale and weight decay can steer transformers toward lower\-complexity reasoning\-based solutions\[[11](https://arxiv.org/html/2607.00452#bib.bib11)\]\. Musat gives a complementary theoretical link between weight norm and Kolmogorov complexity for fixed\-precision looped or recursive transformer\-style models\[[12](https://arxiv.org/html/2607.00452#bib.bib12)\]\. These papers motivate the view of complexity as a controllable training variable; the experiments test that idea on neural critics in actor\-critic RL\.

## 3Problem Formulation and Method

The problem is to turn critic complexity from an informal intuition into a quantity that can be computed, monitored, and intervened on during actor\-critic training\. Before complexity can be measured in training, it must be defined in a way that is computable from neural critic parameters and activations\. LetW∈ℝm×nW\\in\\mathbb\{R\}^\{m\\times n\}denote a weight matrix in the critic, and letσ1≥σ2≥⋯≥σr\\sigma\_\{1\}\\geq\\sigma\_\{2\}\\geq\\cdots\\geq\\sigma\_\{r\}be its singular values\.

### 3\.1Spectral Complexity Metric

The spectral complexity metric is rank entropy\. For each critic weight matrix, define the normalized singular\-value distribution

pi=σi∑jσj\.p\_\{i\}=\\frac\{\\sigma\_\{i\}\}\{\\sum\_\{j\}\\sigma\_\{j\}\}\.\(1\)The layer rank entropy is

H​\(W\)=−∑ipi​log⁡pi,H\(W\)=\-\\sum\_\{i\}p\_\{i\}\\log p\_\{i\},\(2\)and the aggregate critic complexity metric is the average of this entropy across critic layers\. High entropy means that many singular directions contribute meaningfully; low entropy means that the matrix is dominated by a smaller number of directions\. Effective\-rank entropy is the primary complexity metric for three reasons\.

1. 1\.It is based on singular values, so it is easy to compute from critic weight matrices during training\.
2. 2\.The entropy scale is more interpretable than raw rank or norm values: researchers can read it as the spread of mass across spectral directions\.
3. 3\.In preliminary experiments, rank entropy was more correlated with task performance than other spectral metrics such as effective rank or stable rank\.

### 3\.2Additional Metrics

The analysis relates critic complexity to three outcome measurements\. Bias is the signed critic error under the current policy\. At a checkpoint, the policy is frozen, Monte Carlo rollouts estimate the return from sampled states, and the critic prediction is compared to that return:

Bias⁡\(s\)=Qθ​\(s,π​\(s\)\)−VMC​\(s\)\.\\operatorname\{Bias\}\(s\)=Q\_\{\\theta\}\(s,\\pi\(s\)\)\-V\_\{\\mathrm\{MC\}\}\(s\)\.\(3\)Positive values indicate overestimation, while negative values indicate underestimation\.Return volatilityis the standard deviation of episodic returns over the final 25 monitor episodes\.Bias volatilityis the standard deviation of the checkpoint critic\-bias estimates across sampled evaluation states\. These three quantities are compared against critic effective\-rank entropy at the run and checkpoint levels\.

### 3\.3Complexity\-Control Intervention

The control stage introduces one targeted intervention: spectral\-entropy regularization of the critic\. The regularizer penalizes entropy in the critic’s singular\-value distribution\. For critic lossℒcritic\\mathcal\{L\}\_\{\\mathrm\{critic\}\}, the modified objective is

ℒtotal=ℒcritic\+λent​∑ℓ∈𝒞H​\(Wℓ\),\\mathcal\{L\}\_\{\\mathrm\{total\}\}=\\mathcal\{L\}\_\{\\mathrm\{critic\}\}\+\\lambda\_\{\\mathrm\{ent\}\}\\sum\_\{\\ell\\in\\mathcal\{C\}\}H\(W\_\{\\ell\}\),\(4\)where𝒞\\mathcal\{C\}is the set of critic layers andλent\\lambda\_\{\\mathrm\{ent\}\}is implemented ascritic\_entropy\_coef\. Penalizing entropy encourages the critic to concentrate its weight spectrum into fewer dominant directions, thereby reducing rank entropy\.

## 4Results

All experiments use PPO or TD3 actor\-critic runs with critic complexity, return, and checkpoint value\-bias measurements logged during training\. TD3 uses separate actor and critic MLPs with two 256\-unit hidden layers, while PPO uses two 64\-unit hidden layers for both the policy and value networks\. Unless stated otherwise, analyses use completed runs only\. Every run was performed on a single H20 GPU\.

### 4\.1Observational Analysis

The observational analysis asks whether critic complexity has a measurable relationship to performance and bias under real training conditions\. It only uses non\-controlled runs, meaning runs with no spectral\-entropy regularizer applied to the critic\. The subset shown in Figure[2](https://arxiv.org/html/2607.00452#S4.F2)contains 360 PPO and TD3 runs on Pendulum\-v1 and HalfCheetah\-v4\. These runs vary seed \(\{0,1,2,3,4\}\\\{0,1,2,3,4\\\}\), initialization scale \(\{0\.1,1,10\}\\\{0\.1,1,10\\\}\), critic weight decay \(\{0,10−4,10−2\}\\\{0,10^\{\-4\},10^\{\-2\}\\\}\), and critic learning rate \(\{10−4,3⋅10−4\}\\\{10^\{\-4\},3\\cdot 10^\{\-4\}\\\}\)\. Figure[1](https://arxiv.org/html/2607.00452#S4.F1)first checks whether critic effective\-rank entropy changes during training\. Figure[2](https://arxiv.org/html/2607.00452#S4.F2)then shows the run\-level relationship between final complexity and final return\. Together, the plots show that complexity and return are related in a structured but not purely monotonic way: low rank entropy is not automatically better, and high rank entropy is not automatically worse\. Still, the highest\-return runs have lower effective\-rank entropy, suggesting that complexity is a meaningful signal for performance even if it is not a simple monotonic one\.

![Refer to caption](https://arxiv.org/html/2607.00452v1/x1.png)Figure 1:Evolution of critic effective\-rank entropy over normalized training progress for non\-controlled Pendulum\-v1 and HalfCheetah\-v4 runs\. Each subplot summarizes one algorithm/task slice; the line shows the checkpoint median and the shaded band shows the interquartile range\. The plot shows that critic effective\-rank entropy generally decreases slightly during training, although with high variability\.Critic complexity is a dynamic training quantity, not only a final\-run summary\.![Refer to caption](https://arxiv.org/html/2607.00452v1/x2.png)Figure 2:Run\-level relationship between final return and critic effective\-rank entropy for non\-controlled Pendulum\-v1 and HalfCheetah\-v4 runs\. Ant\-v4 and Walker2d\-v4 are omitted to keep the diagnostic focused on the main baseline environments\.The best observed returns tend to occur at lower critic entropy, but the relationship depends on algorithm and task rather than following a universal monotonic rule\.Table 1:Run\-level Spearman correlations for the non\-controlled observational analysis\. The x\-axis variable is final critic effective\-rank entropy; each row is one algorithm/task slice\. Each row summarizesn=90n=90non\-controlled runs from the observational sweep\.Entropy is most strongly associated with bias on Pendulum and with final return for TD3, showing that the signal is structured but task\- and algorithm\-dependent\.Table[1](https://arxiv.org/html/2607.00452#S4.T1)gives a compact summary of the same pattern\. The strongest associations are task\- and algorithm\-dependent: Pendulum shows a strong negative relationship between effective\-rank entropy and signed critic bias, while TD3 shows strong negative relationships between entropy and final return on both baseline tasks\. These results should be read as descriptive rather than causal because the relationships are heterogeneous across tasks and confounded by hyperparameters\. They motivate the intervention below: if complexity is more than a passive correlate, directly changing it should alter training dynamics\.

### 4\.2Complexity Control Intervention

The main intervention subset contains 32 balanced runs: PPO and TD3 on Pendulum\-v1 and HalfCheetah\-v4, using seeds 0 and 1 and entropy coefficients\{0,0\.001,0\.003,0\.01\}\\\{0,0\.001,0\.003,0\.01\\\}\. Table[2](https://arxiv.org/html/2607.00452#S4.T2)reports the results\. Each row averages the same two seeds, 0 and 1, after de\-duplicating repeated completed runs\. Bold entries mark the best coefficient within each algorithm/task group for that metric: higher final return, lower volatility, smaller absolute final bias, and lower rank entropy\.

Table 2:Main\-task regularizer summary across PPO and TD3 on Pendulum\-v1 and HalfCheetah\-v4\. All rows use seeds 0 and 1; values are mean±\\pmSEM\. Bold marks the best value within each algorithm/task block across entropy coefficients\.Entropy regularization reliably lowers TD3 critic entropy, while the clearest performance gain appears for TD3/HalfCheetah\-v4 at a moderate coefficient\.Across the four main slices, the regularizer most clearly moves the targeted metric for TD3: rank entropy decreases on Pendulum\-v1 and HalfCheetah\-v4 as the coefficient increases\. The performance effect is not uniform\. The strongest return improvement appears in Table[2](https://arxiv.org/html/2607.00452#S4.T2): under the balanced seed\-0/1 comparison, TD3/HalfCheetah\-v4 has the best final return at coefficient0\.0010\.001, while its lowest rank entropy occurs at coefficient0\.010\.01\. The PPO rows show much smaller rank\-entropy changes and no comparable return effect\.

The bias result is more subtle\. Final signed bias does not improve reliably across all four slices\. The HalfCheetah TD3 volatility measures are more directionally consistent: final return volatility and final bias volatility both decrease under the best regularized setting\. Therefore, the data do not support the simple story that the regularizer improves return by reducing final bias\. Instead, the more defensible interpretation is that the regularizer changes critic complexity and stability dynamics, and that this change is beneficial most clearly for TD3/HalfCheetah\-v4\.

Cross\-task generalization\.The cross\-task experiment testscritic\_entropy\_coef=0\.01\\texttt\{critic\\\_entropy\\\_coef\}=0\.01on two additional TD3 tasks: Walker2d\-v4 and Ant\-v4\. To match Table[2](https://arxiv.org/html/2607.00452#S4.T2), each condition uses seeds 0 and 1, with the same non\-entropy hyperparameters as the main intervention subset\.

Table 3:Cross\-task regularizer summary\. All rows use seeds 0 and 1; values are mean±\\pmSEM\. Bold marks the best value within each algorithm/task block across entropy coefficients, using the same columns and highlighting convention as Table[2](https://arxiv.org/html/2607.00452#S4.T2)\.The regularizer still reduces critic entropy on Walker2d\-v4 and Ant\-v4, but this control effect does not translate into a consistent return improvement\.Table[3](https://arxiv.org/html/2607.00452#S4.T3)shows that on Walker2d\-v4, rank entropy drops from about2\.822\.82to2\.572\.57at coefficient0\.010\.01, but return and both volatility measures do not improve under the seed\-0/1 comparison\. Signed final bias becomes slightly less negative, but the uncertainty is large\. On Ant\-v4, the rank\-entropy reduction is smaller but still visible; final return is essentially unchanged, return volatility is slightly worse, and final bias volatility decreases\. The cross\-task result therefore separates two claims\. Spectral\-entropy regularization is a robust control knob for critic complexity\. It is not, at the tested coefficient, a task\-general return\-improvement method\.

## 5Conclusion

Critic complexity in actor\-critic reinforcement learning is studied through a three\-part arc: gauging, measuring, and controlling\. The gauging stage defined computable complexity metrics based on singular\-value spectra and local input sensitivity\. The measuring stage built a pipeline that tracks these metrics during full RL training and relates them to return and Monte Carlo bias estimates\. The controlling stage introduced a targeted spectral\-entropy regularizer for the critic\.

The central finding is that critic spectral complexity is actionable\. Rank entropy can be measured over time, it shows structured training dynamics, and it can be directly reduced by a regularizer\. On TD3/HalfCheetah\-v4, this reduction coincides with a large return improvement\. On Walker2d\-v4 and Ant\-v4, complexity reduction still occurs, but the return benefit does not transfer cleanly\.

The main limitation is scope\. The experiments do not establish that the effect is algorithm\-general, because a broader cross\-algorithm campaign was not run\. They also do not establish full hyperparameter robustness, because the robustness sweep over initialization scale and weight decay was not run\. The final claim is therefore precise: spectral entropy regularization is an effective way to control critic rank\-entropy dynamics, and those dynamics reveal training behavior that is not visible from return curves alone\.

## References

- \[1\]Hado van Hasselt\.Double Q\-learning\.In*Advances in Neural Information Processing Systems*, 2010\.
- \[2\]Hado van Hasselt, Arthur Guez, and David Silver\.Deep Reinforcement Learning with Double Q\-learning\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, 2016\.
- \[3\]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov\.Proximal Policy Optimization Algorithms\.*arXiv preprint arXiv:1707\.06347*, 2017\.
- \[4\]Scott Fujimoto, Herke van Hoof, and David Meger\.Addressing Function Approximation Error in Actor\-Critic Methods\.In*Proceedings of the International Conference on Machine Learning*, 2018\.
- \[5\]Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine\.Soft Actor\-Critic: Off\-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor\.In*Proceedings of the International Conference on Machine Learning*, 2018\.
- \[6\]Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals\.Understanding Deep Learning Requires Rethinking Generalization\.*arXiv preprint arXiv:1611\.03530*, 2016\.
- \[7\]Peter L\. Bartlett, Dylan J\. Foster, and Matus J\. Telgarsky\.Spectrally\-normalized margin bounds for neural networks\.In*Advances in Neural Information Processing Systems*, 2017\.
- \[8\]Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida\.Spectral Normalization for Generative Adversarial Networks\.In*International Conference on Learning Representations*, 2018\.
- \[9\]Olivier Roy and Martin Vetterli\.The Effective Rank: A Measure of Effective Dimensionality\.In*15th European Signal Processing Conference*, pages 606–610, 2007\.
- \[10\]Ziming Liu, Eric J\. Michaud, and Max Tegmark\.Omnigrok: Grokking Beyond Algorithmic Data\.In*International Conference on Learning Representations*, 2023\.
- \[11\]Zhongwang Zhang, Pengxiao Lin, Zhiwei Wang, Yaoyu Zhang, and Zhi\-Qin John Xu\.Complexity Control Facilitates Reasoning\-Based Compositional Generalization in Transformers\.*arXiv preprint arXiv:2501\.08537*, 2025\.
- \[12\]Tiberiu Musat\.Neural Weight Norm = Kolmogorov Complexity\.*arXiv preprint arXiv:2605\.10878*, 2026\.