How Maximum Entropy makes Reinforcement Learning Robust
Summary
This article explains how incorporating Shannon entropy into reinforcement learning objectives creates more robust agents capable of handling unexpected or adversarial changes in rewards and dynamics.
<p><em>By <a href="https://ashwinreddy.github.io/">Ashwin Reddy</a></em></p><p>This post explains what <strong>Shannon entropy</strong> is and why adding it to the classic reinforcement learning (RL) formulation creates <strong>robust</strong> agents, ones that tolerate unexpected, even adversarial, changes.</p><p>I’m assuming you’re familiar with general ideas in reinforcement learning so we can focus on intuition for why agents with high entropy are robust, in particular to changes in reward or changes in dynamics.</p><h2>A Brief Review of RL</h2><p>Recall that reinforcement learning aims to solve decision-making problems in games where a <strong>reward function</strong> <em>r(s,a)</em> assigns numerical points to action aaa in state sss of the game. Broadly, the goal is to determine how to score the most points (we’ll make this statement more concrete later).</p><p>The agent’s decision-making strategy, also known as its <strong>policy</strong>, denoted <em>π</em>, can be <strong>deterministic</strong> or <strong>stochastic</strong>. When repeatedly confronted with the same situation, a deterministic policy will always choose the same option whereas a stochastic policy selects according to a probability distribution over options.</p><p>For a concrete example of a stochastic policy, think of rock-paper-scissors. You wouldn’t play a deterministic policy against an intelligent opponent, because then they, knowing which option beats yours, could always win. Instead, you would make each option equiprobable, minimizing the opponent’s ability to exploit your strategy.</p><p>In truth, a deterministic policy <em>is</em> a stochastic policy; it takes a single action with probability 1. Thus, RL formalism typically takes the policy as a distribution over actions given the state, <em>π(a|s )</em>.</p><p>Since the policy is randomized, the total reward over the course of an episode of TTT steps is a random variable. To collapse this into a scalar objective, we take the expected value. The ideal policy maximizes this objective.</p><div class="latex-rendered" data-attrs="{"persistentExpression":"\\pi^\\star = \\arg\\max_\\pi \\mathbb{E}_{\\pi}\\left[\\sum_{t=1}^T r(s_t, a_t) \\right]","id":"KIGUFMIQCV"}" data-component-name="LatexBlockToDOM"></div><h2>Enter Entropy</h2><p>We’ll now introduce the second key ingredient.</p><p>In 1948, Claude Shannon, father of information theory, introduced the idea of entropy, a measurement of information for random variables, in his seminal paper “A Mathematical Theory of Communication.”</p><p>For Shannon, an event’s information content relates to how surprising it is. Information inversely correlates with probability of occurence since highly likely events are expected to happen anyway.</p><p>Say you’re playing Hangman. If you guess “e” and the word has an “e,” you didn’t discover much; “e” is the most frequent character in the English language. However, if you find that a “z” is in the word, that event provides lots of information since that drastically narrows the set of words under consideration.</p><p>Shannon’s desiderata for entropy also includes the following: supposing <em>p<sub>1</sub></em> and <em>p<sub>2</sub></em> are probabilities for independent events, the information <em>I</em> received from seeing both events is the information of seeing each event added together.</p><div class="latex-rendered" data-attrs="{"persistentExpression":"I(p_1p_2) = I(p_1) + I(p_2)","id":"LTIWEUNKEZ"}" data-component-name="LatexBlockToDOM"></div><p>See if you can deduce why these criteria imply that the information of an event with probability <em>p</em> can be suitably defined as </p><div class="latex-rendered" data-attrs="{"persistentExpression":"I(p) \\equiv \\log \\frac{1}{p}","id":"TFHLOYZZEP"}" data-component-name="LatexBlockToDOM"></div><p>The Shannon entropy <em>H</em> is compactly defined as the <em>expected</em> information content of a random variable <em>X</em>:</p><div class="latex-rendered" data-attrs="{"persistentExpression":"\\mathcal{H}(X) \\equiv \\mathbb{E}_{x \\sim X}\\left[I\\big(p(x)\\big)\\right]","id":"EHJSRTBYQE"}" data-component-name="LatexBlockToDOM"></div><p>For simplicity, I supress details about <strong>differential entropy</strong>, which defines entropy for continuous random variables. For our purposes, the equation above applies to any random variable.</p><h2>Entropy Intuition</h2><p>A Bernoulli coin flip serves as a prototypical example for understanding entropy. Given <em>X∼Bernoulli(p)</em>,</p><div class="latex-rendered" data-attrs="{"persistentExpression":"\\mathcal{H}(X) = p \\log \\frac{1}{p} + (1-p)\\log\\frac{1}{1-p}","id":"ALJNDGTWKK"}" data-component-name="LatexBlockToDOM"></div><p>Let’s look at a plot. Entropy is measured in bits when the logarithm in <em>I(p)</em> has base 2.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!lDad!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff209d71b-7dd7-41b5-b0bc-0e68070c34b7_3165x1733.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!lDad!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff209d71b-7dd7-41b5-b0bc-0e68070c34b7_3165x1733.jpeg 424w, https://substackcdn.com/image/fetch/$s_!lDad!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff209d71b-7dd7-41b5-b0bc-0e68070c34b7_3165x1733.jpeg 848w, https://substackcdn.com/image/fetch/$s_!lDad!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff209d71b-7dd7-41b5-b0bc-0e68070c34b7_3165x1733.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!lDad!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff209d71b-7dd7-41b5-b0bc-0e68070c34b7_3165x1733.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!lDad!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff209d71b-7dd7-41b5-b0bc-0e68070c34b7_3165x1733.jpeg" width="1456" height="797" data-attrs="{"src":"https://substack-post-media.s3.amazonaws.com/public/images/f209d71b-7dd7-41b5-b0bc-0e68070c34b7_3165x1733.jpeg","srcNoWatermark":null,"fullscreen":null,"imageSize":null,"height":797,"width":1456,"resizeWidth":null,"bytes":null,"alt":null,"title":null,"type":null,"href":null,"belowTheFold":true,"topImage":false,"internalRedirect":null,"isProcessing":false,"align":null,"offset":false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!lDad!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff209d71b-7dd7-41b5-b0bc-0e68070c34b7_3165x1733.jpeg 424w, https://substackcdn.com/image/fetch/$s_!lDad!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff209d71b-7dd7-41b5-b0bc-0e68070c34b7_3165x1733.jpeg 848w, https://substackcdn.com/image/fetch/$s_!lDad!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff209d71b-7dd7-41b5-b0bc-0e68070c34b7_3165x1733.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!lDad!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff209d71b-7dd7-41b5-b0bc-0e68070c34b7_3165x1733.jpeg 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg role="img" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><title></title><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>An equal chance of heads or tails maximizes entropy. However, as the probability of heads or tails becomes more and more certain (reaching 0 or 1), entropy decreases — we’re less surprised about the outcome.</p><p>Extending this logic, we see the uniform distribution has the highest entropy over all discrete random variables. Since no one option is more likely to show up, the outcomes give maximal surprise.</p><p><strong>Maximum entropy distributions</strong> often play an important role in applications with uncertainty. For another example, let’s say you know a distribution has variance given by some constant <em>σ<sup>2</sup></em>. Of all continuous distributions with variance <em>σ<sup>2</sup></em>, the appropriate Gaussian distribution will have the highest entropy.</p><p>In rough terms, the uniform distribution and the Gaussian distribution, which maximize entropy subject to some constraints, minimize the amount of prior information they assume because they allow for the most surprise. As a result, qualitatively, higher entropy distributions tend to have a larger <strong>support</strong> (i.e. larger set of points with non-zero probability).</p><p>Concretely, if someone asked you to guess the probability of a heads for a possibly biased coin flip, it would make more sense to guess 50% rather than 100% or 0%. Or, if you had to guess what a random number generator will spit out, it would not be wise to bet it all on a single number.</p><h2>MaxEnt RL</h2><p>If we admit stochastic policies, reinforcement learning becomes a great application area for information theoretic concepts.</p><p>To that end, let’s think about how the entropy of an agent’s policy influences its actions. Entropy creates surprisal, so an agent with high entropy will actually have multiple strategies at its disposal simply by nature of policy’s randomness. In addition, we could interpret a maximum entropy policy distribution as making fewer assumptions about the world it’s in.</p><p>Thus, we hypothesize that encouraging entropy in our agents has beneficial properties. This formulation is exactly MaxEnt RL. We’ll include an entropy term as a bonus in the optimization.</p><div class="latex-rendered" data-attrs="{"persistentExpression":"\\pi^\\star_{\\mathsf{MaxEnt}} = \\arg\\max_{\\pi} \\left( \\mathbb{E}_{\\pi}\\left[\\sum_{t=1}^T r(s_t, a_t) \\right] \\color{red}{+\\mathcal{H}\\left(\\pi\\big(a \\mid s\\big)\\right)} \\color{black}\\right).","id":"KYIPDXHMPM"}" data-component-name="LatexBlockToDOM"></div><p>What properties does this actually guarantee us?</p><h2><a href="https://arxiv.org/pdf/1910.01913.pdf">If MaxEnt RL is the Answer, what is the Question?</a></h2><p>MaxEnt RL solves a variant of reinforcement learning called the <strong>robust reward problem</strong>. In the robust reward problem, the reward function is subject to change.</p><p>In other words, we have an adversarial player who gets to change how the agent is scored. Thus, we have a two-player zero-sum game; the agent needs to maximize its total reward in the worst case. The changes due to the adversarial player are higlighted in red.</p><div class="latex-rendered" data-attrs="{"persistentExpression":"\\arg\\max_\\pi \\color{red}{\\min_{r' \\in R}} \\color{black}\\mathbb{E}_\\pi \\left[ \\sum_{t=1}^T \\color{red}{r'}\\color{black}(s_t, a_t) \\right].","id":"RVQRPWWOCO"}" data-component-name="LatexBlockToDOM"></div><p>where <em>R</em> is the set of reward functions the adversarial player can choose.</p><h3>Proof</h3><p>Let’s sketch the proof. First, we’ll look at <strong>cross-entropy</strong>, which tells us how much surprise to expect when we <em>assume</em> the events follow distribution <em>q</em> when they in fact come from a distribution <em>p</em>:</p><div class="latex-rendered" data-attrs="{"persistentExpression":"\\mathcal{H}(p, q) = \\mathbb{E}_{x \\sim p}\\left[\\log \\frac{1}{q(x)} \\right].","id":"UNTJIBLDUU"}" data-component-name="LatexBlockToDOM"></div><p>Naturally, the lowest cross-entropy comes when our assumption lines up with reality, i.e. <em>p=q</em>.</p><div class="latex-rendered" data-attrs="{"persistentExpression":"\\mathcal{H}(p) = \\min_{q} \\mathcal{H}(p, q).","id":"ZFONEQTINR"}" data-component-name="LatexBlockToDOM"></div><p>With some manipulation, the entropy term in the maximum entropy RL formulation can then be packaged into the reward function!</p><div class="latex-rendered" data-attrs="{"persistentExpression":"\\overbrace{\\mathbb{E}_{\\pi}\\left[\\sum_{t=1}^T r(s_t, a_t) \\right] {+ \\mathcal{H}\\left(\\pi\\big(a \\mid s\\big)\\right)}}^{\\text{MaxEnt Reward}} = \\overbrace {\\min_{r' \\in R}\\mathbb{E}_{\\pi}\\left[\\sum_{t=1}^T r'(s_t, a_t) \\right]}^{\\text{Robust Reward Problem}}","id":"JCJXIZYULO"}" data-component-name="LatexBlockToDOM"></div><p>where </p><div class="latex-rendered" data-attrs="{"persistentExpression":"R = \\{r(s,a) - \\log q(a \\mid s)\\}","id":"IZOSVBJCDS"}" data-component-name="LatexBlockToDOM"></div><p>for any valid distribution <em>q</em>. In other words, MaxEnt RL ensures the highest reward under perturbations to the original reward function <em>r(s,a)</em>.</p><h3>Example</h3><p>Let’s take the example from the paper to understand what’s at play here. Imagine a simple 2-armed bandit with reward</p><div class="latex-rendered" data-attrs="{"persistentExpression":"r(a) = \\begin{cases}2 & a = 1 \\\\ 1 & a = 2\\end{cases}.","id":"KTQVDWQCQR"}" data-component-name="LatexBlockToDOM"></div><p>The robust set is given by adding in the perturbation from arbitrary distribution qqq.</p><div class="latex-rendered" data-attrs="{"persistentExpression":"R = \\left\\{ r'(a) = \\begin{cases}2 -\\log q(a)& a = 1 \\\\ 1- \\log(1-q(a)) & a = 2\\end{cases} \\middle|\\, q \\in [0,1] \\right\\}.","id":"SNMRJJPHFC"}" data-component-name="LatexBlockToDOM"></div><p>We can then plot the robust set and see the associated rewards.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!NUqZ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F228620ea-276b-494d-80bc-eb196819e880_1794x924.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!NUqZ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F228620ea-276b-494d-80bc-eb196819e880_1794x924.png 424w, https://substackcdn.com/image/fetch/$s_!NUqZ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F228620ea-276b-494d-80bc-eb196819e880_1794x924.png 848w, https://substackcdn.com/image/fetch/$s_!NUqZ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F228620ea-276b-494d-80bc-eb196819e880_1794x924.png 1272w, https://substackcdn.com/image/fetch/$s_!NUqZ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F228620ea-276b-494d-80bc-eb196819e880_1794x924.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!NUqZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F228620ea-276b-494d-80bc-eb196819e880_1794x924.png" width="1456" height="750" data-attrs="{"src":"https://substack-post-media.s3.amazonaws.com/public/images/228620ea-276b-494d-80bc-eb196819e880_1794x924.png","srcNoWatermark":null,"fullscreen":null,"imageSize":null,"height":750,"width":1456,"resizeWidth":null,"bytes":null,"alt":null,"title":null,"type":null,"href":null,"belowTheFold":true,"topImage":false,"internalRedirect":null,"isProcessing":false,"align":null,"offset":false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!NUqZ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F228620ea-276b-494d-80bc-eb196819e880_1794x924.png 424w, https://substackcdn.com/image/fetch/$s_!NUqZ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F228620ea-276b-494d-80bc-eb196819e880_1794x924.png 848w, https://substackcdn.com/image/fetch/$s_!NUqZ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F228620ea-276b-494d-80bc-eb196819e880_1794x924.png 1272w, https://substackcdn.com/image/fetch/$s_!NUqZ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F228620ea-276b-494d-80bc-eb196819e880_1794x924.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg role="img" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><title></title><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h2>Robustness to Dynamics</h2><p>We’ve now seen how MaxEnt RL is robust to variations in the reward. This insight actually enables us to make another claim about robustness: MaxEnt RL is also robust to changes in dynamics!</p><p>We won’t go into the proof (if you’re interested, that is covered in this <a href="https://bair.berkeley.edu/blog/2021/03/09/maxent-robust-rl/">blog post</a>), but the high level idea is to convert variations in dynamics to variations in the reward.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Rm1D!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0f80a13-bf64-4747-a45c-6fd499b464db_870x738.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Rm1D!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0f80a13-bf64-4747-a45c-6fd499b464db_870x738.png 424w, https://substackcdn.com/image/fetch/$s_!Rm1D!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0f80a13-bf64-4747-a45c-6fd499b464db_870x738.png 848w, https://substackcdn.com/image/fetch/$s_!Rm1D!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0f80a13-bf64-4747-a45c-6fd499b464db_870x738.png 1272w, https://substackcdn.com/image/fetch/$s_!Rm1D!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0f80a13-bf64-4747-a45c-6fd499b464db_870x738.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!Rm1D!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0f80a13-bf64-4747-a45c-6fd499b464db_870x738.png" width="870" height="738" data-attrs="{"src":"https://substack-post-media.s3.amazonaws.com/public/images/c0f80a13-bf64-4747-a45c-6fd499b464db_870x738.png","srcNoWatermark":null,"fullscreen":null,"imageSize":null,"height":738,"width":870,"resizeWidth":null,"bytes":null,"alt":null,"title":null,"type":null,"href":null,"belowTheFold":true,"topImage":false,"internalRedirect":null,"isProcessing":false,"align":null,"offset":false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Rm1D!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0f80a13-bf64-4747-a45c-6fd499b464db_870x738.png 424w, https://substackcdn.com/image/fetch/$s_!Rm1D!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0f80a13-bf64-4747-a45c-6fd499b464db_870x738.png 848w, https://substackcdn.com/image/fetch/$s_!Rm1D!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0f80a13-bf64-4747-a45c-6fd499b464db_870x738.png 1272w, https://substackcdn.com/image/fetch/$s_!Rm1D!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0f80a13-bf64-4747-a45c-6fd499b464db_870x738.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image"><svg role="img" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><g><title></title><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h2>Conclusion</h2><p>We’ve built up some intuitive understanding of what entropy is and how it connects to RL agents. This connection combines rich insights from probability, information theory, and artificial intelligence. In particular, we can use these insights to encourage robustness to both rewards and dynamics in our agents. Robustness properties and guarantees may be useful for applications like robotics, where sensor information can be noisy and reward functions are not easy to write down. If you find these ideas interesting, I invite you to read the relevant papers!</p><h2>Read More</h2><ul><li><p><a href="https://arxiv.org/pdf/1910.01913.pdf">If MaxEnt RL is the Answer, what is the Question?</a></p></li><li><p><a href="https://arxiv.org/pdf/2103.06257.pdf">Maximum Entropy RL (Provably) Solves Some Robust RL Problems</a></p></li><li><p><a href="https://dibyaghosh.com/blog/probability/kldivergence.html">KL Divergence for Machine Learning</a></p></li></ul><h2>Image References</h2><ul><li><p><a href="https://bair.berkeley.edu/blog/2021/03/09/maxent-robust-rl/">Maximum Entropy RL (Provably) Solves Some Robust RL Problems</a></p></li></ul>
View Cached Full Text
Cached at: 05/08/26, 08:18 AM
# How Maximum Entropy makes Reinforcement Learning Robust
Source: [https://mlberkeley.substack.com/p/max-ent](https://mlberkeley.substack.com/p/max-ent)
*By[Ashwin Reddy](https://ashwinreddy.github.io/)*
This post explains what**Shannon entropy**is and why adding it to the classic reinforcement learning \(RL\) formulation creates**robust**agents, ones that tolerate unexpected, even adversarial, changes\.
I’m assuming you’re familiar with general ideas in reinforcement learning so we can focus on intuition for why agents with high entropy are robust, in particular to changes in reward or changes in dynamics\.
Recall that reinforcement learning aims to solve decision\-making problems in games where a**reward function***r\(s,a\)*assigns numerical points to action aaa in state sss of the game\. Broadly, the goal is to determine how to score the most points \(we’ll make this statement more concrete later\)\.
The agent’s decision\-making strategy, also known as its**policy**, denoted*π*, can be**deterministic**or**stochastic**\. When repeatedly confronted with the same situation, a deterministic policy will always choose the same option whereas a stochastic policy selects according to a probability distribution over options\.
For a concrete example of a stochastic policy, think of rock\-paper\-scissors\. You wouldn’t play a deterministic policy against an intelligent opponent, because then they, knowing which option beats yours, could always win\. Instead, you would make each option equiprobable, minimizing the opponent’s ability to exploit your strategy\.
In truth, a deterministic policy*is*a stochastic policy; it takes a single action with probability 1\. Thus, RL formalism typically takes the policy as a distribution over actions given the state,*π\(a\|s \)*\.
Since the policy is randomized, the total reward over the course of an episode of TTT steps is a random variable\. To collapse this into a scalar objective, we take the expected value\. The ideal policy maximizes this objective\.
\\\(\\pi^\\star = \\arg\\max\_\\pi \\mathbb\{E\}\_\{\\pi\}\\left\[\\sum\_\{t=1\}^T r\(s\_t, a\_t\) \\right\]\\\)
We’ll now introduce the second key ingredient\.
In 1948, Claude Shannon, father of information theory, introduced the idea of entropy, a measurement of information for random variables, in his seminal paper “A Mathematical Theory of Communication\.”
For Shannon, an event’s information content relates to how surprising it is\. Information inversely correlates with probability of occurence since highly likely events are expected to happen anyway\.
Say you’re playing Hangman\. If you guess “e” and the word has an “e,” you didn’t discover much; “e” is the most frequent character in the English language\. However, if you find that a “z” is in the word, that event provides lots of information since that drastically narrows the set of words under consideration\.
Shannon’s desiderata for entropy also includes the following: supposing*p1*and*p2*are probabilities for independent events, the information*I*received from seeing both events is the information of seeing each event added together\.
\\\(I\(p\_1p\_2\) = I\(p\_1\) \+ I\(p\_2\)\\\)
See if you can deduce why these criteria imply that the information of an event with probability*p*can be suitably defined as
\\\(I\(p\) \\equiv \\log \\frac\{1\}\{p\}\\\)
The Shannon entropy*H*is compactly defined as the*expected*information content of a random variable*X*:
\\\(\\mathcal\{H\}\(X\) \\equiv \\mathbb\{E\}\_\{x \\sim X\}\\left\[I\\big\(p\(x\)\\big\)\\right\]\\\)
For simplicity, I supress details about**differential entropy**, which defines entropy for continuous random variables\. For our purposes, the equation above applies to any random variable\.
A Bernoulli coin flip serves as a prototypical example for understanding entropy\. Given*X∼Bernoulli\(p\)*,
\\\(\\mathcal\{H\}\(X\) = p \\log \\frac\{1\}\{p\} \+ \(1\-p\)\\log\\frac\{1\}\{1\-p\}\\\)
Let’s look at a plot\. Entropy is measured in bits when the logarithm in*I\(p\)*has base 2\.
[](https://substackcdn.com/image/fetch/$s_!lDad!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff209d71b-7dd7-41b5-b0bc-0e68070c34b7_3165x1733.jpeg)
An equal chance of heads or tails maximizes entropy\. However, as the probability of heads or tails becomes more and more certain \(reaching 0 or 1\), entropy decreases — we’re less surprised about the outcome\.
Extending this logic, we see the uniform distribution has the highest entropy over all discrete random variables\. Since no one option is more likely to show up, the outcomes give maximal surprise\.
**Maximum entropy distributions**often play an important role in applications with uncertainty\. For another example, let’s say you know a distribution has variance given by some constant*σ2*\. Of all continuous distributions with variance*σ2*, the appropriate Gaussian distribution will have the highest entropy\.
In rough terms, the uniform distribution and the Gaussian distribution, which maximize entropy subject to some constraints, minimize the amount of prior information they assume because they allow for the most surprise\. As a result, qualitatively, higher entropy distributions tend to have a larger**support**\(i\.e\. larger set of points with non\-zero probability\)\.
Concretely, if someone asked you to guess the probability of a heads for a possibly biased coin flip, it would make more sense to guess 50% rather than 100% or 0%\. Or, if you had to guess what a random number generator will spit out, it would not be wise to bet it all on a single number\.
If we admit stochastic policies, reinforcement learning becomes a great application area for information theoretic concepts\.
To that end, let’s think about how the entropy of an agent’s policy influences its actions\. Entropy creates surprisal, so an agent with high entropy will actually have multiple strategies at its disposal simply by nature of policy’s randomness\. In addition, we could interpret a maximum entropy policy distribution as making fewer assumptions about the world it’s in\.
Thus, we hypothesize that encouraging entropy in our agents has beneficial properties\. This formulation is exactly MaxEnt RL\. We’ll include an entropy term as a bonus in the optimization\.
\\\(\\pi^\\star\_\{\\mathsf\{MaxEnt\}\} = \\arg\\max\_\{\\pi\} \\left\( \\mathbb\{E\}\_\{\\pi\}\\left\[\\sum\_\{t=1\}^T r\(s\_t, a\_t\) \\right\] \\color\{red\}\{\+\\mathcal\{H\}\\left\(\\pi\\big\(a \\mid s\\big\)\\right\)\} \\color\{black\}\\right\)\.\\\)
What properties does this actually guarantee us?
MaxEnt RL solves a variant of reinforcement learning called the**robust reward problem**\. In the robust reward problem, the reward function is subject to change\.
In other words, we have an adversarial player who gets to change how the agent is scored\. Thus, we have a two\-player zero\-sum game; the agent needs to maximize its total reward in the worst case\. The changes due to the adversarial player are higlighted in red\.
\\\(\\arg\\max\_\\pi \\color\{red\}\{\\min\_\{r' \\in R\}\} \\color\{black\}\\mathbb\{E\}\_\\pi \\left\[ \\sum\_\{t=1\}^T \\color\{red\}\{r'\}\\color\{black\}\(s\_t, a\_t\) \\right\]\.\\\)
where*R*is the set of reward functions the adversarial player can choose\.
Let’s sketch the proof\. First, we’ll look at**cross\-entropy**, which tells us how much surprise to expect when we*assume*the events follow distribution*q*when they in fact come from a distribution*p*:
\\\(\\mathcal\{H\}\(p, q\) = \\mathbb\{E\}\_\{x \\sim p\}\\left\[\\log \\frac\{1\}\{q\(x\)\} \\right\]\.\\\)
Naturally, the lowest cross\-entropy comes when our assumption lines up with reality, i\.e\.*p=q*\.
\\\(\\mathcal\{H\}\(p\) = \\min\_\{q\} \\mathcal\{H\}\(p, q\)\.\\\)
With some manipulation, the entropy term in the maximum entropy RL formulation can then be packaged into the reward function\!
\\\(\\overbrace\{\\mathbb\{E\}\_\{\\pi\}\\left\[\\sum\_\{t=1\}^T r\(s\_t, a\_t\) \\right\] \{\+ \\mathcal\{H\}\\left\(\\pi\\big\(a \\mid s\\big\)\\right\)\}\}^\{\\text\{MaxEnt Reward\}\} = \\overbrace \{\\min\_\{r' \\in R\}\\mathbb\{E\}\_\{\\pi\}\\left\[\\sum\_\{t=1\}^T r'\(s\_t, a\_t\) \\right\]\}^\{\\text\{Robust Reward Problem\}\}\\\)
where
\\\(R = \\\{r\(s,a\) \- \\log q\(a \\mid s\)\\\}\\\)
for any valid distribution*q*\. In other words, MaxEnt RL ensures the highest reward under perturbations to the original reward function*r\(s,a\)*\.
Let’s take the example from the paper to understand what’s at play here\. Imagine a simple 2\-armed bandit with reward
\\\(r\(a\) = \\begin\{cases\}2 & a = 1 \\\\ 1 & a = 2\\end\{cases\}\.\\\)
The robust set is given by adding in the perturbation from arbitrary distribution qqq\.
\\\(R = \\left\\\{ r'\(a\) = \\begin\{cases\}2 \-\\log q\(a\)& a = 1 \\\\ 1\- \\log\(1\-q\(a\)\) & a = 2\\end\{cases\} \\middle\|\\, q \\in \[0,1\] \\right\\\}\.\\\)
We can then plot the robust set and see the associated rewards\.
[](https://substackcdn.com/image/fetch/$s_!NUqZ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F228620ea-276b-494d-80bc-eb196819e880_1794x924.png)
We’ve now seen how MaxEnt RL is robust to variations in the reward\. This insight actually enables us to make another claim about robustness: MaxEnt RL is also robust to changes in dynamics\!
We won’t go into the proof \(if you’re interested, that is covered in this[blog post](https://bair.berkeley.edu/blog/2021/03/09/maxent-robust-rl/)\), but the high level idea is to convert variations in dynamics to variations in the reward\.
[](https://substackcdn.com/image/fetch/$s_!Rm1D!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc0f80a13-bf64-4747-a45c-6fd499b464db_870x738.png)
We’ve built up some intuitive understanding of what entropy is and how it connects to RL agents\. This connection combines rich insights from probability, information theory, and artificial intelligence\. In particular, we can use these insights to encourage robustness to both rewards and dynamics in our agents\. Robustness properties and guarantees may be useful for applications like robotics, where sensor information can be noisy and reward functions are not easy to write down\. If you find these ideas interesting, I invite you to read the relevant papers\!
- [If MaxEnt RL is the Answer, what is the Question?](https://arxiv.org/pdf/1910.01913.pdf)
- [Maximum Entropy RL \(Provably\) Solves Some Robust RL Problems](https://arxiv.org/pdf/2103.06257.pdf)
- [KL Divergence for Machine Learning](https://dibyaghosh.com/blog/probability/kldivergence.html)
- [Maximum Entropy RL \(Provably\) Solves Some Robust RL Problems](https://bair.berkeley.edu/blog/2021/03/09/maxent-robust-rl/)
#### Discussion about this post
### Ready for more?
Similar Articles
Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning
This paper proposes Adaptive Entropy Regularization (AER), a framework that dynamically balances exploration and exploitation in LLM reinforcement learning by addressing policy entropy collapse through difficulty-aware coefficient allocation and initial-anchored target entropy. Experiments on mathematical reasoning benchmarks demonstrate consistent improvements in both accuracy and exploration capability.
AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learning
This paper introduces AEM, a supervision-free method for agentic reinforcement learning that adapts entropy dynamics at the response level to improve exploration-exploitation trade-offs. It demonstrates performance gains on benchmarks like ALFWorld and SWE-bench by aligning uncertainty estimation with action granularity.
Robust Shielding for Safe Reinforcement Learning
Introduces a novel shielding framework for robust Markov decision processes (RMDPs) that formally guarantees safety under uncertain transition dynamics, proving soundness and optimality. The approach combines with PAC guarantees for learned models, enabling safe reinforcement learning in unknown environments.
Evolving Robustness--Exploration Trade-off in Online Reinforcement Learning via Quantile Bayesian Risk MDPs
This paper proposes a quantile Bayesian risk-aware MDP framework for online RL that adaptively balances robustness and exploration over time, providing theoretical regret bounds and demonstrating strong empirical performance.
Breaking Entropy Bounds: Accelerating RL Training via MTP with Rejection Sampling
Bebop proposes entropy-aware multi-token prediction with rejection sampling and a novel TV loss to accelerate RL training of LLMs, achieving up to 1.8x speedup. The method addresses the degradation of acceptance rates during RL by optimizing training objectives.