Adaptive Probabilistic Shielding by Learning MDPs for Safe Reinforcement Learning

# Adaptive Probabilistic Shielding by Learning MDPs for Safe Reinforcement Learning
Source: [https://arxiv.org/html/2608.19836](https://arxiv.org/html/2608.19836)
Astrid Horn BrorholtOrcID:[0009\-0007\-1824\-0554](https://orcid.org/0009-0007-1824-0554)Affiliation:Aalborg University, Aalborg, DenmarkE\-mail[\{asgerhb,kgl,christianms\}@cs\.aau\.dk](mailto:{asgerhb,kgl,christianms}@cs.aau.dk)Maris F\. L\. GaleslootOrcID:[0009\-0002\-5112\-8584](https://orcid.org/0009-0002-5112-8584)Affiliation:Radboud University, Nijmegen, NetherlandsE\-mail[maris\.galesloot@ru\.nl](mailto:[email protected])Nils JansenOrcID:[0000\-0003\-1318\-8973](https://orcid.org/0000-0003-1318-8973)Affiliation:Radboud University, Nijmegen, NetherlandsE\-mail[maris\.galesloot@ru\.nl](mailto:[email protected])Affiliation:Ruhr University Bochum, Bochum, GermanyE\-mail[n\.jansen@rub\.de](mailto:[email protected])Kim Guldstrand LarsenOrcID:[0000\-0002\-5953\-3384](https://orcid.org/0000-0002-5953-3384)Affiliation:Aalborg University, Aalborg, DenmarkE\-mail[\{asgerhb,kgl,christianms\}@cs\.aau\.dk](mailto:{asgerhb,kgl,christianms}@cs.aau.dk)Christian SchillingOrcID:[0000\-0003\-3658\-1065](https://orcid.org/0000-0003-3658-1065)Affiliation:Aalborg University, Aalborg, DenmarkE\-mail[\{asgerhb,kgl,christianms\}@cs\.aau\.dk](mailto:{asgerhb,kgl,christianms}@cs.aau.dk)

###### Abstract

Probabilistic shielding is a technique for safe reinforcement learning \(RL\)\. Typically, a static observer—called the shield—constrains the learning agent’s actions to those for which acting safely remains feasible\. Traditionally, the shield is computed from the transition probabilities of the underlying Markov decision process \(MDP\)\. Thus, this technique is not applicable when the MDP model is not given a priori, which, unfortunately, is the case in typical RL applications\. In this paper, we study the problem of computing a shield in the setting where the transition graph of the MDP is known, but the transition probabilities are unknown\. Our approach integrates probabilistic shielding with online model learning: as the RL agent explores the environment, we estimate the transition probabilities\. From this estimate, we compute a shield\. While the shield may be conservative initially, it adapts as the model estimate becomes more precise\. Thus, the shield improves in tandem with the RL agent\. This paradigm of*adaptive probabilistic shielding*raises a number of challenges, such as when to recompute the shield and how to balance between exploration and safety during learning\. We empirically evaluate multiple variants of this paradigm across several environments\.

###### Keywords:

Safe reinforcement learning Shielding Model learning Interval Markov decision process\.

## 1Introduction

Markov decision processes \(MDPs\)\[[32](https://arxiv.org/html/2608.19836#bib.bib20)\]are the standard models to capture*decision\-making under uncertainty*in artificial intelligence \(AI\)\[[24](https://arxiv.org/html/2608.19836#bib.bib19)\]\. Factors such as unknown or unpredictable environments, contextual changes at runtime, or incomplete data are commonly referred to as*uncertainty*\. Specifically, MDPs capture settings where agents, in each*state*of their environment, choose to execute*actions*upon which the environment*probabilistically transitions*to a new state\. Upon that transition, the agent receives a*reward*\.

Common objectives for MDPs are to \(1\) maximize the expected cumulative reward and \(2\) adhere to safety constraints specified as temporal logic constraints\[[30](https://arxiv.org/html/2608.19836#bib.bib18)\]\. In the past, the first objective was mostly considered by the AI community, and the latter objective by the formal verification community\. Specifically, reinforcement learning \(RL\) is a major AI technique for decision\-making under uncertainty\[[38](https://arxiv.org/html/2608.19836#bib.bib7)\]\. For an unknown MDP, an RL agent aims to maximize the expected reward by collecting data through exploration of the environment across multiple episodes\. A major limitation in RL is that during exploration, the agent will necessarily execute potentially devastatingly unsafe actions\. In contrast, probabilistic model checking \(PMC\) is a formal verification technique that computes the probability of satisfying a safety constraint in an MDP\[[3](https://arxiv.org/html/2608.19836#bib.bib21)\]\. The key limitation of PMC is that the MDP must be fully specified\.

##### Shielded RL\.

In response to these key limitations, a tremendous body of work has brought together RL and formal methods in the area of safe RL\[[13](https://arxiv.org/html/2608.19836#bib.bib28)\], in particular within shielded RL\[[1](https://arxiv.org/html/2608.19836#bib.bib25),[9](https://arxiv.org/html/2608.19836#bib.bib5)\]\. Specifically, in*probabilistic pre\-shielding*, PMC is used to compute a shield that blocks potentially unsafe actions at runtime\[[25](https://arxiv.org/html/2608.19836#bib.bib22),[21](https://arxiv.org/html/2608.19836#bib.bib24),[17](https://arxiv.org/html/2608.19836#bib.bib3)\]\. Such*runtime verification approach*renders RL \(more\) safe during exploration, yet inherits PMC’s strong assumption that the \(safety\-relevant\) environment model, that is, the MDP, must be fully specified\. One may be tempted to approach this problem by first gathering sufficient training data from RL, then using that data to learn a full MDP model of the environment, and finally computing a shield from that model\. However, safety during the data collection is not considered, making such an approach hardly applicable in real\-world scenarios\.

##### Problem setting\.

In this paper, we overcome the aforementioned key real\-world limitation of shielded RL and propose a practical and adaptive approach\. To that end, we impose mild assumptions about the environment in which the RL agent operates\. First, we assume that simulation access to the true environment MDP is available from the initial state, which is a common assumption in RL\. Second, we assume that the topology, that is, the underlying graph of the MDP, is known, which is much more realistic than knowing the exact transition probabilities\.

![Refer to caption](https://arxiv.org/html/2608.19836v1/Adaptive-shielding-loop.drawio.png)Figure 1:A high\-level overview of our adaptive probabilistic shielding approach\.
##### Our approach: Safe RL via Adaptive Probabilistic Shielding\.

[Fig\.1](https://arxiv.org/html/2608.19836#S1.F1)shows an overview of our approach\. As usual, the RL agent executes an action in the \(unknown\) MDP, yielding a reward and causing the environment to transition to a new state\. A key component of our approach is a*model estimator*, based on approaches from\[[37](https://arxiv.org/html/2608.19836#bib.bib14)\]\. While the agent interacts with the environment, it collects data on the observed states and actions\. From this data, the model estimator then constructs a*learned*MDP model of the environment\. We use the state\-of\-the\-art PMC tool PRISM\[[26](https://arxiv.org/html/2608.19836#bib.bib12)\]to compute a shield based on this learned MDP\[[12](https://arxiv.org/html/2608.19836#bib.bib17)\]\. At any point during this process, the shield can be updated*adaptively*, and additional data can be collected under the updated shield\.

##### Interval MDPs and shields\.

An essential part of our approach is to implement and compare estimators that yield different types of MDP models from moderate amounts of data collected as a byproduct of the RL process itself\. In particular, following\[[37](https://arxiv.org/html/2608.19836#bib.bib14),[12](https://arxiv.org/html/2608.19836#bib.bib17)\], we create so\-called interval MDPs \(iMDPs\)\[[29](https://arxiv.org/html/2608.19836#bib.bib15),[36](https://arxiv.org/html/2608.19836#bib.bib30)\]\. Intuitively, iMDPs capture data uncertainty robustly by defining upper and lower bounds on transition probabilities, based on, for instance, confidence intervals around point estimates of those probabilities\. Then, PMC can provide upper and lower bounds on the safety probabilities for iMDPs\. In\[[12](https://arxiv.org/html/2608.19836#bib.bib17)\], the worst\-case estimates of the bounds are used to compute a*robust*, conservative shield\. Consider the case where a particular action imposes a lower bound of10%10\\%and an upper bound of30%30\\%on the probability of reaching a safety\-critical unsafe state\. The application at hand may allow reaching such a state with a maximum probability of20%20\\%\. A shield with a*robust*uncertainty interpretation would block that action\.

##### Safety vs\. exploration\.

The key strength of our adaptive shielding approach is that gathering more data yields more accurate model estimates, which reduce the size of the probability intervals and allow for less conservative shields\. The challenge, however, is that a too\-conservative shield may, in the extreme, impede any exploration of the environment and thereby prevent the agent from gathering the necessary data to refine the shield and learn a good policy\. One solution to this problem is to use an*optimistic*interpretation of uncertainty, as is common in robust RL and referred to as optimism in the face of uncertainty\[[28](https://arxiv.org/html/2608.19836#bib.bib29)\]\. In the example above, the \(optimistic\) shield would then use the lower probability bound of10%10\\%and allow the critical action\. Another approach is to use common RL exploration techniques to address the exploration\-exploitation dilemma\[[38](https://arxiv.org/html/2608.19836#bib.bib7)\]\.

##### Contributions and research questions\.

The main contribution of this paper is a novel, adaptive shielding algorithm that accounts for the uncertainty in estimating an MDP from data\. We provide a thorough experimental evaluation structured around several concrete research questions\. First, we evaluate if an adaptive shielding approach is beneficial to \(1\) obtain a safe and reward\-optimal policy after training and \(2\) remain safe during training\. Then, we investigate whether the choice of model estimator affects the performance of the shield and how far the quality of the MDP estimate improves over time with respect to the \(conservativeness\) of the shield\. Finally, we take into account various practical considerations, such as the number of model updates in relation to the number of RL episodes\. The paper is structured as follows\. In the remainder of the introduction, we discuss related work\. In[Section2](https://arxiv.org/html/2608.19836#S2), we provide necessary background, followed by the definitions of shields for \(interval\) MDPs and model estimation in[Section3](https://arxiv.org/html/2608.19836#S3)\.[Section4](https://arxiv.org/html/2608.19836#S4)describes our adaptive probabilistic shielding algorithm\. Finally, we present our research questions and experimental analysis in[Section5](https://arxiv.org/html/2608.19836#S5)\.

### 1\.1Related work

##### Probabilistic shielding\.

Classic shields provide unconditional safety guarantees, but this is often too conservative\[[17](https://arxiv.org/html/2608.19836#bib.bib3),[25](https://arxiv.org/html/2608.19836#bib.bib22)\]\. Shielding has been extended to many classes of models\[[10](https://arxiv.org/html/2608.19836#bib.bib36),[5](https://arxiv.org/html/2608.19836#bib.bib35),[8](https://arxiv.org/html/2608.19836#bib.bib34),[23](https://arxiv.org/html/2608.19836#bib.bib37),[1](https://arxiv.org/html/2608.19836#bib.bib25),[33](https://arxiv.org/html/2608.19836#bib.bib40),[19](https://arxiv.org/html/2608.19836#bib.bib39),[6](https://arxiv.org/html/2608.19836#bib.bib41)\]\. We consider probabilistic shields that permit a level of risk of reaching unsafe states\[[16](https://arxiv.org/html/2608.19836#bib.bib27),[21](https://arxiv.org/html/2608.19836#bib.bib24),[17](https://arxiv.org/html/2608.19836#bib.bib3),[25](https://arxiv.org/html/2608.19836#bib.bib22)\]\. In this paper, we focus on practical shields that guarantee the*admissibility*of probabilistic safety guarantees\[[21](https://arxiv.org/html/2608.19836#bib.bib24),[31](https://arxiv.org/html/2608.19836#bib.bib26),[12](https://arxiv.org/html/2608.19836#bib.bib17),[25](https://arxiv.org/html/2608.19836#bib.bib22)\]\. Specifically, our shielding method is based on\[[12](https://arxiv.org/html/2608.19836#bib.bib17)\]\.

##### Adaptive shielding\.

Several works have considered shields that change over time\. Pranger et al\. adaptively construct a finite\-state environment abstraction from past observations to maintain a shield\[[31](https://arxiv.org/html/2608.19836#bib.bib26)\]\. Similarly, Tappler et al\. present an iterative approach using automata learning\[[39](https://arxiv.org/html/2608.19836#bib.bib6)\]\. In other work, the shield adapts to changes in the environment, assuming white\-box access to the \(parametric\) dynamics\[[34](https://arxiv.org/html/2608.19836#bib.bib33),[11](https://arxiv.org/html/2608.19836#bib.bib31)\]\. Goodall et al\. estimate probabilistic safety by simulating future trajectories in a learned latent model\[[14](https://arxiv.org/html/2608.19836#bib.bib2)\]\. Bethell et al\. learn a shield using an auto\-encoder and adjust the safety threshold in a subsequent RL phase\[[4](https://arxiv.org/html/2608.19836#bib.bib32)\]\.

##### Shielding based on model estimates\.

Galesloot et al\. estimate iMDPs for shielding in*offline*RL\[[12](https://arxiv.org/html/2608.19836#bib.bib17)\]\. Suilen et al\. obtain iMDP estimates via optimistic exploration and then analytically compute a robust policy\[[37](https://arxiv.org/html/2608.19836#bib.bib14)\]\. Another recent work collects environment data offline and then computes a robust shield for shielded RL, with the main contribution being a new algorithm to compute the shield\[[15](https://arxiv.org/html/2608.19836#bib.bib23)\]\.

##### Delimitation\.

The main differences of our problem setting are as follows\. We assume a static environment with a known, finite state space and transition structure, but with unknown transition probabilities\. We further assume black\-box access instead of a settable simulator, which is why we specifically care about safety during the whole process, including data collection and exploration\. Our method differs from previous work in that we integrate model estimation and updates of both the shield and the policy into a single integrated*online*RL procedure with adaptive probabilistic shields\.

## 2Preliminaries

##### Probability distributions\.

Given a setXX, a probability distributionp:X→\[0,1\]p\\colon X\\rightarrow\[0,1\]satisfies∑x∈Xp⁡\(x\)=1\\sum\_\{x\\in X\}p\(x\)=1\. LetΔ⁡\(X\)\\Delta\(X\)denote the set of probability distributions overXXand letUnifX∈Δ⁡\(X\)\\mathrm\{Unif\}\_\{X\}\\in\\Delta\(X\)denote the uniform distribution overXX\.

##### Intervals\.

A closed interval\[a,b\]⊆ℝ\[a,b\]\\subseteq\\mathbb\{R\}fora≤ba\\leq bdescribes the set\{x∈ℝ∣a≤x≤b\}\\\{x\\in\\mathbb\{R\}\\mid a\\leq x\\leq b\\\}\. An open interval\(a,b\)\(a,b\)describes\{x∈ℝ∣a<x<b\}\\\{x\\in\\mathbb\{R\}\\mid a<x<b\\\}\. Half\-open intervals are defined analogously\. Let𝕀=\{\[a,b\]∣0<a≤b≤1\}\\mathbb\{I\}=\\\{\[a,b\]\\mid 0<a\\leq b\\leq 1\\\}denote the set of uncertain nonzero probabilities\.

##### Markov decision processes\.

A*Markov decision process*\(MDP\) is a tupleM=\(S,A,s0,T\)M=\(S,A,s\_\{0\},T\)whereSSis the finite set of states,AAis the finite set of actions,s0∈Ss\_\{0\}\\in Sis the initial state, andT:S×A×S→\[0,1\]T\\colon S\\times A\\times S\\to\[0,1\]is the probabilistic transition function satisfying∑s′∈ST⁡\(s,a,s′\)=1\\sum\_\{s^\{\\prime\}\\in S\}T\(s,a,s^\{\\prime\}\)=1for allssandaa\. A*run*is an alternating sequences0​a0​s1​a1​…s\_\{0\}a\_\{0\}s\_\{1\}a\_\{1\}\\dotsof states and actions such thatT⁡\(si,ai,si\+1\)\>0T\(s\_\{i\},a\_\{i\},s\_\{i\+1\}\)\>0for allii\.

We consider two types of policies\. A*deterministic policy*π:S→A\\pi\\colon S\\to Amaps each state to an action\. A*nondeterministic policy*πN:S→𝟤A\\pi\_\{N\}\\colon S\\to\\mathsf\{2\}^\{A\}maps each state to a set of actions\. A runs0​a0​s1​a1​…s\_\{0\}a\_\{0\}s\_\{1\}a\_\{1\}\\dotsis an*outcome*of a deterministic policyπ\\pi\(resp\. nondeterministic policyπN\\pi\_\{N\}\) ifai=π⁡\(si\)a\_\{i\}=\\pi\(s\_\{i\}\)\(resp\.ai∈πN​\(si\)a\_\{i\}\\in\\pi\_\{N\}\(s\_\{i\}\)\) for allii\.

An*unknown*MDP \(uMDP\) is a tupleMU=\(S,A,s0,TU\)M\_\{U\}=\(S,A,s\_\{0\},T\_\{U\}\)whereSS,AA, ands0s\_\{0\}are defined as for MDPs andTU:S×A→𝟤ST\_\{U\}\\colon S\\times A\\to\\mathsf\{2\}^\{S\}is a nondeterministic transition function \(i\.e\., uMDPs are ordinary transition systems\)\. Each MDP induces an unknown MDP by removing impossible transitions and dropping the probabilities; formally:TU​\(s,a\)=\{s′∈S∣T⁡\(s,a,s′\)\>0\}T\_\{U\}\(s,a\)=\\\{s^\{\\prime\}\\in S\\mid T\(s,a,s^\{\\prime\}\)\>0\\\}\. An*interval*MDP \(iMDP\)\[[22](https://arxiv.org/html/2608.19836#bib.bib1),[29](https://arxiv.org/html/2608.19836#bib.bib15),[35](https://arxiv.org/html/2608.19836#bib.bib11),[20](https://arxiv.org/html/2608.19836#bib.bib4),[37](https://arxiv.org/html/2608.19836#bib.bib14),[36](https://arxiv.org/html/2608.19836#bib.bib30)\]is a tupleMI=\(S,A,s0,TI\)M\_\{I\}=\(S,A,s\_\{0\},T\_\{I\}\)where againSS,AA, ands0s\_\{0\}are defined as for MDPs andTI:S×A×S→𝕀∪\{0\}T\_\{I\}\\colon S\\times A\\times S\\to\\mathbb\{I\}\\cup\\\{0\\\}is an*interval transition function*\. Consider an MDPM=\(S,A,s0,T\)M=\(S,A,s\_\{0\},T\)and an iMDPMI=\(S,A,s0,TI\)M\_\{I\}=\(S,A,s\_\{0\},T\_\{I\}\)overSSandAA\. We say thatTIT\_\{I\}*abstracts*TT, writtenT∈TIT\\in T\_\{I\}, ifTTis a probabilistic transition function and each interval inTIT\_\{I\}contains the corresponding probability inTT; formally:∀s∈S​∀a∈A:∑s′∈ST⁡\(s,a,s′\)=1∧∀s′∈S:T⁡\(s,a,s′\)∈TI​\(s,a,s′\)\\forall s\\in S~\\forall a\\in A\\colon\\sum\_\{s^\{\\prime\}\\in S\}T\(s,a,s^\{\\prime\}\)=1\\land\\forall s^\{\\prime\}\\in S\\colon T\(s,a,s^\{\\prime\}\)\\in T\_\{I\}\(s,a,s^\{\\prime\}\)\. Analogously, we say thatMIM\_\{I\}abstractsMM, writtenM∈MIM\\in M\_\{I\}\. Note that, because intervals in𝕀\\mathbb\{I\}must not include00, ifT∈TIT\\in T\_\{I\}andT′∈TIT^\{\\prime\}\\in T\_\{I\}, then the same transitions inTTandT′T^\{\\prime\}have a non\-zero probability; formally:∀T,T′∈TI​∀s,s′∈S​∀a∈A:T⁡\(s,a,s′\)\>0⟹T′​\(s,a,s′\)\>0\\forall T,T^\{\\prime\}\\in T\_\{I\}~\\forall s,s^\{\\prime\}\\in S~\\forall a\\in A\\colon T\(s,a,s^\{\\prime\}\)\>0\\implies T^\{\\prime\}\(s,a,s^\{\\prime\}\)\>0\.

##### Reinforcement learning\.

We assume that the reader is familiar with reinforcement learning \(RL\)\[[38](https://arxiv.org/html/2608.19836#bib.bib7)\]and only recall some basic commonalities\. LetR:S×A×S→ℝR\\colon S\\times A\\times S\\to\\mathbb\{R\}be the reward function andγ∈\[0,1\)\\gamma\\in\[0,1\)a discount factor\. Given a deterministic policyπ\\pi, the*expected cumulative discounted reward*from the initial states0s\_\{0\}isVπ​\(s0\)=𝔼sπ​\[∑t=0∞γt​R​\(st,at,st\+1\)\],V^\{\\pi\}\(s\_\{0\}\)=\\mathbb\{E\}^\{\\pi\}\_\{s\}\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}R\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\)\\right\],where at each stept∈ℕt\\in\\mathbb\{N\}, the action isat=π⁡\(st\)a\_\{t\}=\\pi\(s\_\{t\}\)and the expectation is taken over the probabilistic state transitions governed by the transition functionTT\. In RL, an agent explores an environment \(which is assumed to be an MDP\) with the aim to learn a policy that maximizes the expected cumulative discounted reward from the exploration experience\. This “training” takes place in episodes of multiple steps each\. RL requires only black\-box sampling access to the environment MDP from an initial state\. Two major paradigms are*model\-free*and*model\-based*RL\[[38](https://arxiv.org/html/2608.19836#bib.bib7)\]\. The latter learns an approximation of the MDP as the agent explores\. On the one hand, the algorithm proposed in this paper learns such an approximation to construct the shield, and is therefore model\-based\. On the other hand, the algorithm also includes an RL component, for which our prototype implementation uses Q\-learning \(which is model\-free\)\. We note that most other RL algorithms \(model\-free or model\-based\) could also be used in place of Q\-learning\.

During training, we use an*ε\\varepsilon\-greedy exploration/exploitation*strategy\[[38](https://arxiv.org/html/2608.19836#bib.bib7)\]: at each step, the agent chooses a random action with probabilityε\\varepsilon\(“exploration”\) and follows the \(partially\) learned policy with probability1−ε1\-\\varepsilon\(“exploitation”\)\.

##### Safety and shielding\.

We consider safety propertiesφ⊆S\\varphi\\subseteq Sgiven as a set of safe states\. A runs0​a0​s1​a1​…s\_\{0\}a\_\{0\}s\_\{1\}a\_\{1\}\\dotsis*safe*ifsi∈φs\_\{i\}\\in\\varphifor allii\.

Given a deterministic policyπ:S→A\\pi\\colon S\\to A, a*shield*is any nondeterministic policy∇:S→𝟤A\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-15\.00002pt \\hbox to5pt\{\\vbox to6\.5pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-1\.25pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 6\.92 L 6\.23 6\.92 L 6\.23 1\.84 L 3\.11 \-1\.38 L 0 1\.84 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-6\.49994pt \}\\colon S\\to\\mathsf\{2\}^\{A\}over the same states and actions\. The*shielded policy*π∇\\pi\_\{\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-11\.95836pt \\hbox to3\.5pt\{\\vbox to4\.55pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.175pt\\lower\-0\.875pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 4\.84 L 4\.36 4\.84 L 4\.36 1\.29 L 2\.18 \-0\.97 L 0 1\.29 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-5\.1819pt \}\}is a deterministic policy that acts similarly toπ\\pibut only chooses actions allowed by the shield, i\.e\.,∀s∈S∀a∈A:π∇\(s\)=a⟹a∈∇\(s\)\\forall s\\in S~\\forall a\\in A\\colon\\pi\_\{\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-11\.95836pt \\hbox to3\.5pt\{\\vbox to4\.55pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.175pt\\lower\-0\.875pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 4\.84 L 4\.36 4\.84 L 4\.36 1\.29 L 2\.18 \-0\.97 L 0 1\.29 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-5\.1819pt \}\}\(s\)=a\\implies a\\in\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-15\.00002pt \\hbox to5pt\{\\vbox to6\.5pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-1\.25pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 6\.92 L 6\.23 6\.92 L 6\.23 1\.84 L 3\.11 \-1\.38 L 0 1\.84 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-6\.49994pt \}\(s\)\. Additionally, the shield only alters actions when necessary:∀s∈S:π\(s\)∈∇\(s\)⟹π∇\(s\)=π\(s\)\\forall s\\in S\\colon\\pi\(s\)\\in\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-15\.00002pt \\hbox to5pt\{\\vbox to6\.5pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-1\.25pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 6\.92 L 6\.23 6\.92 L 6\.23 1\.84 L 3\.11 \-1\.38 L 0 1\.84 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-6\.49994pt \}\(s\)\\implies\\pi\_\{\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-11\.95836pt \\hbox to3\.5pt\{\\vbox to4\.55pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.175pt\\lower\-0\.875pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 4\.84 L 4\.36 4\.84 L 4\.36 1\.29 L 2\.18 \-0\.97 L 0 1\.29 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-5\.1819pt \}\}\(s\)=\\pi\(s\)\. The action chosen byπ∇\(s\)\\pi\_\{\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-11\.95836pt \\hbox to3\.5pt\{\\vbox to4\.55pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.175pt\\lower\-0\.875pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 4\.84 L 4\.36 4\.84 L 4\.36 1\.29 L 2\.18 \-0\.97 L 0 1\.29 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-5\.1819pt \}\}\(s\)whenπ\(s\)∉∇\(s\)\\pi\(s\)\\notin\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-15\.00002pt \\hbox to5pt\{\\vbox to6\.5pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-1\.25pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 6\.92 L 6\.23 6\.92 L 6\.23 1\.84 L 3\.11 \-1\.38 L 0 1\.84 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-6\.49994pt \}\(s\)depends on the implementation of the RL agent\. In our implementation, we use Q\-learning\[[41](https://arxiv.org/html/2608.19836#bib.bib9)\], for which it is straightforward to read out the best admissible action in∇\(s\)\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-15\.00002pt \\hbox to5pt\{\\vbox to6\.5pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-1\.25pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 6\.92 L 6\.23 6\.92 L 6\.23 1\.84 L 3\.11 \-1\.38 L 0 1\.84 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-6\.49994pt \}\(s\)\.

## 3Probabilistic Shielding Using an Estimator

In this paper, we construct shields based on the method described in\[[12](https://arxiv.org/html/2608.19836#bib.bib17)\], which was originally developed for the*offline*RL problem where a fixed dataset is given\. Our approach differs in that we continuously adapt the shield based on newly generated data, and we must consider exploration to collect new data within and beyond the shield’s allowed actions\. As we will see later, such a setting is particularly challenging and yields trade\-offs between safety and exploration\.

### 3\.1Probabilistic shielding approaches for \(interval\) MDPs

Next, we recall how to obtain a shield∇\\nablafor an \(interval\) MDPMMand a safety specificationφ\\varphi\. We assume a horizonh∈ℕh\\in\\mathbb\{N\}and parametersθ,κ∈\[0,1\]\\theta,\\kappa\\in\[0,1\], which we explain below\. Before the formalization, we first describe the high\-level idea\. Intuitively, the shield ensures the existence of a series ofhhactions from the current statesssuch that the chance of a safety violation along these steps is belowθ\\theta\. For tractability, the shield is memoryless and thus ignores accumulated past risk before reaching the current statess\. Since the restriction is probabilistic and memoryless, there may still be a chance of reaching a state where this guarantee does not hold\[[12](https://arxiv.org/html/2608.19836#bib.bib17),[25](https://arxiv.org/html/2608.19836#bib.bib22),[17](https://arxiv.org/html/2608.19836#bib.bib3)\]\. Whenever no sufficiently safe action is available, the shield only allows actions that areκ\\kappa\-close to the safest available action\.

Now we formalize this idea\. Letφ\|h\\varphi\|\{h\}be the set of all*hh\-safe runs*s0​a0​s1​a1​…s\_\{0\}a\_\{0\}s\_\{1\}a\_\{1\}\\dotswith safehh\-prefix, i\.e\.,si∈φs\_\{i\}\\in\\varphifori≤hi\\leq h\. LetℙπM,φ\|h​\(s\)\\mathbb\{P\}\_\{\\pi\}^\{M,\\varphi\|\{h\}\}\(s\)be the probability of an MDPMMproducing anhh\-safe run by following policyπ\\pistarting in statess\. We denote the related safety optimization problem byℙmaxM,φ\|h​\(s\)=maxπ⁡ℙπM,φ\|h​\(s\)\\mathbb\{P\}\_\{\\max\}^\{M,\\varphi\|h\}\(s\)=\\max\_\{\\pi\}\\mathbb\{P\}\_\{\\pi\}^\{M,\\varphi\|h\}\(s\)\. Moreover, let the probability of producing a run inφ\|h\\varphi\|hafter taking actionaain statessbeℙmaxM,φ\|h​\(s,a\)=∑s′∈ST⁡\(s,a,s′\)​ℙmaxM,φ\|h−1​\(s′\)\\mathbb\{P\}\_\{\\max\}^\{M,\\varphi\|h\}\(s,a\)=\\sum\_\{s^\{\\prime\}\\in S\}T\(s,a,s^\{\\prime\}\)\\mathbb\{P\}\_\{\\max\}^\{M,\\varphi\|h\-1\}\(s^\{\\prime\}\)\.

Normally, the shield allows all actions guaranteeing a safehh\-step run with probability at least1−θ1\-\\theta; formally:∇θ\(s\)=\{a∈A∣ℙmaxM,φ\|h\(s,a\)≥1−θ\}\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-15\.00002pt \\hbox to5pt\{\\vbox to6\.5pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-1\.25pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 6\.92 L 6\.23 6\.92 L 6\.23 1\.84 L 3\.11 \-1\.38 L 0 1\.84 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-6\.49994pt \}\_\{\\theta\}\(s\)=\\\{a\\in A\\mid\\mathbb\{P\}\_\{\\max\}^\{M,\\varphi\|h\}\(s,a\)\\geq 1\-\\theta\\\}\. However, a shielded policy may still reach a statesswhere this shield definition would not allow any action \(i\.e\.,∇θ\(s\)=∅\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-15\.00002pt \\hbox to5pt\{\\vbox to6\.5pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-1\.25pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 6\.92 L 6\.23 6\.92 L 6\.23 1\.84 L 3\.11 \-1\.38 L 0 1\.84 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-6\.49994pt \}\_\{\\theta\}\(s\)=\\emptyset\)\. In that case, the shield instead allows all actions that areκ\\kappa\-close to the safest available action; formally:∇κ\(s\)=\{a∈A∣ℙmaxM,φ\|h\(s,a\)≥maxa′ℙmaxM,φ\|h\(s,a′\)−κ\}\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-15\.00002pt \\hbox to5pt\{\\vbox to6\.5pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-1\.25pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 6\.92 L 6\.23 6\.92 L 6\.23 1\.84 L 3\.11 \-1\.38 L 0 1\.84 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-6\.49994pt \}\_\{\\kappa\}\(s\)=\\\{a\\in A\\mid\\mathbb\{P\}\_\{\\max\}^\{M,\\varphi\|h\}\(s,a\)\\geq\\max\_\{a^\{\\prime\}\}\\mathbb\{P\}\_\{\\max\}^\{M,\\varphi\|h\}\(s,a^\{\\prime\}\)\-\\kappa\\\}\. Our shield combines these two cases:

∇\(s\)=\{∇θ\(s\)if∇θ\(s\)≠∅∇κ\(s\)otherwise\.\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-15\.00002pt \\hbox to5pt\{\\vbox to6\.5pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-1\.25pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 6\.92 L 6\.23 6\.92 L 6\.23 1\.84 L 3\.11 \-1\.38 L 0 1\.84 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-6\.49994pt \}\(s\)=\\begin\{cases\}\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-15\.00002pt \\hbox to5pt\{\\vbox to6\.5pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-1\.25pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 6\.92 L 6\.23 6\.92 L 6\.23 1\.84 L 3\.11 \-1\.38 L 0 1\.84 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-6\.49994pt \}\_\{\\theta\}\(s\)&\\mathrm\{if\}~\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-15\.00002pt \\hbox to5pt\{\\vbox to6\.5pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-1\.25pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 6\.92 L 6\.23 6\.92 L 6\.23 1\.84 L 3\.11 \-1\.38 L 0 1\.84 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-6\.49994pt \}\_\{\\theta\}\(s\)\\neq\\emptyset\\\\ \\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-15\.00002pt \\hbox to5pt\{\\vbox to6\.5pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-1\.25pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 6\.92 L 6\.23 6\.92 L 6\.23 1\.84 L 3\.11 \-1\.38 L 0 1\.84 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-6\.49994pt \}\_\{\\kappa\}\(s\)&\\mathrm\{otherwise\.\}\\end\{cases\}\(1\)
Following\[[12](https://arxiv.org/html/2608.19836#bib.bib17)\], we extend shields to iMDPs by defining the probability of producing a run for iMDPs as follows\. LetℙmaxoptMI,φ\|h\(s\)=maxπoptM∈MIℙπM,φ\|h\(s\)\\mathbb\{P\}\_\{\\max\\opt\}^\{M\_\{I\},\\varphi\|h\}\(s\)=\\max\_\{\\pi\}\\opt\_\{M\\in M\_\{I\}\}\\mathbb\{P\}\_\{\\pi\}^\{M,\\varphi\|h\}\(s\)be the probability of producing a run under the modelM∈MIM\\in M\_\{I\}corresponding to the optimization directionopt∈\{min,max\}\\opt\\in\\\{\\min,\\max\\\}\. Moreover, let the iMDP version of the probability producing a run inφ\|h\\varphi\|hafter taking actionaain statessbe

ℙmaxoptMI,φ\|h\(s,a\)\\displaystyle\\mathbb\{P\}\_\{\\max\\opt\}^\{M\_\{I\},\\varphi\|h\}\(s,a\)=optT†∈TI∑s′∈ST†\(s,a,s′\)ℙmaxoptMI,φ\|h−1\(s′\)\.\\displaystyle=\\opt\_\{\{T\}^\{\\dagger\}\\in T\_\{I\}\}\\,\\sum\_\{s^\{\\prime\}\\in S\}T^\{\\dagger\}\(s,a,s^\{\\prime\}\)\\mathbb\{P\}\_\{\\max\\opt\}^\{M\_\{I\},\\varphi\|h\-1\}\(s^\{\\prime\}\)\.\(2\)We use the probabilistic model checker PRISM\[[26](https://arxiv.org/html/2608.19836#bib.bib12)\]to efficiently compute such probabilities on iMDPs\. Statistically speaking, a worst\-case assumption \(opt=min\\opt=\\min\) makes the shield*robust*against estimation errors\. Following\[[12](https://arxiv.org/html/2608.19836#bib.bib17)\], we define a \(pessimistic\)*robust shield*for an iMDPMIM\_\{I\}similarly to[Eq\.1](https://arxiv.org/html/2608.19836#S3.E1)usingℙmax⁡minMI,φ\|h​\(s,a\)\\mathbb\{P\}\_\{\\max\\min\}^\{M\_\{I\},\\varphi\|h\}\(s,a\)\. Instead of assuming the worst\-case MDPMMfrom an estimateMIM\_\{I\}, one can also assume the best case\(opt=max\)\(\\opt=\\max\), specified asℙmax⁡maxMI,φ\|h​\(s\)\\mathbb\{P\}\_\{\\max\\max\}^\{M\_\{I\},\\varphi\|h\}\(s\)\. We call this alternative an*optimistic shield*\. We say that the*attitude*of a shield is either robust or optimistic, depending on how it was constructed from an iMDP\.

### 3\.2Estimators from data for unknown MDPs

We recall three existing estimators, based on those that appeared in\[[37](https://arxiv.org/html/2608.19836#bib.bib14)\]\. While exploring the black\-box MDP, we count how many times a transition triple\(s,a,s′\)\(s,a,s^\{\\prime\}\)has been observed\. For that, we use a transition databaseD:S×A×S→ℕD\\colon S\\times A\\times S\\to\\mathbb\{N\}\. LetD⁡\(s,a\)=∑s′D⁡\(s,a,s′\)D\(s,a\)=\\sum\_\{s^\{\\prime\}\}D\(s,a,s^\{\\prime\}\)be the*total count*for a state\-action pair\(s,a\)\(s,a\)\.

An*estimator*is a functionE⁡\(MU,D\)E\(M\_\{U\},D\)that maps a uMDPMUM\_\{U\}and a transition databaseDDto either an estimated MDPM^\\hat\{M\}or iMDPM^I\\hat\{M\}\_\{I\}, depending on the estimator\. Below, we describe three estimators that learn \(i\.e\., estimate the transition function of\) MDPs or iMDPs, with or without guarantees:*MAP*\(EMAPE\_\{\\textrm\{MAP\}\}\),*PAC*\(EPACE\_\{\\textrm\{PAC\}\}\), and*LUI*\(ELUIE\_\{\\textrm\{LUI\}\}\)\. These approaches estimate the transition function locally for each\(s,a\)\(s,a\)\-pair using knowledge of the graph in the form of a given uMDPMUM\_\{U\}, whereTU​\(s,a\)T\_\{U\}\(s,a\)denotes the set of successor states\.

##### MAP\.

The first approach finds a*point estimate*T^\\hat\{T\}of the MDP’s transition functionTTbased on the dataDD\. Following\[[37](https://arxiv.org/html/2608.19836#bib.bib14)\], we define point estimates as maximum a\-posteriori \(MAP\) estimation with respect to a symmetric prior weight assigned to each successor state, which we denote as a singlew∈ℕw\\in\\mathbb\{N\}\. Then

T^​\(s,a,s′\)=w\+D⁡\(s,a,s′\)−1\(∑t∈TU​\(s,a\)w\+D⁡\(s,a,t\)\)−\|TU​\(s,a\)\|\\hat\{T\}\(s,a,s^\{\\prime\}\)=\\frac\{w\+D\(s,a,s^\{\\prime\}\)\-1\}\{\\left\(\\sum\_\{t\\in T\_\{U\}\(s,a\)\}w\+D\(s,a,t\)\\right\)\-\|T\_\{U\}\(s,a\)\|\}defines the MAP point estimate\. It can be viewed as a maximum\-likelihood probability with some additive smoothing fromww\. The MAP estimatorEMAPE\_\{\\textrm\{MAP\}\}maps\(MU,D\)\(M\_\{U\},D\)to an estimated MDPM^\\hat\{M\}with point estimatesT^\\hat\{T\}\.

##### PAC\.

The point estimates of the MAP estimator do not account for the uncertainty arising from estimating probabilities from data\. Point estimates can be turned into*probably approximately correct*\(PAC\) intervals via Hoeffding’s inequality\[[18](https://arxiv.org/html/2608.19836#bib.bib13)\], such that the estimated iMDP contains the true MDP with high probability1−δ1\-\\delta, forδ∈\[0,1\]\\delta\\in\[0,1\]\[[2](https://arxiv.org/html/2608.19836#bib.bib8),[37](https://arxiv.org/html/2608.19836#bib.bib14),[12](https://arxiv.org/html/2608.19836#bib.bib17)\]\. As such, by the union bound, we distributeδ\\deltaover all transitions asδT=δ/∑s,ak⁡\(s,a\)\\delta\_\{T\}=\\nicefrac\{\{\\delta\}\}\{\{\\sum\_\{s,a\}k\(s,a\)\}\}, wherek⁡\(s,a\)k\(s,a\)denotes the number of successor statesk⁡\(s,a\)=\|TU​\(s,a\)\|k\(s,a\)=\|T\_\{U\}\(s,a\)\|if\|TU​\(s,a\)\|\>1\|T\_\{U\}\(s,a\)\|\>1andk⁡\(s,a\)=0k\(s,a\)=0otherwise\. Then,ηs,a=log⁡\[2/δT\]/2⋅D⁡\(s,a\)\\eta\_\{s,a\}=\\nicefrac\{\{\\log\[\\nicefrac\{\{2\}\}\{\{\\delta\_\{T\}\}\}\]\}\}\{\{2\\cdot D\(s,a\)\}\}denotes the range of the PAC interval around the point estimate for\(s,a\)\(s,a\)\. Using eachηs,a\\eta\_\{s,a\}, we construct the intervals

T^I​\(s,a,s′\)=\[max⁡\(ξ,T^​\(s,a,s′\)−ηs,a\),min⁡\(1,T^​\(s,a,s′\)\+ηs,a\)\]\\hat\{T\}\_\{I\}\(s,a,s^\{\\prime\}\)=\\left\[\\max\(\\xi,\\hat\{T\}\(s,a,s^\{\\prime\}\)\-\\eta\_\{s,a\}\),\\min\(1,\\hat\{T\}\(s,a,s^\{\\prime\}\)\+\\eta\_\{s,a\}\)\\right\]\(3\)whereξ∈\(0,1\)\\xi\\in\(0,1\)is a small constant that ensures intervals for transitions with nonzero probability \(as given by the unknown MDPMUM\_\{U\}\) map to\[ξ,1\]\[\\xi,1\]\. The PAC estimatorEPACE\_\{\\textrm\{PAC\}\}maps\(MU,D\)\(M\_\{U\},D\)to an iMDPM^I\\hat\{M\}\_\{I\}with intervalsT^I\\hat\{T\}\_\{I\}from[Eq\.3](https://arxiv.org/html/2608.19836#S3.E3)\.

*LUI\.*The third approach that we consider is the*linearly updating intervals*\(LUI\) estimator from\[[37](https://arxiv.org/html/2608.19836#bib.bib14)\], which in turn is based on\[[40](https://arxiv.org/html/2608.19836#bib.bib16)\]\. While it does not retain PAC guarantees, it iteratively learns probabilities by updating intervals\. We assign each unknown transition a prior intervalT~I​\(s,a,si′\)=\[T¯i,T¯i\]\\tilde\{T\}\_\{I\}\(s,a,s\_\{i\}^\{\\prime\}\)=\[\\underline\{T\}\_\{i\},\\overline\{T\}\_\{i\}\]and prior strength\[n¯i,n¯i\]\[\\underline\{n\}\_\{i\},\\overline\{n\}\_\{i\}\]\. The strength influences the prior’s effect on the updated intervals\. At any point, we find new intervals given the databaseDDby distinguishing cases based on whether the current intervals agree with the new data\. For any\(s,a\)\(s,a\)andsj′s^\{\\prime\}\_\{j\}, letFj=D⁡\(s,a,sj′\)/D⁡\(s,a\)F\_\{j\}=\\nicefrac\{\{D\(s,a,s^\{\\prime\}\_\{j\}\)\}\}\{\{D\(s,a\)\}\}denote the relative occurrence of transition\(s,a,sj′\)\(s,a,s^\{\\prime\}\_\{j\}\)in the databaseDD\. Then, the updates are:

T¯i←\{n¯i​T¯i\+D⁡\(s,a,si′\)n¯i\+D⁡\(s,a\)if​Fj≥T¯j​for all​sj′,n¯i​T¯i\+D⁡\(s,a,si′\)n¯i\+D⁡\(s,a\)otherwise\.\\underline\{T\}\_\{i\}\\leftarrow\\begin\{cases\}\\vskip 5\.0pt\\frac\{\\overline\{n\}\_\{i\}\\underline\{T\}\_\{i\}\+D\(s,a,s^\{\\prime\}\_\{i\}\)\}\{\\overline\{n\}\_\{i\}\+D\(s,a\)\}\\quad\\text\{if \}F\_\{j\}\\geq\\underline\{T\}\_\{j\}\\text\{ for all \}s\_\{j\}^\{\\prime\},\\\\ \\frac\{\\underline\{n\}\_\{i\}\\underline\{T\}\_\{i\}\+D\(s,a,s^\{\\prime\}\_\{i\}\)\}\{\\underline\{n\}\_\{i\}\+D\(s,a\)\}\\quad\\text\{otherwise\}\.\\end\{cases\}\(4\)T¯i←\{n¯i​T¯i\+D⁡\(s,a,si′\)n¯i\+D⁡\(s,a\)if​Fj≤T¯j​for all​sj′,n¯i​T¯i\+D⁡\(s,a,si′\)n¯i\+D⁡\(s,a\)otherwise\.\\overline\{T\}\_\{i\}\\leftarrow\\begin\{cases\}\\vskip 5\.0pt\\frac\{\\overline\{n\}\_\{i\}\\overline\{T\}\_\{i\}\+D\(s,a,s^\{\\prime\}\_\{i\}\)\}\{\\overline\{n\}\_\{i\}\+D\(s,a\)\}\\quad\\text\{if \}F\_\{j\}\\leq\\overline\{T\}\_\{j\}\\text\{ for all \}s\_\{j\}^\{\\prime\},\\\\ \\frac\{\\underline\{n\}\_\{i\}\\overline\{T\}\_\{i\}\+D\(s,a,s^\{\\prime\}\_\{i\}\)\}\{\\underline\{n\}\_\{i\}\+D\(s,a\)\}\\quad\\text\{otherwise\}\.\\end\{cases\}\(5\)
The \(strength\) intervals are found from total counts\[n¯i\+D⁡\(s,a\),n¯i\+D⁡\(s,a\)\]\[\\underline\{n\}\_\{i\}\+D\(s,a\),\\overline\{n\}\_\{i\}\+D\(s,a\)\]\. Initial intervals are valid when0<T¯i≤T¯j≤10<\\underline\{T\}\_\{i\}\\leq\\overline\{T\}\_\{j\}\\leq 1andn¯i≥n¯i≥1\\overline\{n\}\_\{i\}\\geq\\underline\{n\}\_\{i\}\\geq 1\. Similarly toEPACE\_\{\\textrm\{PAC\}\},ELUIE\_\{\\textrm\{LUI\}\}maps\(MU,D\)\(M\_\{U\},D\)to an iMDPM~I\\tilde\{M\}\_\{I\}with intervalsT~I\\tilde\{T\}\_\{I\}using[Eq\.4](https://arxiv.org/html/2608.19836#S3.E4)\.

##### Guarantees and convergence\.

While onlyEPACE\_\{\\textrm\{PAC\}\}provides statistical guarantees from finite data\[[2](https://arxiv.org/html/2608.19836#bib.bib8)\], all three estimators converge to the true probabilities as the number of visits to each transition tends to infinity\[[37](https://arxiv.org/html/2608.19836#bib.bib14)\]\.

## 4Adaptive Probabilistic Shielding

In this section, we develop the paradigm that we call*adaptive probabilistic shielding*\. Before we present our algorithm, we motivate the problem it addresses\.

### 4\.1Problem Statement

We consider an RL application in a safety\-critical real\-world scenario\. In particular, we do not know the underlying MDP model \(only the underlying uMDP\) and hence do not assume access to a settable simulator\. While safety violations may not be entirely avoidable, we place great importance on them, as they may occur during real\-world data collection outside a simulator\[[27](https://arxiv.org/html/2608.19836#bib.bib10)\]\. Hence, our goal is to obtain a policyπ\\pisubject to three sub\-goals: \(i\) achieve a given admissibility threshold of the safety specification, \(ii\) achieve a high expected reward, and \(iii\) achieve a low number of safety violations during training\.

To highlight the intricacy of our problem, we point out that goal \(iii\) is in direct competition with the other two goals\. This is because higher safety during training requires more conservative exploration, which may prevent the discovery of a better\-performing policy \(e\.g\., a faster and/or safer route to a goal state\)\. Since we do not assume prior knowledge of the environment’s transition dynamics, one may have to take more risks to learn them; thus, an action deemed less safe due to higher uncertainty may only be determined to be safer after enough exploration\. While RL solves \(ii\), it does not achieve \(i\), and it may also perform poorly regarding \(iii\), since it typically relies on \(random\) exploration of the environment\. To additionally achieve \(i\), we could apply shielded RL; however, since we do not know the MDP, we would need to learn an MDP or iMDP model, which would itself require exploration for the data collection and thus again fail to achieve goal \(iii\)\.

### 4\.2Adaptive Probabilistic Shielding

Our answer to this dilemma is to interweave all three procedures \(policy learning, shield construction, and model estimation\) into a single adaptive learning loop\. Generally, we collect data from the RL agent’s exploration of the environment\. From time to time, we use that data to update our model estimate, and from that improved estimate, we obtain a refined shield that allows us to continue exploring the environment more safely and/or less conservatively\.

One may be tempted to think that this process will, given enough episodes, converge to the ideal solution of learning the underlying MDP and thus the best possible shield\. However, this is not necessarily the case, and indeed, we observed that such an approach can fail in practice\. The issue is that the conservative shield, from the beginning, may simply prevent exploration of large parts of the state space, even if the corresponding actions were to be perfectly safe under the true MDP model\. More specifically, the shield cannot distinguish between actions that are already known to be risky \(which it should indeed block\) and actions for which the model estimate is too coarse to make a definite judgement\. This is particularly pronounced for the more pessimistic robust shields\.

Our final step is to extend theε\\varepsilon\-greedy exploration strategy to explore beyond the shield’s boundaries\. When the exploration strategy decides to explore a random action in statess, we choose this action from the full set of actionsAA, rather than just from∇\(s\)\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-15\.00002pt \\hbox to5pt\{\\vbox to6\.5pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-1\.25pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 6\.92 L 6\.23 6\.92 L 6\.23 1\.84 L 3\.11 \-1\.38 L 0 1\.84 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-6\.49994pt \}\(s\)allowed by the shield\. We show the impact of this extension empirically in the next section in[\(RQ6\)](https://arxiv.org/html/2608.19836#S5.I1.i6)\.

Algorithm 1Safe RL via Adaptive Probabilistic Shielding1:black\-box MDPM=\(S,A,s0,T\)M=\(S,A,s\_\{0\},T\); uMDPMU=\(S,A,s0,TU\)M\_\{U\}=\(S,A,s\_\{0\},T\_\{U\}\); reward functionRR; safety specificationφ\\varphi; estimatorEE; shield update delayu∈ℕu\\in\\mathbb\{N\}; shield parametersθ,κ∈\[0,1\]\\theta,\\kappa\\in\[0,1\]andh∈ℕh\\in\\mathbb\{N\}; exploration rateε∈\[0,1\]\\varepsilon\\in\[0,1\]; number of episodesN∈ℕN\\in\\mathbb\{N\}; maximum episode lengthL∈ℕL\\in\\mathbb\{N\}

2:π←\\pi\\leftarrowInitialize RL policy

3:D⁡\(s,a,s′\)←0,∀\(s,a,s′\)D\(s,a,s^\{\\prime\}\)\\leftarrow 0,\\quad\\forall\\,\(s,a,s^\{\\prime\}\)⊳\\trianglerightInitialize transition database

4:foriiin00toN−1N\-1do

5:ifi≡0modui\\equiv 0\\mod uthen

6:M^←E⁡\(MU,D\)\\hat\{M\}\\leftarrow E\(M\_\{U\},D\)⊳\\trianglerightSee[Section3\.2](https://arxiv.org/html/2608.19836#S3.SS2)

7:∇←\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-15\.00002pt \\hbox to5pt\{\\vbox to6\.5pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-1\.25pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 6\.92 L 6\.23 6\.92 L 6\.23 1\.84 L 3\.11 \-1\.38 L 0 1\.84 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-6\.49994pt \}\\leftarrowSynthesize shield fromM^\\hat\{M\},θ\\theta,κ\\kappa, andhhforφ\\varphi⊳\\trianglerightSee[Section3\.1](https://arxiv.org/html/2608.19836#S3.SS1)

8:s←s0s\\leftarrow s\_\{0\}

9:forjjin00toL−1L\-1do

10:ifRandom bit with probabilityε\\varepsilonof being truethen

11:a∼UnifAa\\sim\\text\{Unif\}\_\{A\}⊳\\triangleright“Explore” – uniform choice among all actions

12:else

13:a=π∇\(s\)a=\\pi\_\{\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-11\.95836pt \\hbox to3\.5pt\{\\vbox to4\.55pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.175pt\\lower\-0\.875pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 4\.84 L 4\.36 4\.84 L 4\.36 1\.29 L 2\.18 \-0\.97 L 0 1\.29 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-5\.1819pt \}\}\(s\)⊳\\triangleright“Exploit” – agent chooses the best safe action

14:s′∼T⁡\(s,a\)s^\{\\prime\}\\sim T\(s,a\)⊳\\trianglerightTake a step with actionaain the environment

15:π←\\pi\\leftarrowUpdate policy with transition\(s,a,s′\)\(s,a,s^\{\\prime\}\)and rewardR⁡\(s,a,s′\)R\(s,a,s^\{\\prime\}\)

16:D⁡\(s,a,s′\)←D⁡\(s,a,s′\)\+1D\(s,a,s^\{\\prime\}\)\\leftarrow D\(s,a,s^\{\\prime\}\)\+1

17:s←s′s\\leftarrow s^\{\\prime\}return\(

π,∇,M^\\pi,\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-15\.00002pt \\hbox to5pt\{\\vbox to6\.5pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-1\.25pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 6\.92 L 6\.23 6\.92 L 6\.23 1\.84 L 3\.11 \-1\.38 L 0 1\.84 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-6\.49994pt \},\\hat\{M\}\)

[Algorithm1](https://arxiv.org/html/2608.19836#alg1)shows the pseudocode of our proposed approach\. Notably, we only use white\-box access to the environment via the uMDPMUM\_\{U\}, while we only access the underlying MDPMMimplicitly via the black\-box functionsTTandRR\.

In our implementation, we use Q\-learning\[[41](https://arxiv.org/html/2608.19836#bib.bib9)\]to find a policyπ\\pi\. For that, we initializeπ\\pias an empty Q\-table and an empty databaseDDof observed transition triples\. In the first iteration of the outer for\-loop in Line[4](https://arxiv.org/html/2608.19836#alg1.l4),i=0i=0satisfies the condition in Line[5](https://arxiv.org/html/2608.19836#alg1.l5), which triggers the computation of the first model estimateM^\\hat\{M\}and shield∇\\nabla\. Since the transition database is still empty, this estimate and shield are most conservative\. The inner for\-loop in Line[9](https://arxiv.org/html/2608.19836#alg1.l9)represents Q\-learning for a single episode with the extendedε\\varepsilon\-greedy exploration strategy described above under∇\\nabla\. In particular, we either select a random action \(“explore”\) or select the current best action that is allowed by the shield \(“exploit”\)\. The selected action is then executed in the environment MDPMM\. Its output is recorded in the transition databaseDDand used along with the immediate rewardrrto update the Q\-table\. Everyuuepisodes, we update the model estimate and shield\. After exceeding the training budget ofNNepisodes, we return the final versions of the policy, the shield, and the model estimate\.

## 5Experimental Evaluation

In this section, we evaluate our proposed adaptive probabilistic shielding approach from different angles\. We aim to answer the following research questions:

1. \(RQ1\)Can our approach learn a safe and optimal policy?
2. \(RQ2\)What is the effect on the environment exploration?
3. \(RQ3\)Does the model estimate improve over time?
4. \(RQ4\)Do the model estimates become sufficiently precise?
5. \(RQ5\)What is the impact of the specific model estimator?
6. \(RQ6\)Is unshielded exploration beneficial?
7. \(RQ7\)How often should the shield be updated?
8. \(RQ8\)What is the impact of the shield lookahead \(hh\)?

### 5\.1Implementation and Baseline Methods

The implementation is available online\[[7](https://arxiv.org/html/2608.19836#bib.bib38)\]\. We train the agent using standard Q\-learning\[[41](https://arxiv.org/html/2608.19836#bib.bib9)\]with the hyperparametersα=0\.1\\alpha=0\.1\(learning rate\),γ=0\.9\\gamma=0\.9\(discount factor\), andε=0\.05\\varepsilon=0\.05\(exploration probability\)\. We use reward shaping to penalize safety violations, with the penalty varying by environment\. Instead, we separately record the number of episodes that were unsafe\.

By default, we use the LUI estimatorELUIE\_\{\\textrm\{LUI\}\}with prior strengths\[n¯i,n¯i\]=\[5,10\]\[\\underline\{n\}\_\{i\},\\overline\{n\}\_\{i\}\]=\[5,10\]and a \(“pessimistic”\) robust shield attitude with parametersθ=0\.05\\theta=0\.05,κ=0\.01\\kappa=0\.01, andh=100h=100, and update the estimate and shield everyu=1000u=1000episodes\. We underline these defaults in the following figures and tables\.

We compare to two baselines\. The first baseline is a standard unshielded RL agent trained with reward shaping; this baseline is expected to perform poorly in terms of safety due to the lack of a shield\. The second baseline is a shielded RL agent that uses a probabilistic shield computed with the same method but given the ground\-truth MDP \(which our method cannot access\); this “oracle” baseline acts as a benchmark and is expected to outperform all other methods\.

### 5\.2Description of Environments

In total, we consider five different environments with mixed safety and optimization objectives\. The first environment is described in previous literature while the remaining were developed as additional benchmarks for our problem setting\.

Theaircraftenvironment\[[37](https://arxiv.org/html/2608.19836#bib.bib14),[24](https://arxiv.org/html/2608.19836#bib.bib19)\]\(\|S\|=1665\|S\|=1665\) represents a collision avoidance system of an aircraft that must keep a minimum vertical distance to another plane that passes horizontally\. The other plane changes altitude at random, and the agent’s aircraft may fail to follow the instructions with a small probability\.

Theantlionenvironment \(\|S\|=400\|S\|=400\) requires the agent \(an ant\) to circumnavigate a stationary predator in order to reach a goal on the other side\. Instead of moving in the intended direction, the agent may slip toward the predator, with the probability increasing with proximity to it\. Reaching the goal yields a reward, while taking a step has a cost that decreases with proximity to the goal\.

Thesinkholesenvironment \(\|S\|=400\|S\|=400\) features multiple goals with varying rewards and multiple holes that must be avoided\. If the agent falls into a hole, it either escapes with a small probability or returns to the starting state\. As the agent moves, it may slip in a random direction instead, with the probability varying by the state\. The cost of moving decreases with proximity to the goal\.

Thecrossroadsenvironment \(\|S\|=202\|S\|=202\) illustrates deferred risk\. In the initial state, the agent must choose between two roads, which are then followed for100100steps\. One route is safe, while the other route is more rewarding but has a risk of slipping into an unsafe state at each step\.

Thegravityenvironment \(\|S\|=2000\|S\|=2000\) rewards the agent for visiting a sequence of checkpoints near a gravity well without crashing into the latter\. Every step has a small cost, and there is a probability that the agent is dragged towards the well, which increases with proximity\. Later checkpoints are riskier to visit, and the agent can end the episode early by going to one of two exit points\.

### 5\.3Experimental Results and Discussion of Research Questions

Our experiments consists of2121hyperparameter configurations \(e\.g\., the choice of the model estimator\), and we repeated each run100100times\.

##### [\(RQ1\)](https://arxiv.org/html/2608.19836#S5.I1.i1): Can our approach learn a safe and optimal policy?

I II III IV

![Refer to caption](https://arxiv.org/html/2608.19836v1/graphics/Aircraft_safety.png)

![Refer to caption](https://arxiv.org/html/2608.19836v1/graphics/Aircraft_reward.png)

![Refer to caption](https://arxiv.org/html/2608.19836v1/graphics/Aircraft_fallback_actions.png)

![Refer to caption](https://arxiv.org/html/2608.19836v1/graphics/Aircraft_TV.png)

![Refer to caption](https://arxiv.org/html/2608.19836v1/graphics/Antlion_safety.png)

![Refer to caption](https://arxiv.org/html/2608.19836v1/graphics/Antlion_reward.png)

![Refer to caption](https://arxiv.org/html/2608.19836v1/graphics/Antlion_fallback_actions.png)

![Refer to caption](https://arxiv.org/html/2608.19836v1/graphics/Antlion_TV.png)

![Refer to caption](https://arxiv.org/html/2608.19836v1/graphics/Sinkholes_safety.png)

![Refer to caption](https://arxiv.org/html/2608.19836v1/graphics/Sinkholes_reward.png)

![Refer to caption](https://arxiv.org/html/2608.19836v1/graphics/Sinkholes_fallback_actions.png)

![Refer to caption](https://arxiv.org/html/2608.19836v1/graphics/Sinkholes_TV.png)

![Refer to caption](https://arxiv.org/html/2608.19836v1/graphics/Crossroads_safety.png)

![Refer to caption](https://arxiv.org/html/2608.19836v1/graphics/Crossroads_reward.png)

![Refer to caption](https://arxiv.org/html/2608.19836v1/graphics/Crossroads_fallback_actions.png)

![Refer to caption](https://arxiv.org/html/2608.19836v1/graphics/Crossroads_TV.png)

![Refer to caption](https://arxiv.org/html/2608.19836v1/graphics/Gravity_safety.png)

![Refer to caption](https://arxiv.org/html/2608.19836v1/graphics/Gravity_reward.png)

![Refer to caption](https://arxiv.org/html/2608.19836v1/graphics/Gravity_fallback_actions.png)

![Refer to caption](https://arxiv.org/html/2608.19836v1/graphics/Gravity_TV.png)

Figure 2:Mean outcome of 100 repetitions for different configurations\. We plot the standard deviation as a ribbon around the lines\. Vertical grid lines mark updates of the adaptive shield\. Column I: Cumulative safety violations during training\. Column II: Reward during training\. Column III: Per\-episode rate of using the fallback∇κ\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-15\.00002pt \\hbox to5pt\{\\vbox to6\.5pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-1\.25pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 6\.92 L 6\.23 6\.92 L 6\.23 1\.84 L 3\.11 \-1\.38 L 0 1\.84 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-6\.49994pt \}\_\{\\kappa\}\. Column IV: Average total variation between true model and model estimate during training\.![Refer to caption](https://arxiv.org/html/2608.19836v1/graphics/Antlion_Unshielded_Heatmap.png)

![Refer to caption](https://arxiv.org/html/2608.19836v1/graphics/Antlion_Oracle_Heatmap.png)

![Refer to caption](https://arxiv.org/html/2608.19836v1/graphics/Antlion_LUI_Heatmap.png)

![Refer to caption](https://arxiv.org/html/2608.19836v1/graphics/Antlion_PAC_Heatmap.png)

![Refer to caption](https://arxiv.org/html/2608.19836v1/graphics/Antlion_MAP_Heatmap.png)

![Refer to caption](https://arxiv.org/html/2608.19836v1/graphics/Antlion_Heatmap_Legend.png)

Figure 3:Heat map showing how frequently a state is visited in the antlion environment\. The initial state, predator, and goal are at the bottom, center, and top, respectively\.We are interested in both safety \([Fig\.2](https://arxiv.org/html/2608.19836#S5.F2)column I\) and performance \([Fig\.2](https://arxiv.org/html/2608.19836#S5.F2)column II\) over time\. \(Note that the penalty stemming from the reward shaping is not included in these plots\.\) For most environments, the adaptive shield leads to about the same number of safety violations as the oracle shield\. The unshielded baseline is generally less safe, especially in the crossroads and gravity environments, despite a strong reward penalty\. Still, the negative outcomes were overshadowed by the more frequent positive rewards in the Q\-learning algorithm\. We also note that the oracle baseline can generally explore more freely than the adaptive method because of its less conservative shield\. Yet, in the aircraft environment, the adaptive method actually achieves a slightly more rewarding policy, profiting from slightly riskier behavior\. This may seem counterintuitive given the robust attitude of the shield, which is generally more conservative than the oracle baseline\. The reason we still see this behavior is that, during the exploration, profitable states are visited more often, making these seem safer as compared to less explored states with wider interval estimates\. Indeed, column III reveals that the shield mostly falls back to the∇κ\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-15\.00002pt \\hbox to5pt\{\\vbox to6\.5pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-1\.25pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 6\.92 L 6\.23 6\.92 L 6\.23 1\.84 L 3\.11 \-1\.38 L 0 1\.84 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-6\.49994pt \}\_\{\\kappa\}variant in this case\.

Updates to the adaptive shield can be seen to temporarily affect performance negatively in the antlion, sinkholes, and gravity environments\. At the10001000episode mark, the shield is updated for the first time, leading to a drop in the reward performance\. This drop is to be expected: as seen in column III, the initial shield typically uses the∇κ\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-15\.00002pt \\hbox to5pt\{\\vbox to6\.5pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-1\.25pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 6\.92 L 6\.23 6\.92 L 6\.23 1\.84 L 3\.11 \-1\.38 L 0 1\.84 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-6\.49994pt \}\_\{\\kappa\}variant because every action seems unsafe\. The first model update is most impactful and hence typically leads to a very different shield, and hence the agent effectively experiences a different environment\. Over time, the reward performance recovers as the policy adapts to the new shield\.

Overall, the shields have a significant positive impact on safety and explore the environment more safely\. This safety may come at a cost when risky behavior is profitable, but this is a desirable trade\-off in many applications\.

##### [\(RQ2\)](https://arxiv.org/html/2608.19836#S5.I1.i2): What is the effect on the environment exploration?

To assess how adaptive shields affect exploration,[Fig\.3](https://arxiv.org/html/2608.19836#S5.F3)visualizes the number of times a state has been visited in the antlion environment during one algorithm execution over10,00010\{,\}000episodes, comparing the unshielded and oracle baselines as well as adaptive shields with the estimators*robust LUI*\(default\),*robust PAC*, and*MAP*\.

The LUI and PAC interval estimators both find the goal by following some narrow paths during learning\. Meanwhile, the MAP estimator almost never leaves the area around the initial states\. This is consistent with the low average performance of the MAP estimator in[Table1](https://arxiv.org/html/2608.19836#S5.T1)and further investigated below in[\(RQ5\)](https://arxiv.org/html/2608.19836#S5.I1.i5)\. The unshielded agent passes close to the antlion but takes a more circuitous route, similar to what the oracle shield allows\. Both appear to explore more freely around the paths they take, rather than staying on a narrow route permitted by the adaptive shield\. This highlights the need for exploring states outside of what the shield allows, as also investigated in[\(RQ6\)](https://arxiv.org/html/2608.19836#S5.I1.i6)below\.

##### [\(RQ3\)](https://arxiv.org/html/2608.19836#S5.I1.i3): Does the model estimate improve over time?

We plot mean*total variation*\(TV\), i\.e\.,1/\|S\|​\|A\|​∑s,a1/2​∑s′∈S\|T†​\(s,a,s′\)−T∗​\(s,a,s′\)\|\\nicefrac\{\{1\}\}\{\{\|S\|\|A\|\}\}\\sum\_\{s,a\}\\nicefrac\{\{1\}\}\{\{2\}\}\\sum\_\{s^\{\\prime\}\\in S\}\|\{T^\{\\dagger\}\}\(s,a,s^\{\\prime\}\)\-T^\{\*\}\(s,a,s^\{\\prime\}\)\|, between the true MDP’s transition probabilitiesT∗T^\{\*\}and the probabilitiesT†T^\{\\dagger\}returned by PMC used for the shield \(e\.g\., for robust/optimistic shields, as found from[Eq\.2](https://arxiv.org/html/2608.19836#S3.E2)\)\. We omit transitions with probability 1, since their probability is known\.

As the agent explores the environment, we obtain a more precise estimate of the model\. Exploration also comes with risk; thus, it is not desirable to obtain a perfect estimate of all transitions – in particular not those transitions that are rarely visited by the policy\. The mean TV after each model update is shown in column IV of[Fig\.2](https://arxiv.org/html/2608.19836#S5.F2)\. The most significant change occurs in the first update after10001000episodes, indicating that this is sufficient to collect data for key transitions, which will be further explored during the rest of the training\. While the estimates for these transitions keep improving, this has little impact on the \(global\) metric\.

##### [\(RQ4\)](https://arxiv.org/html/2608.19836#S5.I1.i4): Do the model estimates become sufficiently precise?

We examine how often the shield allows an action because it satisfies theθ\\thetathreshold \(∇θ\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-15\.00002pt \\hbox to5pt\{\\vbox to6\.5pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-1\.25pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 6\.92 L 6\.23 6\.92 L 6\.23 1\.84 L 3\.11 \-1\.38 L 0 1\.84 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-6\.49994pt \}\_\{\\theta\}\) respectively how often it has to use the fallback \(∇κ\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-15\.00002pt \\hbox to5pt\{\\vbox to6\.5pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-1\.25pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 6\.92 L 6\.23 6\.92 L 6\.23 1\.84 L 3\.11 \-1\.38 L 0 1\.84 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-6\.49994pt \}\_\{\\kappa\}\) \(cf\.[Eq\.1](https://arxiv.org/html/2608.19836#S3.E1)\)\. Column III of[Fig\.2](https://arxiv.org/html/2608.19836#S5.F2)shows the fallback frequency\. Until the first model update \(10001000episodes\), the robust and MAP estimates primarily use∇κ\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-15\.00002pt \\hbox to5pt\{\\vbox to6\.5pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-1\.25pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 6\.92 L 6\.23 6\.92 L 6\.23 1\.84 L 3\.11 \-1\.38 L 0 1\.84 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-6\.49994pt \}\_\{\\kappa\}, whereas the optimistic estimates, which consider almost all actions safe from the beginning, primarily use∇θ\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-15\.00002pt \\hbox to5pt\{\\vbox to6\.5pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-1\.25pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 6\.92 L 6\.23 6\.92 L 6\.23 1\.84 L 3\.11 \-1\.38 L 0 1\.84 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-6\.49994pt \}\_\{\\theta\}\. The slowest improvement of the estimate is observed in the aircraft environment\. This is because, unlike the static obstacles in the other environments, learning the behavior of the randomly moving opponent requires more data\. In all other environments,∇θ\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-15\.00002pt \\hbox to5pt\{\\vbox to6\.5pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.25pt\\lower\-1\.25pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 6\.92 L 6\.23 6\.92 L 6\.23 1\.84 L 3\.11 \-1\.38 L 0 1\.84 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-6\.49994pt \}\_\{\\theta\}is used most of the time after one or two model updates for all but the robust PAC estimator, which sometimes fails to obtain a sufficiently precise estimate \(due to its higher data requirements\)\.

##### [\(RQ5\)](https://arxiv.org/html/2608.19836#S5.I1.i5): What is the impact of the specific model estimator?

Table 1:Comparison for different model estimators\. The numbers in each cell respectively denote the reward in the final evaluation \(left\) and the probability that the final policy produces an unsafe episode \(right\)\. The numbers are the mean outcomes of100100repetitions and bold entries mark the best result for each column\.EstimatorAircraftAntlionSinkholesCrossroadsGravityRobust LUI14\.89 8\.3%5\.27 4\.5%57\.05 3\.6%5\.120\.0%8\.27 3\.9%Robust PAC15\.548\.2%6\.332\.8%46\.59 3\.8%5\.120\.0%−\-2\.560\.0%MAP13\.244\.0%−\-8\.99 8\.0%49\.163\.0%5\.120\.0%−\-2\.350\.0%Optimistic LUI14\.17 5\.2%6\.66 6\.0%67\.78 4\.1%5\.110\.0%21\.40 19\.6%Optimistic PAC14\.35 6\.8%6\.36 10\.0%65\.35 3\.8%5\.120\.0%23\.51 47\.8%Unshielded14\.36 7\.3%6\.62 9\.0%65\.62 3\.7%9\.5740\.1%30\.3599\.2%Oracle13\.98 4\.1%6\.785\.6%73\.323\.8%5\.120\.0%19\.86 4\.5%Table 2:Comparison for the two exploration variants\. Setup as in[Table1](https://arxiv.org/html/2608.19836#S5.T1)\.ExplorationAircraftAntlionSinkholesCrossroadsGravityUnifA\\mathrm\{Unif\}\_\{A\}14\.898\.3%5\.274\.5%57\.053\.6%5\.120\.0%8\.273\.9%Unif∇\(s\)\\mathrm\{Unif\}\_\{\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-11\.95836pt \\hbox to3\.5pt\{\\vbox to4\.55pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.175pt\\lower\-0\.875pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 4\.84 L 4\.36 4\.84 L 4\.36 1\.29 L 2\.18 \-0\.97 L 0 1\.29 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-5\.1819pt \}\(s\)\}15\.049\.8%1\.93 5\.1%57\.293\.7%5\.000\.0%1\.580\.6%We investigate how the different model estimators impact the results\. Specifically, we compare the iMDP estimators PAC and LUI, both with the robust and the optimistic attitudes, and the MDP estimator MAP\. We fix the priors of MAP tow=10w=10and of LUI to\[n¯i,n¯i\]=\[5,10\]\[\\underline\{n\}\_\{i\},\\overline\{n\}\_\{i\}\]=\[5,10\], and the parameters of PAC toδ=0\.1\\delta=0\.1andξ=10−8\\xi=10^\{\-8\}\. Varying priors as an additional dimension of experimental parameters is left for future research\.

In[Table1](https://arxiv.org/html/2608.19836#S5.T1), we show the evaluation of the final policies\. Each cell shows the reward, evaluated empirically as the mean over10001000episodes, and the relative safety of this policy, computed analytically using the PRISM model checker\[[26](https://arxiv.org/html/2608.19836#bib.bib12)\]\.111We did not use PRISM to compute the reward because it does not support a mix of positive and negative rewards\.

Counterintuitively, robust estimators do not always lead to safer policies, as seen in the aircraft environment\. Since the agent initially finds an imperfect route, which is considered relatively safe compared to less\-explored states, the shield later forces the agent to stay on it, even if an unexplored yet safer alternative may exist\. Instead, the optimistic shield specifications allow the agent to explore more states unless prior experience indicates that doing so is unsafe\. The sinkholes and gravity environments show the biggest variation in the results\. This is because the goals \(checkpoints\) in the sinkholes \(gravity\) environment are far apart, and hence finding a good route strongly depends on the exploration\.

##### [\(RQ6\)](https://arxiv.org/html/2608.19836#S5.I1.i6): Is unshielded exploration beneficial?

In Line[11](https://arxiv.org/html/2608.19836#alg1.l11)of[Algorithm1](https://arxiv.org/html/2608.19836#alg1), theε\\varepsilon\-greedy exploration chooses a random action from the set of all actionsUnifA\\mathrm\{Unif\}\_\{A\}, thus disregarding the shield\. In[Table2](https://arxiv.org/html/2608.19836#S5.T2), we compare this strategy to the alternative that only admissible actions are allowed in statesswithUnif∇\(s\)\\mathrm\{Unif\}\_\{\\text\{\{\\color\[rgb\]\{1,1,1\} $\\nabla$\} \\hskip\-11\.95836pt \\hbox to3\.5pt\{\\vbox to4\.55pt\{\\pgfpicture\\makeatletter\\hbox\{\\hskip 0\.175pt\\lower\-0\.875pt\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@begingroup@\{stroke\} \\lxSVG@begingroup@\{fill\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\\nullfont\\hbox to0\.0pt\{\\lxSVG@begingroup@\{\_scopebegin\} \{ \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\\lx@inpgf@ignorespaces\}\{\} \{\{\\lx@inpgf@ignorespaces\}\{\}\}\{\} \{\\lx@inpgf@ignorespaces\}\\lxSVG@begingroup@\{\_scopebegin\} \\lxSVG@setlinewidth\{\\the\\pgflinewidth\}\\lxSVG@begingroup@\{stroke\-width\} \\lx@inpgf@ignorespaces\{\}\\lxSVG@stroke\\lxSVG@drawpath@unclipped\{M 0 4\.84 L 4\.36 4\.84 L 4\.36 1\.29 L 2\.18 \-0\.97 L 0 1\.29 Z\}\{fill:none\} \\lx@inpgf@ignorespaces \\lxSVG@closescope \} \\lxSVG@closescope \{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\{\\lx@inpgf@ignorespaces\}\\hss\}\\lxSVG@discardpath\\lxSVG@closescope \\hss\}\}\\lxSVG@closescope\\endpgfpicture\}\} \\hskip\-5\.1819pt \}\(s\)\}\. Generally, with the latter variant, the agent eventually stops exploring new actions once the shield has found at least one admissible route\. For some environments, this change has no significant effect on the reward because the route that was identified first was sufficiently good, while in the antlion and gravity environments, the restricted variant prevents the agent from uncovering more promising routes\.

##### [\(RQ7\)](https://arxiv.org/html/2608.19836#S5.I1.i7): How often should the shield be updated?

Table 3:Comparison for different update delaysuu\. Setup as in[Table1](https://arxiv.org/html/2608.19836#S5.T1)\.uuAircraftAntlionSinkholesCrossroadsGravity25014\.94 12\.9%−\-10\.74 9\.7%78\.71 5\.4%5\.120\.0%−\-2\.320\.0%50014\.9910\.5%−\-10\.46 11\.0%84\.784\.1%5\.120\.0%2\.95 1\.9%1 00014\.89 8\.3%5\.27 4\.5%57\.053\.6%5\.120\.0%8\.27 3\.9%1 50014\.81 7\.8%6\.173\.7%54\.403\.6%5\.120\.0%11\.02 3\.5%2 00014\.727\.0%6\.06 4\.0%57\.933\.6%5\.120\.0%12\.928\.2%Table 4:Comparison for varying shield horizon \(hhinφ\|h\\varphi\|h\)\. Setup as in[Table1](https://arxiv.org/html/2608.19836#S5.T1)\.hhAircraftAntlionSinkholesCrossroadsGravity614\.508\.2%6\.423\.1%48\.503\.3%9\.5740\.1%24\.6147\.1%1214\.788\.2%5\.90 3\.8%54\.81 3\.4%9\.5740\.1%23\.16 33\.3%2514\.898\.3%5\.71 4\.2%56\.31 3\.7%9\.5740\.1%22\.16 21\.5%5014\.898\.3%4\.67 4\.9%57\.12 3\.4%9\.5740\.1%19\.36 13\.2%7514\.898\.3%4\.39 4\.4%56\.35 3\.7%9\.5740\.1%13\.55 6\.9%10014\.898\.3%5\.27 4\.4%57\.05 3\.7%5\.120\.0%8\.27 4\.0%12514\.898\.3%5\.37 4\.0%57\.09 3\.7%5\.120\.0%5\.21 2\.7%15014\.898\.3%5\.14 3\.6%57\.07 3\.9%5\.120\.0%3\.81 2\.5%17514\.898\.3%5\.42 3\.6%57\.193\.7%5\.120\.0%2\.90 2\.0%20014\.898\.3%5\.333\.1%57\.18 3\.6%5\.120\.0%1\.241\.5%The model estimate and subsequent shield update are the most expensive operations in our approach\. We examine the effect of the update delayuu\.[Table3](https://arxiv.org/html/2608.19836#S5.T3)shows that the choice ofuucan be impactful, but no single choice is more preferable across the environments\.

##### [\(RQ8\)](https://arxiv.org/html/2608.19836#S5.I1.i8): What is the impact of the shield lookahead \(hh\)?

We examine the effect of varying the lookahead horizonhhof the shield\.[Table4](https://arxiv.org/html/2608.19836#S5.T4)shows the results\. Aircraft episodes end after2020steps, making longer horizons redundant\. The gravity environment is less safe at lower lookahead\. In the crossroads environment, forh≤75h\\leq 75, the agent prefers the more rewarding \(but less safe\) route\.

## 6Conclusion and Future Work

In this paper, we have proposed the paradigm of*adaptive probabilistic shielding*for safe reinforcement learning\. We assume only access to a nondeterministic environment model \(i\.e\., we do not know the transition probabilities\) and consequently do not have access to a simulator\. In settings where safety violations are costly, safe exploration is a challenge\. We tackle this challenge with a practical, integrated procedure that simultaneously explores the environment, updates a model estimate, maintains a probabilistic shield, and learns a policy under that shield\. Our focus has been on the empirical evaluation, in which we investigated various research questions regarding the success and impact of our design choices\.

While our shield implementation is based on a recent approach for interval MDPs\[[12](https://arxiv.org/html/2608.19836#bib.bib17)\], the procedure can be extended to support other shields\. One direction is to vary the safety thresholds of the shields computed during the course of the algorithm, which we have kept to a user\-defined constant in our approach\. For instance, we imagine first using an optimistic shield that encourages exploration and then gradually raising the safety threshold to obtain a safer shield in the end\. As another direction, one can incorporate more elements from model\-based RL algorithms\. For instance, we may replace the uniformly random exploration step with a biased choice toward under\-explored states and actions\.

#### Acknowledgements

This research was partly supported by the European Research Council \(ERC\) Starting Grant 101077178 \(DEUCE\), the Villum Investigator Grant S4OS under reference number 37819, and the Independent Research Fund Denmark under reference number 10\.46540/3120\-00041B\.

#### Disclosure of Interests\.

The authors have no competing interests to declare that are relevant to the content of this article\.

## References

- \[1\]\(2018\)Safe reinforcement learning via shielding\.InAAAI,pp\. 2669–2678\.External Links:[Document](https://dx.doi.org/10.1609/AAAI.V32I1.11797)Cited by:[§1](https://arxiv.org/html/2608.19836#S1.SS0.SSS0.Px1.p1.1),[§1\.1](https://arxiv.org/html/2608.19836#S1.SS1.SSS0.Px1.p1.1)\.
- \[2\]P\. Ashok, J\. Kretínský, and M\. Weininger\(2019\)PAC statistical model checking for Markov decision processes and stochastic games\.InCAV,LNCS, Vol\.11561,pp\. 497–519\.External Links:[Document](https://dx.doi.org/10.1007/978-3-030-25540-4%5F29)Cited by:[§3\.2](https://arxiv.org/html/2608.19836#S3.SS2.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2608.19836#S3.SS2.SSS0.Px3.p1.1)\.
- \[3\]C\. Baier and J\. Katoen\(2008\)Principles of model checking\.The MIT Press\.Cited by:[§1](https://arxiv.org/html/2608.19836#S1.p2.1)\.
- \[4\]D\. Bethell, S\. Gerasimou, R\. Calinescu, and C\. Imrie\(2025\)Safe reinforcement learning in black\-box environments via adaptive shielding\.InECAI,FAIA,pp\. 2450–2457\.External Links:[Document](https://dx.doi.org/10.3233/FAIA251092)Cited by:[§1\.1](https://arxiv.org/html/2608.19836#S1.SS1.SSS0.Px2.p1.1)\.
- \[5\]A\. H\. Brorholt, P\. G\. Jensen, K\. G\. Larsen, F\. Lorber, and C\. Schilling\(2023\)Shielded reinforcement learning for hybrid systems\.InAISoLA,LNCS,pp\. 33–54\.External Links:[Document](https://dx.doi.org/10.1007/978-3-031-46002-9%5F3)Cited by:[§1\.1](https://arxiv.org/html/2608.19836#S1.SS1.SSS0.Px1.p1.1)\.
- \[6\]A\. H\. Brorholt, K\. G\. Larsen, and C\. Schilling\(2025\)Compositional shielding and reinforcement learning for multi\-agent systems\.InAAMAS,pp\. 399–407\.External Links:[Link](https://dl.acm.org/doi/10.5555/3709347.3743554)Cited by:[§1\.1](https://arxiv.org/html/2608.19836#S1.SS1.SSS0.Px1.p1.1)\.
- \[7\]Code and data for “Adaptive probabilistic shielding by learning MDPs for safe reinforcement learning”External Links:[Document](https://dx.doi.org/10.5281/zenodo.21874278)Cited by:[§5\.1](https://arxiv.org/html/2608.19836#S5.SS1.p1.1)\.
- \[8\]S\. Carr, G\. Bakirtzis, and U\. Topcu\(2025\)Compositional shield synthesis for safe reinforcement learning in partial observability\.CoRRabs/2509\.12085\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.2509.12085),2509\.12085Cited by:[§1\.1](https://arxiv.org/html/2608.19836#S1.SS1.SSS0.Px1.p1.1)\.
- \[9\]A\. David, P\. G\. Jensen, K\. G\. Larsen, A\. Legay, D\. Lime, M\. G\. Sørensen, and J\. H\. Taankvist\(2014\)On time with minimal expected cost\!\.InATVA,LNCS, Vol\.8837,pp\. 129–145\.External Links:[Document](https://dx.doi.org/10.1007/978-3-319-11936-6%5F10)Cited by:[§1](https://arxiv.org/html/2608.19836#S1.SS0.SSS0.Px1.p1.1)\.
- \[10\]I\. Elsayed\-Aly, S\. Bharadwaj, C\. Amato, R\. Ehlers, U\. Topcu, and L\. Feng\(2021\)Safe multi\-agent reinforcement learning via shielding\.InAAMAS,pp\. 483–491\.External Links:[Document](https://dx.doi.org/10.5555/3463952.3464013)Cited by:[§1\.1](https://arxiv.org/html/2608.19836#S1.SS1.SSS0.Px1.p1.1)\.
- \[11\]Y\. Feng, J\. Zhu, A\. Platzer, and J\. Laurent\(2025\)Adaptive shielding via parametric safety proofs\.Proc\. ACM Program\. Lang\.9\(OOPSLA1\),pp\. 816–843\.External Links:[Document](https://dx.doi.org/10.1145/3720450)Cited by:[§1\.1](https://arxiv.org/html/2608.19836#S1.SS1.SSS0.Px2.p1.1)\.
- \[12\]M\. F\. L\. Galesloot, T\. Rhemrev, and N\. Jansen\(2026\)Robust probabilistic shielding for safe offline reinforcement learning\.CoRRabs/2605\.10293\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.2605.10293),2605\.10293Cited by:[§1](https://arxiv.org/html/2608.19836#S1.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2608.19836#S1.SS0.SSS0.Px4.p1.1),[§1\.1](https://arxiv.org/html/2608.19836#S1.SS1.SSS0.Px1.p1.1),[§1\.1](https://arxiv.org/html/2608.19836#S1.SS1.SSS0.Px3.p1.1),[§3\.1](https://arxiv.org/html/2608.19836#S3.SS1.p1.1),[§3\.1](https://arxiv.org/html/2608.19836#S3.SS1.p4.1),[§3\.1](https://arxiv.org/html/2608.19836#S3.SS1.p4.2),[§3\.2](https://arxiv.org/html/2608.19836#S3.SS2.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2608.19836#S3.p1.1),[§6](https://arxiv.org/html/2608.19836#S6.p2.1)\.
- \[13\]J\. García and F\. Fernández\(2015\)A comprehensive survey on safe reinforcement learning\.J\. Mach\. Learn\. Res\.16,pp\. 1437–1480\.External Links:[Document](https://dx.doi.org/10.5555/2789272.2886795)Cited by:[§1](https://arxiv.org/html/2608.19836#S1.SS0.SSS0.Px1.p1.1)\.
- \[14\]A\. W\. Goodall and F\. Belardinelli\(2023\)Approximate model\-based shielding for safe reinforcement learning\.InECAI,FAIA, Vol\.372,pp\. 883–890\.External Links:[Document](https://dx.doi.org/10.3233/FAIA230357)Cited by:[§1\.1](https://arxiv.org/html/2608.19836#S1.SS1.SSS0.Px2.p1.1)\.
- \[15\]E\. Hamel\-De le Court, T\. Badings, A\. Abate, F\. Belardinelli, and F\. Fabiano\(2026\)Robust shielding for safe reinforcement learning\.CoRRabs/2606\.00270\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.2606.00270),2606\.00270Cited by:[§1\.1](https://arxiv.org/html/2608.19836#S1.SS1.SSS0.Px3.p1.1)\.
- \[16\]E\. Hamel\-De le Court, F\. Belardinelli, and A\. W\. Goodall\(2025\)Probabilistic shielding for safe reinforcement learning\.InAAAI,pp\. 16091–16099\.External Links:[Document](https://dx.doi.org/10.1609/AAAI.V39I15.33767)Cited by:[§1\.1](https://arxiv.org/html/2608.19836#S1.SS1.SSS0.Px1.p1.1)\.
- \[17\]L\. Heck, F\. Macák, R\. Andriushchenko, M\. Češka, and S\. Junges\(2026\)Shields to guarantee probabilistic safety in mdps\.InCAV,External Links:[Link](https://arxiv.org/abs/2605.10888)Cited by:[§1](https://arxiv.org/html/2608.19836#S1.SS0.SSS0.Px1.p1.1),[§1\.1](https://arxiv.org/html/2608.19836#S1.SS1.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2608.19836#S3.SS1.p1.1)\.
- \[18\]W\. Hoeffding\(1963\)Probability inequalities for sums of bounded random variables\.J\. Am\. Stat\. Assoc\.58\(301\),pp\. 13–30\.External Links:ISSN 01621459, 1537274X,[Link](http://www.jstor.org/stable/2282952)Cited by:[§3\.2](https://arxiv.org/html/2608.19836#S3.SS2.SSS0.Px2.p1.1)\.
- \[19\]K\. V\. Huynh, D\. Parker, and L\. Feng\(2025\)Robust permissive controller synthesis for interval mdps\.CoRRabs/2510\.03481\.Cited by:[§1\.1](https://arxiv.org/html/2608.19836#S1.SS1.SSS0.Px1.p1.1)\.
- \[20\]M\. Jaeger, G\. Bacci, G\. Bacci, K\. G\. Larsen, and P\. G\. Jensen\(2020\)Approximating Euclidean by imprecise Markov decision processes\.InISoLA,LNCS, Vol\.12476,pp\. 275–289\.External Links:[Document](https://dx.doi.org/10.1007/978-3-030-61362-4%5F15)Cited by:[§2](https://arxiv.org/html/2608.19836#S2.SS0.SSS0.Px3.p3.1)\.
- \[21\]N\. Jansen, B\. Könighofer, S\. Junges, A\. Serban, and R\. Bloem\(2020\)Safe reinforcement learning using probabilistic shields \(invited paper\)\.InCONCUR,LIPIcs,pp\. 3:1–3:16\.External Links:[Document](https://dx.doi.org/10.4230/LIPICS.CONCUR.2020.3)Cited by:[§1](https://arxiv.org/html/2608.19836#S1.SS0.SSS0.Px1.p1.1),[§1\.1](https://arxiv.org/html/2608.19836#S1.SS1.SSS0.Px1.p1.1)\.
- \[22\]B\. Jonsson and K\. G\. Larsen\(1991\)Specification and refinement of probabilistic processes\.InLICS,pp\. 266–277\.External Links:[Document](https://dx.doi.org/10.1109/LICS.1991.151651)Cited by:[§2](https://arxiv.org/html/2608.19836#S2.SS0.SSS0.Px3.p3.1)\.
- \[23\]K\. Kim, D\. Corsi, A\. Rodríguez, J\. Lanier, B\. Parellada, P\. Baldi, C\. Sánchez, and R\. Fox\(2025\)Realizable continuous\-space shields for safe reinforcement learning\.InL4DC,PMLR,pp\. 932–945\.External Links:[Link](https://proceedings.mlr.press/v283/kim25c.html)Cited by:[§1\.1](https://arxiv.org/html/2608.19836#S1.SS1.SSS0.Px1.p1.1)\.
- \[24\]M\. Kochenderfer\(2015\)Decision making under uncertainty\.MIT Press\.Cited by:[§1](https://arxiv.org/html/2608.19836#S1.p1.1),[§5\.2](https://arxiv.org/html/2608.19836#S5.SS2.p2.1)\.
- \[25\]B\. Könighofer, R\. Bloem, N\. Jansen, S\. Junges, and S\. Pranger\(2025\)Shields for safe reinforcement learning\.Commun\. ACM68\(11\),pp\. 80–90\.External Links:[Document](https://dx.doi.org/10.1145/3715958)Cited by:[§1](https://arxiv.org/html/2608.19836#S1.SS0.SSS0.Px1.p1.1),[§1\.1](https://arxiv.org/html/2608.19836#S1.SS1.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2608.19836#S3.SS1.p1.1)\.
- \[26\]M\. Z\. Kwiatkowska, G\. Norman, and D\. Parker\(2011\)PRISM 4\.0: verification of probabilistic real\-time systems\.InCAV,LNCS, Vol\.6806,pp\. 585–591\.External Links:[Document](https://dx.doi.org/10.1007/978-3-642-22110-1%5F47)Cited by:[§1](https://arxiv.org/html/2608.19836#S1.SS0.SSS0.Px3.p1.1),[§3\.1](https://arxiv.org/html/2608.19836#S3.SS1.p4.2),[§5\.3](https://arxiv.org/html/2608.19836#S5.SS3.SSS0.Px5.p2.1)\.
- \[27\]B\. Lacerda, F\. Faruq, D\. Parker, and N\. Hawes\(2019\)Probabilistic planning with formal performance guarantees for mobile service robots\.Int\. J\. Robotics Res\.38\(9\)\.External Links:[Document](https://dx.doi.org/10.1177/0278364919856695)Cited by:[§4\.1](https://arxiv.org/html/2608.19836#S4.SS1.p1.1)\.
- \[28\]J\. Moos, K\. Hansel, H\. Abdulsamad, S\. Stark, D\. Clever, and J\. Peters\(2022\)Robust reinforcement learning: A review of foundations and recent advances\.Mach\. Learn\. Knowl\. Extr\.4\(1\),pp\. 276–315\.External Links:[Document](https://dx.doi.org/10.3390/MAKE4010013)Cited by:[§1](https://arxiv.org/html/2608.19836#S1.SS0.SSS0.Px5.p1.1)\.
- \[29\]A\. Nilim and L\. E\. Ghaoui\(2005\)Robust control of Markov decision processes with uncertain transition matrices\.Oper\. Res\.53\(5\),pp\. 780–798\.External Links:[Document](https://dx.doi.org/10.1287/OPRE.1050.0216)Cited by:[§1](https://arxiv.org/html/2608.19836#S1.SS0.SSS0.Px4.p1.1),[§2](https://arxiv.org/html/2608.19836#S2.SS0.SSS0.Px3.p3.1)\.
- \[30\]A\. Pnueli\(1977\)The temporal logic of programs\.InFOCS,pp\. 46–57\.External Links:[Document](https://dx.doi.org/10.1109/SFCS.1977.32)Cited by:[§1](https://arxiv.org/html/2608.19836#S1.p2.1)\.
- \[31\]S\. Pranger, B\. Könighofer, M\. Tappler, M\. Deixelberger, N\. Jansen, and R\. Bloem\(2021\)Adaptive shielding under uncertainty\.InACC,pp\. 3467–3474\.External Links:[Document](https://dx.doi.org/10.23919/ACC50511.2021.9482889)Cited by:[§1\.1](https://arxiv.org/html/2608.19836#S1.SS1.SSS0.Px1.p1.1),[§1\.1](https://arxiv.org/html/2608.19836#S1.SS1.SSS0.Px2.p1.1)\.
- \[32\]M\. L\. Puterman\(1994\)Markov decision processes: discrete stochastic dynamic programming\.Wiley Series in Probability and Statistics,Wiley\.External Links:[Document](https://dx.doi.org/10.1002/9780470316887),ISBN 978\-0\-47161977\-2Cited by:[§1](https://arxiv.org/html/2608.19836#S1.p1.1)\.
- \[33\]R\. Reed and M\. Lahijanian\(2025\)Learning\-based shielding for safe autonomy under unknown dynamics\.InACC,pp\. 4940–4946\.Cited by:[§1\.1](https://arxiv.org/html/2608.19836#S1.SS1.SSS0.Px1.p1.1)\.
- \[34\]P\. Senthilvelan, J\. Li, and K\. Tei\(2023\)Similarity\-based shield adaptation under dynamic environment\.InSEAI,pp\. 33–39\.External Links:[Document](https://dx.doi.org/10.1109/SEAI59139.2023.10217461)Cited by:[§1\.1](https://arxiv.org/html/2608.19836#S1.SS1.SSS0.Px2.p1.1)\.
- \[35\]A\. L\. Strehl and M\. L\. Littman\(2008\)An analysis of model\-based interval estimation for Markov decision processes\.J\. Comput\. Syst\. Sci\.74\(8\),pp\. 1309–1331\.External Links:[Document](https://dx.doi.org/10.1016/J.JCSS.2007.08.009)Cited by:[§2](https://arxiv.org/html/2608.19836#S2.SS0.SSS0.Px3.p3.1)\.
- \[36\]M\. Suilen, T\. Badings, E\. M\. Bovy, D\. Parker, and N\. Jansen\(2024\)Robust Markov decision processes: A place where AI and formal methods meet\.InPrinciples of Verification \(3\),LNCS,pp\. 126–154\.External Links:[Document](https://dx.doi.org/10.1007/978-3-031-75778-5%5F7)Cited by:[§1](https://arxiv.org/html/2608.19836#S1.SS0.SSS0.Px4.p1.1),[§2](https://arxiv.org/html/2608.19836#S2.SS0.SSS0.Px3.p3.1)\.
- \[37\]M\. Suilen, T\. D\. Simão, D\. Parker, and N\. Jansen\(2022\)Robust anytime learning of Markov decision processes\.InNeurIPS,External Links:[Link](https://doi.org/10.52202/068431-2087)Cited by:[§1](https://arxiv.org/html/2608.19836#S1.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2608.19836#S1.SS0.SSS0.Px4.p1.1),[§1\.1](https://arxiv.org/html/2608.19836#S1.SS1.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2608.19836#S2.SS0.SSS0.Px3.p3.1),[§3\.2](https://arxiv.org/html/2608.19836#S3.SS2.SSS0.Px1.p1.1),[§3\.2](https://arxiv.org/html/2608.19836#S3.SS2.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2608.19836#S3.SS2.SSS0.Px2.p2.1),[§3\.2](https://arxiv.org/html/2608.19836#S3.SS2.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2608.19836#S3.SS2.p1.1),[§5\.2](https://arxiv.org/html/2608.19836#S5.SS2.p2.1)\.
- \[38\]R\. S\. Sutton and A\. G\. Barto\(2018\)Reinforcement learning: an introduction\.MIT Press\.External Links:[Link](http://www.incompleteideas.net/book/the-book-2nd.html)Cited by:[§1](https://arxiv.org/html/2608.19836#S1.SS0.SSS0.Px5.p1.1),[§1](https://arxiv.org/html/2608.19836#S1.p2.1),[§2](https://arxiv.org/html/2608.19836#S2.SS0.SSS0.Px4.p1.1),[§2](https://arxiv.org/html/2608.19836#S2.SS0.SSS0.Px4.p2.1)\.
- \[39\]M\. Tappler, S\. Pranger, B\. Könighofer, E\. Muskardin, R\. Bloem, and K\. G\. Larsen\(2022\)Automata learning meets shielding\.InISoLA,LNCS, Vol\.13701,pp\. 335–359\.External Links:[Document](https://dx.doi.org/10.1007/978-3-031-19849-6%5F20)Cited by:[§1\.1](https://arxiv.org/html/2608.19836#S1.SS1.SSS0.Px2.p1.1)\.
- \[40\]G\. Walter and T\. Augustin\(2009\)Imprecision and prior\-data conflict in generalized Bayesian inference\.J\. Stat\. Theory Pract\.3\(1\),pp\. 255–271\.Cited by:[§3\.2](https://arxiv.org/html/2608.19836#S3.SS2.SSS0.Px2.p2.1)\.
- \[41\]C\. J\. C\. H\. Watkins and P\. Dayan\(1992\)Q\-learning\.Mach\. Learn\.8,pp\. 279–292\.External Links:[Document](https://dx.doi.org/10.1007/BF00992698)Cited by:[§2](https://arxiv.org/html/2608.19836#S2.SS0.SSS0.Px5.p2.1),[§4\.2](https://arxiv.org/html/2608.19836#S4.SS2.p5.1),[§5\.1](https://arxiv.org/html/2608.19836#S5.SS1.p1.1)\.