Structured Reinforcement Learning for Bayesian Persuasion : Application to Intelligent Interactive Driving

# Structured Reinforcement Learning for Bayesian Persuasion : Application to Intelligent Interactive Driving
Source: [https://arxiv.org/html/2607.13576](https://arxiv.org/html/2607.13576)
###### Abstract

Interactive driving, wherein an intelligent lead vehicle equipped with real\-time traffic data coordinates route choices of connected vehicles, offers a promising approach to dynamic traffic management\. To address the challenge of harmonising decisions — the lead vehicle prioritizes global traffic flow, while the connected vehicle seeks the shortest path, this paper considers the strategic information revealing framework of Bayesian persuasion\. Here, the principal \(lead vehicle\) aims to guide the agent’s \(connected vehicle\) partially observable sequential decision making towards its own objectives by selectively revealing information, such as real\-time traffic ahead, using signals\. However, the agent’s farsighted response to maximize its long\-term reward, accounting for future state transitions and signal disclosures, renders the principal’s signaling strategy design computationally challenging\. Moreover, in real\-world settings, both the principal and agent may not have complete knowledge of their reward and transition dynamics\. We propose an online structured reinforcement learning framework to synthesize computationally efficient signaling strategy which is*persuasive*for a far\-sighted agent\. In this context, the main contributions of the paper are as follows:\(i\)For a monotonic agent with approximate best response, we propose MAPL, a structured policy learning algorithm which utilizes the monotonic structure of the agent’s policy for faster online learning in large state spaces,\(ii\)Identification of sufficient conditions on the Bayesian persuasion model for the supermodular structure of the Q \(action value\) function of the principal for a monotonic agent,\(iii\)Identification of sufficient conditions to ensure the persuasiveness of the principal’s signaling strategy for a monotonic agent,\(iv\)Supermodular Q learning for Principal \(SQP\), which leverages the supermodular structure of principal’s action value to synthesize computationally efficient signaling strategy that is persuasive for a monotonic learning agent,\(v\)Numerical analysis considering a real\-time application of Bayesian persuasive driving for lane selection demonstrates that the proposed method is 30% cost efficient for optimising travelling rewards of both the lead and connected vehicle compared to the existing methodologies for signaling strategy design\. Moreover, online structured learning of connected vehicle’s lane choices under Bayesian persuasion outperforms existing approaches for lane selection\.

## IIntroduction

Intelligent traffic routing, which integrates real\-time data into vehicle routing recommendations, has emerged as a promising approach to enhance adaptability and responsiveness to dynamic traffic challenges\. Connected vehicles using an intelligent leader\-based architecture offer a state\-of\-the\-art solution for traffic optimization \- refer to Fig\.[4](https://arxiv.org/html/2607.13576#S6.F4)for an example\. Here, the lead vehicle equipped with real\-time traffic data can coordinate connected vehicles through speed modulation, route guidance, and platoon formation for efficient traffic management\[[50](https://arxiv.org/html/2607.13576#bib.bib217)\]\. This facilitates intelligent transportation networks, where the lead vehicle and the connected vehicle continuously collaborate through real\-time interaction and learning\. Conventional approaches for interactive driving face the critical challenge of harmonising decisions: — the connected vehicle seeks the shortest path, while the lead vehicle prioritizes global traffic flow and congestion avoidance, requiring coordination in real\-time routing\.

To address this challenge, this paper adopts the selective information disclosure framework of Bayesian persuasion to provide route guidance to individual vehicles while ensuring efficient traffic flow\. Bayesian persuasion is a strategic framework in which a principal influences an agent’s actions by selectively revealing information through signals\. This framework has found applications in several domains involving information asymmetry, including traffic routing\[[13](https://arxiv.org/html/2607.13576#bib.bib1)\], recommendation systems\[[29](https://arxiv.org/html/2607.13576#bib.bib48)\], grading systems in educational settings\[[6](https://arxiv.org/html/2607.13576#bib.bib33)\], queuing\[[24](https://arxiv.org/html/2607.13576#bib.bib49)\], etc\.

In the sequential decision\-making framework of Bayesian persuasion for intelligent interactive driving, illustrated in Fig\. 1, both principal and agent observe the environmental state, while an additional hidden external state is known only to the principal\. In the context of persuasive driving, this hidden state may represent real\-time traffic conditions ahead\. The principal strategically reveals information about the hidden state through signals—alerts about the traffic ahead or possible route selections—to influence the agent’s actions\. A signaling strategy is a probabilistic mapping from the hidden external state and the agent’s state to a distribution over signals\. A simple example is a direct revelation signaling strategy; however, directly revealing real\-time traffic information may prompt many vehicles to choose the same route, ultimately causing congestion \[8\]\. This underscores the need for carefully designed signaling strategies that guide routing decisions while maintaining efficient traffic flow\.

Accordingly, the principal controls the flow of information by selectively disclosing these external states, aiming to align the agent’s action with its objectives\. The agent, on the other hand, faces a sequential decision\-making problem and chooses actions based on its preferences\. A short\-sighted agent maximizes the expected immediate reward, while a far\-sighted agent takes into account the future state transitions and signal disclosures\. As a result, the principal must effectively solve a nested planning problem — choosing signaling strategy while anticipating how the agent will respond to it\. In general, the design of signaling strategy while considering a far\-sighted agent is NP\-Hard\[[13](https://arxiv.org/html/2607.13576#bib.bib1)\]\. Additionally, in real\-world settings, both the principal and agent may not have complete knowledge of their reward and transition dynamics\. Hence, we propose an online learning framework in which we model a reinforcement learning agent that approximately best responds\.

![Refer to caption](https://arxiv.org/html/2607.13576v1/x1.png)Figure 1:The schematic representation of intelligent interactive driving based on Bayesian persuasion is illustrated in Section[III](https://arxiv.org/html/2607.13576#S3)\. Here, both the principal \(intelligent lead vehicle\) and the agent \(connected vehicle\) are aware of the vehicle’s state,ss\. An external state,θ\\theta\(real\-time traffic\), known to the principal but unknown to the agent, influences the decision outcome\. The principal uses this information to persuade the agent through a signalgg, guiding actions toward its objectives\. Given the signaling strategy, the agent acts on its posterior belief aboutθ\\theta, incurring an immediate reward and inducing a stochastic transition to the next state\.This paper considers a monotonic agent \- an agent that responds to ‘higher’ signals by choosing ‘higher’ actions, inducing an ordered and predictable response to signal disclosures\. Such behaviour is natural in many practical settings— e\.g\., transmission control, sensor scheduling, traffic routing — where stronger signals lead agents to better actions\[[28](https://arxiv.org/html/2607.13576#bib.bib219)\],\[[36](https://arxiv.org/html/2607.13576#bib.bib206)\]\. Many real\-world applications—such as credit rating, queuing, traffic routing — where Bayesian persuasion is widely used, often exhibit monotonic threshold policies, such as assigning higher ranks to higher\-quality firms\[[14](https://arxiv.org/html/2607.13576#bib.bib3)\]or increasing travelling speed as the road traffic improves\[[36](https://arxiv.org/html/2607.13576#bib.bib206)\]\.

We propose an online structured reinforcement learning framework to synthesize computationally efficient signaling strategy that is*persuasive*for a far\-sighted agent\. The main contributions of the paper are:

1. 1\.Monotonic Agent Policy Learning \(MAPL\): We assume an agent with monotonic policy that approximately best responds and propose a structured reinforcement learning algorithm that utilizes the monotonic structure of the agent policy\. Algorithm 1 \(MAPL\) enables faster online learning of agent policy in large state spaces while achieving higher average reward\.
2. 2\.Characterisation of principal’s Q factors:Theorem 1 gives sufficient conditions on the Bayesian persuasion model to ensure supermodular Q value function of the principal, considering a monotonic agent\.
3. 3\.Structural results for persuasive signaling strategy:We present sufficient conditions that ensure the persuasiveness of the signaling strategy for a monotonic agent\. Specifically, Theorem 2 characterizes the principal’s signaling strategy, while Theorem 3 derives sufficient conditions on the agent’s reward function which ensure that the designed signaling strategy is persuasive for a monotonic agent\.
4. 4\.Supermodular Q learning for Principal \(SQP\): We propose Algorithm 2 \(SQP\), which leverages the supermodular structure of the principal’s Q\-value to synthesize computationally efficient signaling strategy which is persuasive for a monotonic agent\. Furthermore, Algorithm 3 presents the online reinforcement learning framework for the repeated interaction of the principal with approximately best responding agent in dynamic Bayesian persuasion\.
5. 5\.Case study of Bayesian persuasive driving: We illustrate\(i\)the computational efficiency of the online learning of the connected vehicle’s policy using MAPL compared to conventional dynamic vehicle routing recommendations,\(ii\)cost effectiveness as well as computational efficiency of the proposed structured reinforcement learning framework in the design of signaling strategy for the lead vehicle’s intelligent interactive driving against existing methodologies\[[46](https://arxiv.org/html/2607.13576#bib.bib74)\],\[[13](https://arxiv.org/html/2607.13576#bib.bib1)\],\[[3](https://arxiv.org/html/2607.13576#bib.bib75)\]\.

*To the best of our knowledge, this is the first work that explores the structural results for online reinforcement learning in dynamic Bayesian persuasion framework, with applications in intelligent traffic routing, transmission control, credit rating, sensor scheduling, and queuing\.*

## IIRelated Work

Interactive driving approaches that anticipate and adapt to surrounding vehicle’s lane\-change behaviours have gained increasing attention in intelligent traffic management\[[37](https://arxiv.org/html/2607.13576#bib.bib54)\]\. In this work, we focus on*lane\-based routing*, where decisions are made at the granularity of individual lanes, enabled by advances in vehicle\-to\-vehicle \(V2V\) communication, real\-time traffic sensing, and precize localization\. This fine\-grained control facilitates coordinated lane assignment and platoon formation, allowing a well\-informed lead vehicle with a global view of traffic conditions to guide connected vehicles into high\-speed formations\. Such coordination improves traffic throughput, enhances energy efficiency through reduced aerodynamic drag, and promotes road safety via synchronized vehicle behavior\[[50](https://arxiv.org/html/2607.13576#bib.bib217)\]\.

##### Existing lane\-changing approaches

Classical lane change approaches for autonomous vehicles rely primarily on rule\-based decisions that evaluate current traffic conditions against predefined safety constraints, such as minimum gap distance and time\-to\-collision thresholds\[[48](https://arxiv.org/html/2607.13576#bib.bib224)\]\. However, these methods depend on the instantaneous traffic state rather than anticipating future traffic evolution\. Model Predictive Control \(MPC\) addresses this limitation by optimising vehicle trajectories over a finite prediction horizon while satisfying the dynamic and safety constraints\[[52](https://arxiv.org/html/2607.13576#bib.bib218)\]\. However, its performance is highly sensitive to model accuracy and becomes computationally demanding in complex traffic environments, limiting its practicality for real\-time deployment\.

##### Online reinforcement learning approaches

Deep neural networks have been used to model complex lane\-changing behaviour\[[45](https://arxiv.org/html/2607.13576#bib.bib222)\],\[[52](https://arxiv.org/html/2607.13576#bib.bib218)\]\. However, these methods typically require large\-scale training data and substantial computational resources\[[52](https://arxiv.org/html/2607.13576#bib.bib218)\]\. Furthermore, recent advances have shifted toward online reinforcement learning methods that learn lane choices through repeated environmental interactions\[[1](https://arxiv.org/html/2607.13576#bib.bib215)\]\. However, in multi\-vehicle settings, fully shared information can paradoxically lead to traffic congestion\[[26](https://arxiv.org/html/2607.13576#bib.bib47)\]\. This has motivated the exploration of strategic information revealing framework, with Bayesian persuasion emerging as a promising approach for intelligent traffic management\[[37](https://arxiv.org/html/2607.13576#bib.bib54)\]\.

##### Bayesian persuasive driving

Bayesian persuasion provides a strategic framework in which a lead vehicle \(the principal\) selectively reveals real\-time traffic information to influence the lane selections of a connected vehicle \(the agent\)\. In the context of Bayesian persuasive driving, it is typically assumed that the principal has complete prior knowledge of the environment, allowing signaling strategies to be precomputed\[[13](https://arxiv.org/html/2607.13576#bib.bib1)\]\. Moreover, the signaling strategy of the lead vehicle as well as the posterior belief of the connected vehicle are assumed to be Gaussian distributed for computational tractability\. In addition, a simpler agent model is assumed without stochastic state transitions\. In this paper, we consider the online reinforcement learning setting, in which the principal as well as the agent are unaware of the reward, the state transition dynamics, and the prior belief over external states, which makes the design of signaling strategy challenging\.

##### Myopic agent assumption

Furthermore, for computational tractability of design of signaling strategy, numerous works restrict attention to myopic agent, which optimizes its immediate rewards without considering future transitions and signaling disclosures\. For instance,\[[13](https://arxiv.org/html/2607.13576#bib.bib1)\]designs signaling strategies under a fully known model with a myopic agent assumption\. More recently,\[[46](https://arxiv.org/html/2607.13576#bib.bib74)\]extends this line of work to an online reinforcement setting, where the principal must simultaneously learn the prior distribution, reward function, and transition dynamics\. However, there is growing interest toward dynamic Bayesian persuasion, in which the principal and agent interact repeatedly over time\[[13](https://arxiv.org/html/2607.13576#bib.bib1)\],\[[26](https://arxiv.org/html/2607.13576#bib.bib47)\],\[[12](https://arxiv.org/html/2607.13576#bib.bib51)\], which we consider in this paper\.

##### Dynamic Bayesian Persuasion

Determining the optimal signaling strategy for dynamic persuasion, assuming the optimum policy of the agent is generally NP\-Hard\[[13](https://arxiv.org/html/2607.13576#bib.bib1)\],\[[46](https://arxiv.org/html/2607.13576#bib.bib74)\]\. To address farsightedness, under the assumption of model dynamics,\[[5](https://arxiv.org/html/2607.13576#bib.bib198)\]employs history\-dependent strategies like promise form schemes, which ensure future reward promises\.\[[4](https://arxiv.org/html/2607.13576#bib.bib202)\]explores online learning framework in which the principal, without any knowledge of the prior belief of the agent about the hidden external state, repeatedly interacts with the far\-sighted agent and gradually learns the persuasive signaling strategy\. Moreover, it assumes that all other underlying model parameters are known and the agent possesses perfect recall of memory\. This paper considers a far\-sighted agent and develops computationally and memory\-efficient signaling strategy design as compared to\[[4](https://arxiv.org/html/2607.13576#bib.bib202)\]\.

##### Online Bayesian persuasion for large state spaces

Online learning of signaling strategy in large state, external state, action, and signal spaces is often computationally intractable due to the vast exploration domain\. Existing literature on dynamic Bayesian persuasion in continuous spaces considers a simplified model with myopic agent\[[37](https://arxiv.org/html/2607.13576#bib.bib54)\], linear reward, and transition dynamics\[[46](https://arxiv.org/html/2607.13576#bib.bib74)\]assumptions\. However, we consider a far\-sighted agent under general reward and transition dynamics\. In the context of dynamic Bayesian persuasion, we explore structural results for faster online learning of both agent’s and principal’s strategies in large state spaces\.

##### Structural results

Many real world applications naturally satisfy sufficient conditions that guarantee the optimality of monotonic policies\[[19](https://arxiv.org/html/2607.13576#bib.bib23)\],\[[47](https://arxiv.org/html/2607.13576#bib.bib78)\],\[[2](https://arxiv.org/html/2607.13576#bib.bib164)\]\. In the context of Bayesian persuasion, monotonic signaling—where higher actions are recommended under higher external states—has found applications in diverse areas such as credit rating and traffic routing\. In such settings, agents often exhibit threshold policies, such as assigning higher ranks\[[14](https://arxiv.org/html/2607.13576#bib.bib3)\]to higher\-quality firms or increasing speed as the road traffic improves\[[34](https://arxiv.org/html/2607.13576#bib.bib211)\]\. Similarly,\[[10](https://arxiv.org/html/2607.13576#bib.bib204)\]utilizes supermodularity as a linear constraint on Q factors to improve learning\. Additionally,\[[11](https://arxiv.org/html/2607.13576#bib.bib203)\]characterizes signaling strategy when the principal’s utility function is supermodular\. However, these works restrict the interaction between the principal and agent in one\-shot settings\.

##### Novelty of our work

This paper utilizes structural results for online reinforcement learning in dynamic Bayesian persuasion\. Our approach provides a joint characterisation of both the agent’s policy and the principal’s value function\. Additionally, structured learning ensures faster convergence\[[7](https://arxiv.org/html/2607.13576#bib.bib209)\]in high\-dimensional or model\-free dynamic environments\. In particular, the monotonic characterisation of the agent’s policy substantially reduces computational complexity and memory requirements by restricting the search space to policies that satisfy the monotone structure\. This enables faster online algorithms for computing the agent’s optimal policy\. Furthermore, by exploiting the supermodularity of the principal’s Q\-value function, we develop a structured online reinforcement learning framework that synthesizes computationally efficient signaling strategy while ensuring persuasiveness for a monotonic agent\.

### Outline

Section[III](https://arxiv.org/html/2607.13576#S3)presents the Bayesian persuasion framework and formulates the Bellman dynamic programming equation to compute the optimal policy of the agent and principal’s signaling strategy\. Section[IV](https://arxiv.org/html/2607.13576#S4)presents the main result, Theorem[1](https://arxiv.org/html/2607.13576#Thmtheorem1), which gives the sufficient conditions for the supermodular structure of the Q value function of the principal for a farsighted monotonic agent\. Additionally, Theorem[2](https://arxiv.org/html/2607.13576#Thmtheorem2)characterizes the structural results on the principal’s signaling strategy, while Theorem[3](https://arxiv.org/html/2607.13576#Thmtheorem3)derives the sufficient conditions on the agent’s reward function that ensure that the designed signaling strategy is persuasive for a monotonic agent\. Section[V](https://arxiv.org/html/2607.13576#S5)presents a structured reinforcement learning framework with computationally efficient algorithms, MAPL for structured learning of the agent’s monotonic policy, and SQP for supermodular Q learning for the principal’s action value, respectively\. Section[VI](https://arxiv.org/html/2607.13576#S6)illustrates the practical applicability of the proposed model assumptions for Bayesian persuasive driving\. The numerical results highlight the usefulness of structural results in enhancing computational efficiency in learning the optimal policy of the connected vehicle as well as the signaling strategy of the intelligent lead vehicle\. Concluding remarks are offered in Section[VII](https://arxiv.org/html/2607.13576#S7)\. The Appendix at the end contains supporting lemmas, assumptions, proofs of algorithms, and theorems\.

## IIIProblem formulation

![Refer to caption](https://arxiv.org/html/2607.13576v1/x2.png)Figure 2:Dynamic framework of Bayesian persuasion in which the principal guides the sequential decision making of the agent by revealing the unknown external states through strategic signaling\.Fig\.[2](https://arxiv.org/html/2607.13576#S3.F2)illustrates the dynamic framework of Bayesian persuasion in which the principal guides the sequential decision making of the agent by revealing the unknown external states using a signaling strategy\. At each time step, the principal observes the external state of the environment and samples a signal according to the committed signaling strategy\. Upon receiving the signal, the agent updates its posterior belief about the unknown state and selects an action by maximizing its expected reward\. This accounts for both immediate payoff and state transitions, which are observed by the principal and agent\. Subsequently, a new external state of the environment evolves, which is known only to the principal\. This dynamic repeated interaction between the principal and agent continues over time\. The goal of the agent is to learn to choose actions that maximize its expected average reward under the committed signaling strategy of the principal\. On the other hand, the objective of the principal is to design signaling strategy to maximize its average reward while ensuring that the persuaded learning agent also benefits from the signal disclosures\.

### III\-ASystem Model

The model can be represented by the tupleM=⟨𝒮,Θ,𝒢,𝒜,P,\(μs\),\(πs,θ\),RA,RP⟩M~=~\\langle\\ \\mathcal\{S\},\\Theta,\\mathcal\{G\},\\mathcal\{A\},P,\(\\mu\_\{s\}\),\(\\pi\_\{s,\\theta\}\),R\_\{A\},R\_\{P\}\\ \\rangle

- •State \(s∈𝒮=\{0,1,2\.\.S\}s\\in\\mathcal\{S\}=\\\{0,1,2\.\.S\\\}\):State refers to the known environment of the agent\. The state of the agent is also observable by the principal\. In the context of traffic routing, it represents measurable factors such as vehicle’s location, velocity, acceleration, or energy consumption\[[18](https://arxiv.org/html/2607.13576#bib.bib55)\]\.
- •External State \(θ∈Θ=\{0,1,2\.\.O\}\\theta\\in\\Theta=\\\{0,1,2\.\.O\\\}\):It denotes hidden factors which are not directly known to the agent but observed by the principal, such as road accidents, construction works, which have a significant impact on the agent’s decision outcomes\.
- •Signal \(g∈𝒢=\{1,2,…,G\}g\\in\\mathcal\{G\}=\\\{1,2,\\dots,G\\\}\):The signal space denotes the set of signals, which the principal sends to the agent based on the observed state and external state\. In the context of traffic routing, signaling may be binary, discrete, or continuous—for example, indicating whether to switch lanes, alerting about upcoming congestion, recommending an appropriate driving speed, or specifying the current traffic condition\[[37](https://arxiv.org/html/2607.13576#bib.bib54)\]\.
- •Action \(a∈𝒜=\{1,2,…,A\}a\\in\\mathcal\{A\}=\\\{1,2,\\dots,A\\\}\):Action space refers to the set of possible actions that the agent can choose based on the signal received\. For example, in traffic routing, accelerating, decelerating, selection of different lanes, etc\., are taken as actions\[[18](https://arxiv.org/html/2607.13576#bib.bib55)\]\.
- •Prior belief \(μ​\(s,θ\)=P​\(θ∣s\)\\mu\(s,\\theta\)=P\(\\theta\\mid s\)\):It is the conditional probability density of the external stateθ\\thetagiven the statess\. It serves as the prior belief distribution of the external stateθ\\thetain statess, known to the agent before any interaction with the principal, and captures the correlation between state and external state\.
- •Signaling strategy \(πs,θ​\(g\)=P​\(g∣s,θ\)\\pi\_\{s,\\theta\}\(g\)=P\(g\\mid s,\\theta\)\):It defines the conditional probability distribution of signals used by the principal, based on the joint realization ofssandθ\\theta\. In our model, the signaling strategy is assumed to be Markovian\[[13](https://arxiv.org/html/2607.13576#bib.bib1)\]in which the principal samples a signalggbased on the current statessand external stateθ\\theta\.
- •Reward \(RA​\(s,θ,a\),RP​\(s,θ,a\)R\_\{A\}\(s,\\theta,a\),R\_\{P\}\(s,\\theta,a\)\):It denotes the immediate reward to the agent and principal, respectively, determined by the agent’s action as well as state and external state\.
- •State transition \(P​\(s′∣s,a\)P\(s^\{\\prime\}\\mid s,a\)\):It specifies the stationary probability distribution over the next states′s^\{\\prime\}, given the current statessand the actionaataken by the agent\.

Classical MDP settings involve an agent that either knows or learns the environment through direct interaction, whereas in Bayesian persuasion, the agent’s action is influenced by the principal’s signaling strategy\.

### III\-BAgent’s average reward policy under Bayesian persuasion

##### Agent’s belief over the external state,θ\\theta

The agent does not have direct knowledge of the external stateθ\\theta\. Upon observing a signalgg, while being in statess, the agent updates its belief aboutθ\\thetausing Bayes’ rule, resulting in the posterior belief,bs,g​\(θ\)b\_\{s,g\}\(\\theta\), given by

bs,g​\(θ\)\\displaystyle b\_\{s,g\}\(\\theta\)=P​\(θ∣s,g\)=πs,θ​\(g\)⋅μs​\(θ\)∑θ∈Θπs,θ​\(g\)⋅μs​\(θ\)\\displaystyle=P\(\\theta\\mid s,g\)=\\frac\{\\pi\_\{s,\\theta\}\(g\)\\cdot\\mu\_\{s\}\(\\theta\)\}\{\\sum\_\{\\theta\\in\\Theta\}\\pi\_\{s,\\theta\}\(g\)\\cdot\\mu\_\{s\}\(\\theta\)\}\(1\)At each instanttt, based on the signaling strategy, the principal samples signalgtg\_\{t\}, depending on the current statests\_\{t\}, and external stateθt\\theta\_\{t\}\. Since the external state is unobserved by the agent, the expected immediate reward,r~A\\tilde\{r\}\_\{A\}, obtained on taking action,ata\_\{t\}, can be computed by taking the expectation with respect to the posterior belief,bst,gt​\(θt\)b\_\{s\_\{t\},g\_\{t\}\}\(\\theta\_\{t\}\), which is given by

r~A​\(st,gt,at\)=∑θ∈Θbst,gt​\(θt\)⋅RA​\(st,θt,at\)\\displaystyle\\tilde\{r\}\_\{A\}\(s\_\{t\},g\_\{t\},a\_\{t\}\)=\\sum\_\{\\theta\\in\\Theta\}b\_\{s\_\{t\},g\_\{t\}\}\(\\theta\_\{t\}\)\\cdot R\_\{A\}\(s\_\{t\},\\theta\_\{t\},a\_\{t\}\)\(2\)A policy,ρ:𝒮×𝒢→𝒜\\rho:\\mathcal\{S\}\\times\\mathcal\{G\}\\rightarrow\\mathcal\{A\}, is the decision rule that determines the actionaato be taken at a particular statesson receiving a signalgg\. In this paper, the objective of the agent is to maximize expected average reward from the starting statesswith signalggexpressed as

R~A=limt→∞1t𝔼\[∑t=0∞r~A\(st,gt,at\)\|s0=s,g0=g\]\\displaystyle\\tilde\{R\}\_\{A\}\\\!=\\lim\_\{t\\xrightarrow\{\}\\infty\}\\frac\{1\}\{t\}\\mathbb\{E\}\\biggl\[\\sum\_\{t=0\}^\{\\infty\}\\tilde\{r\}\_\{A\}\(s\_\{t\},g\_\{t\},a\_\{t\}\)\|s\_\{0\}\\\!=\\\!s,g\_\{0\}\\\!=\\\!g\\biggl\]\(3\)Under the ergodic Markov chain assumption of the agent’s policy, this limit exists and is independent of the initial state\(s,g\)\(s,g\)\[[39](https://arxiv.org/html/2607.13576#bib.bib17)\]\. Here, the expectation is taken with respect to state transition dynamics, signaling strategy, and prior belief distribution\. Agent’s actionaain statessresults in an expected immediate reward as well as a stochastic transition to a new states′s^\{\\prime\}\. The new external state is given byθ′∼μs′\(\.\)\\theta^\{\\prime\}\\sim~\\mu\_\{s^\{\\prime\}\}\(\.\)\. The principal who is aware of the realisation ofθ′\\theta^\{\\prime\}, samples signalg′∼πs′,θ′\(\.\)g^\{\\prime\}\\sim\\pi\_\{s^\{\\prime\},\\theta^\{\\prime\}\}\(\.\)and transmits it to the agent\. This transition,\(s,g\)\(s,g\)to\(s′,g′\)\(s^\{\\prime\},g^\{\\prime\}\), happens with probability

P​\(\(s,g\),\(s′,g′\),a\)=P​\(s,s′,a\)⋅∑θ′∈Θμs′​\(θ′\)⋅πs′,θ′​\(g′\)\\displaystyle P\(\(s,g\),\(s^\{\\prime\},g^\{\\prime\}\),a\)=P\(s,s^\{\\prime\},a\)\\cdot\\sum\_\{\\theta^\{\\prime\}\\in\\Theta\}\\mu\_\{s^\{\\prime\}\}\(\\theta^\{\\prime\}\)\\cdot\\pi\_\{s^\{\\prime\},\\theta^\{\\prime\}\}\(g^\{\\prime\}\)VA​\(s,g\)V\_\{A\}\(s,g\)is the differential or relative value function of the agent at statesswith signalgg\. It denotes the reward obtained relative to the average reward while following policyρ\\rhostarting from the state–signal pair\(s,g\)\(s,g\)\. The corresponding action value functionQA​\(s,g,a\)Q\_\{A\}\(s,g,a\), represents the expected relative reward of taking actionaain\(s,g\)\(s,g\)under policyρ\\rho, and can be considered as the sum of the expected immediate reward in statessand the expected value of the subsequent state–signal pair\(s′,g′\)\(s^\{\\prime\},g^\{\\prime\}\)\. The Bellman equation for the agent’s policy is given by

R~A\+QA​\(s,g,a\)=r~A​\(s,a\)\+\\displaystyle\\tilde\{R\}\_\{A\}\+Q\_\{A\}\(s,g,a\)=\\tilde\{r\}\_\{A\}\(s,a\)\+\(4\)∑s′∈𝒮P​\(s′∣s,a\)​∑g′∈𝒢∑θ′∈Θμs′​\(θ′\)⋅πs′,θ′​\(g′\)⋅VA​\(s′,g′\)\\displaystyle\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}P\(s^\{\\prime\}\\mid s,a\)\\sum\_\{g^\{\\prime\}\\in\\mathcal\{G\}\}\\sum\_\{\\theta^\{\\prime\}\\in\\Theta\}\\mu\_\{s^\{\\prime\}\}\(\\theta^\{\\prime\}\)\\cdot\\pi\_\{s^\{\\prime\},\\theta^\{\\prime\}\}\(g^\{\\prime\}\)\\cdot V\_\{A\}\(s^\{\\prime\},g^\{\\prime\}\)LetQA∗,VA∗,ρ∗,RA∗Q\_\{A\}^\{\*\},V\_\{A\}^\{\*\},\\rho^\{\*\},R\_\{A\}^\{\*\}be the optimal action value, value function, policy, and average reward of the agent\. The Bellman optimality equation can be written as

RA∗\+QA∗​\(s,g,a\)=r~A​\(s,a\)\+\\displaystyle R\_\{A\}^\{\*\}\+Q\_\{A\}^\{\*\}\(s,g,a\)=\\tilde\{r\}\_\{A\}\(s,a\)\+\(5\)∑s′∈𝒮P​\(s′∣s,a\)​∑g′∈𝒢∑θ′∈Θμs′​\(θ′\)⋅πs′,θ′​\(g′\)⋅maxa∈A⁡QA∗​\(s′,g′,a\)\\displaystyle\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}P\(s^\{\\prime\}\\mid s,a\)\\sum\_\{g^\{\\prime\}\\in\\mathcal\{G\}\}\\sum\_\{\\theta^\{\\prime\}\\in\\Theta\}\\mu\_\{s^\{\\prime\}\}\(\\theta^\{\\prime\}\)\\cdot\\pi\_\{s^\{\\prime\},\\theta^\{\\prime\}\}\(g^\{\\prime\}\)\\cdot\\max\_\{a\\in A\}Q\_\{A\}^\{\*\}\(s^\{\\prime\},g^\{\\prime\},a\)ρ∗​\(s,g\)=arg⁡maxa∈𝒜⁡QA∗​\(s,g,a\)\\displaystyle\\rho^\{\*\}\(s,g\)=\\arg\\max\_\{a\\in\\mathcal\{A\}\}Q\_\{A\}^\{\*\}\(s,g,a\)\(6\)VA∗​\(s,g\)=maxa∈𝒜⁡QA∗​\(s,g,a\)\\displaystyle V\_\{A\}^\{\*\}\(s,g\)=\\max\_\{a\\in\\mathcal\{A\}\}Q\_\{A\}^\{\*\}\(s,g,a\)In the next section, we demonstrate how this agent’s policy is taken into account in the principal’s signaling strategy design\.

### III\-CSignaling strategy of the principal for Bayesian persuasion

Bayesian persuasion differs from the classical MDP in the fact that the actions are not taken directly by the principal, but by the agent under the influence of the information shared\. The goal of the principal is to strategically reveal the unknown external state to guide the agent’s action to align with its objectives\. In this paper, we consider a Markovian signaling strategy that maps the external hidden state for each of the agent’s state to a probability distribution of signals, i\.e\.,π:𝒮×Θ→Δ​𝒢\.\\pi:\\mathcal\{S\}\\times\\Theta\\rightarrow\\Delta\\mathcal\{G\}\.Moreover, in the context of reinforcement learning, where both the principal and agent learn together, probabilistic recommendations reduce the risk that signals are completely ignored by the agent\[[22](https://arxiv.org/html/2607.13576#bib.bib220)\]\. At each step, given the statessand external stateθ\\theta, the principal samples a signal,g∼πs,θ\(\.\)g\\sim\\pi\_\{s,\\theta\}\(\.\)\. The objective of the principal is to select a signaling strategy which maximizes its long run average reward from a starting state \(ss,θ\\theta\) while taking into account the agent’s action given by \([6](https://arxiv.org/html/2607.13576#S3.E6)\) expressed as

R~P=limt→∞1t𝔼π\[∑t=0∞RP\(st,θt,at\)\|s0=s,θ0=θ\]\\displaystyle\\tilde\{R\}\_\{P\}=\\lim\_\{t\\xrightarrow\{\}\\infty\}\\frac\{1\}\{t\}\\mathbb\{E\}\_\{\\pi\}\\biggl\[\\sum\_\{t=0\}^\{\\infty\}R\_\{P\}\(s\_\{t\},\\theta\_\{t\},a\_\{t\}\)\|s\_\{0\}\\\!=\\\!s,\\theta\_\{0\}\\\!=\\\!\\theta\\biggl\]\(7\)
Here, the expectation is taken with respect to the agent’s policy, state transition dynamics, signaling strategy, and prior belief distribution\. LetVP​\(s,θ\)V\_\{P\}\(s,\\theta\)be the differential value function of the principal in statessassociated withθ\\theta, which gives the reward obtained relative to the average rewardR~P\\tilde\{R\}\_\{P\}, under signaling strategyπ\\pistarting from\(s,θ\)\(s,\\theta\)\. The transition probability from\(s,θ\)\(s,\\theta\)to\(s′,θ′\)\(s^\{\\prime\},\\theta^\{\\prime\}\)is given by:

P​\(\(s,θ\),\(s′,θ′\),a′\)=P​\(s,s′,a′\)⋅μs′​\(θ′\)\\displaystyle P\(\(s,\\theta\),\(s^\{\\prime\},\\theta^\{\\prime\}\),a^\{\\prime\}\)=P\(s,s^\{\\prime\},a^\{\\prime\}\)\\cdot\\mu\_\{s^\{\\prime\}\}\(\\theta^\{\\prime\}\)QP​\(s,θ,g\)Q\_\{P\}\(s,\\theta,g\)is the signal value of the principal in\(s,θ\)\(s,\\theta\)on sending a signalggunder the signaling strategyπ\.\\pi\.It can be expressed as the sum of the principal’s immediate reward based on the agent’s action prompted by the signalggin\(s,θ\)\(s,\\theta\)and the expected value on the transition to the future state\(s′,θ′\)\(s^\{\\prime\},\\theta^\{\\prime\}\)\.

We consider a learning agent that approximately best responds\[[8](https://arxiv.org/html/2607.13576#bib.bib221)\], selecting stochastic actions that are near\-optimal with high probability during learning\. Letρ~​\(a\|s,g\)\\tilde\{\\rho\}\(a\|s,g\)be the probability that the learning agent takes an actionaa, on receiving a signalg\.g\.The Q value function of the principal with an average rewardR~P\\tilde\{R\}\_\{P\}can be formulated as

R~P\+QP​\(s,θ,g\)=∑a′∈𝒜ρ~​\(a′\|s,g\)⋅RP​\(s,θ,a′\)\+\\displaystyle\\tilde\{R\}\_\{P\}\+Q\_\{P\}\(s,\\theta,g\)=\\sum\_\{a^\{\\prime\}\\in\\mathcal\{A\}\}\\tilde\{\\rho\}\(a^\{\\prime\}\|s,g\)\\cdot R\_\{P\}\(s,\\theta,a^\{\\prime\}\)\+\(8\)∑a′∈𝒜ρ~​\(a′\|s,g\)⋅∑s′∈𝒮P​\(s,s′,a′\)​∑θ′∈Θμs′​\(θ′\)⋅VP​\(s′,θ′\)\\displaystyle\\sum\_\{a^\{\\prime\}\\in\\mathcal\{A\}\}\\tilde\{\\rho\}\(a^\{\\prime\}\|s,g\)\\cdot\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}P\(s,s^\{\\prime\},a^\{\\prime\}\)\\sum\_\{\\theta^\{\\prime\}\\in\\Theta\}\\mu\_\{s^\{\\prime\}\}\(\\theta^\{\\prime\}\)\\cdot V\_\{P\}\(s^\{\\prime\},\\theta^\{\\prime\}\)VP​\(s,θ\)=∑g∈𝒢QP​\(s,θ,g\)⋅πs,θ​\(g\)\\displaystyle V\_\{P\}\(s,\\theta\)=\\sum\_\{g\\in\\mathcal\{G\}\}Q\_\{P\}\(s,\\theta,g\)\\cdot\\pi\_\{s,\\theta\}\(g\)

### III\-DPersuasive signaling : Incentive compatibility of the agent

The principal sends a signal that selectively discloses the privately observed external state, anticipating that the agent will take the signal into account and act in a way beneficial to the principal\. A simple signaling strategy is to reveal the true external state\. However, the direct revelation of external state, for example, real\-time traffic, may prompt connected vehicles to choose the same route and may finally lead to traffic congestion\[[26](https://arxiv.org/html/2607.13576#bib.bib47)\]\. Moreover, a key challenge in information design is that such communication can be effective only if the agent finds it profitable to follow the signal\. Hence, the signal cannot serve the principal’s interest solely; it must also improve the agent’s expected outcome so that adhering to it is rational from the agent’s perspective\.

##### Signaling strategy as action recommendation

Without loss of generality, a signaling strategy can restrict the number of signals equal to the receiver’s action set\[[13](https://arxiv.org/html/2607.13576#bib.bib1)\], i\.e\.,GA=\{ga:a∈𝒜\}G\_\{A\}=\\\{g\_\{a\}:a\\in\\mathcal\{A\}\\\}\. LetQA​\(s,θ,ga\)Q\_\{A\}\(s,\\theta,g\_\{a\}\)be the action value of the agent in\(s,θ\)\(s,\\theta\)while recommended by a signalgag\_\{a\}, for taking an actionaa, under the signaling strategyπ\.\\pi\.Thus, for the persuaded agent to benefit from strategic signaling of the principal, its expected average reward for the recommended action has to be greater as compared to other actions, i\.e\.,

QA​\(s,ga,a\)≥QA​\(s,ga,a′\),∀a,a′∈𝒜\\displaystyle Q\_\{A\}\(s,g\_\{a\},a\)\\geq Q\_\{A\}\(s,g\_\{a\},a^\{\\prime\}\),\\forall a,a^\{\\prime\}\\in\\mathcal\{A\}\(9\)Hence, in order to ensure a persuasive signaling strategy, the incentive compatibility of the agent is taken into account while maximizing the principal’s expected average reward\. Letβ​\(s,θ\)\\beta\(s,\\theta\)be the probability distribution of the initial state being\(s,θ\)\(s,\\theta\)\. The objective of the principal is to maximize the average reward given by \([7](https://arxiv.org/html/2607.13576#S3.E7)\) subject to the constraints in \([8](https://arxiv.org/html/2607.13576#S3.E8)\) and \([9](https://arxiv.org/html/2607.13576#S3.E9)\)\. Thus, the optimization of the principal can be formulated\[[20](https://arxiv.org/html/2607.13576#bib.bib18)\]as

minVP​∑s∈𝒮∑θ∈Θβ​\(s,θ\)⋅VP​\(s,θ\)\\displaystyle\{\\min\_\{V\_\{P\}\}\}\\sum\_\{s\\in\\mathcal\{S\}\}\\sum\_\{\\theta\\in\\Theta\}\\beta\(s,\\theta\)\\cdot V\_\{P\}\(s,\\theta\)\(10\)subject to

R~P\+QP​\(s,θ,ga\)≥∑a′∈𝒜ρ~​\(a′\|s,ga\)⋅RP​\(s,θ,a′\)\+\\displaystyle\\tilde\{R\}\_\{P\}\+Q\_\{P\}\(s,\\theta,g\_\{a\}\)\\geq\\sum\_\{a^\{\\prime\}\\in\\mathcal\{A\}\}\\tilde\{\\rho\}\(a^\{\\prime\}\|s,g\_\{a\}\)\\cdot R\_\{P\}\(s,\\theta,a^\{\\prime\}\)\+∑a′∈𝒜ρ~​\(a′\|s,ga\)⋅∑s′∈𝒮P​\(s,s′,a′\)​∑θ′∈Θμs′​\(θ′\)⋅VP​\(s′,θ′\)\\displaystyle\\sum\_\{a^\{\\prime\}\\in\\mathcal\{A\}\}\\tilde\{\\rho\}\(a^\{\\prime\}\|s,g\_\{a\}\)\\cdot\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}P\(s,s^\{\\prime\},a^\{\\prime\}\)\\sum\_\{\\theta^\{\\prime\}\\in\\Theta\}\\mu\_\{s^\{\\prime\}\}\(\\theta^\{\\prime\}\)\\cdot V\_\{P\}\(s^\{\\prime\},\\theta^\{\\prime\}\)QA​\(s,ga,a\)≥QA​\(s,ga,a′\),∀a,a′∈𝒜\\displaystyle Q\_\{A\}\(s,g\_\{a\},a\)\\geq Q\_\{A\}\(s,g\_\{a\},a^\{\\prime\}\),\\forall a,a^\{\\prime\}\\in\\mathcal\{A\}When the agent is far\-sighted, its optimal action does not only depend on the expected immediate reward but also on future state transitions and signal disclosures\. Consequently, the principal’s value function depends on the signaling strategy as well as the induced agent policy, which is the solution of a dynamic optimisation as illustrated in \([5](https://arxiv.org/html/2607.13576#S3.E5)\)\. This nested dependence makes the principal’s optimization nonlinear, and the design of signaling strategy NP\-Hard\[[13](https://arxiv.org/html/2607.13576#bib.bib1)\]\. Hence, in the next section, we explore the structural results to compute efficiently the optimal agent policy as well as the signaling strategy\.

## IVStructural Results

Here, we investigate how structural results on the principal can be leveraged for the design of a persuasive signaling strategy for a far\-sighted agent\. We restrict attention to a monotone agent, which enables us to use the Monotone Likelihood Ratio \(MLR\) ordered signaling strategy, which is*persuasive*, as illustrated in the following subsection\.

### IV\-AMonotonic agent policy

The optimal policy of the agent in a Bayesian persuasion environment often exhibits monotone behaviour with respect to the state and signal space\. This is widely observed in various real\-world applications, including traffic routing\[[36](https://arxiv.org/html/2607.13576#bib.bib206)\], sensor scheduling\[[20](https://arxiv.org/html/2607.13576#bib.bib18)\], inventory management\[[39](https://arxiv.org/html/2607.13576#bib.bib17)\], transmission scheduling\[[2](https://arxiv.org/html/2607.13576#bib.bib164)\], etc\. The monotonic policy can be characterized as

ρ​\(s,g\)≤ρ​\(s′,g\),s≤s′,∀s,s′∈𝒮,g∈𝒢,\\displaystyle\\rho\(s,g\)\\leq\\rho\(s^\{\\prime\},g\),\\ s\\leq s^\{\\prime\},\\ \\forall s,s^\{\\prime\}\\in\\mathcal\{S\},\\ g\\in\\mathcal\{G\},\(11\)ρ​\(s,g\)≤ρ​\(s,g′\),g≤g′,∀s∈𝒮,g,g′∈𝒢\.\\displaystyle\\rho\(s,g\)\\leq\\rho\(s,g^\{\\prime\}\),\\ g\\leq g^\{\\prime\},\\ \\forall s\\in\\mathcal\{S\},\\ g,g^\{\\prime\}\\in\\mathcal\{G\}\.Subsequently, in the context of Bayesian persuasion, assuming monotonic agent policy, we characterize the structure of the principal’s Q value function for a persuasive signaling strategy as detailed below\.

### IV\-BSupermodularity of Q function and Incentive compatible signaling strategy design:

In this section, we explore structural properties of principal’s Q\-functions to ensure that the induced signaling strategy is incentive compatible for a monotonic agent\. The main result is Theorem[1](https://arxiv.org/html/2607.13576#Thmtheorem1), which gives sufficient conditions for supermodularity of the Q value of the principal with respect to the state and external state\. Consider the following assumptions on the Bayesian persuasion model:

1. \(A1\)RP​\(s,θ,a\)R\_\{P\}\(s,\\theta,a\)is increasing in the state of agentss, external stateθ\\theta, actionaataken by the agent\.
2. \(A2\)RP​\(s,θ,a\)R\_\{P\}\(s,\\theta,a\)is supermodular in\(s,a\)\(s,a\)for fixedθ\\thetaandRP​\(s,θ,a\)R\_\{P\}\(s,\\theta,a\)is supermodular in\(θ,a\)\(\\theta,a\)for fixedss\.
3. \(A3\)μsi​\(θ\)≤rμsj​\(θ\),∀i<j,\\mu\_\{\{s\_\{i\}\}\}\(\\theta\)\\leq\_\{r\}\\mu\_\{\{s\_\{j\}\}\}\(\\theta\),\\ \\forall i<j,i\.e\., prior belief of the external states of higher states MLR dominates the lower\.
4. \(A4\)P\(si,\.,a\)≤sP\(sj,\.,a\),∀s∈𝒮,a∈𝒜,i≤jP\(s\_\{i\},\.,a\)\\leq\_\{s\}P\(s\_\{j\},\.,a\),\\ \\forall s\\in\\mathcal\{S\},a\\in\\mathcal\{A\},i\\leq j, i\.e\., each state of the transition probability stochastically dominates the previous state\.
5. \(A5\)P​\(s,s′,a\+1\)P\(s,s^\{\\prime\},a\+1\)is tail sum supermodular ins,as,ain the sense∑s¯S\(P​\(s,s′,a\+1\)−P​\(s,s′,a\)\)\\sum\_\{\\bar\{s\}\}^\{S\}\(P\(s,s^\{\\prime\},a\+1\)\-P\(s,s^\{\\prime\},a\)\)is increasing ins,∀s¯∈𝒮s,\\ \\forall\\ \\bar\{s\}\\in\\mathcal\{S\}\.
6. \(A6\)The learning agent’s policy,ρ~​\(a\|s,g\)\\tilde\{\\rho\}\(a\|s,g\)stochastically dominates the previous state and signal spaces and is tail sum supermodular ins,gs,g\.

###### Theorem 1\.

Assuming that the Bayesian persuasion model satisfies \(A1\) to \(A6\), then there exists an optimal principal’s action value function which is supermodular in state and external space with respect to signals\.

The discussion of the assumptions is given in the Appendix[VII\-E](https://arxiv.org/html/2607.13576#Sx1.SS5)\. In the following section, we investigate how the signaling strategy of the principal can be made incentive compatible for a monotonic agent and state it as follows:

###### Theorem 2\.

When the principal commits to an MLR\-ordered signaling strategy, it ensures monotone, incentive\-compatible agent behaviour\.

MLR signaling results in the agent’s monotone belief about external states\. Since the agent updates posterior belief of unknown external state, based on the signal transmitted by the principal using Bayes’ rule—and MLR is preserved under Bayesian updating—the principal must commit to an MLR\-ordered signaling strategy to guarantee monotone, incentive\-compatible agent behaviour\. This is proved in detail in the Appendix[VII\-C](https://arxiv.org/html/2607.13576#Sx1.SS3)\. Next, we illustrate how the supermodular structure of the principal’s Q value guarantees monotone, incentive\-compatible agent behaviour in the following lemma\.

###### Lemma 1\.

Softmax strategy over the supermodular Q values of the principal ensures MLR ordered signaling strategy\.

We compute the signaling strategy as the softmax policy considered for the average reward setting as

π​\(g\|s,θ\)=exp⁡\(Q​\(s,θ,g\)/τ\)∑g′∈𝒢exp⁡\(Q​\(s,θ,g′\)/τ\)\\displaystyle\\pi\(g\|s,\\theta\)=\\frac\{\\exp\\left\(Q\(s,\\theta,g\)/\\tau\\right\)\}\{\\sum\_\{g^\{\\prime\}\\in\\mathcal\{G\}\}\\exp\\left\(Q\(s,\\theta,g^\{\\prime\}\)/\\tau\\right\)\}\(12\)whereτ\\tauis the temperature parameter controlling exploration\. It is proved in the Appendix[VII\-F](https://arxiv.org/html/2607.13576#Sx1.SS6)that the softmax strategy preserves MLR\.

### IV\-CStructural results for agent’s policy learning

In addition to MLR ordered signaling strategies committed by the principal, \(A3\) and \(A4\) satisfied by the Bayesian persuasion model, the following assumptions on the reward structure of the agent are required to ensure a monotone best response of the agent\.

1. \(B1\)RA​\(s,θ,a\)R\_\{A\}\(s,\\theta,a\)is increasing in state of agentss, external stateθ\\theta\.
2. \(B2\)RA​\(s,θ,a\)R\_\{A\}\(s,\\theta,a\)is supermodular in\(s,a\)\(s,a\)for fixedθ\\thetaandRA​\(s,θ,a\)R\_\{A\}\(s,\\theta,a\)is supermodular in\(θ,a\)\(\\theta,a\)for fixedss\.

Now, we state the existence of a monotone best response of the agent in the following theorem\.

###### Theorem 3\.

In a Bayesian persuasion model with an MLR signaling strategy, there exists a sufficient set of conditions \(B1\), \(B2\), \(A3\), \(A4\) by which the following hold simultaneously:

1. 1\.The agent’s action value function,QAQ\_\{A\}, is supermodular: QA​\(s,g,a\)​has increasing differences in​\(s,a\)​and​\(g,a\)Q\_\{A\}\(s,g,a\)\\text\{ has increasing differences in \}\(s,a\)\\text\{ and \}\(g,a\)
2. 2\.There exists a monotone agent policy: a∗:S×G→A​such that​a∗​\(s′,g′\)≥a∗​\(s,g\)\\displaystyle a^\{\*\}:S\\times G\\to A\\ \\textnormal\{such that\}\\ a^\{\*\}\(s^\{\\prime\},g^\{\\prime\}\)\\geq a^\{\*\}\(s,g\)∀s′≥s,g′≥g\\displaystyle\\forall s^\{\\prime\}\\geq s,g^\{\\prime\}\\geq g
3. 3\.The agent’s best response is monotone, and the signaling strategy is incentive compatible\. For everys,gs,gand every deviation from monotone action,a′∈𝒜a^\{\\prime\}\\in\\mathcal\{A\}, QA​\(s,g,a∗​\(s,g\)\)≥QA​\(s,g,a′\)\.Q\_\{A\}\(s,g,a^\{\*\}\(s,g\)\)\\geq Q\_\{A\}\(s,g,a^\{\\prime\}\)\.

Subsequently, in the following section, we utilize the structural results for a computationally efficient learning of a monotonic agent policy as well as a signaling strategy in dynamic Bayesian persuasion\.

## VStructured Learning

In this section, we present Algorithm 1 \(MAPL\), which exploits the monotonic structure for faster learning of the agent policy\. Then, Algorithm 2, Supermodular Q learning of Principal \(SQP\) utilizes the supermodular structure of the principal’s Q\-value to enable computationally efficient design of the signaling strategy\. Furthermore, Algorithm 3 presents an online reinforcement learning framework for signaling strategy that remains persuasive for a monotonic learning agent\.

### V\-AStructured policy learning of the monotone agent under Bayesian persuasion

Structure\-based learning leverages the threshold characteristics of the optimal agent policy\[[30](https://arxiv.org/html/2607.13576#bib.bib201)\], focusing solely on the set of monotonic policies to approximate the optimal solution\. It specifically learns the threshold where the optimal action shifts\. The threshold is updated based on the gradient of the expected reward received by the agent\. The average reward defined in \([3](https://arxiv.org/html/2607.13576#S3.E3)\) can be rewritten based on the evaluation of different policies parameterized byσ\\sigma, as follows:

R~A=limt→∞1t​𝔼σ​\[∑i=0tr~A​\(si;σ\)\]\\displaystyle\\tilde\{R\}\_\{A\}=\\lim\_\{t\\to\\infty\}\\frac\{1\}\{t\}\\mathbb\{E\}\_\{\\sigma\}\[\\sum\_\{i=0\}^\{t\}\\tilde\{r\}\_\{A\}\(s\_\{i\};\\sigma\)\]\(13\)Here,sis\_\{i\}denotes the state at a particular time,r~A​\(si;σ\)\\tilde\{r\}\_\{A\}\(s\_\{i\};\\sigma\)is the expected immediate reward based on the state, posterior belief of external state and agent’s actions underρσ\\rho\_\{\\sigma\}and the expectation is computed with respect to transition probability, signaling strategy and prior belief distribution\.

###### Lemma 2\.

The gradient of the average reward of the monotone agent following a policy parameterized byσ\\sigmais given by

∇σR~A=∑s∈𝒮d​\(s;σ\)\\displaystyle\\nabla\_\{\\sigma\}\\tilde\{R\}\_\{A\}=\\sum\_\{s\\in\\mathcal\{S\}\}d\(s;\\sigma\)⋅∇σr~A\(s\)\+\\displaystyle\\cdot\\nabla\_\{\\sigma\}\\tilde\{r\}\_\{A\}\(s\)\\ \+\(14\)∑s∈𝒮∑s′∈𝒮d​\(s;σ\)​∇σP​\(s′∣s;σ\)⋅V~A​\(s′\)\\displaystyle\\sum\_\{s\\in\\mathcal\{S\}\}\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}d\(s;\\sigma\)\\nabla\_\{\\sigma\}P\(s^\{\\prime\}\\mid s;\\sigma\)\\cdot\\tilde\{V\}\_\{A\}\(s^\{\\prime\}\)whered​\(s\)d\(s\)is the stationary state distribution, V~A​\(s′\)=∑g′∈𝒢∑θ′∈Θμs′​\(θ′\)⋅πs′,θ′​\(g′\)⋅VA​\(s′,g′\)\\tilde\{V\}\_\{A\}\(s^\{\\prime\}\)=\\sum\_\{g^\{\\prime\}\\in\\mathcal\{G\}\}\\sum\_\{\\theta^\{\\prime\}\\in\\Theta\}\\mu\_\{s^\{\\prime\}\}\(\\theta^\{\\prime\}\)\\cdot\\pi\_\{s^\{\\prime\},\\theta^\{\\prime\}\}\(g^\{\\prime\}\)\\cdot V\_\{A\}\(s^\{\\prime\},g^\{\\prime\}\)

The proof of the lemma is given in the Appendix\. The above expression shows that the gradient of the long\-term average reward depends on expected immediate reward, how the transition dynamics shift underσ\\sigma\(i\.e\.,∇σP\\nabla\_\{\\sigma\}P\), and how the states being transitioned are \(i\.e\.,V~A​\(s′\)\\tilde\{V\}\_\{A\}\(s^\{\\prime\}\)\), weighed by the stationary distributiond​\(s\)d\(s\)\. This is used in the following section for policy gradient methods for learning agent’s policy under Bayesian persuasion\.

### V\-BOnline learning of an agent’s monotonic policy under Bayesian persuasion

In this section, we introduce an algorithm that updates the parameterized agent policy at each time step\. This approach enables us to decompose the process into a series of incremental updates performed at every time step\[[30](https://arxiv.org/html/2607.13576#bib.bib201)\]\. In the context of Bayesian persuasion, we consider the expected reward of the agent’s action based on the transmitted signal, under the principal’s committed signaling strategy\. Here, we substitute the gradient with a biased estimate derived from simulating a single sample path\. It is known that this bias diminishes asymptotically, resulting in convergence\[[30](https://arxiv.org/html/2607.13576#bib.bib201)\]\. We impose the following assumption regarding the transition probabilities, i\.e\. for every parameterσ\\sigmaand statei,ji,j, there exists a bounded functionLi​jL\_\{ij\}such that,∇Pi​j​\(σ\)=Pi​j​\(σ\)⋅Li​j​\(σ\)\\nabla P\_\{ij\}\(\\sigma\)=P\_\{ij\}\(\\sigma\)\\cdot L\_\{ij\}\(\\sigma\)The gradient in \([14](https://arxiv.org/html/2607.13576#S5.E14)\) can be written as :

∇R~A​\(σ\)=∑n=0T−1∇r~A​\(sn;σ\)\+∑n=1T−1\(V~Asn​\(σ,R~A\)​Lsn−1​sn​\(σ\)\)\\displaystyle\\nabla\\tilde\{R\}\_\{A\}\(\\sigma\)=\\sum\_\{n=0\}^\{T\-1\}\\nabla\\tilde\{r\}\_\{A\}\(\{s\_\{n\}\};\\sigma\)\+\\sum\_\{n=1\}^\{T\-1\}\(\\tilde\{V\}\_\{A\_\{s\_\{n\}\}\}\(\\sigma,\\tilde\{R\}\_\{A\}\)L\_\{s\_\{n\-1\}s\_\{n\}\}\(\\sigma\)\)∇R~A​\(σ\)=\\displaystyle\\nabla\\tilde\{R\}\_\{A\}\(\\sigma\)=∇r~A​\(s0;σ\)\+∑k=1T−1∇r~A​\(sk;σ\)\+\(r~A​\(sk;σ\)−R~A\)​zk\\displaystyle\\nabla\\tilde\{r\}\_\{A\}\(\{s\_\{0\}\};\\sigma\)\+\\sum\_\{k=1\}^\{T\-1\}\\nabla\\tilde\{r\}\_\{A\}\(\{s\_\{k\}\};\\sigma\)\+\(\\tilde\{r\}\_\{A\}\(\{s\_\{k\}\};\\sigma\)\-\\tilde\{R\}\_\{A\}\)z\_\{k\}where,zk=∑n=1kLsn−1​sn​\(σ\)=∑n=1k∇Psn−1​sn​\(σ\)Psn−1​sn​\(σ\)\\displaystyle\\text\{ where\},z\_\{k\}=\\sum\_\{n=1\}^\{k\}L\_\{s\_\{n\-1\}s\_\{n\}\}\(\\sigma\)=\\sum\_\{n=1\}^\{k\}\\frac\{\\nabla P\_\{s\_\{n\-1\}s\_\{n\}\}\(\\sigma\)\}\{P\_\{s\_\{n\-1\}s\_\{n\}\(\\sigma\)\}\}The parameters can be updated at each step,tt, recursively as,

z0=\\displaystyle z\_\{0\}=0\\displaystyle 0\(15\)zt\+1=\\displaystyle z\_\{t\+1\}=zt\+Lst​st\+1​\(σ\)\\displaystyle z\_\{t\}\+L\_\{s\_\{t\}s\_\{t\+1\}\}\(\\sigma\)=\\displaystyle=zt\+∇Pst​st\+1​\(σ\)Pst​st\+1\\displaystyle z\_\{t\}\+\\frac\{\\nabla P\_\{s\_\{t\}s\_\{t\+1\}\}\(\\sigma\)\}\{P\_\{s\_\{t\}s\_\{t\+1\}\}\}\\σt\+1=\\displaystyle\\sigma\_\{t\+1\}=σt\+\(∇r~A​\(st;σt\)\+\(r~A​\(st;σt\)−R~A,t\)​zt\)\\displaystyle\\sigma\_\{t\}\+\(\\nabla\\tilde\{r\}\_\{A\}\(s\_\{t\};\\sigma\_\{t\}\)\+\(\\tilde\{r\}\_\{A\}\(s\_\{t\};\\sigma\_\{t\}\)\-\\tilde\{R\}\_\{A,t\}\)z\_\{t\}\)R~A,t\+1=\\displaystyle\\tilde\{R\}\_\{A,t\+1\}=R~A,t\+η​\(r~A​\(st;σt\)−R~A,t\)\\displaystyle\\tilde\{R\}\_\{A,t\}\+\\eta\(\\tilde\{r\}\_\{A\}\(s\_\{t\};\\sigma\_\{t\}\)\-\\tilde\{R\}\_\{A,t\}\)When the horizon is substantial,ztz\_\{t\}will also grow large before being reset to zero, leading to increased variance in the updates\. This is a common challenge encountered with likelihood ratio methods\. Therefore, it may be advantageous to incorporate a forgetting factor\[[30](https://arxiv.org/html/2607.13576#bib.bib201)\]\. In order to reduce the variance, we introduce a discount factor ofβ\\betaforztz\_\{t\}, and it becomes,zt\+1=β∗zt\+Lst​st\+1​\(σ\)z\_\{t\+1\}=\\beta\*z\_\{t\}\+L\_\{s\_\{t\}s\_\{t\+1\}\}\(\\sigma\)\\

### V\-CParameterization of agent’s monotonic policy

In this section, we consider the approximation of the optimal agent policy, which is monotonically increasing in the state and signal spaces, using parametric curves\. When there are two actions, the optimal monotonic policy in continuous state and signal spaces can be approximated by a threshold curve characterized by a set of parameters,σ,\\sigma,which partitions the region spanned by the state and signal space into two\. The optimal policy, which increases with state and signal space, can be characterized by a monotonic threshold curve\. The parametric representation of monotonic policy can utilize step, piecewise linear, or sigmoidal approximations\. The continuity and differentiability of the sigmoid at all points make it an ideal choice to represent a monotonic curve\. The sigmoidal approximation of an optimal monotonic policy in state and signal spaces can be given by

s~​\(σ,g\)=ϕ2−ϕ11\+e−ϕ3\+g⋅ϕ4\+ϕ1\\tilde\{s\}\(\\sigma,g\)=\\frac\{\\phi\_\{2\}\-\\phi\_\{1\}\}\{1\+e^\{\-\\phi\_\{3\}\+g\\cdot\\phi\_\{4\}\}\}\+\\phi\_\{1\}\(16\)where each sigmoid is characterized by a set of parameters given byσ=\{ϕ1,ϕ2,ϕ3,ϕ4\}\\sigma=\\\{\\phi\_\{1\},\\phi\_\{2\},\\phi\_\{3\},\\phi\_\{4\}\\\}\[[41](https://arxiv.org/html/2607.13576#bib.bib20)\]\.ϕ1\\phi\_\{1\},ϕ2\\phi\_\{2\}represent the floor and ceiling of the sigmoid, respectively\. The choice ofϕ4\\phi\_\{4\}determines the direction and spread of the sigmoid\. The curve is decreasing in the state\-signal space ifϕ4\>0\\phi\_\{4\}\>0, constant ifϕ4=0\\phi\_\{4\}=0, and increasing ifϕ4<0\\phi\_\{4\}<0\. The change in curvature of the sigmoid occurs at\[ϕ1\+ϕ22,ϕ3ϕ4\]\[\\frac\{\\phi\_\{1\}\+\\phi\_\{2\}\}\{2\},\\frac\{\\phi\_\{3\}\}\{\\phi\_\{4\}\}\]\. The optimal threshold policy approximation is the solution of

σ∗=arg⁡maxσ⁡R~A​\(ρ​\(σ\)\)\\sigma^\{\*\}=\\arg\\max\_\{\\sigma\}\\tilde\{R\}\_\{A\}\(\\rho\(\\sigma\)\)\(17\)whereR~A\\tilde\{R\}\_\{A\}is the expected average reward given in \([3](https://arxiv.org/html/2607.13576#S3.E3)\)\.

Algorithm 1Monotonic Agent Policy Learning \(MAPL\)1:Horizon length

NN
2:Signaling strategy

←𝝅\\leftarrow\\boldsymbol\{\\pi\}
3:forsteps

t=1,2,3​…\.Nt=1,2,3\.\.\.\.Ndo

4:Observe the external parameter

θt←μst\(\.\)\\theta\_\{t\}\\leftarrow\\mu\_\{s\_\{t\}\}\(\.\)
5:Obtain the signal

gt←πst,θt\(\.\)g\_\{t\}\\leftarrow\\pi\_\{s\_\{t\},\\theta\_\{t\}\}\(\.\)
6:Obtain the action

ata\_\{t\}based on

ρσ​\(st,gt\)\\rho\_\{\\sigma\}\(s\_\{t\},g\_\{t\}\)
7:Compute expected immediate reward

r~A​\(st;σt\)\\tilde\{r\}\_\{A\}\(s\_\{t\};\\sigma\_\{t\}\)
8:Update the agent’s threshold policy parameter

zt\+1=zt\+∇Pst​st\+1​\(σ\)Pst​st\+1\\displaystyle z\_\{t\+1\}=z\_\{t\}\+\\frac\{\\nabla P\_\{s\_\{t\}s\_\{t\+1\}\}\(\\sigma\)\}\{P\_\{s\_\{t\}s\_\{t\+1\}\}\}\\σt\+1=σt\+\(∇r~A​\(st;σt\)\+\(r~A​\(st,σt\)−R~A,t\)​zt\)\\displaystyle\\sigma\_\{t\+1\}=\\sigma\_\{t\}\+\(\\nabla\\tilde\{r\}\_\{A\}\(s\_\{t\};\\sigma\_\{t\}\)\+\(\\tilde\{r\}\_\{A\}\(s\_\{t\},\\sigma\_\{t\}\)\-\\tilde\{R\}\_\{A,t\}\)z\_\{t\}\)R~A,t\+1=R~A,t\+η​\(r~A​\(st;σt\)−R~A,t\)\\displaystyle\\tilde\{R\}\_\{A,t\+1\}=\\tilde\{R\}\_\{A,t\}\+\\eta\(\\tilde\{r\}\_\{A\}\(s\_\{t\};\\sigma\_\{t\}\)\-\\tilde\{R\}\_\{A,t\}\)
9:Get the next state

s′s^\{\\prime\}given by

P\(st,\.,at\)P\(s\_\{t\},\.,a\_\{t\}\)
10:

st\+1←s′s\_\{t\+1\}\\leftarrow s^\{\\prime\}
11:endfor

Algorithm[1](https://arxiv.org/html/2607.13576#alg1)\(MAPL\) describes the structured online learning of the agent policy\. Given an initial state, depending on the prior belief, an external state is sampled\. Based on the signaling strategy, a signal is then sampled for the realised state and external state\. The principal transmits the signal and this updates the agent’s posterior belief of external state\. Based on the expected immediate reward, the agent updates the threshold policy and chooses an action\. This process is repeated until the agent’s policy converges for the committed signaling strategy of the principal\. In the next subsection, we investigate how the Q structure of the principal can be leveraged for the design of a computationally efficient signaling strategy for a monotonic agent\.

### V\-DSupermodular Q learning of principal \(SQP\)

Here we discuss how standard relative Q learning\[[49](https://arxiv.org/html/2607.13576#bib.bib205)\]is modified to handle constraints so that it can adaptively learn a signaling strategy persuasive for a monotonic agent\. We can approximate the Bellman equation for the principal given by \([8](https://arxiv.org/html/2607.13576#S3.E8)\) for relative Q learning as :

QP\(s,θ,g\)=QP\(s,θ,g\)\+α′\[rP\(s,θ,g\)\+\\displaystyle Q\_\{P\}\(s,\\theta,g\)=Q\_\{P\}\(s,\\theta,g\)\+\\alpha^\{\\prime\}\[r\_\{P\}\(s,\\theta,g\)\+maxg′QP\(s′,θ′,g′\)−QP\(s,θ,g\)−maxg′QP\(s0,θ0,g′\)\]\\displaystyle\\max\_\{g^\{\\prime\}\}Q\_\{P\}\(s^\{\\prime\},\\theta^\{\\prime\},g^\{\\prime\}\)\-Q\_\{P\}\(s,\\theta,g\)\-\\max\_\{g^\{\\prime\}\}Q\_\{P\}\(s\_\{0\},\\theta\_\{0\},g^\{\\prime\}\)\]whereα′\\alpha^\{\\prime\}is the learning rate and\(s0,θ0\)\(s\_\{0\},\\theta\_\{0\}\)is the reference state\. The optimal solution can be computed as

𝔼\[rP\(s,θ,g\)\+maxg′QP∗\(s′,θ′,g′\)\)−QP∗\(s,θ,g\)\]=0\\displaystyle\\mathbb\{E\}\\left\[r\_\{P\}\(s,\\theta,g\)\+\\max\_\{g^\{\\prime\}\}Q\_\{P\}^\{\*\}\(s^\{\\prime\},\\theta^\{\\prime\},g^\{\\prime\}\)\)\-Q\_\{P\}^\{\*\}\(s,\\theta,g\)\\right\]=0\(18\)According to Theorem 3, principal’s Q\-values exhibit supermodularity\. Imposing this supermodular structure enables efficient learning while bypassing non\-optimal strategies\[[20](https://arxiv.org/html/2607.13576#bib.bib18)\]\. The convergence of supermodular Q\-learning is well\-understood\[[10](https://arxiv.org/html/2607.13576#bib.bib204)\]and extends naturally to our context\. The supermodularity as a linear inequality constraint onQPQ\_\{P\}is :

QP​\(s,θ,g\+1\)−QP​\(s,θ,g\)≤QP​\(s′,θ,g\+1\)−QP​\(s′,θ,g\)\\displaystyle Q\_\{P\}\(s,\\theta,g\+1\)\-Q\_\{P\}\(s,\\theta,g\)\\leq Q\_\{P\}\(s^\{\\prime\},\\theta,g\+1\)\-Q\_\{P\}\(s^\{\\prime\},\\theta,g\)Qp​\(s,θ,g\+1\)−QP​\(s,θ,g\)≤QP​\(s,θ′,g\+1\)−QP​\(s,θ′,g\)\\displaystyle Q\_\{p\}\(s,\\theta,g\+1\)\-Q\_\{P\}\(s,\\theta,g\)\\leq Q\_\{P\}\(s,\\theta^\{\\prime\},g\+1\)\-Q\_\{P\}\(s,\\theta^\{\\prime\},g\)∀s≤s′,θ≤θ′\.\\forall s\\leq s^\{\\prime\},\\theta\\leq\\theta^\{\\prime\}\.The constraint on the supermodularity of Q factors can be given as:

QP​I≤0\\displaystyle Q\_\{P\}I\\leq 0\(19\)where≤\\leqis elementwise\. Letf​\(QP​\(s,θ,g\)\)f\(Q\_\{P\}\(s,\\theta,g\)\)be taken as

f​\(QP​\(s,θ,g\)\)=\(rP​\(s,θ,g\)\+maxg′⁡QP​\(s′,θ′,g′\)\)−QP​\(s,θ,g\)\\displaystyle f\(Q\_\{P\}\(s,\\theta,g\)\)=\(r\_\{P\}\(s,\\theta,g\)\+\\max\_\{g^\{\\prime\}\}Q\_\{P\}\(s^\{\\prime\},\\theta^\{\\prime\},g^\{\\prime\}\)\)\-Q\_\{P\}\(s,\\theta,g\)Also letf​\(QP\)=∇QPh​\(QP\)f\(Q\_\{P\}\)=\\nabla\_\{Q\_\{P\}\}h\(Q\_\{P\}\) Then, by \([18](https://arxiv.org/html/2607.13576#S5.E18)\), we have𝔼​\(f​\(QP∗\)\)=∇Q𝔼​\(h​\(QP∗\)\)=0\\mathbb\{E\}\(\{f\(Q\_\{P\}^\{\*\}\)\}\)=\\nabla\_\{Q\}\\mathbb\{E\}\(\{h\(Q\_\{P\}^\{\*\}\)\}\)=0\. Hence, we consider Q learning as stochastic gradient algorithm\[[20](https://arxiv.org/html/2607.13576#bib.bib18)\]that maximizes an objectiveh​\(Q\)h\(Q\)subject to \([19](https://arxiv.org/html/2607.13576#S5.E19)\), i\.e\.

QP∗=arg⁡maxQP⁡𝔼​\(h​\(QP\)\)\\displaystyle Q\_\{P\}^\{\*\}=\\arg\\max\_\{Q\_\{P\}\}\\mathbb\{E\}\(h\(Q\_\{P\}\)\)\(20\)Now, we use primal\-dual stochastic algorithm\[[20](https://arxiv.org/html/2607.13576#bib.bib18)\]to approximateQP∗Q\_\{P\}^\{\*\}withλ≥0\\lambda\\geq 0as Lagrange multipliers given by:

QP,t\+1=\\displaystyle Q\_\{P,t\+1\}=QP,t\+α′​\[f​\(QP,t\)−maxg′⁡QP,t​\(s0,θ0,g′\)−λt​I\]\\displaystyle Q\_\{P,t\}\+\\alpha^\{\\prime\}\[f\(Q\_\{P,t\}\)\-\\max\_\{g^\{\\prime\}\}Q\_\{P,t\}\(s\_\{0\},\\theta\_\{0\},g^\{\\prime\}\)\-\\lambda\_\{t\}I\]\(21\)λt\+1=\\displaystyle\\lambda\_\{t\+1\}=m​a​x​\[λt\+α′​QP,t​I,0\]\\displaystyle max\[\\lambda\_\{t\}\+\\alpha^\{\\prime\}Q\_\{P,t\}I,0\]
Algorithm 2Supermodular Q learning of Principal \(SQP\)1:Learning agent’s best response policy

\{ρ~k\}\\\{\\tilde\{\\rho\}\_\{k\}\\\}, Principal’s Q value

QP,kQ\_\{P,k\}
2:Initial state

s0s\_\{0\}
3:for

t=0,1,2,…​Tt=0,1,2,\\dots Tdo

4:

θt←μ\(⋅∣st\)\\theta\_\{t\}\\leftarrow\\mu\(\\cdot\\mid s\_\{t\}\)
5:

π\(\.\|s,θ\)=exp⁡\(QP​\(s,θ,g\)/τ\)∑g′exp⁡\(QP​\(s,θ,g′\)/τ\)\\pi\(\.\|s,\\theta\)=\\frac\{\\exp\\left\(Q\_\{P\}\(s,\\theta,g\)/\\tau\\right\)\}\{\\sum\_\{g^\{\\prime\}\}\\exp\\left\(Q\_\{P\}\(s,\\theta,g^\{\\prime\}\)/\\tau\\right\)\}
6:

gt←π\(⋅∣st,θt\)g\_\{t\}\\leftarrow\\pi\(\\cdot\\mid s\_\{t\},\\theta\_\{t\}\)
7:Choose action

at∼ρk~\(⋅∣st,gt\)a\_\{t\}\\sim\\tilde\{\\rho\_\{k\}\}\(\\cdot\\mid s\_\{t\},g\_\{t\}\)
8:Observe next state

st\+1←P\(⋅∣st,at\)s\_\{t\+1\}\\leftarrow P\(\\cdot\\mid s\_\{t\},a\_\{t\}\)
9:Observe

RP​\(st,θt,at\),RA​\(st,θt,at\)R\_\{P\}\(s\_\{t\},\\theta\_\{t\},a\_\{t\}\),R\_\{A\}\(s\_\{t\},\\theta\_\{t\},a\_\{t\}\)
10:Update

QP,t\+1Q\_\{P,t\+1\}and ensure supermodularity using \([21](https://arxiv.org/html/2607.13576#S5.E21)\)

11:endfor

Algorithm 2 \(SQP\) describes the online learning of the principal’s Q value for the agent’s best response policyρ~\\tilde\{\\rho\}\. At each time step, for the given\(st,θt\)\(s\_\{t\},\\theta\_\{t\}\), a signalgtg\_\{t\}is sampled by taking the soft max of the principal’s Q values\. On receiving the signal, the agent takes an action based onρ~\\tilde\{\\rho\}and the rewards of both the principal and agent are observed\. The Q values of the principal are then updated using \([21](https://arxiv.org/html/2607.13576#S5.E21)\)\. This is continued till the end of the episode\.

### V\-ERepeated principal agent interaction

![Refer to caption](https://arxiv.org/html/2607.13576v1/x3.png)Figure 3:The figure illustrates an online learning framework in which the principal and the agent iteratively adapt their strategies through repeated interactions\. At each episode, given the signaling strategy announced by the principal, the agent learns its optimal policy\. The principal then observes the agent’s best response and updates the signaling strategy accordingly for the subsequent episode\.In this paper, we consider the dynamic Bayesian persuasion as illustrated in Fig 3, wherein the principal repeatedly interacts with an approximately best responding learning agent\. LetR~P​\(π,ρ\)\\tilde\{R\}\_\{P\}\(\\pi,\\rho\)andR~A​\(π,ρ\)\\tilde\{R\}\_\{A\}\(\\pi,\\rho\)denote the principal’s and agent’s long\-run expected average rewards, respectively, under the signaling strategyπ\\piand agent’s policyρ\\rho\. For any fixed signaling strategyπk\\pi^\{k\}, the agent selects a best response atkt​hk^\{th\}episode by choosing

ρk=arg⁡maxρ~⁡R~A​\(πk,ρ~\)\\rho^\{k\}=\\arg\\max\_\{\\tilde\{\\rho\}\}\\tilde\{R\}\_\{A\}\(\\pi^\{k\},\\tilde\{\\rho\}\)Given the induced agent response, the principal updates the signaling strategy accordingly as

πk\+1=arg⁡maxπ~⁡R~P​\(π~,ρk\)\\pi^\{k\+1\}=\\arg\\max\_\{\\tilde\{\\pi\}\}\\tilde\{R\}\_\{P\}\(\\tilde\{\\pi\},\\rho^\{k\}\)Initially, given the principal’s signaling strategyπk\\pi^\{k\}, the agent learn’s over an episode and finds its best response,ρk\\rho^\{k\}\. Observing the agent’s best response, the principal modifies its signaling strategy for the next round,πk\+1\\pi^\{k\+1\}, and the repeated iteration continues\.π∗\\pi^\{\*\}is said to be dynamically stable if∇πR~P​\(π,ρ\)=0\\nabla\_\{\\pi\}\\tilde\{R\}\_\{P\}\(\\pi,\\rho\)=0asπk→π∗\\pi^\{k\}\\to\\pi^\{\*\}\.

Algorithm 3Online learning of signaling strategy and agent’s policy1:Initialize:Signaling strategy

π0\\pi\_\{0\}, agent policy

ρ0~\\tilde\{\\rho\_\{0\}\}, step sizes

\{α′\}\\\{\\alpha^\{\\prime\}\\\}
2:for

k=0,1,2,…​Nk=0,1,2,\\ldots Ndo

3:Principal commits to signaling strategy

πk\\pi\_\{k\}
4:for

t=0,1,2,…​Tt=0,1,2,\\ldots Tdo

5:Agent policy update using MAPL

σt\+1=σt\+\(∇r~st​\(σt\)\+\(r~st​\(σt\)−R~At\)\)​zt\\sigma\_\{t\+1\}=\\sigma\_\{t\}\+\(\\nabla\\tilde\{r\}\_\{s\_\{t\}\}\(\\sigma\_\{t\}\)\+\(\\tilde\{r\}\_\{s\_\{t\}\}\(\\sigma\_\{t\}\)\-\\tilde\{R\}\_\{A\_\{t\}\}\)\)z\_\{t\}
6:endfor

7:for

t′=0,1,2,…​Tt^\{\\prime\}=0,1,2,\\ldots Tdo

8:Update principal’s Q value under agent’s best response using SQP

QP,t′\+1=QP,t′\+α′​\[f​\(QP,t′\)−λt′​I\]\\displaystyle Q\_\{P,t^\{\\prime\}\+1\}=Q\_\{P,t^\{\\prime\}\}\+\\alpha^\{\\prime\}\[f\(Q\_\{P,t^\{\\prime\}\}\)\-\\lambda\_\{t^\{\\prime\}\}I\]
9:endfor

10:Output signaling strategy

π\\piusing \([12](https://arxiv.org/html/2607.13576#S4.E12)\)

11:endfor

### V\-FStructured learning for repeated principal agent interactions

Algorithm 3 describes the online learning of the signaling strategy and agent’s policy\. At the start of each round, the sender publicly commits to an incentive\-compatible signaling strategyπ\\pifor a monotonic agent\. Knowing the signaling strategy, agent learn to select an optimal policy that maximizes expected reward using MAPL which updates the threshold at each step based on the expected reward\. After observing the agent’s action for an episode, the principal computes the average reward and updates its Q value under agent’s best response using SQP\. Thus, using structured learning, Algorithm 3 learns the agent’s monotonic policy and leverage the supermodular structure of the value function to ensure that the sender’s signaling strategy remains persuasive throughout the repeated interaction\.

## VINumerical Results: Bayesian Persuasive Driving

![Refer to caption](https://arxiv.org/html/2607.13576v1/x4.png)Figure 4:The figure gives an illustration of Bayesian persuasive driving\. Here, the red coloured vehicle symbolises the lead vehicle and blue represents the connected vehicle\. Road side Units\(RSU\) transmit traffic data to the lead vehicle\. The lead vehicle who is aware of the real\-time traffic updates guides the connected vehicle through the dark line ensuring congestion\-free traffic\.In this work, we consider lane\-based routing—where decisions are made not just at the road level but at the granularity of individual lanes— which is becoming increasingly viable due to advancements in vehicle\-to\-vehicle \(V2V\) communication, real\-time traffic sensing and precise localization technologies\.

### VI\-AReal\-time traffic routing via Bayesian persuasion

We model a dynamic Bayesian persuasion framework for intelligent, lane\-based traffic routing involving an intelligent lead vehicle \(the principal\) and connected vehicle \(agent\), as depicted in Fig\.[4](https://arxiv.org/html/2607.13576#S6.F4)\. The framework includes:

1. 1\.Intelligent lead vehicle: The lead vehicle has access to real\-time traffic conditions—including accidents and road closures—through GPS, RSUs, and other external sources\[[9](https://arxiv.org/html/2607.13576#bib.bib6)\]\. Based on this information, it strategically selects signaling strategies to guide vehicles into specific lanes, aiming to optimize overall traffic flow\.
2. 2\.Autonomous Connected Vehicle: Vehicle use onboard sensors to monitor their speed and location, while receiving external updates from the lead vehicle\. Based on the observed signal and knowledge of the signaling strategy, it updates the belief about congestion ahead and makes lane\-selection locally and in real\-time\.
3. 3\.Continuous feedback: The routing process is adaptive and continuous, with connected vehicles being updated regularly based on real\-time traffic conditions\. The dynamic change in the state of the connected vehicle is also continuously monitored by the lead vehicle to provide further updates, improving overall efficiency and service quality\.

We consider the following model for Bayesian persuasion\[[9](https://arxiv.org/html/2607.13576#bib.bib6),[37](https://arxiv.org/html/2607.13576#bib.bib54)\]\.

- •*State Space*: The vehicle state represents the speed levels of the connected vehicle\. Heres∈𝒮=\{0,1,2\.,10\}s\\in\\mathcal\{S\}=\\\{0,1,2\.,10\\\}
- •*External State Space*: Under typical traffic conditions, vehicles move at free\-flow speed, which is the speed attainable when traffic density is low and movement is not affected by surrounding vehicles\. However, if many vehicles choose the same route at the same time, their collective demand can surpass the lane’s capacity, causing congestion\. In our model, the external state represents the extent of this congestion\. The external state quantifies traffic congestion based on the traffic flow model in\[[35](https://arxiv.org/html/2607.13576#bib.bib189)\], where: θ=vdvf=\(1−\(ddjam\)p\)q\\displaystyle\\theta=\\frac\{v\_\{d\}\}\{v\_\{f\}\}=\\left\(1\-\\left\(\\frac\{d\}\{d\_\{\\text\{jam\}\}\}\\right\)^\{p\}\\right\)^\{q\}\(22\)Here,ddis traffic density,djamd\_\{\\text\{jam\}\}the jam density,vdv\_\{d\}the effective speed at densitydd, andvfv\_\{f\}the free\-flow speed\. Here, increasing levels of congestion\-free traffic is represented asθ∈Θ=\{0,1,2\.\.10\}\\theta\\in\\Theta=\\\{0,1,2\.\.10\\\}
- •*Action Space*: In our traffic routing framework, we analyze a single\-link setup that may contain several parallel lanes, each characterized by distinct speed profiles or congestion levels\. The available actions correspond to choosing among these lanes\. Specifically, in the two\-lane case, we define action 1 as selecting the low\-speed lane and action 2 as selecting the high\-speed lane\.
- •*Agent’s travel reward function*: The reward function for choosing lane 1 and lane 2 are assumed asRA1​\(s,θ,1\)R\_\{A\_\{1\}\}\(s,\\theta,1\)as175\+s2\+θ2175\+s^\{2\}\+\\theta^\{2\}andRA2​\(s,θ,2\)R\_\{A\_\{2\}\}\(s,\\theta,2\)as155\+1\.5⋅s2\+1\.5⋅θ2155\+1\.5\\cdot s^\{2\}\+1\.5\\cdot\\theta^\{2\}\. The reward structure ensures that the first lane is better for lower speed and lower traffic\-free flow\. In addition, the second lane becomes preferable at higher speeds and congestion\-free traffic due to its greater rate of reward in comparison with the first\.
- •*Principal’s reward function*: The reward function for choosing lane 1 and lane 2 are assumed asRP1​\(s,θ,1\)R\_\{P\_\{1\}\}\(s,\\theta,1\)to beRA1​\(s,θ,1\)R\_\{A\_\{1\}\}\(s,\\theta,1\)andRP2​\(s,θ,2\)R\_\{P\_\{2\}\}\(s,\\theta,2\)as500\+RA2​\(s,θ,1\)500\+R\_\{A\_\{2\}\}\(s,\\theta,1\)\. This accounts for the extra reward the lead vehicle receives when the connected vehicle chooses the high\-speed lane\. It satisfies \(A1\) and \(A2\)\.
- •*Vehicle dynamics*: The transition probability density reflects the likelihood of transitioning to different speeds based on the chosen lane\. Low and high\-speed ranges are referred to as speed which is less than 50% of the maximum level possible and the rest respectively\. High\-speed lane offers higher chances of transitioning to higher speeds than low\-speed lanes\. The transition probability density captures how likely a vehicle is to shift between different speeds depending on the selected lane, as illustrated in Fig\.[5](https://arxiv.org/html/2607.13576#S6.F5)\. It reflects the assumption that vehicles traveling at higher speeds tend to maintain those speeds, and similarly for lower speeds\. Compared to low\-speed lanes, high\-speed lanes provide a greater probability of transitioning into and staying within higher speed ranges\. The speed transition dynamics of the connected vehicle are consistent with \(A4\) and \(A5\)\.
- •*Connected vehicle’s objective*: In the proposed framework, the objective of the principal \(lead vehicle\) is to monitor the speed levels of connected vehicles across different lanes and provide routing signals based on real\-time congestion\[[16](https://arxiv.org/html/2607.13576#bib.bib88)\]\. Its signaling strategy is designed to guide vehicles predominantly toward high\-speed lanes, thereby enhancing overall traffic efficiency while also maximizing the travel reward for the connected vehicle\. The agent \(connected vehicle\) on the other hand aims to determine a policy that selects the optimal lane to maximize its expected average reward favouring higher speeds and minimal congestion\.

LowspeedHighspeedLANE 10\.90\.20\.10\.8LowspeedHighspeedLANE 20\.30\.80\.70\.2Figure 5:State transition diagram for the two\-lane system\.We consider a Bayesian persuasion model with a completely unknown reward and transition dynamics, which fit the model assumed for monotonic agent policies, and consider the agent’s prior belief as uniform, which satisfies \(A3\)\. The assumed Bayesian persuasive driving context satisfies assumptions for a monotonic agent policy and the principal’s supermodular Q function\. Hence, we utilize the structural results and demonstrate its advantage in the efficient online learning of the connected vehicle policy as well as signaling strategy of the lead vehicle with the existing methods in the following\.

### VI\-BComparison with existing dynamic vehicle routing algorithms

Here, we compare the travelling reward of the connected vehicle, which uses the Bayesian persuasion framework for lane selection, employing the MAPL with the existing vehicle routing methodologies\. The dynamic vehicle routing algorithms considered are:

1. 1\.Model Predictive Control \(MPC\): Here we use the transition model described in Fig\.[5](https://arxiv.org/html/2607.13576#S6.F5)\. The model is iteratively learned during the learning to simulate the future trajectory over a horizon of 5\[[40](https://arxiv.org/html/2607.13576#bib.bib156)\],\[[38](https://arxiv.org/html/2607.13576#bib.bib184)\],\[[17](https://arxiv.org/html/2607.13576#bib.bib216)\]\. At each decision point, MPC computes the cumulative travel reward over the prediction horizon and chooses the action that minimizes it\[[51](https://arxiv.org/html/2607.13576#bib.bib214)\],\[[35](https://arxiv.org/html/2607.13576#bib.bib189)\]\. This is continued till the end of the episode\.
2. 2\.Stochastic Shortest Path \(SSP\): This approach utilizes the prior assumption of the agent regarding the congestion of the lane ahead\. The transition probabilities capture stochastic speed variations, and then one\-step prediction is employed to choose a lane which maximizes the reward of travel\[[21](https://arxiv.org/html/2607.13576#bib.bib188)\]\.
3. 3\.Tree\-based learning: Tree\-based learning approximates the value function in dynamic vehicle routing by using trees to estimate the future reward of different routing decisions based on current states\. The tree is trained on samples collected through simulation\[[44](https://arxiv.org/html/2607.13576#bib.bib212)\]\.

TABLE I:Comparison of the travelling reward of connected vehicle for various vehicle routing algorithms

![Refer to caption](https://arxiv.org/html/2607.13576v1/Agent_reward.png)Figure 6:Average reward accumulated by the connected vehicle![Refer to caption](https://arxiv.org/html/2607.13576v1/Prin_reward.png)Figure 7:Average reward accumulated by the lead vehicleHere, the proposed structured learning method yields better average rewards considering the farsightedness of the agent, as illustrated in Table 1\. SSP, MPC and Tree based learning obtain lower rewards due to their reliance on prior traffic assumptions without real\-time updates\. This shows the benefit of utilising the threshold policy of the connected vehicle navigating using the Bayesian persuasion framework for lane selection over the existing dynamic vehicle routing recommendations\.

#### VI\-B1Comparison with existing methodology for the design of signaling strategy

In the structured learning, we utilize the monotonic structure of the agent optimal policy and supermodular Q learning of principal for computationally efficient learning of signaling strategy persuasive for connected vehicle\. For comparison with existing reinforcement learning framework, the signaling strategy is evaluated over 100 episodes of 100 steps\. We compare with existing signaling strategies used in the context of Bayesian persuasion which include

1. 1\.Optimism\-Pessimism Principle for Persuasion Process\(OP4\)\[[46](https://arxiv.org/html/2607.13576#bib.bib74)\]: The principal designs signals to optimize agent’s expected immediate reward, without accounting for future state transitions\. Here, pessimism tackles the uncertainty in the prior estimation by selecting a signaling policy that is persuasive for a myopic agent with respect to all the priors in the confidence region, while optimism in principal’s Q\-function estimation encourages exploration\[[46](https://arxiv.org/html/2607.13576#bib.bib74)\]\. It perform poorly with far\-sighted drivers who anticipate future congestion and signal disclosures\.
2. 2\.History based online persuasion: In traffic routing, the lead vehicle learns a decision tree mapping histories to signals, anticipating drivers’ future reactions\. The IC constraint requires that at every node in this tree, the agent with full recall — knowing exactly which path led to the current node — still finds it rational to follow the history based signaling recommendation rather than deviate\[[3](https://arxiv.org/html/2607.13576#bib.bib75)\]\. This approach assumes that model assumptions except the agent’s prior belief of the external state are known, and the agent needs to retain the history of past interaction, making it slow to learn, adapt, and plan ahead\.
3. 3\.Uniform signaling: Here the principal sends signals that are independent of the realized state of nature\. As a result, every signal induces the same posterior belief for the agent, equal to the prior\. This means the principal does not exploit informational asymmetries and exerts no influence on the agent’s action through belief manipulation\. Uniform signaling is often used as a baseline\[[23](https://arxiv.org/html/2607.13576#bib.bib223)\]\.
4. 4\.Q learning: In Bayesian persuasion, the principal and the agent can use Q\-learning independently to maximize their own long\-term rewards under repeated interactions
5. 5\.Full Revelation: Full\-revelation signaling discloses complete traffic state information \(e\.g\., exact congestion levels on all routes\)\[[23](https://arxiv.org/html/2607.13576#bib.bib223)\]\.

Here, the proposed structured learning yields better average rewards considering the farsightedness of the agent and principal, as illustrated in the Table 2\. The faster and enhanced learning of the proposed method, which takes into account the structure of the agent’s policy and principal’s value function, compared with the existing online reinforcement learning for signaling strategy, is shown in Fig 6 and 7\.

TABLE II:Comparison of the travelling reward of connected vehicle for existing signaling strategies

### VI\-CComputational efficiency as compared to existing methodology

Fig\.[8](https://arxiv.org/html/2607.13576#S6.F8)illustrates the computational efficiency of the proposed approach with the existing methodology in terms of the computational time\. The existing methodology scales with an increase in the dimension of state and external state space, whereas for the proposed methodology, the agent’s policy learning remains independent of the increase in dimensionality of state space\. By incorporating prior knowledge of the threshold structure of the agent’s policy and the principal’s Q\-values, it improves learning efficiency while reducing computational complexity\.

![Refer to caption](https://arxiv.org/html/2607.13576v1/bar_computation_time.png)Figure 8:Comparison of computational time of existing and structured learning methods

## VIIConclusion

Motivated by the efficiency of Bayesian persuasion for intelligent interactive driving, this paper proposes an online structured reinforcement learning framework for design of computationally efficient signaling strategy, which is persuasive for a far\-sighted agent\. The main contributions are:\(i\)MAPL\- online structured policy learning algorithm which utilizes the monotonic structure of agent policy for faster learning in large state and signal spaces,\(ii\)Identification of sufficient condition on the Bayesian persuasion model to ensure the supermodular structure of the Q function of the principal for a monotonic agent,\(iii\)Identification of sufficient conditions for a persuasive signaling strategy for a far\-sighted agent\(iv\)Supermodular Q learning of Principal \(SQP\) for the design of a persuasive signaling strategy considering a monotonic agent,\(v\)Case study of Bayesian persuasive driving, which illustrates the practicality of the structural results in addressing Bayesian persuasion for dynamic vehicle routing\.By incorporating prior knowledge of the threshold structure of the agent’s policy and the principal’s Q\-values, it improves learning efficiency while reducing computational complexity\. This approach is especially effective in highly dynamic real\-world applications such as traffic routing, sensor scheduling, inventory management, and transmission scheduling, where the proposed model fits naturally\. The current work considers the homogeneous and monotonic driving behaviour of connected vehicles\. However, incorporating the risk\-sensitive nature of connected vehicles remains a promising direction for future research\.

## Appendix:

### VII\-ABasic definitions

The following definitions are required for the proof of the main results\.

###### Definition 1\.

\(Supermodularity\): The functionr​\(s,a\)r\(s,a\)is said to be supermodular if

r​\(s,a¯\)−r​\(s,a\)≤r​\(s¯,a¯\)−r​\(s¯,a\)\\displaystyle r\(s,\\bar\{a\}\)\-r\(s,\{a\}\)\\leq r\(\\bar\{s\},\\bar\{a\}\)\-r\(\\bar\{s\},\{a\}\)\(23\)wheres¯≥s\\bar\{s\}\\geq sanda¯≥a,∀s,s¯∈𝒮,a,a¯∈𝒜\\bar\{a\}\\geq a,\\forall s,\\bar\{s\}\\in\\mathcal\{S\},a,\\bar\{a\}\\in\\mathcal\{A\}\. In other words,r​\(s,a¯\)−r​\(s,a\)r\(s,\\bar\{a\}\)\-r\(s,a\)is increasing with respect toss\.

###### Definition 2\.

\(Tail sum supermodularity\): The probability mass function, pmf,P​\(s,s′,a\)P\(s,s^\{\\prime\},a\), is tail sum supermodular in\(s,a\)\(s,a\)if

∑s¯S\(\(P\(s,s′,a\+1\)−P\(s,s′,a\)\)is increasing ins\\displaystyle\\sum\_\{\\bar\{s\}\}^\{S\}\(\(P\(s,s^\{\\prime\},a\+1\)\-P\(s,s^\{\\prime\},a\)\)\\text\{ is increasing in \}s\(24\)for∀s,s¯∈𝒮,a∈\{0,A\}\\ \\forall s,\\bar\{s\}\\in\\mathcal\{S\},a\\in\\\{0,A\\\}

###### Definition 3\.

\(Stochastic dominance\): Letp1p\_\{1\}andp2p\_\{2\}be two probability mass functions, thenp1p\_\{1\}is said to first\-order stochastically dominatep2p\_\{2\}, denoted asp1≥sp2p\_\{1\}\\geq\_\{s\}p\_\{2\}if

∑s¯Sp1​\(s\)≥∑s¯Sp2​\(s\)\\displaystyle\\sum\_\{\\bar\{s\}\}^\{S\}p\_\{1\}\(s\)\\geq\\sum\_\{\\bar\{s\}\}^\{S\}p\_\{2\}\(s\)\(25\)

###### Definition 4\.

\(Monotone Likelihood Ratio \(MLR\) ordering\): Letp1p\_\{1\}andp2p\_\{2\}be two probability mass functions, thenp1p\_\{1\}is said to dominatep2p\_\{2\}with respect to the MLR order, denoted asp1≥rp2p\_\{1\}\\geq\_\{r\}p\_\{2\}if

p1​\(g1\)⋅p2​\(g2\)≤p2​\(g1\)⋅p1​\(g2\),g1≤g2,∀g1,g2∈𝒢\\displaystyle p\_\{1\}\(g\_\{1\}\)\\cdot p\_\{2\}\(g\_\{2\}\)\\leq p\_\{2\}\(g\_\{1\}\)\\cdot p\_\{1\}\(g\_\{2\}\),g\_\{1\}\\leq g\_\{2\},\\ \\forall g\_\{1\},g\_\{2\}\\in\\mathcal\{G\}\(26\)

###### Theorem 4\.

For any increasing function off\(\.\)f\(\.\)\.p1≥sp2p\_\{1\}\\geq\_\{s\}p\_\{2\}iff∑s∈𝒮f​\(s\)⋅p1​\(s\)​d​s≥∑s∈𝒮f​\(s\)⋅p2​\(s\)​d​s\\sum\_\{s\\in\\mathcal\{S\}\}f\(s\)\\cdot p\_\{1\}\(s\)ds\\geq\\sum\_\{s\\in\\mathcal\{S\}\}f\(s\)\\cdot p\_\{2\}\(s\)ds\[[20](https://arxiv.org/html/2607.13576#bib.bib18)\]\.

###### Theorem 5\.

For any increasing functionf\(\.\)f\(\.\), theP​\(s,s′,a\)P\(s,s^\{\\prime\},a\)is tail sum supermodular iff∑s′∈𝒮P​\(s,s′,a\)⋅f​\(s′\)\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}P\(s,s^\{\\prime\},a\)\\cdot f\(s^\{\\prime\}\)is supermodular in\(s,a\)\(s,a\)\[[20](https://arxiv.org/html/2607.13576#bib.bib18)\]\.

###### Lemma 3\.

Considerγk\\gamma\_\{k\}be an increasing sequence of discount factors\{γk\}\\\{\\gamma\_\{k\}\\\}such thatγk→1\\gamma\_\{k\}\\rightarrow 1, and let\{ργk\}\\\{\\rho\_\{\\gamma\_\{k\}\}\\\}be the corresponding stationary discounted optimal policies\. Then there exists a subsequence\{ζk\}\\\{\\zeta\_\{k\}\\\}ofγk\\gamma\_\{k\}and a stationary policyρ∗\\rho^\{\*\}which is a limit point of\{ρζk\}\\\{\\rho\_\{\\zeta\_\{k\}\}\\\}\. That is, for every\(s,g\)\(s,g\), there exists an integerN​\(s,g\)N\(s,g\)such that for alln≥N​\(s,g\),ρζk​\(s,g\)=ρ∗​\(s,g\)n\\geq N\(s,g\),\\rho\_\{\\zeta\_\{k\}\}\(s,g\)=~\\rho^\{\*\}\(s,g\)\.

For the Bayesian persuasion model we assume :

- \(C1\)Markov chain induced by agent’s policy is ergodic\.
- \(C2\)For everys∈S,g∈Gs\\in S,g\\in Gand discount factorγ\\gamma, discounted value function,Vργ​\(s,g\)<∞V^\{\\gamma\}\_\{\\rho\}\(s,g\)<\\infty
- \(C3\)Assuming a reference state\(s0,g0\)\(s\_\{0\},g\_\{0\}\), there exists a nonnegativeLLsuch that −L≤hγ​\(s,g\)=△VAγ​\(s,g\)−VAγ​\(s0,g0\)​∀\(s,g\),γ\-L\\leq h^\{\\gamma\}\(s,g\)\\stackrel\{\{\\scriptstyle\\triangle\}\}\{\{=\}\}V\_\{A\}^\{\\gamma\}\(s,g\)\-V\_\{A\}^\{\\gamma\}\(s\_\{0\},g\_\{0\}\)\\ \\forall\(s,g\),\\gamma\.
- \(C4\)There exists aM​\(s,g\)M\(s,g\)such thathγ​\(s,g\)≤M​\(s,g\)h^\{\\gamma\}\(s,g\)\\leq M\(s,g\)for every\(s,g\)\(s,g\)andγ\\gammaand an actiona0a\_\{0\}such that∑s′∈𝒮P\(s,s′,a0\)⋅M\(s′,\.\)<∞\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}P\(s,s^\{\\prime\},a\_\{0\}\)\\cdot M\(s^\{\\prime\},\.\)<\\infty\. Also∑s′∈𝒮P\(s,s′,a\)⋅M\(s′,\.\)<∞\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}P\(s,s^\{\\prime\},a\)\\cdot M\(s^\{\\prime\},\.\)<\\inftyfor all\(s,g\),a\(s,g\),a\.

Under \(C1\), \(C2\), \(C3\), \(C4\), any policy identified as a limit point in Lemma[3](https://arxiv.org/html/2607.13576#Thmlemma3), qualifies as an average reward optimal policy\[[42](https://arxiv.org/html/2607.13576#bib.bib210)\]\. This is established in the following as :

###### Lemma 4\.

Any stationary deterministic policyρ∗\\rho^\{\*\}identified through Lemma[3](https://arxiv.org/html/2607.13576#Thmlemma3)is average reward optimal\. More precisely, there exists a constantRA∗=l​i​mγ→1​\(1−γ\)​VAγ​\(s,g\)R\_\{A\}^\{\*\}=lim\_\{\\gamma\\rightarrow 1\}\(1\-\\gamma\)V\_\{A\}^\{\\gamma\}\(s,g\)for every\(s,g\)\(s,g\)andVA​\(s,g\)V\_\{A\}\(s,g\)with−L≤VA​\(s,g\)≤M​\(s,g\),\-L\\leq V\_\{A\}\(s,g\)\\leq M\(s,g\),such that the Bellman optimality equation

RA∗\+VA\(s,g\)=maxa∈A\[r~A\(s,a\)\+\\displaystyle R\_\{A\}^\{\*\}\+V\_\{A\}\(s,g\)=\\max\_\{a\\in A\}\\biggl\[\\tilde\{r\}\_\{A\}\(s,a\)\+\(27\)∑s′∈𝒮P\(s′∣s,a\)∑g∈𝒢∑θ∈Θμs′\(θ′\)⋅πs′,θ′\(g′\)⋅VA\(s′,g′\)\]\\displaystyle\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}P\(s^\{\\prime\}\\mid s,a\)\\sum\_\{g\\in\\mathcal\{G\}\}\\sum\_\{\\theta\\in\\Theta\}\\mu\_\{s^\{\\prime\}\}\(\\theta^\{\\prime\}\)\\cdot\\pi\_\{s^\{\\prime\},\\theta^\{\\prime\}\}\(g^\{\\prime\}\)\\cdot V\_\{A\}\(s^\{\\prime\},g^\{\\prime\}\)\\biggl\]is satisfied where the average reward under the optimal policyρ∗\\rho^\{\*\}isRA∗R\_\{A\}^\{\*\}\.

### VII\-BDiscussion

\(C1\) ensures a unique stationary distribution and hence the average payoff does not depend on the initial state\. \(C2\) requires that the optimal discounted reward is finite for every state and discount factor\. \(C3\) assumes a reference state \(e\.g\.,\(s0,g0\)\(s\_\{0\},g\_\{0\}\)\) and a nonnegative functionVA​\(s,g\)V\_\{A\}\(s,g\)uniformly bounding the expected reward to reach it\. \(C4\) states that for each state, there is an action keeping the expected next\-stage value within a nonnegative boundM​\(s,g\)M\(s,g\)\. It also ensures that one\-step transitions are uniformly bounded across all states and actions\. These are largely observed in transmission scheduling, traffic routing and queuing models\[[33](https://arxiv.org/html/2607.13576#bib.bib100)\]\.

### VII\-CMonotone best responses and incentive compatibility

We need to prove that under sufficient set of conditions, for MLR signaling strategyπ\\pi, the monotone policya∗​\(⋅,⋅\)a^\{\*\}\(\\cdot,\\cdot\)from Theorem[3](https://arxiv.org/html/2607.13576#Thmtheorem3)is incentive compatible: for everys,gs,gand every deviation from monotone action,a′∈𝒜a^\{\\prime\}\\in\\mathcal\{A\},

QA​\(s,g,a∗​\(s,g\)\)≥QA​\(s,g,a′\)\.Q\_\{A\}\(s,g,a^\{\*\}\(s,g\)\)\\geq Q\_\{A\}\(s,g,a^\{\\prime\}\)\.The proof proceeds with the following steps:

#### VII\-C1Step 1: Posterior belief ordering from MLR

###### Lemma 5\.

Ifπ\\pisatisfies MLR ordering w\.r\.tggthen bs,gi​\(θ\)<rbs,gj​\(θ\),∀gi≤gjb\_\{s,g\_\{i\}\}\(\\theta\)<\_\{r\}b\_\{s,g\_\{j\}\}\(\\theta\),\\ \\forall g\_\{i\}\\leq g\_\{j\}

###### Proof\.

MLR ordering of signal distributions implies that posterior belief of external states are ordered in the MLR sense as the signal increases\.

πs,θ​\(gi\)\\displaystyle\\pi\_\{s,\\theta\}\(g\_\{i\}\)≤rπs,θ​\(gj\)\\displaystyle\\leq\_\{r\}\\pi\_\{s,\\theta\}\(g\_\{j\}\)πs,θm​\(gi\)⋅πs,θn​\(gj\)\\displaystyle\\pi\_\{s,\\theta\_\{m\}\}\(g\_\{i\}\)\\cdot\\pi\_\{s,\\theta\_\{n\}\}\(g\_\{j\}\)≥πs,θn​\(gi\)⋅πs,θm​\(gj\)\\displaystyle\\geq\\pi\_\{s,\\theta\_\{n\}\}\(g\_\{i\}\)\\cdot\\pi\_\{s,\\theta\_\{m\}\}\(g\_\{j\}\)i\.e\.​πs,θm​\(gi\)⋅μs​\(θm\)∑θ∈Θπs,θ​\(gi\)⋅μs​\(θ\)πs,θn​\(gi\)⋅μs​\(θn\)∑θ∈Θπs,θ​\(gi\)⋅μs​\(θ\)\\displaystyle\\textnormal\{i\.e\. \\ \}\\frac\{\\frac\{\\pi\_\{s,\\theta\_\{m\}\}\(g\_\{i\}\)\\cdot\\mu\_\{s\}\(\\theta\_\{m\}\)\}\{\\sum\_\{\\theta\\in\\Theta\}\\pi\_\{s,\\theta\}\(g\_\{i\}\)\\cdot\\mu\_\{s\}\(\\theta\)\}\}\{\\frac\{\\pi\_\{s,\\theta\_\{n\}\}\(g\_\{i\}\)\\cdot\\mu\_\{s\}\(\\theta\_\{n\}\)\}\{\\sum\_\{\\theta\\in\\Theta\}\\pi\_\{s,\\theta\}\(g\_\{i\}\)\\cdot\\mu\_\{s\}\(\\theta\)\}\}≥πs,θm​\(gj\)⋅μs​\(θm\)∑θ∈Θπs,θ​\(gj\)⋅μs​\(θ\)πs,θn​\(gj\)⋅μs​\(θn\)∑θ∈Θπs,θ​\(gj\)⋅μs​\(θ\)\\displaystyle\\geq\\frac\{\\frac\{\\pi\_\{s,\\theta\_\{m\}\}\(g\_\{j\}\)\\cdot\\mu\_\{s\}\(\\theta\_\{m\}\)\}\{\\sum\_\{\\theta\\in\\Theta\}\\pi\_\{s,\\theta\}\(g\_\{j\}\)\\cdot\\mu\_\{s\}\(\\theta\)\}\}\{\\frac\{\\pi\_\{s,\\theta\_\{n\}\}\(g\_\{j\}\)\\cdot\\mu\_\{s\}\(\\theta\_\{n\}\)\}\{\\sum\_\{\\theta\\in\\Theta\}\\pi\_\{s,\\theta\}\(g\_\{j\}\)\\cdot\\mu\_\{s\}\(\\theta\)\}\}bs,gi​\(θm\)bs,gi​\(θn\)≥bs,gj​\(θm\)bs,gj​\(θn\),\\displaystyle\\frac\{b\_\{s,g\_\{i\}\}\(\\theta\_\{m\}\)\}\{b\_\{s,g\_\{i\}\}\(\\theta\_\{n\}\)\}\\geq\\frac\{b\_\{s,g\_\{j\}\}\(\\theta\_\{m\}\)\}\{b\_\{s,g\_\{j\}\}\(\\theta\_\{n\}\)\},i\.e\.,bs,gi​\(θ\)<rbs,gj​\(θ\),∀gi≤gj\\displaystyle\\textnormal\{i\.e\.\},\\ b\_\{s,g\_\{i\}\}\(\\theta\)<\_\{r\}b\_\{s,g\_\{j\}\}\(\\theta\),\\ \\forall g\_\{i\}\\leq g\_\{j\}∎

#### VII\-C2Step 2: Ordering of agent’s value function

###### Lemma 6\.

Whenπ\\pisatisfies MLR ordering with respect toggthen the normalized measure of Bayesian update,σ′​\(s\)\\sigma^\{\\prime\}\(s\)is first\-order stochastically increasing inss, i\.e\.,

σ′​\(si\)≤sσ′​\(sj\),∀si≤sj\.\\sigma^\{\\prime\}\(s\_\{i\}\)\\leq\_\{s\}\\sigma^\{\\prime\}\(s\_\{j\}\),\\quad\\forall s\_\{i\}\\leq s\_\{j\}\.

###### Proof\.

Fix any thresholdg¯∈𝒢\\bar\{g\}\\in\\mathcal\{G\},

∑g¯Gσs′​\(g\)\\displaystyle\\sum\_\{\\bar\{g\}\}^\{G\}\\sigma^\{\\prime\}\_\{s\}\(g\)\\=∑g¯G∑θ∈Θπs,θ​\(g\)​μs​\(θ\)\\displaystyle=\\sum\_\{\\bar\{g\}\}^\{G\}\\sum\_\{\\theta\\in\\Theta\}\\pi\_\{s,\\theta\}\(g\)\\,\\mu\_\{s\}\(\\theta\)\\We haveπs,θ​\(gi\)≤rπs,θ​\(gj\)\.\\pi\_\{s,\\theta\}\(g\_\{i\}\)\\leq\_\{r\}\\pi\_\{s,\\theta\}\(g\_\{j\}\)\.Sinceμs\\mu\_\{s\}is first\-order stochastically increasing inss, it follows that

∑g¯Gσsi′​\(g\)≤∑g¯Gσsj′​\(g\)∀si≤sj\.\\sum\_\{\\bar\{g\}\}^\{G\}\\sigma^\{\\prime\}\_\{s\_\{i\}\}\(g\)\\ \\;\\leq\\;\\sum\_\{\\bar\{g\}\}^\{G\}\\sigma^\{\\prime\}\_\{s\_\{j\}\}\(g\)\\,\\quad\\forall s\_\{i\}\\leq s\_\{j\}\.Since the inequality holds for allg¯\\bar\{g\}, we conclude thatσ′​\(si\)≤sσ′​\(sj\)\\sigma^\{\\prime\}\(s\_\{i\}\)\\leq\_\{s\}\\sigma^\{\\prime\}\(s\_\{j\}\)\. This result can be used to order the agent’s value function as illustrated below:

###### Lemma 7\.

LetVA​\(s,g\)V\_\{A\}\(s,g\)be a increasing function w\.r\.tssandgg, thenV~A​\(si\)≤V~A​\(sj\)\\tilde\{V\}\_\{A\}\(s\_\{i\}\)\\leq\\tilde\{V\}\_\{A\}\(s\_\{j\}\)where

V~A​\(s\)=∑g∈𝒢∑θ∈Θμs​\(θ\)⋅πs,θ​\(g\)⋅V~A​\(s,g\)\\displaystyle\\tilde\{V\}\_\{A\}\(s\)=\\sum\_\{g\\in\\mathcal\{G\}\}\\sum\_\{\\theta\\in\\Theta\}\\mu\_\{s\}\(\\theta\)\\cdot\\pi\_\{s,\\theta\}\(g\)\\cdot\\tilde\{V\}\_\{A\}\(s,g\)σ′​\(si\)≤sσ′​\(sj\),∀si≤sj\\sigma^\{\\prime\}\(s\_\{i\}\)\\leq\_\{s\}\\sigma^\{\\prime\}\(s\_\{j\}\),\\ \\forall s\_\{i\}\\leq s\_\{j\} SinceV~A​\(s,g\)\\tilde\{V\}\_\{A\}\(s,g\)is increasing inggthen

∑g∈𝒢σ′​\(si,g\)⋅VA​\(si,g\)≤∑g∈𝒢σ′​\(sj,g\)⋅VA​\(si,g\)\\displaystyle\\sum\_\{g\\in\\mathcal\{G\}\}\\\!\\sigma^\{\\prime\}\(s\_\{i\},g\)\\\!\\cdot V\_\{A\}\(s\_\{i\},g\)\\\!\\leq\\\!\\sum\_\{g\\in\\mathcal\{G\}\}\\\!\\sigma^\{\\prime\}\(s\_\{j\},\\\!g\)\\\!\\cdot\\\!V\_\{A\}\(s\_\{i\},g\)\(28\)∑g∈𝒢σ′​\(si,g\)⋅VA​\(sj,g\)≤∑g∈𝒢σ′​\(sj,g\)⋅VA​\(sj,g\)\\displaystyle\\sum\_\{g\\in\\mathcal\{G\}\}\\sigma^\{\\prime\}\(s\_\{i\},g\)\\cdot V\_\{A\}\(s\_\{j\},g\)\\leq\\sum\_\{g\\in\\mathcal\{G\}\}\\sigma^\{\\prime\}\(s\_\{j\},g\)\\cdot V\_\{A\}\(s\_\{j\},g\)\(29\)AssumeVA​\(s,g\)V\_\{A\}\(s,g\)is increasing inssandσ′​\(si,g\)\>0\\sigma^\{\\prime\}\(s\_\{i\},g\)\>0

∑g∈𝒢σ′​\(si,g\)⋅VA​\(si,g\)≤∑g∈𝒢σ′​\(si,g\)⋅VA​\(sj,g\)\\displaystyle\\sum\_\{g\\in\\mathcal\{G\}\}\\\!\\sigma^\{\\prime\}\(s\_\{i\},g\)\\\!\\cdot V\_\{A\}\(s\_\{i\},g\)\\leq\\sum\_\{g\\in\\mathcal\{G\}\}\\\!\\sigma^\{\\prime\}\(s\_\{i\},g\)\\\!\\cdot\\\!V\_\{A\}\(s\_\{j\},g\)\(30\)Thus combining \([28](https://arxiv.org/html/2607.13576#Sx1.E28)\), \([29](https://arxiv.org/html/2607.13576#Sx1.E29)\) and \([30](https://arxiv.org/html/2607.13576#Sx1.E30)\)we have

∑g∈𝒢σ′​\(si,g\)⋅VA​\(si,g\)≤∑g∈𝒢σ′​\(sj,g\)⋅VA​\(sj,g\)\\displaystyle\\sum\_\{g\\in\\mathcal\{G\}\}\\sigma^\{\\prime\}\(s\_\{i\},g\)\\cdot V\_\{A\}\(s\_\{i\},g\)\\leq\\sum\_\{g\\in\\mathcal\{G\}\}\\sigma^\{\\prime\}\(s\_\{j\},g\)\\cdot V\_\{A\}\(s\_\{j\},g\)i\.e\.,V~A​\(si\)≤V~A​\(sj\)\\tilde\{V\}\_\{A\}\(s\_\{i\}\)\\leq\\tilde\{V\}\_\{A\}\(s\_\{j\}\)

#### VII\-C3Step 3: Supermodularity of agent’s Q value function

Under \(B1\) and \(B2\), \(A3\) and \(A4\), we haveQA​\(s,g,a\)Q\_\{A\}\(s,g,a\)is supermodular in both\(s,a\)\(s,a\)and\(g,a\)\(g,a\)\. The proof is direct from\[[36](https://arxiv.org/html/2607.13576#bib.bib206)\]with the corresponding change while considering supermodularity in reward structures\.

#### VII\-C4Step 4: Existence of monotonic policy of agent

According to Theorem[3](https://arxiv.org/html/2607.13576#Thmtheorem3), sinceQA​\(s,g,a\)Q\_\{A\}\(s,g,a\)is supermodular in\(s,a\)\(s,a\)and\(g,a\)\(g,a\), there exists an optimal policy that is monotonically increasing in the state and signal space\.

#### VII\-C5Step 5: IC compatibility of MLR signaling strategy

The agent’s best response :a∗​\(s,g\)=arg⁡maxa∈A⁡QA​\(s,g,a\)a^\{\*\}\(s,g\)=\\arg\\max\_\{a\\in A\}Q\_\{A\}\(s,g,a\)can be chosen to be monotone in bothssandgg\. Thus, given the MLR signaling strategyπ\\pi, following the principal’s recommendation is optimal and action deviation cannot yield a higher expected reward\. ThusQA​\(s,g,a∗​\(s,g\)\)≥QA​\(s,g,a′\)Q\_\{A\}\(s,g,a^\{\*\}\(s,g\)\)\\geq Q\_\{A\}\(s,g,a^\{\\prime\}\)for all action deviations from monotone responsesa′a^\{\\prime\}\. This yields*dynamic incentive compatibility*: for every timettand every realized\(st,gt\)\(s\_\{t\},g\_\{t\}\),a∗​\(st,gt\)a^\{\*\}\(s\_\{t\},g\_\{t\}\)is the best response\. Hence, the MLR signaling scheme with the corresponding monotone policy is incentive compatible for the agent\. ∎

### VII\-DProof of Theorem[1](https://arxiv.org/html/2607.13576#Thmtheorem1)

LetVP0​\(s,g\)V\_\{P\_\{0\}\}\(s,g\)be the arbitrary initial value function\. Then updatedQPQ\_\{P\}for thek\+1t​hk\+1^\{th\}iteration is given as:

QPk\+1​\(s,θ,g\)=∑a∈𝒜ρ~​\(a\|s,g\)⋅RP​\(s,θ,a\)\+\\displaystyle Q\_\{P\_\{k\+1\}\}\(s,\\theta,g\)=\\sum\_\{a\\in\\mathcal\{A\}\}\\tilde\{\\rho\}\(a\|s,g\)\\cdot R\_\{P\}\(s,\\theta,a\)\+∑a∈𝒜ρ~​\(a\|s,g\)⋅∑s′∈𝒮P​\(s,s′,a\)​∑θ′∈Θμs′​\(θ′\)⋅VPk​\(s′,θ′\)\\displaystyle\\sum\_\{a\\in\\mathcal\{A\}\}\\tilde\{\\rho\}\(a\|s,g\)\\cdot\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}P\(s,s^\{\\prime\},a\)\\sum\_\{\\theta^\{\\prime\}\\in\\Theta\}\\mu\_\{s^\{\\prime\}\}\(\\theta^\{\\prime\}\)\\cdot V\_\{P\_\{k\}\}\(s^\{\\prime\},\\theta^\{\\prime\}\)In order to prove thatQP​\(s,θ,g\)Q\_\{P\}\(s,\\theta,g\)is supermodular in\(s,g\)\(s,g\)for fixedθ\\thetaand also supermodular in\(θ,g\)\(\\theta,g\)for fixedss, using mathematical induction we proceed in two steps:

#### VII\-D1To prove value function is monotonically increasing inssandθ\\theta

###### Proof\.

Let us assume thatVPk​\(s,θ\)V\_\{P\_\{k\}\}\(s,\\theta\)is increasing inssandθ\\theta\. By Theorem[4](https://arxiv.org/html/2607.13576#Thmtheorem4),∑θ′∈Θμs′​\(θ′\)⋅VPk​\(s′,θ′\)\\sum\_\{\\theta^\{\\prime\}\\in\\Theta\}\\mu\_\{s^\{\\prime\}\}\(\\theta^\{\\prime\}\)\\cdot V\_\{P\_\{k\}\}\(s^\{\\prime\},\\theta^\{\\prime\}\)is increasing ins′s^\{\\prime\}\. By \(A4\),P\(si,\.,a\)≤sP\(sj,\.,a\),∀i≤jP\(s\_\{i\},\.,a\)\\leq\_\{s\}P\(s\_\{j\},\.,a\),\\forall i\\leq j\. Hence,∑s′∈𝒮P​\(s,s′,a\)​∑θ′∈Θμs′​\(θ′\)​VPk​\(s′,θ′\)\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}P\(s,s^\{\\prime\},a\)\\sum\_\{\\theta^\{\\prime\}\\in\\Theta\}\\mu\_\{s^\{\\prime\}\}\(\\theta^\{\\prime\}\)V\_\{P\_\{k\}\}\(s^\{\\prime\},\\theta^\{\\prime\}\)is increasing in s\. AlsoRP​\(s,θ,a\)R\_\{P\}\(s,\\theta,a\)is increasing inss,θ\\thetaandaa\. By \(A6\),ρ~​\(a\|si,g\)≤sρ~​\(a\|sj,g\),∀i<j,\\tilde\{\\rho\}\(a\|s\_\{i\},g\)\\leq\_\{s\}\\tilde\{\\rho\}\(a\|s\_\{j\},g\),\\ \\forall i<j,∑a∈𝒜ρ~​\(a\|s,g\)⋅RP​\(s,θ,a\)\\sum\_\{a\\in\\mathcal\{A\}\}\\tilde\{\\rho\}\(a\|s,g\)\\cdot R\_\{P\}\(s,\\theta,a\)is increasing withss\. Similarly∑a∈𝒜ρ~​\(a\|s,g\)⋅RP​\(s,θ,a\)\\sum\_\{a\\in\\mathcal\{A\}\}\\tilde\{\\rho\}\(a\|s,g\)\\cdot R\_\{P\}\(s,\\theta,a\)is increasing withθ\\theta\. Hence their sum,QPk​\(s,θ,g\)Q\_\{P\_\{k\}\}\(s,\\theta,g\)is also increasing inssandθ\\theta\.VPk\+1​\(s,θ\)=∑g∈𝒢QPk​\(s,θ,g\)⋅πk​\(g\|s,θ\)V\_\{P\_\{k\+1\}\}\(s,\\theta\)=\\sum\_\{g\\in\\mathcal\{G\}\}Q\_\{P\_\{k\}\}\(s,\\theta,g\)\\cdot\\pi\_\{k\}\(g\|s,\\theta\) Assumingπk​\(g\|s,θ\)\\pi\_\{k\}\(g\|s,\\theta\)is MLR with respect tossandθ\\theta,VPk\+1​\(s,θ\)V\_\{P\_\{k\+1\}\}\(s,\\theta\)is also increasing inssandθ\.\\theta\. Thus, the mathematical induction is complete\.The optimal value function is the fixed point solution of the value iteration algorithm, and hence the optimal value function is monotone and increasing inssandθ\\thetais proved\. ∎

#### VII\-D2To prove thatQP​\(s,θ,g\)Q\_\{P\}\(s,\\theta,g\)is supermodular in\(s,g\)\(s,g\)and\(θ,g\)\(\\theta,g\)

###### Proof\.

We haveμsi​\(θ\)≤rμsj​\(θ\),∀i<j\\mu\_\{\{s\_\{i\}\}\}\(\\theta\)\\leq\_\{r\}\\mu\_\{\{s\_\{j\}\}\}\(\\theta\),\\ \\forall i<jandVPk​\(s,θ,g\)V\_\{P\_\{k\}\}\(s,\\theta,g\)is increasing in\(s,θ\)\(s,\\theta\)\. Hence by Theorem[4](https://arxiv.org/html/2607.13576#Thmtheorem4), V~Pk​\(s′\)=∑θ′∈Θμs′​\(θ′\)⋅VPk​\(s′,θ′\)\\tilde\{V\}\_\{P\_\{k\}\}\(s^\{\\prime\}\)=\\sum\_\{\\theta^\{\\prime\}\\in\\Theta\}\\mu\_\{s^\{\\prime\}\}\(\\theta^\{\\prime\}\)\\cdot V\_\{P\_\{k\}\}\(s^\{\\prime\},\\theta^\{\\prime\}\)is increasing ins′s^\{\\prime\}\.P\(s,\.,a\)P\(s,\.,a\)is tail sum supermodular inssandaa\. Hence by Theorem[5](https://arxiv.org/html/2607.13576#Thmtheorem5),∑s′∈𝒮P​\(s,s′,a\)⋅V~Pk​\(s′\)\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}P\(s,s^\{\\prime\},a\)\\cdot\\tilde\{V\}\_\{P\_\{k\}\}\(s^\{\\prime\}\)is supermodular in\(s,a\)\(s,a\)\. Using \(A6\) and Theorem 5, we have ∑aρ~​\(a\|s,g\)⋅∑s′∈𝒮P​\(s,s′,a\)​∑θ′∈Θμs′​\(θ′\)⋅VPk​\(s′,θ′\)\\sum\_\{a\}\\tilde\{\\rho\}\(a\|s,g\)\\cdot\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}P\(s,s^\{\\prime\},a\)\\sum\_\{\\theta^\{\\prime\}\\in\\Theta\}\\mu\_\{s^\{\\prime\}\}\(\\theta^\{\\prime\}\)\\cdot V\_\{P\_\{k\}\}\(s^\{\\prime\},\\theta^\{\\prime\}\)is supermodular inssandgg\. By \(A2\),RP​\(s,θ,a\)R\_\{P\}\(s,\\theta,a\)is supermodular in\(s,a\)\(s,a\)and\(θ,a\)\(\\theta,a\)\. Again, we have∑aρ~​\(a\|s,g\)⋅RP​\(s,θ,a\)\\sum\_\{a\}\\tilde\{\\rho\}\(a\|s,g\)\\cdot R\_\{P\}\(s,\\theta,a\)is supermodular inssandgg\. Similarly, we have∑a∈𝒜ρ~​\(a\|s,g\)⋅RP​\(s,θ,a\)\\sum\_\{a\\in\\mathcal\{A\}\}\\tilde\{\\rho\}\(a\|s,g\)\\cdot R\_\{P\}\(s,\\theta,a\)is supermodular inθ\\thetaandgg\. Since the sum of supermodular function is supermodular we have,QPk\+1​\(s,θ,g\)=∑a∈𝒜ρ~​\(a\|s,g\)⋅RP​\(s,θ,a\)\+∑a∈𝒜ρ~​\(a\|s,g\)⋅∑s′∈𝒮P​\(s,s′,a\)​∑θ′∈Θμs′​\(θ′\)⋅VPk​\(s′,θ′\)Q\_\{P\_\{k\+1\}\}\(s,\\theta,g\)=\\sum\_\{a\\in\\mathcal\{A\}\}\\tilde\{\\rho\}\(a\|s,g\)\\cdot R\_\{P\}\(s,\\theta,a\)\+\\sum\_\{a\\in\\mathcal\{A\}\}\\tilde\{\\rho\}\(a\|s,g\)\\cdot\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}P\(s,s^\{\\prime\},a\)\\sum\_\{\\theta^\{\\prime\}\\in\\Theta\}\\mu\_\{s^\{\\prime\}\}\(\\theta^\{\\prime\}\)\\cdot V\_\{P\_\{k\}\}\(s^\{\\prime\},\\theta^\{\\prime\}\)is supermodular in\(s,θ\)\(s,\\theta\)andgg\. Hence, by mathematical induction, the Q function of the principal is supermodular in\(s,g\)\(s,g\)and\(θ,g\)\(\\theta,g\)\. ∎

### VII\-EDiscussion of assumptions for supermodular Q function

These assumptions are widely observed in transmission control, sensor scheduling, and traffic routing applications, such as prioritizing transmission in good channels, activating sensors when uncertainty is high\[[33](https://arxiv.org/html/2607.13576#bib.bib100)\],\[[36](https://arxiv.org/html/2607.13576#bib.bib206)\]\. To illustrate when these assumptions are applicable, we present a representative example of lane\-based Bayesian persuasive driving detailed in Section[VI](https://arxiv.org/html/2607.13576#S6)\. This involves a principal, i\.e\., lead vehicle, and an agent, i\.e\., vehicle connected for selection of low speed and high speed lane\. The state refers to the speed of the vehicle, and the external state indicates the extent of congestion\-free traffic\.

- •\(A1\) ensures that there is an increase in reward of the principal at higher speeds, higher congestion\-free traffic, and selection of high\-speed lanes\. It formalizes the intuition that travel rewards increase with higher vehicle speed \(state\), congestion free traffic \(external state\), and lane selection \(action\)\. Related works\[[31](https://arxiv.org/html/2607.13576#bib.bib169)\]and\[[27](https://arxiv.org/html/2607.13576#bib.bib185)\]also exploit monotonic reward structures to rank paths in dynamic traffic routing\.
- •\(A2\) assures the supermodular reward for the principal, which indicates the increased reward for higher speeds and higher congestion\-free traffic at higher actions\. It ensures the marginal benefit of taking higher action at higher states and external states\. It plays a key role in numerous applications, including traffic routing and resource allocation problems such as joint replenishment and inventory routing\[[31](https://arxiv.org/html/2607.13576#bib.bib169)\], determining optimal locations for mobile recharging stations in unmanned vehicle networks\[[43](https://arxiv.org/html/2607.13576#bib.bib171)\], and tasks like coverage maximization, adaptive ranking, and routing\[[32](https://arxiv.org/html/2607.13576#bib.bib172)\]\.
- •\(A3\) indicates the monotonic correlation between state and external state, which captures the correlation between speed and congestion\-free traffic\.
- •The higher probability of transitioning to more favourable states from higher states and actions are captured in \(A4\) and \(A5\)\. This tendency to remain at higher speeds as current speed increases— under stochastic dominance over states—is commonly modelled in predictive control frameworks\[[15](https://arxiv.org/html/2607.13576#bib.bib170)\],\[[25](https://arxiv.org/html/2607.13576#bib.bib97)\]\. Moreover transition model in\[[38](https://arxiv.org/html/2607.13576#bib.bib184)\]similarly reflects the higher likelihood of reaching greater speeds associated with different speed lanes\.
- •\(A6\) ensures that the agent’s policy is monotonically increasing with respect to state and signal spaces\[[36](https://arxiv.org/html/2607.13576#bib.bib206)\]\. In other words, higher states and signal more effectively favour higher action, which also helps in faster convergence while considering a monotonic agent\.

### VII\-FProof of Lemma 1: Softmax policy is MLR under supermodularity

###### Proof\.

We haveπ​\(g∣s,θ\)=exp⁡\(QP​\(s,θ,g\)/τ\)Z​\(s,θ\),\\pi\(g\\mid s,\\theta\)=\\frac\{\\exp\\big\(Q\_\{P\}\(s,\\theta,g\)/\\tau\\big\)\}\{Z\(s,\\theta\)\},whereZ​\(s,θ\)Z\(s,\\theta\)is∑g′∈𝒢exp⁡\(QP​\(s,θ,g′\)/τ\)\.\\sum\_\{g^\{\\prime\}\\in\\mathcal\{G\}\}\\exp\\big\(Q\_\{P\}\(s,\\theta,g^\{\\prime\}\)/\\tau\\big\)\.By supermodularity, QP​\(s,g¯\)−QP​\(s,g\)≤QP​\(s¯,g¯\)−QP​\(s¯,g\),Q\_\{P\}\(s,\\bar\{g\}\)\-Q\_\{P\}\(s,g\)\\leq Q\_\{P\}\(\\bar\{s\},\\bar\{g\}\)\-Q\_\{P\}\(\\bar\{s\},g\),and similarly withθ\\thetareplacingss\. We need to prove thatπ\\piis MLR in\(s,g\)\(s,g\)and in\(θ,g\)\(\\theta,g\)\. , i\.e\. for anys¯≥s\\bar\{s\}\\geq sand anyg1≤g2g\_\{1\}\\leq g\_\{2\}, π​\(g1∣s¯,θ\)​π​\(g2∣s,θ\)≤π​\(g1∣s,θ\)​π​\(g2∣s¯,θ\)\\pi\(g\_\{1\}\\mid\\bar\{s\},\\theta\)\\,\\pi\(g\_\{2\}\\mid s,\\theta\)\\leq\\pi\(g\_\{1\}\\mid s,\\theta\)\\,\\pi\(g\_\{2\}\\mid\\bar\{s\},\\theta\),and likewise withθ¯≥θ\\bar\{\\theta\}\\geq\\theta\. Fixθ\\thetaand lets¯≥s\\bar\{s\}\\geq s\. For anyg1≤g2g\_\{1\}\\leq g\_\{2\}, write the two products using the softmax form:

π​\(g1∣s¯,θ\)​π​\(g2∣s,θ\)=e\(Q​\(s¯,θ,g1\)\+Q​\(s,θ,g2\)\)/τZ​\(s¯,θ\)​Z​\(s,θ\),\\pi\(g\_\{1\}\\mid\\bar\{s\},\\theta\)\\pi\(g\_\{2\}\\mid s,\\theta\)=\\frac\{e^\{\(Q\(\\bar\{s\},\\theta,g\_\{1\}\)\+Q\(s,\\theta,g\_\{2\}\)\)/\\tau\}\}\{Z\(\\bar\{s\},\\theta\)Z\(s,\\theta\)\},π​\(g1∣s,θ\)​π​\(g2∣s¯,θ\)=e\(Q​\(s,θ,g1\)\+Q​\(s¯,θ,g2\)\)/τZ​\(s,θ\)​Z​\(s¯,θ\)\.\\pi\(g\_\{1\}\\mid s,\\theta\)\\pi\(g\_\{2\}\\mid\\bar\{s\},\\theta\)=\\frac\{e^\{\(Q\(s,\\theta,g\_\{1\}\)\+Q\(\\bar\{s\},\\theta,g\_\{2\}\)\)/\\tau\}\}\{Z\(s,\\theta\)Z\(\\bar\{s\},\\theta\)\}\.The normalizers cancel, so the desired inequality is equivalent toQ​\(s,θ,g2\)−Q​\(s,θ,g1\)≤Q​\(s¯,θ,g2\)−Q​\(s¯,θ,g1\),Q\(s,\\theta,g\_\{2\}\)\-Q\(s,\\theta,g\_\{1\}\)\\leq Q\(\\bar\{s\},\\theta,g\_\{2\}\)\-Q\(\\bar\{s\},\\theta,g\_\{1\}\),which is exactly the assumed supermodularity withg¯=g2\\bar\{g\}=g\_\{2\}andg=g1g=g\_\{1\}\. Similarly, withssfixed andθ¯≥θ\\bar\{\\theta\}\\geq\\thetaproves MLR in\(θ,g\)\(\\theta,g\)\. ∎

### VII\-GProof of Lemma 2

###### Proof\.

The stationary distribution satisfies the balance equation:d​\(s′;σ\)=∑s∈𝒮d​\(s;σ\)​P​\(s′∣s;σ\)\.d\(s^\{\\prime\};\\sigma\)=\\sum\_\{s\\in\\mathcal\{S\}\}d\(s;\\sigma\)P\(s^\{\\prime\}\\mid s;\\sigma\)\.Taking the gradient with respect toσ\\sigma, and then multiplying both sides withV~A​\(s′\)\\tilde\{V\}\_\{A\}\(s^\{\\prime\}\)and taking the sum we get

∑s∈𝒮∇σd​\(s′;σ\)⋅V~A​\(s′\)=\\displaystyle\\sum\_\{s\\in\\mathcal\{S\}\}\\nabla\_\{\\sigma\}d\(s^\{\\prime\};\\sigma\)\\cdot\\tilde\{V\}\_\{A\}\(s^\{\\prime\}\)=∑s∈𝒮∑s′∈𝒮∇σd​\(s;σ\)​P​\(s′∣s;σ\)⋅V~A​\(s′\)\+\\displaystyle\\sum\_\{s\\in\\mathcal\{S\}\}\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}\\nabla\_\{\\sigma\}d\(s;\\sigma\)P\(s^\{\\prime\}\\mid s;\\sigma\)\\cdot\\tilde\{V\}\_\{A\}\(s^\{\\prime\}\)\+∑s∈𝒮∑s′∈𝒮d​\(s;σ\)​∇σP​\(s′∣s;σ\)⋅V~A​\(s′\)\\displaystyle\\sum\_\{s\\in\\mathcal\{S\}\}\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}d\(s;\\sigma\)\\nabla\_\{\\sigma\}P\(s^\{\\prime\}\\mid s;\\sigma\)\\cdot\\tilde\{V\}\_\{A\}\(s^\{\\prime\}\)\\,whereV~A​\(s′\)=∑g′∈𝒢∑θ′∈Θμs′​\(θ′\)⋅πs′,θ′​\(g′\)⋅VA​\(s′,g′\)\\tilde\{V\}\_\{A\}\(s^\{\\prime\}\)=\\sum\_\{g^\{\\prime\}\\in\\mathcal\{G\}\}\\sum\_\{\\theta^\{\\prime\}\\in\\Theta\}\\mu\_\{s^\{\\prime\}\}\(\\theta^\{\\prime\}\)\\cdot\\pi\_\{s^\{\\prime\},\\theta^\{\\prime\}\}\(g^\{\\prime\}\)\\cdot V\_\{A\}\(s^\{\\prime\},g^\{\\prime\}\)We have,​r~A​\(s\)−R~A=V~A​\(s\)−∑s′∈𝒮P​\(s′∣s\)​V~A​\(s′\)\\text\{We have, \}\\tilde\{r\}\_\{A\}\(s\)\-\\tilde\{R\}\_\{A\}=\\tilde\{V\}\_\{A\}\(s\)\-\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}P\(s^\{\\prime\}\\mid s\)\\tilde\{V\}\_\{A\}\(s^\{\\prime\}\)\\,

Therefore,∑s∈𝒮∇d​\(s\)⋅r~A​\(s\)−∑s∈𝒮∇d​\(s\)⋅R~A=\\displaystyle\\sum\_\{s\\in\\mathcal\{S\}\}\\nabla d\(s\)\\cdot\\tilde\{r\}\_\{A\}\(s\)\-\\sum\_\{s\\in\\mathcal\{S\}\}\\nabla d\(s\)\\cdot\\tilde\{R\}\_\{A\}=∑s∈𝒮∇d​\(s\)⋅V~A​\(s\)−∑s∈𝒮∇d​\(s\)⋅∑P​\(s′∣s\)​V~A​\(s′\)\\displaystyle\\sum\_\{s\\in\\mathcal\{S\}\}\\nabla d\(s\)\\cdot\\tilde\{V\}\_\{A\}\(s\)\-\\sum\_\{s\\in\\mathcal\{S\}\}\\nabla d\(s\)\\cdot\\sum P\(s^\{\\prime\}\\mid s\)\\tilde\{V\}\_\{A\}\(s^\{\\prime\}\)\\,Also,R~A​\(σ\)=∑s∈𝒮d​\(s;σ\)​r~A​\(s\)​, and hence\\displaystyle\\tilde\{R\}\_\{A\}\(\\sigma\)=\\sum\_\{s\\in\\mathcal\{S\}\}d\(s;\\sigma\)\\tilde\{r\}\_\{A\}\(s\)\\text\{, and hence\}∇σR~A=\\displaystyle\\nabla\_\{\\sigma\}\\tilde\{R\}\_\{A\}=∑s∈𝒮d​\(s;σ\)⋅∇σr~A​\(s\)\+∑s∈𝒮∑s′∈𝒮d​\(s;σ\)​∇σP​\(s′∣s;σ\)⋅V~A​\(s′\)\\displaystyle\\sum\_\{s\\in\\mathcal\{S\}\}d\(s;\\sigma\)\\cdot\\nabla\_\{\\sigma\}\\tilde\{r\}\_\{A\}\(s\)\+\\sum\_\{s\\in\\mathcal\{S\}\}\\sum\_\{s^\{\\prime\}\\in\\mathcal\{S\}\}d\(s;\\sigma\)\\nabla\_\{\\sigma\}P\(s^\{\\prime\}\\mid s;\\sigma\)\\cdot\\tilde\{V\}\_\{A\}\(s^\{\\prime\}\)∎

## References

- \[1\]A\. Alizadeh, M\. Moghadam, Y\. Bicer, N\. K\. Ure, U\. Yavas, and C\. Kurtulus\(2019\)Automated lane change decision making using deep reinforcement learning in dynamic and uncertain highway environment\.InProc\. Intelligent Transportation Systems Conference \(ITSC\), 1399–1404,Cited by:[§II](https://arxiv.org/html/2607.13576#S2.SS0.SSS0.Px2.p1.1)\.
- \[2\]A\. Aprem, C\. R\. Murthy, and N\. B\. Mehta\(2013\)Transmit power control policies for energy harvesting sensors with retransmissions\.IEEE Journal of Selected Topics in Signal Processing7\(5\),pp\. 895–906\.Cited by:[§II](https://arxiv.org/html/2607.13576#S2.SS0.SSS0.Px7.p1.1),[§IV\-A](https://arxiv.org/html/2607.13576#S4.SS1.p1.1)\.
- \[3\]M\. Bernasconi, M\. Castiglioni, A\. Marchesi, N\. Gatti, and F\. Trovò\(2022\)Sequential information design: learning to persuade in the dark\.Advances in Neural Information Processing Systems35,pp\. 15917–15928\.Cited by:[item 5ii](https://arxiv.org/html/2607.13576#S1.I1.i5.I1.i2.1),[item 2](https://arxiv.org/html/2607.13576#S6.I4.i2.p1.1)\.
- \[4\]M\. Bernasconi, M\. Castiglioni, A\. Marchesi, N\. Gatti, and F\. Trovò\(2025\)Online learning in sequential bayesian persuasion: handling unknown priors\.Artificial Intelligence338,pp\. 104245\.Cited by:[§II](https://arxiv.org/html/2607.13576#S2.SS0.SSS0.Px5.p1.1)\.
- \[5\]M\. Bernasconi, M\. Castiglioni, A\. Marchesi, and M\. Mutti\(2023\)Persuading farsighted receivers in MDPs: the power of honesty\.Advances in Neural Information Processing Systems36,pp\. 14987–15014\.Cited by:[§II](https://arxiv.org/html/2607.13576#S2.SS0.SSS0.Px5.p1.1)\.
- \[6\]R\. Boleslavsky and C\. Cotton\(2015\)Grading standards and education quality\.American Economic Journal: Microeconomics7\(2\),pp\. 248–279\.Cited by:[§I](https://arxiv.org/html/2607.13576#S1.p2.1)\.
- \[7\]S\. J\. Brams and M\. Ismail\(2019\)Farsightedness in games: stabilizing cooperation in international conflict\.Available at SSRN 3313025\.Cited by:[§II](https://arxiv.org/html/2607.13576#S2.SS0.SSS0.Px8.p1.1)\.
- \[8\]Y\. Chen and T\. Lin\(2023\)Persuading a behavioral agent: approximately best responding and learning\.arXiv preprint\.Cited by:[§III\-C](https://arxiv.org/html/2607.13576#S3.SS3.p3.4)\.
- \[9\]S\. Das, E\. Kamenica, and R\. Mirka\(2017\)Reducing congestion through information design\.InProc\. 55th Annual Allerton Conference on Communication, Control and Computing \(Allerton\), 1279–1284,Cited by:[item 1](https://arxiv.org/html/2607.13576#S6.I1.i1.p1.1),[§VI\-A](https://arxiv.org/html/2607.13576#S6.SS1.p1.2)\.
- \[10\]D\. V\. Djonin and V\. Krishnamurthy\(2007\)Q\-learning algorithms for constrained Markov decision processes with randomized monotone policies: application to MIMO transmission control\.IEEE Transactions on Signal Processing55\(5\),pp\. 2170–2181\.Cited by:[§II](https://arxiv.org/html/2607.13576#S2.SS0.SSS0.Px7.p1.1),[§V\-D](https://arxiv.org/html/2607.13576#S5.SS4.p1.3)\.
- \[11\]S\. Dughmi and H\. Xu\(2016\)Algorithmic Bayesian persuasion\.InProceedings of the Forty\-Eighth Annual ACM Symposium on Theory of Computing, 412–425,Cited by:[§II](https://arxiv.org/html/2607.13576#S2.SS0.SSS0.Px7.p1.1)\.
- \[12\]J\. C\. Ely\(2017\)Beeps\.American Economic Review107\(1\),pp\. 31–53\.Cited by:[§II](https://arxiv.org/html/2607.13576#S2.SS0.SSS0.Px4.p1.1)\.
- \[13\]J\. Gan, R\. Majumdar, G\. Radanovic, and A\. Singla\(2022\)Bayesian persuasion in sequential decision\-making\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.36\.Cited by:[item 5ii](https://arxiv.org/html/2607.13576#S1.I1.i5.I1.i2.1),[§I](https://arxiv.org/html/2607.13576#S1.p2.1),[§I](https://arxiv.org/html/2607.13576#S1.p4.1),[§II](https://arxiv.org/html/2607.13576#S2.SS0.SSS0.Px3.p1.1),[§II](https://arxiv.org/html/2607.13576#S2.SS0.SSS0.Px4.p1.1),[§II](https://arxiv.org/html/2607.13576#S2.SS0.SSS0.Px5.p1.1),[6th item](https://arxiv.org/html/2607.13576#S3.I1.i6.p1.6),[§III\-D](https://arxiv.org/html/2607.13576#S3.SS4.SSS0.Px1.p1.10),[§III\-D](https://arxiv.org/html/2607.13576#S3.SS4.SSS0.Px1.p1.6)\.
- \[14\]I\. Goldstein and Y\. Leitner\(2018\)Stress tests and information disclosure\.Journal of Economic Theory177,pp\. 34–69\.Cited by:[§I](https://arxiv.org/html/2607.13576#S1.p5.1),[§II](https://arxiv.org/html/2607.13576#S2.SS0.SSS0.Px7.p1.1)\.
- \[15\]C\. Guo, R\. Xu, B\. Yang, Y\. Yuan, T\. Kieu, Y\. Zhao, and C\. S\. Jensen\(2024\)Efficient stochastic routing in path\-centric uncertain road networks\.Proceedings of the VLDB Endowment17\(11\),pp\. 2893–2905\.Cited by:[4th item](https://arxiv.org/html/2607.13576#Sx1.I3.i4.p1.1)\.
- \[16\]K\. He, Z\. Xu, P\. Wang, L\. Deng, and L\. Tu\(2015\)Congestion avoidance routing based on large\-scale social signals\.IEEE Transactions on Intelligent Transportation Systems17\(9\),pp\. 2613–2626\.Cited by:[7th item](https://arxiv.org/html/2607.13576#S6.I2.i7.p1.1)\.
- \[17\]L\. Hewing, K\. P\. Wabersich, M\. Menner, and M\. N\. Zeilinger\(2020\)Learning\-based model predictive control: toward safe learning in control\.Annual Review of Control, Robotics, and Autonomous Systems3\(1\),pp\. 269–296\.Cited by:[item 1](https://arxiv.org/html/2607.13576#S6.I3.i1.p1.1)\.
- \[18\]M\. Kamrani, A\. R\. Srinivasan, S\. Chakraborty, and A\. J\. Khattak\(2020\)Applying Markov decision process to understand driving decisions using basic safety messages data\.Transportation Research Part C: Emerging Technologies115,pp\. 102642\.Cited by:[1st item](https://arxiv.org/html/2607.13576#S3.I1.i1.p1.1),[4th item](https://arxiv.org/html/2607.13576#S3.I1.i4.p1.1)\.
- \[19\]V\. Krishnamurthy, A\. Aprem, and S\. Bhatt\(2018\)Multiple stopping time POMDPs: Structural results & application in interactive advertising on social media\.Automatica95,pp\. 385–398\.Cited by:[§II](https://arxiv.org/html/2607.13576#S2.SS0.SSS0.Px7.p1.1)\.
- \[20\]V\. Krishnamurthy\(2016\)Partially observed Markov decision processes\.Cambridge university press\.Cited by:[§III\-D](https://arxiv.org/html/2607.13576#S3.SS4.SSS0.Px1.p1.8),[§IV\-A](https://arxiv.org/html/2607.13576#S4.SS1.p1.1),[§V\-D](https://arxiv.org/html/2607.13576#S5.SS4.p1.11),[§V\-D](https://arxiv.org/html/2607.13576#S5.SS4.p1.3),[§V\-D](https://arxiv.org/html/2607.13576#S5.SS4.p1.9),[Theorem 4](https://arxiv.org/html/2607.13576#Thmtheorem4.p1.3.3),[Theorem 5](https://arxiv.org/html/2607.13576#Thmtheorem5.p1.4.4)\.
- \[21\]N\. Levering, M\. Boon, M\. Mandjes, and R\. Núñez\-Queija\(2022\)A framework for efficient dynamic routing under stochastically varying conditions\.Transportation research part B: methodological160,pp\. 97–124\.Cited by:[item 2](https://arxiv.org/html/2607.13576#S6.I3.i2.p1.1)\.
- \[22\]Y\. Lin, W\. Li, H\. Zha, and B\. Wang\(2023\)Information design in multi\-agent reinforcement learning\.Advances in Neural Information Processing Systems36,pp\. 25584–25597\.Cited by:[§III\-C](https://arxiv.org/html/2607.13576#S3.SS3.p1.6)\.
- \[23\]D\. Lingenbrink and K\. Iyer\(2018\)Optimal signaling mechanisms in unobservable queues\.Available at SSRN 2964093\.Cited by:[item 3](https://arxiv.org/html/2607.13576#S6.I4.i3.p1.1),[item 5](https://arxiv.org/html/2607.13576#S6.I4.i5.p1.1)\.
- \[24\]D\. Lingenbrink and K\. Iyer\(2019\)Optimal signaling mechanisms in unobservable queues\.Operations research67\(5\),pp\. 1397–1416\.Cited by:[§I](https://arxiv.org/html/2607.13576#S1.p2.1)\.
- \[25\]K\. Liu, Z\. Asher, X\. Gong, M\. Huang, and I\. Kolmanovsky\(2019\)Vehicle velocity prediction and energy management strategy part 1: deterministic and stochastic vehicle velocity prediction using machine learning\.Technical reportSAE Technical Paper\.Cited by:[4th item](https://arxiv.org/html/2607.13576#Sx1.I3.i4.p1.1)\.
- \[26\]Y\. Liu and A\. B\. Whinston\(2019\)Efficient real\-time routing for autonomous vehicles through Bayes correlated equilibrium: An information design framework\.Information Economics and Policy47,pp\. 14–26\.Cited by:[§II](https://arxiv.org/html/2607.13576#S2.SS0.SSS0.Px2.p1.1),[§II](https://arxiv.org/html/2607.13576#S2.SS0.SSS0.Px4.p1.1),[§III\-D](https://arxiv.org/html/2607.13576#S3.SS4.p1.1)\.
- \[27\]R\. P\. Loui\(1983\)Optimal paths in graphs with stochastic or multidimensional weights\.Communications of the ACM26\(9\),pp\. 670–676\.Cited by:[1st item](https://arxiv.org/html/2607.13576#Sx1.I3.i1.p1.1)\.
- \[28\]A\. Ma and L\. Xiao\(2025\)When the line says more: optimal signaling in queues with congestion\-driven expectations\.International Applied Mathematics, Modelling and Simulation Conference\.Cited by:[§I](https://arxiv.org/html/2607.13576#S1.p5.1)\.
- \[29\]Y\. Mansour, A\. Slivkins, V\. Syrgkanis, and Z\. S\. Wu\(2022\)Bayesian exploration: incentivizing exploration in Bayesian games\.Operations Research70\(2\),pp\. 1105–1127\.Cited by:[§I](https://arxiv.org/html/2607.13576#S1.p2.1)\.
- \[30\]P\. Marbach and J\. N\. Tsitsiklis\(2001\)Simulation\-based optimization of Markov reward processes\.IEEE Transactions on Automatic Control46\(2\),pp\. 191–209\.Cited by:[§V\-A](https://arxiv.org/html/2607.13576#S5.SS1.p1.1),[§V\-B](https://arxiv.org/html/2607.13576#S5.SS2.p1.4),[§V\-B](https://arxiv.org/html/2607.13576#S5.SS2.p1.9)\.
- \[31\]V\. Nagarajan and C\. Shi\(2016\)Approximation algorithms for inventory problems with submodular or routing costs\.Mathematical Programming160,pp\. 225–244\.Cited by:[1st item](https://arxiv.org/html/2607.13576#Sx1.I3.i1.p1.1),[2nd item](https://arxiv.org/html/2607.13576#Sx1.I3.i2.p1.1)\.
- \[32\]F\. Navidi, P\. Kambadur, and V\. Nagarajan\(2020\)Adaptive submodular ranking and routing\.Operations Research68\(3\),pp\. 856–877\.Cited by:[2nd item](https://arxiv.org/html/2607.13576#Sx1.I3.i2.p1.1)\.
- \[33\]M\. H\. Ngo and V\. Krishnamurthy\(2009\)Monotonicity of constrained optimal transmission policies in correlated fading channels with ARQ\.IEEE Transactions on Signal Processing58\(1\),pp\. 438–451\.Cited by:[§VII\-B](https://arxiv.org/html/2607.13576#Sx1.SS2.p1.3),[§VII\-E](https://arxiv.org/html/2607.13576#Sx1.SS5.p1.1)\.
- \[34\]M\. H\. Ngo and V\. Krishnamurthy\(2009\)Optimality of threshold policies for transmission scheduling in correlated fading channels\.IEEE Transactions on Communications57\(8\),pp\. 2474–2483\.Cited by:[§II](https://arxiv.org/html/2607.13576#S2.SS0.SSS0.Px7.p1.1)\.
- \[35\]H\. Pan, X\. Yu, L\. Gao, Y\. Li, and W\. Hua\(2022\)Model predictive control\-based multivariable controller for traffic flows in automated freeway systems\.IEEE Intelligent Transportation Systems Magazine15\(2\),pp\. 149–161\.Cited by:[2nd item](https://arxiv.org/html/2607.13576#S6.I2.i2.p1.7),[item 1](https://arxiv.org/html/2607.13576#S6.I3.i1.p1.1)\.
- \[36\]M\. Paul and A\. Aprem\(2024\)Monotonic agent policies in dynamic bayesian persuasion: application to intelligent traffic routing\.TechRxiv\.Cited by:[§I](https://arxiv.org/html/2607.13576#S1.p5.1),[§IV\-A](https://arxiv.org/html/2607.13576#S4.SS1.p1.1),[5th item](https://arxiv.org/html/2607.13576#Sx1.I3.i5.p1.1),[§VII\-C3](https://arxiv.org/html/2607.13576#Sx1.SS3.SSS3.p1.3),[§VII\-E](https://arxiv.org/html/2607.13576#Sx1.SS5.p1.1)\.
- \[37\]C\. Peng and M\. Tomizuka\(2019\)Bayesian persuasive driving\.In2019 American Control Conference \(ACC\),Cited by:[§II](https://arxiv.org/html/2607.13576#S2.SS0.SSS0.Px2.p1.1),[§II](https://arxiv.org/html/2607.13576#S2.SS0.SSS0.Px6.p1.1),[§II](https://arxiv.org/html/2607.13576#S2.p1.1),[3rd item](https://arxiv.org/html/2607.13576#S3.I1.i3.p1.1),[§VI\-A](https://arxiv.org/html/2607.13576#S6.SS1.p1.2)\.
- \[38\]A\. Prabu, N\. Ravi, and L\. Li\(2022\)A novel method for lane\-change maneuver in urban driving using predictive Markov decision process\.arXiv preprint\.Cited by:[item 1](https://arxiv.org/html/2607.13576#S6.I3.i1.p1.1),[4th item](https://arxiv.org/html/2607.13576#Sx1.I3.i4.p1.1)\.
- \[39\]M\. L\. Puterman\(1990\)Markov Decision Processes\.Handbooks in operations research and management science2,pp\. 331–434\.Cited by:[§III\-B](https://arxiv.org/html/2607.13576#S3.SS2.SSS0.Px1.p1.27),[§IV\-A](https://arxiv.org/html/2607.13576#S4.SS1.p1.1)\.
- \[40\]M\. Ramezani and N\. Geroliminis\(2012\)On the estimation of arterial route travel time distribution with Markov chains\.Transportation Research Part B: Methodological46\(10\),pp\. 1576–1590\.Cited by:[item 1](https://arxiv.org/html/2607.13576#S6.I3.i1.p1.1)\.
- \[41\]A\. Sauré, J\. Patrick, and M\. L\. Puterman\(2015\)Simulation\-based approximate policy iteration with generalized logistic functions\.INFORMS Journal on Computing27\(3\),pp\. 579–595\.Cited by:[§V\-C](https://arxiv.org/html/2607.13576#S5.SS3.p1.9)\.
- \[42\]L\. I\. Sennott\(1989\)Average cost optimal stationary policies in infinite state Markov decision processes with unbounded costs\.Operations Research37\(4\),pp\. 626–633\.Cited by:[§VII\-A](https://arxiv.org/html/2607.13576#Sx1.SS1.p3.1)\.
- \[43\]G\. Shi and P\. Tokekar\(2023\)Decision\-oriented learning with differentiable submodular maximization for vehicle routing problem\.In2023 IEEE/RSJ International Conference on Intelligent Robots and Systems \(IROS\),Cited by:[2nd item](https://arxiv.org/html/2607.13576#Sx1.I3.i2.p1.1)\.
- \[44\]M\. W\. Ulmer, D\. C\. Mattfeld, and F\. Köster\(2018\)Budgeting time for dynamic vehicle routing with stochastic customer requests\.Transportation Science52\(1\),pp\. 20–37\.Cited by:[item 3](https://arxiv.org/html/2607.13576#S6.I3.i3.p1.1)\.
- \[45\]C\. Wei, F\. Hui, and A\. J\. Khattak\(2021\)Driver lane\-changing behavior prediction based on deep learning\.Journal of advanced transportation\(1\),pp\. 6676092\.Cited by:[§II](https://arxiv.org/html/2607.13576#S2.SS0.SSS0.Px2.p1.1)\.
- \[46\]J\. Wu, Z\. Zhang, Z\. Feng, Z\. Wang, Z\. Yang, M\. I\. Jordan, and H\. Xu\(2022\)Sequential information design: Markov persuasion process and its efficient reinforcement learning\.arXiv:2202\.10678\.Cited by:[item 5ii](https://arxiv.org/html/2607.13576#S1.I1.i5.I1.i2.1),[§II](https://arxiv.org/html/2607.13576#S2.SS0.SSS0.Px4.p1.1),[§II](https://arxiv.org/html/2607.13576#S2.SS0.SSS0.Px5.p1.1),[§II](https://arxiv.org/html/2607.13576#S2.SS0.SSS0.Px6.p1.1),[item 1](https://arxiv.org/html/2607.13576#S6.I4.i1.p1.1)\.
- \[47\]S\. Wu, K\. Ding, P\. Cheng, and L\. Shi\(2020\)Optimal scheduling of multiple sensors over lossy and bandwidth limited channels\.IEEE Transactions on Control of Network Systems7\(3\),pp\. 1188–1200\.Cited by:[§II](https://arxiv.org/html/2607.13576#S2.SS0.SSS0.Px7.p1.1)\.
- \[48\]M\. Yang, X\. Wang, and M\. Quddus\(2019\)Examining lane change gap acceptance, duration and impact using naturalistic driving data\.Transportation research part C: emerging technologies104,pp\. 317–331\.Cited by:[§II](https://arxiv.org/html/2607.13576#S2.SS0.SSS0.Px1.p1.1)\.
- \[49\]X\. Yang, J\. Hu, and J\. Hu\(2024\)Relative Q\-learning for average\-reward Markov decision processes with continuous states\.IEEE Transactions on Automatic Control\.Cited by:[§V\-D](https://arxiv.org/html/2607.13576#S5.SS4.p1.12)\.
- \[50\]Y\. Zhang, M\. Hu, Y\. Chen, and C\. Shi\(2023\)Cooperative platoon forming strategy for connected autonomous vehicles in mixed traffic flow\.Physica A: Statistical Mechanics and its Applications623,pp\. 128828\.Cited by:[§I](https://arxiv.org/html/2607.13576#S1.p1.1),[§II](https://arxiv.org/html/2607.13576#S2.p1.1)\.
- \[51\]Y\. Zhang, Q\. Lin, J\. Wang, S\. Verwer, and J\. M\. Dolan\(2018\)Lane\-change intention estimation for car\-following control in autonomous driving\.IEEE Transactions on Intelligent Vehicles3\(3\),pp\. 276–286\.Cited by:[item 1](https://arxiv.org/html/2607.13576#S6.I3.i1.p1.1)\.
- \[52\]J\. Zhao, Y\. Wu, R\. Deng, S\. Xu, J\. Gao, and A\. Burke\(2025\)A survey of autonomous driving from a deep learning perspective\.ACM Computing Surveys57\(10\),pp\. 1–60\.Cited by:[§II](https://arxiv.org/html/2607.13576#S2.SS0.SSS0.Px1.p1.1),[§II](https://arxiv.org/html/2607.13576#S2.SS0.SSS0.Px2.p1.1)\.