Smart charging of large fleets of Electric Vehicles: Independent Multi-Agent Reinforcement Learning approaches

# Smart charging of large fleets of Electric Vehicles: Independent Multi-Agent Reinforcement Learning approaches
Source: [https://arxiv.org/html/2606.31347](https://arxiv.org/html/2606.31347)
11institutetext:Orange Research11email:@orange\.com22institutetext:ENS Rennes22email:@ens\-rennes\.fr33institutetext:IMT Atlantique33email:@imt\-atlantique\.fr44institutetext:IRIT44email:@irit\.frEloann Le GuernRaphaël FéraudFatma SalemMelissa ChiknounEymeric GiabicaniMehdi FekiPatrick MailléGuy CamilleriAnne BlavetteHamid Benhamed

###### Abstract

The electrification of transportation through electric vehicles introduces new challenges for power grid management, such as increased peak demand, voltage fluctuations, line overloads, and the integration of variable renewable energy sources\. To enable efficient integration of EVs while minimizing costs for users and avoiding network overloads, implicit coordination between EVs is required\. This work compares two independent multi\-agent reinforcement learning approaches for optimizing such decentralized EV charging: contextual combinatorial bandits and policy gradient algorithms\. Using a realistic simulation environment with autonomous agents making decisions based on local environmental information \(including price signals, state\-of\-charge, and temporal constraints\), we evaluate their performance across varying congestion levels, and mixed\-strategy configurations with heterogeneous agent groups under dynamic electricity pricing derived from real photovoltaic production data\.

## 1Introduction

The widespread adoption of Electric Vehicles \(EVs\) poses significant challenges for modern electrical grids\. With 17 million units sold in 2024, representing over 20% of the market share, EVs are expected to continue their growth and account for 40% of automotive sales by 2030\[iea2025\]\. This rapid expansion has consequences for electrical networks because the global adoption of EVs comes with a rising electricity demand\. In 2024, the global fleet of EVs consumed around 180 TWh of electricity, equivalent to Argentina’s annual electricity consumption\[iea2025\]\.

While this growth is essential for global commitments to achieve carbon neutrality by 2050, it also raises concerns about grid stability\. Uncoordinated charging during peak hours can cause voltage drops, line overloads, and network instability\[tirunagari2022\]\.

Moreover, renewable energy sources often experience curtailment, with several GWh of clean electricity lost due to insufficient demand during peak production periods and the lack of energy storage solutions\. For example, California’s Independent System Operator regularly curtails renewable generation when supply exceeds demand, with solar curtailment reaching over 900 GWh in peak months during spring and fall\[caiso2025\]\. In extreme cases, energy producers are even paid to stop generating electricity to prevent grid overload\[bbc2024\]\. EV batteries can be leveraged as flexible storage to absorb excess energy and mitigate such curtailment issues\. Smart charging is therefore a promising solution, as it can reduce grid pressure while maximizing renewable energy utilization by utilizing the storage capacity of EV batteries\. However, optimizing charging schedules for EV fleets requires handling two major issues:

- •scalability: the number of EVs is very large, which requires scalable algorithms,
- •uncertainty: the production of renewable energy and the consumption of energy depend on the weather, while the availability of EVs depends on unknown human constraints\.

While centralized optimization, such as mixed\-integer linear programming\[Franco2015\], requires full information about all EVs \(arrival times, departure times, state of charges\), prices, grid load, which is often impractical due to temporal, scalability, privacy, and communication constraints, fully decentralized approaches can scale with any number of EVs by allowing each EV to take decisions based on local and partial information\. That is why, in this paper we will focus on fully decentralized approaches\.

Reinforcement learning \(RL\)\[sutton2018\]can optimize the scheduling of charging EVs, while handling uncertainties arising from EV owners’ behavior, variability in renewable production and energy consumption\. Multi\-Agent Reinforcement Learning \(MARL\) has become a popular approach for EV charging coordination\[xie2025\]\. MARL algorithms based on the concept of centralized training and decentralized exploration\[rashid2018,lowe2017\]are efficient in collaborative setting\. However, the centralized critic needs the observations and the actions of all agents, which limits their scalability when the number of agents becomes large\. Independent Multi\-Agent Reinforcement Learning\[tan1993\]is a fully decentralized approach that can scale with any number of EVs, and is well suited for competitive setting: each agent has its own policy\. Proximal Policy Optimization \(PPO\)\[schulman2017proximalpolicyoptimizationalgorithms\], which is an improvement over Advantage Actor Critic \(A2C\)\[a2C\], is one of the most popular reinforcement algorithm due to its excellent compromise between performance and simplicity, making it a good candidate for each independent agent, while Simple Policy Optimization \(SPO\)\[SPO\]is a recent version of PPO equipped with theoretical guarantees\. Based on a simplified version of reinforcement learning problem, Multi\-armed bandits\[lattimore2020\]benefit from a lighter and faster training that is a key advantage for real\-world use in a changing environment\. Multi\-Agent Multi\-armed bandits have been successfully tested for optimizing communications in IoT networks\[Bonnefoi2018,Kerkouche2018,DAKDOUK2023\], and then for EV charging\[zafar2026,leguerndallo:hal\-05262742\]\. However, previous studies focus on just one approach without comparing different algorithms\. Moreover, real\-world deployments will inevitably involve heterogeneous vehicle populations using different charging algorithms\. As in Independent MARL, other agents are seen as a part of the environment, this heterogeneity between charging algorithms can change their performances\.

In this paper, for bridging the gap, we investigate and compare the performances of Independent MARL based on modern policy gradient algorithms such as PPO\[schulman2017proximalpolicyoptimizationalgorithms\], A2C\[a2C\], and SPO\[SPO\]with Linear Thompson Sampling\[DBLP:journals/corr/abs\-1209\-3352\]for charging large fleets of vehicles\. To assess algorithm robustness in real\-world environment, we also evaluate mixed deployments of algorithms\.

## 2Problem Setting

An electrical grid is a large graph that connects power plants to each customer\. It involves different actors \(producers, transporters, distributors, and consumers\), each having different goals and constraints\. This study focuses on the decentralized smart charging of multiple electric vehicles connected to an electrical grid\. We consider only one actor: the owners of electric vehicles, who seek to charge their vehicles at the lowest possible rate, when available\. The electric grid constraints are taken into account through congestion\.

### 2\.1System Model

#### 2\.1\.1Electrical grid\.

We consider an elementary electrical grid that consists ofNNelectric vehicles connected to a transformer\. The electrical grid is modeled without losses to focus on vehicle coordination, with total load being the sum of charging powers\. The transformer has the capacity to charge at the same timeKKEVs, leading to congestion when demand exceeds this capacity \(N\>KN\>K\)\.

#### 2\.1\.2Electric Vehicle\.

We consider an episodic structure of time, where each dayd∈\[D\]d\\in\[D\]is discretized intoTTintervals\. An EVn∈\[N\]n\\in\[N\]is available for charging during the time slots\{adn,…,bdn\}∈\[T\]2\\\{a\_\{d\}^\{n\},\.\.\.,b\_\{d\}^\{n\}\\\}\\in\[T\]^\{2\}, whereadna\_\{d\}^\{n\}andbdnb\_\{d\}^\{n\}are respectively the arrival and departure time slots,adn<bdna\_\{d\}^\{n\}<b\_\{d\}^\{n\}\. Arrival and departure time slots are dependent random variables\. Each EVn∈\[N\]n\\in\[N\]needsmdnm\_\{d\}^\{n\}time slots to complete its charge during the day\.mdnm\_\{d\}^\{n\}is also a random variable\.

#### 2\.1\.3Pricing Model\.

We model the price at time slotttas:

p​\(t\)=1−PP​V​\(t\)P¯p\(t\)=1\-\\frac\{P\_\{PV\}\(t\)\}\{\\bar\{P\}\}\(1\)wherePP​V​\(t\)P\_\{PV\}\(t\)is the photovoltaic production at time slottt, normalized to the maximum of photovoltaic productionP¯\\bar\{P\}\. This pricing mechanism is designed to create incentives for renewable energy utilization: when solar production is at its peak \(PP​V​\(t\)=P¯P\_\{PV\}\(t\)=\\bar\{P\}\), the price is minimal \(p​\(t\)=0p\(t\)=0\), and when solar production is zero, the price reaches its maximum \(p​\(t\)=1p\(t\)=1\)\. Notice, that the price depends on weather, and hence it is a random variable\.

### 2\.2Problem Formulation

At the first glance, the decentralized smart charging problem can be viewed as a repeated planning problem overDDdays, under uncertainty\.

Each dayd∈\[D\]d\\in\[D\], each EVn∈\[N\]n\\in\[N\]aims to find a schedule of charging that allows to complete its charge at the end of the day, while minimizing its cost of charge\. Due to the availability of each EVnn, and its initial state of charge, only some schedules𝐬dn∈\{0,1\}T\\mathbf\{s\}^\{n\}\_\{d\}\\in\\\{0,1\\\}^\{T\}are feasible:

𝒮dn=\{𝐬dn∈\{0,1\}T,supp​\(𝐬dn\)⊆\{adn,…,bdn\},∥𝐬dn∥1=mdn\},\\mathcal\{S\}\_\{d\}^\{n\}=\\left\\\{\\mathbf\{s\}^\{n\}\_\{d\}\\in\\\{0,1\\\}^\{T\},\\text\{supp\}\(\\mathbf\{s\}^\{n\}\_\{d\}\)\\subseteq\\\{a^\{n\}\_\{d\},\.\.\.,b^\{n\}\_\{d\}\\\},\\lVert\\mathbf\{s\}^\{n\}\_\{d\}\\rVert\_\{1\}=m^\{n\}\_\{d\}\\right\\\},\(2\)wheresupp​\(𝐬dn\)=\{t∈\[T\],𝐬dn​\(t\)=1\}\\text\{supp\}\(\\mathbf\{s\}^\{n\}\_\{d\}\)=\\\{t\\in\[T\],\\mathbf\{s\}^\{n\}\_\{d\}\(t\)=1\\\}, andmdnm\_\{d\}^\{n\}denotes the number of required charging time slots to reach the target state\-of\-charge\. Without taking into account the congestion due to other charging EVs, the optimal schedule for dayd∈\[D\]d\\in\[D\], and agentn∈\[N\]n\\in\[N\]is:

𝐬dn,∗=argminsdn∈𝒮dn\(𝐬dn\)⊤𝐩d,\\mathbf\{s\}^\{n,\*\}\_\{d\}=\\arg\\min\_\{s^\{n\}\_\{d\}\\in\\mathcal\{S\}\_\{d\}^\{n\}\}\{\(\\mathbf\{s\}^\{n\}\_\{d\}\}\)^\{\\top\}\\mathbf\{p\}\_\{d\},\(3\)where𝐩d∈\[0,1\]T\\mathbf\{p\}\_\{d\}\\in\[0,1\]^\{T\}is the vector of price for daydd\. The availability, the required number of time slots for charging, and the prices for the day are random variables\. Moreover, the schedules of other EVsi≠ni\\neq nare unknown to EVnn\. If more thanKKEVs choose the same time slotttfor charging, a congestion occurs𝐜d​\(t\)=1\\mathbf\{c\}\_\{d\}\(t\)=1and no EV can charge111congestion management can involve random disconnection of excess vehicles when the aggregate demand surpasses the transformer capacity, or congestion management can be done by available energy sharing\.,𝐜d​\(t\)∈\{0,1\}T\\mathbf\{c\}\_\{d\}\(t\)\\in\\\{0,1\\\}^\{T\}, else no congestion occurs𝐜d​\(t\)=0\\mathbf\{c\}\_\{d\}\(t\)=0\. We consider the case where all the EVs use a stochastic policy for choosing their charging time slots, and hence𝐜d​\(t\)\\mathbf\{c\}\_\{d\}\(t\)is a random variable\.

This random variable𝐜d​\(t\)\\mathbf\{c\}\_\{d\}\(t\)drifts the decentralized charging problem from planning under uncertainty or model\-based reinforcement learning to model free reinforcement learning\. Indeed, choosing a good planning at the beginning of the day is not sufficient\. When congestion occurs, for reaching their target state\-of\-charge, EVs need to replan their charging during the day\. Moreover, the model free reinforcement learning framework allows to take into account imperfect observation of the state𝐱d,tn∈ℝl\\mathbf\{x\}^\{n\}\_\{d,t\}\\in\\mathds\{R\}^\{l\},l∈ℕ\+l\\in\\mathds\{N\}^\{\+\}, through the policy that chooses charging of EVnnat time slottt\.

At each time slott∈\[T\]t\\in\[T\]of episoded∈\[D\]d\\in\[D\], each EVn∈\[N\]n\\in\[N\]chooses an action𝐬dn​\(t\)∈\{0,1\}\\mathbf\{s\}^\{n\}\_\{d\}\(t\)\\in\\\{0,1\\\}until the end of the episode or until it is charged, i\.e\.\(𝐬d,\.<tn\)⊤​\(𝟏−𝐜d,\.<t\)=mdn\(\{\\mathbf\{s\}^\{n\}\_\{d,\.<t\}\}\)^\{\\top\}\(\\mathbf\{1\-c\}\_\{d,\.<t\}\)=m\_\{d\}^\{n\}, where𝐜d\\mathbf\{c\}\_\{d\}is the vector ofTTdimensions where each coordinate is the event congestion, and the subscript𝐬d,\.<t\\mathbf\{s\}\_\{d,\.<t\}means the firstttdimensions of the vector𝐬d\\mathbf\{s\}\_\{d\}\.

The vector of reward𝐫dn∈\[0,1\]T\\mathbf\{r\}^\{n\}\_\{d\}\\in\[0,1\]^\{T\}of EVnnis provided at the end of the episodedd:

𝐫dn=\(𝐬dn∘\(𝟏−𝐜d\)\)⊤​\(𝟏−𝐩d\),\\mathbf\{r\}^\{n\}\_\{d\}=\(\\mathbf\{s\}^\{n\}\_\{d\}\\circ\(\\mathbf\{1\}\-\\mathbf\{c\}\_\{d\}\)\)^\{\\top\}\(\\mathbf\{1\}\-\\mathbf\{p\}\_\{d\}\),\(4\)where∘\\circdenotes the Hadamard product\. The goal of each EV is to maximize its expected daily gain:

Gn​\(D\)=1D​∑d=1D∑t=1T𝔼​\[𝐫dn​\(t\)\],G^\{n\}\(D\)=\\frac\{1\}\{D\}\\sum\_\{d=1\}^\{D\}\\sum\_\{t=1\}^\{T\}\\mathds\{E\}\[\\mathbf\{r\}^\{n\}\_\{d\}\(t\)\],\(5\)while minimizing the number of times the state of charge its not reached:

∑d=1D𝟙​\{\(𝐬d,\.<Tn\)⊤​\(𝟏−𝐜d,\.<T\)<mdn\}\.\\sum\_\{d=1\}^\{D\}\\mathds\{1\}\\\{\(\{\\mathbf\{s\}^\{n\}\_\{d,\.<T\}\}\)^\{\\top\}\(\\mathbf\{1\-c\}\_\{d,\.<T\}\)<m\_\{d\}^\{n\}\\\}\.\(6\)

## 3Independent Multi\-Agent Reinforcement Learning approaches

In this section, we present and adapt to our problem setting two reinforcement learning approaches that have been proposed in the literature for EV charging: Contextual Combinatorial Thompson Sampling\[zafar2026,leguerndallo:hal\-05262742\], and policy gradient algorithms: PPO\[schulman2017proximalpolicyoptimizationalgorithms\], A2C\[a2C\], and SPO\[SPO\]\.

### 3\.1Contextual Combinatorial Linear Bandits

Algorithm 1Contextual Combinatorial Bandits based on Linear Thompson Sampling for EVnn0:

σ\>0\\sigma\>0, for all

t∈\[T\],At:=Il,l,𝐛t:=0l,𝜽^t:=0lt\\in\[T\],A\_\{t\}:=I\_\{l,l\},\\mathbf\{b\}\_\{t\}:=0\_\{l\},\\hat\{\\boldsymbol\{\\theta\}\}\_\{t\}:=0\_\{l\}
1:for

d=1,…,Dd=1,\.\.\.,Ddo

2:

m:=min⁡\(mdn,bdn−adn\)m:=\\min\(m\_\{d\}^\{n\},b^\{n\}\_\{d\}\-a^\{n\}\_\{d\}\)
3:for

t=1,…,Tt=1,\.\.\.,Tdo

4:Observe context

𝐱d,tn\\mathbf\{x\}^\{n\}\_\{d,t\}// Sampling parameter for each charging time slot

5:Sample

𝜽t~∼𝒩​\(𝜽^t,σ2​At−1\)\\tilde\{\\boldsymbol\{\\theta\}\_\{t\}\}\\sim\\mathcal\{N\}\(\\hat\{\\boldsymbol\{\\theta\}\}\_\{t\},\\sigma^\{2\}A^\{\-1\}\_\{t\}\)// Compute the likelihood of reward for each time slot

6:Compute

𝝁~dn​\(t\):=\(𝐱d,tn\)⊤​𝜽~t\\tilde\{\\boldsymbol\{\\mu\}\}^\{n\}\_\{d\}\(t\):=\(\\mathbf\{x\}^\{n\}\_\{d,t\}\)^\{\\top\}\\boldsymbol\{\\tilde\{\\theta\}\}\_\{t\}
7:endfor

8:

t:=adnt:=a^\{n\}\_\{d\},

𝐬dn:=0T\\mathbf\{s\}\_\{d\}^\{n\}:=0\_\{T\},

𝐜d:=0T\\mathbf\{c\}\_\{d\}:=0\_\{T\}
9:while\(

t≤bdn​​and​​m\>0t\\leq b^\{n\}\_\{d\}\\text\{ \}\\and\\text\{ \}m\>0\)do

10:Define set

𝒮​\(t\):=\{0,1\}bdn−t\+1\\mathcal\{S\}\(t\):=\\\{0,1\\\}^\{b^\{n\}\_\{d\}\-t\+1\}// Choice of charging schedule

11:Choose

𝐬t:=arg⁡max𝐬∈𝒮\(t\),\|\|𝐬\|\|1=m⁡𝐬⊤​𝝁~d,adn≤\.<bdn−t\+1n\\mathbf\{s\}\_\{t\}:=\\arg\\max\_\{\\mathbf\{s\}\\in\\mathcal\{S\}\(t\),\\lvert\\lvert\\mathbf\{s\}\\lvert\\rvert\_\{1\}=m\}\\mathbf\{s\}^\{\\top\}\\tilde\{\\boldsymbol\{\\mu\}\}^\{n\}\_\{d,a^\{n\}\_\{d\}\\leq\.<b^\{n\}\_\{d\}\-t\+1\}
12:Play action

𝐬t​\(t\)\\mathbf\{s\}\_\{t\}\(t\), observe congestion

𝐜d​\(t\)\\mathbf\{c\}\_\{d\}\(t\)
13:

𝐬dn​\(t\):=𝐬dn​\(t\)\+𝐬t​\(t\)\\mathbf\{s\}\_\{d\}^\{n\}\(t\):=\\mathbf\{s\}\_\{d\}^\{n\}\(t\)\+\\mathbf\{s\}\_\{t\}\(t\)
14:if

𝐬t​\(t\)=1​​and​​𝐜d​\(t\)=0\\mathbf\{s\}\_\{t\}\(t\)=1\\text\{ \}\\and\\text\{ \}\\mathbf\{c\}\_\{d\}\(t\)=0then

15:m:=m\-1

16:endif

17:

t:=t\+1t:=t\+1
18:endwhile

19:Observe price

𝐩d\\mathbf\{p\}\_\{d\}// The linear models are updated for selected slots𝐬dn​\(t\)\\mathbf\{s\}\_\{d\}^\{n\}\(t\)

20:for

t=1,…,Tt=1,\.\.\.,Tdo

21:if

𝐬dn​\(t\)=1\\mathbf\{s\}\_\{d\}^\{n\}\(t\)=1then

22:

𝐫dn​\(t\):=𝐬dn​\(t\)​\(1−𝐜d​\(t\)\)​\(𝟏−𝐩d​\(t\)\)\\mathbf\{r\}^\{n\}\_\{d\}\(t\):=\\mathbf\{s\}\_\{d\}^\{n\}\(t\)\(1\-\\mathbf\{c\}\_\{d\}\(t\)\)\(\\mathbf\{1\}\-\\mathbf\{p\}\_\{d\}\(t\)\)
23:

At:=At\+\(𝐱d,tn\)⊤​𝐱d,tnA\_\{t\}:=A\_\{t\}\+\(\\mathbf\{x\}^\{n\}\_\{d,t\}\)^\{\\top\}\\mathbf\{x\}^\{n\}\_\{d,t\}
24:

bt:=𝐛t\+𝐫dn​\(t\)​𝐱d,tn\\textbf\{b\}\_\{t\}:=\\mathbf\{b\}\_\{t\}\+\\mathbf\{r\}^\{n\}\_\{d\}\(t\)\\mathbf\{x\}^\{n\}\_\{d,t\}
25:

𝜽^t:=bt⊤​At−1\\hat\{\\boldsymbol\{\\theta\}\}\_\{t\}:=\\textbf\{b\}\_\{t\}^\{\\top\}A\_\{t\}^\{\-1\}
26:endif

27:endfor

28:endfor

A Multi\-Armed Bandit \(MAB\) problem involves repeatedly selecting among actions, receiving rewards drawn from unknown probability distributions corresponding to the chosen actions\[lattimore2020\]\. The goal is to maximize expected cumulative reward over time\.

As at the beginning of the game the player does not know which is the action with the highest mean reward, the player faces exploration\-exploitation dilemma: in one hand, it has to explore actions with loosely estimated mean reward to built better estimates, and on the other hand, it has to exploit actions with highest estimated mean reward to gather rewards\.

The decentralized charging problem can be viewed as a special case of the combinatorial bandit problem\[chen13\]\. Each daydd, each EVnnhas to choosemdnm^\{n\}\_\{d\}time slots fromTTtime slots\. When a congestion occurs, EVnnhas to reselect the remaining time slots to reach the target state\-of\-charge\. Moreover, at the beginning of each daydd, EVnncan observe a context𝐱d,tn∈ℝl\\mathbf\{x\}\_\{d,t\}^\{n\}\\in\\mathds\{R\}^\{l\}for each time slottt,l∈ℕ\+l\\in\\mathds\{N\}^\{\+\}\.

Algorithm[1](https://arxiv.org/html/2606.31347#alg1)uses a Linear Thompson Sampling \(LinTS\)\[DBLP:journals/corr/abs\-1209\-3352\]for taking into account the context to evaluate the highest likelihood of reward at time slottt\(line 6\)\. Then, the schedule of charging is selected using the time slots with the highest likelihood of reward \(line 11\), where𝝁~d,adn≤\.<bdnn\\tilde\{\\boldsymbol\{\\mu\}\}^\{n\}\_\{d,a^\{n\}\_\{d\}\\leq\.<b^\{n\}\_\{d\}\}means the dimensions betweenadna\_\{d\}^\{n\}andbdnb\_\{d\}^\{n\}of the vector𝝁~dn\\tilde\{\\boldsymbol\{\\mu\}\}^\{n\}\_\{d\}\. The computational complexity of the selection of the best schedule is linear in term of number of time slotsTT\. Moreover, this selection could be done at the first time slot and then only if a congestion occurs at the previous time slot\. The price of each time slot is revealed at the end of the day \(line 19\)\. It allows to update the linear model of each selected time slot \(lines 22\-25\)\.

### 3\.2Proximal Policy Optimization \(PPO\)

In contrast to Contextual Combinatorial Linear Bandits, standard Reinforcement Learning addresses sequential decision\-making problems modeled as Markov Decision Processes: an agent observes a state, select an action according to its policy, and receives a reward that depends on the current state, the played action, and the following state, while in Contextual Combinatorial Linear Bandits the reward depends only on the state and on the played action\.

Algorithm 2PPO0:Initial policy

π𝜽0\\pi\_\{\\boldsymbol\{\\theta\}\_\{0\}\}, value function

Vϕ0V\_\{\\boldsymbol\{\\phi\}\_\{0\}\}
0:Hyperparameters:

ϵ\\epsilon\(clipping\),

λ\\lambda\(GAE\),

KK\(epochs\),

DD\(horizon\),

MM\(mini\-batch size\)

1:

𝒟:=∅\\mathcal\{D\}:=\\emptyset// Collect trajectories

2:foriteration

d=1,2,…,Dd=1,2,\\ldots,Ddo

3:Collect trajectories

𝒟:=\{\(𝐱d,tn,𝐬dn​\(t\),𝐜d​\(t\),𝐩d​\(t\)\)\}\\mathcal\{D\}:=\\\{\(\\mathbf\{x\}^\{n\}\_\{d,t\},\\mathbf\{s\}\_\{d\}^\{n\}\(t\),\\mathbf\{c\}\_\{d\}\(t\),\\mathbf\{p\}\_\{d\}\(t\)\)\\\}by running

πθo​l​d\\pi\_\{\\theta\_\{old\}\}// Compute returns and advantages for each timestep

4:foreach trajectory

𝒯\\mathcal\{T\}in

𝒟\\mathcal\{D\}do

5:foreach

t∈𝒯t\\in\\mathcal\{T\}do

6:

𝐫dn\(t\):=𝐬dn\(t\)\(1−𝐜d\(t\)\(1−𝐩d\(t\)\)\\mathbf\{r\}^\{n\}\_\{d\}\(t\):=\\mathbf\{s\}\_\{d\}^\{n\}\(t\)\(1\-\\mathbf\{c\}\_\{d\}\(t\)\(1\-\\mathbf\{p\}\_\{d\}\(t\)\)
7:

R​\(t\):=∑l=0\|𝒯\|−1−t𝐫dn​\(t\+l\)R\(t\):=\\sum\_\{l=0\}^\{\|\\mathcal\{T\}\|\-1\-t\}\\mathbf\{r\}^\{n\}\_\{d\}\(t\+l\)
8:

δ​\(t\):=𝐫dn​\(t\)\+Vϕ​\(𝐱d,t\+1n\)−Vϕ​\(𝐱d,tn\)\\delta\(t\):=\\mathbf\{r\}^\{n\}\_\{d\}\(t\)\+V\_\{\\phi\}\(\\mathbf\{x\}^\{n\}\_\{d,t\+1\}\)\-V\_\{\\phi\}\(\\mathbf\{x\}^\{n\}\_\{d,t\}\)
9:

A^​\(t\):=∑l=0\|𝒯\|−1−t\(λ\)l​δ​\(t\+l\)\\hat\{A\}\(t\):=\\sum\_\{l=0\}^\{\|\\mathcal\{T\}\|\-1\-t\}\(\\lambda\)^\{l\}\\delta\(t\+l\)
10:endfor

11:endfor// Optimize over multiple epochs

12:forepoch

k=1k=1to

KKdo

13:Split

𝒟\\mathcal\{D\}into

MMmini\-batches colorblue// Compute loss averaged over timesteps in batch

14:foreach mini\-batch

ℬ=\{\(𝐱d,tn,𝐬dn​\(t\),A^​\(t\),R​\(t\)\)\}t∈ℬ\\mathcal\{B\}=\\\{\(\\mathbf\{x\}^\{n\}\_\{d,t\},\\mathbf\{s\}\_\{d\}^\{n\}\(t\),\\hat\{A\}\(t\),\{R\}\(t\)\)\\\}\_\{t\\in\\mathcal\{B\}\}do

15:

ρ𝜽​\(t\):=π𝜽​\(𝐬d​\(t\)\|𝐱d,tn\)π𝜽o​l​d​\(𝐬d​\(t\)\|𝐱d,tn\)\\rho\_\{\\boldsymbol\{\\theta\}\}\(t\):=\\frac\{\\pi\_\{\\boldsymbol\{\\theta\}\}\(\\mathbf\{s\}\_\{d\}\(t\)\|\\mathbf\{x\}^\{n\}\_\{d,t\}\)\}\{\\pi\_\{\\boldsymbol\{\\theta\}\_\{old\}\}\(\\mathbf\{s\}\_\{d\}\(t\)\|\\mathbf\{x\}^\{n\}\_\{d,t\}\)\}
16:

LC​L​I​P:=1\|ℬ\|​∑t∈ℬmin⁡\(ρ𝜽​\(t\)​A^​\(t\),clip​\(ρ𝜽​\(t\),1−ϵ,1\+ϵ\)​A^​\(t\)\)L^\{CLIP\}:=\\frac\{1\}\{\|\\mathcal\{B\}\|\}\\sum\_\{t\\in\\mathcal\{B\}\}\\min\(\\rho\_\{\\boldsymbol\{\\theta\}\}\(t\)\\hat\{A\}\(t\),\\text\{clip\}\(\\rho\_\{\\boldsymbol\{\\theta\}\}\(t\),1\-\\epsilon,1\+\\epsilon\)\\hat\{A\}\(t\)\)
17:

LV​F:=1\|ℬ\|​∑t∈ℬ\(Vϕ​\(𝐱d,tn\)−R​\(t\)\)2L^\{VF\}:=\\frac\{1\}\{\|\\mathcal\{B\}\|\}\\sum\_\{t\\in\\mathcal\{B\}\}\(V\_\{\\phi\}\(\\mathbf\{x\}^\{n\}\_\{d,t\}\)\-\{R\}\(t\)\)^\{2\}
18:Update

𝜽\\boldsymbol\{\\theta\}and

ϕ\\boldsymbol\{\\phi\}using gradients of

LC​L​I​PL^\{CLIP\}and

LV​FL^\{VF\}
19:endfor

20:endfor

21:

𝒟:=∅\\mathcal\{D\}:=\\emptyset,

𝜽o​l​d:=𝜽\{\\boldsymbol\{\\theta\}\}\_\{old\}:=\\boldsymbol\{\\theta\}
22:endfor

The decentralized charging problem can be viewed as a reinforcement learning problem, where the environment is composed by the considered EVs, and the electrical grid, where the terminal states are the maximum lengthTTof the episode and the event "target state\-of\-charge is reached", and where the discount factor is11: the reward of the episode is the sum of rewards of charging time slots\.

Proximal Policy Optimization \(PPO\)\[schulman2017proximalpolicyoptimizationalgorithms\]is a policy gradient algorithm that addresses the instability of traditional policy gradient methods by limiting policy updates\. Algorithm[2](https://arxiv.org/html/2606.31347#alg2)collects trajectories using the previously updated policyπo​l​d\\pi\_\{old\}\(line 3\), computes advantages using Generalized Advantage Estimation \(GAE\) \(line 6\-9\)\. PPO uses collected trajectories for multiple optimization epochs with mini\-batch gradient descent \(lines 14\-19\) The key innovation is the clipping mechanism: the probability ratio between new and old policiesρ𝜽\\rho\_\{\\boldsymbol\{\\theta\}\}\(line 15\) is constrained within\[1−ϵ,1\+ϵ\]\[1\-\\epsilon,1\+\\epsilon\], ensuring conservative updates \(line 16\)\. PPO jointly trains an actor \(policy\)π𝜽\\pi\_\{\\boldsymbol\{\\theta\}\}and a critic \(value function\)V𝚽V\_\{\\boldsymbol\{\\Phi\}\}\(line 18\)\.

### 3\.3Advantage Actor Critic \(A2C\)

PPO and A2C differ only by the clipping mechanism \(lines 15\-16 Algorithm[2](https://arxiv.org/html/2606.31347#alg2)\)\. This gives PPO greater stability during gradient descent, thus reducing overfitting and allowing it to replayKKtimes the buffer of trajectories𝒟\\mathcal\{D\}\(line 12\), thereby improving sample efficiency\.

### 3\.4Simple Policy Optimization \(SPO\)

While PPO uses hard clipping \(line 16 Algorithm[2](https://arxiv.org/html/2606.31347#alg2)\), resulting in zero gradients for data points outside the interval\[1−ϵ,1\+ϵ\]\[1\-\\epsilon,1\+\\epsilon\], SPO\[SPO\]replaces this with soft quadratic regularization, which maintains non\-zero gradients everywhere\. In practice, the only difference with the PPO algorithm lies in line 16:

Lp:=−1\|ℬ\|​∑t∈ℬ\{ρθ​\(t\)​A^​\(t\)−\|A^​\(t\)\|2​ϵ⋅\[ρθ​\(t\)−1\]2\}L\_\{p\}:=\-\\frac\{1\}\{\|\\mathcal\{B\}\|\}\\sum\_\{t\\in\\mathcal\{B\}\}\\left\\\{\\rho\_\{\\theta\}\(t\)\\hat\{A\}\(t\)\-\\frac\{\|\\hat\{A\}\(t\)\|\}\{2\\epsilon\}\\cdot\\left\[\\rho\_\{\\theta\}\(t\)\-1\\right\]^\{2\}\\right\\\}

## 4Experimental setup

### 4\.1Simulation Environment

We developed a multi\-agent EV charging simulation environment in Python222a gitlab can be found here: https://gitlab\.com/lpplc/anon, leveraging our custom multi\-agent platform built onGymnasium\[towers2024gymnasiumstandardinterfacereinforcement\], building upon the work of\[leguerndallo:hal\-05262742\]\. Key features include:

- •configurable capacityKKof electrical grid,
- •support for heterogeneous strategies across the vehicle population,
- •integration of historical photovoltaic production data,
- •customizable distributions for EV arrivals and departures,
- •congestion management by random disconnection of excess vehicles when the aggregate demand surpasses the transformer capacity,

### 4\.2Electric Vehicle Parameters

Each vehicle has a2222kW charging power and a9292kWh battery capacity\[statista2025\]\. Vehicle arrivals follow a truncated normal distribution centered at 9:00 AM \(τiarr∼TruncNorm​\(μ=9,σ=1\.5,a=6,b=12\)\\tau\_\{i\}^\{\\text\{arr\}\}\\sim\\text\{TruncNorm\}\(\\mu=9,\\sigma=1\.5,a=6,b=12\)\), while parking durations follow a truncated exponential distribution \(Di∼TruncExp​\(λ=1,a=7,b=10\)D\_\{i\}\\sim\\text\{TruncExp\}\(\\lambda=1,a=7,b=10\)hours\)\. The initial state of charge varies uniformly between 10% and 70%, with a target SoC of 80%\.

### 4\.3Data and Electricity Prices

Four years \(2021\-2024\) of photovoltaic production data available on Elia’s portal\[elia2025data\]are used to derive electricity price using equation \([1](https://arxiv.org/html/2606.31347#S2.E1)\)\. Two information are considered: one with perfect real\-time solar data and another with realistic day\-ahead forecasts including uncertainty\. The data are then normalized between the 1st and 99th percentiles with a 15\-minute granularity\.40004000days are generated by sampling uniformly these four years of electricity price\.30003000days are used for training for training dataset and10001000days are used for test dataset\.

### 4\.4Algorithms and baselines Implementation

#### 4\.4\.1Greedy\.

This strategy sorts all available time slots between arrival and departure times in ascending order of electricity price, then selects the cheapest time slots to meet the charging demand\. If the current time falls within this selection, the vehicle charges; otherwise, it waits\. Without congestion and the true electricity price, this policy is optimal\. If a charging attempt fails due to congestion, the selection process is repeated to find the most suitable time slots\. The electricity price may be based on a forecast or the actual price\.

#### 4\.4\.2Bernoulli Thompson Sampling \(TS\)\.

Bernoulli Thompson Sampling is an optimal algorithm for the stochastic Multi\-Armed Bandits\[kaufmann2012\]\. It does not take into account the context𝐱d,tn∈ℝl\\mathbf\{x\}^\{n\}\_\{d,t\}\\in\\mathds\{R\}^\{l\}\. It is used as a baseline to evaluate the gain of contextual information in Linear Thompson Sampling\. At each timestep, Bernoulli Thompson Sampling models the probability of successfully charging \(i\.e\., not being congested\) as a Beta distribution:

psuccess​\(t\)∼Beta​\(α,β\)\.p\_\{\\text\{success\}\}\(t\)\\sim\\text\{Beta\}\(\\alpha,\\beta\)\.
Parameters are updated according to observed outcomes:

α:=α\+1−𝐜d​\(t\),β​\[t\]←β​\[t\]\+𝐜d​\(t\)\.\\alpha:=\\alpha\+1\-\\mathbf\{c\}\_\{d\}\(t\),\\qquad\\beta\[t\]\\leftarrow\\beta\[t\]\+\\mathbf\{c\}\_\{d\}\(t\)\.
At each time slot, the algorithm samples from these Beta distributions to estimate the probability of successful charging and computes an expected gain for each time slot as:

𝐠^dn​\(t\)=psuccess​\(t\)​\(1−𝐩^d​\(t\)\),\\hat\{\\mathbf\{g\}\}^\{n\}\_\{d\}\(t\)=p\_\{\\text\{success\}\}\(t\)\(1\-\\hat\{\\mathbf\{p\}\}\_\{d\}\(t\)\),where𝐩^d​\(t\)\\hat\{\\mathbf\{p\}\}\_\{d\}\(t\)is forecast price at time slottt\. The algorithm then selectsmdnm^\{n\}\_\{d\}time slots with the highest expected gain\. If congestion occurs, the selection process is repeated\. The parametersα\\alphaandβ\\betaare initialized to11\.

#### 4\.4\.3Linear Thompson Sampling \(LinTS\)\.

Linear Thompson Sampling\[DBLP:journals/corr/abs\-1209\-3352\]is used in Algorithm[1](https://arxiv.org/html/2606.31347#alg1)for selecting the best schedules\. The variance parametervvis set to11\. The context𝐱d,tn\\mathbf\{x\}^\{n\}\_\{d,t\}contains the forecast price, the current time step, the current week\. Given the cyclic nature of time step and week as​i​n/c​o​ssin/coscoding is used\.

#### 4\.4\.4Neural Thompson Sampling \(NeuralTS\)\.

This algorithm is the same that Algorithm[1](https://arxiv.org/html/2606.31347#alg1), except that instead using LinTS it employs a neural networkf𝜽f\_\{\\boldsymbol\{\\theta\}\}to model the relationship between contextual features𝐱d,tn\\mathbf\{x\}^\{n\}\_\{d,t\}and expected reward𝐫dn​\(t\)\\mathbf\{r\}^\{n\}\_\{d\}\(t\)\. The neural network uses a single hidden layer with6464neurons, a learning rate of10−310^\{\-3\}, and Sherman\-Morrison updates for the Bayesian linear head\. The context𝐱d,tn\\mathbf\{x\}^\{n\}\_\{d,t\}is the same as the one used in LinTS\.

#### 4\.4\.5Proximal Policy Optimization \(PPO\)\.

PPO\[schulman2017proximalpolicyoptimizationalgorithms\]is used in Algorithm[2](https://arxiv.org/html/2606.31347#alg2)for selecting the best schedules\. PPO uses2525epochs per update, a batch size of512512, a cliping range of0\.30\.3, and a learning rate of5×10−45\\times 10^\{\-4\}\. PPO is implemented with Stable\-Baselines3\[stable\-baselines3\]\. The state𝐱d,tn\\mathbf\{x\}^\{n\}\_\{d,t\}used in PPO contains the same information as that of LinTS, plus plug status, which is an indicator of connection to the electrical grid, the state of charge, and the remaining number of time steps\. This additional contextual information is directly included in Contextual Combinatorial MAB \(Algorithm[1](https://arxiv.org/html/2606.31347#alg1)\)\.

#### 4\.4\.6Advantage Actor Critic \(A2C\)\.

A2C\[a2C\]uses the same parameters, contexts and implementation as PPO\.

#### 4\.4\.7Simple Policy Optimization \(SPO\)\.

SPO\[SPO\]uses the same parameters, contexts and implementation as PPO\.

## 5Experimental Results

### 5\.1Evaluation Protocol

All algorithms are trained over30003000episodes, each representing a one\-day scenario, and tested on10001000unseen episodes\. Performance is assessed using two complementary metrics: the average daily rewards achieved by the agents \(equation[5](https://arxiv.org/html/2606.31347#S2.E5)\) and the number of "SOC failure" events \(equation[6](https://arxiv.org/html/2606.31347#S2.E6)\)\. Each experiment is done55times\.

We consider two scenarios within this unified protocol\. The first scenario corresponds to homogeneous deployments, where all 20 EVs use the same algorithm under different capacity levels and both information regimes \(perfect real\-time prices versus day\-ahead forecasts\)\. The second scenario involves heterogeneous deployments, where multiple algorithms coexist within the same environment to assess robustness and interoperability\.

### 5\.2Homogeneous Groups

#### 5\.2\.1Learning curves\.

![Refer to caption](https://arxiv.org/html/2606.31347v1/learning_curves_homo.png)Figure 1:Learning curves\. Left panel70%70\\%capacity\. Right panel30%30\\%capacity\.We focus first on the learning curves to study the convergence time of different learning algorithms, which is crucial in a constantly evolving real\-world environment\.

In Figure[1](https://arxiv.org/html/2606.31347#S5.F1), for each algorithm, the mean reward evaluated on the test dataset versus the number of training episodes is plotted for a grid capacity allowing simultaneous charging70%70\\%of EVs \(left panel\) and30%30\\%of EVs \(right panel\)\. In low congestion regime \(capacity of70%70\\%\), EVs equipped with LinTS and and TS converge to a reasonably good policy in few episodes\. Then, due to additional contextual information, EVs equipped with LinTS outperforms those with TS in term of mean reward\. EVs equipped with PPO, SPO and A2C do not have time to converge since their test mean reward still increase after30003000episodes\. Neural Thompson Sampling exhibits a very slow convergence\. In high congestion regime \(capacity of30%30\\%\), EVs equipped with LinTS also quickly converge to a reasonably good policy that is improved after400400episodes\. After500500episodes, EVs equipped with PPO, SPO or A2C exhibit slightly better performance than those using LinTS\. EVS equipped with TS converge very fast, but are outperformed by policies that use contextual information\. NeuralTS would require more episodes to converge\.

#### 5\.2\.2Performance Analysis\.

To analyze the performance of algorithms, in Figure[2](https://arxiv.org/html/2606.31347#S5.F2), we plot the average reward and the percentage of failures in the state\-of\-charge measurement against capacity levels \(percentage of electric vehicles that can be charged simultaneously\)\. Results are presented for both measured prices in Figure[2\(a\)](https://arxiv.org/html/2606.31347#S5.F2.sf1)\(perfect real\-time information\) and day\-ahead forecasted prices in Figure[2\(b\)](https://arxiv.org/html/2606.31347#S5.F2.sf2)\(realistic uncertainty\)\.

![Refer to caption](https://arxiv.org/html/2606.31347v1/capacity_results_measured.png)\(a\)Measured prices
![Refer to caption](https://arxiv.org/html/2606.31347v1/capacity_results_day_ahead.png)\(b\)Day\-ahead forecasted prices

Figure 2:Algorithm performance for homogeneous groups across capacity levels\. Left panels: average rewards per vehicle; right panels: SOC failure rates \(log scale\)\.While the performance of greedy baseline is optimal when the price information is perfect and when there is no congestion \(capacity level100%100\\%\), we observe that its performance in terms of mean reward and SOC failures are the worst for low capacity level \(Figure[2\(a\)](https://arxiv.org/html/2606.31347#S5.F2.sf1)\)\. Notice that in perfect condition \(capacity level100%100\\%and measured price\), TS and LinTS obtain near optimal performance, while PPO, SPO and A2C do not achieve this performance\. This is consistent with their lower convergence rate observed in Figure[1](https://arxiv.org/html/2606.31347#S5.F1)a\. For low capacity \(between20%20\\%and30%30\\%\), EVs equipped with PPO, SPO or A2C slightly outperform those equipped with LinTS, while TS that does not use contextual information and NeuralTS that does not finish its convergence are outperformed\. In terms of SOC failures, EVs equipped with A2C, PPO and SPO outperform those equipped with LinTS for low to medium capacity \(between20%20\\%and50%50\\%\)\. This suggests that EV equipped with policy gradient algorithms coordinate better than other algorithms\. For high capacity \(between70%70\\%and100%100\\%\), EVs equipped with LinTS and A2C dominate other algorithms\. The worst algorithm is Greedy that chooses the lower prices and hence, as soon as there is congestion, fails to charge EVs\. Finally, while the performances of LinTS and TS decrease when the electricity prices are forecasted, those of PPO, SPO, and A2C are the same\.

Globally, LinTS exhibits a fast convergence \(Figure[1](https://arxiv.org/html/2606.31347#S5.F1)\), and good performances in terms of electricity price and SOC failures for all network capacity levels \(Figure[2](https://arxiv.org/html/2606.31347#S5.F2)\), while policy gradient algorithms are better for low capacity, which suggests that EV equipped with PPO, SPO, or A2C coordinate better than other algorithms\.

### 5\.3Heterogeneous Groups

Real\-world deployments will inevitably involve heterogeneous vehicle populations using different charging algorithms\. To assess algorithm robustness in such environments, we evaluate mixed deployments with1010PPO agents and1010LinTS agents sharing the same network constraints\.

For high network capacity \(70%70\\%\), the conclusion of the previous experiment with homogeneous groups of EVs is reinforced \(Figure[1](https://arxiv.org/html/2606.31347#S5.F1)\): EVs equipped with LinTS converge faster to a good solution that is slightly improved during time \(figure[3](https://arxiv.org/html/2606.31347#S5.F3)left panel\)\. However, for low network capacity \(30%30\\%\) the behavior of the learning curve of EVs equipped with LinTS radically change: after quickly reaching a good solution, the mean reward on the test set decreases \(Figure[3](https://arxiv.org/html/2606.31347#S5.F3)right panel\)\. This is due to the learning of electric vehicles equipped with PPO, which, when LinTS\-equipped electric vehicles focus on certain time slots, manage to find a better compromise between price and congestion, thus polluting the solution obtained by LinTS\. Indeed, LinTS is based on a stationary stochastic assumption that does not hold in high congestion regime in face of another group of learning agents, while standard reinforcement learning handles dynamic environment\. It should be noted, however, that EVs equipped with PPO only demonstrate superior performance to those equipped with LinTS after500500days, or nearly a year and a half\. Furthermore, EVs equipped with LinTS outperform those equipped with PPO on medium to high capacity grids \(Figure[4](https://arxiv.org/html/2606.31347#S5.F4)\)\.

![Refer to caption](https://arxiv.org/html/2606.31347v1/learning_curves_hetero.png)Figure 3:Learning curves for heterogeneous groups \(1010PPO agents,1010LinTS agents\)\. Left panel70%70\\%capacity\. Right panel30%30\\%capacity\.![Refer to caption](https://arxiv.org/html/2606.31347v1/capacity_results_day_ahead_hetero.png)Figure 4:Left panel: average rewards per vehicle; right panel: SOC failure rates \(log scale\)\. Heterogeneous groups \(1010PPO agents,1010LinTS agents\) across capacity levels\.

## 6Conclusion

To develop a scalable approach for optimizing electric vehicle charging, we proposed two approaches based on independent multi\-agent reinforcement learning: contextual and combinatorial linear bandits, and classical reinforcement learning algorithms such as PPO, SPO, and A2C\. When the agents use the same learning algorithm, Contextual and combinatorial linear bandits based on LinTS find a good solution faster, which, in the absence of congestion, achieves performance close to the one of the optimal policy, and, in the presence of significant congestion, closely matches those of the best classical reinforcement learning algorithms\. However, when two teams of agents compete to recharge their batteries on the same highly congested electrical grid, we have observed that strong learners such as PPO, SPO or A2C can significantly outperform LinTS\. However, the convergence speed of these algorithms remains far too slow to be learned online in even a slightly changing environment \(one year and a half to do better than LinTS\)\. Using a sliding window for LinTS might seem like a good solution for improving performance against an opposing team\. Indeed, the sliding window allows players to forget part of their past performance and thus adapt to a change\[Garivier2011\]\. However, electricity consumption and photovoltaic electricity production are seasonally dependent\. This means that to learn this dependence, a sliding window of at least one year is necessary, which severely limits the ability to adapt to a change in the opposing team’s strategy\. Future works will focus on adversarial linear bandits\[neu2020\], which should allow for adaptation to changes and could also provide theoretical guarantees\.

## References