Can Reinforcement Learning Efficiently Discover Price Manipulation?

# Can Reinforcement Learning Efficiently Discover Price Manipulation?
Source: [https://arxiv.org/html/2607.06121](https://arxiv.org/html/2607.06121)
###### Abstract

In this paper, we investigate whether a model\-free RL agent can identify and exploit price manipulation opportunities more effectively than a traditional model\-based approach that assumes correct specification of the data\-generating process but relies on noisy parameter estimates\. We consider a single\-asset market in which prices evolve according to an Almgren–Chriss framework with non\-linear permanent impact and linear temporary impact\. We first establish the existence of price\-manipulative strategies in discrete time and compute the optimal benchmark strategy using Sequential Least Squares Quadratic Programming under full information\. We then compare two finite\-sample learning approaches: a model\-based procedure that estimates impact parameters from simulated execution data and an agnostic RL approach based on Deep Deterministic Policy Gradient, trained directly on the same amount of data\. For intermediate volatility, the RL agent successfully discovers profitable manipulative strategies without explicit knowledge of the underlying model, even when training data are quite limited\. More importantly, RL consistently outperforms the model\-based approach when parameter estimates are affected by sampling error, despite the latter benefiting from the correct model specification\. For large volatility, all methods are unable to identify manipulation opportunities, while for small volatility, the model based approach outperforms RL\. These findings highlight both the effectiveness of RL in complex control problems and the risks associated with deploying learning algorithms in financial markets without appropriate safeguards\.

## 1Introduction

The increasing use of autonomous algorithmic trading systems has transformed modern financial markets\. Reinforcement Learning \(RL\) methods, in particular, have attracted growing attention because of their ability to learn adaptive trading policies directly from market interaction\. While these techniques offer significant advantages in high\-dimensional and partially observed environments, they also raise important concerns for regulators\. Since RL agents optimize rewards without explicit economic understanding or regulatory awareness, they may discover trading behaviours that are profitable but potentially manipulative\. As emphasized byWellman and Rajan\([2017](https://arxiv.org/html/2607.06121#bib.bib28)\)andAzzutti et al\.\([2021](https://arxiv.org/html/2607.06121#bib.bib4)\)autonomous trading agents may create new forms of market abuse and challenge traditional regulatory frameworks designed around human intent and accountability\. Recent research in market manipulation and market microstructure has highlighted the vulnerability of electronic markets to algorithmic strategies capable of distorting price formation and exploiting structural weaknesses of limit order book environmentsLee et al\.\([2013](https://arxiv.org/html/2607.06121#bib.bib13)\)\. In modern electronic markets, even relatively simple algorithmic trading rules can artificially influence short\-term prices and induce predictable reactions from other market participants\. Such behaviours may undermine market integrity and generate misleading signals regarding supply and demand\. Moreover, recent studies suggest that reinforcement learning agents may autonomously discover manipulative trading strategies purely through reward maximization and repeated interaction with the environmentMartínez\-Miranda et al\.\([2016](https://arxiv.org/html/2607.06121#bib.bib19)\);Mizuta\([2020](https://arxiv.org/html/2607.06121#bib.bib21)\)\. Specifically inMartínez\-Miranda et al\.\([2016](https://arxiv.org/html/2607.06121#bib.bib19)\)they predefine honest and manipulative actions, such as spoofing and pinging and use RL to determine when those already\-defined actions are optimal under different transaction costs, penalties, and liquidity conditions\. The actual price\-manipulation mechanism is assumed in the model rather than learned by the agent\. On the other hand, inMizuta\([2020](https://arxiv.org/html/2607.06121#bib.bib21)\)a genetic algorithm discovers a pump\-and\-dump pattern through feedback with trend\-following agents in an artificial market\. This differentiates us since we explicitly model non\-linear permanent market impact and show that a model\-free DDPG agent can recover such strategies in a continuous\-action environment\.

One potential source of manipulation arises from the presence of non\-linear market impact, which is the reaction of prices to trades\. A large empirical literature such asLillo et al\.\([2003](https://arxiv.org/html/2607.06121#bib.bib15)\);Potters and Bouchaud\([2003](https://arxiv.org/html/2607.06121#bib.bib25)\);Lillo et al\.\([2004](https://arxiv.org/html/2607.06121#bib.bib16)\)andZhou\([2012](https://arxiv.org/html/2607.06121#bib.bib30)\)documents that instantaneous market impact is typically a non\-linear function of volume for individual trades, typically with an odd and concave shape\. While these effects are well established empirically, they may also create environments in which trading activity can influence prices in exploitable ways\. In particular, non\-linear impact can reintroduce the possibility of dynamic arbitrage111Dynamic arbitrage exists if a trader can buy and sell over time, end with no inventory, and still make positive expected profit purely from the market\-impact dynamics\., allowing round\-trip trading strategies, those with zero initial and terminal inventory, to generate positive expected cash flow purely through their impact on prices\. This phenomenon, commonly referred to as price manipulation, was formalized byHuberman and Stanzl\([2004](https://arxiv.org/html/2607.06121#bib.bib10)\)and further explored in continuous\-time settings byGatheral\([2010](https://arxiv.org/html/2607.06121#bib.bib7)\)andGueant\([2016](https://arxiv.org/html/2607.06121#bib.bib8)\)\. Understanding when and how such manipulation arises remains a central question in market microstructure\.

Although one can often find mathematically price manipulations for a given impact model, their identification from market data poses several challenges\. First, the true Data Generating Process \(DGP\) of the price and trade dynamics is unknown, and models represent at most a stylized description of reality\. Second, even when the DGP is known, its parameters must be estimated from a finite sample of data\. The error in their estimation can severely undermine the manipulation opportunities\. Thus, the mere existence of manipulations does not imply a systematically profitable strategy\.

RL offers an intriguing alternative since it is learning based and model\-free\.Rather than postulating a model and estimating the parameters on a finite sample, it uses this sample to directly identify the policy that maximizes the reward\. It is therefore natural to ask whether RL based manipulation identification outperforms in finite sample the model based approach, even under the correct DGP specification\. This is the main question we answer in this paper\.

The use of RL methods for optimal execution and trading, where actions correspond to trading rates, is widespread, both in academia and in industry\. The Deterministic Policy Gradient framework introduced bySilver et al\.\([2014](https://arxiv.org/html/2607.06121#bib.bib26)\)and its deep learning extension, Deep Deterministic Policy Gradient \(DDPG\) byLillicrap et al\.\([2016](https://arxiv.org/html/2607.06121#bib.bib14)\), enable efficient learning in continuous action spaces\. These methods have been successfully applied to optimal execution and portfolio management tasks \(see, e\.g\.,Jiang et al\.\([2017](https://arxiv.org/html/2607.06121#bib.bib11)\);Moody and Saffell\([2001](https://arxiv.org/html/2607.06121#bib.bib22)\);Micheli and Monod\([2024](https://arxiv.org/html/2607.06121#bib.bib20)\)\)\. Early applications of RL to trading includeNevmyvaka et al\.\([2006](https://arxiv.org/html/2607.06121#bib.bib23)\), who applied Q\-learning to execution problems, and more recent work such asHendricks and Wilcox\([2014](https://arxiv.org/html/2607.06121#bib.bib9)\)andNing et al\.\([2021](https://arxiv.org/html/2607.06121#bib.bib24)\), which demonstrate the effectiveness of deep RL methods in complex market environments\. Extensions incorporating realistic features such as stochastic liquidity and partial observability have been studied inMacrì and Lillo\([2025](https://arxiv.org/html/2607.06121#bib.bib17)\)andMacrì et al\.\([2025](https://arxiv.org/html/2607.06121#bib.bib18)\), respectively\.

The use of RL in financial markets presents several criticalities, since algorithmic trading systems must operate within strict regulatory frameworks designed to prevent manipulation and ensure market integrity\. In fact, as pointed out in a recent work byCartea et al\.\([2023](https://arxiv.org/html/2607.06121#bib.bib5)\), the risk of unintentional market manipulation exists, especially when learning algorithms converge to strategies that artificially influence prices in order to increase profits\. This phenomenon is closely related to the broader problem of reward misspecification in machine learning, where the objective function does not fully capture the desired behaviour of the agent\. In the context of optimal execution, an RL agent trained to maximize trading profits may inadvertently learn to exploit non\-linear impact effects, effectively engaging in dynamic arbitrage even when such behaviour is undesirable or unrealistic, simply following a reward maximisation logic while learning from the environment\.

Motivated by these considerations, this paper examines whether an RL agent is more effective than a traditional model\-based strategy, which assumes correct model specification but faces parameter uncertainty arising from finite\-sample estimation, in generating positive expected cash flows through price manipulation in a market characterized by non\-linear price impact\. We consider a single\-asset setting, where the price evolves according to an Almgren\-Chriss \(AC\) type model\(Almgren and Chriss,[2000](https://arxiv.org/html/2607.06121#bib.bib1)\)with non\-linear permanent impact and linear temporary impact \(i\.e\. quadratic temporary impact costs\)\. The agent is constrained to follow a round\-trip strategy, meaning that it starts and ends with zero inventory over a finite trading horizon\. This constraint ensures that any profit must arise purely from trading dynamics rather than directional exposure\. Leveraging on this framework, and extendingGueant\([2016](https://arxiv.org/html/2607.06121#bib.bib8)\), by considering discrete time and non\-vanishing temporary impact costs, we first prove the existence of manipulative strategies from a theoretical perspective, showing that non\-linear impact functions can indeed admit solutions with positive expected cash flow even when only two trading velocities are allowed\. Due to the non\-convexity of the optimization problem, we numerically find the general optimal strategy by using Sequential Least Squares Quadratic Programming \(SLSQP\), assuming full knowledge of the model parameters\. The SLSQP algorithm has been used previously for the solution of optimal execution problem for the non linear transient impact model inCurato et al\.\([2017](https://arxiv.org/html/2607.06121#bib.bib6)\)\. This provides a benchmark solution against which learning\-based methods can be compared\.

We then consider the problem of learning to manipulate the price from a finite sample of data and we compare two approaches\. The first is the model based one: we use a finite sample of simulated TWAP execution to estimate the temporary and permanent impact parameters following the approach proposed byAlmgren et al\.\([2005](https://arxiv.org/html/2607.06121#bib.bib2)\)\. Notice that in this approach the correct DGP is known and only the parameters are estimated\. We then use them to implement the optimal manipulation strategies obtained with the SLSQP optimization on a test set of simulations\. The second approach is model free and based on RL: we train a DDPG algorithm to learn directly to manipulate the price on a finite sample of the same size of the one of TWAP executions\. Then, as before, we test the learned strategy on a test set\. For both approaches, the critical variable is volatility since, depending on its level, it preventsan accurate estimation of impact parameters or an effective learning of the strategy, respectively\.

Our first finding is that the RL agent is capable of discovering trading strategies that yield positive expected cash flow, effectively identifying price\-manipulative behaviors without explicit knowledge of the model, also when the training set is relatively small and volatility is not too large\. More importantly, the numerical results show that the performance of the RL\-based strategy is superior to the one of the model based approach when impact parameters are noisily estimated from data\. The superiority of RL is observed when volatility is not too low, because otherwise the parameter estimation of the model based approach is extremely accurate and therefore the method is equivalent to an exact optimization of a well specified DGP\. It is important to stress that the central question of the paper is therefore not whether manipulation exists theoretically, but whether an RL algorithm can independently learn to exploit these market vulnerabilities, when they are present, and if it can outperform a traditional model based approach\.

These results highlight both the effectiveness of RL in solving complex control problems and the potential risks associated with its unsupervised deployment in financial markets\. In particular, they underscore the need for a careful design of reward functions and regulatory safeguards to prevent potentiallyunintendedmarket manipulation adopted by learning algorithms\. From a broader perspective, this work extends the literature on non\-linear market impact and dynamic arbitrage by demonstrating how modern learning algorithms can exploit these features in practice\. At the same time, it connects to the growing body of research on the safety and robustness of machine learning systems, emphasizing the importance of aligning algorithmic objectives with market stability and regulatory requirements\.

The rest of the paper is organized as follows\. In Section[2](https://arxiv.org/html/2607.06121#S2), we formulate the problem and demonstrate the existence of a manipulative solution involving only two trading velocities\. In Section[3](https://arxiv.org/html/2607.06121#S3), we introduce the SLSQP numerical optimizer and present its manipulative solution under full information of the trading environment\. We also illustrate the complexity of the problem and assess its performance under partial information\. In Section[4](https://arxiv.org/html/2607.06121#S4), we introduce the DDPG algorithm\. Finally, in Section[5](https://arxiv.org/html/2607.06121#S5), we compare all the methods\.

## 2Framework of the Problem

In this section, we provide a general framework for the problem of dynamic arbitrage\. We define the market model with non\-linear permanent impact in discrete time, the trading cost structure, and finally we provide a solution to the case where the trader can trade with just two velocities\.

We consider the problem of a risk\-neutral agent seeking an optimal trading strategy over a finite time horizon, subject to zero initial and terminal inventory\. The agent is allowed to both buy and sell during the trading period, and therefore aims to construct around\-tripstrategy\. The trading horizon is divided intoTTsubintervals, each assumed, for simplicity, to have lengthΔ​t=1\\Delta t=1\.

We assume that the price dynamics follow an AC model with non\-linear permanent impact, namely:

St\\displaystyle S\_\{t\}=St−1\+f​\(vt\)​Δ​t\+σ​Δ​t​ξt,ξt∼𝒩​\(0,1\)\\displaystyle=S\_\{t\-1\}\+f\(v\_\{t\}\)\\Delta t\+\\sigma\\sqrt\{\\Delta t\}\\xi\_\{t\},\\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\xi\_\{t\}\\sim\\mathcal\{N\}\(0,1\)\(1\)S~t\\displaystyle\\tilde\{S\}\_\{t\}=St−1\+κ​vt,κ\>0\\displaystyle=S\_\{t\-1\}\+\\kappa v\_\{t\},\\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\kappa\>0\(2\)whereStS\_\{t\}is the market price andS~t\\tilde\{S\}\_\{t\}is the execution price at timett\.

Assuming that the trader trades at eacht∈\[T\]t\\in\[T\],222We use the notation\[T\]:=\{1,⋯,T\}\[T\]:=\\\{1,\\cdots,T\\\}\.we denote byqtq\_\{t\}andvtv\_\{t\}the inventory and the trading velocity at timettrespectively\. The trading velocityvtv\_\{t\}represents the traded volume at timett\. Clearly, it holds thatq0=qT=0q\_\{0\}=q\_\{T\}=0\. The permanent market impact is non\-linear and it is

f​\(vt\)=θ​sign​\(vt\)​\|vt\|δ,θ\>0,δ∈\(0,1\)f\(v\_\{t\}\)=\\theta\\penalty 10000\\ \\text\{sign\}\(v\_\{t\}\)\|v\_\{t\}\|^\{\\delta\},\\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\theta\>0,\\delta\\in\(0,1\)\(3\)whenδ=1\\delta=1, the permanent impact is linear as in the standard AC framework\. The coefficientκ\\kappais the temporary impact coefficient, hence we assume a linear temporary impact\. We observe that a buy order pushes the price up, both the market and the execution price, sincef​\(vt\)\>0f\(v\_\{t\}\)\>0andκ​vt\>0\\kappa v\_\{t\}\>0\. Similarly, a sell order pushes both prices down\.

Now, as in other optimal execution problems, we have that:

∑t=1Tvt=q0=0\.\\sum\_\{t=1\}^\{T\}v\_\{t\}=q\_\{0\}=0\.In such a problem, the final cash flow, will be:

XT=−∑t=1Tvt​S~tX\_\{T\}=\-\\sum\_\{t=1\}^\{T\}v\_\{t\}\\tilde\{S\}\_\{t\}The expected cash flow is then \(see Appendix[A](https://arxiv.org/html/2607.06121#A1)\):

𝔼​\[XT\]=−\[∑t=1T−1f​\(vt\)​∑j=t\+1Tvj\+κ​∑t=1Tvt2\]\.\\mathbb\{E\}\[X\_\{T\}\]=\-\\Big\[\\sum\_\{t=1\}^\{T\-1\}f\(v\_\{t\}\)\\sum\_\{j=t\+1\}^\{T\}v\_\{j\}\+\\kappa\\sum\_\{t=1\}^\{T\}v\_\{t\}^\{2\}\\Big\]\.\(4\)
As the main aim of the paper is to study the existence, feasibility and learnability of dynamic arbitrages, we briefly recall its main definition used throughout the text\.

###### Definition 2\.1\(Huberman and Stanzl\([2004](https://arxiv.org/html/2607.06121#bib.bib10)\)\)\.

InT∈ℕT\\in\\mathbb\{N\}trading steps there is a dynamic arbitrage if there exists\{vt\}t∈\[T\]\\\{v\_\{t\}\\\}\_\{t\\in\[T\]\}such that

- •∑t=1Tvt=0\\sum\_\{t=1\}^\{T\}v\_\{t\}=0and
- •𝔼​\[XT\]\>0\\mathbb\{E\}\[X\_\{T\}\]\>0\.

Therefore, finding an optimal solution to the trading problem defined above amounts to finding a solutionv→∗∈ℝT\\vec\{v\}\_\{\*\}\\in\\mathbb\{R\}^\{T\}such that the function𝔼​\[XT\]:=c​\(v→∗\)\\mathbb\{E\}\[X\_\{T\}\]:=c\(\\vec\{v\}\_\{\*\}\)is maximized, where byc​\(v→∗\)c\(\\vec\{v\}\_\{\*\}\)we denote a function of the optimal trading trajectoryv→∗\\vec\{v\}\_\{\*\}\. To this end, we first need to make sure that there exists a solution to the problem we pose\. The following proposition guarantees that the problem is not ill\-posed\.

###### Proposition 2\.1\.

The expected cash flow forδ<1\\delta<1, is a continuous function of the strategy\. Moreover searching for a strategy that maximizes the expected cash flow is a well\-posed problem since

limλ→±∞c​\(λ​v→\)=−∞\.\\lim\_\{\\lambda\\rightarrow\\pm\\infty\}c\(\\lambda\\vec\{v\}\)=\-\\infty\.Therefore, there exists a solution that maximizes𝔼​\[XT\]\\mathbb\{E\}\[X\_\{T\}\]withδ<1\\delta<1\.

The above proposition does not guarantee that there is a round\-trip strategy but that the problem we want to solve is not ill\-posed, since the best strategy might still be to not trade\.

## 3Dynamic arbitrage strategies

In this section, we investigate the existence and characteristics of dynamic arbitrage strategies of the previously introduced impact model\. Due to the non\-convexity of the problem, the investigation is mainly numerical\. However in the next subsection we show that, by restricting to trading strategies with only two velocities, it is possible to find and characterize price manipulation strategies\.

### 3\.1Price\-manipulation with only two trading velocities

Before considering more complex cases, we start by analysing the case where the trader can only select two trading velocities\. Leveraging onGueant\([2016](https://arxiv.org/html/2607.06121#bib.bib8)\), we denote by𝒜⊂\[−b,b\]⊂ℝ\\mathcal\{A\}\\subset\[\-b,b\]\\subset\\mathbb\{R\}the set of all possible trading actions across time step and by\|𝒜\|\|\\mathcal\{A\}\|the cardinality of𝒜\\mathcal\{A\}\. Notice that for a trading strategy\{vt\}t∈\[T\]\\\{v\_\{t\}\\\}\_\{t\\in\[T\]\}there are at mostTTdifferent trading velocities thus\|𝒜\|≤T\|\\mathcal\{A\}\|\\leq T\. We observe that for a round\-trip strategy it must haveT≥2T\\geq 2\(i\.e\. at least two time steps\)\. In the rest of this subsection we consider that\|𝒜\|=2\|\\mathcal\{A\}\|=2\. In this case it is

vt=\{α,if​t∈S⊂\[T\]−β,if​t∈\[T\]∖Sv\_\{t\}=\\begin\{cases\}\\alpha,\\penalty 10000\\ \\penalty 10000\\ \\text\{if\}\\penalty 10000\\ t\\in S\\subset\[T\]\\\\ \-\\beta,\\penalty 10000\\ \\penalty 10000\\ \\text\{if\}\\penalty 10000\\ t\\in\[T\]\\setminus S\\end\{cases\}whereSSis any set of indexes in\[T\]\[T\]\. To be a round\-trip strategy,α\\alphaandβ\\betamust have the same sign and it should be that:

∑t=1Tvt=\|S\|​α−\(T−\|S\|\)​β=0⇔\|S\|​α=\(T−\|S\|\)​β\.\\sum\_\{t=1\}^\{T\}v\_\{t\}=\|S\|\\alpha\-\(T\-\|S\|\)\\beta=0\\Leftrightarrow\|S\|\\alpha=\(T\-\|S\|\)\\beta\.\(5\)In the following lemma, we compute the expected cash flow when one trading velocity is used up to a certain time and then replaced by a different trading velocity for the remainder of the trading horizon\.

###### Lemma 3\.1\.

Let\|𝒜\|=2\|\\mathcal\{A\}\|=2andn∈\[T−1\]n\\in\[T\-1\]such thatvt=αv\_\{t\}=\\alphafor every1≤t≤n1\\leq t\\leq nandvt=−βv\_\{t\}=\-\\betafor everyn<t≤Tn<t\\leq T, then

𝔼​\[XT\]=n​α2​\[\(n\+1\)​f​\(α\)\+f​\(−nT−n​α\)​\(T−n−1\)\]−κ​\[n​α2\+n2​α2T−n\]\\mathbb\{E\}\[X\_\{T\}\]=\\frac\{n\\alpha\}\{2\}\[\(n\+1\)f\(\\alpha\)\+f\(\-\\frac\{n\}\{T\-n\}\\alpha\)\(T\-n\-1\)\]\-\\kappa\[n\\alpha^\{2\}\+\\frac\{n^\{2\}\\alpha^\{2\}\}\{T\-n\}\]\(6\)

Now that we have the expected cash flow for the case when there are only two trading velocities we can compute which are the optimal ones that maximize the cash flow\. The following lemma proves this result\.

###### Lemma 3\.2\.

Assume the conditions of Lemma[3\.1](https://arxiv.org/html/2607.06121#S3.Thmlemma1)hold andα\>0\\alpha\>0, then the maximum expected cash flow at the end of the trading period is

maxn∈\[T−1\]⁡𝔼​\[XT\]​\(α∗,n\)withα∗=\(\(δ\+1\)​A​\(n\)2​B​\(n\)\)11−δ\\max\_\{n\\in\[T\-1\]\}\\mathbb\{E\}\[X\_\{T\}\]\(\\alpha^\{\*\},n\)\\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\text\{with\}\\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\alpha^\{\*\}=\\bigg\(\\frac\{\(\\delta\+1\)A\(n\)\}\{2B\(n\)\}\\bigg\)^\{\\frac\{1\}\{1\-\\delta\}\}\(7\)and

A\(n\)=n​θ2\[\(n\+1\)−\(T−n−1\)\(nT−n\)δ\],B\(n\)=κ\[n\+n2T−n\]\.A\(n\)=\\frac\{n\\theta\}\{2\}\\Big\[\(n\+1\)\-\(T\-n\-1\)\\Big\(\\frac\{n\}\{T\-n\}\\Big\)^\{\\delta\}\\Big\]\\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ ,\\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ B\(n\)=\\kappa\\Big\[n\+\\frac\{n^\{2\}\}\{T\-n\}\\Big\]\.

![Refer to caption](https://arxiv.org/html/2607.06121v1/figures/long_delta.png)Figure 1:The optimal long position,n​α∗n\\alpha^\{\*\}, as a function ofδ\\deltafor the PnD strategy\.Notice that we do not have a closed\-form analytical solution for the optimal value ofnn, but only for the optimal value ofα\\alpha, which itself depends onnn\. Therefore, when\|𝒜\|=2\|\\mathcal\{A\}\|=2, we first determine the optimal trading velocityα∗\\alpha^\{\*\}for each fixednn, and then choose the value ofnnthat maximizes the expected cash flow\. Finally, for a fixednn, there exists a unique strategy that maximizes the expected cash flow whenα\>0\\alpha\>0\. Throughout the remainder of the paper, we refer to this strategy as the pump\-and\-dump strategy \(PnD\)\.

It is important to stress that nonlinearity is the main source of manipulation\. Fig\. \([1](https://arxiv.org/html/2607.06121#S3.F1)\) shows that the long position in the optimal PnD strategy decreases asd​e​l​t​adeltaincreases\. This relationship illustrates how stronger nonlinear price impact increases the size of the position selected by the optimal manipulation strategy\.

### 3\.2Numerical optimization via SLSQP

Assuming only two distinct trading velocities, as in subsection[3\.1](https://arxiv.org/html/2607.06121#S3.SS1), clearly restricts the space of possible strategies and the revenues generated from trading\. A richer set of trading volumes and strategies could emerge if\|𝒜\|\|\\mathcal\{A\}\|were allowed to include more than just two velocities\. However, in that case the problem is not convex any more and there are no closed\-form solutions\. We therefore resort to numerical methods to obtain an approximate solution to the problem\. We solve the constrained optimization problem using the Sequential Least Squares Programming \(SLSQP\) algorithm proposed byKraft\([1988](https://arxiv.org/html/2607.06121#bib.bib12)\)\. Therefore, the resulting optimization problem reads as follows:

v→∗=argmaxv→∈ℝT​𝔼​\[XT\],s\.t​∑t=1Tvt=0,\\displaystyle\\vec\{v\}\_\{\*\}=\\text\{argmax\}\_\{\\vec\{v\}\\in\\mathbb\{R\}^\{T\}\}\\mathbb\{E\}\[X\_\{T\}\],\\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ s\.t\\penalty 10000\\ \\penalty 10000\\ \\sum\_\{t=1\}^\{T\}v\_\{t\}=0,The objective function𝔼​\[XT\]\\mathbb\{E\}\[X\_\{T\}\]is defined in Eq\.\([4](https://arxiv.org/html/2607.06121#S2.E4)\), and the parameter values used to determine it are reported on Table[1](https://arxiv.org/html/2607.06121#S3.T1)\. We initialize the optimization fromN=500N=500different starting points, sampling uniformly at random fromU​\[−b,b\]TU\[\-b,b\]^\{T\}, in order to identify the best solution\.

Table 1:Fixed Parameters used both for the RL environment and for the definition of the objective for the SLSQP\.Clearly, in this first step, to use the numerical solution we assume complete knowledge of the parameters governing the evolution of the trading environment\.

![Refer to caption](https://arxiv.org/html/2607.06121v1/figures/SLSQP_steps21_lam0_sigma01.png)Figure 2:Top\-Left:The optimal trading strategy by SLSQP and the corresponding inventory \(red line\)\.Bottom\-Left:The expected cash\-flow\. The green \(red\) area indicates the region for which𝔼​\[Xt\]\>0\\mathbb\{E\}\[X\_\{t\}\]\>0\(𝔼​\[Xt\]<0\\mathbb\{E\}\[X\_\{t\}\]<0\)\.Right:The histogram of𝔼​\[XT\]\\mathbb\{E\}\[X\_\{T\}\]from the different starting points\.Note:The parameters used are the ones reported on Table[1](https://arxiv.org/html/2607.06121#S3.T1)\.Fig\. \([2](https://arxiv.org/html/2607.06121#S3.F2)\) illustrates the best numerical solution obtained via SLSQP\. The top\-left panel shows the optimal trading velocity together with the corresponding inventory trajectory, depicted by the red line\. The bottom\-left panel reports the associated expected cash flow, from which we observe the existence of a manipulative strategy since𝔼​\[XT\]\>0\\mathbb\{E\}\[X\_\{T\}\]\>0\. The right panel displays the histogram of𝔼​\[XT\]\\mathbb\{E\}\[X\_\{T\}\]obtained from 500 different SLSQP initializations\. These results highlight the complexity of the optimization problem, which stems from the non\-linear form of the permanent market impact function\. In Appendix[C](https://arxiv.org/html/2607.06121#A3)we shed light on the landscape of the various trading strategies found by the SLSQP starting from theNNdifferent initial points\.

### 3\.3Effect of impact parameter estimation on profitability

The previous numerical results were based on the assumption that the parameters governing the market were fully known when optimizing with SLSQP\. In practice, this corresponds to an idealized setting that cannot realistically be solved under full information\. In Section[3\.3\.1](https://arxiv.org/html/2607.06121#S3.SS3.SSS1), we study the sensitivity of SLSQP to parameter misspecification, while in Section[3\.3\.2](https://arxiv.org/html/2607.06121#S3.SS3.SSS2), we present a method of impact parameters estimation on a finite sample of optimal execution strategies, based onAlmgren et al\.\([2005](https://arxiv.org/html/2607.06121#bib.bib2)\)\.

#### 3\.3\.1Sensitivity of PnL to parameter misspecification

Usually, most of the times in real market scenarios, impact parameters values are not available, as by definition they are latent quantities and therefore difficult to be estimated directly\. Therefore, numerical solutions obtained with estimated parameters might be affected by estimation errors as magnitudes might be misspecified\. To see this, Fig\. \([3](https://arxiv.org/html/2607.06121#S3.F3)\) illustrates the expected cash as a function of parameter misspecification across all three parameters\. For the computation of the cash flow, we find the optimal strategy by usingδ^\\hat\{\\delta\}and then we substitute this strategy to Eq\.\([4](https://arxiv.org/html/2607.06121#S2.E4)\) where we use the parameterδ\\delta, similarly for the other two cases when we do not have knowledge of parametersθ\\thetaandκ\\kappa\. Specifically, panel \(a\) shows the impact of misspecification inδ\\delta, panel \(b\) inθ\\theta, and panel \(c\) inκ\\kappa\.

Two key observations emerge\. First, clearly the highest expected cash is achieved in the absence of misspecification, and performance deteriorates as the magnitude of the misspecification increases\. Second, performance improves when the true value ofδ\\deltais closer to 1 rather than to 0\. A similar pattern is observed in the remaining panels, where the highest cash is again attained under no or minimal misspecification\.

![Refer to caption](https://arxiv.org/html/2607.06121v1/figures/cash_vs_delta_overall_reversed.png)
![Refer to caption](https://arxiv.org/html/2607.06121v1/figures/cash_vs_theta_overall_reversed.png)
![Refer to caption](https://arxiv.org/html/2607.06121v1/figures/cash_vs_kappa_overall_reversed.png)

Figure 3:Expected cash vs the true parameters;δ,θ\\delta,\\thetaandκ\\kappawhen the optimal strategy is found usingδ^,θ^\\hat\{\\delta\},\\hat\{\\theta\}andκ^\\hat\{\\kappa\}\.
#### 3\.3\.2Estimation of impact parameters

Here we present a parameter estimation method based onAlmgren et al\.\([2005](https://arxiv.org/html/2607.06121#bib.bib2)\)\. InAlmgren et al\.\([2005](https://arxiv.org/html/2607.06121#bib.bib2)\), the authors used real metaorders333Metaorders are large trading orders that are split into pieces and executed incrementally\., thus here we estimate the parameters bysimulatingmetaorders from a TWAP strategy in a time window\[1,T\]\[1,T\]leaving the dynamics of the market unchanged from previous numerical experiments\. Specifically, we assume that for the metaorderiithe fundamental and the execution price, respectively, are

St\(i\)\\displaystyle S\_\{t\}^\{\(i\)\}=St−1\(i\)\+θ​sign​\(v\(i\)\)​\|v\(i\)\|δ\+σ​ξt\(i\)\\displaystyle=S\_\{t\-1\}^\{\(i\)\}\+\\theta\\text\{sign\}\(v^\{\(i\)\}\)\|v^\{\(i\)\}\|^\{\\delta\}\+\\sigma\\xi\_\{t\}^\{\(i\)\}S~t\(i\)\\displaystyle\\tilde\{S\}\_\{t\}^\{\(i\)\}=St−1\(i\)\+κ​v\(i\),t=1,⋯,T​and​i=1,⋯,N\.\\displaystyle=S\_\{t\-1\}^\{\(i\)\}\+\\kappa v^\{\(i\)\},\\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ t=1,\\cdots,T\\penalty 10000\\ \\penalty 10000\\ \\text\{and\}\\penalty 10000\\ \\penalty 10000\\ i=1,\\cdots,N\.We assume that at theii\-th metaorder we want to executeqiq\_\{i\}shares444InAlmgren et al\.\([2005](https://arxiv.org/html/2607.06121#bib.bib2)\)this variable is denoted byXX\.at a constant speed for any orderiiand it isv\(i\)=qiTv^\{\(i\)\}=\\frac\{q\_\{i\}\}\{T\}withqi∼U​\[1,20\]q\_\{i\}\\sim U\[1,20\]\.

For each metaorderiithe fundamental price at timettcan be written as:

St\(i\)=S0\+t​θ​\(v\(i\)\)δ\+σ​∑s=1tξs\(i\)S\_\{t\}^\{\(i\)\}=S\_\{0\}\+t\\theta\(v^\{\(i\)\}\)^\{\\delta\}\+\\sigma\\sum\_\{s=1\}^\{t\}\\xi\_\{s\}^\{\(i\)\}thus, at the end of metaorderii, at timeTTthe fundamental price it is

ST\(i\)\\displaystyle S\_\{T\}^\{\(i\)\}=S0\+T​θ​\(v\(i\)\)δ\+σ​∑s=1Tξs\(i\)⇔\\displaystyle=S\_\{0\}\+T\\theta\(v^\{\(i\)\}\)^\{\\delta\}\+\\sigma\\sum\_\{s=1\}^\{T\}\\xi\_\{s\}^\{\(i\)\}\\LeftrightarrowST\(i\)−S0S0⏟Ii\\displaystyle\\underbrace\{\\frac\{S\_\{T\}^\{\(i\)\}\-S\_\{0\}\}\{S\_\{0\}\}\}\_\{I\_\{i\}\}=T​θ​\(v\(i\)\)δS0\+noise\\displaystyle=\\frac\{T\\theta\(v^\{\(i\)\}\)^\{\\delta\}\}\{S\_\{0\}\}\+\\text\{noise\}The execution price at timettcan be written as:

S~t\(i\)=S0\+\(t−1\)​θ​\(v\(i\)\)δ\+σ​∑s=1t−1ξs\(i\)\+κ​v\(i\)\\tilde\{S\}\_\{t\}^\{\(i\)\}=S\_\{0\}\+\(t\-1\)\\theta\(v^\{\(i\)\}\)^\{\\delta\}\+\\sigma\\sum\_\{s=1\}^\{t\-1\}\\xi\_\{s\}^\{\(i\)\}\+\\kappa v^\{\(i\)\}and the realised price \(following the terminology inAlmgren et al\.\([2005](https://arxiv.org/html/2607.06121#bib.bib2)\)\) for the metaorderii, due to the TWAP strategy, is

S¯i=∑t=1Tv\(i\)​S~t\(i\)∑t=1Tv\(i\)=∑t=1TqiT​S~t\(i\)∑t=1TqiT=1T​qi​∑t=1TS~t\(i\)qi\\bar\{S\}\_\{i\}=\\frac\{\\sum\_\{t=1\}^\{T\}v^\{\(i\)\}\\tilde\{S\}\_\{t\}^\{\(i\)\}\}\{\\sum\_\{t=1\}^\{T\}v^\{\(i\)\}\}=\\frac\{\\sum\_\{t=1\}^\{T\}\\frac\{q\_\{i\}\}\{T\}\\tilde\{S\}\_\{t\}^\{\(i\)\}\}\{\\sum\_\{t=1\}^\{T\}\\frac\{q\_\{i\}\}\{T\}\}=\\frac\{\\frac\{1\}\{T\}q\_\{i\}\\sum\_\{t=1\}^\{T\}\\tilde\{S\}\_\{t\}^\{\(i\)\}\}\{q\_\{i\}\}S¯i\\displaystyle\\bar\{S\}\_\{i\}=1T​∑t=1TS~t\(i\)\\displaystyle=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\tilde\{S\}\_\{t\}^\{\(i\)\}=1T​∑t=1T\[S0\+\(t−1\)​θ​\(v\(i\)\)δ\+κ​v\(i\)\+σ​∑s=1t−1ξs\(i\)\]\\displaystyle=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\Big\[S\_\{0\}\+\(t\-1\)\\theta\(v^\{\(i\)\}\)^\{\\delta\}\+\\kappa v^\{\(i\)\}\+\\sigma\\sum\_\{s=1\}^\{t\-1\}\\xi\_\{s\}^\{\(i\)\}\\Big\]=S0\+θ​T−12​\(v\(i\)\)δ\+κ​v\(i\)\+noise⇔\\displaystyle=S\_\{0\}\+\\theta\\frac\{T\-1\}\{2\}\(v^\{\(i\)\}\)^\{\\delta\}\+\\kappa v^\{\(i\)\}\+\\text\{noise\}\\LeftrightarrowS¯i−S0S0⏟Ji\\displaystyle\\underbrace\{\\frac\{\\bar\{S\}\_\{i\}\-S\_\{0\}\}\{S\_\{0\}\}\}\_\{J\_\{i\}\}=T−12​T​Ii\+κ​v\(i\)S0\+noise\\displaystyle=\\frac\{T\-1\}\{2T\}I\_\{i\}\+\\frac\{\\kappa v^\{\(i\)\}\}\{S\_\{0\}\}\+\\text\{noise\}Therefore, we end up with the following two regressions555Notice that inAlmgren et al\.\([2005](https://arxiv.org/html/2607.06121#bib.bib2)\)they use the approximationT−1T≈1\\frac\{T\-1\}\{T\}\\approx 1, to simplify the second regression equation but we observe, without this approximation, that the estimations for the temporary impact are more robust, since forT=21T=21the approximation is not sufficiently accurate\.:

Ii\\displaystyle I\_\{i\}=T​θ​\(v\(i\)\)δS0\+noise\\displaystyle=\\frac\{T\\theta\(v^\{\(i\)\}\)^\{\\delta\}\}\{S\_\{0\}\}\+\\text\{noise\}\(8\)Ji−T−12​T​Ii\\displaystyle J\_\{i\}\-\\frac\{T\-1\}\{2T\}I\_\{i\}=κ​v\(i\)S0\+noise\\displaystyle=\\frac\{\\kappa v^\{\(i\)\}\}\{S\_\{0\}\}\+\\text\{noise\}\(9\)To find an estimate forδ\\deltawe minimize the residuals sum of squares \(RSS\):

δ^=argminδ∈\(0,1\)​∑i=1N\(Ii−θ^​\(δ\)​TS0​\(qiT\)δ\)2\\hat\{\\delta\}=\\text\{argmin\}\_\{\\delta\\in\(0,1\)\}\\sum\_\{i=1\}^\{N\}\\Big\(I\_\{i\}\-\\hat\{\\theta\}\(\\delta\)\\frac\{T\}\{S\_\{0\}\}\\Big\(\\frac\{q\_\{i\}\}\{T\}\\Big\)^\{\\delta\}\\Big\)^\{2\}\(10\)whereθ^​\(δ\)=S0T​∑i=1N\(qiT\)δ​Ii∑i=1N\(qiT\)2​δ\\hat\{\\theta\}\(\\delta\)=\\frac\{S\_\{0\}\}\{T\}\\frac\{\\sum\_\{i=1\}^\{N\}\(\\frac\{q\_\{i\}\}\{T\}\)^\{\\delta\}I\_\{i\}\}\{\\sum\_\{i=1\}^\{N\}\(\\frac\{q\_\{i\}\}\{T\}\)^\{2\\delta\}\}\(see Appendix[D](https://arxiv.org/html/2607.06121#A4)for a derivation of this formula\)\. Thenθ^=θ^​\(δ^\)\\hat\{\\theta\}=\\hat\{\\theta\}\(\\hat\{\\delta\}\)\. Similarly, for the estimation ofκ\\kappait is

κ^=S0​∑i=1N\(qiT\)​\(Ji−T−12​T​Ii\)∑i=1N\(qiT\)2\.\\hat\{\\kappa\}=S\_\{0\}\\frac\{\\sum\_\{i=1\}^\{N\}\(\\frac\{q\_\{i\}\}\{T\}\)\(J\_\{i\}\-\\frac\{T\-1\}\{2T\}I\_\{i\}\)\}\{\\sum\_\{i=1\}^\{N\}\(\\frac\{q\_\{i\}\}\{T\}\)^\{2\}\}\.

### 3\.4Numerical experiments

We now perform numerical experiments to show how the estimation method proposed above works in finite sample\. The parameter of the impact model are those in Table[1](https://arxiv.org/html/2607.06121#S3.T1)\. For the choice of the volatilityσ\\sigma, we adopt a realistic estimate by using real data\. First we use an estimate of the daily realized volatility \(RV\) \(seeAndersen et al\.\([2003](https://arxiv.org/html/2607.06121#bib.bib3)\)\) of MSFT from April 2018 and we setσdaily=0\.026\\sigma\_\{\\text\{daily\}\}=0\.026\. Similarly toAlmgren et al\.\([2005](https://arxiv.org/html/2607.06121#bib.bib2)\)we assume that an average metaorder lasts0\.390\.39days, thus

σmetaorder:=σ=σdaily​0\.39=0\.0168\.\\sigma\_\{\\text\{metaorder\}\}:=\\sigma=\\sigma\_\{\\text\{daily\}\}\\sqrt\{0\.39\}=0\.0168\.Moreover for the impact parameters;θ\\thetaandκ\\kappa, we choose them in a way such that the ratiosσθ\\frac\{\\sigma\}\{\\theta\}andσκ\\frac\{\\sigma\}\{\\kappa\}are 56 and 167 respectively\. These ratios are the same with the ones reported on Table B fromAlmgren et al\.\([2005](https://arxiv.org/html/2607.06121#bib.bib2)\)in which they report the summary statistics from a sample of real metaorders\. Finally, we consider the estimation on samples of sizeN∈\{500,5000\}N\\in\\\{500,5000\\\}\.

The results are shown in Table[2](https://arxiv.org/html/2607.06121#S3.T2)\. As expected, the estimation is more precise for larger samples666We conducted experiments for larger values ofNN, which provide indications that the estimators are asymptotically consistent\.and for smaller volatility\. The interesting aspect is that the estimation of the parameterδ\\deltais very noisy and usually upward biased\. As suggested by Fig\. \([1](https://arxiv.org/html/2607.06121#S3.F1)\), this overestimation leads to more conservative strategies \(i\.e\. with smaller long position\), reducing the potential PnL of the dynamic arbitrage strategy when the DGP is known but the impact parameters must be estimated from a finite sample of data\.

Table 2:Estimate and Standard Error \(SE\) of the parameters whenT=21T=21\. Note that the values for the first 4 columns are in basis points\.

## 4Dynamic Arbitrage via deep RL

Having set the stage with numerical techniques, we now pass to studying the problem with deep RL\. By means of RL we aim at assessing whether a learner, who has no direct knowledge of either the impacts structure or of the values for the impacts, would be able tolearna dynamic arbitrage strategy, employing as little information on the market as possible\. In a spirit similar to that ofNing et al\.\([2021](https://arxiv.org/html/2607.06121#bib.bib24)\)andMacrì and Lillo\([2025](https://arxiv.org/html/2607.06121#bib.bib17)\), we solve the round trip optimal trading problem in the context of the model reported in Eq\.\([1](https://arxiv.org/html/2607.06121#S2.E1)\) with DDPG\.

### 4\.1Environment, states and reward

The deep RL experiment runs over a numberMMof training episodes of trading where the agent learns the optimal strategy\. The agent, for each time steptt, has access to a simple state representation of the environment called𝒮t=\(qt,t,St\)\\mathcal\{S\}\_\{t\}=\(q\_\{t\},t,S\_\{t\}\), that is the tuple made by current inventory at time steptt, time stepttand permanently impacted price at time steptt, i\.e\.StS\_\{t\}\.

The price dynamics evolve as in Eq\.\([1](https://arxiv.org/html/2607.06121#S2.E1)\) and the values of the parameters are the same as the ones described on Table[1](https://arxiv.org/html/2607.06121#S3.T1), therefore the agent has to make a trading decisionvtv\_\{t\}\(i\.e\. buy/sell action to adopt\) based on the knowledge of the state𝒮t\\mathcal\{S\}\_\{t\}, such that the reward at each time step is maximal\. The reward, seen at each trading step is defined as:

rt=St​vt−κ​vt2r\_\{t\}=S\_\{t\}v\_\{t\}\-\\kappa v\_\{t\}^\{2\}\(11\)Wherevtv\_\{t\}is the traded volume at timett,StS\_\{t\}is the permanently impacted price as in Eq\. \([1](https://arxiv.org/html/2607.06121#S2.E1)\) andκ\\kappais the temporary impact parameter\. The reward and the actions are added to the tuple𝒮t\\mathcal\{S\}\_\{t\}when they are stored in the memory used by the algorithm to train its neural networks \(further details below\)\.

### 4\.2The learning algorithm

The DDPG algorithm is composed by two main neural networks: acriticand anactornetwork\. It allows for continuous controls and combines deterministic policy gradients with deep function approximation\. Originally introduced byLillicrap et al\.\([2016](https://arxiv.org/html/2607.06121#bib.bib14)\), DDPG extends the deterministic policy gradient framework ofSilver et al\.\([2014](https://arxiv.org/html/2607.06121#bib.bib26)\)by leveraging neural networks to approximate both the policy \(actor\) and the action\-value function \(critic\)\. The actor network deterministically maps states to actions, while the critic estimates the expected return\. To stabilize training, DDPG employs target networks and an experience replay buffer, borrowing key ideas from Deep Q\-Network\. The critic is trained by minimizing the Bellman error using targets constructed from the reward signal and the discounted estimate of future returns, while the actor is updated via the deterministic policy gradient, maximizing the critic’s evaluation of the chosen actions\.

#### 4\.2\.1The actor and critic networks

The critic and actor neural networks are feed\-forward networks with four and three input neurons, respectively, a number oflNNl\_\{\\text\{NN\}\}layers ofdNNd\_\{\\text\{NN\}\}hidden nodes, each with`SiLu`activation function, in the Actor network, the final layer has`tanh`activation function\.

We refer to the actor network asπ\\pi, since it is the network that parametrises the policy directly, and to the critic network asQQ, as the architecture of the critic part of the algorithm relies fully on double deep Q\-learning\.

At the beginning of the training loop we initialise the networksπ,Q\\pi,Qwith random weightsμπ,μQ\\mu\_\{\\pi\},\\mu\_\{Q\}\. As training unfolds,\(μπ,μQ\)→\(μπ∗,μQ∗\)\(\\mu\_\{\\pi\},\\mu\_\{Q\}\)\\to\(\\mu^\{\*\}\_\{\\pi\},\\mu^\{\*\}\_\{Q\}\), where\(μπ∗,μQ∗\)\(\\mu^\{\*\}\_\{\\pi\},\\mu^\{\*\}\_\{Q\}\)are the optimal weights that correspond to the optimal policy that maximises the the sum of discounted rewards\.

Therefore, during training theQQnetwork, in order to avoid an overestimation of theQQ\-values for the considered state\-action pairs, we use aQtgtQ\_\{\\text\{tgt\}\}network that is initialised at the beginning of the training routine as a copy of theQQnetwork with weightsμQtgt=μQ\\mu\_\{Q\_\{\\text\{tgt\}\}\}=\\mu\_\{Q\}, thus employing double deep Q\-learning for the Critic part of the algorithm, the weights of theμQtgt\\mu\_\{Q\_\{\\text\{tgt\}\}\}are then periodically set equal toμQ\\mu\_\{Q\}\. In order to do so, we have adopted the soft update technique as described in the original DDPG paper byLillicrap et al\.\([2016](https://arxiv.org/html/2607.06121#bib.bib14)\), with the soft update parameter set to0\.0010\.001\.

During training, as the trading unfolds we chose to keep memory of the states seen and the actions, volumes traded, at each time step into a memory of lengthmm\. As soon as the memory has at least lengthBB, which is the length of a batch, the algorithm starts training the two networks via gradient descent\. In any case, every time the networks are trained, batches of past trading experiences are randomly sampled from the memory\.

##### The critic\.

For each training iteration, we train a critic networkQQ\. To this end, we add an additive exploration zero mean noise to the output of the actor networkπ\\pi\(whose training is described below\), whose variance declines at each iteration777Specifically, before learning loop starts we seta=1/Ta=1/Tandεmin=0\.1\\varepsilon\_\{\\text\{min\}\}=0\.1Then, for each train iterationmmwe setε=max⁡\(ε−a,εmin\)\\varepsilon=\\max\(\\varepsilon\-a,\\varepsilon\_\{\\text\{min\}\}\)\. In this way, the more we iterate in the learning routine the lower theε\\varepsilonvalue\.ε\\varepsilonis then used as an additive noise in the Actor\-Critic architecture used in both the learning algorithms\.\. The additive noise,ε\\varepsilon, helps the Actor neural network to explore the range of inventory holdings to be held depending on the state\. Thus, while training theQQnetwork,vt\+1=π​\(𝒮t\|μπ\)\+𝒩​\(0,ε\)v\_\{\{t\+1\}\}=\\pi\(\\mathcal\{S\}\_\{t\}\|\\mu\_\{\\pi\}\)\+\\mathcal\{N\}\(0,\\varepsilon\), whereπ​\(𝒮t\|μπ\)\\pi\(\\mathcal\{S\}\_\{t\}\|\\mu\_\{\\pi\}\)is the output of the Actor network\. Clearly, as the training unfolds andε\\varepsilonbecomes smaller, the exploration noise diminishes and the policy chosen by the networkπ\\pibecomes deterministic\. In this way, given the weightsμπ∗\\mu^\{\*\}\_\{\\pi\}, found at the end of the training, the Critic network directly maps a state into an action such that:

vt∗=π​\(𝒮t\|μπ∗\)v^\{\*\}\_\{t\}=\\pi\(\\mathcal\{S\}\_\{t\}\|\\mu^\{\*\}\_\{\\pi\}\)\(12\)Once the volume is obtained, it is used to calculate the reward for each time\-step using the equation Eq\.\([11](https://arxiv.org/html/2607.06121#S4.E11)\)\.

To train theQQnetwork, we update the weightsμQ\\mu\_\{Q\}such thatμQ=argminμQ​Ł1​\(μQ;μQtgt\)\\mu\_\{Q\}=\\text\{argmin\}\_\{\\mu\_\{Q\}\}\\L \_\{1\}\(\\mu\_\{Q\};\\mu\_\{Q\_\{\\text\{tgt\}\}\}\), by taking a single gradient step on the loss

Ł1\\displaystyle\\L \_\{1\}=1b​∑i=1b\(Q​\(𝒮t,vt\+1\|μQ\)−yt\)2\\displaystyle=\\frac\{1\}\{b\}\\sum^\{b\}\_\{i=1\}\(Q\(\\mathcal\{S\}\_\{t\},v\_\{\{t\+1\}\}\|\\mu\_\{Q\}\)\-y\_\{t\}\)^\{2\}\(13\)yt\\displaystyle y\_\{t\}=rt\+γ​Qtgt​\(𝒮t′,π​\(𝒮t′\|μπ\)\|μQtgt\)\\displaystyle=r\_\{t\}\+\\gamma Q\_\{\\text\{tgt\}\}\(\\mathcal\{S\}\_\{t\}^\{\\prime\},\\pi\(\\mathcal\{S\}\_\{t\}^\{\\prime\}\|\\mu\_\{\\pi\}\)\|\\mu\_\{Q\_\{\\text\{tgt\}\}\}\)\(14\)
Where𝒮t′\\mathcal\{S\}\_\{t\}^\{\\prime\}is the next state where the environment evolves after the agent has taken the actionvtv\_\{t\}\. The weights of the Critic networkμQ\\mu\_\{Q\}are then updated using gradient descent\.

##### The actor\.

For each training iteration we train the actor networkπ\\pi, which is the neural network responsible for choosing the actionvtv\_\{t\}\. To do this, we feed theπ\\pinetwork with a batch ofBBstates𝒮t\\mathcal\{S\}\_\{t\}\. Theπ\\pinetwork returns a volume to be sold or boughtvtv\_\{t\}, which is in turn fed to the CriticQQnetwork, along with the state𝒮t\\mathcal\{S\}\_\{t\}\. The output of theQQnetwork is theQQvalue which is maximized during training\. Specifically, the weightsμπ\\mu\_\{\\pi\}of the Actor network are found by maximizing the output of the critic network with fixed weightsμQ\\mu\_\{Q\}\. As before we perform a single gradient step to minimise the loss:

Ł2=−1b​∑i=1bQ​\(𝒮t,π​\(𝒮t′\|μπ\)\|μQ\)\\L \_\{2\}=\-\\frac\{1\}\{b\}\\sum^\{b\}\_\{i=1\}Q\(\\mathcal\{S\}\_\{t\},\\pi\(\\mathcal\{S\}\_\{t\}^\{\\prime\}\|\\mu\_\{\\pi\}\)\|\\mu\_\{Q\}\)\(15\)whereπ​\(𝒮t\|μπ\)\\pi\(\\mathcal\{S\}\_\{t\}\|\\mu\_\{\\pi\}\)is the output of the actor network with current weightsμπ\\mu\_\{\\pi\}\. The weightsμπ\\mu\_\{\\pi\}are updated using the policy gradient theorem\. The derivative of the loss functionŁ2\\L \_\{2\}with respect to the weights of the Actor network is:

∇μπŁ2=−1b​∑i=1b\[∇aQ​\(𝒮t,a\(i\)\|μQ\)\|a\(i\)=π​\(𝒮t′\|μπ\)​∇μπ\(𝒮t\|μπ\)\]\\nabla\_\{\\mu\_\{\\pi\}\}\\L \_\{2\}=\-\\frac\{1\}\{b\}\\sum^\{b\}\_\{i=1\}\\left\[\\nabla\_\{a\}Q\(\\mathcal\{S\}\_\{t\},a^\{\(i\)\}\|\\mu\_\{Q\}\)\|\_\{a^\{\(i\)\}=\\pi\(\\mathcal\{S\}\_\{t\}^\{\\prime\}\|\\mu\_\{\\pi\}\)\}\\nabla\_\{\\mu\_\{\\pi\}\}\(\\mathcal\{S\}\_\{t\}\|\\mu\_\{\\pi\}\)\\right\]\(16\)where the first part in the square brackets tells us how sensitive theQQnetwork is to changes in the action chosen by the Actorπ\\pi, this is the gradient of theQQ\-value with respect to the action and evaluated at the action suggested by the Actor network, in this context for ease of notation we usea\(i\)=vt\(i\)a^\{\(i\)\}=v^\{\(i\)\}\_\{t\}to denote the action for theiistate among thebbconsidered in the batch\. The second part inside the brackets is the gradient of the the actor’s policy and accounts for how much the actionvtv\_\{t\}changes with respect to changes in theπ\\pinetwork parameters\. We update the weights for theπ\\pinetwork with gradient descent\. We repeat this training routine a numberllof times in order to assess how the quality of the actions chosen by the Actor changes and, in this way, to thoroughly explore weights combinations for theπ\\pinetwork\.

The complete training algorithm for both the Critic and the Actor neural networks is reported in Algorithm[1](https://arxiv.org/html/2607.06121#alg1)\. Both the Actor and the Critic neural networks are feed\-forward fully connected neural networks, for the gradient descent we use the Weighted ADAM with learning rate of0\.10\.1\. As training unfolds, we adopt a scheduler for the learning rate, thus diminishing its value as the agent makes experience and the neural networks are trained, once very100100episodic iterations \(i\.e\. time step over training episodes\) the learning rate is diminished following the rulel​r←l​r×0\.99lr\\leftarrow lr\\times 0\.99\. The features for the DDPG algorithm are normalised in the domain\[−1,1\]\[\-1,1\]\. Relevant parameters for the networks and the market model are reported in Table[3](https://arxiv.org/html/2607.06121#S4.T3)\.

Table 3:Symbol and value used for each parameter in the DDPG algorithm\.![Refer to caption](https://arxiv.org/html/2607.06121v1/figures/action_heatmap_split_episode_500.png)Figure 4:Top:Heatmap of the executed actions chosen by DDPG during the training phase of 500 episodes\.Bottom:The actions chosen from DDPG along with the corresponding inventory during the final training episode\.Algorithm 1Training of DDPG agent for optimal tradingInitialise with random weights

QQand make a copy

QtgtQ\_\{\\text\{tgt\}\};

Initialise with random weights

π\\pi
Initialise the memory with max length

mm\. Set

ε=1\\varepsilon=1,

BBbatch size,

MMtrain episodes;

Set market dynamics parameters;

fori in Mdo

Set

S0i=S0S^\{i\}\_\{0\}=S\_\{0\};

fort in Tdo

𝒮t←\(qt,t,Sti\)\\mathcal\{S\}\_\{t\}\\leftarrow\(q\_\{t\},t,S^\{i\}\_\{t\}\);

vt=π​\(𝒮t\|μπ\);v\_\{t\}=\\pi\(\\mathcal\{S\}\_\{t\}\|\\mu\_\{\\pi\}\);
rt←St−1i​vt−κ​vt2r\_\{t\}\\leftarrow S^\{i\}\_\{t\-1\}v\_\{t\}\-\\kappa v^\{2\}\_\{t\};

St−1i→StiS^\{i\}\_\{t\-1\}\\to S^\{i\}\_\{t\};⊳\\trianglerightGenerateStiS^\{i\}\_\{t\}fromSt−1iS^\{i\}\_\{t\-1\}

𝒮t′←\(qt\+1,t\+1,St\+1\)\\mathcal\{S\}\_\{t\}^\{\\prime\}\\leftarrow\(q\_\{t\+1\},t\+1,S\_\{t\+1\}\);

Memory

←\(𝒮t,rt,vt,𝒮t′\)\\leftarrow\(\\mathcal\{S\}\_\{t\},r\_\{t\},v\_\{t\},\\mathcal\{S\}\_\{t\}^\{\\prime\}\);⊳\\trianglerightMemory storing

ifLength of memory

≥\\geqBBthen

forj in

BBdo

Sample a batch of

\(𝒮tj,rtj,vtj,𝒮t,′j\)\(\\mathcal\{S\}\_\{t\}^\{j\},r^\{j\}\_\{t\},v^\{j\}\_\{t\},\\mathcal\{S\}\_\{t\}^\{\{\}^\{\\prime\},j\}\)from memory;

train

π\\pi;

train

QQ
endfor

ifLength of memory =

LLthenhalve the length of memory

endif

endif

After

mmiterations decay

ϵ=ϵ−a\\epsilon=\\epsilon\-a;

After

mmiterations

θtgt←θmain\\theta\_\{\\text\{tgt\}\}\\leftarrow\\theta\_\{\\text\{main\}\};

endfor

endfor

![Refer to caption](https://arxiv.org/html/2607.06121v1/figures/action_metrics_4_runs.png)Figure 5:The 25\-episode moving averagesLeft:of the action aggressiveness andRight:of signed action directionality during training\.
##### Training results\.

We train the agent using the procedure described above and terminate training after episode 500, since no noticeable improvement in the learned policy is observed beyond this point\. The top panel of Fig\. \([4](https://arxiv.org/html/2607.06121#S4.F4)\) shows a heatmap of the actions selected at each time step and in each episode during one run of training\. At the beginning of the training phase, the agent alternates between exploratory buy and sell actions\. As training progresses, however, the policy converges toward a directional buying strategy over the first 20 time steps, followed by a sell execution at the final step to satisfy the round\-trip constraint\. The bottom panel shows the final learned strategy\.

To characterize how the strategies learned by DDPG evolve across episodes, we define the action aggressiveness in episodeeeas:

Ae=120​∑t=120\|ve,t\|A\_\{e\}=\\frac\{1\}\{20\}\\sum\_\{t=1\}^\{20\}\|v\_\{e,t\}\|\(17\)and the signed directionality in episodeeeas:

De=∑t=120ve,t∑t=120\|ve,t\|,with−1<De<1D\_\{e\}=\\frac\{\\sum\_\{t=1\}^\{20\}v\_\{e,t\}\}\{\\sum\_\{t=1\}^\{20\}\|v\_\{e,t\}\|\},\\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\text\{with\}\\penalty 10000\\ \\penalty 10000\\ \-1<D\_\{e\}<1\(18\)whereve,tv\_\{e,t\}is the action chosen \(i\.e\. the volume traded\) in episodeeeat timett\. The quantityAeA\_\{e\}measures how aggressive in absolute terms is the pump phase of the manipulative strategy, whileDeD\_\{e\}tells us how consistently the strategy buys or sells in the pump phase\. The left panel of Fig\. \([5](https://arxiv.org/html/2607.06121#S4.F5)\) displays the evolution ofAeA\_\{e\}during 4 training runs\. We observe that, as training progresses, DDPG converges toward less aggressive strategies\. The right panel of Fig\. \([5](https://arxiv.org/html/2607.06121#S4.F5)\) showsDeD\_\{e\}across the same runs\. During the first episodes, this quantity is close to zero, indicating no clear directional preference in the first 20 time steps\. As training proceeds, however, the strategy converges toward a positive direction over the first 20 time steps\. Notice that this symmetry breaking is consistent with the form of the reward function in Eq\. \([11](https://arxiv.org/html/2607.06121#S4.E11)\)\. At each time steptt, DDPG seeks an actionvtv\_\{t\}that maximizes the instantaneous reward\. Therefore, by differentiating the reward with respect tovtv\_\{t\}and setting the derivative equal to zero, we obtain the action that maximizes the reward at timett:

vt∗=St2​κ\>0\.v\_\{t\}^\{\*\}=\\frac\{S\_\{t\}\}\{2\\kappa\}\>0\.\(19\)If on the other hand, one defines the reward as:

rt=−St​vt−κ​vt2r\_\{t\}=\-S\_\{t\}v\_\{t\}\-\\kappa v\_\{t\}^\{2\}then we will observe a negative direction at the final episodes \(see Fig\. \([11](https://arxiv.org/html/2607.06121#A5.F11)\) in Appendix[E](https://arxiv.org/html/2607.06121#A5)\)\.

## 5Discovering price manipulation strategies in finite data samples

We now consider the problem of a trader who tries to find a price manipulation strategy in a market where dynamic arbitrage is allowed by the nonlinearity of permanent impact\. Critically, we assume that the trader has a finite amount of data to identify the strategy, and we compare two approaches\.

In the first one, the agent hasNNTWAP metaorders to estimate the parameters of the AC nonlinear model with the regression method presented above\. Then the agent uses the estimated parameters in an SLSQP optimizer to find the optimal price manipulation and it tests it on 1000 out of sample simulations of manipulative strategies\. We considerN=500N=500andN=5000N=5000and we term these strategiesregSLSQP500\\text\{regSLSQP\}\_\{500\}andregSLSQP5000\\text\{regSLSQP\}\_\{5000\}, respectively\. In the second approach, the agent is agnostic about the underlying impact model and uses the RL approach described above to learn the manipulation strategy onN=500N=500episodes\. As before, the agent tests it on 1000 out of sample simulations of manipulative strategies\. We term this strategy DDPG\.

As benchmarks we use the PnL obtained \(i\) with the optimal strategy with only two trading velocities and \(ii\) with the optimal strategy obtained with SLSQP\. Clearly, in both cases we use the correct DGP and the true parameters\. The two strategies are termed PnD and SLSQP, respectively\. As above, we consider three levels of volatility\.

Table[4](https://arxiv.org/html/2607.06121#S5.T4)summarizes, for each method, the average terminal cashXTX\_\{T\}, the corresponding standard error, and the results of the t\-test together with the associated p\-values over 1000 out of sample implementations of the learned strategies\. For the regSLSQP methods, we follow a different approach\. Since there is a large variability in the estimated impact parameters from a finite number of TWAP executions, the PnL in the out of sample set depends strongly on the specific realization of 500 or 5000 TWAPs used in the estimation\. To mitigate this effect, we replicate the whole procedure 100 times and we report the mean and the average standard deviation in these 100 experiments\. Moreover, the t\-test column reports the percentage of the replications \(among the 100\) that reject the null\-hypothesis of zero expected cash\.

The results show that when volatility is high \(σ=0\.0168\\sigma=0\.0168\), none of the methods among PnD, SLSQP and DDPG produces a statistically significant positive terminal cash\. This is due to the finiteness of the test set and the large volatility\. The regSLSQPs produce a statistically significant negative expected terminal cash in46%46\\%and33%33\\%of the cases when500500and50005000metaorders are used, respectively, for the estimation of the impact parameters\. Thus in these conditions, even if manipulation is possible, large volatility prevents its exploitation\.

For intermediate volatility values \(σ=0\.003\\sigma=0\.003\) PnD,regSLSQP​5000\\text\{regSLSQP\}\{5000\}, and DDPG achieve statistically significant positive cash flows\. PnD is the best\-performing method followed by DDPG and then byregSLSQP​5000\\text\{regSLSQP\}\{5000\}\. Moreover, SLSQP even though it has complete knowledge of both the functional form of the market\-impact model and of the parameters does not produce statistically significant positive cash\. On the contrary, the model based approachregSLSQP​500\\text\{regSLSQP\}\{500\}estimated on a smaller sample provides negative expected cash flow\. Importantly, notice that the same sample size is sufficient for the RL based approach DDPG to learn a systematically profitable strategy\.

Finally, when volatility is low \(σ=0\.002\\sigma=0\.002\), all methods exceptregSLSQP500\\text\{regSLSQP\}\_\{500\}generate statistically significant positive terminal cash\. However, among the two sample based methods,regSLSQP500\\text\{regSLSQP\}\_\{500\}outperforms DDPG\. This is somewhat expected because with low volatility and relatively large estimation set \(N=5000N=5000\), the parameters of the impact model are accurately estimated, while the DDPG is still trained onN=500N=500episodes\.

MethodMean\(10−4\)\(10^\{\-4\}\)Stdt\-testp\-valueσ\\sigmaPnD32320\.14420\.14420\.700\.700\.4800\.4800\.01680\.0168SLSQP48480\.28050\.28050\.540\.540\.5800\.580regSLSQP500\\text\{regSLSQP\}\_\{500\}−5942\-59421\.59241\.592446%46\\%<0\.05<0\.05regSLSQP5000\\text\{regSLSQP\}\_\{5000\}−4082\-40821\.241\.2433%33\\%<0\.05<0\.05DDPG11110\.04340\.04340\.830\.830\.4000\.400PnD14140\.02570\.02571\.751\.750\.0700\.0700\.00300\.0030SLSQP25250\.05010\.05011\.581\.580\.1100\.110regSLSQP500\\text\{regSLSQP\}\_\{500\}−2562\-25620\.19780\.197834%34\\%<0\.05<0\.05regSLSQP5000\\text\{regSLSQP\}\_\{5000\}110\.05610\.05615%5\\%<0\.05<0\.05DDPG550\.00770\.00772\.342\.340\.0100\.010PnD13130\.01710\.01712\.392\.390\.0100\.0100\.00200\.0020SLSQP23230\.03340\.03342\.222\.220\.0200\.020regSLSQP500\\text\{regSLSQP\}\_\{500\}−1071\-10710\.08520\.085254%54\\%<0\.05<0\.05regSLSQP5000\\text\{regSLSQP\}\_\{5000\}13130\.03490\.034936%36\\%<0\.05<0\.05DDPG550\.00510\.00513\.263\.260\.0010\.001Table 4:Comparison among the methods after10001000simulations of the price evolution as in Eq\.\([1](https://arxiv.org/html/2607.06121#S2.E1)\)\-\([2](https://arxiv.org/html/2607.06121#S2.E2)\)\.As expected, volatility is the key parameter for the effective profitability of dynamic arbitrage strategies\. When the model and the parameters are known \(SLSQP and PnD\), large volatility prevents the exploitation of the arbitrage, at least in a finite testing sample\. When the model and/or the parameters are unknown, the volatility also enters in the estimation \(regSLSQP\) and learning \(DDPG\)\. The surprising result is that for intermediate volatility and small sample a model agnostic method is more effective of one knowing the model but not the parameter in identifying profitable strategies\. Even more surprising is that in this volatility regime a ten\-fold larger training set of the regSLSQP is not sufficient to beat the performance of DDPG\. Finally, even for small volatility, DDPG outperforms regSLSQP when the training tests are equally small\.

In conclusion, the DDPG algorithm generates positive cash flows while relying on only minimal information: the current time step, the remaining inventory, and the permanently impacted asset price\. Despite this simple state representation, the agent achieves performance comparable to numerical optimization methods, with relatively low variability in terminal cashXTX\_\{T\}\. This suggests that reinforcement learning algorithms can discover round\-trip trading strategies that create dynamic arbitrage opportunities\.

## 6Conclusion

This paper studied whether a model\-free reinforcement learning algorithm can learn to exploit price\-manipulation opportunities more effectively than a traditional model\-based approach when both methods are trained on finite samples of market data\. We considered a single\-asset optimal execution problem with concave permanent market impact, a setting in which dynamic arbitrage may arise through round\-trip trading strategies\.

We first established, both analytically and numerically, the existence of profitable manipulative strategies in discrete time\. While these strategies can be derived under full knowledge of the market\-impact model, implementing them in practice requires estimating latent impact parameters from data, introducing an additional source of uncertainty\. This observation motivates the central comparison of the paper: a model\-based optimization procedure with correctly specified dynamics but estimated parameters versus a model\-free reinforcement learning agent that learns trading policies directly from interaction with the environment\.

Our numerical experiments show that the DDPG agent is capable of learning profitable round\-trip strategies without explicit knowledge of the underlying market\-impact model\. More importantly, when both approaches are constrained by the same amount of training data, reinforcement learning frequently outperforms the model\-based procedure because it avoids the intermediate and error\-prone parameter\-estimation step\. The advantage is most pronounced in regimes where parameter uncertainty materially affects optimization, whereas the model\-based approach naturally regains superiority when estimation becomes sufficiently accurate\.

The broader implication of these findings is methodological as much as financial\. The paper does not argue that reinforcement learning creates manipulation where none exists; rather, it shows that, whenever the market environment admits exploitable dynamic arbitrage, a model\-free learning algorithm can identify such opportunities directly from data and may do so more efficiently than a correctly specified parametric model estimated on finite samples\. This highlights an important distinction between the theoretical existence of manipulation and its practical discoverability\.

From a market\-design perspective, the results also carry regulatory implications\. Reinforcement learning agents optimize the objective they are given, without distinguishing between economically desirable profits and profits generated through behaviour that may undermine market integrity\. Consequently, environments that inadvertently admit manipulative incentives may be particularly vulnerable to increasingly sophisticated learning algorithms\. As reinforcement learning becomes more widely adopted in automated trading systems, ensuring that reward functions, market design, and regulatory constraints are aligned with no\-manipulation principles will become increasingly important\.

Several directions for future research naturally follow from this work\. Extending the analysis to more realistic market\-impact models, including transient or stochastic impact, would help assess the robustness of the results beyond the Almgren\-Chriss framework\. Likewise, studying richer market environments with limit\-order\-book dynamics, multiple strategic agents, or cross\-asset interactions would provide a more comprehensive understanding of how learning algorithms exploit market structure\. Finally, integrating explicit no\-manipulation constraints or safe reinforcement learning techniques into optimal execution algorithms represents a promising avenue for developing trading systems that remain both profitable and market compliant\.

## References

- Almgren and Chriss \[2000\]R\. Almgren and N\. Chriss\.Optimal execution of portfolio transactions\.Journal of Risk, 3:5–39, 2000\.
- Almgren et al\. \[2005\]Robert Almgren, Chee Thum, Emmanuel Hauptmann, and Hong Li\.Direct estimation of equity market impact\.Risk, 18:58–62, 2005\.
- Andersen et al\. \[2003\]T\.G\. Andersen, T\. Bollerslev, F\.X\. Diebold, and P\. Labys\.Modelling and forecasting realized volatility\.Econometrica, 71:579–625, 2003\.
- Azzutti et al\. \[2021\]Alessio Azzutti, Wolf\-Georg Ringe, and H\. Siegfried Stiehl\.Machine learning, market manipulation, and collusion on capital markets: Why the “black box” matters\.University of Pennsylvania Journal of International Law, 43:79–136, 2021\.
- Cartea et al\. \[2023\]Alvaro Cartea, Patrick Chang, and Gabriel Garcıa\-Arenas\.Spoofing and manipulating order books with learning algorithms\.Available at SSRN: https://ssrn\.com/abstract=4639959, 2023\.
- Curato et al\. \[2017\]Gianbiagio Curato, Jim Gatheral, and Fabrizio Lillo\.Optimal execution with non\-linear transient market impact\.Quantitative Finance, 17:41–54, 2017\.
- Gatheral \[2010\]J\. Gatheral\.No\-dynamic\-arbitrage and market impact\.Quantitative Finance, 10:749–759, 2010\.
- Gueant \[2016\]Olivier Gueant\.The Financial Mathematics of Market Liquidity\.Chapman & Hall/CRC Financial Mathematics Series, 2016\.
- Hendricks and Wilcox \[2014\]Dieter Hendricks and Diane Wilcox\.A reinforcement learning extension to the almgren\-chriss framework for optimal trade execution\.In 2014 IEEE Conference on Computational Intelligence for Financial Engineering & Economics \(CIFEr\), pages 457–464\. IEEE, 2014\.
- Huberman and Stanzl \[2004\]G\. Huberman and W\. Stanzl\.Price manipulation and quasi\-arbitrage\.Econometrica, 72:1247–1275, 2004\.
- Jiang et al\. \[2017\]Zhengyao Jiang, Dixing Xu, and Jinjun Liang\.A deep reinforcement learning framework for the financial portfolio management problem\.arXiv preprint arXiv:1706\.10059, 2017\.
- Kraft \[1988\]Dieter Kraft\.A software package for sequential quadratic programming\.Technical Report DFVLR\-FB 88\-28, Institut für Dynamik der Flugsysteme, Deutsche Forschungs\- und Versuchsanstalt für Luft\- und Raumfahrt, 1988\.
- Lee et al\. \[2013\]Eun Jung Lee, Kyoung\-Soo Eom, and Kyung Suh Park\.Microstructure\-based manipulation: Strategic behavior and performance of spoofing traders\.Journal of Financial Markets, 16:227–252, 2013\.
- Lillicrap et al\. \[2016\]T\. P\. Lillicrap, Jonathan J\. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra\.Continuous control with deep reinforcement learning\.In Proceedings of the 4th International Conference on Learning Representations, Proceedings of Machine Learning Research\. PMLR, 2016\.
- Lillo et al\. \[2003\]Fabrizio Lillo, J\.D\. Farmer, and Rosario N\. Mantegna\.Master curve for price\-impact function\.Nature, 421:129–130, 2003\.
- Lillo et al\. \[2004\]Fabrizio Lillo, J\. Doyne Farmer, and Rosario N\. Mantegna\.Price impact function of a single transaction\.In The Complex Dynamics of Economic Interaction, pages 153–160, Berlin, Heidelberg, 2004\. Springer Berlin Heidelberg\.
- Macrì and Lillo \[2025\]Andrea Macrì and Fabrizio Lillo\.Reinforcement learning for optimal execution when liquidity is time\-varying\.Applied Mathematical Finance, 31:312–342, 2025\.
- Macrì et al\. \[2025\]Andrea Macrì, Sebastian Jaimungal, and Fabrizio Lillo\.Deep reinforcement learning for optimal trading with partial information\.arXiv:2511\.00190, 2025\.
- Martínez\-Miranda et al\. \[2016\]Enrique Martínez\-Miranda, Frank McGroarty, and Edward Tsang\.Learning unfair trading: A market manipulation analysis from the reinforcement learning perspective\.In IEEE Conference on Evolving and Adaptive Intelligent Systems, pages 103–108, 2016\.
- Micheli and Monod \[2024\]Alessandro Micheli and Mélodie Monod\.Deep reinforcement learning for online optimal execution strategies\.arXiv preprint arXiv:2410\.13493, 2024\.
- Mizuta \[2020\]Takanobu Mizuta\.Can an ai perform market manipulation at its own discretion? – a genetic algorithm learns in an artificial market simulation\.In IEEE Symposium Series on Computational Intelligence, pages 407–414, 2020\.
- Moody and Saffell \[2001\]John Moody and Matthew Saffell\.Learning to trade via direct reinforcement\.IEEE Transactions on Neural Networks, 12:875–889, 2001\.
- Nevmyvaka et al\. \[2006\]Yuriy Nevmyvaka, Yi Feng, and Michael Kearns\.Reinforcement learning for optimized trade execution\.In Proceedings of the 23rd International Conference on Machine Learning \(ICML\), pages 673–680, 2006\.
- Ning et al\. \[2021\]Brian Ning, Franco Ho Ting Lin, and Sebastian Jaimungal\.Double deep q\-learning for optimal execution\.Applied Mathematical Finance, 28:361–380, 2021\.
- Potters and Bouchaud \[2003\]M\. Potters and J\.P\. Bouchaud\.More statistical properties of order books and price impact\.Physica A: Statistical Mechanics and its Applications, 324:133–140, 2003\.
- Silver et al\. \[2014\]David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller\.Deterministic policy gradient algorithms\.In Eric P\. Xing and Tony Jebara, editors, Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 387–395, Bejing, China, 22\-24 Jun 2014\. PMLR\.
- Tan et al\. \[2013\]Pang\-Ning Tan, Michael Steinbach, and Vipin Kumar\.Introduction to Data Mining\.Pearson Education Limited, 2013\.
- Wellman and Rajan \[2017\]Michael P\. Wellman and Uday Rajan\.Ethical issues for autonomous trading agents\.Minds and Machines, 27:609–624, 2017\.
- Zhelezniak et al\. \[2019\]Vitalii Zhelezniak, Aleksandar Savkov, April Shen, and Nils Hammerla\.Correlation coefficients and semantic textual similarity\.In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\), pages 951–962\. Association for Computational Linguistics, 2019\.
- Zhou \[2012\]Wei\-Xing Zhou\.Universal price impact functions of individual trades in an order\-driven market\.Quantitative Finance, 12\(8\):1253–1263, 2012\.

## Appendix ADerivations for section[2](https://arxiv.org/html/2607.06121#S2)

In this appendix we first derive the expected cash flow and then we give the proof of Proposition \([2\.1](https://arxiv.org/html/2607.06121#S2.Thmproposition1)\)\.

We derive Eq\.\([4](https://arxiv.org/html/2607.06121#S2.E4)\) by substituting the execution price into the expression for the terminal cash flow and expanding the result:

XT\\displaystyle X\_\{T\}=−∑t=1Tvt​S~t\\displaystyle=\-\\sum\_\{t=1\}^\{T\}v\_\{t\}\\tilde\{S\}\_\{t\}=−∑t=1Tvt​\(St−1\+κ​vt\)\\displaystyle=\-\\sum\_\{t=1\}^\{T\}v\_\{t\}\(S\_\{t\-1\}\+\\kappa v\_\{t\}\)=−\[∑t=1Tvt​St−1\+∑t=1Tκ​vt2\]\\displaystyle=\-\\Big\[\\sum\_\{t=1\}^\{T\}v\_\{t\}S\_\{t\-1\}\+\\sum\_\{t=1\}^\{T\}\\kappa v\_\{t\}^\{2\}\\Big\]The recursive Eq\.\([1](https://arxiv.org/html/2607.06121#S2.E1)\) is written as

St−1=S0\+∑j=1t−1f​\(vj\)\+σ​∑j=1t−1ξjS\_\{t\-1\}=S\_\{0\}\+\\sum\_\{j=1\}^\{t\-1\}f\(v\_\{j\}\)\+\\sigma\\sum\_\{j=1\}^\{t\-1\}\\xi\_\{j\}hence the terminal cash\-flow can be written as:

XT=−∑t=1Tvt​\[∑j=1t−1f​\(vj\)\+σ​∑j=1t−1ξj\]−κ​∑t=1Tvt2\.X\_\{T\}=\-\\sum\_\{t=1\}^\{T\}v\_\{t\}\\Big\[\\sum\_\{j=1\}^\{t\-1\}f\(v\_\{j\}\)\+\\sigma\\sum\_\{j=1\}^\{t\-1\}\\xi\_\{j\}\\Big\]\-\\kappa\\sum\_\{t=1\}^\{T\}v\_\{t\}^\{2\}\.We observe that

∑t=1Tvt​∑j=1t−1f​\(vj\)=∑j=1T−1f​\(vj\)​∑t=j\+1Tvt,v0=0\\sum\_\{t=1\}^\{T\}v\_\{t\}\\sum\_\{j=1\}^\{t\-1\}f\(v\_\{j\}\)=\\sum\_\{j=1\}^\{T\-1\}f\(v\_\{j\}\)\\sum\_\{t=j\+1\}^\{T\}v\_\{t\},\\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ \\penalty 10000\\ v\_\{0\}=0thus it is

𝔼​\[XT\]=−\[∑t=1T−1f​\(vt\)​∑j=t\+1Tvj\+κ​∑t=1Tvt2\]\.\\mathbb\{E\}\[X\_\{T\}\]=\-\\Big\[\\sum\_\{t=1\}^\{T\-1\}f\(v\_\{t\}\)\\sum\_\{j=t\+1\}^\{T\}v\_\{j\}\+\\kappa\\sum\_\{t=1\}^\{T\}v\_\{t\}^\{2\}\\Big\]\.\(20\)The following proof establishes Proposition \([2\.1](https://arxiv.org/html/2607.06121#S2.Thmproposition1)\)\.

###### Proof\.

Let the vectorλ​v∈ℝT\\lambda v\\in\\mathbb\{R\}^\{T\}then the cost function is written as:

c​\(λ​v\)\\displaystyle c\(\\lambda v\)=−\[∑t=1T−1f​\(λ​vt\)​∑j=t\+1Tλ​vj⏟F​\(λ​v\)\+κ​∑t=1Tλ2​vt2⏟S​\(λ​v\)\]\\displaystyle=\-\\Big\[\\underbrace\{\\sum\_\{t=1\}^\{T\-1\}f\(\\lambda v\_\{t\}\)\\sum\_\{j=t\+1\}^\{T\}\\lambda v\_\{j\}\}\_\{F\(\\lambda v\)\}\+\\underbrace\{\\kappa\\sum\_\{t=1\}^\{T\}\\lambda^\{2\}v\_\{t\}^\{2\}\}\_\{S\(\\lambda v\)\}\\Big\]\(21\)the first term inside the parenthesis is written as

F​\(λ​v\)\\displaystyle F\(\\lambda v\)=\[f​\(λ​v1\)f​\(λ​v2\)⋮f​\(λ​vT−1\)\]⊺​\[∑t=2Tλ​vt000⋯00∑t=3Tλ​vt00⋯0⋮⋮⋮⋮⋯⋮0000⋯λ​vT\]​\[11⋮1\]\\displaystyle=\\begin\{bmatrix\}f\(\\lambda v\_\{1\}\)\\\\ f\(\\lambda v\_\{2\}\)\\\\ \\vdots\\\\ f\(\\lambda v\_\{T\-1\}\)\\end\{bmatrix\}^\{\\intercal\}\\begin\{bmatrix\}\\sum\_\{t=2\}^\{T\}\\lambda v\_\{t\}&0&0&0&\\cdots&0\\\\ 0&\\sum\_\{t=3\}^\{T\}\\lambda v\_\{t\}&0&0&\\cdots&0\\\\ \\vdots&\\vdots&\\vdots&\\vdots&\\cdots&\\vdots\\\\ 0&0&0&0&\\cdots&\\lambda v\_\{T\}\\end\{bmatrix\}\\begin\{bmatrix\}1\\\\ 1\\\\ \\vdots\\\\ 1\\end\{bmatrix\}\(22\)=\|λ\|δ\+1​F​\(v\)\\displaystyle=\|\\lambda\|^\{\\delta\+1\}F\(v\)\(23\)while for the second term it is clearly true that

S​\(λ​v\)=λ2​S​\(v\)\.S\(\\lambda v\)=\\lambda^\{2\}S\(v\)\.hence

c​\(λ​v\)=−\[\|λ\|δ\+1​F​\(v\)\+λ2​S​\(v\)\]c\(\\lambda v\)=\-\[\|\\lambda\|^\{\\delta\+1\}F\(v\)\+\\lambda^\{2\}S\(v\)\]sinceδ<1\\delta<1the second term will dominate the convergence, thus the result in Eq\.\([21](https://arxiv.org/html/2607.06121#A1.E21)\) follows\. For the continuity we simply observe thatc​\(v\)c\(v\)is a sum of continuous functions\. ∎

## Appendix BDerivations for section[3\.1](https://arxiv.org/html/2607.06121#S3.SS1)

We first prove lemma[3\.1](https://arxiv.org/html/2607.06121#S3.Thmlemma1)

###### Proof\.

∑t=1T−1f​\(vt\)​∑j=t\+1Tvj\\displaystyle\\sum\_\{t=1\}^\{T\-1\}f\(v\_\{t\}\)\\sum\_\{j=t\+1\}^\{T\}v\_\{j\}=f​\(α\)​\(−β\)​\(T−n\)​n\+f​\(α\)​α​∑k=1n−1\(n−k\)\+f​\(−β\)​\(−β\)​∑k=1T−n−1\(T−n−k\)\\displaystyle=f\(\\alpha\)\(\-\\beta\)\(T\-n\)n\+f\(\\alpha\)\\alpha\\sum\_\{k=1\}^\{n\-1\}\(n\-k\)\+f\(\-\\beta\)\(\-\\beta\)\\sum\_\{k=1\}^\{T\-n\-1\}\(T\-n\-k\)=f​\(α\)​\(−β\)​\(T−n\)​n\+f​\(α\)​α​\[n​\(n−1\)−n​\(n−1\)2\]\\displaystyle=f\(\\alpha\)\(\-\\beta\)\(T\-n\)n\+f\(\\alpha\)\\alpha\[n\(n\-1\)\-\\frac\{n\(n\-1\)\}\{2\}\]\+f​\(−β\)​\(−β\)​\[\(T−n\)​\(T−n−1\)−\(T−n\)​\(T−n−1\)2\]\\displaystyle\+f\(\-\\beta\)\(\-\\beta\)\[\(T\-n\)\(T\-n\-1\)\-\\frac\{\(T\-n\)\(T\-n\-1\)\}\{2\}\]=n​f​\(α\)​\(T−n\)​\(−β\)\+f​\(α\)​α​n​\(n−1\)2\+f​\(−β\)​\(−β\)​\(T−n\)​\(T−n−1\)2\\displaystyle=nf\(\\alpha\)\(T\-n\)\(\-\\beta\)\+f\(\\alpha\)\\alpha\\frac\{n\(n\-1\)\}\{2\}\+f\(\-\\beta\)\(\-\\beta\)\\frac\{\(T\-n\)\(T\-n\-1\)\}\{2\}=−n​α2​\[\(n\+1\)​f​\(α\)\+f​\(−β\)​\(T−n−1\)\]\\displaystyle=\-\\frac\{n\\alpha\}\{2\}\[\(n\+1\)f\(\\alpha\)\+f\(\-\\beta\)\(T\-n\-1\)\]For the temporary impact term

κ​∑t=1Tvt2\\displaystyle\\kappa\\sum\_\{t=1\}^\{T\}v\_\{t\}^\{2\}=κ​\[n​α2\+\(T−n\)​β2\]\\displaystyle=\\kappa\[n\\alpha^\{2\}\+\(T\-n\)\\beta^\{2\}\]thus

𝔼​\[XT\]=−\[−n​α2​\[\(n\+1\)​f​\(α\)\+f​\(−β\)​\(T−n−1\)\]\+κ​\[n​α2\+\(T−n\)​β2\]\]\.\\mathbb\{E\}\[X\_\{T\}\]=\-\\bigg\[\-\\frac\{n\\alpha\}\{2\}\[\(n\+1\)f\(\\alpha\)\+f\(\-\\beta\)\(T\-n\-1\)\]\+\\kappa\[n\\alpha^\{2\}\+\(T\-n\)\\beta^\{2\}\]\\bigg\]\.∎

We then prove lemma[3\.2](https://arxiv.org/html/2607.06121#S3.Thmlemma2)

###### Proof\.

From Lemma[3\.1](https://arxiv.org/html/2607.06121#S3.Thmlemma1), for anyn∈\[T−1\]n\\in\[T\-1\]the expected cash flow is written as

𝔼​\[XT\]\\displaystyle\\mathbb\{E\}\[X\_\{T\}\]=n​α2​\[\(n\+1\)​θ​αδ−\(T−n−1\)​\(n​αT−n\)δ\]−κ​\[n​α2\+n2​α2T−n\]\\displaystyle=\\frac\{n\\alpha\}\{2\}\\Big\[\(n\+1\)\\theta\\alpha^\{\\delta\}\-\(T\-n\-1\)\\Big\(\\frac\{n\\alpha\}\{T\-n\}\\Big\)^\{\\delta\}\\Big\]\-\\kappa\\Big\[n\\alpha^\{2\}\+\\frac\{n^\{2\}\\alpha^\{2\}\}\{T\-n\}\\Big\]=αδ\+1​n​θ2​\[\(n\+1\)−\(T−n−1\)​\(nT−n\)δ\]⏟A​\(n\)−α2​κ​\[n\+n2T−n\]⏟B​\(n\)\\displaystyle=\\alpha^\{\\delta\+1\}\\underbrace\{\\frac\{n\\theta\}\{2\}\\Big\[\(n\+1\)\-\(T\-n\-1\)\\Big\(\\frac\{n\}\{T\-n\}\\Big\)^\{\\delta\}\\Big\]\}\_\{A\(n\)\}\-\\alpha^\{2\}\\underbrace\{\\kappa\\Big\[n\+\\frac\{n^\{2\}\}\{T\-n\}\\Big\]\}\_\{B\(n\)\}\(24\)We observe that𝔼​\[XT\]\>0\\mathbb\{E\}\[X\_\{T\}\]\>0if and only if

A​\(n\)\>0⇔n\+1\>\(T−n−1\)​\(nT−n\)δA\(n\)\>0\\Leftrightarrow n\+1\>\(T\-n\-1\)\\Big\(\\frac\{n\}\{T\-n\}\\Big\)^\{\\delta\}and in that case𝔼​\[XT\]\\mathbb\{E\}\[X\_\{T\}\]is a concave function ofα\\alpha, thus to find the optimalα\\alphawe compute the derivative with respect toα\\alphain Eq\.\([B](https://arxiv.org/html/2607.06121#A2.Ex36)\):

∂𝔼​\[XT\]∂α\\displaystyle\\frac\{\\partial\\mathbb\{E\}\[X\_\{T\}\]\}\{\\partial\\alpha\}=\(δ\+1\)​αδ​A​\(n\)−2​α​B​\(n\)⇔\\displaystyle=\(\\delta\+1\)\\alpha^\{\\delta\}A\(n\)\-2\\alpha B\(n\)\\Leftrightarrowα∗\\displaystyle\\alpha^\{\*\}=\(\(δ\+1\)​A​\(n\)2​B​\(n\)\)11−δ\.\\displaystyle=\\bigg\(\\frac\{\(\\delta\+1\)A\(n\)\}\{2B\(n\)\}\\bigg\)^\{\\frac\{1\}\{1\-\\delta\}\}\.Hence the cash flow of the optimal strategy is then

maxn∈\[T−1\]⁡𝔼​\[XT\]​\(α∗,n\)\.\\max\_\{n\\in\[T\-1\]\}\\mathbb\{E\}\[X\_\{T\}\]\(\\alpha^\{\*\},n\)\.\(25\)∎

## Appendix CProperties of the local maxima of the optimization problem

We analyze the trading strategies obtained by SLSQP fromNNdifferent initializations, focusing on how similar the resulting strategies are in terms of their final cash flow\. To compare two strategiesv→i,v→j∈ℝT\\vec\{v\}\_\{i\},\\vec\{v\}\_\{j\}\\in\\mathbb\{R\}^\{T\}, we use the cosine similarity888The cosine similarity between two round\-trip strategies corresponds to their Pearson correlation \(see alsoZhelezniak et al\.\[[2019](https://arxiv.org/html/2607.06121#bib.bib29)\]\)\.\[Tan et al\.,[2013](https://arxiv.org/html/2607.06121#bib.bib27)\]

c​\(i,j\)=⟨v→i,v→j⟩‖v→i‖​‖v→j‖,c\(i,j\)=\\frac\{\\langle\\vec\{v\}\_\{i\},\\vec\{v\}\_\{j\}\\rangle\}\{\\\|\\vec\{v\}\_\{i\}\\\|\\\|\\vec\{v\}\_\{j\}\\\|\},which measures the angle between the two vectors and lies in\[−1,1\]\[\-1,1\]\. Since a strategy and its sign\-reversed counterpart produce the same final cash position, we use the sign\-invariant distance

d​\(i,j\)=1−\|c​\(i,j\)\|\.d\(i,j\)=1\-\|c\(i,j\)\|\.This distance treats aligned and anti\-aligned strategies as equivalent\. We then examine the pairwise distance matrices for all SLSQP terminal strategies\. As expected \(see Fig\. \([6](https://arxiv.org/html/2607.06121#A3.F6)\)\), nearby strategies in the ordering tend to have smaller distances\. However, the plots alone do not reveal whether similar strategies necessarily yield similar expected cash\. Therefore, for all pairs satisfyingd​\(i,j\)≤ηd\(i,j\)\\leq\\etawithη=0\.1\\eta=0\.1, we compute\|𝔼i​\[XT\]−𝔼j​\[XT\]\|\|\\mathbb\{E\}\_\{i\}\[X\_\{T\}\]\-\\mathbb\{E\}\_\{j\}\[X\_\{T\}\]\|\(see Fig\. \([7](https://arxiv.org/html/2607.06121#A3.F7)\)\)\. The resulting scatter plot shows that strategies can be close under the sign\-invariant distance while still producing different expected cash, indicating structural differences despite high similarity\. Finally in Fig\. \([8](https://arxiv.org/html/2607.06121#A3.F8)\) are illustrated two strategies with small sign\-invariant distance but relatively large cash discrepancy\.

![Refer to caption](https://arxiv.org/html/2607.06121v1/figures/CosineSim_SignInv.png)Figure 6:The heatmap of the sign\-invariant distance on the left for all the strategies and on the right with precision of two colors for all the strategies in Fig\. \([2](https://arxiv.org/html/2607.06121#S3.F2)\)\.![Refer to caption](https://arxiv.org/html/2607.06121v1/figures/discrepancy_eta0.1.png)Figure 7:Each point represents a pair of strategies whose sign\-invariant distance is less than or equal to the thresholdη\\eta\.![Refer to caption](https://arxiv.org/html/2607.06121v1/figures/inventory_cash_plot_bk_213.png)
![Refer to caption](https://arxiv.org/html/2607.06121v1/figures/inventory_cash_plot_bk_452.png)

Figure 8:The two strategies with the largest discrepancy in expected cash for whichd​\(i,j\)=0\.0855d\(i,j\)=0\.0855\.
## Appendix DEstimating the Parameters in SLSQP

In this Appendix we justify the derivation for the estimation of the permanent impact parameterθ\\thetaas a function of the exponent impact parameterδ\\delta\. Since

δ^=argminδ∈\(0,1\)​∑i=1N\(Ii−θ^​\(δ\)​TS0​\(qiT\)δ\)2⏟RSS​\(θ^\)\\hat\{\\delta\}=\\text\{argmin\}\_\{\\delta\\in\(0,1\)\}\\underbrace\{\\sum\_\{i=1\}^\{N\}\\Big\(I\_\{i\}\-\\hat\{\\theta\}\(\\delta\)\\frac\{T\}\{S\_\{0\}\}\\Big\(\\frac\{q\_\{i\}\}\{T\}\\Big\)^\{\\delta\}\\Big\)^\{2\}\}\_\{\\text\{RSS\}\(\\hat\{\\theta\}\)\}\(26\)for the simplification in the notation we use:θ^​\(δ\):=θ^\\hat\{\\theta\}\(\\delta\):=\\hat\{\\theta\}\. The RSS is a concave function inθ^\\hat\{\\theta\}thus there exist unique minimizer, it is then enough to consider the first derivative:

∂RSS∂θ^\\displaystyle\\frac\{\\partial\\text\{RSS\}\}\{\\partial\\hat\{\\theta\}\}=0⇔\\displaystyle=0\\Leftrightarrow−2​TS0​∑i=1N\(qiT\)δ​\(Ii−θ^​TS0​\(qiT\)δ\)\\displaystyle\-2\\frac\{T\}\{S\_\{0\}\}\\sum\_\{i=1\}^\{N\}\\Big\(\\frac\{q\_\{i\}\}\{T\}\\Big\)^\{\\delta\}\\Big\(I\_\{i\}\-\\hat\{\\theta\}\\frac\{T\}\{S\_\{0\}\}\\Big\(\\frac\{q\_\{i\}\}\{T\}\\Big\)^\{\\delta\}\\Big\)=0⇔\\displaystyle=0\\Leftrightarrowθ^​\(δ\)\\displaystyle\\hat\{\\theta\}\(\\delta\)=S0T​∑i=1N\(qiT\)δ​Ii∑i=1N\(qiT\)2​δ\.\\displaystyle=\\frac\{S\_\{0\}\}\{T\}\\frac\{\\sum\_\{i=1\}^\{N\}\(\\frac\{q\_\{i\}\}\{T\}\)^\{\\delta\}I\_\{i\}\}\{\\sum\_\{i=1\}^\{N\}\(\\frac\{q\_\{i\}\}\{T\}\)^\{2\\delta\}\}\.
![Refer to caption](https://arxiv.org/html/2607.06121v1/figures/consist_estim.png)
![Refer to caption](https://arxiv.org/html/2607.06121v1/figures/consist_estim_sigma003.png)
![Refer to caption](https://arxiv.org/html/2607.06121v1/figures/consist_estim_sigma002.png)

Figure 9:Estimation of the impact parameters along with their standard errors as a function of the number of metaordersNN\(a\) whenσ=0\.0168\\sigma=0\.0168, \(b\) whenσ=0\.003\\sigma=0\.003and \(c\) whenσ=0\.002\\sigma=0\.002\.Fig\. \([9](https://arxiv.org/html/2607.06121#A4.F9)\) depicts the estimated impact parameters as a function of the number of metaordersNNfor all three values ofσ\\sigma\. We observe that the estimators are consistent since the standard errors get decrease asNNincreases and approach the true value of the parameters\. Fig\. \([10](https://arxiv.org/html/2607.06121#A4.F10)\) illustrates the strategy obtained with theregSLSQPN\\text\{regSLSQP\}\_\{N\}forN=500N=500andN=5000N=5000metaorders when the volatility is0\.01680\.0168in panels \(a\) and \(d\), when the volatility is0\.0030\.003in panels \(b\) and \(e\) and when the volatility is0\.0020\.002in panels \(c\) and \(f\)\. Clearly the expected cash\-flow is positive since we use the complete information of the price dynamics for its computation\.

![Refer to caption](https://arxiv.org/html/2607.06121v1/figures/regSLSQP_sigma0168_N500.png)
![Refer to caption](https://arxiv.org/html/2607.06121v1/figures/regSLSQP_sigma003_N500.png)
![Refer to caption](https://arxiv.org/html/2607.06121v1/figures/regSLSQP_sigma002_N500.png)
![Refer to caption](https://arxiv.org/html/2607.06121v1/figures/regSLSQP_sigma0168_N5000.png)
![Refer to caption](https://arxiv.org/html/2607.06121v1/figures/regSLSQP_sigma003_N5000.png)
![Refer to caption](https://arxiv.org/html/2607.06121v1/figures/regSLSQP_sigma002_N5000.png)

Figure 10:Plots\(a\-c\)whenN=500N=500and\(d\-f\)whenN=5000N=5000metaorders\. In each panel,Top\-Left:The optimal trading strategy byregSLSQPN\\text\{regSLSQP\}\_\{N\}and the corresponding inventory \(red line\)\.Bottom\-Left:The expected cash\-flow\. The green \(red\) area indicates the region for which𝔼​\[Xt\]\>0\\mathbb\{E\}\[X\_\{t\}\]\>0\(𝔼​\[Xt\]<0\\mathbb\{E\}\[X\_\{t\}\]<0\)\.Right:The histogram of𝔼​\[XT\]\\mathbb\{E\}\[X\_\{T\}\]from the different starting points\.
## Appendix EThe training phase of DDPG

Here we plot the actions’ aggressiveness \(see Eq\.\([17](https://arxiv.org/html/2607.06121#S4.E17)\)\) on the left plot of Fig\. \([11](https://arxiv.org/html/2607.06121#A5.F11)\) and the actions’ directionality \(see Eq\.\([18](https://arxiv.org/html/2607.06121#S4.E18)\)\) on the right plot of Fig\. \([11](https://arxiv.org/html/2607.06121#A5.F11)\) during the training phase of DDPG when the reward is

rt=−St​vt−κ​vt2r\_\{t\}=\-S\_\{t\}v\_\{t\}\-\\kappa v^\{2\}\_\{t\}\(27\)
![Refer to caption](https://arxiv.org/html/2607.06121v1/figures/action_metrics_4_runs_rev.png)Figure 11:The 25\-episode moving averagesLeft:of the action aggressiveness andRight:of signed action directionality during training and the reward is defined as in Eq\. \([27](https://arxiv.org/html/2607.06121#A5.E27)\)\.We observe that, although the level of aggressiveness remains approximately the same throughout the training episodes with that of Fig\. \([5](https://arxiv.org/html/2607.06121#S4.F5)\), the directionality of the actions is reversed due to the negative sign in the reward function, relative to the directionality obtained when the reward is defined as in Eq\.\([11](https://arxiv.org/html/2607.06121#S4.E11)\)\.