Do LLMs Understand Limit Order Book Dynamics?

# Do LLMs Understand Limit Order Book Dynamics?
Source: [https://arxiv.org/html/2608.23706](https://arxiv.org/html/2608.23706)
Paul GlassermanAffiliation:\{J1Chen27, pg20\}@gsb\.columbia\.edu

###### Abstract

A large language model \(LLM\) trained on synthetic limit order book \(LOB\) data achieves near perfect scores in generating valid sequences of LOB events\. However, the LLM’s implicit world model fails to learn the state of the LOB\. This deficiency leads to biased estimates and spurious predictability in using the LLM to forecast future LOB events\. Our analysis uses novel tests of an LLM’s world model, extending prior work from deterministic settings to the stochastic dynamics needed for the LOB\.

## 1Introduction

If a large language model \(LLM\) learns to recognize and generate valid sequences of limit order book \(LOB\) events — order arrivals, execution, and cancellations — has it developed a correct world model of the LOB? Does such an understanding matter? These are the questions we investigate\.

We train a transformer\-based LLM from scratch using events in a simulated LOB\. The LLM performs extremely well in generating valid sequences and in finding sequences that move the LOB from a starting state to a target state\. By these measures, the LLM appears to understand key concepts underlying an LOB: transactions can only occur at the best bid or ask price; market orders can only execute against opposite side resting orders; a cancellation order requires the presence of a limit order to be canceled\.

At the same time, we find evidence of systematic errors in the LLM’s implicit world model of the LOB\. A correct understanding of the LOB’s dynamics becomes important in applying the trained LLM to a task related to but different from the task on which it was trained\.

In our setting, the related task is forecasting\. We have in mind a trader or market\-maker using the trained LLM to predict, e\.g\., whether the next order will be a buy or a sell order\. We find consistent evidence of spurious predictability: the LLM finds predictability where none exists, and thus delivers misleading forecasts\. This is where the abstract concept of an incorrect world model takes on practical consequences\.

By using simulated data, we are able to ensure that the state of the data\-generating LOB is a Markov chain\. Given the current state, the past history of the LOB is therefore irrelevant to its future evolution, yet the LLM’s forecasts are influenced by past events\. Our framework thus evaluates the LLM under the null hypothesis of no predictability\. Events in an actual LOB may well exhibit some predictability, but an LLM’s ability to provide reliable forecasts is suspect if it finds predictability under the null\. More fundamentally, its implicit world model of an LOB appears to be flawed\.

Prior work on inferring and evaluating LLMs’ world models includes\[[4](https://arxiv.org/html/2608.23706#bib.bib4),[7](https://arxiv.org/html/2608.23706#bib.bib7),[10](https://arxiv.org/html/2608.23706#bib.bib10)\]in the setting of games,\[[3](https://arxiv.org/html/2608.23706#bib.bib3)\]in task planning,\[[8](https://arxiv.org/html/2608.23706#bib.bib8),[12](https://arxiv.org/html/2608.23706#bib.bib11)\]in deterministic finite automata \(DFA\), and\[[11](https://arxiv.org/html/2608.23706#bib.bib12)\]in the setting of physical laws\. In these cases, the “true” model is deterministic whereas the evolution of an LOB is inherently stochastic\. We adapt and apply some tests studied in the general framework of\[[12](https://arxiv.org/html/2608.23706#bib.bib11)\]\.

Our focus on biased forecasts and spurious predictability is new and particularly relevant to the financial setting\. We introduce new tests for this property\. Ourkernel\-level total variation \(TV\)measures deviations between an LLM’s next\-token probabilities and the true distribution kernel\. Ourhistory\-level total variation \(TV\)measures the erroneous influence of past events on an LLM’s forecast distribution\. We also apply a regression test to evaluate spurious predictability or miscalibration in the probabilities of future events using information on past events\.

From a practical perspective, our results indicate that valid\-sequence tests are insufficient in training an LLM for downstream tasks \(such as forecasting events\) in LOB\. Larger training datasets or new training methods are needed to improve an LLM’s world model of an LOB\. Our results use small LOB settings to make training from scratch feasible; the issues we document would likely be more severe with a larger state space\. A growing literature focuses on generating large synthetic financial datasets; see, e\.g\.,\[[1](https://arxiv.org/html/2608.23706#bib.bib1),[6](https://arxiv.org/html/2608.23706#bib.bib6),[9](https://arxiv.org/html/2608.23706#bib.bib9)\]\.

Section[2](https://arxiv.org/html/2608.23706#S2)reviews LOBs and formulates our LOB dynamics\. Section[3](https://arxiv.org/html/2608.23706#S3)constructs the datasets\. Section[4](https://arxiv.org/html/2608.23706#S4)describes our model training process\. Section[5](https://arxiv.org/html/2608.23706#S5)introduces a simple baseline for comparison\. Section[6](https://arxiv.org/html/2608.23706#S6)uses existing diagnostics to evaluate our models\. Section[7](https://arxiv.org/html/2608.23706#S7)introduces and applies our new tests\. Section[8](https://arxiv.org/html/2608.23706#S8)tests larger LOB settings\. Section[9](https://arxiv.org/html/2608.23706#S9)concludes\.

## 2Limit Order Book

### 2\.1States

Figure[1](https://arxiv.org/html/2608.23706#S2.F1)illustrates a limit order book\. This example has five fixed price levelsp1\>⋯\>p5p\_\{1\}\>\\cdots\>p\_\{5\}\. There arev1=1v\_\{1\}=1orders to sell at pricep1p\_\{1\}andv2=2v\_\{2\}=2orders to sell atp2p\_\{2\}; there arev4=3v\_\{4\}=3orders to buy at pricep4p\_\{4\}andv5=2v\_\{5\}=2orders to buy atp5p\_\{5\}\. The notationℓa=2\\ell^\{a\}=2indicates that the lowest asking price isp2p\_\{2\}, and similarlyhb=4h^\{b\}=4indicates that the highest bid price isp4p\_\{4\}\.

Figure 1:Representation of a limit order bookMore generally, we represent the state of the LOB by the vector\(ℓa,hb,v1,…,vP\)\(\\ell^\{a\},h^\{b\},v\_\{1\},\\dots,v\_\{P\}\), wherePPis the fixed number of possible price levels\. Fori=1,…,ℓai=1,\\dots,\\ell^\{a\},viv\_\{i\}is the quantity of sell orders askingpip\_\{i\}\. Fori=hb,…,Pi=h^\{b\},\\dots,P,viv\_\{i\}is the quantity of buy orders biddingpip\_\{i\}\. If there are no bids \(buy orders\), thenhb=P\+1h^\{b\}=P\+1, and if there are no ask \(sell\) orders thenℓa=0\\ell^\{a\}=0\.

Any price levels \(p3p\_\{3\}in the figure\) between the bids at the bottom and the asks at the top constitute the spread region and should havevi=0v\_\{i\}=0\. To keep the state space finite, we cap the total number of orders at any price level atV1V\_\{1\}, and the size of each individual order atV2V\_\{2\}\. \(See, e\.g\.,\[[2](https://arxiv.org/html/2608.23706#bib.bib2)\]for general background on limit order books\.\)

### 2\.2Events and State Transitions

The state of the LOB evolves through the arrival of limit orders, cancellation orders, and market orders\. We associate a token with each possible event\. The tokenask\_p\_i\_vrepresents the arrival of an ask order to sellvvshares at pricepip\_\{i\}\. A limit buy order is similarly written asbid\_p\_i\_v\. The cancellation ordersw\_ask\_p\_i\_vandw\_bid\_p\_i\_vremovevvunits from the corresponding side at pricepip\_\{i\}\. A market sell orderm\_ask\_vconsumesvvunits from the bid side starting from the best bidphbp\_\{h^\{b\}\}; a market buy orderm\_bid\_vconsumesvvunits from the ask side starting from the best askpℓap\_\{\\ell^\{a\}\}\.

Limit orders follow standard matching logic\. If an arriving limit buy order bids at a pricepip\_\{i\}below the lowest ask pricepℓap\_\{\\ell^\{a\}\}, the order rests in the book at pricepip\_\{i\}, provided that the resulting volume does not exceedV1V\_\{1\}\. If instead the submitted limit buy order bids atpi≥pℓap\_\{i\}\\geq p\_\{\\ell^\{a\}\}\(a crossing bid\), it executes against the opposite side starting from the lowest asking pricepℓap\_\{\\ell^\{a\}\}\. If the submitted size is larger than the available sell volume at prices less than or equal to the bidpip\_\{i\}, the unfilled residual rests atpip\_\{i\}\. The arrival of limit sell orders works analogously\. Cancellation orders are valid only when the corresponding side and price level currently contain at least the requested cancellation volume\. Market orders are valid only if the opposite side has enough total volume to fully execute the order\.

The rules of the LOB determine the set of valid events𝒜⁡\(s\)\\mathcal\{A\}\(s\)in each statess\. Each valid eventa∈𝒜⁡\(s\)a\\in\\mathcal\{A\}\(s\)deterministically maps the LOB statessto a new states′=T⁡\(s,a\)s^\{\\prime\}=T\(s,a\), as in a deterministic finite automaton \(DFA\), the framework used in\[[12](https://arxiv.org/html/2608.23706#bib.bib11)\]\. There may be more than one event that moves the state fromsstos′s^\{\\prime\}\.

Once we specify the probabilityμ⁡\(a\|s\)\\mu\(a\|s\)of each eventaain each statess, the evolution of the state becomes a Markov chain\. The transition probability fromsstos′s^\{\\prime\}is given by

K\(s,s′\)=∑a∈𝒜⁡\(s\):T⁡\(s,a\)=s′μ\(a\|s\)\.K\(s,s^\{\\prime\}\)=\\sum\_\{a\\in\\mathcal\{A\}\(s\):T\(s,a\)=s^\{\\prime\}\}\\mu\(a\|s\)\.The resulting finite\-state Markov chain is irreducible and aperiodic, and therefore admits a unique stationary distributionπ\\pi, which satisfiesπ​K=π\\pi K=\\pi\. For simplicity, we assume that, in each LOB state, all valid events \(tokens\) are equally likely,μ⁡\(a\|s\)=1/\|𝒜⁡\(s\)\|\\mu\(a\|s\)=1/\|\\mathcal\{A\}\(s\)\|\. We have obtained consistent results using non\-uniform kernels that assign higher probabilities to buy or sell orders, inducing upward or downward price pressure\.

## 3Synthetic Datasets Construction

We associate a state ID token with every LOB state, and, as described above, we associate a token with every event\. Each training sequence is represented as

sstartsgoala1a2⋯aTend,s\_\{\\mathrm\{start\}\}\\;\\;s\_\{\\mathrm\{goal\}\}\\;\\;a\_\{1\}\\;\\;a\_\{2\}\\;\\;\\cdots\\;\\;a\_\{T\}\\;\\;\\texttt\{end\},\(1\)wheresstarts\_\{\\mathrm\{start\}\}is the initial state ID,sgoals\_\{\\mathrm\{goal\}\}is a target state ID, anda1,…,aTa\_\{1\},\\ldots,a\_\{T\}are valid LOB events\. The special tokenendis valid only when current state equals the target state\. We train an LLM to respond to a prompt of the form\(sstart,sgoal\)\(s\_\{\\mathrm\{start\}\},s\_\{\\mathrm\{goal\}\}\)by generating a sequence of valid event tokens that move the LOB fromsstarts\_\{\\mathrm\{start\}\}tosgoals\_\{\\mathrm\{goal\}\}\.

This format requires the LLM to learn which sequences of events are valid from any LOB statesstarts\_\{\\mathrm\{start\}\}\. But it goes further in requiring the model to learn how to reach a target statesgoals\_\{\\mathrm\{goal\}\}, which suggests an ability to plan or anticipate the impact of multiple events and thus a deeper understanding of the LOB’s transition structure\.111The goal state acts as a conditioning signal, analogous to contextual conditioning in standard language modeling, requiring the model to learn multi\-step reachability rather than only local transition probabilities\. This does not conflict with our subsequent prediction tests, since this conditioning is averaged out over goal states drawn from the same data\-generation process\.

Following\[[12](https://arxiv.org/html/2608.23706#bib.bib11)\], we generate two types of datasets of sequences \([1](https://arxiv.org/html/2608.23706#S3.E1)\): random walks and shortest paths\.

For the random walk dataset, we drawsstarts\_\{\\mathrm\{start\}\}from the stationary distributionπ\\piof the LOB Markov chain\. We generatea1a\_\{1\}fromμ\(⋅\|sstart\)\\mu\(\\cdot\|s\_\{\\mathrm\{start\}\}\), the next\-token distribution in the current state\. The eventa1a\_\{1\}moves the LOB to a new states′s^\{\\prime\}\. With probabilitypstopp\_\{\\text\{stop\}\}, the sequence terminates and we setsgoal=s′s\_\{\\mathrm\{goal\}\}=s^\{\\prime\}\. With probability1−pstop1\-p\_\{\\text\{stop\}\}, we generatea2a\_\{2\}froms′s^\{\\prime\}, and the process repeats\. This construction gives the sequence lengthTTa geometric distribution with mean1/pstop1/p\_\{\\text\{stop\}\}\. The memoryless property of the geometric distribution implies that the number of events observed thus far along a sequence carries no information about how many more events will be observed before the sequence terminates\. In our experiments, we cap the maximum sequence length to keep the length bounded\.

In the shortest\-path dataset, we require that \([1](https://arxiv.org/html/2608.23706#S3.E1)\) move from the start to the goal state with the fewest possible transitionsTT\. Details of the construction are given in Appendix[A\.1](https://arxiv.org/html/2608.23706#A1.SS1)\.

The random walk dataset is intended to approximate observational data\. Given a history of events in an actual LOB, one could randomly divide the historical sequence into segments, record the start and end states for each segment, and train an LLM to generate similar segments\. The shortest\-path dataset is for comparison; we expect the shortest paths to reveal more about the LOB dynamics but to explore the state space less thoroughly\.

## 4LLM Training

For each dataset, we train a decoder\-only transformer model on token sequences of the form \([1](https://arxiv.org/html/2608.23706#S3.E1)\)\. We use a GPT\-style causal decoder architecture with 12 transformer layers, embedding dimension 768, and 12 attention heads, giving approximately 86\.4 million trainable parameters\. We have obtained very similar qualitative results using a larger transformer structure with 48 layers, embedding dimension 1600, and 25 attention heads, which gives approximately 1\.5 billion trainable parameters\.

Each transformer model is trained with the standard next\-token prediction objective using a cross\-entropy loss function\. Given a token sequencea1,…,aNa\_\{1\},\\dots,a\_\{N\}, the model estimatespθ​\(at\|a<t\)p\_\{\\theta\}\(a\_\{t\}\|a\_\{<t\}\), wherea<ta\_\{<t\}denotes the sequence of tokens precedingata\_\{t\}, andθ\\thetadenotes the model’s parameters\. Interpretpθ​\(a\|a<t\)p\_\{\\theta\}\(a\|a\_\{<t\}\)as the probability the model assigns to tokenaato followa<ta\_\{<t\}\. The parametersθ\\thetaare chosen through approximate minimization of the loss function

L\(θ\)=−∑i=1Nlogpθ\(at\|a<t\)\.L\(\\theta\)=\-\\sum\_\{i=1\}^\{N\}\\log p\_\{\\theta\}\(a\_\{t\}\|a\_\{<t\}\)\.For batched training over many sequences, this loss is summed over all non\-padding tokens in the batch\.

For the experiments in Sections[6](https://arxiv.org/html/2608.23706#S6)to[7](https://arxiv.org/html/2608.23706#S7), we use the base LOB setting\(P,V1,V2\)=\(3,5,2\)\(P,V\_\{1\},V\_\{2\}\)=\(3,5,2\), and we train three models:

- •SP: trained on 500K shortest\-path segments with geometric mean length 5 and cap 20\.
- •RW\-S: trained on 10M random\-walk segments with geometric mean length 20 and cap 60\.
- •RW\-L: trained on 10M random\-walk segments with geometric mean length 50 and cap 150\.

We use 99% of each dataset for training and reserve 1% for heldout evaluation\. All training is performed using mixed precision on a Supermicro AS\-4124GO\-NART\+ server with eight NVIDIA A100\-SXM4\-80GB GPUs\. The SP model is trained for 10 epochs\. We run a validation every 5000 parameter updates and select the checkpoint with lowest validation loss\. The RW models are trained for one epoch, and the final checkpoint is used\. The training time is 4\-6 hours for the SP model, and 1\-2 days for the RW models\.

In Section[8](https://arxiv.org/html/2608.23706#S8), we repeat the same pipeline on two larger LOB settings\. One increases the depth capacity, using\(P,V1,V2\)=\(3,12,3\)\(P,V\_\{1\},V\_\{2\}\)=\(3,12,3\), and the other increases the number of price levels, using\(P,V1,V2\)=\(8,2,2\)\(P,V\_\{1\},V\_\{2\}\)=\(8,2,2\)\. For these scaling experiments, the training datasets are enlarged to two and three times the size of the corresponding base datasets, respectively, and the training time increases approximately proportionally with the amount of training data\.

## 5Empirical\-Kernel Baseline

To separate errors caused by the training data from errors caused by the LLM’s learned representation, we also construct a simple empirical\-kernel baseline\. This baseline has no hidden state or attention mechanism, it predicts the next event only based on the current LOB state by using empirical next\-token frequencies in the training data\. For each statessand event tokenaa\(including theendtoken\), letNtrain​\(s,a\)N\_\{\\mathrm\{train\}\}\(s,a\)be the number of times eventaaappears immediately after statessin the training data, and letNtrain​\(s\)=∑a′∈𝒜Ntrain​\(s,a′\)N\_\{\\mathrm\{train\}\}\(s\)=\\sum\_\{a^\{\\prime\}\\in\\mathcal\{A\}\}N\_\{\\mathrm\{train\}\}\(s,a^\{\\prime\}\)be the total number of observed outgoing events fromss, using the RW\-L dataset\. The empirical\-kernel baseline predicts

p^emp​\(a∣s\)=Ntrain​\(s,a\)Ntrain​\(s\)\\widehat\{p\}\_\{\\mathrm\{emp\}\}\(a\\mid s\)=\\frac\{N\_\{\\mathrm\{train\}\}\(s,a\)\}\{N\_\{\\mathrm\{train\}\}\(s\)\}For each LOB setting, the baseline is trained on the same dataset as the RW\-L model\. In our experiments, more than 99\.8% of evaluation states are observed in the training data and baseline results are computed over these covered states\. This simple model provides a useful comparison: if this empirical\-kernel baseline does not exhibit the same errors we find in the LLM, then those patterns are unlikely to be a consequence of the training data\. Instead, they arise from the representation the LLM learns\.

## 6Model Diagnostics

### 6\.1Valid Traversal Test

As a first test of the trained LLM’s understanding of the LOB, we evaluate its ability to generate \(only\) valid sequences of tokens\. We prompt the model with a heldout pair\(s0,sg\)\(s\_\{0\},s\_\{g\}\)\. Recall that state IDs are themselves tokens, so we can evaluate the model’s next\-token distributionpθ\(⋅\|s0,sg\)p\_\{\\theta\}\(\\cdot\|s\_\{0\},s\_\{g\}\)and choose tokena^1\\hat\{a\}\_\{1\}to maximize the next\-token probability\. We then choosea^2\\hat\{a\}\_\{2\}to maximizepθ\(⋅\|s0,sg,a^1\)p\_\{\\theta\}\(\\cdot\|s\_\{0\},s\_\{g\},\\hat\{a\}\_\{1\}\)and so on to generate a sequences0,sg,a^1,a^2,⋯,a^Ts\_\{0\},s\_\{g\},\\hat\{a\}\_\{1\},\\hat\{a\}\_\{2\},\\cdots,\\hat\{a\}\_\{T\},end\. We refer to this as greedy generation\.

The generated trajectory is valid if every generated event is valid under the true LOB transition rules and the terminal token appears only when the current state equals the target\. Formally, \(recalling𝒜\\mathcal\{A\}andT⁡\(⋅,⋅\)T\(\\cdot,\\cdot\)from the end of Section[2\.2](https://arxiv.org/html/2608.23706#S2.SS2)\) starting froms^0=s0\\hat\{s\}\_\{0\}=s\_\{0\}, each generated event must satisfy

a^t∈𝒜⁡\(s^t−1\),s^t=T⁡\(s^t−1,a^t\),\\hat\{a\}\_\{t\}\\in\\mathcal\{A\}\(\\hat\{s\}\_\{t\-1\}\),\\quad\\hat\{s\}\_\{t\}=T\(\\hat\{s\}\_\{t\-1\},\\hat\{a\}\_\{t\}\),andendis emitted only whens^T=sg\\hat\{s\}\_\{T\}=s\_\{g\}\. We evaluate the fraction of valid sequences for each model\.

In the experiments below, this evaluation is carried out on the testing split of each dataset\. In addition to the valid traversal rate, we also report two shortest\-path diagnostics\. The SP Rate is the fraction of generated trajectories that reachsgs\_\{g\}using a shortest path in the LOB state graph\. The Extra Steps statistic is the generated path length minus the shortest\-path distance betweens0s\_\{0\}andsgs\_\{g\}, averaged over valid generated trajectories\.

Success in generating \(only\) valid sequences suggests an understanding of LOB dynamics\. In contrast, executing a market buy order at a higher price than the lowest resting sell order would suggest a lack of understanding; so would generating a cancellation of a nonexistent limit order\. Using sequences of the form \([1](https://arxiv.org/html/2608.23706#S3.E1)\) requires substantially more than this basic understanding of LOB rules by requiring the LLM to steer the state to a fixed goal in generation\.

Table[1](https://arxiv.org/html/2608.23706#S6.T1)shows that that all three LLMs score very high on the valid traversal test, nearly always generating valid sequences\. As expected, the baseline performs poorly on this task because it has no mechanism for goal\-directed sequence generation\. TheSP Ratecolumn shows that 99\.4% of sequences generated by the SP model also achieve the shortest length\. Interestingly, the RW models also tend to generate short sequences\. In the RW\-S model, 90\.6% of generated sequences achieve the shortest path to the goal, even though only 10\.4% of sequences in its training set have this property\. Also, the generated sequences of RW\-S model exceed the shortest path by only 0\.1 steps on average, whereas in its training data the value is 16\.3\. The RW\-L results in the table show a similar pattern\. The fact that RW models generate shorter sequences to reachsgs\_\{g\}further suggests an understanding of the LOB dynamics, since their training sequences are not constructed to be the shortest paths\.

Table 1:Diagnostic resultsModelValidSPExtraCompress\.TraversalRateStepsScoreSP99\.4%99\.4% \(100%\)0\.0 \(0\.0\)61\.1%RW\-S98\.5%90\.6% \(10\.4%\)0\.1 \(16\.3\)42\.0%RW\-L95\.5%83\.7% \(4\.2%\)0\.3 \(43\.8\)48\.0%Baseline0\.7%0\.7% \(4\.2%\)0\.0 \(43\.8\)99\.9%
### 6\.2Compression Test

The compression test, introduced in\[[12](https://arxiv.org/html/2608.23706#bib.bib11)\], examines an LLM’s understanding of the state of the real system\. Given the current state, an LLM with a correct world model should not be influenced by the history leading to that state in generating future sequences\. The test in\[[12](https://arxiv.org/html/2608.23706#bib.bib11)\]was formulated in a deterministic setting; we adapt it to our setting for consistency with our data generating process\.

We draw an initial states0s\_\{0\}from the stationary distributionπ\\pi\. We simulate forward a geometric number of steps using the true LOB dynamics and record the state reachedsgs\_\{g\}; this will serve as the goal state\. We then generate event histories by simulating backwards from states0s\_\{0\}\. By Bayes’ rule, states0s\_\{0\}is preceded by states′s^\{\\prime\}and eventaa, whereT⁡\(s′,a\)=s0T\(s^\{\\prime\},a\)=s\_\{0\}, with probability proportional toπ⁡\(s′\)​μ​\(a\|s′\)\\pi\(s^\{\\prime\}\)\\mu\(a\|s^\{\\prime\}\)\. Repeating this backward\-sampling process produces an event history; by running the procedure twice we produce two histories \(valid token sequences\)h1h\_\{1\}andh2h\_\{2\}that terminate ins0s\_\{0\}\.

From the historyh1h\_\{1\}we use the LLM to generate suffix sequencesz=\(z1,…,zm\)z=\(z\_\{1\},\\dots,z\_\{m\}\)withz1z\_\{1\}drawn frompθ\(⋅\|h1,sg\)p\_\{\\theta\}\(\\cdot\|h\_\{1\},s\_\{g\}\),z2z\_\{2\}drawn frompθ\(⋅\|h1,sg,z1\)p\_\{\\theta\}\(\\cdot\|h\_\{1\},s\_\{g\},z\_\{1\}\), and so on\. If the LLM’s world model aligns with the LOB state, any suffixzzgenerated fromh1h\_\{1\}should be feasible \(according to the LLM\) followingh2h\_\{2\}\. Given a thresholdϵ\>0\\epsilon\>0, we say that a suffixzzis accepted underh2h\_\{2\}ifpθ​\(zj\|h2,sg,z<j\)\>ϵp\_\{\\theta\}\(z\_\{j\}\|h\_\{2\},s\_\{g\},z\_\{<j\}\)\>\\epsilon\. The compression score is the fraction of statesssin which all suffixes generated from one incoming history with probability\>ϵ\>\\epsilonare also accepted under a second incoming history\. A high compression score suggests that the LLM should understand, given the fixed current state, the set of feasible suffixes does not depend on the path by which the current state was reached\.

In our experiments, we use 1000 sampled state\-goal\-history pairs and 30 suffixes per pair\. The prefix length is sampled from the capped geometric segment distribution\. We setϵ=0\.01\\epsilon=0\.01, but the results are not sensitive to this threshold choice\.

The compression scores of all the three LLMs in Table[1](https://arxiv.org/html/2608.23706#S6.T1)are low, whereas the baseline score is close to 100%\. For the RW models, in more than half the states the set of valid suffixes is deemed history\-dependent, even though in the true LOB dynamics the set of valid suffixes is identical for two histories ending in the same states0s\_\{0\}\.

We are thus left with a mixed picture\. The LLMs perform remarkably well in generating valid sequences, but they have failed to understand a key feature of the LOB dynamics\.222We have also applied the probe test of\[[5](https://arxiv.org/html/2608.23706#bib.bib5)\]and the detour and distinction tests of\[[12](https://arxiv.org/html/2608.23706#bib.bib11)\]\. The RW models perform well on the probe test, which checks if a model’s hidden state encodes information about the true state\. They also perform well on detour and distinction tests, while the SP model does not\. These tests are less directly relevant to our focus on forecast distributions so we do not discuss them further\.We turn next to the implications of this misunderstanding\.

## 7Evaluating LLM Forecast Distributions

We will show that deficiencies in LLMs’ implicit world view of LOB dynamics translate into biased forecasts and spurious predictability\. This analysis requires refining the valid traversal and compression tests to account for the stochastic nature of the LOB dynamics\.

Under greedy generation, an LLM will achieve a perfect score on the valid traversal test so long as the event that maximizespθ\(⋅\|a<t\)p\_\{\\theta\}\(\\cdot\|a\_\{<t\}\)is always a valid event in the state determined bya<ta\_\{<t\}; beyond that requirement, the probabilitiespθ​\(a\|a<t\)p\_\{\\theta\}\(a\|a\_\{<t\}\)do not affect the test\. Similarly, the compression test checks that the suffix distributions that follow historiesh1h\_\{1\}andh2h\_\{2\}have common support, but the threshold is a hard cap, and it ignores the probabilities assigned to valid suffixes\. In the setting of a deterministic model, these probabilities are not necessarily relevant; in our stochastic setting, they form an important part of the LOB’s dynamics\. We therefore develop stochastic counterparts to the validity and compression tests\.

### 7\.1Kernel\-Level TV Distance

Under our LOB dynamics, an event sequencea1:Ha\_\{1:H\}of lengthHHfollowing statesshas probability

PH∗\(a1:H∣s\)=∏t=1Hμ\(at∣st−1\),P\_\{H\}^\{\*\}\(a\_\{1:H\}\\mid s\)=\\prod\_\{t=1\}^\{H\}\\mu\(a\_\{t\}\\mid s\_\{t\-1\}\),wheres0=ss\_\{0\}=sandst=T⁡\(st−1,at\)s\_\{t\}=T\(s\_\{t\-1\},a\_\{t\}\)\. We want to compare this probability with the LLM’s implied probability\. As before,pθ​\(a\|h,sg\)p\_\{\\theta\}\(a\|h,s\_\{g\}\)denotes the LLM’s probability of eventaagiven historyhhand goal statesgs\_\{g\}\. Then the LLM’s implied probability of the token sequencea1:Ha\_\{1:H\}given\(h,sg\)\(h,s\_\{g\}\)is

Qθ,H\(a1:H\|h,sg\)=pθ\(a1\|h,sg\)⋯pθ\(aH\|h,sg,a1:H−1\)Q\_\{\\theta,H\}\(a\_\{1:H\}\|h,s\_\{g\}\)=p\_\{\\theta\}\(a\_\{1\}\|h,s\_\{g\}\)\\cdots p\_\{\\theta\}\(a\_\{H\}\|h,s\_\{g\},a\_\{1:H\-1\}\)To remove the condition on the goal state while maintaining consistency with the training\-data generation process, we simulateMgM\_\{g\}independent LOB paths from statess, producingMgM\_\{g\}goal statessgjs\_\{g\_\{j\}\}\. The resulting LLM probability, conditional on historyhh, is then estimated by the average over goal states

Qθ,H\(⋅∣h\)=1Mg∑j=1MgQθ,H\(⋅∣h,sg,j\)\.Q\_\{\\theta,H\}\(\\cdot\\mid h\)=\\frac\{1\}\{M\_\{g\}\}\\sum\_\{j=1\}^\{M\_\{g\}\}Q\_\{\\theta,H\}\(\\cdot\\mid h,s\_\{g,j\}\)\.\(2\)The conditional kernel\-level total variation compares the model’s goal\-averagedHH\-step path distribution with the trueHH\-step kernel:

TVHkernel\(h,s\)=12∑a1:H\|Qθ,H\(a1:H∣h\)−PH∗\(a1:H∣s\)\|\.\\mathrm\{TV\}^\{\\mathrm\{kernel\}\}\_\{H\}\(h,s\)=\\frac\{1\}\{2\}\\sum\_\{a\_\{1:H\}\}\\left\|Q\_\{\\theta,H\}\(a\_\{1:H\}\\mid h\)\-P\_\{H\}^\{\*\}\(a\_\{1:H\}\\mid s\)\\right\|\.\(3\)Here, we retainpθ​\(a\|h,sg\)p\_\{\\theta\}\(a\|h,s\_\{g\}\)assigned to invalid tokensa∉𝒜⁡\(s\)a\\notin\\mathcal\{A\}\(s\), which have zero probability underPH∗\(⋅∣s\)P\_\{H\}^\{\*\}\(\\cdot\\mid s\)and therefore contribute directly to the TV distance\. We average \([3](https://arxiv.org/html/2608.23706#S7.E3)\) over draws ofssfrom the LOB’s stationary distributionπ\\piand from historieshhof geometric length ending in statess, using the backward\-sampling procedure described in Section[6\.2](https://arxiv.org/html/2608.23706#S6.SS2)\.

The resulting quantity measures how well the LLM recovers the true event\-token transition probabilities at horizonHH\. The value lies in\[0,1\]\[0,1\], with smaller values indicating stronger agreement\. Whereas the compression test basically checks if the two distributions have the same support,TVHkernel\\mathrm\{TV\}^\{\\mathrm\{kernel\}\}\_\{H\}checks if they are close\.

If the two distributions are not close, then there will be states in which the LLM predicts some events \(e\.g\., a buy order at a particular price\) has high probabilities while in fact they should not\. We will illustrate this point with an example\.

### 7\.2Example 1: Kernel\-Level TV

With the notation of Figure[1](https://arxiv.org/html/2608.23706#S2.F1), suppose the current state iss=\(ℓa,hb,v1,v2,v3\)=\(3,4,2,2,2\)s=\(\\ell\_\{a\},h\_\{b\},v\_\{1\},v\_\{2\},v\_\{3\}\)=\(3,4,2,2,2\)\. There are two limit sell orders at each pricep1p\_\{1\},p2p\_\{2\}, andp3p\_\{3\}, and there are no limit buy orders\. Suppose this state was reached from state \(2,4,4,1,0\) through the sequence of events and states in Table[2](https://arxiv.org/html/2608.23706#S7.T2)\.

Table 2:Prefix used in kernel\-level TVStepEventResulting State0Start\(2,4,4,1,0\)\(2,4,4,1,0\)1ask\_p3\_2\(3,4,4,1,2\)\(3,4,4,1,2\)2bid\_p2\_1\(3,4,4,1,1\)\(3,4,4,1,1\)3ask\_p2\_1\(3,4,4,2,1\)\(3,4,4,2,1\)4w\_ask\_p1\_2\(3,4,2,2,1\)\(3,4,2,2,1\)5ask\_p3\_1\(3,4,2,2,2\)\(3,4,2,2,2\)In the current statessthere are 20 valid events\. The true one\-step event kernel assigns probability1/20=0\.051/20=0\.05to each of them\. The model’s implied distribution is calculated using \([2](https://arxiv.org/html/2608.23706#S7.E2)\), and the resulting one\-step kernel\-level TV is 0\.665 in this example\.

To illustrate why the value is so large, Table[3](https://arxiv.org/html/2608.23706#S7.T3)shows the five most likely events according to the model\. The LLM assigns nearly a 50% chance\(0\.245\+0\.241\)\(0\.245\+0\.241\)to the next event being a cancellation at pricep1p\_\{1\}, although the actual probability is only 10%\. It also overestimates the probability of a limit sell order atp1p\_\{1\}orp2p\_\{2\}\.

In this example, the LLM’s forecasts are severely biased\. The errors illustrated in Table[3](https://arxiv.org/html/2608.23706#S7.T3)would not prevent the LLM from attaining a perfect score on the valid traversal test, since that test is simply too coarse to quantify errors in forecasting distributions, which is why we introduced the kernel\-level TV measure\.

Table 3:Next\-event probabilities in kernel\-level TVAction TokenNext StateTrue Prob\.LLM’s Prob\.w\_ask\_p1\_2\(3,4,0,2,2\)\(3,4,0,2,2\)0\.0500\.245w\_ask\_p1\_1\(3,4,1,2,2\)\(3,4,1,2,2\)0\.0500\.241ask\_p2\_2\(3,4,2,4,2\)\(3,4,2,4,2\)0\.0500\.190ask\_p1\_1\(3,4,3,2,2\)\(3,4,3,2,2\)0\.0500\.132ask\_p1\_2\(3,4,4,2,2\)\(3,4,4,2,2\)0\.0500\.106
### 7\.3History\-Level TV Distance

The kernel\-level TV is essentially a measure of bias: it measures systematic errors in the LLM’s forecast distribution when compared with the true dynamics\. On the other hand, we also introduce the history\-level TV, which measures whether the LLM gives different predictions for different histories ending at the same current state \(see Figure[2](https://arxiv.org/html/2608.23706#S7.F2)\)\. For each current statess, suppose we observe historiesh1,…,hmh\_\{1\},\\ldots,h\_\{m\}, all ending atss\. Define the within\-state mean model distributionQ¯θ,Hs=1m∑i=1mQθ,H\(⋅∣hi\)\\bar\{Q\}\_\{\\theta,H\}^\{s\}=\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}Q\_\{\\theta,H\}\(\\cdot\\mid h\_\{i\}\), then the history\-level TV for one historyhih\_\{i\}and current statessis

TVHhistory\(hi,s\)=12∑a1:H\|Qθ,H\(a1:H∣hi\)−Q¯θ,Hs\(a1:H\)\|\.\\mathrm\{TV\}^\{\\mathrm\{history\}\}\_\{H\}\(h\_\{i\},s\)=\\frac\{1\}\{2\}\\sum\_\{a\_\{1:H\}\}\\left\|Q\_\{\\theta,H\}\(a\_\{1:H\}\\mid h\_\{i\}\)\-\\bar\{Q\}\_\{\\theta,H\}^\{s\}\(a\_\{1:H\}\)\\right\|\.\(4\)As withTVHkernel\\mathrm\{TV\}^\{\\mathrm\{kernel\}\}\_\{H\}, we average this measure over draws ofssfromπ\\piand historieshih\_\{i\}ending inss\. This measure quantifies dispersion: the larger the value, the more the LLM’s forecast is erroneously influenced by past history, which violates the Markov property\.

![Refer to caption](https://arxiv.org/html/2608.23706v1/TV_figure.png)Figure 2:Illustration of history\- and kernel\-level TVIn the experiments, the kernel\-level TV is computed from 1000 sampled state\-prefix pairs\. For the history\-level TV, it is computed from 200 sampled current state, andm=10m=10prefixes for each state\. The LLM’s implied probabilityQθ,H\(a1:H\|h,sg\)Q\_\{\\theta,H\}\(a\_\{1:H\}\|h,s\_\{g\}\)for each state\-prefix pair is averaged overMg=32M\_\{g\}=32sampled future goals\.

Table[4](https://arxiv.org/html/2608.23706#S7.T4)shows the results for our two TV measures across all models and three forecast horizons\. The results are more relevant for the RW models\. AtH=1H=1, the TV measures are rather small, but they clearly increase at longer forecasting horizons as error accumulates, which is a pattern we expect to see continue for largerHH\. Moreover, the variances of both TV measures are very small, indicating that the LLMs consistently exhibit forecast errors and excessive dependence on past events across all states\.

Table 4:Multi\-step forecast TV distancesModelTV TypeH=1H=1H=2H=2H=3H=3SPKernel\-level0\.460\.720\.84History\-level0\.330\.510\.59RW\-SKernel\-level0\.110\.190\.28History\-level0\.070\.140\.23RW\-LKernel\-level0\.080\.150\.21History\-level0\.060\.110\.16BaselineKernel\-level0\.000\.010\.01History\-level0\.000\.000\.01
### 7\.4Example 2: History\-Level TV

To illustrate the history\-level TV measure, we consider two different histories that reach the same current state\. The shared current state iss=\(ℓa,hb,v1,v2,v3\)=\(1,3,5,0,2\)s=\(\\ell\_\{a\},h\_\{b\},v\_\{1\},v\_\{2\},v\_\{3\}\)=\(1,3,5,0,2\)\. The two event sequences leading to this state are shown in Appendix[A\.2](https://arxiv.org/html/2608.23706#A1.SS2), Table[7](https://arxiv.org/html/2608.23706#A1.T7)\.

We calculateQθ,H\(⋅\|hi\)Q\_\{\\theta,H\}\(\\cdot\|h\_\{i\}\),i=1,2i=1,2, as in \([2](https://arxiv.org/html/2608.23706#S7.E2)\) for the two histories, using the same sampled goal statessg,js\_\{g,j\}; this ensures the differences inQθ,H\(⋅\|hi\)Q\_\{\\theta,H\}\(\\cdot\|h\_\{i\}\)are due only to differences in the historieshih\_\{i\}, which should be irrelevant given the Markov property\. The history\-level TV distance \([4](https://arxiv.org/html/2608.23706#S7.E4)\) using these histories is 0\.385 in this example\.

Table[5](https://arxiv.org/html/2608.23706#S7.T5)provides more detail by showing the model\-based probabilities of certain events following the two histories\. If the LLM’s implicit world view of the LOB recognized the current state, the last two columns would be identical\. Instead, we see that the LLM assigns a rather large probability \(0\.278\) to a limit buy order at pricep2p\_\{2\}followingh1h\_\{1\}and a rather large probability \(0\.313\) of a limit sell order at pricep3p\_\{3\}followingh2h\_\{2\}\.

These are examples of spurious predictability: the LLM operates as if the history leading to the current state helps predict the next event, even though there should be no such predictability in our Markov\-generated training data\.

Table 5:Next\-event probabilities in history\-level TVEvent TokenNext StateQθ,1​\(a∣h1\)Q\_\{\\theta,1\}\(a\\mid h\_\{1\}\)Qθ,1​\(a∣h2\)Q\_\{\\theta,1\}\(a\\mid h\_\{2\}\)bid\_p2\_2\(1,2,5,2,2\)\(1,2,5,2,2\)0\.2780\.002ask\_p3\_2\(1,4,5,0,0\)\(1,4,5,0,0\)0\.0560\.313m\_ask\_2\(1,4,5,0,0\)\(1,4,5,0,0\)0\.0470\.233w\_bid\_p3\_2\(1,4,5,0,0\)\(1,4,5,0,0\)0\.0280\.199ask\_p2\_2\(2,3,5,2,2\)\(2,3,5,2,2\)0\.0530\.208ask\_p2\_1\(2,3,5,1,2\)\(2,3,5,1,2\)0\.1280\.002
### 7\.5Regression Test

A low compression score and a high history\-level TV score indicate that the LLM has failed to understand the state of the LOB\. A direct way to see the implications of this failure is to run a forecasting regression on features of the event history, which should again have no predictive power under the Markov property\.

For our forecasting analysis, we group our target future events into four categories:

- •Buy orders: limit buy and market buy orders\.
- •Cancel orders: all cancellation orders\.
- •Market orders: all market orders\.
- •Big orders: orders with size greater thanV2/2V\_\{2\}/2\.

These categories correspond to economically meaningful order\-flow quantities\. Buy orders capture directional buy pressure\. Cancel orders capture liquidity withdrawal from the book\. Market orders capture aggressive liquidity\-taking behavior\. Big orders capture large\-flow intensity\. By studying the model’s forecasts for these quantities, we can identify which forms of financial predictability the LLM has incorrectly learned\.

For each sampled visit, we compute the model’s implied category ratios overHH\-step event paths\. For a categorycc, define

rc\(a1:H\)=1H∑t=1H𝟏\{at∈c\}\.r\_\{c\}\(a\_\{1:H\}\)=\\frac\{1\}\{H\}\\sum\_\{t=1\}^\{H\}\\mathbf\{1\}\\\{a\_\{t\}\\in c\\\}\.The true expected category ratio is𝔼PH∗​\[rc\]\\mathbb\{E\}\_\{P\_\{H\}^\{\*\}\}\[r\_\{c\}\], and the model\-implied expected category ratio is𝔼Qθ,H​\[rc\]\\mathbb\{E\}\_\{Q\_\{\\theta,H\}\}\[r\_\{c\}\], withPH∗P\_\{H\}^\{\*\}andQθ,HQ\_\{\\theta,H\}as defined in Section[7\.1](https://arxiv.org/html/2608.23706#S7.SS1)\. We use the difference between these two quantities as the regression target:

Bc,H​\(h,s\)=𝔼Qθ,H​\[rc\]−𝔼PH∗​\[rc\]\.B\_\{c,H\}\(h,s\)=\\mathbb\{E\}\_\{Q\_\{\\theta,H\}\}\[r\_\{c\}\]\-\\mathbb\{E\}\_\{P\_\{H\}^\{\*\}\}\[r\_\{c\}\]\.It is important to note that the raw model’s implied ratio𝔼Qθ,H​\[rc\]\\mathbb\{E\}\_\{Q\_\{\\theta,H\}\}\[r\_\{c\}\]would not be the right object for detecting spurious predictability or miscalibration, because the true ratio𝔼PH∗​\[rc\]\\mathbb\{E\}\_\{P\_\{H\}^\{\*\}\}\[r\_\{c\}\]should itself depend on the current statess\. Since different LOB states have different feasible actions and liquidity conditions, the true probabilities of buy orders, cancellations, market orders, and big orders may not be constant across states\. For example, a state with little ask\-side liquidity or a wide spread can have a different true distribution over future order types than a balanced, liquid state\.

The ratio biasBc,H​\(h,s\)B\_\{c,H\}\(h,s\)removes this true state\-dependent component\. It measures whether the LLM overpredicts or underpredicts a future order\-flow quantity relative to the correct Markov benchmark of the true LOB dynamics\.

We clarify the interpretation of the regression coefficients\. Under the Markov LOB dynamics, the historyhhshould have no incremental predictive content once the current statessis given\. Significant prefix\-history coefficients therefore reveal spurious predictability: the model assigns forecasting value to historical order flow that should be irrelevant\. On the other hand, if current\-state features predictBc,H​\(h,s\)B\_\{c,H\}\(h,s\), this does not violate the Markov property directly, since both𝔼Qθ,H​\[rc\]\\mathbb\{E\}\_\{Q\_\{\\theta,H\}\}\[r\_\{c\}\]and𝔼PH∗​\[rc\]\\mathbb\{E\}\_\{P\_\{H\}^\{\*\}\}\[r\_\{c\}\]are allowed to depend on the current LOB state\. However, because the target has already subtracted the true Markov benchmark, such coefficients indicate a state\-dependent model miscalibration: the LLM is systematically too optimistic or pessimistic about future order flow in certain types of book configurations\.

Based on that, for each categoryccand horizonH∈\{1,2,3\}H\\in\\\{1,2,3\\\}, we run an OLS regression,

Bc,H​\(h,s\)=α\+βs⊤​Fs​\(s\)\+βh⊤​Fh​\(h\)\+ε,B\_\{c,H\}\(h,s\)=\\alpha\+\\beta\_\{s\}^\{\\top\}F\_\{s\}\(s\)\+\\beta\_\{h\}^\{\\top\}F\_\{h\}\(h\)\+\\varepsilon,whereFs​\(s\)F\_\{s\}\(s\)contains current\-state features andFh​\(h\)F\_\{h\}\(h\)contains prefix\-history features\. This gives 12 regression equations per model\.

Here, we useDDto represent the depth of the LOB\. Recall the state definitions=\(ℓa,hb,v1,…,vP\)s=\(\\ell^\{a\},h^\{b\},v\_\{1\},\\dots,v\_\{P\}\)in Section[2](https://arxiv.org/html/2608.23706#S2), we have:

Dask​\(s\)=∑i=1ℓa​\(s\)vi​\(s\),Dbestask​\(s\)=vℓa​\(s\)\.D^\{\\mathrm\{ask\}\}\(s\)=\\sum\_\{i=1\}^\{\\ell^\{a\}\(s\)\}v\_\{i\}\(s\),\\quad D^\{\\mathrm\{bestask\}\}\(s\)=v\_\{\\ell^\{a\}\}\(s\)\.With similarly for bid depth, we also defineDtotal​\(s\)=Dask​\(s\)\+Dbid​\(s\)\.D^\{\\mathrm\{total\}\}\(s\)=D^\{\\mathrm\{ask\}\}\(s\)\+D^\{\\mathrm\{bid\}\}\(s\)\.The current\-state feature vectorFs​\(s\)F\_\{s\}\(s\)includes:

- •Current depth imbalance: Dbid​\(s\)−Dask​\(s\)Dbid​\(s\)\+Dask​\(s\)\.\\frac\{D^\{\\mathrm\{bid\}\}\(s\)\-D^\{\\mathrm\{ask\}\}\(s\)\}\{D^\{\\mathrm\{bid\}\}\(s\)\+D^\{\\mathrm\{ask\}\}\(s\)\}\.This measures whether displayed liquidity is concentrated on the bid side or the ask side, which is a standard proxy for pressure at the book level\.
- •Current spread width:ℓa​\(s\)−hb​\(s\)\\ell^\{a\}\(s\)\-h^\{b\}\(s\)\. A wider spread corresponds to lower immediacy and liquidity\.
- •Current total depth:Dtotal​\(s\)D^\{\\mathrm\{total\}\}\(s\)\. This is the total displayed liquidity on the bid and ask sides\. A deeper book is typically more resilient to incoming orders\.
- •Current best\-depth imbalance: Dbestbid​\(s\)−Dbestask​\(s\)Dbestbid​\(s\)\+Dbestask​\(s\)\.\\frac\{D^\{\\mathrm\{bestbid\}\}\(s\)\-D^\{\\mathrm\{bestask\}\}\(s\)\}\{D^\{\\mathrm\{bestbid\}\}\(s\)\+D^\{\\mathrm\{bestask\}\}\(s\)\}\.This measures the imbalance at the top of the book, focusing on the most immediately executable liquidity\.

The prefix\-history feature vectorFh​\(h\)F\_\{h\}\(h\)includes:

- •Prefix length: This is the number of past events in the segment\. Under the Markov setting, segment age should not matter once the current state is known\.
- •Prefix buy ratio: This is the fraction of past events that are buy orders, which measures buy\-side order\-flow intensity\.
- •Prefix take ratio: This is the fraction of past events that are market orders or crossing\-limit orders, which \(partially\) execute immediately and consume liquidity from the opposite side of the LOB\. It measures the order\-flow aggressiveness\.
- •Prefix cancel ratio: This is the fraction of past events that are cancellations\. It measures liquidity withdrawal and reflects active quote management\.
- •Prefix rest\-order depletion: ∑t=1\|h\|𝟏\{Dtotal\(st\)<Dtotal\(st−1\)\},\\sum\_\{t=1\}^\{\|h\|\}\\mathbf\{1\}\\left\\\{D^\{\\mathrm\{total\}\}\(s\_\{t\}\)<D^\{\\mathrm\{total\}\}\(s\_\{t\-1\}\)\\right\\\},wherest−1s\_\{t\-1\}is the state before eventata\_\{t\}, andsts\_\{t\}is the state after eventata\_\{t\}\. This counts how often events in the prefix reduce the total depth of the LOB, which captures liquidity removal through active trades or cancellations\.
- •Prefix event entropy: This measures the diversity of event types in the prefix\. Low entropy corresponds to a concentrated, repetitive order\-flow history, while high entropy corresponds to more varied flow\.

Figure[3](https://arxiv.org/html/2608.23706#S7.F3)shows results for the RW\-S model\. Each column corresponds to a separate regression for a given event category and forecasting horizon\. The first four rows are state features, and the remaining six rows are prefix\-history features\. Red cells indicate positive coefficients, while blue cells indicate negative coefficients\. A star marks coefficients that are statistically significant at the 5% level333We use the 5% significance level as a standard threshold and to provide a clear visual illustration\. The results should be interpreted with the usual caution regarding multiple testing\. Even under more conservative thresholds, like the Bonferroni correction, many features remain statistically significant\., and darker cells in either color indicate smallerpp\-values\.

From the upper\-left corner, we see that the LLM systematically overestimates the probability of a buy order in states with larger depth imbalance, and market orders in states with larger spreads, while it underestimates market orders in states with greater depth forH=2,3H=2,3\. These suggest that the model has state\-dependent miscalibrations, that its forecast bias are wrongly related to current liquidity conditions\. More importantly, in eight out of the twelve regressions, one or more prefix\-history features appear to be statistically significant\. For example, a higher historical buy ratio predicts a lower future buy probability, resembling a reversal signal in order flow; while greater rest\-order depletion shifts the model’s forecasts away from buy orders and toward cancellations, resembling a liquidity\-withdrawal signal\. Yet under the Markov property, these history features should be irrelevant to future events conditional on the current state\. In other words, the predictability found by the LLM is spurious\.

The results for RW\-L in Figure[4](https://arxiv.org/html/2608.23706#S7.F4)are broadly similar\.

![Refer to caption](https://arxiv.org/html/2608.23706v1/rw_s_spurious_predictability_heatmap.png)Figure 3:Regression results for RW\-S![Refer to caption](https://arxiv.org/html/2608.23706v1/rw_l_spurious_predictability_heatmap.png)Figure 4:Regression results for RW\-L

## 8Scaling to Larger LOB Settings

As introduced in Section[4](https://arxiv.org/html/2608.23706#S4), the base setting uses\(P,V1,V2\)=\(3,5,2\)\(P,V\_\{1\},V\_\{2\}\)=\(3,5,2\), which gives a finite LOB space with 756 states\. To check whether the main findings are specific to small environments, we repeat the same pipeline on two larger settings:\(P,V1,V2\)=\(3,12,3\)\(P,V\_\{1\},V\_\{2\}\)=\(3,12,3\)and\(P,V1,V2\)=\(8,2,2\)\(P,V\_\{1\},V\_\{2\}\)=\(8,2,2\)\. These settings substantially expand the number of reachable LOB states, to 8281 and 41553 states respectively\.

Table[6](https://arxiv.org/html/2608.23706#S8.T6)shows that the qualitative conclusions are stable across these larger environments\. The LLMs maintain high valid traversal rates and also tend to generate short sequence\. However, the compression scores decline as the LOB environment becomes larger, especially for the RW models\. The multi\-step TV results point in the same direction, as kernel\-level and history\-level TV distances both increase in the larger settings\. However, the baseline performs stably well\. This suggests that the LLM’s ability to represent the correct Markov state abstraction weakens as the LOB becomes more complex\. Thus, the scaling experiments provide direct support for our claim that the issues we document would likely be more severe in larger settings, especially when LLMs are applied to real\-market LOB data with larger price levels and size limits, and richer order\-flow patterns\.

Table 6:Diagnostic and multi\-step forecast results for larger LOB settingsLOBSetting\# ofStatesModelValidTraversalSPRateCompress\.ScoreKernel\-level TVHistory\-level TVH=1H=1H=2H=2H=3H=3H=1H=1H=2H=2H=3H=3\(P,V1,V2\)\(P,V\_\{1\},V\_\{2\}\)=\(3,12,3\)=\(3,12,3\)8281SP99\.4%99\.4% \(100%\)47\.0%0\.640\.860\.930\.340\.450\.57RW\-S97\.8%88\.6% \(11\.5%\)29\.0%0\.140\.220\.350\.100\.130\.26RW\-L94\.1%81\.2% \(4\.0%\)34\.0%0\.130\.190\.320\.080\.140\.18Baseline0\.3%0\.3% \(4\.0%\)99\.8%0\.000\.020\.010\.010\.000\.01\(P,V1,V2\)\(P,V\_\{1\},V\_\{2\}\)=\(8,2,2\)=\(8,2,2\)41553SP97\.1%96\.8% \(100%\)53\.6%0\.610\.790\.920\.390\.560\.62RW\-S98\.6%90\.2% \(10\.1%\)23\.0%0\.170\.240\.380\.120\.160\.27RW\-L99\.3%83\.3% \(6\.0%\)24\.0%0\.160\.200\.310\.110\.180\.23Baseline0\.4%0\.4% \(6\.0%\)99\.9%0\.000\.010\.020\.000\.010\.01
## 9Conclusion

Our investigation contributes to the study of world models implicit in LLMs, with a specific focus on understanding the operation of a financial market — a limit order book\. We have shown that an LLM trained on LOB event sequences may perform well in generating valid sequences yet fail to understand the dynamics of the LOB\. This deficiency leads to biased forecasts and spurious predictability in LOB events\. We introduce new tests to measure these effects\.

To train an LLM from scratch, we keep our LOB settings small\. A larger LOB would likely require vastly more training data to perform well on our tests, consistent with our scaling\-up experiments\. By using synthetic data, we are able to evaluate LLM forecasts in a setting without predictability\. Events in an actual LOB may exhibit some predictability, but an LLM’s ability to provide reliable forecasts is suspect if it finds predictability where none exists\.

## References

- \[1\]S\. A\. Assefa, D\. Dervovic, M\. Mahfouz, R\. E\. Tillman, P\. Reddy, and M\. Veloso\(2020\)Generating synthetic data in finance: opportunities, challenges and pitfalls\.InProceedings of the first ACM international conference on AI in finance,pp\. 1–8\.Cited by:[§1](https://arxiv.org/html/2608.23706#S1.p8.1)\.
- \[2\]Á\. Cartea, S\. Jaimungal, and J\. Penalva\(2015\)Algorithmic and high\-frequency trading\.Cambridge University Press\.Cited by:[§2\.1](https://arxiv.org/html/2608.23706#S2.SS1.p3.1)\.
- \[3\]L\. Guan, K\. Valmeekam, S\. Sreedharan, and S\. Kambhampati\(2023\)Leveraging pre\-trained large language models to construct and utilize world models for model\-based task planning\.Advances in Neural Information Processing Systems36,pp\. 79081–79094\.Cited by:[§1](https://arxiv.org/html/2608.23706#S1.p6.1)\.
- \[4\]D\. Hazineh, Z\. Zhang, and J\. Chiu\(2023\)Linear latent world models in simple transformers: a case study on othello\-gpt\.InSocially Responsible Language Modelling Research,Cited by:[§1](https://arxiv.org/html/2608.23706#S1.p6.1)\.
- \[5\]J\. Hewitt and P\. Liang\(2019\)Designing and interpreting probes with control tasks\.InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing \(emnlp\-ijcnlp\),pp\. 2733–2743\.Cited by:[footnote 2](https://arxiv.org/html/2608.23706#footnote2)\.
- \[6\]Y\. Kong, Y\. Nie, X\. Dong, J\. M\. Mulvey, H\. V\. Poor, Q\. Wen, and S\. Zohren\(2024\)Large language models for financial and investment management: applications and benchmarks\.Journal of Portfolio Management51\(2\),pp\. 162–210\.Cited by:[§1](https://arxiv.org/html/2608.23706#S1.p8.1)\.
- \[7\]K\. Li, A\. K\. Hopkins, D\. Bau, F\. Viégas, H\. Pfister, and M\. Wattenberg\(2023\)Emergent world representations: exploring a sequence model trained on a synthetic task\.ICLR\.Cited by:[§1](https://arxiv.org/html/2608.23706#S1.p6.1)\.
- \[8\]B\. Liu, J\. T\. Ash, S\. Goel, A\. Krishnamurthy, and C\. Zhang\(2023\)Transformers learn shortcuts to automata\.InThe Eleventh International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.23706#S1.p6.1)\.
- \[9\]P\. Nagy, S\. Frey, S\. Sapora, K\. Li, A\. Calinescu, S\. Zohren, and J\. N\. Foerster\(2023\)Generative ai for end\-to\-end limit order book modelling: a token\-level autoregressive generative model of message flow using a deep state space network\.InProceedings of the 4th ACM International Conference on AI in Finance \(ICAIF 2023\),pp\. 91–99\.External Links:[Link](https://doi.org/10.1145/3604237.3626898)Cited by:[§1](https://arxiv.org/html/2608.23706#S1.p8.1)\.
- \[10\]S\. Toshniwal, S\. Wiseman, K\. Livescu, and K\. Gimpel\(2022\)Chess as a testbed for language model state tracking\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.36,pp\. 11385–11393\.Cited by:[§1](https://arxiv.org/html/2608.23706#S1.p6.1)\.
- \[11\]K\. Vafa, P\. G\. Chang, A\. Rambachan, and S\. Mullainathan\(2025\)What has a foundation model found? using inductive bias to probe for world models\.InInternational Conference on Machine Learning,pp\. 60727–60747\.Cited by:[§1](https://arxiv.org/html/2608.23706#S1.p6.1)\.
- \[12\]K\. Vafa, J\. Y\. Chen, A\. Rambachan, J\. Kleinberg, and S\. Mullainathan\(2024\)Evaluating the world model implicit in a generative model\.Advances in Neural Information Processing Systems37,pp\. 26941–26975\.Cited by:[§1](https://arxiv.org/html/2608.23706#S1.p6.1),[§2\.2](https://arxiv.org/html/2608.23706#S2.SS2.p3.1),[§3](https://arxiv.org/html/2608.23706#S3.p3.1),[§6\.2](https://arxiv.org/html/2608.23706#S6.SS2.p1.1),[footnote 2](https://arxiv.org/html/2608.23706#footnote2)\.

## Appendix ASupporting Information

### A\.1Shortest Paths Datasets Construction

To generate a shortest\-path training sequence \([1](https://arxiv.org/html/2608.23706#S3.E1)\), we first draw the goal statesgs\_\{g\}from the stationary distributionπ\\pi, and we then run a "reverse breadth\-first search \(BFS\)" on the LOB state transition graph\. Firstly, we calculate the distancedsg​\(s\)d\_\{s\_\{g\}\}\(s\)from every statesstosgs\_\{g\}using normal BFS\. Then, starting fromdsg​\(sg\)=0d\_\{s\_\{g\}\}\(\{s\_\{g\}\}\)=0, we work backwards from the endpoint using the fact that if there exists a valid eventaasuch thatT⁡\(s,a\)=s′T\(s,a\)=s^\{\\prime\}, thendsg​\(s′\)=dsg​\(s\)−1d\_\{s\_\{g\}\}\(s^\{\\prime\}\)=d\_\{s\_\{g\}\}\(s\)\-1\. We repeatedly choose an incoming event\-labeled predecessor whose distance is exactly one larger than the current distance\. After each reverse step, we stop with probabilitypstopp\_\{\\mathrm\{stop\}\}\. Reversing the sampled event sequence gives a forward shortest path fromsstarts\_\{\\mathrm\{start\}\}tosg\{s\_\{g\}\}\. When multiple event\-labeled edges satisfy the distance condition, uniform sampling is done at the event level\.

### A\.2Two Prefixes in History\-Level TV Example

We show the two histories used in Section[7\.4](https://arxiv.org/html/2608.23706#S7.SS4)in Table[7](https://arxiv.org/html/2608.23706#A1.T7)\.

Table 7:Two prefixes in history\-level TVPrefixStart StateEvent Sequenceh1h\_\{1\}\(2,3,2,2,1\)\(2,3,2,2,1\)ask\_p1\_2,w\_bid\_p3\_1,ask\_p2\_2,ask\_p1\_1,bid\_p3\_2,bid\_p2\_1,m\_ask\_1,bid\_p3\_1,w\_bid\_p3\_2,ask\_p2\_1,ask\_p3\_2,bid\_p1\_1,bid\_p3\_2,m\_ask\_1,m\_bid\_2,m\_bid\_1,m\_bid\_2,bid\_p1\_1,ask\_p1\_2,bid\_p3\_2h2h\_\{2\}\(1,2,3,1,2\)\(1,2,3,1,2\)bid\_p3\_2,bid\_p2\_1,bid\_p2\_2,m\_bid\_2,ask\_p1\_1,w\_bid\_p3\_1,w\_bid\_p3\_1,ask\_p1\_2,w\_bid\_p3\_2,ask\_p3\_1,m\_ask\_2,bid\_p3\_1,ask\_p3\_1,w\_ask\_p1\_1,ask\_p2\_2,ask\_p1\_1,m\_ask\_1,ask\_p1\_1,m\_bid\_2,bid\_p3\_2