A Foundation Model for Multimodal Event Sequences in Financial Applications

# A Foundation Model for Multimodal Event Sequences in Financial Applications
Source: [https://arxiv.org/html/2607.09955](https://arxiv.org/html/2607.09955)
\(2026\)

###### Abstract\.

Predictive modeling is a core component of modern financial services, where a wide range of tasks are traditionally addressed using separate models trained on manually engineered tabular features\. This task\-specific approach limits reuse and makes it difficult to fully exploit heterogeneous data sources such as transaction histories and digital interaction signals\. In this paper, we present an approach based on pretraining a foundation transformer model on multimodal sequences of user events\. Events from multiple data sources are unified into a single chronological sequence, enabling early fusion of heterogeneous modalities and learning of general\-purpose representations via a next\-event prediction objective\. These representations are combined with existing engineered user features, on top of which lightweight neural models are trained for multiple downstream tasks\. The proposed system outperforms traditional task\-specific models while reducing development overhead\. The approach was deployed in production at one of the biggest banks in Eastern Europe, resulting in measurable improvements in business metrics\.

foundation models; multimodal event sequences; self\-supervised learning; transactional data; financial applications

††journalyear:2026††copyright:cc††conference:Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\.2; August 09–13, 2026; Jeju Island, Republic of Korea††booktitle:Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\.2 \(KDD ’26\), August 09–13, 2026, Jeju Island, Republic of Korea††doi:10\.1145/3770855\.3818311††isbn:979\-8\-4007\-2259\-2/2026/08††ccs:Information systems Data mining††ccs:Computing methodologies Neural networks††ccs:Computing methodologies Learning latent representations## 1\.Introduction

Machine learning is a core component of modern financial services, supporting a wide range of predictive tasks such as risk assessment, fraud detection, product recommendations, and customer analytics\(Braithwaiteet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib18); Yehet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib17); Skalskiet al\.,[2023](https://arxiv.org/html/2607.09955#bib.bib16); Babaevet al\.,[2022](https://arxiv.org/html/2607.09955#bib.bib15); Jhaet al\.,[2012](https://arxiv.org/html/2607.09955#bib.bib25)\)\. In large financial institutions, these tasks are typically addressed by building separate models for each use case, trained on manually engineered features derived from user transaction histories and digital interaction data\. While this task\-specific approach has demonstrated strong performance in individual applications, it leads to significant duplication of effort, limited reuse of learned representations, and increasing system complexity as the number of predictive tasks grows\.

A key limitation of this paradigm is its inability to fully leverage the scale and richness of available data\(Braithwaiteet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib18); Yehet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib17)\)\. In practice, user behavior is recorded across multiple heterogeneous sources, including financial transactions, online interactions, communications, and other behavioral signals, each characterized by different structures and temporal dynamics\(Mollaevet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib21)\)\. Large financial institutions collect massive volumes of such event\-level data over long time horizons\. Transforming these multimodal event streams into task\-specific tabular features requires substantial manual effort and often leads to information loss\. As a result, only a fraction of the available data is effectively utilized, and insights learned for one task are difficult to transfer or reuse across others\.

To address these challenges, we explore an approach based on pretraining of a transformer\-based foundation model on sequences of user events\(Braithwaiteet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib18); Yehet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib17); Karpukhin and Savchenko,[2025](https://arxiv.org/html/2607.09955#bib.bib22); Ostroukhovet al\.,[2026](https://arxiv.org/html/2607.09955#bib.bib23); Douet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib24)\)\. The model is trained in a self\-supervised manner to learn general\-purpose representations from large\-scale heterogeneous data that can be reused across multiple downstream tasks\. Events from different data sources are unified into a single chronological sequence, enabling early fusion of multimodal signals, and the model is pretrained using a next\-event prediction objective\. The pretrained model serves as a shared representation backbone: its embeddings are frozen and combined with existing engineered user features, on top of which lightweight task\-specific tabular neural models are trained for individual applications\.

We evaluate the proposed approach across multiple real\-world financial prediction tasks at Sber, one of the biggest banks in Eastern Europe, serving more than 100 million customers\. In our experiments, we demonstrate that self\-supervised pretraining and the use of multiple data modalities consistently improve predictive performance compared to traditional task\-specific models\. We further analyze the impact of different fusion strategies and study scaling behavior with respect to model size and sequence length under industrial inference constraints\. The proposed system has been deployed in production, resulting in measurable improvements in business metrics while reducing model development time and overall system complexity\.

## 2\.Related Work

### 2\.1\.Foundation models for user behavior modeling and recommendation systems

Several recent industrial works explore pretrained transformer\-based models for modeling user behavior\. These approaches focus on learning reusable user representations from sequences of user actions, which can be applied across multiple downstream tasks\. Pinnerformer\(Panchaet al\.,[2022](https://arxiv.org/html/2607.09955#bib.bib6)\)introduces a transformer\-based model trained on user actions to produce high\-quality user embeddings that can be used for many recommendation tasks at Pinterest\. TransAct\(Xiaet al\.,[2023](https://arxiv.org/html/2607.09955#bib.bib7)\)proposes a hybrid ranking framework that combines short\-term user interests captured by a transformer\-based real\-time sequential model with long\-term user representations computed offline using PinnerFormer\. Finally, PinFM\(Chenet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib8)\)explicitly adopts the foundation model paradigm and pretraining\-finetuning approach for user activity modeling at Pinterest\. The authors pretrain a sequential model on massive user activity sequence data collected across multiple applications\. Downstream ranking models reuse the pretrained transformer and embedding tables, which are further fine\-tuned with application\-specific data and features\. ARGUS\(Khrylchenkoet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib9)\)studies the scalability of autoregressively pretrained transformer models for recommendations\. The model is pretrained on user histories using next\-item and feedback prediction objectives and then fine\-tuned for various ranking tasks\. The work demonstrates that this approach scales effectively across a wide range of transformer sizes\. DV365\(Lyuet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib10)\)focuses on encoding extremely long user histories into compact user embeddings\. The proposed offline foundational model precomputes user representations from long behavior sequences, which are then shared across a large number of downstream models, improving performance across corresponding tasks\.

Universal behavioral modeling was also the focus of the RecSys Challenge 2025\(Dabrowskiet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib11)\)\. The goal was to learn task\-independent user representations from rich multi\-event user logs such as purchases, cart additions, page visits, and search queries, which are applicable across different downstream tasks, including churn prediction, product propensity, and category propensity prediction\. The work\(Sawadaet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib12)\)proposes a contrastive learning approach with a transformer model pretrained to maximize agreement between representations of past and future segments of the same user’s behavior, while pushing apart representations of different users\. Another solution\(Makeevet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib13)\)is based on pretraining a large transformer model using the next\-event prediction objective, where several attributes of the next event are predicted jointly\. The paper\(Klenitskiyet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib14)\)proposes learning universal representations using a sequential autoencoder that reconstructs the whole user history from the user embedding\. Beyond the RecSys Challenge, several other works also explore pretraining approaches for user behavior modeling\(Wanget al\.,[2023](https://arxiv.org/html/2607.09955#bib.bib29); Gonget al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib30); Liuet al\.,[2022](https://arxiv.org/html/2607.09955#bib.bib31)\)\.

### 2\.2\.Representation learning from transaction sequences for financial applications

Autoregressive pretraining and contrastive learning are among the most widely used approaches for representation learning on transactional data\(Bazarovaet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib32); Sakhnoet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib33); Moskvoretskiiet al\.,[2024](https://arxiv.org/html/2607.09955#bib.bib35)\)\. Early work on learning general\-purpose embeddings from user transaction histories primarily relied on recurrent neural networks\. CoLES\(Babaevet al\.,[2022](https://arxiv.org/html/2607.09955#bib.bib15)\)represents an example of the contrastive learning paradigm for event sequence modeling\. The method learns embeddings for subsequences of user event histories by encouraging representations of subsequences from the same user to be similar, while pushing apart representations of different users\. Embeddings learned from transaction datasets are shown to be effective for downstream financial tasks\. NPPR\(Skalskiet al\.,[2023](https://arxiv.org/html/2607.09955#bib.bib16)\)follows an autoregressive pretraining approach, training an RNN\-based model on large corpora of transaction sequences using a combination of next\-transaction prediction and past reconstruction objectives\. The authors demonstrate the success of the next\-event prediction task for financial applications\.

More recent works apply transformer\-based autoregressive pretraining to large\-scale transactional datasets\(Yehet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib17); Braithwaiteet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib18); Douet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib24)\)\. For example, TREASURE\(Yehet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib17)\)introduces a transformer foundation model specifically designed for transaction data\. The model is pretrained using next\-transaction prediction along with an auxiliary objective capturing current transaction network signals\. In nuFormer\(Braithwaiteet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib18)\), the authors investigate transformer\-based representation learning for transaction data with both textual and structured attributes\. The authors propose a tokenization scheme where each transaction is represented by multiple tokens and pretrain causal transformers using the standard next\-token prediction objective\. An end\-to\-end fine\-tuning approach is introduced to combine learned user embeddings with existing tabular features for downstream financial tasks\. Another line of work explores the use of large language models for transactional data\(Polletiet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib26); Guoet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib27); Ramanet al\.,[2024](https://arxiv.org/html/2607.09955#bib.bib28); Shestovet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib34)\)\.

In summary, prior work has demonstrated the effectiveness of pretrained sequential models for user behavioral modeling and transaction\-based financial applications\. Existing approaches either focus on non\-financial user interaction data or model transaction sequences in isolation\. In contrast, our work addresses the practical setting of large\-scale financial systems, where the data is inherently multimodal, since user behavior is observed across multiple heterogeneous data sources\.

## 3\.Problem Formulation

We consider the problem of learning general\-purpose user representations from large\-scale, heterogeneous behavioral data in a financial system\. User behavior is recorded across multiple data sources, each capturing complementary aspects of user activity\. For a given useruu, each modalitymmfrom a set of available modalitiesMMproduces a temporal sequence of events

Su\(m\)=\[eu​1\(m\),eu​2\(m\),…\]\.S\_\{u\}^\{\(m\)\}=\[e\_\{u1\}^\{\(m\)\},e\_\{u2\}^\{\(m\)\},\\dots\]\.Events from different modalities are heterogeneous in structure and attributes, but share a common timeline\. A key challenge is to integrate these heterogeneous sequences into a unified representation\. Existing approaches typically rely on either late fusion, in which each modality is modeled independently, or early fusion, which merges events from all modalities into a single chronological sequence\. While late fusion limits cross\-modal temporal interactions, early fusion requires a unified event representation that can handle heterogeneous inputs\.

In addition to raw event sequences, production systems rely on a rich set of engineered user features𝐟u∈RK\\mathbf\{f\}\_\{u\}\\in R^\{K\}\. These features encode domain knowledge, have strong predictive value in existing tabular models, and are already deeply integrated into downstream pipelines\. A practical solution should therefore complement, rather than replace them\.

The learned representations should apply to diverse downstream business tasks without task\-specific adaptation\. In practical settings, this task set is not fixed in advance, is typically large, and evolves over time, since new tasks may emerge as business objectives change\. Training and maintaining a separate sequence model for each task is therefore computationally expensive and operationally impractical\. To summarize, our objective is to learn a universal user embedding function

Φcore:\{Su\(1\),…,Su\(M\)\}→𝐳u∈ℝD,\\Phi\_\{\\text\{core\}\}:\\\{S\_\{u\}^\{\(1\)\},\\dots,S\_\{u\}^\{\(M\)\}\\\}\\rightarrow\\mathbf\{z\}\_\{u\}\\in\\mathbb\{R\}^\{D\},such that:

- •𝐳u\\mathbf\{z\}\_\{u\}captures behavioral patterns and temporal dynamics across all modalities;
- •the embedding is task\-independent and can be reused across diverse downstream objectives;
- •Φcore\\Phi\_\{\\text\{core\}\}can be frozen after pretraining, enabling lightweight downstream models to be trained efficiently on top of\(𝐳u,𝐟u\)\(\\mathbf\{z\}\_\{u\},\\mathbf\{f\}\_\{u\}\)without fine\-tuning the core model\.

## 4\.Approach

We adopt an early fusion strategy to capture cross\-modal interactions in user behavior better\. Events from all available data sources are merged into a single chronological sequence, and we pretrain a transformer\-based sequence encoder on top of it using the next\-event prediction objective\. This pretrained encoder serves as a shared representation backbone: its parameters are frozen and reused without fine\-tuning across all downstream tasks\. For downstream applications, the sequence embedding𝐳u\\mathbf\{z\}\_\{u\}is combined with existing engineered features𝐟u\\mathbf\{f\}\_\{u\}\. Features𝐟u\\mathbf\{f\}\_\{u\}are processed by a separate transformer\-based tabular model, whose output embedding is concatenated with the sequence embedding\. A lightweight MLP head is added to predict the final objective\. The following sections describe the components of this pipeline in detail\.

### 4\.1\.Pretrained Sequence Encoder

![Refer to caption](https://arxiv.org/html/2607.09955v1/images/ssl_full_.png)Figure 1\.Pretrained sequence model architecture\. The sequential processing of heterogeneous user events \(transactions, clicks, communications\) via an event encoder and a shared Transformer backbone, followed by event\-specific predictive heads that jointly model multiple target attributes for each next\-event type\.Block diagram of the pretrained sequence model, with data flowing from bottom to top\. At the input there is a chronological sequence of heterogeneous user events of different types: transactions, clicks, and communications\. Each event first passes through a shared event encoder that maps the different event types into a common embedding space\. The resulting sequence of event embeddings is fed into a Transformer backbone, which processes the events jointly and produces a representation for each position\. On top of the backbone, multiple event\-specific predictive heads sit, one per next\-event type\. Each head jointly predicts several target attributes that characterize the corresponding next event\.We use a three\-stage architecture to process heterogeneous, multimodal event sequence data\. The architecture consists of anEmbedding Layer, anEvent Encoder, and aTransformer Backbone, as shown in Figure[1](https://arxiv.org/html/2607.09955#S4.F1)\.

#### 4\.1\.1\.Embedding Layer

Events are represented by a set of numerical and categorical attributes\. The embedding layer maps individual event attributes into a common embedding dimension, producing fixed\-size representations that are later combined at the event level\.

For numerical attributes, we first apply z\-score normalization:

x~num=xnum−μnumσnum,\\tilde\{x\}\_\{\\text\{num\}\}=\\frac\{x\_\{\\text\{num\}\}\-\\mu\_\{\\text\{num\}\}\}\{\\sigma\_\{\\text\{num\}\}\},wherexnum∈ℝx\_\{\\text\{num\}\}\\in\\mathbb\{R\}denotes a raw numerical value, andμnum\\mu\_\{\\text\{num\}\}andσnum\\sigma\_\{\\text\{num\}\}are the corresponding mean and standard deviation\. The normalized value is then projected into add\-dimensional embedding space using a learnable linear transformation:

𝐚num=𝐰num⋅x~num\+𝐛num,\\mathbf\{a\}\_\{\\text\{num\}\}=\\mathbf\{w\}\_\{\\text\{num\}\}\\cdot\\tilde\{x\}\_\{\\text\{num\}\}\+\\mathbf\{b\}\_\{\\text\{num\}\},where𝐰num∈ℝd\\mathbf\{w\}\_\{\\text\{num\}\}\\in\\mathbb\{R\}^\{d\}and𝐛num∈ℝd\\mathbf\{b\}\_\{\\text\{num\}\}\\in\\mathbb\{R\}^\{d\}are learnable parameters\.

Categorical attributes are mapped using learnable embedding tables with the same embedding dimensiondd\.

#### 4\.1\.2\.Event Encoder: heterogeneous attribute fusion

A key challenge in modeling heterogeneous event data is that different event types are characterized by different sets of attributes\. For example, a transaction event may include amount, location, and category attributes, whereas a clickstream event may contain application section, device type, and related metadata\. The proposed Event Encoder addresses this heterogeneity by padding, modeling intra\-event interactions between attributes, and aggregating them into a single fixed\-size event representation\.

##### Padding

Each eventeeis represented by a variable\-length set of attributes, where the number of attributes depends on the modality\. After embedding each attribute intoℝd\\mathbb\{R\}^\{d\}, the set is padded to a fixed attribute dimensionNattrN\_\{\\text\{attr\}\}, allowing events of different types to be stacked into a tensor of fixed shape for batch training\. Formally, each event is represented as a matrix

𝐀e=\[𝐚1,𝐚2,…,𝐚Nattr\]∈ℝNattr×d,\\mathbf\{A\}\_\{e\}=\[\\mathbf\{a\}\_\{1\},\\mathbf\{a\}\_\{2\},\\ldots,\\mathbf\{a\}\_\{N\_\{\\text\{attr\}\}\}\]\\in\\mathbb\{R\}^\{N\_\{\\text\{attr\}\}\\times d\},where𝐚i∈ℝd\\mathbf\{a\}\_\{i\}\\in\\mathbb\{R\}^\{d\}is the embedding ofii\-th attribute, and padded positions correspond to missing attributes and are masked during subsequent computations\. Here,NattrN\_\{\\text\{attr\}\}denotes the total number of unique attributes among all events used to form a representation\.

##### Intra\-event self\-attention across attributes\.

To capture dependencies between attributes and to identify the most informative ones, we apply self\-attention along the attribute dimension\. This mechanism is conceptually related to intra\-feature attention used in TP\-BERTa\(Yanet al\.,[2024](https://arxiv.org/html/2607.09955#bib.bib3)\)\. Given the attribute matrix𝐀e\\mathbf\{A\}\_\{e\}, we compute

𝐀e′=Attn​\(𝐀e\)=Softmax​\(𝐐𝐊⊤d\)​𝐕,\\mathbf\{A\}^\{\\prime\}\_\{e\}=\\mathrm\{Attn\}\(\\mathbf\{A\}\_\{e\}\)=\\mathrm\{Softmax\}\\\!\\left\(\\frac\{\\mathbf\{Q\}\\mathbf\{K\}^\{\\top\}\}\{\\sqrt\{d\}\}\\right\)\\mathbf\{V\},where𝐐,𝐊,𝐕∈ℝNattr×d\\mathbf\{Q\},\\mathbf\{K\},\\mathbf\{V\}\\in\\mathbb\{R\}^\{N\_\{\\text\{attr\}\}\\times d\}are learned linear projections of𝐀e\\mathbf\{A\}\_\{e\}\. Attention scores corresponding to padded positions are masked before the softmax operation so that missing attributes do not influence the resulting representations\.

##### Event aggregation

The refined attribute representations are aggregated into a single event\-level embedding by pooling over the attribute dimension, followed by LayerNorm:

𝐞=LayerNorm​\(1∑i=1Nattrmi​∑i=1Nattrmi​𝐚i′\),\\mathbf\{e\}=\\mathrm\{LayerNorm\}\\\!\\left\(\\frac\{1\}\{\\sum\_\{i=1\}^\{N\_\{\\text\{attr\}\}\}m\_\{i\}\}\\sum\_\{i=1\}^\{N\_\{\\text\{attr\}\}\}m\_\{i\}\\,\\mathbf\{a\}^\{\\prime\}\_\{i\}\\right\),where𝐚i′∈ℝd\\mathbf\{a\}^\{\\prime\}\_\{i\}\\in\\mathbb\{R\}^\{d\}denotes the attribute representation after self\-attention \(ii\-th row of𝐀e′\\mathbf\{A\}^\{\\prime\}\_\{e\}\), andmi∈\{0,1\}m\_\{i\}\\in\\\{0,1\\\}is a binary mask indicating non\-padding positions\. The normalization ensures that padding does not affect the aggregated representation\.

Finally, we concatenate event embeddings to construct the event sequence:

𝐗=\[𝐞1,𝐞2,…,𝐞T\]∈ℝT×d,\\mathbf\{X\}=\[\\mathbf\{e\}\_\{1\},\\mathbf\{e\}\_\{2\},\\ldots,\\mathbf\{e\}\_\{T\}\]\\in\\mathbb\{R\}^\{T\\times d\},whereTTis the number of events in the sequence\. This yields a homogeneous sequence representation despite heterogeneous event attribute structure\.

#### 4\.1\.3\.Transformer Backbone: sequence modeling

The sequence of event embeddings is processed by a causal transformer backbone that closely follows the GPT\-2 architecture\(Radfordet al\.,[2019](https://arxiv.org/html/2607.09955#bib.bib4)\)\. It consists of masked multi\-head self\-attention and position\-wise feedforward layers with residual connections and layer normalization, enabling autoregressive modeling of event sequences\.

The output of the final transformer layer produces contextualized representations for each event in the sequence\. The final user representation𝐳seq\\mathbf\{z\}\_\{\\text\{seq\}\}is obtained by mean pooling over the temporal dimension of these outputs\.

#### 4\.1\.4\.Pretraining objective

The model is trained using anext\-event predictionobjective\. Given the hidden state \(the output of the last transformer layer\)𝐡t∈ℝd\\mathbf\{h\}\_\{t\}\\in\\mathbb\{R\}^\{d\}corresponding to the event at positiontt, the model predicts the attributes of the subsequent event at positiont\+1t\+1\.

For each categorical attributeccof the next event, we apply a linear projection followed by a softmax to obtain a vector of predicted probabilities for all categories:

𝐩^t\+1\(c\)=Softmax​\(Wcat\(c\)​𝐡t\+𝐛cat\(c\)\),\\hat\{\\mathbf\{p\}\}^\{\(c\)\}\_\{t\+1\}=\\mathrm\{Softmax\}\\\!\\left\(W^\{\(c\)\}\_\{\\mathrm\{cat\}\}\\mathbf\{h\}\_\{t\}\+\\mathbf\{b\}^\{\(c\)\}\_\{\\mathrm\{cat\}\}\\right\),whereWcat\(c\)∈ℝCc×dW^\{\(c\)\}\_\{\\mathrm\{cat\}\}\\in\\mathbb\{R\}^\{C\_\{c\}\\times d\},𝐛cat\(c\)∈ℝCc\\mathbf\{b\}^\{\(c\)\}\_\{\\mathrm\{cat\}\}\\in\\mathbb\{R\}^\{C\_\{c\}\}, andCcC\_\{c\}denotes the number of categories for the categorical attributecc\.

For each numerical attributenn, we use a linear projection to obtain a scalar prediction:

y^t\+1\(n\)=𝐰num\(n\)​𝐡t\+bnum\(n\),\\hat\{y\}^\{\(n\)\}\_\{t\+1\}=\\mathbf\{w\}^\{\(n\)\}\_\{\\text\{num\}\}\\mathbf\{h\}\_\{t\}\+b^\{\(n\)\}\_\{\\text\{num\}\},where𝐰num\(n\)∈ℝd\\mathbf\{w\}^\{\(n\)\}\_\{\\text\{num\}\}\\in\\mathbb\{R\}^\{d\}andbnum\(n\)∈ℝb^\{\(n\)\}\_\{\\text\{num\}\}\\in\\mathbb\{R\}\.

The training loss combines cross\-entropy \(CE\) losses for categorical attributes and mean squared error \(MSE\) losses for numerical attributes\. For each positiontt, the loss is computed only over attributes actually present in the next event, ignoring missing attributes associated with other modalities\. Let𝒞t\+1\\mathcal\{C\}\_\{t\+1\}and𝒩t\+1\\mathcal\{N\}\_\{t\+1\}denote the sets of observed categorical and numerical attributes of the event att\+1t\+1\. The loss at positionttis defined as:

ℒt=∑c∈𝒞t\+1CE​\(𝐲t\+1\(c\),𝐩^t\+1\(c\)\)\+∑n∈𝒩t\+1MSE​\(yt\+1\(n\),y^t\+1\(n\)\),\\mathcal\{L\}\_\{t\}=\\sum\_\{c\\in\\mathcal\{C\}\_\{t\+1\}\}\\text\{CE\}\\\!\\left\(\\mathbf\{y\}^\{\(c\)\}\_\{t\+1\},\\hat\{\\mathbf\{p\}\}^\{\(c\)\}\_\{t\+1\}\\right\)\+\\sum\_\{n\\in\\mathcal\{N\}\_\{t\+1\}\}\\text\{MSE\}\\\!\\left\(y^\{\(n\)\}\_\{t\+1\},\\hat\{y\}^\{\(n\)\}\_\{t\+1\}\\right\),where𝐲t\+1\(c\)\\mathbf\{y\}^\{\(c\)\}\_\{t\+1\}is the one\-hot target for categorical attributecc, andyt\+1\(m\)y^\{\(m\)\}\_\{t\+1\}is the target value for numerical attributenn\. The final objective aggregatesℒt\\mathcal\{L\}\_\{t\}over all valid positions in a sequence\.

### 4\.2\.Tabular Model

We apply a transformer\-based neural network for tabular data, hereafter referred to asTabNN, to process engineered user\-level features and produce a tabular embedding for downstream prediction\. The tabular representation is constructed from two types of input features: categorical and numerical\. The model architecture is illustrated in Figure[2](https://arxiv.org/html/2607.09955#S4.F2)\.

#### 4\.2\.1\.Input preprocessing

Numerical features undergo a two\-stage normalization procedure\. First, a signed logarithmic transform is applied to reduce skewness:x′=sign​\(x\)⋅log⁡\(1\+\|x\|\)x^\{\\prime\}=\\text\{sign\}\(x\)\\cdot\\log\(1\+\|x\|\)\. The result is then standardized using the mean \(μ\\mu\) and standard deviation \(σ\\sigma\) calculated from the training data:x~=\(x′−μ\)/\(σ\+ε\)\\tilde\{x\}=\(x^\{\\prime\}\-\\mu\)/\(\\sigma\+\\varepsilon\), whereε\>0\\varepsilon\>0is a small constant added for numerical stability\.

![Refer to caption](https://arxiv.org/html/2607.09955v1/images/tabular_pipeline.png)Figure 2\.Schematic overview of the proposed downstream model architecture\. The pipeline integrates a frozen sequential model with a tabular Transformer\. Sequential events are encoded into embeddings and refined via an MLP adapter\. Simultaneously, the tabular Transformer processes both categorical and numerical engineered features, and its output is aggregated\. In the end, two representations are concatenated, and a task\-specific head is added\.Block diagram of the downstream model architecture, showing two parallel branches that merge before the output\. In the first branch, a frozen pretrained sequential model encodes user events into embeddings, which are then refined by an MLP adapter\. In the second branch, a tabular Transformer processes engineered categorical and numerical features, and its output is aggregated into a single vector\. The two resulting representations are concatenated, and a task\-specific head is added on top to produce the prediction\.
#### 4\.2\.2\.Embedding Layer

Each numerical feature is associated with a trainable vector𝐰numj∈ℝdemb\\mathbf\{w\}\_\{\\text\{num\}\}^\{j\}\\in\\mathbb\{R\}^\{d\_\{\\text\{emb\}\}\}, which is scaled by the corresponding normalized feature value:

𝐞numj=𝐰numj⋅x~j,\\mathbf\{e\}\_\{\\text\{num\}\}^\{j\}=\\mathbf\{w\}\_\{\\text\{num\}\}^\{j\}\\cdot\\tilde\{x\}^\{j\},wherex~j∈ℝ\\tilde\{x\}^\{j\}\\in\\mathbb\{R\}denotes the normalized value of thejj\-th numerical feature\. The total number of numerical features is denoted byNnumN\_\{\\text\{\\text\{num\}\}\}\.

For each categorical feature, a trainable embedding table is learned, assigning a vector of dimensiondembd\_\{\\text\{emb\}\}to each category\. Let us denote the embedding of thejj\-th categorical feature as𝐞catj\\mathbf\{e\}\_\{\\text\{cat\}\}^\{j\}, and the number of categorical features isNcatN\_\{\\text\{cat\}\}\.

#### 4\.2\.3\.Concatenation & Noise Injection

All feature embeddings are concatenated into a single sequence:

𝐄concat=\[𝐞cat1,…,𝐞catNcat,𝐞num1,…,𝐞numNnum\]\.\\mathbf\{E\}\_\{\\text\{concat\}\}=\[\\mathbf\{e\}^\{1\}\_\{\\text\{\\text\{cat\}\}\},\\dots,\\mathbf\{e\}^\{N\_\{\\text\{cat\}\}\}\_\{\\text\{cat\}\},\\mathbf\{e\}^\{1\}\_\{\\text\{num\}\},\\dots,\\mathbf\{e\}^\{N\_\{\\text\{num\}\}\}\_\{\\text\{num\}\}\]\.To improve robustness and provide regularization, a small random noise vector is added to each embedding in the sequence during training\.

#### 4\.2\.4\.Transformer Processing

The resulting sequence of feature embeddings is processed by a Transformer encoder\. Each layer applies multi\-head self\-attentionacross the feature dimension, allowing the model to learn interactions between different user features\.

#### 4\.2\.5\.Weighted Aggregation

The sequence of feature representations𝐇=\[𝐡1,𝐡2,…,𝐡Ntotal\]\\mathbf\{H\}=\[\\mathbf\{h\}\_\{1\},\\mathbf\{h\}\_\{2\},\\dots,\\mathbf\{h\}\_\{N\_\{\\text\{total\}\}\}\]from the final Transformer layer is aggregated into a single fixed\-size vector𝐳tab\\mathbf\{z\}\_\{\\text\{tab\}\}using a weighted sum with learnable weights𝜶∈ℝNtotal\\boldsymbol\{\\alpha\}\\in\\mathbb\{R\}^\{N\_\{\\text\{total\}\}\}:

𝐳tab=∑i=1Ntotalαi⋅𝐡i,\\mathbf\{z\}\_\{\\text\{tab\}\}=\\sum\_\{i=1\}^\{N\_\{\\text\{total\}\}\}\\alpha\_\{i\}\\cdot\\mathbf\{h\}\_\{i\},whereNtotal=Nnum\+NcatN\_\{\\text\{total\}\}=N\_\{\\text\{num\}\}\+N\_\{\\text\{cat\}\}is the total number of features\.

### 4\.3\.Downstream pipeline

For downstream tasks, the final user representation is obtained by combining embeddings from the sequential and tabular models, as illustrated in Figure[2](https://arxiv.org/html/2607.09955#S4.F2)\. The sequential user embedding is first adapted to the target dimensionality using a lightweight MLP adapter consisting of two linear layers with LayerNorm, SeLU activation\(Klambaueret al\.,[2017](https://arxiv.org/html/2607.09955#bib.bib5)\), and dropout:

𝐳seq′=MLPadapter​\(𝐳seq\)\.\\mathbf\{z\}\_\{\\text\{seq\}\}^\{\\prime\}=\\text\{MLP\}\_\{\\text\{adapter\}\}\(\\mathbf\{z\}\_\{\\text\{seq\}\}\)\.
The adapted sequential embedding𝐳seq′\\mathbf\{z\}\_\{\\text\{seq\}\}^\{\\prime\}is then concatenated with the tabular embedding𝐳tab\\mathbf\{z\}\_\{\\text\{tab\}\}produced by TabNN:

𝐳u=\[𝐳seq′,𝐳tab\]\.\\mathbf\{z\}\_\{u\}=\[\\mathbf\{z\}\_\{\\text\{seq\}\}^\{\\prime\},\\mathbf\{z\}\_\{\\text\{tab\}\}\]\.
The combined representation𝐳u\\mathbf\{z\}\_\{u\}is passed to a task\-specific head \(e\.g\., a linear layer or MLP\) for final prediction:

y^=Headtask​\(𝐳u\)\.\\hat\{y\}=\\text\{Head\}\_\{\\text\{task\}\}\(\\mathbf\{z\}\_\{u\}\)\.
To adapt the system to a particular downstream task, the pretrained sequential encoder is keptfrozen\. Training is performed only on the parameters of the TabNN and the MLP adapter, as well as the task\-specific head\. This approach ensures that task\-relevant patterns are extracted from the aggregate features, while the large, general\-purpose sequential embedding provides a stable, information\-rich foundation\.

As a baseline for evaluating the contribution of the pretrained sequential model, we use the same TabNN architecture trained solely on engineered features with the corresponding task\-specific head, without incorporating the sequential embedding𝐳seq′\\mathbf\{z\}\_\{\\text\{seq\}\}^\{\\prime\}\. In addition, for ablation analysis, we consider a complementary setting in which only the sequential embedding𝐳seq′\\mathbf\{z\}\_\{\\text\{seq\}\}^\{\\prime\}is used together with a task\-specific head, excluding the tabular embedding𝐳tab\\mathbf\{z\}\_\{\\text\{tab\}\}\.

### 4\.4\.Design Choices

#### 4\.4\.1\.Pretraining objective

The choice of the next\-event prediction \(NEP\) objective for pretraining the sequential model is motivated by prior work demonstrating the effectiveness of next\-step prediction for learning transferable representations from behavioral sequences\(Khrylchenkoet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib9); Braithwaiteet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib18); Skalskiet al\.,[2023](https://arxiv.org/html/2607.09955#bib.bib16); Yehet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib17)\)\. In addition, we empirically evaluated several alternative self\-supervised objectives, including CoLES\(Babaevet al\.,[2022](https://arxiv.org/html/2607.09955#bib.bib15)\), masked language modeling \(MLM\), and DeTPP\(Karpukhin and Savchenko,[2024](https://arxiv.org/html/2607.09955#bib.bib20)\), and NEP yielded the strongest performance\.

#### 4\.4\.2\.Tabular model architecture

The described tabular processing pipeline was selected through extensive evaluation on our proprietary datasets\. We benchmarked several state\-of\-the\-art approaches, including standard Feed\-Forward Neural Networks, Deep & Cross Network v2 \(DCNv2\)\(Wanget al\.,[2021](https://arxiv.org/html/2607.09955#bib.bib1)\), and the recently proposed TabM model\(Gorishniyet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib2)\)\. Our architecture is closely related to the FT\-Transformer\(Daiet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib19)\)but differs in the choice of activation functions and the order of normalization operations\. In our experiments, the selected TabNN architecture outperformed other approaches across multiple tasks\.

## 5\.Offline Experiments

We evaluate the proposed approach through a series of offline experiments, including comparison with a strong feature\-based baseline, analysis of scaling behavior, assessment of self\-supervised pretraining, and ablations on input modalities and fusion design\.

### 5\.1\.Experimental Setup

#### 5\.1\.1\.Datasets

We use three behavioral data domains to construct multimodal sequential inputs for foundation model pretraining:

- •Transactions \(Txn\): financial operations \(debits, credits, transfers\) represented by attributes including transaction amount, category, and timestamp\.
- •Clickstream \(Click\): time\-ordered sequences of user interactions within the bank’s mobile application, including visited application sections and associated timestamps\.
- •Communications \(Comm\): records of customer interactions with the bank, including chat and call\-center requests, push notifications, emails, and product applications\.

Their feature composition and sequence statistics are summarized in Table[1](https://arxiv.org/html/2607.09955#S5.T1)\. All datasets cover 2 years\.

Table 1\.Feature set statistics by data domain\. Seq\. length/year denotes the length of a customer event sequence per year\.In addition to sequential event data, we use a set of pre\-existing tabular user features available in the production system\. For each user and month, these features are basically computed by aggregating raw behavioral signals over predefined temporal windows aligned with the given month and include both numerical and categorical variables\. No additional feature construction is performed as part of this work\.

#### 5\.1\.2\.Tasks and evaluation

We evaluate the proposed approach on four downstream binary classification tasks \(Task 1–Task 4\), each predicting customer response to communications for a specific bank product, a key business objective for customer engagement\. Performance is measured using ROC AUC on a held\-out test set\. All experiments are conducted on the same set of tasks and metrics, and we additionally report the average ROC AUC across all tasks for each configuration\.

#### 5\.1\.3\.Pretrained model configurations

We consider multiple configurations of the pretrained sequence encoder to analyze the effect of model scale on downstream performance\. Specifically, we study three model sizes that vary in depth, width, and maximum context length:

- •2M parameters \(small\): 6 layers, 4 attention heads, hidden size 128, context length 256
- •8M parameters \(medium\): 8 layers, 6 attention heads, hidden size 256, context length 1024
- •42M parameters \(large\): 12 layers, 6 attention heads, hidden size 512, context length 2048

Unless stated otherwise, we use the medium\-sized 8M configuration as the default pretrained backbone in our experiments\.

#### 5\.1\.4\.Pretraining setup

We train the sequence encoder using a batch size of 32 with gradient accumulation over 16 steps, dropout rate of 0\.15, and the AdamW optimizer \(learning rate10−310^\{\-3\}, weight decay10−210^\{\-2\},ϵ=10−4\\epsilon=10^\{\-4\}\)\. We apply a cosine learning\-rate schedule with 10,000 warmup steps and train the model for 2\-10 epochs depending on the model size\. Gradient clipping with a maximum norm of 1\.0 is applied throughout training\.

Training is distributed across 4–32 NVIDIA A100 80GB GPUs using Distributed Data Parallel \(DDP\)\. The best model checkpoint is selected based on validation loss\.

#### 5\.1\.5\.Downstream training setup

All downstream tabular models are trained using a fixed set of hyperparameters across all experiments\. Specifically, we use a batch size of 4096, an embedding dimension ofdemb=64d\_\{\\text\{emb\}\}=64, three transformer layers with four attention heads, a learning rate of0\.0040\.004, weight decay of0\.010\.01, 6,000 warmup steps\. Models are trained for 40 epochs\.

Training is distributed across 4 NVIDIA A100 80GB GPUs using Distributed Data Parallel \(DDP\)\. For evaluation, we select the model checkpoint with the best validation performance and report the corresponding test\-set metric\.

### 5\.2\.Comparison with a Feature\-Based Baseline

We first compare the proposed approach against a strong baseline built on hand\-engineered user features that are widely used in our production systems\. They have been developed and refined over an extended period of time and capture domain knowledge accumulated across multiple teams and use cases\. As a result, models trained on these features present a highly competitive baseline that is difficult to outperform\. Even relatively small improvements in terms of ROC AUC translate into measurable gains in business value\.

Table[2](https://arxiv.org/html/2607.09955#S5.T2)reports results on four downstream prediction tasks, as well as the average performance across all tasks\. We consider three variants of input representation: hand\-engineered features, frozen embeddings produced by a pretrained model with 42M parameters, and a combination of pretrained embeddings and engineered features\. When used in isolation, pretrained embeddings can outperform or underperform engineered features depending on the task\. However, their combination is consistently better than feature\-based baseline, leading to an average ROC AUC improvement of 0\.009\.

Table 2\.Comparison of pretrained model \(42M large\) with previously used approach based on engineered features\.
### 5\.3\.Scaling Properties

We further investigate the scaling properties of the pretrained model to assess whether increasing model size leads to performance improvements, as observed in other domains and prior work on foundation models\(Khrylchenkoet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib9); Yehet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib17); Braithwaiteet al\.,[2025](https://arxiv.org/html/2607.09955#bib.bib18)\)\.

#### 5\.3\.1\.Scaling model capacity

We compare three model sizes \(2M, 8M, and 42M\) and evaluate each in isolation and in combination with TabNN operating on engineered features\. Table[3](https://arxiv.org/html/2607.09955#S5.T3)reports the average ROC AUC across all downstream tasks\. As expected, larger models achieve higher ROC AUC, and adding TabNN yields consistent improvements at every scale\. Since engineered features already capture a substantial amount of relevant information, increasing the capacity of the pretrained model has a smaller effect in the combined setting \(from 0\.792 to 0\.801 when scaling from 2M to 42M\) compared to using pretrained embeddings alone \(from 0\.750 to 0\.787\)\. Notably, improvements over the feature\-based baseline \(0\.792\) become significant only at a sufficient model scale\.

While further increasing model size could potentially lead to additional gains, practical constraints in the production environment currently limit the feasibility of deploying larger models\. We therefore choose the 42M configuration as a reasonable trade\-off between performance and operational cost\.

Table 3\.The effect of the pretrained model size on the average ROC AUC metric with and without TabNN operating on engineered features\.
#### 5\.3\.2\.Effect of context length scaling

We additionally study the effect of scaling only context length using the medium\-sized \(8M\) model, varying the maximum sequence length from 256 to 1024 events\. As shown in Table[4](https://arxiv.org/html/2607.09955#S5.T4), longer user histories consistently improve ROC AUC across all four tasks, indicating that the model benefits from access to extended temporal context\.

Table 4\.The effect of the transformer context length on the ROC AUC metric for various tasks\. The results are shown for the 8M medium model\.

### 5\.4\.Impact of pretraining

To analyze the effect of self\-supervised pretraining, we compare our approach against the same model architecture trained end\-to\-end in a supervised manner to directly predict downstream labels from user histories and engineered features\. Table[5](https://arxiv.org/html/2607.09955#S5.T5)shows that self\-supervised pretraining consistently outperforms supervised training across all four downstream tasks, with an average ROC AUC improvement of 0\.004\. The largest gain is observed on Task 1, while Tasks 2–4 exhibit smaller but stable improvements\. This behavior is expected in the considered data regime, where the amount of labeled data available for downstream tasks is substantially smaller than the unlabeled data used for pretraining\.

Table 5\.Comparison of the ROC AUC metric for self\-supervised pretraining and supervised learning on the downstream tasks with the same architecture from scratch\. The results are shown for the 8M medium model\.As a result, a single pretrained backbone can be reused across multiple downstream tasks in a frozen form, reducing model development and maintenance costs while simultaneously improving predictive quality\.

### 5\.5\.Contribution of Multiple Behavioral Modalities

We analyze the contribution of different input data sources, using a model trained on Transactions \(Txn\-only\) as a baseline, since it is the primary and most informative data source for the considered downstream tasks\. Table[6](https://arxiv.org/html/2607.09955#S5.T6)compares three configurations: a Txn\-only model, a model combining Transactions and Communications \(Txn \+ Comm\), and a model that additionally incorporates Clickstream \(Txn \+ Comm \+ Click\), which corresponds to our final multimodal setup\.

As expected, the impact of additional modalities varies across tasks, reflecting differences in the relevance of behavioral signals for each prediction objective\. Nevertheless, multimodal models match or outperform the Txn\-only baseline across all tasks and achieve the strongest gains on Task 2, where the Txn \+ Comm \+ Click configuration outperforms the Txn\-only model by more than 0\.01 ROC AUC\. These results indicate that communication and clickstream data provide complementary signals beyond transactional data\.

Table 6\.The impact of combining different data sources on the ROC AUC metric\. The results are shown for the 8M medium model\.
### 5\.6\.Comparison of Fusion Strategies

We compare the early fusion strategy adopted in our main experiments with two late fusion approaches for combining heterogeneous sequential modalities\.

In the first late fusion variant \(Late fusion with hiddens\), we pretrain separate sequential encoders for each modality, extract their hidden representations, and concatenate them before passing the resulting representation to the downstream model together with standard tabular features\. In the second variant \(Late fusion with scores\), we again pretrain separate sequential encoders, but pair each of them with its own downstream model\. The resulting prediction scores are then combined using a logistic regression layer trained on top of the individual model outputs\.

In contrast, the early fusion model uses a single multimodal sequential encoder that jointly processes all modalities and produces a shared representation, which is subsequently consumed by a downstream model\.

Table[7](https://arxiv.org/html/2607.09955#S5.T7)shows that early fusion achieves the highest ROC AUC on three of the four tasks \(Task1, Task3, Task4\), with noticeable gains on Task 1 and Task 4 and comparable performance to the best late fusion variant on Task2 and Task 3\. On average across all four tasks, early fusion delivers the best overall performance\.

Table 7\.Comparison of the ROC AUC metric for different fusion strategies\. The results are shown for the 8M medium model\.

## 6\.Online Results

We evaluated the proposed approach in A/B tests on core banking products, where communications were delivered through three channels: push notifications, SMS, and telemarketing\. As the target online metric, we used the Net Present Value \(NPV\), and we report the relative improvement of the treatment group over the control group:

Δ=NPVtreat−NPVctrlNPVctrl\.\\Delta=\\frac\{\\mathrm\{NPV\}\_\{\\mathrm\{treat\}\}\-\\mathrm\{NPV\}\_\{\\mathrm\{ctrl\}\}\}\{\\mathrm\{NPV\}\_\{\\mathrm\{ctrl\}\}\}\.
Model scores produced by our approach were consumed by an existing production optimizer\. For each client–product–channel candidate, a proxy for expected incremental profit was computed asscore×NPV\\text\{score\}\\times\\text\{NPV\}\. The optimizer then built a communication plan that maximizes total NPV subject to business constraints \(e\.g\., channel capacity limits, contact policies\)\.

For online evaluation, we deployed the proposed model for 6 core products \(out of more than 40 products offered by the company\)\. These products were selected based on the highest historical NPV, as they contribute the most to total revenue and provide sufficient traffic for statistically reliable A/B evaluation\. We report the impact on total NPV across all products because the optimizer jointly allocates limited contact capacity across competing offers, so changes in scores for a subset of products can shift exposure and revenue among the remaining products\. As the control group, we used the production Gradient Boosting Decision Trees \(GBDT\) models deployed at the time of the experiment\. The treatment group used the proposed architecture, while keeping all remaining business rules and operational constraints unchanged\. Clients were randomly assigned to control and treatment groups, with approximately 10% of the total client base in the treatment group and the remaining population in the control group\. Statistical significance was assessed using a t\-test with a significance level of 0\.05, and the experiment duration \(2 months\) was chosen to ensure sufficient power given the variability of NPV\.

Across products and communication channels, the proposed architecture achieved an average\+1%\+1\\%upliftin total NPV relative to the GBDT baseline\. While the absolute uplift value may appear modest, it corresponds to a substantial absolute revenue gain at the system scale\. Following successful A/B testing, we transitioned the production scoring pipelines for the evaluated products to the proposed approach, which has since been adopted as the default scoring model\.

## 7\.Conclusion

In this work, we introduced a foundation model for multimodal behavioral data that jointly leverages transactional histories, clickstream signals, and communication events\. The proposed approach is based on pretraining a transformer\-based sequence encoder on unified event sequences and reusing it as a frozen shared representation backbone across diverse downstream tasks\. Extensive offline experiments demonstrate the effectiveness of the proposed approach\. The proposed system has been successfully deployed at scale in production and validated through online A/B testing, resulting in measurable improvements in business metrics and confirming the practical value of foundation models for large\-scale financial applications\.

## References

- D\. Babaev, N\. Ovsov, I\. Kireev, M\. Ivanova, G\. Gusev, I\. Nazarov, and A\. Tuzhilin \(2022\)CoLES: contrastive learning for event sequences with self\-supervision\.InProceedings of the 2022 International Conference on Management of Data,pp\. 1190–1199\.Cited by:[§1](https://arxiv.org/html/2607.09955#S1.p1.1),[§2\.2](https://arxiv.org/html/2607.09955#S2.SS2.p1.1),[§4\.4\.1](https://arxiv.org/html/2607.09955#S4.SS4.SSS1.p1.1)\.
- A\. Bazarova, M\. Kovaleva, I\. Kuleshov, E\. Romanenkova, A\. Stepikin, A\. Yugay, D\. Mollaev, I\. Kireev, A\. Savchenko, and A\. Zaytsev \(2025\)Learning transactions representations for information management in banks: mastering local, global, and external knowledge\.International Journal of Information Management Data Insights5\(1\),pp\. 100323\.Cited by:[§2\.2](https://arxiv.org/html/2607.09955#S2.SS2.p1.1)\.
- D\. Braithwaite, M\. Cavalcanti, R\. A\. McEver, H\. Udagawa, D\. Silva, R\. Ramanath, F\. Meneses, A\. Yoshida, E\. Wingert, M\. Ramos,et al\.\(2025\)Your spending needs attention: modeling financial habits with transformers\.arXiv preprint arXiv:2507\.23267\.Cited by:[§1](https://arxiv.org/html/2607.09955#S1.p1.1),[§1](https://arxiv.org/html/2607.09955#S1.p2.1),[§1](https://arxiv.org/html/2607.09955#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.09955#S2.SS2.p2.1),[§4\.4\.1](https://arxiv.org/html/2607.09955#S4.SS4.SSS1.p1.1),[§5\.3](https://arxiv.org/html/2607.09955#S5.SS3.p1.1)\.
- X\. Chen, K\. Rajesh, M\. Lawhon, Z\. Wang, H\. Li, H\. Li, S\. V\. Joshi, P\. Eksombatchai, J\. Yang, Y\. Hsu,et al\.\(2025\)PinFM: foundation model for user activity sequences at a billion\-scale visual discovery platform\.InProceedings of the Nineteenth ACM Conference on Recommender Systems,pp\. 381–390\.Cited by:[§2\.1](https://arxiv.org/html/2607.09955#S2.SS1.p1.1)\.
- J\. Dabrowski, M\. Janicka, L\. Sienkiewicz, G\. Stomfai, D\. Jannach, F\. Barile, M\. Polignano, C\. Pomo, and A\. Srivastava \(2025\)RecSys challenge 2025: universal behavioral profiles for recommender systems\.InProceedings of the Nineteenth ACM Conference on Recommender Systems,pp\. 1389–1393\.Cited by:[§2\.1](https://arxiv.org/html/2607.09955#S2.SS1.p2.1)\.
- H\. Dai, S\. Wu, H\. Zhao, J\. Huang, Z\. Jian, Y\. Zhu, and H\. Hu \(2025\)Cited by:[§4\.4\.2](https://arxiv.org/html/2607.09955#S4.SS4.SSS2.p1.1)\.
- Y\. Dou, Z\. Jiang, T\. Zhang, M\. Hu, Z\. Xu, S\. Jain, U\. S\. Saini, X\. Fan, J\. Sun, M\. Pan,et al\.\(2025\)TransactionGPT\.arXiv preprint arXiv:2511\.08939\.Cited by:[§1](https://arxiv.org/html/2607.09955#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.09955#S2.SS2.p2.1)\.
- J\. Gong, J\. Ding, F\. Meng, C\. Yang, H\. Chen, Z\. Wang, H\. Lu, and Y\. Li \(2025\)BehaveGPT: a foundation model for large\-scale user behavior modeling\.arXiv preprint arXiv:2505\.17631\.Cited by:[§2\.1](https://arxiv.org/html/2607.09955#S2.SS1.p2.1)\.
- Y\. Gorishniy, A\. Kotelnikov, and A\. Babenko \(2025\)TabM: advancing tabular deep learning with parameter\-efficient ensembling\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Sd4wYYOhmY)Cited by:[§4\.4\.2](https://arxiv.org/html/2607.09955#S4.SS4.SSS2.p1.1)\.
- Y\. Guo, W\. Zhang, X\. Zhang, V\. W\. Zheng, and Y\. Yang \(2025\)Efficient multi\-expert tabular language model for banking\.InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\. 1,pp\. 2271–2281\.Cited by:[§2\.2](https://arxiv.org/html/2607.09955#S2.SS2.p2.1)\.
- S\. Jha, M\. Guillen, and J\. C\. Westland \(2012\)Employing transaction aggregation strategy to detect credit card fraud\.Expert systems with applications39\(16\),pp\. 12650–12657\.Cited by:[§1](https://arxiv.org/html/2607.09955#S1.p1.1)\.
- I\. Karpukhin and A\. Savchenko \(2024\)Detecting the future: all\-at\-once event sequence forecasting with horizon matching\.arXiv preprint arXiv:2408\.13131\.Cited by:[§4\.4\.1](https://arxiv.org/html/2607.09955#S4.SS4.SSS1.p1.1)\.
- I\. Karpukhin and A\. Savchenko \(2025\)HT\-transformer: event sequences classification by accumulating prefix information with history tokens\.arXiv preprint arXiv:2508\.01474\.Cited by:[§1](https://arxiv.org/html/2607.09955#S1.p3.1)\.
- K\. Khrylchenko, A\. Matveev, S\. Makeev, and V\. Baikalov \(2025\)Scaling recommender transformers to one billion parameters\.arXiv preprint arXiv:2507\.15994\.Cited by:[§2\.1](https://arxiv.org/html/2607.09955#S2.SS1.p1.1),[§4\.4\.1](https://arxiv.org/html/2607.09955#S4.SS4.SSS1.p1.1),[§5\.3](https://arxiv.org/html/2607.09955#S5.SS3.p1.1)\.
- G\. Klambauer, T\. Unterthiner, A\. Mayr, and S\. Hochreiter \(2017\)Self\-normalizing neural networks\.External Links:1706\.02515,[Link](https://arxiv.org/abs/1706.02515)Cited by:[§4\.3](https://arxiv.org/html/2607.09955#S4.SS3.p1.1)\.
- A\. Klenitskiy, A\. Fatkulin, D\. Denisova, A\. Pembek, and A\. Vasilev \(2025\)Encode me if you can: learning universal user representations via event sequence autoencoding\.InProceedings of the Recommender Systems Challenge 2025,pp\. 26–30\.Cited by:[§2\.1](https://arxiv.org/html/2607.09955#S2.SS1.p2.1)\.
- C\. Liu, Y\. Gao, L\. Sun, J\. Feng, H\. Yang, and X\. Ao \(2022\)User behavior pre\-training for online fraud detection\.InProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,pp\. 3357–3365\.Cited by:[§2\.1](https://arxiv.org/html/2607.09955#S2.SS1.p2.1)\.
- W\. Lyu, D\. Tyagi, Y\. Yang, Z\. Li, A\. Somani, K\. Shanmugasundaram, N\. Andrejevic, F\. Adeputra, C\. Zeng, A\. K\. Singh,et al\.\(2025\)DV365: extremely long user history modeling at instagram\.InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\. 2,pp\. 4717–4727\.Cited by:[§2\.1](https://arxiv.org/html/2607.09955#S2.SS1.p1.1)\.
- S\. Makeev, A\. Andreev, V\. Baikalov, V\. Tytskiy, A\. Krasilnikov, and K\. Khrylchenko \(2025\)Blending sequential embeddings, graphs, and engineered features: 4th place solution in recsys challenge 2025\.InProceedings of the Recommender Systems Challenge 2025,pp\. 21–25\.Cited by:[§2\.1](https://arxiv.org/html/2607.09955#S2.SS1.p2.1)\.
- D\. Mollaev, I\. Kireev, M\. Orlov, A\. Kostin, I\. Karpukhin, M\. Postnova, G\. Gusev, and A\. Savchenko \(2025\)Multimodal banking dataset: understanding client needs through event sequences\.InProceedings of the 34th ACM International Conference on Information and Knowledge Management,pp\. 6476–6480\.Cited by:[§1](https://arxiv.org/html/2607.09955#S1.p2.1)\.
- V\. Moskvoretskii, D\. Osin, E\. Shvetsov, I\. Udovichenko, M\. Zhelnin, A\. Dukhovny, A\. Zhimerikina, and E\. Burnaev \(2024\)MLEM: generative and contrastive learning as distinct modalities for event sequences\.arXiv preprint arXiv:2401\.15935\.Cited by:[§2\.2](https://arxiv.org/html/2607.09955#S2.SS2.p1.1)\.
- M\. Ostroukhov, R\. Mikhailov, V\. Iashin, A\. Sokolov, A\. Akshonov, V\. Protasov, D\. Beloborodov, V\. Mullin, R\. Y\. Enzmann, G\. Kolovos,et al\.\(2026\)PRAGMA: revolut foundation model\.arXiv preprint arXiv:2604\.08649\.Cited by:[§1](https://arxiv.org/html/2607.09955#S1.p3.1)\.
- N\. Pancha, A\. Zhai, J\. Leskovec, and C\. Rosenberg \(2022\)PinnerFormer: sequence modeling for user representation at pinterest\.InProceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining,pp\. 3702–3712\.Cited by:[§2\.1](https://arxiv.org/html/2607.09955#S2.SS1.p1.1)\.
- G\. Polleti, M\. Santana, and E\. Fontes \(2025\)Open banking foundational model: learning language representations from few financial transactions\.arXiv preprint arXiv:2511\.12154\.Cited by:[§2\.2](https://arxiv.org/html/2607.09955#S2.SS2.p2.1)\.
- A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. Sutskever \(2019\)Language models are unsupervised multitask learners\.OpenAI\.Note:Accessed: 2024\-11\-15External Links:[Link](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)Cited by:[§4\.1\.3](https://arxiv.org/html/2607.09955#S4.SS1.SSS3.p1.1)\.
- N\. Raman, S\. Ganesh, and M\. Veloso \(2024\)Scalable representation learning for multimodal tabular transactions\.arXiv preprint arXiv:2410\.07851\.Cited by:[§2\.2](https://arxiv.org/html/2607.09955#S2.SS2.p2.1)\.
- A\. Sakhno, I\. Kireev, D\. Babaev, M\. Savchenko, G\. Gusev, and A\. Savchenko \(2025\)PyTorch\-lifestream: learning embeddings on discrete event sequences\.InProceedings of the Thirty\-Fourth International Joint Conference on Artificial Intelligence,pp\. 11104–11108\.Cited by:[§2\.2](https://arxiv.org/html/2607.09955#S2.SS2.p1.1)\.
- Y\. Sawada, R\. Hasegawa, Y\. Nagatsuma, S\. Takei, K\. Yonekawa, and H\. Auchi \(2025\)Toward universal user representations: contrastive learning with transformers and embedding ensembles\.InProceedings of the Recommender Systems Challenge 2025,pp\. 51–55\.Cited by:[§2\.1](https://arxiv.org/html/2607.09955#S2.SS1.p2.1)\.
- A\. Shestov, O\. Zoloev, M\. Makarenko, M\. Orlov, E\. Fadeev, I\. Kireev, and A\. Savchenko \(2025\)LLM4ES: learning user embeddings from event sequences via large language models\.InProceedings of the 34th ACM International Conference on Information and Knowledge Management,pp\. 5238–5242\.Cited by:[§2\.2](https://arxiv.org/html/2607.09955#S2.SS2.p2.1)\.
- P\. Skalski, D\. Sutton, S\. Burrell, I\. Perez, and J\. Wong \(2023\)Towards a foundation purchasing model: pretrained generative autoregression on transaction sequences\.InProceedings of the Fourth ACM International Conference on AI in Finance,pp\. 141–149\.Cited by:[§1](https://arxiv.org/html/2607.09955#S1.p1.1),[§2\.2](https://arxiv.org/html/2607.09955#S2.SS2.p1.1),[§4\.4\.1](https://arxiv.org/html/2607.09955#S4.SS4.SSS1.p1.1)\.
- R\. Wang, R\. Shivanna, D\. Cheng, S\. Jain, D\. Lin, L\. Hong, and E\. Chi \(2021\)DCN v2: improved deep & cross network and practical lessons for web\-scale learning to rank systems\.pp\. 1785–1797\.External Links:[Document](https://dx.doi.org/10.1145/3442381.3450078)Cited by:[§4\.4\.2](https://arxiv.org/html/2607.09955#S4.SS4.SSS2.p1.1)\.
- Z\. Wang, Q\. Wu, B\. Zheng, J\. Wang, K\. Huang, and Y\. Shi \(2023\)Sequence as genes: an user behavior modeling framework for fraud transaction detection in e\-commerce\.InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,pp\. 5194–5203\.Cited by:[§2\.1](https://arxiv.org/html/2607.09955#S2.SS1.p2.1)\.
- X\. Xia, P\. Eksombatchai, N\. Pancha, D\. D\. Badani, P\. Wang, N\. Gu, S\. V\. Joshi, N\. Farahpour, Z\. Zhang, and A\. Zhai \(2023\)TransAct: transformer\-based realtime user action model for recommendation at pinterest\.InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,pp\. 5249–5259\.Cited by:[§2\.1](https://arxiv.org/html/2607.09955#S2.SS1.p1.1)\.
- J\. Yan, B\. Zheng, H\. Xu, Y\. Zhu, D\. Z\. Chen, J\. Sun, J\. Wu, and J\. Chen \(2024\)Making pre\-trained language models great on tabular prediction\.External Links:2403\.01841,[Link](https://arxiv.org/abs/2403.01841)Cited by:[§4\.1\.2](https://arxiv.org/html/2607.09955#S4.SS1.SSS2.Px2.p1.1)\.
- C\. M\. Yeh, U\. S\. Saini, X\. Dai, X\. Fan, S\. Jain, Y\. Fan, J\. Sun, J\. Wang, M\. Pan, Y\. Dou,et al\.\(2025\)TREASURE: a transformer\-based foundation model for high\-volume transaction understanding\.arXiv preprint arXiv:2511\.19693\.Cited by:[§1](https://arxiv.org/html/2607.09955#S1.p1.1),[§1](https://arxiv.org/html/2607.09955#S1.p2.1),[§1](https://arxiv.org/html/2607.09955#S1.p3.1),[§2\.2](https://arxiv.org/html/2607.09955#S2.SS2.p2.1),[§4\.4\.1](https://arxiv.org/html/2607.09955#S4.SS4.SSS1.p1.1),[§5\.3](https://arxiv.org/html/2607.09955#S5.SS3.p1.1)\.