SeqLLM: Augmenting LLMs with Behavioral-Sequence Modeling for High-Stakes Decisions at WeChat Pay

# Augmenting LLMs with Behavioral-Sequence Modeling for High-Stakes Decisions at WeChat Pay
Source: [https://arxiv.org/html/2608.03063](https://arxiv.org/html/2608.03063)
,Jiaxing ZhangShanghai Jiao Tong UniversityChina,Matthias Hwai Yong TanCity University of Hong KongHong Kong SAR, China,Bo WangWeChat Pay, TencentChinaandWeiran HuangShanghai Jiao Tong UniversityChina

###### Abstract\.

Merchant risk control at large payment platforms screens tens of millions of merchants per day, where errors are costly in both directions: a false positive may harm a legitimate merchant, while a false negative leaves harmful activity undetected\. The hardest cases cannot be settled from either modality alone; they require reading a merchant’s textual profile jointly with its long behavioral sequence\. Large language models \(LLMs\) excel at the former but cannot natively model behavioral sequences, and endowing them with this ability typically erodes their language and reasoning skills through catastrophic forgetting\. We present SeqLLM, a framework that injects behavioral\-sequence modeling into a pretrained LLM while preserving its language ability, enabling joint use of content and behavioral sequences\. SeqLLM has three components: a compact discrete behavior vocabulary that represents each event as native tokens; a lightweight projector, trained with a two\-stage alignment curriculum, that grounds behavior tokens in the LLM’s semantic space; and prefix\-guided capability injection, which learns sequence modeling through task\-prefixed supervised fine\-tuning rather than continual pre\-training\. SeqLLM is fully deployed at WeChat Pay, screening millions of merchants per day\. Compared with the production DeepSeek\-based LLM baseline, it improves screening precision from 92\.0% to 97\.5%; its pretrained behavior\-token embeddings also lift Precision@Top\-0\.01% by 26\.8 pp in a production fraud detector serving billion\-scale transaction traffic\. Beyond payment behavior, SeqLLM achieves state\-of\-the\-art results against strong public baselines on open recommendation benchmarks\. On MovieLens and Amazon, it outperforms the strong User\-LLM baseline by up to 32% relative Recall@5 while retaining markedly stronger language ability\. On the RecIF benchmark, SeqLLM improves Pass@32 by 14\.2% over the full OneRec\-8B pipeline, while using only one\-fifth of OneRec\-8B’s GPU\-days\. Our code is available at[https://github\.com/125jx/SeqLLM\.git](https://github.com/125jx/SeqLLM.git)\.

††journalyear:2027††conference:Proceedings of the 33rd ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August 2027; TBD## 1\.Introduction

At WeChat Pay, merchants include both offline businesses and online payment\-accepting entities, such as e\-commerce platforms, mini\-programs, and apps\. Merchant risk control screens millions of merchants each day for fraud, money laundering, and other illicit fund flows\(Phuaet al\.,[2010](https://arxiv.org/html/2608.03063#bib.bib17); Ngaiet al\.,[2011](https://arxiv.org/html/2608.03063#bib.bib15)\)\. Criminals may create online stores that appear legitimate and use them to collect illicit payments\. Errors are costly in both directions: a false positive may harm a legitimate merchant, while a false negative leaves harmful activity undetected, so each decision must be precise\.

The hardest cases are still left to human experts\. Consider a shop registered under “clothing” that in fact sells jade via livestreams, luring buyers to pay by a QR code sent in private chat: suspicious wording, but not conclusive on its own\. Its payments look ordinary too, until its behavioral sequence reveals that they cluster at night, span many provinces, and target elderly buyers, with repeated deleted bills\. Either clue alone is weak, and a pipeline that inspects text and behavioral statistics separately misses the case; only reading the wording jointly with the behavior reveals the fraud\. Such expert review is accurate but slow, and does not scale to tens of millions of merchants\.

This need reflects a broader pattern\. Entities on modern platforms are described along two axes: their content, which captures what an entity is, and their behavior, long sequences of timestamped actions that reveal how it acts over time\(Caffagniet al\.,[2024](https://arxiv.org/html/2608.03063#bib.bib3); Kang and McAuley,[2018](https://arxiv.org/html/2608.03063#bib.bib9); Wanget al\.,[2019](https://arxiv.org/html/2608.03063#bib.bib26)\)\. These axes drive applications from recommendation\(Bobadillaet al\.,[2013](https://arxiv.org/html/2608.03063#bib.bib2); Rendle,[2010](https://arxiv.org/html/2608.03063#bib.bib19); Liuet al\.,[2020](https://arxiv.org/html/2608.03063#bib.bib12); Linet al\.,[2025](https://arxiv.org/html/2608.03063#bib.bib11)\)and e\-commerce search\(Van Gyselet al\.,[2016](https://arxiv.org/html/2608.03063#bib.bib25); Aiet al\.,[2017](https://arxiv.org/html/2608.03063#bib.bib1); Sarviet al\.,[2020](https://arxiv.org/html/2608.03063#bib.bib20)\)to risk control\(Liet al\.,[2025](https://arxiv.org/html/2608.03063#bib.bib10)\), which increasingly demand reasoning over both at once\. Yet the two capabilities have long lived in disjoint model families: large language models \(LLMs\) reason fluently over text but cannot natively consume behavioral sequences\(Caffagniet al\.,[2024](https://arxiv.org/html/2608.03063#bib.bib3); Zhanget al\.,[2024](https://arxiv.org/html/2608.03063#bib.bib28)\), while behavior models capture temporal patterns but lack language, reasoning, and explanation\(Kang and McAuley,[2018](https://arxiv.org/html/2608.03063#bib.bib9); Liet al\.,[2025](https://arxiv.org/html/2608.03063#bib.bib10)\)\. Recent work begins to bridge them, injecting discretized behavior tokens into an LLM\(Denget al\.,[2025](https://arxiv.org/html/2608.03063#bib.bib4); Liuet al\.,[2025](https://arxiv.org/html/2608.03063#bib.bib14); Team,[2026](https://arxiv.org/html/2608.03063#bib.bib23)\)or contextualizing it with a behavior\-derived user embedding\(Ninget al\.,[2025](https://arxiv.org/html/2608.03063#bib.bib16)\)\. This progress has so far centered on engagement\-oriented recommendation, and its extension to high\-stakes decisions, where reliable outcomes require jointly leveraging complementary evidence from text and behavioral sequences, remains largely unexplored\.

Equipping an LLM with behavioral\-sequence modeling, however, is harder than it looks\. The obvious route—serializing a sequence as natural language—creates long inputs and overloads word tokens with behavioral meanings\. It also represents temporal patterns in an embedding space designed for lexical semantics, weakening the sequence signal\.

Learning behavior with dedicated tokens avoids these pitfalls but raises a second challenge: acquiring the new ability without eroding language competence\. OpenOneRec\(Team,[2025a](https://arxiv.org/html/2608.03063#bib.bib22)\)acquires sequence capability through recommendation\-oriented continual pre\-training \(CPT\), then relies on mixed\-domain training and general\-ability distillation to counteract general\-language degradation\. Yet this adapt\-then\-repair pipeline does not eliminate forgetting: despite using roughly44M auxiliary language examples for only156156K behavioral sequences and a separate distillation stage, OpenOneRec still loses general language ability\. In our setting, sequences number in the tens of millions \(∼107\\sim\\\!10^\{7\}\), a scale at which direct CPT drives a strong LLM’s general language ability to near\-collapse \(C\-Eval0\.78→0\.270\.78\\\!\\rightarrow\\\!0\.27\)\. General\-language degradation therefore cannot be reliably offset simply by scaling auxiliary language data\.

To meet both challenges, we proposeSeqLLM, a framework that can be applied to pretrained LLMs, endowing them with native behavioral\-sequence modeling ability while preserving their language ability and supporting downstream tasks that jointly use textual and behavioral information\. SeqLLM rests on three components \(Figure[1](https://arxiv.org/html/2608.03063#S1.F1)\)\. First, a discrete behavior vocabulary encodes each event as field\-level tokens, compressing it from dozens of tokens to about nine on average \(roughly6×6\\times\) while avoiding collisions with the original word vocabulary\. Second, inspired by visual–language alignment\(Liuet al\.,[2023](https://arxiv.org/html/2608.03063#bib.bib13)\), a lightweight behavior projector with text\-grounded initialization aligns these tokens into the LLM’s semantic space through a two\-stage, translation\-then\-reasoning curriculum, so the model first reads behavior tokens and then reasons over them\. Third, and central to our findings, prefix\-guided capability injection learns sequence modeling through instruction\-conditioned SFT rather than CPT: by recasting next\-event prediction as conditional generation under a task prefix, sequence learning is confined to the parameter pathways the prefix activates, avoiding the broad general\-language degradation caused by behavior\-oriented CPT\.

![Refer to caption](https://arxiv.org/html/2608.03063v1/main.png)Figure 1\.Overview of SeqLLM\.\(a\)At inference, projected behavior tokens are interleaved with the task prefix and textual context for joint text–behavior reasoning\.\(b\)Phase 1 grounds text\-initialized behavior tokens through token translation and multi\-event reasoning\. Phase 2 jointly mixes task\-prefixed sequence completion with general\-instruction SFT, applying loss only to response tokens\. This injects sequence modeling without behavior\-oriented CPT or a separate capability\-recovery stage\. The resulting behavior\-capable SeqLLM backbone is adapted through risk\-control SFT for deployment\.We evaluate SeqLLM in two large\-scale business scenarios at WeChat Pay and on three public recommendation benchmarks\. The results demonstrate three main findings\. First, sequence capability can be injected at scale without eroding language ability: trained on approximately2020M unlabeled merchant behavior sequences, prefix\-guided SFT matches CPT on next\-event prediction \(HR@100\.800\.80\) while retaining a C\-Eval score of0\.780\.78, versus0\.270\.27under CPT\. Second, the resulting model delivers measurable production impact at WeChat Pay\. SeqLLM is deployed in two complementary roles: as an end\-to\-end merchant screening model that processes millions of merchants per day, and as a behavior\-embedding provider for an existing fraud detector\. In the first role, a three\-month shadow evaluation confirms97\.5%97\.5\\%of flagged merchants as risky versus92\.0%92\.0\\%for the production baseline, while the post\-launch appeal rate drops from12%12\\%to∼2%\\sim 2\\%with no exonerations\. In the second role, an online A/B test shows that initializing the fraud detector’s behavior\-token embeddings with those learned by SeqLLM yields gains of 26\.8 pp in Precision@Top\-0\.01% and 33\.1 pp in Recall@Top\-1%—the largest improvement across historical iterations of this online fraud\-detection model\. Third, the methodology generalizes beyond payment behavior to public recommendation benchmarks\. On MovieLens and Amazon, SeqLLM improves Recall@5 over User\-LLM\(Ninget al\.,[2025](https://arxiv.org/html/2608.03063#bib.bib16)\)by up to32%32\\%\. On RecIF, SeqLLM\+RL improves Pass@32 by14\.2%14\.2\\%over OpenOneRec’s full OneRec\-8B pipeline\(Team,[2025a](https://arxiv.org/html/2608.03063#bib.bib22)\)while using4\.8×4\.8\\timesfewer GPU\-days and no separate distillation stage\. To test behavior\-token understanding beyond ranking accuracy, we construct RecProbe, a four\-task evaluation suite; SeqLLM outperforms OneRec\-8B on all four tasks, raising Video–Topic Matching accuracy from0\.4650\.465to0\.7450\.745\.

Our contributions are fourfold:

1. \(1\)We formulate*joint text–behavior modeling*for high\-stakes decisions: a regime where risk emerges only from the interaction between textual evidence and long event streams, and is therefore systematically missed by text\-only or behavior\-only pipelines\. Controlled modality experiments confirm the value of joint modeling \(Table[2](https://arxiv.org/html/2608.03063#S4.T2)\)\.
2. \(2\)We introduce SeqLLM, a general framework that endows a pretrained LLM with native behavioral\-sequence modeling through a compact field\-level vocabulary, a text\-grounded residual projector with a translation\-to\-reasoning curriculum, and prefix\-guided capability injection\.
3. \(3\)We establish prefix\-guided SFT as a scalable alternative to adapt\-then\-repair CPT\. It matches CPT on sequence modeling while preserving language ability, without a separate general\-ability distillation stage\.
4. \(4\)We demonstrate impact at both industrial and public scale\. SeqLLM is fully deployed at WeChat Pay in two production systems for merchant screening and fraud detection\. On public benchmarks, it substantially outperforms User\-LLM and OpenOneRec’s full pipeline in both recommendation accuracy and behavior\-token semantic understanding, while preserving general language ability and using up to4\.8×4\.8\\timesfewer GPU\-days\.

## 2\.Related Work

#### Behavioral sequence modeling\.

Behavioral sequence models learn temporal patterns from interaction histories\(Wanget al\.,[2019](https://arxiv.org/html/2608.03063#bib.bib26)\), from recurrent and Transformer architectures such as GRU4Rec\(Hidasiet al\.,[2016](https://arxiv.org/html/2608.03063#bib.bib7)\), SASRec\(Kang and McAuley,[2018](https://arxiv.org/html/2608.03063#bib.bib9)\), and BERT4Rec\(Sunet al\.,[2019](https://arxiv.org/html/2608.03063#bib.bib21)\)to large generative systems like HSTU\(Zhaiet al\.,[2024](https://arxiv.org/html/2608.03063#bib.bib27)\)and TIGER\(Rajputet al\.,[2023](https://arxiv.org/html/2608.03063#bib.bib18)\)\. These models are strong at behavioral prediction but do not natively support free\-form language reasoning over textual evidence\.

#### LLMs for behavioral sequences\.

Recent work brings behavior into LLMs\(Linet al\.,[2025](https://arxiv.org/html/2608.03063#bib.bib11)\)\. Text\-based systems such as P5\(Genget al\.,[2022](https://arxiv.org/html/2608.03063#bib.bib5)\)serialize interactions as language, providing a natural interface at the cost of long contexts and overloaded word tokens\. Generative recommenders instead represent items with dedicated codes: OneRec\(Denget al\.,[2025](https://arxiv.org/html/2608.03063#bib.bib4)\)unifies retrieval and ranking, OpenOneRec\(Team,[2025a](https://arxiv.org/html/2608.03063#bib.bib22)\)grounds itemic tokens in language, and OneRec\-Think\(Liuet al\.,[2025](https://arxiv.org/html/2608.03063#bib.bib14)\)and OneReason\(Team,[2026](https://arxiv.org/html/2608.03063#bib.bib23)\)develop explicit reasoning over itemic histories\. A complementary approach, User\-LLM\(Ninget al\.,[2025](https://arxiv.org/html/2608.03063#bib.bib16)\), conditions an LLM on a compact user vector injected via cross\-attention\.

#### Positioning SeqLLM

These methods show LLMs can model behavioral histories, but our setting poses a different task: high\-stakes risk control must jointly weigh heterogeneous textual evidence, such as merchant profiles and complaints, against field\-rich transaction events to produce an auditable decision\. OpenOneRec is our closest generative counterpart: it aligns newly added itemic embeddings directly in the expanded embedding table without an explicit modality projector, then acquires sequence modeling through full\-parameter co\-pretraining, with mixed\-domain data and general\-ability distillation repairing the resulting general\-language degradation\. Yet this adapt\-then\-repair pipeline does not eliminate forgetting: even with roughly44M auxiliary language examples for only156156K behavioral sequences plus a separate distillation stage, OpenOneRec still loses general language ability, and the gap widens as the behavioral corpus grows\. SeqLLM instead uses a text\-grounded projector and prefix\-guided SFT to inject sequence capability without a recovery stage, breaking this coupling\. User\-LLM avoids drift only in its frozen\-backbone \(Enc\) variant; its default Full strategy finetunes the backbone, and either way the entire history is compressed into one vector\. We report the matched quantitative comparison in Sections[4\.2](https://arxiv.org/html/2608.03063#S4.SS2)and[4\.5](https://arxiv.org/html/2608.03063#S4.SS5)\.

#### Risk detection\.

Fraud and risk detection has progressed from rules and statistical scoring to learned models over transaction graphs and behavioral sequences\(Phuaet al\.,[2010](https://arxiv.org/html/2608.03063#bib.bib17); Ngaiet al\.,[2011](https://arxiv.org/html/2608.03063#bib.bib15)\), e\.g\., PANTHER\(Liet al\.,[2025](https://arxiv.org/html/2608.03063#bib.bib10)\)pretrains on transaction histories and transfers to downstream risk tasks\. Existing systems nevertheless tend to score textual and behavioral signals in separate pipelines\. SeqLLM instead reasons over both within one model, targeting risks that become conclusive only through their interaction\.

## 3\.Methodology

SeqLLM injects behavioral\-sequence modeling into a pretrained LLM through three components, each addressing one challenge of joint text–behavior modeling: how to represent behavior compactly, how to make the LLM understand it, and how to inject sequence modeling without eroding language ability\. First, a discrete behavior vocabulary encodes each event as compact field\-level tokens \(Section[3\.2](https://arxiv.org/html/2608.03063#S3.SS2)\)\. Second, a lightweight behavior projector aligns these tokens into the LLM’s semantic space via a translation\-then\-reasoning curriculum \(Section[3\.3](https://arxiv.org/html/2608.03063#S3.SS3)\)\. Third, prefix\-guided capability injection acquires sequence modeling through task\-prefixed SFT rather than CPT, preserving the backbone’s language ability \(Section[3\.4](https://arxiv.org/html/2608.03063#S3.SS4)\)\.

### 3\.1\.Problem Formulation

We consider an entityee—a merchant, user, or item—with textual content𝒯e\\mathcal\{T\}\_\{e\}describing what it is and a timestamped behavioral sequence𝒮e=⟨s1,…,sLe⟩\\mathcal\{S\}\_\{e\}=\\langle s\_\{1\},\\ldots,s\_\{L\_\{e\}\}\\rangledescribing how it acts\. Each eventst∈ℰs\_\{t\}\\in\\mathcal\{E\}contains fields such as time, amount, channel, and event status \(Section[3\.2](https://arxiv.org/html/2608.03063#S3.SS2)\)\. Given a natural\-language instructionℐ\\mathcal\{I\}, a single modelΘ\\Thetajointly uses both signals to model the task\-specific output𝐲e\\mathbf\{y\}\_\{e\}as

\(1\)pΘ​\(𝐲e∣ℐ,𝒯e,𝒮e\)\.p\_\{\\Theta\}\(\\mathbf\{y\}\_\{e\}\\mid\\mathcal\{I\},\\mathcal\{T\}\_\{e\},\\mathcal\{S\}\_\{e\}\)\.Depending onℐ\\mathcal\{I\},𝐲e\\mathbf\{y\}\_\{e\}may be a risk decision with a natural\-language rationale or the next eventsLe\+1s\_\{L\_\{e\}\+1\}\. In high\-stakes screening, neither modality may suffice alone\. Figure[1](https://arxiv.org/html/2608.03063#S1.F1)\(a\) illustrates the jade\-shop case: a category mismatch and private\-QR complaint become conclusive only when read with nighttime, cross\-province, high\-value payments and a deleted bill\. SeqLLM must therefore consume long event streams without compromising the LLM’s language and reasoning ability\.

### 3\.2\.Behavioral Vocabulary

The first design choice is how to represent each event\. Instead of writing it out as text, we split the event into its fields—time, amount, event status, and so on—and turn each field value into its own dedicated token\. For example, one payment is encoded as follows:

<Time:Thu\_00h\> <Scene:App pay\> <Channel:Scan QR\> <Amount:200\-500 CNY\> … <Status:Success\>

Each angle\-bracketed unit is one token\. Formally, a fieldfif\_\{i\}takes discretized valuesViV\_\{i\}\(e\.g\., log\-scale amount buckets or an event\-status code\); each valuev∈Viv\\in V\_\{i\}maps to a token⟨fi:v⟩\\langle f\_\{i\}\\\!:\\\!v\\rangle, and the behavioral vocabulary is their union𝒱b=⋃i\{⟨fi:v⟩:v∈Vi\}\\mathcal\{V\}\_\{b\}=\\bigcup\_\{i\}\\\{\\langle f\_\{i\}\\\!:\\\!v\\rangle:v\\in V\_\{i\}\\\}\. An eventsts\_\{t\}is the concatenation of its field tokens and a sequence𝒮e\\mathcal\{S\}\_\{e\}the concatenation of its events; the event spaceℰ\\mathcal\{E\}collects all such field\-token strings\. These tokens occupy a disjoint sub\-range of the embedding table, insulating behavior from the word vocabulary while reusing the same backbone\.

Field\-level factorization offers three benefits\. First, it is compact: an additive vocabulary of∑i\|Vi\|\\sum\_\{i\}\|V\_\{i\}\|entries represents the∏i\|Vi\|\\prod\_\{i\}\|V\_\{i\}\|combinatorial event space using about99field tokens per event on average—each event type encodes its applicable field subset \(Appendix[B\.1](https://arxiv.org/html/2608.03063#A2.SS1)\)—far fewer than serialized text and without reusing word tokens\. Second, it supports compositional generalization: unlike whole\-event IDs, as used by PANTHER\(Liet al\.,[2025](https://arxiv.org/html/2608.03063#bib.bib10)\), events differing in one field share all remaining tokens, so novel combinations remain expressible and decisions can be attributed to individual fields\. Third, the tokens are semantically readable: explicit field–value names enable text\-based initialization and traceable decisions, whereas RQ\-VAE codes carry no intrinsic textual meaning and require a separate quantizer \(Section[3\.3](https://arxiv.org/html/2608.03063#S3.SS3)\)\.

### 3\.3\.Behavior Projector and Sequence\-Language Alignment

The new field tokens enter with no trained embedding\. A model can learn to predict them as a bare sequence without any grounding, but understanding and reasoning over them requires aligning them to the LLM’s semantic space\. OpenOneRec\(Team,[2025a](https://arxiv.org/html/2608.03063#bib.bib22)\), for instance, trains newly added itemic embeddings through item–text alignment before full\-parameter co\-pretraining, without an explicit projection interface\.

SeqLLM instead casts grounding as an interface\-and\-curriculum problem: a lightweight behavior projector supplies each token a dedicated route into the LLM, and a two\-stage translation\-then\-reasoning curriculum first anchors that route in language, then opens it to reasoning \(Stages 1–2 below\)\. Alignment thereby advances from understanding individual tokens to reasoning over them\.

Behavior projector\.Every newly added behavior token reaches the LLM through a shared projector

\(2\)gψ​\(𝐞\)=𝐞\+MLPψ​\(𝐞\),g\_\{\\psi\}\(\\mathbf\{e\}\)=\\mathbf\{e\}\+\\text\{MLP\}\_\{\\psi\}\(\\mathbf\{e\}\),a two\-layer MLP \(Linear–GELU–Linear\) with a residual \(skip\) connection whose final linear layer is zero\-initialized, sogψg\_\{\\psi\}starts as the identity and learns only a small correction to each input embedding𝐞\\mathbf\{e\}\. Applying the same transformation to every behavior token imposes a shared alignment constraint: related field tokens are mapped consistently into the LLM’s semantic space, allowing the backbone to compose them across fields and events\. Directly tuning each token embedding provides no such cross\-token constraint\. This distinction is empirical, not merely architectural\. In a controlled industrial ablation, the w/o\-projector variant learns to translate individual tokens but fails to transfer their meanings to multi\-event reasoning and downstream decisions; the shared projector closes this gap \(Appendix[E](https://arxiv.org/html/2608.03063#A5), Table[15](https://arxiv.org/html/2608.03063#A5.T15)\)\. The same pattern appears on RecIF, where SeqLLM outperforms OpenOneRec’s direct\-alignment pipeline on all four semantic\-understanding and preference\-reasoning probes \(Table[6](https://arxiv.org/html/2608.03063#S4.T6)\)\.

Because our field tokens are readable, we ground each in the text it denotes rather than initializing at random\. LetZvZ\_\{v\}be the backbone tokenizer’s segmentation of tokenvv’s readable text \(e\.g\.,<Amount:200\-500 CNY\>→\\toAmount 200\-500 CNY\); we mean\-pool the corresponding embeddings and rescale the result to the backbone’s embedding statistics,

\(3\)𝐞¯v=1\|Zv\|​∑z∈Zv𝐄​\[z\],𝐞v\(0\)=𝝁\+𝝈⊙𝐞¯v−𝝁^𝝈^,\\bar\{\\mathbf\{e\}\}\_\{v\}=\\frac\{1\}\{\|Z\_\{v\}\|\}\\sum\_\{z\\in Z\_\{v\}\}\\mathbf\{E\}\[z\],\\qquad\\mathbf\{e\}\_\{v\}^\{\(0\)\}=\\boldsymbol\{\\mu\}\+\\boldsymbol\{\\sigma\}\\odot\\frac\{\\bar\{\\mathbf\{e\}\}\_\{v\}\-\\hat\{\\boldsymbol\{\\mu\}\}\}\{\\hat\{\\boldsymbol\{\\sigma\}\}\},where𝐄\\mathbf\{E\}is the backbone embedding table,\(𝝁,𝝈\)\(\\boldsymbol\{\\mu\},\\boldsymbol\{\\sigma\}\)its per\-dimension mean and standard deviation, and\(𝝁^,𝝈^\)\(\\hat\{\\boldsymbol\{\\mu\}\},\\hat\{\\boldsymbol\{\\sigma\}\}\)those of the pooled vectors\{𝐞¯v\}\\\{\\bar\{\\mathbf\{e\}\}\_\{v\}\\\}\. This rescaling step is essential: without it, the pooled vectors fall outside the backbone’s embedding distribution and disrupt the model; rescaling them to match its mean and variance places each token in the same region as its own descriptive words\.

Recall from Section[3\.2](https://arxiv.org/html/2608.03063#S3.SS2)that eventsts\_\{t\}concatenates field tokens; we writeΦ​\(st\)\\Phi\(s\_\{t\}\)for its token string \(Φ​\(𝒮e\)\\Phi\(\\mathcal\{S\}\_\{e\}\)for a multi\-event window\) and train the projector in two stages\.

Stage 1: Translation\.We first optimize a translation objective that reconstructs the natural\-language reading of behavior tokens, training the projectorψ\\psijointly with the LLM\. Each instance pairs an instructionℐ\\mathcal\{I\}, an inputx=Φ​\(st\)x=\\Phi\(s\_\{t\}\)\(or a short window\), and a field\-wise targetyy; an illustrative example from our translation corpus is:

Stage 1 example: translationInstruction:Translate these behavior tokens\.Input:<Time:Sun\_14h\> <Channel:Msg link\> <Status:Success\>Output:Sunday at 14:00; successful message\-link payment\.

The bridge loss is

\(4\)ℒbridge=−∑\(ℐ,x,y\)∈𝒟translog⁡pΘ,ψ​\(y∣ℐ,x\),\\mathcal\{L\}\_\{\\text\{bridge\}\}=\-\\\!\\\!\\\!\\sum\_\{\(\\mathcal\{I\},x,y\)\\in\\mathcal\{D\}\_\{\\text\{trans\}\}\}\\\!\\\!\\\!\\log p\_\{\\Theta,\\psi\}\(y\\mid\\mathcal\{I\},x\),where embeddings of tokens inxxpass throughgψg\_\{\\psi\}before entering the LLM\. This objective anchors each field token to its meaning, grounding the vocabulary before the model is asked to reason over it\.

Stage 2: Reasoning\.We then continue with SFT on multi\-event reasoning queries over real merchant sequences\. Each instance providesℐ\\mathcal\{I\}, a windowx=Φ​\(𝒮e\)x=\\Phi\(\\mathcal\{S\}\_\{e\}\)of tokenized transactions, and a free\-text answer that aggregates or compares fields—customer profiling, anomaly spotting, or risk commentary\. A representative example is:

Stage 2 example: reasoningInstruction:What pattern is shared by these transactions?Input:Tx1: <Time:Tue\_15h\> <Channel:Scan QR\>; Tx2: <Time:Tue\_16h\> <Channel:Scan QR\>Output:Both are Tuesday\-afternoon QR payments\.

Together, translation grounds each new token in the LLM’s semantic space, while reasoning alignment enables the model to interpret and use this new vocabulary across contexts, preparing it for joint text–behavior decisions in Section[3\.4](https://arxiv.org/html/2608.03063#S3.SS4)\. We present the two as stages for clarity; since translation is lightweight, their data can also be mixed into a single pass with equivalent effect\.

### 3\.4\.Prefix\-Guided Capability Injection

Token grounding teaches the model what each behavior token denotes, but not what sequences of such tokens reveal about an entity\. Platform\-specific regularities—such as spending rhythms, amount transitions, channel preferences, and deviations from routine—are largely absent from language pretraining\. Learning them requires modeling event co\-occurrence and temporal evolution\. Future\-event prediction provides this supervision: generating a continuation requires the model to infer regularities from the observed history\. Standard CPT applies next\-token loss at every position in the behavior stream\. Although this learns sequence structure, it also updates the backbone broadly toward behavior\-token prediction, risking interference with pretrained language knowledge\.

Our key idea is to retain future\-event prediction while turning it from a default modeling objective into an instruction\-conditioned capability\. The instruction acts as a task condition: behavior continuation is optimized only when this condition is present, rather than being imposed on every input\. For each sequence, we choose a cutkkat about70%70\\%of its length\. The first part,c=Φ​\(s1,…,sk\)c=\\Phi\(s\_\{1\},\\ldots,s\_\{k\}\), is provided as input under a natural\-language instructionℐ\\mathcal\{I\}, while the remaining events,y=Φ​\(sk\+1,…,sLe\)y=\\Phi\(s\_\{k\+1\},\\ldots,s\_\{L\_\{e\}\}\), form the answer\. Loss is computed only on this answer\. Letb1:N=Φ​\(𝒮e\)b\_\{1:N\}=\\Phi\(\\mathcal\{S\}\_\{e\}\)denote the resulting behavior\-token sequence andmmthe token boundary induced by the event cutoffkk\. Figure[1](https://arxiv.org/html/2608.03063#S1.F1)\(b\) illustrates the construction; at the token level, the two objectives differ in where the loss is applied:

\(5\)ℒCPT\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{CPT\}\}=−∑t=1Nlog⁡pΘ​\(bt∣b<t\),\\displaystyle=\-\\sum\_\{t=1\}^\{N\}\\log p\_\{\\Theta\}\(b\_\{t\}\\mid b\_\{<t\}\),ℒPrefix\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{Prefix\}\}=−∑t=m\+1Nlog⁡pΘ,ψ​\(bt∣ℐ,b<t\)\.\\displaystyle=\-\\sum\_\{t=m\+1\}^\{N\}\\log p\_\{\\Theta,\\psi\}\(b\_\{t\}\\mid\\mathcal\{I\},b\_\{<t\}\)\.
Why this injects sequence ability\.The sequence supervision is retained rather than removed\. The remaining∼30%\\sim 30\\%of events form the response and receive the same autoregressive next\-token loss used by CPT\. Predicting each response token from the preceding history directly trains temporal dependencies, both from the observed prefixs1:ks\_\{1\{:\}k\}to the target suffixsk\+1:Les\_\{k\+1\{:\}L\_\{e\}\}and within the suffix itself\. Prefix masking only removes loss on the observed70%70\\%; it leaves the future\-event targets—and hence the core sequence\-learning signal of CPT— intact\.

Why this mitigates forgetting\.CPT treats behavior prediction as an unconditional objective and applies loss at almost every position, causing the behavior corpus to update the backbone broadly\. Prefix\-guided SFT instead conditions behavior prediction on an explicit instruction and applies supervision only to the response suffix\. This confines the behavior objective to a specific task context, reducing interference with pretrained language capabilities while preserving the future\-prediction signal\. Consistent with this interpretation, prefix\-guided SFT produces smaller and more uniform per\-layer weight changes than CPT \(Figure[2](https://arxiv.org/html/2608.03063#S4.F2)\)\. It matches sample\-aligned CPT on sequence modeling \(HR@100\.8010\.801vs\.0\.8020\.802\) while preserving substantially more language ability \(C\-Eval0\.7830\.783vs\.0\.2710\.271; Table[1](https://arxiv.org/html/2608.03063#S4.T1)\)\.

We optimize the backbone and projector with response\-only loss:

\(6\)ℒinject=−∑\(ℐ,c,y\)∈𝒟injlog⁡pΘ,ψ​\(y∣ℐ,c\),\\mathcal\{L\}\_\{\\text\{inject\}\}=\-\\\!\\\!\\\!\\sum\_\{\(\\mathcal\{I\},c,y\)\\in\\mathcal\{D\}\_\{\\text\{inj\}\}\}\\\!\\\!\\\!\\log p\_\{\\Theta,\\psi\}\(y\\mid\\mathcal\{I\},c\),For sequence examples, all behavior tokens pass throughgψg\_\{\\psi\}\.𝒟inj\\mathcal\{D\}\_\{\\text\{inj\}\}also includes general instruction examples, whose response loss preserves language capabilities during sequence injection, avoiding a separate capability\-recovery stage\.

## 4\.Experiments & Results

We evaluate SeqLLM on large\-scale payment data from WeChat Pay and on public recommendation benchmarks, complemented by evidence from production deployments\. Our experiments address four questions:

- •RQ1: Can SeqLLM enable a pretrained LLM to model behavioral sequences while retaining its general language capabilities?
- •RQ2: Does joint text–behavior modeling improve merchant risk screening over text\-only and behavior\-only modeling?
- •RQ3: Does deploying SeqLLM yield measurable production gains in merchant screening and fraud detection?
- •RQ4: Does SeqLLM generalize beyond payment behavior to diverse recommendation domains, and how does it compare with strong public sequence–language baselines in recommendation performance, semantic understanding, and language retention?

We answerRQ1through a controlled comparison with CPT on WeChat Pay data,RQ2through a controlled modality comparison, andRQ3through two production deployments\. ForRQ4, we compare with User\-LLM on MovieLens and Amazon and with OpenOneRec’s OneRec\-8B on RecIF, including semantic and preference probes\.

### 4\.1\.Experiment Setup

#### Settings\.

We consider three settings:\(i\)behavioral\-sequence modeling on industrial payment data and two downstream deployments at WeChat Pay: merchant screening and fraud detection;\(ii\)next\-item prediction, preference inference, and review generation on MovieLens\-20M and Amazon Reviews; and\(iii\)sequential recommendation and item understanding on the RecIF benchmark, where we compare with the unified sequence–language model OneRec\-8B released by OpenOneRec\(Team,[2025a](https://arxiv.org/html/2608.03063#bib.bib22)\)\.

#### Backbone & compute\.

Unless stated otherwise, SeqLLM uses Qwen3\-8B as its pretrained backbone\. We report training cost in GPU\-days; full hardware and training configurations are provided in Appendix[C\.1](https://arxiv.org/html/2608.03063#A3.SS1)\. Code for all public\-benchmark experiments, including the RecProbe construction scripts, will be open\-sourced\.

#### Datasets\.

We evaluate on industrial data from two WeChat Pay production scenarios and three public recommendation datasets; full statistics and preprocessing are provided in Appendix[B](https://arxiv.org/html/2608.03063#A2)\.

- •WeChat Pay\.Each merchant includes a profile, complaint text, and transaction sequence\. We inject sequence capability using∼\\sim20M unlabeled sequences without risk\-positive merchants, then perform risk SFT on 4\.06M labeled merchants from a preceding multi\-month window\. Evaluation covers a subsequent 30\-day window \(∼\\sim0\.99M merchants/day\); features are cut at scoring time\. After scoring each merchant, we wait 30 days to collect subsequent evidence of risk before assigning its final label\. Histories are capped at1,0001\{,\}000events and inputs at10,00010\{,\}000tokens\.
- •MovieLens\-20M & Amazon Reviews\.We evaluate next\-item prediction, preference classification, and review generation following the User\-LLM protocol\(Ninget al\.,[2025](https://arxiv.org/html/2608.03063#bib.bib16)\); see Appendix[B\.2](https://arxiv.org/html/2608.03063#A2.SS2)\.
- •RecIF\.This OpenOneRec benchmark contains 96M interactions from 160K users and evaluates sequential recommendation and item understanding under its official protocol\(Team,[2025a](https://arxiv.org/html/2608.03063#bib.bib22)\); see Appendix[B\.3](https://arxiv.org/html/2608.03063#A2.SS3)\.

#### Metrics\.

For next\-transaction prediction, HR@10 is the fraction of examples whose ground\-truth next event appears in the top 10, following PANTHER\(Liet al\.,[2025](https://arxiv.org/html/2608.03063#bib.bib10)\)\. Recall@5 analogously tests whether the held\-out next item appears in the top 5\. For merchant risk, we report the risky fraction among the highest\-scoredr%r\\%of merchants \(r∈\{1,0\.1,0\.01\}r\\in\\\{1,0\.1,0\.01\\\}\)\. Following OpenOneRec\(Team,[2025a](https://arxiv.org/html/2608.03063#bib.bib22)\), RecIF reports whether the target appears amongkkgenerated candidates \(Pass@kk\) and the fraction of relevant items retrieved \(Recall@kk\)\. Other task metrics are Accuracy, NDCG@3, and ROUGE; general language ability is measured by MMLU\(Hendryckset al\.,[2020](https://arxiv.org/html/2608.03063#bib.bib6)\), C\-Eval\(Huanget al\.,[2023](https://arxiv.org/html/2608.03063#bib.bib8)\), and AGIEval\(Zhonget al\.,[2024](https://arxiv.org/html/2608.03063#bib.bib29)\)\. Higher is better throughout; implementation details are in Appendix[D](https://arxiv.org/html/2608.03063#A4)\.

### 4\.2\.Sequence Modeling without Catastrophic Forgetting \(RQ1\)

This section compares prefix\-guided SFT with CPT to show that the former injects behavioral\-sequence capability without catastrophic forgetting\. Sample\-aligned CPT matches our behavioral examples and updates, while gradient\-aligned CPT matches our loss\-bearing behavior tokens\. We test both under no behavior\-token alignment and the full alignment stack\. All variants share the backbone, corpora, and optimizer; CPT additionally receives both raw\-text and chat\-format language replay, giving it strictly more language supervision \(Appendix[C\.2](https://arxiv.org/html/2608.03063#A3.SS2)\)\. PANTHER\(Liet al\.,[2025](https://arxiv.org/html/2608.03063#bib.bib10)\)and zero\-shot Qwen3\-8B provide sequence\-only and language\-only references\.

Table 1\.Sequence modeling and language retention for prefix\-guided SFT and matched CPT controls under two alignment settings\. “No alignment stack” disables semantic initialization, the projectorgψg\_\{\\psi\}, and the translation→\\toreasoning curriculum; “full alignment stack” enables all three\. Component\-level ablations: Table[4](https://arxiv.org/html/2608.03063#S4.T4), Table[15](https://arxiv.org/html/2608.03063#A5.T15)\.MethodHR@10C\-EvalMMLUAGIEvalPANTHER\(Liet al\.,[2025](https://arxiv.org/html/2608.03063#bib.bib10)\)\(sequence\-only\)0\.680–––Qwen3\-8B\(Team,[2025b](https://arxiv.org/html/2608.03063#bib.bib24)\)\(zero\-shot\)0\.3120\.7790\.7690\.636Qwen3\-8B \+ CPT\(sample\-aligned\)0\.8020\.2710\.2640\.324Qwen3\-8B \+ CPT\(gradient\-aligned\)0\.7840\.4350\.4670\.379Prefix\-guided SFT\(ours\)0\.8010\.7830\.7450\.640Qwen3\-8B \+ CPT \+ align\(sample\-aligned\)0\.8040\.2930\.2980\.367Qwen3\-8B \+ CPT \+ align\(gradient\-aligned\)0\.7930\.4560\.4890\.377SeqLLM \(ours\)0\.8060\.7890\.7650\.643Table[1](https://arxiv.org/html/2608.03063#S4.T1)shows that SeqLLM matches CPT on sequence modeling \(HR@100\.8060\.806vs\.0\.8040\.804\) while preserving the original language ability \(C\-Eval0\.7890\.789vs\.0\.2930\.293\)\. Two further controls rule out alternative explanations: the gradient\-aligned CPT row matches the cumulative number of loss\-bearing behavior tokens yet still collapses language ability, so the retention advantage is not explained by supervision volume; and the same pattern holds in the no\-alignment setting, so it is not contingent on the alignment stack\. The high\-replay\-ratio OpenOneRec comparison—where OneRec\-8B also uses continual pre\-training—reaches the same conclusion \(Section[4\.5](https://arxiv.org/html/2608.03063#S4.SS5)\)\.

#### Parameter\-level evidence: prefix\-guided SFT better preserves the backbone\.

With the same behavioral and replay corpora, sample\-aligned CPT produces much larger per\-layer changes, especially in the middle Transformer blocks, whereas prefix\-guided SFT produces smaller, more uniform updates \(Figure[2](https://arxiv.org/html/2608.03063#S4.F2)\)\. These smaller updates provide parameter\-level evidence for why prefix\-guided SFT better preserves general language ability\.

![Refer to caption](https://arxiv.org/html/2608.03063v1/gradient1.png)Figure 2\.Per\-layer relative weight change after sample\-aligned CPT and prefix\-guided SFT training from the same Qwen3\-8B backbone\.

### 4\.3\.Multimodal Merchant Risk Modeling \(RQ2\)

To answerRQ2, we compare controlled text\-only, behavior\-only, and joint variants that use the same SeqLLM architecture, training data, and supervision\. This isolates whether text and behavioral sequences provide complementary evidence for merchant risk screening\.

#### Task and downstream adaptation\.

We first train the SeqLLM backbone through sequence–language alignment and prefix\-guided sequence\-capability injection \(Sections[3\.3](https://arxiv.org/html/2608.03063#S3.SS3)and[3\.4](https://arxiv.org/html/2608.03063#S3.SS4)\), then fine\-tune it on the labeled merchant risk\-SFT corpus\. Each example asks the model whether risk control is required, with the answer beginning with either the “control” or “no control” label\. For ranking, rather than using only the generated answer, we score each merchant by the normalized probability of the positive label token at the first response position,s=exp⁡\(ℓ\+\)/\[exp⁡\(ℓ\+\)\+exp⁡\(ℓ−\)\]s=\\exp\(\\ell^\{\+\}\)/\[\\exp\(\\ell^\{\+\}\)\+\\exp\(\\ell^\{\-\}\)\], and report precision among the highest\-scored merchants\. The controlled modality setup is detailed in Appendix[C\.3](https://arxiv.org/html/2608.03063#A3.SS3)\.

Table 2\.Offline merchant\-risk precision under controlled input modalities\.Input modalityP@Top\-1%P@Top\-0\.1%P@Top\-0\.01%Behavior only9\.0%20\.1%70\.0%Text only23\.0%70\.1%88\.0%Text \+ Behavior32\.6%79\.2%97\.0%Table[2](https://arxiv.org/html/2608.03063#S4.T2)isolates the contribution of joint modeling by holding the model and supervision fixed while varying only the input\. Behavior alone misses merchant context, while text alone misses temporal transaction patterns; combining both raises precision substantially at every cutoff, reaching97\.0%97\.0\\%at Top\-0\.01%0\.01\\%\. This confirms that the two modalities provide complementary evidence for merchant risk screening\.

### 4\.4\.Two Production Deployments at WeChat Pay \(RQ3\)

To answerRQ3, we evaluate SeqLLM in two complementary production roles\. The first uses the joint model directly for merchant risk screening; the second uses SeqLLM\-pretrained behavior\-token embeddings to initialize the embedding layer for behavior\-sequence events in a downstream fraud detector\.

#### Deployment I: merchant risk screening\.

SeqLLM is deployed as a 0\.6B–8B cascade: a 0\.6B scanner scores approximately 50 million active merchants daily and retrieves a fixed candidate set, which the 8B model ranks for risk\-control action; the full service runs on 128 GPUs\. The production baseline is a DeepSeek\-based LLM adapted through SFT and RL, using merchant profiles and complaints but no behavioral sequence\.

Before launch, we ran a three\-month matched prospective shadow evaluation of three systems—the baseline, DeepSeek with text\-serialized behavior \(same backbone and recipe, differing only in input\), and SeqLLM’s 8B ranking stage\. Each day, all three ranked the same neutral candidate pool from an upstream pre\-screen independent of all evaluated systems, and returned the same number of top candidates; shadow outputs did not affect review, enforcement, or labels\. A candidate is labeled positive if confirmed within 30 days either by the existing production risk system—an ensemble of expert strategies operated independently of all three evaluated scorers, including the DeepSeek baseline—or through subsequently confirmed user\-reported harm\.

Table 3\.Merchant screening: three\-month shadow evaluation and subsequent production\. All metrics are weekly\-averaged; appeal and exoneration rates are operational indicators collected after the two systems are officially deployed\.System \(input\)Risk precision\(paired shadow;↑\\uparrow\)Appeal ratepost\-launch, among actioned;↓\\downarrowExoneration ratepost\-launch, among appeals;↓\\downarrowDeepSeek baseline \(text only\)92\.0%92\.0\\%12%12\\%8%8\\%DeepSeek \+ serialization83\.0%83\.0\\%––SeqLLM \(text \+ behavior\)97\.5%97\.5\\%∼2%\\sim 2\\%0%0\\%Improvement\+5\.5\+5\.5pp1010pp reduction88pp reduction

As shown in Table[3](https://arxiv.org/html/2608.03063#S4.T3), SeqLLM improves weekly\-averaged 30\-day risk precision from92\.0%92\.0\\%to97\.5%97\.5\\%, outperforming the baseline in every weekly cohort \(p<10−3p<10^\{\-3\}, paired sign test\)\. Text serialization instead reduces precision to83\.0%83\.0\\%: without sequence\-capability injection, serializing up to1,0001\{,\}000events into tens of thousands of low\-density tokens dilutes attention and interferes with the backbone’s text reasoning rather than aiding it\. Thus merely appending behavior does not explain SeqLLM’s gain\. The serialized variant was not deployed\.

After launch, SeqLLM and the baseline were deployed concurrently in production\. In the deployed 0\.6B–8B cascade, the 0\.6B scanner retains91\.24%91\.24\\%of confirmed risky merchants at a0\.4%0\.4\\%screening ratio over a six\-day monitoring window \(Table[14](https://arxiv.org/html/2608.03063#A3.T14)\), so the 8B ranker’s precision gain is realized on top of near\-complete funnel coverage rather than at its expense\. Over the same concurrent window, SeqLLM’s appeal rate is∼2%\\sim 2\\%versus the baseline’s12%12\\%, and its exoneration rate is0%0\\%versus8%8\\%\(zero observed exonerations\)\. Full details are provided in Appendix[C\.3](https://arxiv.org/html/2608.03063#A3.SS3)\.

#### Deployment II: behavior\-token embeddings for a downstream fraud detector\.

The production baseline is a discriminative transaction\-pair model whose behavior embeddings are learned end\-to\-end from fraud labels\. We replace their initialization with SeqLLM’s pretrained behavior embeddings, leaving all other features, model components, and serving unchanged\. In a concurrent three\-month A/B test with users randomly assigned to two equal arms \(billions of transactions per day; 14\-day label maturation\), Precision@Top\-0\.01%/0\.1%0\.01\\%/0\.1\\%increases by26\.8/7\.626\.8/7\.6pp and Recall@Top\-0\.1%/1%0\.1\\%/1\\%by12\.9/33\.112\.9/33\.1pp—the largest improvement achieved across all historical iterations of this production model\. Intuitively, SeqLLM embeddings complement sparse fraud labels with text\-grounded semantics and sequence patterns, providing stronger representations for long\-tail behaviors\.

\+26\.8 pp\+7\.6 pp\+12\.9 pp\+33\.1 ppPrecision@Top\-0\.01%Precision@Top\-0\.1%Recall@Top\-0\.1%Recall@Top\-1%Baseline\(normalized\)SeqLLM init\.Figure 3\.Online A/B gains from SeqLLM\-initialized behavior embeddings\. Blue bars show absolute percentage\-point gains over the baseline; absolute values are withheld for business confidentiality\.

### 4\.5\.Generalization and Comparison with Strong Public Baselines \(RQ4\)

We assess generalization along three dimensions: MovieLens/Amazon test transfer from payment behavior to standard recommendation; RecIF provides a stage\-matched comparison with OpenOneRec; and our constructed RecProbe probes semantic understanding and preference reasoning beyond ranking accuracy\.

Table 4\.Public recommendation and language results\. Recommendation baselines from User\-LLM\(Ninget al\.,[2025](https://arxiv.org/html/2608.03063#bib.bib16)\); language scores reproduced by us\.MethodMovieLens\-20MAmazon ReviewLanguage AbilityNext\-item \(R@5\)Fav\. genre \(Acc\.\)Next\-item \(R@5\)Fav\. category \(Acc\.\)Review \(ROUGE\)MMLUC\-EvalAGIEvalVanilla\-Sequence0\.1520\.3720\.0500\.437––––Textualized0\.1400\.7870\.0420\.88522\.82–––User\-LLM\(baseline\)0\.1540\.7870\.0470\.89026\.380\.2970\.2760\.307SeqLLM0\.174\(\+13\.0%\)0\.973\(\+23\.6%\)0\.062\(\+31\.9%\)0\.987\(\+10\.9%\)28\.62\(\+8\.5%\)0\.7380\.7860\.657SeqLLM w/o SemInit0\.1690\.9550\.0520\.95627\.990\.7370\.7860\.656SeqLLM w/o projector0\.1720\.9340\.0610\.92527\.230\.7350\.7840\.655

#### Cross\-domain transfer on MovieLens and Amazon\.

User\-LLM\(Ninget al\.,[2025](https://arxiv.org/html/2608.03063#bib.bib16)\)is a representative purpose\-built baseline that conditions the LLM on a separate user encoder via cross\-attention with the backbone finetuned jointly\. Table[4](https://arxiv.org/html/2608.03063#S4.T4)shows SeqLLM outperforms User\-LLM and its Vanilla\-Sequence/Textualized controls on all five recommendation and preference tasks—Recall@5 gains of13\.0%13\.0\\%on MovieLens and31\.9%31\.9\\%on Amazon—while retaining substantially higher MMLU/C\-Eval/AGIEval\. The two ablations show complementary benefits: semantic initialization helps next\-item prediction, while the projector helps semantic preference and review generation\.

#### Stage\-matched comparison with OpenOneRec on RecIF

Open OneRec\(Team,[2025a](https://arxiv.org/html/2608.03063#bib.bib22)\)grounds itemic tokens and acquires sequence modeling through continual pretraining\. We match its 156K behavioral sequences and 13M\-caption alignment pool\. Despite using only 100K general\-instruction examples versus OneRec’s∼\\sim28\.6M general\-domain and general\-SFT examples, SeqLLM outperforms OneRec\-8B\-Pretrain on all six recommendation and language metrics at the sequence\-acquisition stage using4\.4×4\.4\\timesfewer GPU\-days \(Table[5](https://arxiv.org/html/2608.03063#S4.T5)\)\.111We use the released OneRec\-8B checkpoint; OneReason\-8B\(Team,[2026](https://arxiv.org/html/2608.03063#bib.bib23)\)was unavailable at evaluation time\.For final adaptation, OneRec adds SFT, general\-ability distillation, and RL, whereas SeqLLM adds only RL; SeqLLM\+RL still matches the full OneRec\-8B pipeline on P@1 and language ability, improves P@32 by14\.2%14\.2\\%, and uses4\.8×4\.8\\timesfewer GPU\-days\. See Appendix[B\.3](https://arxiv.org/html/2608.03063#A2.SS3)and[C\.3](https://arxiv.org/html/2608.03063#A3.SS3)for data construction, training budgets, and GPU\-day estimation\.

Table 5\.RecIF results: sequential recommendation, language ability, and efficiency\. OneRec\-PT/OneRec\-full = Pretrain→\\to\(SFT→\\toDistill→\\to\)RL; SeqLLM/SeqLLM\+RL = Align→\\toSFT\(→\\toRL\)\. CE = C\-Eval; AGI = AGIEval; GPU = GPU\-days\. Gray subscripts: relative change vs\. the OneRec baseline in the same block \(compute\-reduction factor for GPU\-days\)\.MethodP@1P@32R@32MMLUCEAGIGPUOneRec\-PT0\.02040\.15210\.02350\.71120\.73700\.6080460SeqLLM0\.0411\(\+101\.5%\)0\.2398\(\+57\.7%\)0\.0354\(\+50\.6%\)0\.7221\(\+1\.5%\)0\.7667\(\+4\.0%\)0\.6646\(\+9\.3%\)105\(4\.4×4\.4\\timesfewer\)OneRec\-full0\.05420\.21010\.03560\.71760\.74890\.6411∼\\sim994SeqLLM\+RL0\.0540\(−\-0\.4%\)0\.2399\(\+14\.2%\)0\.0360\(\+1\.1%\)0\.7201\(\+0\.3%\)0\.7485\(−\-0\.1%\)0\.6465\(\+0\.8%\)206\(4\.8×4\.8\\timesfewer\)

#### Behavior\-token understanding and preference reasoning\.

We construct four RecProbe tasks covering video semantics and user preferences \(Table[6](https://arxiv.org/html/2608.03063#S4.T6)\)\. SeqLLM substantially outperforms OneRec\-8B on all four—e\.g\., Video–Topic Matching accuracy rises from0\.4650\.465to0\.7450\.745and Video Interest Ranking NDCG@3 from0\.1120\.112to0\.8960\.896\. The three option\-based probes are format\-robust, so OneRec\-8B’s lower accuracies reflect limited semantic grounding rather than formatting failures; the ranking probe additionally exposes an instruction\-following failure of OneRec\-8B \(Appendix[B\.3](https://arxiv.org/html/2608.03063#A2.SS3)\)\. These results indicate that SeqLLM not only grounds new behavior tokens but also flexibly applies them across tasks—a capability we attribute to the behavior projector, which provides a shared semantic interface between the new vocabulary and the LLM\. Our controlled projector ablation on industrial data supports this view: removing the projector preserves single\-token translation but degrades multi\-event reasoning and downstream transfer \(Table[15](https://arxiv.org/html/2608.03063#A5.T15)\), consistent with its role as a compositional interface rather than per\-token storage\.

Table 6\.Semantic understanding and preference reasoning on RecIF\. Gray text: relative improvement over OneRec\-8B\.TaskMetricOneRec\-8BOurs*Video semantics*Video–Topic MatchingAcc0\.46530\.7450\(\+60\.1%\)Audience TargetingAcc0\.47330\.6160\(\+30\.2%\)*User preference*Video Interest RankingNDCG@30\.11210\.8961\(8\.0×8\.0\\times\)Interest–Category ConsistencyAcc0\.49730\.7420\(\+49\.2%\)

Together, SeqLLM outperforms encoder\-based User\-LLM and delivers stronger overall recommendation and token\-understanding performance than OpenOneRec at comparable language ability and substantially lower cost, demonstrating generalization across datasets, tasks, and sequence–language paradigms \(RQ4\)\.

### 4\.6\.Ablation Summary

Our ablations isolate each design choice: the training objective \(prefix\-guided SFT vs\. CPT, Table[1](https://arxiv.org/html/2608.03063#S4.T1)\), the alignment stack as a whole \(on/off, Table[1](https://arxiv.org/html/2608.03063#S4.T1)\) and its components individually—semantic initialization and the projector \(Table[4](https://arxiv.org/html/2608.03063#S4.T4), Table[15](https://arxiv.org/html/2608.03063#A5.T15)\)—the input modality \(text/behavior/joint, Table[2](https://arxiv.org/html/2608.03063#S4.T2)\), and behavior serialization \(none/text/native tokens, Table[3](https://arxiv.org/html/2608.03063#S4.T3)\)\.

## 5\.Conclusion

We presented SeqLLM, a framework that gives pretrained LLMs native behavioral\-sequence modeling while preserving language ability\. Its three components—a compact field\-level vocabulary, a text\-grounded projector, and prefix\-guided capability injection—match CPT on sequence modeling while avoiding catastrophic forgetting \(C\-Eval0\.780\.78vs\.0\.270\.27under CPT\) and the need for a separate recovery stage\. Deployed at WeChat Pay, SeqLLM raises risk precision from92\.0%92\.0\\%to97\.5%97\.5\\%and yields the largest recall gain across historical iterations of the online fraud\-detection model\. On public benchmarks it outperforms User\-LLM and OpenOneRec’s full pipeline at substantially lower training cost, establishing capability injection as a scalable route to unified language and behavioral\-sequence models\.

## References

- Q\. Ai, Y\. Zhang, K\. Bi, X\. Chen, and W\. B\. Croft \(2017\)Learning a hierarchical embedding model for personalized product search\.InProceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7\-11, 2017,N\. Kando, T\. Sakai, H\. Joho, H\. Li, A\. P\. de Vries, and R\. W\. White \(Eds\.\),pp\. 645–654\.External Links:[Link](https://doi.org/10.1145/3077136.3080813),[Document](https://dx.doi.org/10.1145/3077136.3080813)Cited by:[§1](https://arxiv.org/html/2608.03063#S1.p3.1)\.
- J\. Bobadilla, F\. Ortega, A\. Hernando, and A\. Gutiérrez \(2013\)Recommender systems survey\.Knowledge\-based systems46,pp\. 109–132\.Cited by:[§1](https://arxiv.org/html/2608.03063#S1.p3.1)\.
- D\. Caffagni, F\. Cocchi, L\. Barsellotti, N\. Moratelli, S\. Sarto, L\. Baraldi, M\. Cornia, and R\. Cucchiara \(2024\)The revolution of multimodal large language models: a survey\.Findings of the association for computational linguistics: ACL 2024,pp\. 13590–13618\.Cited by:[§1](https://arxiv.org/html/2608.03063#S1.p3.1)\.
- J\. Deng, S\. Wang, K\. Cai, L\. Ren, Q\. Hu, W\. Ding, Q\. Luo, and G\. Zhou \(2025\)Onerec: unifying retrieve and rank with generative recommender and iterative preference alignment\.arXiv preprint arXiv:2502\.18965\.Cited by:[§1](https://arxiv.org/html/2608.03063#S1.p3.1),[§2](https://arxiv.org/html/2608.03063#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Geng, S\. Liu, Z\. Fu, Y\. Ge, and Y\. Zhang \(2022\)Recommendation as language processing \(RLP\): A unified pretrain, personalized prompt & predict paradigm \(P5\)\.InRecSys ’22: Sixteenth ACM Conference on Recommender Systems, Seattle, WA, USA, September 18 \- 23, 2022,J\. Golbeck, F\. M\. Harper, V\. Murdock, M\. D\. Ekstrand, B\. Shapira, J\. Basilico, K\. T\. Lundgaard, and E\. Oldridge \(Eds\.\),pp\. 299–315\.External Links:[Link](https://doi.org/10.1145/3523227.3546767),[Document](https://dx.doi.org/10.1145/3523227.3546767)Cited by:[§2](https://arxiv.org/html/2608.03063#S2.SS0.SSS0.Px2.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2020\)Measuring massive multitask language understanding\.arXiv preprint arXiv:2009\.03300\.Cited by:[§4\.1](https://arxiv.org/html/2608.03063#S4.SS1.SSS0.Px4.p1.5)\.
- B\. Hidasi, A\. Karatzoglou, L\. Baltrunas, and D\. Tikk \(2016\)Session\-based recommendations with recurrent neural networks\.In4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2\-4, 2016, Conference Track Proceedings,Y\. Bengio and Y\. LeCun \(Eds\.\),External Links:[Link](http://arxiv.org/abs/1511.06939)Cited by:[§2](https://arxiv.org/html/2608.03063#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Huang, Y\. Bai, Z\. Zhu, J\. Zhang, J\. Zhang, T\. Su, J\. Liu, C\. Lv, Y\. Zhang, Y\. Fu,et al\.\(2023\)C\-eval: a multi\-level multi\-discipline chinese evaluation suite for foundation models\.Advances in neural information processing systems36,pp\. 62991–63010\.Cited by:[§4\.1](https://arxiv.org/html/2608.03063#S4.SS1.SSS0.Px4.p1.5)\.
- W\. Kang and J\. J\. McAuley \(2018\)Self\-attentive sequential recommendation\.InIEEE International Conference on Data Mining, ICDM 2018, Singapore, November 17\-20, 2018,pp\. 197–206\.External Links:[Link](https://doi.org/10.1109/ICDM.2018.00035),[Document](https://dx.doi.org/10.1109/ICDM.2018.00035)Cited by:[§1](https://arxiv.org/html/2608.03063#S1.p3.1),[§2](https://arxiv.org/html/2608.03063#S2.SS0.SSS0.Px1.p1.1)\.
- G\. Li, Y\. Zhang, X\. Chen, C\. Li, B\. Wang, L\. Kong, W\. Wang, W\. Huang, and M\. H\. Y\. Tan \(2025\)PANTHER: generative pretraining beyond language for sequential user behavior modeling\.CoRRabs/2510\.10102\.External Links:[Link](https://doi.org/10.48550/arXiv.2510.10102),[Document](https://dx.doi.org/10.48550/ARXIV.2510.10102),2510\.10102Cited by:[§C\.3](https://arxiv.org/html/2608.03063#A3.SS3.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.03063#S1.p3.1),[§2](https://arxiv.org/html/2608.03063#S2.SS0.SSS0.Px4.p1.1),[§3\.2](https://arxiv.org/html/2608.03063#S3.SS2.p4.3),[§4\.1](https://arxiv.org/html/2608.03063#S4.SS1.SSS0.Px4.p1.5),[§4\.2](https://arxiv.org/html/2608.03063#S4.SS2.p1.1),[Table 1](https://arxiv.org/html/2608.03063#S4.T1.11.2.1)\.
- J\. Lin, X\. Dai, Y\. Xi, W\. Liu, B\. Chen, H\. Zhang, Y\. Liu, C\. Wu, X\. Li, C\. Zhu,et al\.\(2025\)How can recommender systems benefit from large language models: a survey\.ACM Transactions on Information Systems43\(2\),pp\. 1–47\.Cited by:[§1](https://arxiv.org/html/2608.03063#S1.p3.1),[§2](https://arxiv.org/html/2608.03063#S2.SS0.SSS0.Px2.p1.1)\.
- B\. Liu, C\. Zhu, G\. Li, W\. Zhang, J\. Lai, R\. Tang, X\. He, Z\. Li, and Y\. Yu \(2020\)Autofis: automatic feature interaction selection in factorization models for click\-through rate prediction\.Inproceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining,pp\. 2636–2645\.Cited by:[§1](https://arxiv.org/html/2608.03063#S1.p3.1)\.
- H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee \(2023\)Visual instruction tuning\.Advances in neural information processing systems36,pp\. 34892–34916\.Cited by:[§1](https://arxiv.org/html/2608.03063#S1.p6.1)\.
- Z\. Liu, S\. Wang, X\. Wang, R\. Zhang, J\. Deng, H\. Bao, J\. Zhang, W\. Li, P\. Zheng, X\. Wu, Y\. Hu, Q\. Hu, X\. Luo, L\. Ren, Z\. Zhang, Q\. Wang, K\. Cai, Y\. Wu, H\. Cheng, Z\. Cheng, L\. Ren, H\. Wang, Y\. Su, R\. Tang, K\. Gai, and G\. Zhou \(2025\)OneRec\-think: in\-text reasoning for generative recommendation\.CoRRabs/2510\.11639\.External Links:[Link](https://doi.org/10.48550/arXiv.2510.11639),[Document](https://dx.doi.org/10.48550/ARXIV.2510.11639),2510\.11639Cited by:[§1](https://arxiv.org/html/2608.03063#S1.p3.1),[§2](https://arxiv.org/html/2608.03063#S2.SS0.SSS0.Px2.p1.1)\.
- E\. W\. Ngai, Y\. Hu, Y\. H\. Wong, Y\. Chen, and X\. Sun \(2011\)The application of data mining techniques in financial fraud detection: a classification framework and an academic review of literature\.Decision support systems50\(3\),pp\. 559–569\.Cited by:[§1](https://arxiv.org/html/2608.03063#S1.p1.1),[§2](https://arxiv.org/html/2608.03063#S2.SS0.SSS0.Px4.p1.1)\.
- L\. Ning, L\. Liu, J\. Wu, N\. Wu, D\. Berlowitz, S\. Prakash, B\. Green, S\. O’Banion, and J\. Xie \(2025\)User\-llm: efficient LLM contextualization with user embeddings\.InCompanion Proceedings of the ACM on Web Conference 2025, WWW 2025, Sydney, NSW, Australia, 28 April 2025 \- 2 May 2025,G\. Long, M\. Blumestein, Y\. Chang, L\. Lewin\-Eytan, Z\. H\. Huang, and E\. Yom\-Tov \(Eds\.\),pp\. 1219–1223\.External Links:[Link](https://doi.org/10.1145/3701716.3715463),[Document](https://dx.doi.org/10.1145/3701716.3715463)Cited by:[§C\.3](https://arxiv.org/html/2608.03063#A3.SS3.SSS0.Px5.p1.1),[§1](https://arxiv.org/html/2608.03063#S1.p3.1),[§1](https://arxiv.org/html/2608.03063#S1.p7.13),[§2](https://arxiv.org/html/2608.03063#S2.SS0.SSS0.Px2.p1.1),[2nd item](https://arxiv.org/html/2608.03063#S4.I2.i2.p1.1),[§4\.5](https://arxiv.org/html/2608.03063#S4.SS5.SSS0.Px1.p1.2),[Table 4](https://arxiv.org/html/2608.03063#S4.T4)\.
- C\. Phua, V\. Lee, K\. Smith, and R\. Gayler \(2010\)A comprehensive survey of data mining\-based fraud detection research\.arXiv preprint arXiv:1009\.6119\.Cited by:[§1](https://arxiv.org/html/2608.03063#S1.p1.1),[§2](https://arxiv.org/html/2608.03063#S2.SS0.SSS0.Px4.p1.1)\.
- S\. Rajput, N\. Mehta, A\. Singh, R\. H\. Keshavan, T\. Vu, L\. Heldt, L\. Hong, Y\. Tay, V\. Q\. Tran, J\. Samost, M\. Kula, E\. H\. Chi, and M\. Sathiamoorthy \(2023\)Recommender systems with generative retrieval\.InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 \- 16, 2023,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2023/hash/20dcab0f14046a5c6b02b61da9f13229-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2608.03063#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Rendle \(2010\)Factorization machines\.In2010 IEEE International conference on data mining,pp\. 995–1000\.Cited by:[§1](https://arxiv.org/html/2608.03063#S1.p3.1)\.
- F\. Sarvi, N\. Voskarides, L\. Mooiman, S\. Schelter, and M\. de Rijke \(2020\)A comparison of supervised learning to match methods for product search\.arXiv preprint arXiv:2007\.10296\.Cited by:[§1](https://arxiv.org/html/2608.03063#S1.p3.1)\.
- F\. Sun, J\. Liu, J\. Wu, C\. Pei, X\. Lin, W\. Ou, and P\. Jiang \(2019\)BERT4Rec: sequential recommendation with bidirectional encoder representations from transformer\.InProceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM 2019, Beijing, China, November 3\-7, 2019,W\. Zhu, D\. Tao, X\. Cheng, P\. Cui, E\. A\. Rundensteiner, D\. Carmel, Q\. He, and J\. X\. Yu \(Eds\.\),pp\. 1441–1450\.External Links:[Link](https://doi.org/10.1145/3357384.3357895),[Document](https://dx.doi.org/10.1145/3357384.3357895)Cited by:[§2](https://arxiv.org/html/2608.03063#S2.SS0.SSS0.Px1.p1.1)\.
- O\. Team \(2025a\)OpenOneRec technical report\.CoRRabs/2512\.24762\.External Links:[Link](https://doi.org/10.48550/arXiv.2512.24762),[Document](https://dx.doi.org/10.48550/ARXIV.2512.24762),2512\.24762Cited by:[§C\.3](https://arxiv.org/html/2608.03063#A3.SS3.SSS0.Px6.p1.8),[§C\.3](https://arxiv.org/html/2608.03063#A3.SS3.SSS0.Px6.p2.8),[Appendix D](https://arxiv.org/html/2608.03063#A4.SS0.SSS0.Px5.p1.3),[§1](https://arxiv.org/html/2608.03063#S1.p5.4),[§1](https://arxiv.org/html/2608.03063#S1.p7.13),[§2](https://arxiv.org/html/2608.03063#S2.SS0.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2608.03063#S3.SS3.p1.1),[3rd item](https://arxiv.org/html/2608.03063#S4.I2.i3.p1.1),[§4\.1](https://arxiv.org/html/2608.03063#S4.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2608.03063#S4.SS1.SSS0.Px4.p1.5),[§4\.5](https://arxiv.org/html/2608.03063#S4.SS5.SSS0.Px2.p1.4)\.
- O\. Team \(2026\)OneReason technical report\.CoRRabs/2606\.06260\.External Links:[Link](https://doi.org/10.48550/arXiv.2606.06260),[Document](https://dx.doi.org/10.48550/ARXIV.2606.06260),2606\.06260Cited by:[§1](https://arxiv.org/html/2608.03063#S1.p3.1),[§2](https://arxiv.org/html/2608.03063#S2.SS0.SSS0.Px2.p1.1),[footnote 1](https://arxiv.org/html/2608.03063#footnote1)\.
- Q\. Team \(2025b\)Qwen3 technical report\.CoRRabs/2505\.09388\.External Links:[Link](https://doi.org/10.48550/arXiv.2505.09388),[Document](https://dx.doi.org/10.48550/ARXIV.2505.09388),2505\.09388Cited by:[§C\.3](https://arxiv.org/html/2608.03063#A3.SS3.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2608.03063#S4.T1.11.3.1)\.
- C\. Van Gysel, M\. de Rijke, and E\. Kanoulas \(2016\)Learning latent vector spaces for product search\.InProceedings of the 25th ACM international on conference on information and knowledge management,pp\. 165–174\.Cited by:[§1](https://arxiv.org/html/2608.03063#S1.p3.1)\.
- S\. Wang, L\. Hu, Y\. Wang, L\. Cao, Q\. Z\. Sheng, and M\. Orgun \(2019\)Sequential recommender systems: challenges, progress and prospects\.arXiv preprint arXiv:2001\.04830\.Cited by:[§1](https://arxiv.org/html/2608.03063#S1.p3.1),[§2](https://arxiv.org/html/2608.03063#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Zhai, L\. Liao, X\. Liu, Y\. Wang, R\. Li, X\. Cao, L\. Gao, Z\. Gong, F\. Gu, J\. He, Y\. Lu, and Y\. Shi \(2024\)Actions speak louder than words: trillion\-parameter sequential transducers for generative recommendations\.InForty\-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21\-27, 2024,R\. Salakhutdinov, Z\. Kolter, K\. A\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research,pp\. 58484–58509\.External Links:[Link](https://proceedings.mlr.press/v235/zhai24a.html)Cited by:[§2](https://arxiv.org/html/2608.03063#S2.SS0.SSS0.Px1.p1.1)\.
- D\. Zhang, Y\. Yu, J\. Dong, C\. Li, D\. Su, C\. Chu, and D\. Yu \(2024\)Mm\-llms: recent advances in multimodal large language models\.Findings of the Association for Computational Linguistics: ACL 2024,pp\. 12401–12430\.Cited by:[§1](https://arxiv.org/html/2608.03063#S1.p3.1)\.
- W\. Zhong, R\. Cui, Y\. Guo, Y\. Liang, S\. Lu, Y\. Wang, A\. Saied, W\. Chen, and N\. Duan \(2024\)Agieval: a human\-centric benchmark for evaluating foundation models\.InFindings of the association for computational linguistics: NAACL 2024,pp\. 2299–2314\.Cited by:[§4\.1](https://arxiv.org/html/2608.03063#S4.SS1.SSS0.Px4.p1.5)\.

## Appendix AReproducibility Overview

This appendix provides everything needed to reproduce every result in the main text, organized by the reproduction workflow: data acquisition and preprocessing \(Appendix[B](https://arxiv.org/html/2608.03063#A2)\), model configuration and training with all baselines \(Appendix[C](https://arxiv.org/html/2608.03063#A3)\), and per\-number computation from a trained checkpoint \(Appendix[D](https://arxiv.org/html/2608.03063#A4)\)\. To avoid duplication, we do not restate the method \(Section[3](https://arxiv.org/html/2608.03063#S3)\) or the metric definitions \(Section[4\.1](https://arxiv.org/html/2608.03063#S4.SS1)\), reporting only the configurations, statistics, and protocols required for reproduction\.

Table 7\.Mapping from main results to the relevant parts of this appendix\. Training configurations \(Appendix[C\.1](https://arxiv.org/html/2608.03063#A3.SS1)\) and evaluation protocols \(Appendix[D](https://arxiv.org/html/2608.03063#A4)\) are shared across all rows\.ResultDatasetData / baseline setupTab\.[1](https://arxiv.org/html/2608.03063#S4.T1)WeChat Pay§[B\.1](https://arxiv.org/html/2608.03063#A2.SS1), §[C\.2](https://arxiv.org/html/2608.03063#A3.SS2), §[C\.3](https://arxiv.org/html/2608.03063#A3.SS3)Tab\.[2](https://arxiv.org/html/2608.03063#S4.T2)WeChat Pay§[B\.1](https://arxiv.org/html/2608.03063#A2.SS1), §[C\.3](https://arxiv.org/html/2608.03063#A3.SS3)Tab\.[3](https://arxiv.org/html/2608.03063#S4.T3)WeChat Pay§[C\.3](https://arxiv.org/html/2608.03063#A3.SS3)Tab\.[13](https://arxiv.org/html/2608.03063#A3.T13)WeChat Pay§[C\.3](https://arxiv.org/html/2608.03063#A3.SS3)Tab\.[14](https://arxiv.org/html/2608.03063#A3.T14)WeChat Pay§[C\.3](https://arxiv.org/html/2608.03063#A3.SS3)Tab\.[4](https://arxiv.org/html/2608.03063#S4.T4)MovieLens/Amazon§[B\.2](https://arxiv.org/html/2608.03063#A2.SS2), §[C\.3](https://arxiv.org/html/2608.03063#A3.SS3)Tab\.[5](https://arxiv.org/html/2608.03063#S4.T5)RecIF§[B\.3](https://arxiv.org/html/2608.03063#A2.SS3), §[C\.3](https://arxiv.org/html/2608.03063#A3.SS3)Tab\.[6](https://arxiv.org/html/2608.03063#S4.T6)RecIF \(RecProbe\)§[B\.3](https://arxiv.org/html/2608.03063#A2.SS3)Fig\.[2](https://arxiv.org/html/2608.03063#S4.F2)WeChat Pay§[C\.2](https://arxiv.org/html/2608.03063#A3.SS2)

## Appendix BData and Preprocessing

### B\.1\.WeChat Pay Dataset

#### Additional sample\-construction details\.

All records are anonymized and stripped of personally identifiable information\. For offline evaluation, all merchants appearing in the risk\-SFT corpus are excluded from the held\-out test cohorts, making the split entity\-disjoint for both positive and negative labels\. Transactions are sorted chronologically before sequence truncation and tokenization\. Table[8](https://arxiv.org/html/2608.03063#A2.T8)consolidates the corpus and stage\-level statistics reported in the main text\.

#### Field\-level vocabulary\.

To respect confidentiality, individual field names are not disclosed\. The full schema spans2828fields \(Table[8](https://arxiv.org/html/2608.03063#A2.T8)\), but not every field applies to every event: each event type is encoded with its own applicable field subset, so an event carries about99field tokens on average\. The behavioral vocabulary \(1,5331\{,\}533entries\) is the union of all field–value tokens across the2828fields\.

Table 8\.WeChat Pay corpus statistics\. Mean sequence length is computed after the1,0001\{,\}000\-event cap\.TrainValidationTestTime spanPreceding multi\-month window–Subsequent 30\-day windowRisk\-SFT entities \(merchants\)4\.064\.06M–0\.990\.99M/day×\\times30Unlabeled source merchants2020M active merchantsMean / max behavior events862 /1,0001\{,\}000–845 /1,0001\{,\}000Maximum input tokens10,00010\{,\}000Behavior fields / vocab\. size2828/1,5331\{,\}533*Stage sample counts*Alignment117117k \(5050k \+1919k×3\\times 3\+1010k\)Capability injection∼20\\sim 20M seq\. \+∼1\.1\\sim 1\.1M textRisk SFT4\.064\.06M–0\.990\.99M/day×\\times30

### B\.2\.MovieLens\-20M and Amazon Reviews

#### Data sources and filtering\.

MovieLens\-20M is the GroupLens 20M\-rating release; for Amazon Reviews we use the*Movies and TV*5\-core subset\. Following User\-LLM, we retain the users and items that meet its interaction\-count thresholds, sort each user’s interactions by timestamp, and represent an interaction by its metadata: movie name, genre, and rating for MovieLens, and product title, category, rating, and review summary for Amazon\. Table[9](https://arxiv.org/html/2608.03063#A2.T9)reports the resulting user, item, and interaction counts\.

Table 9\.Statistics after the User\-LLM preprocessing protocol\. The number of test examples equals the number of users\.DatasetUsersItemsTrainTestMovieLens\-20M82,97727,28013,821,40582,977Amazon Reviews5,756177,978357,2585,756
#### Preprocessing

A sliding window \(length5050\) over each user’s chronological history yields supervised examples, where the events preceding a position form the history and the event at that position is the label\. The most recent interaction of every user is held out for testing and never seen during training, so the test set has exactly one example per user \(82,97782\{,\}977for MovieLens,5,7565\{,\}756for Amazon\), and the*Train*column of Table[9](https://arxiv.org/html/2608.03063#A2.T9)counts the windowed next\-item pairs\. Each distinct movie or product is assigned a dedicated item token, whose readable description \(name and genre for MovieLens, title and category for Amazon\) is used for sequence–language alignment \(Section[3\.3](https://arxiv.org/html/2608.03063#S3.SS3)\)\. At downstream time, a user history is the chronological sequence of these item tokens, and the three tasks differ only in the task prefix and target\.

#### Next\-item prediction\.

In the prefix\-guided capability injection stage, we split each sliding window of length 50 at a ratio of 35:15\. Specifically, the first 35 items are taken as the conditional input to generate the subsequent 15 items for multi\-item sequence continuation, instead of merely scoring individual candidate items\. This operation is applied to both the training and evaluation sets\. The model takes the item\-token interaction history as input and is optimized to predict the next item token\.

#### Favorite genre / category\.

The label is the genre \(MovieLens\) or product category \(Amazon\) that occurs most often across the user’s history, with ties broken by the most recent occurrence\. The input is the same item\-token history and the target is the dominant genre/category, so the task probes long\-range preference rather than the last interaction\. Each user yields one instance per split, scored by accuracy\.

Favorite\-genre example \(MovieLens\)Instruction:Based on the user’s history, what is their favorite genre?Input:<movie\_924\> <genre\_103\> <ml\_rating\_6\> <movie\_919\> <genre\_210\> <ml\_rating\_6\>⋯\\cdots<movie\_3030\> <genre\_131\> <ml\_rating\_5\>Output:<genre\_171\> \(Horror\|\|Mystery\|\|Thriller\)

#### Review generation\.

Defined for Amazon only, as MovieLens has no review text: conditioned on the item\-token history and the target product \(given by name and rating in the instruction\), the model generates the user’s review, and the held\-out review of the test item is the ROUGE reference; empty reviews are discarded\.

Review\-generation example \(Amazon\)Instruction:Please write a review for*The Palace of Versailles*\. Your rating is 3\.Input:<item\_461\> <category\_1\> <az\_rating\_4\> <item\_2649\> <category\_25\> <az\_rating\_4\>⋯\\cdots<item\_51101\> <category\_9\> <az\_rating\_3\>Output:Well done\! Biltz was the original person to put …

### B\.3\.RecIF

#### Data preparation\.

We obtain the data from two official OpenOneRec Hugging Face repositories: the RecIF dataset222[https://huggingface\.co/datasets/OpenOneRec/OpenOneRec\-RecIF](https://huggingface.co/datasets/OpenOneRec/OpenOneRec-RecIF)and the general\-purpose SFT dataset\.333[https://huggingface\.co/datasets/OpenOneRec/OpenOneRec\-General\-SFT](https://huggingface.co/datasets/OpenOneRec/OpenOneRec-General-SFT)Following the official preprocessing scripts,444[https://github\.com/Kuaishou\-OneRec/OpenOneRec](https://github.com/Kuaishou-OneRec/OpenOneRec)we build our splits while faithfully retaining the original semantic IDs \(SIDs\), their associated textual descriptions, and the standard train/test partition\.*Alignment data*pair each SID with its textual caption to teach the projector to map video tokens into language, contain 13MM\+ item\-understanding data\.*Capability\-injection data*come from the Video\-Rec behavioral sequences \(96M interactions from 160K users\): to increase sequence supervision, every training example with a multi\-item output is expanded position\-wise—each output position is used in turn as the target of a separate instance while retaining the associated user\-history context \(RL variant is optimized on the original, unexpanded examples\)\.*Evaluation data*follow the official held\-out test split\. We additionally include 100000 general\-language examples to preserve language ability and 5MM\+ item\-understanding examples derived from RecIF captions and SIDs\.

#### RecProbe: item\-understanding probing tasks\.

RecProbe is constructed in this work from the public RecIF item captions and semantic IDs\. It contains four probing tasks that evaluate video\-token understanding beyond next\-item prediction\. Each task shares the system prompt below and is scored as reported in Table[6](https://arxiv.org/html/2608.03063#S4.T6): Video–Topic Matching, Audience Targeting, and Interest–Category Consistency use accuracy, and Video Interest Ranking uses NDCG@3\. The prompts are shown in English for presentation; semantic ID token strings \(<\|sid\_begin\|\>…\\ldots<\|sid\_end\|\>\) are verbatim\.

We observe that OneRec\-8B follows instructions poorly on this task*only*: rather than producing the requested ranking, it merely re\-emits the candidate SIDs and loops on them, which directly accounts for its low NDCG@3 here\. Its score on this probe therefore reflects grounding and instruction following jointly, and we scope our claims accordingly \(Section[4\.5](https://arxiv.org/html/2608.03063#S4.SS5)\)\. On the three option\-based probes \(Video–Topic Matching, Audience Targeting, and Interest–Category Consistency\), OneRec\-8B consistently emits a valid option under the same answer\-extraction protocol applied to both models, so its lower accuracies on those tasks measure semantic understanding rather than output formatting\.

RecProbe tasks \(shared system prompt and one example each\)System:You are a video semantic\-understanding expert who can infer video content from video tokens\.Video–Topic Matching \(acc\.\):Given videos A\-\-D \(<\|sid\_begin\|\>…\\ldots<\|sid\_end\|\>\), which one is most relevant to the topic ‘‘pets / cute animals’’? Answer with the option only\.Audience Targeting \(acc\.\):Given an ad video, which audience is it best suited for?*\(A\)*humorous chat screenshots / intimate\-relationship interactions;*\(B\)*baking and dessert making;*\(C\)*cartoon characters and warm emotion;*\(D\)*game achievements and phone\-screen demos\. Answer with the option only\.Video Interest Ranking \(NDCG@3\):Given the primary interest ‘‘gaming / esports’’ and recent views, rank candidate videos A\-\-C by recommendation priority \(high to low\)\.Interest–Category Consistency \(acc\.\):Given a behavior sequence and a candidate video, does the candidate match the sequence’s dominant interest?*\(A\)*Matches*\(B\)*Does not match\. Answer with the option only\.

Video Interest Ranking: OneRec\-8B vs\. SeqLLM \(same input as above\)OneRec\-8B:<\|sid\_begin\|\><s\_a\_5719\><s\_b\_513\><s\_c\_7881\><\|sid\_end\|\> <\|sid\_begin\|\><s\_a\_5719\><s\_b\_2395\><s\_c\_3488\><\|sid\_end\|\> <\|sid\_begin\|\><s\_a\_2776\><s\_b\_554\><s\_c\_8066\><\|sid\_end\|\>⋯\\cdots\(re\-emits candidate SIDs and keeps looping instead of ranking\)SeqLLM:‘‘Answer: Video A \> Video B \> Video C’’

## Appendix CModel, Training, and Baselines

The SeqLLM components are defined in Section[3](https://arxiv.org/html/2608.03063#S3); this section adds only reproduction\-level configuration and the exact setup of every baseline and ablation\. All settings share the stage order semantic initialization→\\rightarrowtranslation alignment→\\rightarrowreasoning alignment→\\rightarrowprefix\-guided injection→\\rightarrowdownstream tuning, with the projector and LLM jointly updated unless a baseline requires otherwise\.

### C\.1\.Training Configurations

Table[12](https://arxiv.org/html/2608.03063#A3.T12)reports the stage\-wise configurations for all three settings\. Behavior\-token initialization follows Section[3\.3](https://arxiv.org/html/2608.03063#S3.SS3); tokens whose readable text cannot be segmented fall back to random initialization\.

### C\.2\.Controlled CPT vs\. Prefix\-Guided SFT Protocol

The RQ1 comparison \(Table[1](https://arxiv.org/html/2608.03063#S4.T1), Fig\.[2](https://arxiv.org/html/2608.03063#S4.F2)\) controls every factor except the training objective\. Both runs start from the identical Qwen3\-8B checkpoint, share the same corpora, token budget, and optimizer schedule, and deliberately omit the projector and alignment curriculum\. The sole difference is the objective: CPT uses full\-token next\-token prediction on the packed behavior stream, while prefix\-guided SFT computes loss only on the assistant span under a task prefix\. Both CPT controls receive language replay in both raw\-text and chat\-format renderings \(strictly more language supervision than prefix\-guided SFT\)\. Training uses 50k steps, 96 GPUs, DeepSpeed ZeRO\-3, and BF16 \(Table[10](https://arxiv.org/html/2608.03063#A3.T10)\)\.

Table 10\.Training configuration for CPT vs\. prefix\-guided SFT\. CPT uses same configuration with stage=ptand full\-token loss\.SettingValueBackboneQwen3\-8B \(full FT\)General corporaAlpaca\-GPT4\-ZH \+ Firefly \+ identityPacking / cutofftrue /10,00010\{,\}000Batch size2\*2\*8\*12Steps50​k50kPeak LR2\.0×10−52\.0\\times 10^\{\-5\}LR schedulecosine, warmup=500=500stepsPrecision / parallelismBF16 / DeepSpeed ZeRO\-3CPT objectiveNext\-token LM loss on packed streamPrefix\-guided SFTAssistant\-token loss onlyTable 11\.Mechanistic diagnostics of the two objectives, each measured against the shared Qwen3\-8B base \(WeChat Pay\)\. Lower weight change / KL and higher similarity to base indicate less disruption of the pretrained model\.Diagnostic \(vs\. base\)CPTPrefix\-guided SFTBackbone mean∥Δ​W∥F\\lVert\\Delta W\\rVert\_\{F\}\(rel\.\)0\.0420\.0420\.022\\mathbf\{0\.022\}Stable rank ofΔ​W\\Delta W46\.046\.026\.6\\mathbf\{26\.6\}Next\-token KL from base0\.840\.840\.21\\mathbf\{0\.21\}Last\-layer CKA to base0\.910\.910\.98\\mathbf\{0\.98\}Task\-vector cosinecos⁡\(CPT,Prefix\-guided SFT\)\\cos\(\\text\{CPT\},\\text\{Prefix\-guided SFT\}\)0\.140\.14Next\-transaction HR@100\.8020\.8020\.8010\.801Table 12\.Stage\-wise training configurations across WeChat Pay, MovieLens\-20M/Amazon Reviews, and RecIF\. T\-Align = Translation Alignment; R\-Align = Reasoning Alignment; PGCI = P\-G Capability Injection; RC\-SFT = Risk\-Control SFT; TS\-SFT = Task\-Specific SFT; TIA = Text–Item Alignment; RL = Reinforcement Learning\. Global batch sizes are written as per\-device batch×\\timesgradient\-accumulation steps×\\timesGPUs per node×\\timesnodes; the RL row reports per\-device batch×\\timesnodes\.SettingStageTrainable modulesLRGlobal batchMax lengthSteps/EpochsWeight decayWeChat PayT\-Alignprojector \+ LLM backbone1×10−51\\times 10^\{\-5\}4\*8\*8\*1220482 epochs0\.01R\-Align1×10−51\\times 10^\{\-5\}4\*8\*8\*1220482 epochs0\.01PGCI2×10−52\\times 10^\{\-5\}2\*2\*8\*121000050k steps0\.01RC\-SFT5×10−65\\times 10^\{\-6\}2\*2\*8\*12100002 epochs0\.01MovieLens\-20M& Amazon ReviewsT\-Alignprojector \+ LLM backbone1×10−51\\times 10^\{\-5\}8\*16\*8\*1210242 epochs0\.01R\-Align1×10−51\\times 10^\{\-5\}8\*16\*8\*1210242 epochs0\.01PGCI2×10−52\\times 10^\{\-5\}8\*16\*8\*1210245k steps0\.01TS\-SFT5×10−65\\times 10^\{\-6\}8\*16\*8\*1210242k steps0\.01RecIF\(OpenOneRec\)TIAprojector \+ LLM backbone1×10−51\\times 10^\{\-5\}8\*16\*8\*1210241 epoch0\.01PGCI2×10−52\\times 10^\{\-5\}2\*8\*8\*1230722 epochs0\.01RL1×10−61\\times 10^\{\-6\}64\*1281921 epoch\-

### C\.3\.Baselines

Unless noted otherwise, every baseline shares SeqLLM’s datasets, splits, tokenization, and metrics \(Section[4\.1](https://arxiv.org/html/2608.03063#S4.SS1)\); only the architecture, input modality, or training objective changes, so each comparison isolates a single factor\. We group baselines by experiment\.

#### Sequence modeling on WeChat Pay \(RQ1, Table[1](https://arxiv.org/html/2608.03063#S4.T1)\)\.

PANTHER\(Liet al\.,[2025](https://arxiv.org/html/2608.03063#bib.bib10)\)is the sequence\-only reference \(a behavioral pretraining transformer that outperforms SASRec/HSTU\)\. We retrain it from scratch on the WeChat Pay corpus with whole\-event IDs, matching Qwen3\-8B in parameter count and Transformer configuration and using the capability\-injection optimizer, schedule, and token budget \(Table[12](https://arxiv.org/html/2608.03063#A3.T12)\); it has no language head, so its language cells are dashes\.Qwen3\-8B \(Text, zero\-shot\)\(Team,[2025b](https://arxiv.org/html/2608.03063#bib.bib24)\)serializes each transaction as text \(Section[3\.1](https://arxiv.org/html/2608.03063#S3.SS1)\) and predicts the next one; we map its free\-form output back to the field vocabulary \(Section[3\.2](https://arxiv.org/html/2608.03063#S3.SS2)\) before scoring HR@10\.Qwen3\-8B \+ CPT \(naive\)is the primary matched control \(same corpus, language replay, token budget, and updates as SeqLLM; only the objective differs\)\.

#### Offline comparison of deployment candidates \(RQ3, Table[13](https://arxiv.org/html/2608.03063#A3.T13)\)\.

Both DeepSeek\-based candidates are fine\-tuned on the same downstream merchant\-risk data with the same SFT–RL recipe, optimization setup, and risk\-scoring method\. They differ only in sequence input: the text\-only variant receives the merchant profile and complaint text, while the textualized\-sequence variant additionally receives the transaction sequence serialized as plain text\. P@r%r\\%is the risky fraction among ther%r\\%highest\-scored merchants, and latency is normalized to SeqLLM under the same inference stack\. Relative changes in Table[13](https://arxiv.org/html/2608.03063#A3.T13)use the textualized\-sequence DeepSeek variant as the reference for each cutoff\.

Table 13\.Complete offline comparison of merchant\-ranking deployment candidates\. The textualized\-sequence DeepSeek model was not deployed\.ModelMax tokensLatencyP@0\.01%P@0\.1%P@1%SeqLLM \(Ours\)10,00010\{,\}0001×1\\times0\.970\(\+17\.7%\)0\.792\(\+28\.6%\)0\.326\(\+141\.5%\)DeepSeek\-based \(text only, SFT\+RL\)40,96040\{,\}96020×20\\times0\.930\(\+12\.9%\)0\.556\(−9\.7%\-9\.7\\%\)0\.231\(\+71\.1%\)DeepSeek\-based \(text \+ serialized sequence, SFT\+RL\)40,96040\{,\}96020×20\\times0\.824\(reference\)0\.616\(reference\)0\.135\(reference\)

SeqLLM achieves the best precision at all three ranking cutoffs while requiring one twentieth of the inference time\. The textualized\-sequence DeepSeek model degraded performance relative to the text\-only DeepSeek model and was therefore not deployed\.

#### Production merchant risk screening \(RQ3, Table[3](https://arxiv.org/html/2608.03063#S4.T3)\)\.

The 0\.6B and 8B cascade stages share the identical SeqLLM pipeline, differing only in backbone size \(Qwen3\-0\.6B vs\. Qwen3\-8B; Table[12](https://arxiv.org/html/2608.03063#A3.T12)\)\. The shadow precision evaluates the 8B ranking stage alone on a shared upstream candidate pool; the 0\.6B scanner was introduced for deployment and is evaluated separately below\. The three\-month shadow evaluation, labeling protocol, and appeal/exoneration definitions are described in Section[4\.4](https://arxiv.org/html/2608.03063#S4.SS4); here we note only that exact candidate, action, and appeal counts and traffic composition are withheld for business confidentiality\. The serialized DeepSeek variant was not deployed\.

#### Recall\-side monitoring of the 0\.6B scanner \(Table[14](https://arxiv.org/html/2608.03063#A3.T14)\)\.

After deployment, we monitored the 0\.6B scanner over a six\-day window, scoring all∼\\sim50M active merchants daily\. Daily true positives are confirmed through the same 30\-day maturation pipeline as the shadow labels \(Section[4\.4](https://arxiv.org/html/2608.03063#S4.SS4)\)\. Table[14](https://arxiv.org/html/2608.03063#A3.T14)reports Recall@Top\-NNat three screening depths\. Retaining only the top0\.4%0\.4\\%\(∼\\sim200K candidates\) covers91\.24%91\.24\\%of confirmed risky merchants on average\. This recall result evaluates the deployed scanner separately from the 8B\-only shadow precision reported in Table[3](https://arxiv.org/html/2608.03063#S4.T3)\.

Table 14\.Daily recall of the 0\.6B scanner over a six\-day window\. The scanner scores∼\\sim50M merchants per day; true positives are confirmed within a 30\-day maturation window\. R@Top\-NNis the fraction of confirmed risky merchants captured among theNNhighest\-scored merchants\.DayTrue positivesR@Top\-5wR@Top\-10wR@Top\-20w\(0\.1%\)\(0\.2%\)\(0\.4%\)110,1500\.77780\.85600\.9126210,0460\.77540\.85620\.912339,6110\.77560\.85150\.911949,4750\.77470\.85020\.912759,3550\.77460\.85170\.912369,2790\.77250\.85100\.9123Avg\.9,6530\.77510\.85280\.9124
#### Public recommendation on MovieLens/Amazon \(RQ4, Table[4](https://arxiv.org/html/2608.03063#S4.T4)\)\.

User\-LLM\(Ninget al\.,[2025](https://arxiv.org/html/2608.03063#bib.bib16)\)is the main encoder\-based competitor\. We reproduce its default*Full*training strategy—the configuration used for all experiments reported in the original paper: a Transformer user encoder produces user embeddings that are projected to the LLM hidden size and injected via cross\-attention at intermediate layers, with the Qwen3\-8B backbone finetuned jointly with the encoder, projection, and cross\-attention modules\. Its lower language scores in Table[4](https://arxiv.org/html/2608.03063#S4.T4)are consistent with this full finetuning of the backbone on recommendation data\. We use the same backbone, preprocessing, and splits \(Appendix[B\.2](https://arxiv.org/html/2608.03063#A2.SS2)\) and the TS\-SFT budget \(Table[12](https://arxiv.org/html/2608.03063#A3.T12)\)\.Vanilla\-Sequence\(item\-token history only\) andTextualized\(item history as text\) reuse the sequence\-only and text\-serialization references above\.

#### Item recommendation and understanding on RecIF \(RQ4, Tables[5](https://arxiv.org/html/2608.03063#S4.T5)and[6](https://arxiv.org/html/2608.03063#S4.T6)\)\.

OneRec\-8B\(Team,[2025a](https://arxiv.org/html/2608.03063#bib.bib22)\)is OpenOneRec’s unified item\-token sequence–language model; we use the checkpoint from its official Hugging Face repository and the official RecIF split \(Appendix[B\.3](https://arxiv.org/html/2608.03063#A2.SS3)\)\. At the sequence\-acquisition stage, we compare OneRec\-Pretrain with SeqLLM Align→\\toSFT; for final adaptation, we compare OneRec Pretrain→\\toSFT→\\toDistill→\\toRL with SeqLLM Align→\\toSFT→\\toRL\. Both methods use 156K behavioral sequences and the same 13M\-caption alignment pool\. OneRec’s 33B\-token co\-pretraining allocates 62% to general\-domain text \(∼\\sim26M examples\) and is followed by∼\\sim2\.6M general SFT examples; SeqLLM uses 100K general\-instruction examples\.

The GPU\-days in Table[5](https://arxiv.org/html/2608.03063#S4.T5)are estimated as

\(7\)GPU\-days≈6​N​DMFU⋅Fpeak⋅86400,\\text\{GPU\-days\}\\approx\\frac\{6ND\}\{\\text\{MFU\}\\cdot F\_\{\\text\{peak\}\}\\cdot 86400\},whereN≈8×109N\\\!\\approx\\\!8\\times 10^\{9\},DDis the processed\-token count \(for OneRec\-8B, taken from its officially reported per\-stage token budgets\(Team,[2025a](https://arxiv.org/html/2608.03063#bib.bib22)\); for SeqLLM, measured from our training logs\),FpeakF\_\{\\text\{peak\}\}is the per\-GPU BF16 throughput, andMFU∈\[0\.4,0\.5\]\\text\{MFU\}\\in\[0\.4,0\.5\]\. Since both models share the backbone size and hardware, GPU\-days scale withDD, giving460/∼994460/\\\!\\sim\\\!994\(OneRec\) vs\.105/206105/206\(SeqLLM\), a4\.4×/4\.8×4\.4\\times/4\.8\\timesreduction\.

## Appendix DDetails of Evaluation

#### Next\-transaction HR@kk\(WeChat Pay\)\.

A transaction event is described by2828fields, but next\-event prediction is scored on five core fields that identify a transaction\. For each test position we generate thekkmost probable next events by beam search over the core\-field tokens, ranked by joint probability, and record a hit if the ground\-truth event matches one of the top\-kkcandidates on all five core fields\. HR@kkis the fraction of positions with a hit; we report HR@10\.

#### Next\-item Recall@kk\(MovieLens/Amazon\)\.

Each user contributes a single held\-out last item \(Appendix[B\.2](https://arxiv.org/html/2608.03063#A2.SS2)\)\. We rank items by predicted probability and set Recall@k=1k=1when the held\-out item is among the top\-kk; we report Recall@5\. With one relevant item this coincides with HR@kk\.

#### Top\-ranked precision \(merchant risk\)\.

The held\-out test window covers 30 consecutive days; on each day we rank the daily cohort \(≈0\.99\\approx\\\!0\.99M merchants\) by the risk scoress\(Appendix[C\.3](https://arxiv.org/html/2608.03063#A3.SS3)\)\. Precision@Top\-r%r\\%is the fraction of truly risky merchants among the highest\-scoredr%r\\%of each daily cohort, averaged over the 30 days\. Even at the most selective cutoff \(r=0\.01r=0\.01\), this aggregates≈\\approx99 merchants per day, i\.e\.,≈\\approx3,000 candidates over the window\. We reportr=1/0\.1/0\.01r=1/0\.1/0\.01in Table[2](https://arxiv.org/html/2608.03063#S4.T2)\.

#### Fraud\-detection Precision/Recall@Top\-r%r\\%\.

We conduct a concurrent three\-month A/B test with users randomly assigned to two equal\-sized arms\. The evaluation covers billions of transactions per day\. Fraud labels return through the production feedback pipeline after a 14\-day maturation window, and only fully matured transactions are included; exact traffic counts are withheld for business confidentiality\. Within each online A/B arm, we rank transactions by the fraud detection model’s fraud score and select the highest\-scoredr%r\\%\. Precision is the fraudulent fraction within this set, while recall is the fraction of all labeled fraudulent transactions retrieved by it\. Section[4\.4](https://arxiv.org/html/2608.03063#S4.SS4)reports the SeqLLM arm’s absolute percentage\-point gains over the production baseline: precision atr=0\.01/0\.1r=0\.01/0\.1and recall atr=0\.1/1r=0\.1/1\.

#### RecIF Pass@kkand Recall@kk\.

Following OpenOneRec\(Team,[2025a](https://arxiv.org/html/2608.03063#bib.bib22)\),555[https://github\.com/Kuaishou\-OneRec/OpenOneRec/tree/main/benchmarks](https://github.com/Kuaishou-OneRec/OpenOneRec/tree/main/benchmarks)Pass@kkmeasures whether the ground\-truth item appears amongkkgenerated candidates and Recall@kkthe fraction of all relevant items retrieved; we report P@1, P@32, and R@32\.

#### Preference and understanding metrics\.

Favorite\-genre/category prediction and the RecProbe classification tasks \(Appendix[B\.3](https://arxiv.org/html/2608.03063#A2.SS3)\) are scored by accuracy over the extracted option; Video Interest Ranking uses NDCG@3 of the three candidates against the reference order; review generation uses ROUGE\-L \(F1F\_\{1\}\) between the generated and held\-out reviews\.

#### General language ability\.

MMLU, C\-Eval, and AGIEval are evaluated with the lm\-evaluation\-harness666[https://github\.com/EleutherAI/lm\-evaluation\-harness/](https://github.com/EleutherAI/lm-evaluation-harness/)on vLLM as log\-likelihood completion without a chat template, matching the Qwen3\-8B base protocol; C\-Eval uses 5\-shot, whereas MMLU and AGIEval are evaluated 0\-shot\. We report multiple\-choice accuracy\.

## Appendix EControlled Ablation of the Behavior Projector

We isolate whether the projector helps the LLM merely memorize new token meanings or compose and apply them\. The full model is compared with a*w/o\-projector*variant \(named consistently with Table[4](https://arxiv.org/html/2608.03063#S4.T4)\) that removesgψg\_\{\\psi\}and tunes the behavior\-token embeddings directly\. Both variants use the same backbone, text\-based token initialization, translation\-to\-reasoning data, optimization schedule, and number of updates; the projector is the sole controlled difference\.

We evaluate two capability levels\.*Token translation*measures whether the model can recover the textual description of an individual behavior token\.*Compositional application*measures multi\-event reasoning and downstream risk decisions, where the model must combine token meanings across fields and time\. The w/o\-projector variant learns the translation task but transfers poorly to compositional application\. Adding the shared residual projector leaves token translation intact while substantially improving multi\-event reasoning and downstream transfer\. This separation supports the projector’s intended role as a shared semantic interface rather than additional per\-token storage\.

Table 15\.Controlled ablation of the behavior projector on industrial data\. All factors except the projection interface are held fixed\. Field Acc\. is the fraction of behavior fields translated correctly; higher is better for all metrics\.CapabilityMetricw/o projectorw/ projectorToken translationField Acc\.0\.870\.93Multi\-event Q&AAccuracy0\.320\.86Risk\-assessment Q&AAccuracy0\.210\.78