Towards a Densing Law for User Representation Learning at Billion-Scale Capacity
Summary
The paper proposes a densing law for user representation learning that quantifies the relationship between data scale and tokenization capacity, and introduces an adaptive tokenization method ALGN to improve efficiency in billion-scale scenarios.
View Cached Full Text
Cached at: 08/25/26, 12:36 PM
Paper page - Towards a Densing Law for User Representation Learning at Billion-Scale Capacity
Source: https://huggingface.co/papers/2608.23392
Abstract
The study proposes a scaling law linking data size to tokenization capacity for user behavior modeling and introduces an adaptive tokenization method to improve efficiency.
User representation learningin real-world industrial scenarios is commonly scaled by increasing user amount, behavioral sequence length and model size. However, existing methods face two challenges: (i) Bottleneck for raw data scaling at billion-scale capacity, as performance exhibit diminishing performance gains with larger-scale raw text user behavioral input, which can be mitigated bytokenization. (ii) Lack of quantitative analysis of howtokenizationconfigurations should scale with data size. In this report, we proposeUser Behavioral Densing Lawfor characterizing the quantitative relationship between data scale and the minimum sufficienttokenizationcapacity. Firstly, we conduct a pilot study on raw & tokenized scaling comparison on billion-scale Alipay dataset, revealing the raw data scaling bottleneck and the sustained gains enabled bytokenization. To derive the scaling pattern governing the minimum sufficienttokenizationconfiguration at different data scales, theoretical analysis and systematic experiments are employed to summarize the quantitative scaling pattern. We find an approximately linear relationship between the logarithms of minimum sufficienttokenizationcapacity and input data size measured by tokens, and the scaling slope varies systematically with thetokenizationmethod and data source, reflecting differences in representation-space redundancy and intra-source uniqueness. Guided by the proposed law, we further developALGN, anadaptive variable-length tokenizationmethod that improves capacity allocation. Extensive experiments across diverse data sources,tokenizationmethods, and downstream tasks demonstrate the generalizability and reliability of theUser Behavioral Densing Law, providing practical guidance fortokenizationconfiguration selection in large-scaleuser representation learning. Moreover,ALGNoutperforms existing baselines.
View arXiv pageView PDFGitHub0Add to collection
Get this paper in your agent:
hf papers read 2608\.23392
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.23392 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.23392 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.23392 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
ScaleToT: Generalizing Structured LLM Reasoning for Billion-Scale Low-Activity User Modeling
ScaleToT proposes a method to generalize structured LLM reasoning for low-activity user modeling at billion scale, using tree-of-thought refinement and training a student model to reduce cost. An online A/B test in advertising deployment showed a 6.738% increase in LT30.
Compute Optimal Tokenization (2 minute read)
This paper systematically derives compression-aware neural scaling laws by training nearly 1,300 models, demonstrating that the widely used heuristic of 20 tokens per parameter is an artifact of specific tokenizers. The authors propose a tokenizer-agnostic scaling law based on bytes, offering a new framework for compute-efficient training across diverse languages and modalities.
How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size
Proposes a three-term scaling law that decouples model size, training steps, and batch size, enabling robust fitting with fewer runs and deriving scaling laws for suboptimal batch sizes.
InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition
InfoLaw is a data-aware scaling framework that predicts model loss based on token consumption, model size, data mixture weights, and repetition, enabling efficient data-recipe selection under varying compute budgets.
@rosinality: https://arxiv.org/abs/2606.29858 Why does power-law scaling occur? Loss of individual tokens follows a sigmoidal curve,…
This paper presents a token-level framework showing that power-law scaling in language model loss arises from the aggregation of sigmoidal learning curves of individual tokens, and demonstrates that reshaping training distributions based on token learning times can accelerate validation loss reduction by 11%.