The PyTorch AOTI backend delivered a 1.14x–1.28x speedup over the Python backend in NVIDIA’s HSTU inference tests when deployed with Triton Inference Server.

With the PyTorch AOTI backend and KV cache, @nvidia reports a 2.20x–2.38x speedup in an ideal all-GPU cache-hit scenario.

The results come from NVIDIA’s recsys-examples repository, a collection of examples demonstrating best practices for training and deploying generative recommenders on NVIDIA GPUs using PyTorch. It includes optimized HSTU and Semantic ID implementations covering training and inference workflows.

For HSTU inference, recsys-examples supports PyTorch AOTInductor to execute the model in the Torch C++ runtime. The NVIDIA developer blog also covers nv-embedding-cache, which provides PyTorch-compatible modules for accelerating large-scale embedding lookups.

🔗 Read the full post:

How Generative Recommenders Are Redefining RecSys at Scale

Source: https://developer.nvidia.com/blog/how-generative-recommenders-are-redefining-recsys-at-scale/ Recommender systems (RecSys) are one of the most ubiquitous machine learning problems in the consumer internet industry yet notoriously difficult to train and serve at scale. The advent of LLMs has inspired a shift from the traditional embedding-similarity-based objective to a generative one, where the goal is to predict the next action or item in a large catalog given a sequence of user histories.

This post covers the architectural shift toward generative recommenders (GRs), the production challenges it introduces, and how the NVIDIArecsys\-examplesandnv\-embedding\-cacheaddress them.

Why traditional RecSys breaks at scalehttps://developer.nvidia.com/blog/how-generative-recommenders-are-redefining-recsys-at-scale/#why_traditional_recsys_breaks_at_scale

Data type and volumehttps://developer.nvidia.com/blog/how-generative-recommenders-are-redefining-recsys-at-scale/#data_type_and_volume

User histories, the primary RecSys data type, represent a record of how users interact with items in a catalog. Unlike modalities such as text or images, user histories involve a mix of categorical and continuous features that change frequently over time. At industry scale, this data can get to the order of terabytes or petabytes every day. Even on the most high-end hardware accelerators, data of this size will be unable to fit GPU high-bandwidth memory (HBM), introducing many bottlenecks during training and inference.

Sparsity and the long-tail problemhttps://developer.nvidia.com/blog/how-generative-recommenders-are-redefining-recsys-at-scale/#sparsity_and_the_long-tail_problem

The long-tail problem in recommender systems describes the phenomenon where a small number of popular items in a catalog get the majority of the interaction. This issue arises because the catalog of items can far exceed the number of users, leading to very sparse user-item interaction data. Because the probability distribution of interacting with an item is heavily skewed towards how popular that item is, training data provides little signal to a vast majority of niche items that represent ground truth for user preferences.

Cold start problemhttps://developer.nvidia.com/blog/how-generative-recommenders-are-redefining-recsys-at-scale/#cold_start_problem

As new users or items join a RecSys platform, there is no interaction history to immediately produce a quality embedding. Rather, it has to be deduced from a smaller set of features, which can negatively impact the trajectory of recommendations. This can be partially resolved by correlating with similar items and semantic descriptions, but there is a risk that initial recommendations are poor and degrade user experience.

Strict latency requirementshttps://developer.nvidia.com/blog/how-generative-recommenders-are-redefining-recsys-at-scale/#strict_latency_requirements

In production, RecSys models are often served online to millions of users under strict service-level agreements (SLAs), where small increases in latency can impact user experience. Unlike LLM workloads that may tolerate autoregressive decoding latency, RecSys models must frequently retrieve and rank thousands of candidate items in a few milliseconds.

Generative recommendershttps://developer.nvidia.com/blog/how-generative-recommenders-are-redefining-recsys-at-scale/#generative_recommenders

Unlike traditional embedding-based recommenders that model user-item preference with geometric similarity, GRs reframe recommendation as a sequence modelling problem similar to LLMs.

The objective is to model the probability distribution of next action or item conditioned on a sequence of user history:

P(next_item | user_history)

The shift toward more homogeneous, transformer-like architectures can better leverage scaling laws, potentially unify retrieval and ranking within a single model, and more naturally integrate with the rapidly evolving LLM ecosystem.

The two prevalent approaches that implement this objective in recommenders are Hierarchical Sequential Transduction Units (HSTU) and Semantic IDs:

HSTUhttps://developer.nvidia.com/blog/how-generative-recommenders-are-redefining-recsys-at-scale/#hstu

HSTU, introduced by Meta in 2024, is a foundational GR model architecture that reframes RecSys under the generative objective and introduces key innovations to enable efficient training and serving at production scale.

HSTU represents input data as a per-user sequence of interleaved items and actions (like, click, etc.) ordered by timestamp. It also removes the dependence on explicit feature engineering, a common practice in traditional RecSys models, in favor of learned sequential representations from attention over user-item interactions. This formulation makes user histories analogous to next token prediction in LLMs, where in training you can get a meaningful learning signal both within a sequence and across batches.

Unlike standard Transformer attention, HSTU modifies the attention aggregation mechanism by replacing softmax normalization with SiLU-based weighting, incorporating relative attention bias, and applying elementwise gating before output projection. These modifications preserve stronger magnitude information across long sequences while enabling more efficient kernel fusion and lower-latency inference.

Semantic IDshttps://developer.nvidia.com/blog/how-generative-recommenders-are-redefining-recsys-at-scale/#semantic_ids

Modeling next item prediction over a large corpus of items with a sparse set of user-item interactions can introduce many challenges: bottlenecks from full softmax calculation, weak training signal from long tail items, and poor generalization to semantically similar items.

Semantic IDs (SIDs), introduced by Google, alleviate these issues by producing a smaller set of new vocabulary tokens based on hierarchical clustering of item embeddings. Architectures such as TIGER, PLUM, OneRec v1/v2, and many modern GR architectures use Semantic IDs as the foundation for scalable autoregressive recommendation.

Figure 1. Semantic ID GR end-to-end architecture Unlike traditional RecSys, autoregressive decoding of Semantic IDs directly generates recommendations instead of searching within an embedding space, and naturally provides a ranking via output logits. This allows search methodologies like beam search to produce multiple Semantic IDs in a single forward pass, improving throughput and allowing niche items in a cluster to be chosen.

recsys\-examplesrepositoryhttps://developer.nvidia.com/blog/how-generative-recommenders-are-redefining-recsys-at-scale/#recsys-examples_repository

Therecsys-examplesrepository is a collection of examples to demonstrate the best practice for training and deploying generative recommenders on NVIDIA GPUs using PyTorch. It includes optimized implementations of HSTU and semantic ID models, covering both training and inference workflows. The repository also consolidates three modular components: DynamicEmb for embedding layers, a KV cache and storage manager tailored for recommender systems, and efficient CUDA ops for HSTU and semantic ID beam search decoding.

Dynamic embeddinghttps://developer.nvidia.com/blog/how-generative-recommenders-are-redefining-recsys-at-scale/#dynamic_embedding

Traditional embedding tables in RecSys assume a fixed, static vocabulary. However, in production, new users and items appear continuously, and the long-tail of the item catalog grows much faster than any single GPU’s HBM can keep up with. Over-provisioning a static table wastes memory on rows that will never be touched, while under-provisioning causes expensive copies that can degrade model performance and quality.

DynamicEmb replaces static tables with a GPU-optimized scored hash table that maps arbitrary feature IDs to embedding rows on demand. Rows are allocated only for IDs the model actually sees, and the table lives across HBM and pinned host memory so it can grow well beyond a single GPU’s capacity. The implementation builds on the algorithms from theHierarchicalKVhash-table design. The combination of admission control and score-based eviction allows capacity to be spent on IDs that matter to model training, and makes the long-tail problem tractable at scale.

It is provided as a TorchRec backend, with tables sharded row-wise across ranks using theEmbeddingBagCollectionandEmbeddingCollectionAPIs. Fused CUDA kernels handle lookup and gradient reduction forSUM,MEAN, and sequence pooling modes. Prefetching techniques keep frequently accessed embeddings resident in HBM for efficient access.

HSTU supporthttps://developer.nvidia.com/blog/how-generative-recommenders-are-redefining-recsys-at-scale/#hstu_support

Recsys-examples provides a production-style training and inference stack for HSTU. Item, user, action, and contextual embedding tables are managed through TorchRec, with DynamicEmb providing dynamic capacity and caching for the high-cardinality tables. For dense layers, the HSTU backbone uses Megatron-Core so a single training run can leverage data, tensor, sequence, and pipeline parallelism. The library is designed to be modular across the entire stack so that components can be plug and play with custom architectures.

During training,TorchRec/DynamicEmbandMegatron\-Coreare seamlessly integrated to coordinate sharding and parallelism across both embedding and dense modules. The training pipeline incorporates dynamic shuffling to balance workloads across ranks, overlaps embedding communication and prefetching with dense computation, and includes a HSTU layer with fused ops and FBGEMM attention kernels optimized for NVIDIA Ampere, Hopper, and Blackwell GPUs. Together, these optimizations improve end-to-end Model FLOP Utilization (MFU) to 31.40% from 7.65% on two DGX H100 nodes, demonstrating substantialgains in trainingefficiency.

For inference,recsys\-examplesis designed to meet strict low-latency requirements by supporting PyTorch AOTInductor to execute the model in the Torch C++ runtime while remaining compatible with NVIDIA Triton Inference Server.

Frequently accessed embeddings are kept close to the GPU usingnv\-embedding\-cache, and computation is further reduced through a customizedFlexKV-enabled KV cache that distributes cache entries across multiple memory tiers.

When deployed with Triton Inference Server, inference with the Pytorch AOTI backend & no KV cache delivers a 1.14x ~ 1.28x speedup over the Python backend, and inference with the Pytorch AOTI backend with KV cache delivers a 2.20x ~ 2.38x speedup in an ideal all GPU cache-hit scenario.

Figure 2. HSTU Performance using recsys-examples Triton backend configurations

Semantic ID-GRhttps://developer.nvidia.com/blog/how-generative-recommenders-are-redefining-recsys-at-scale/#semantic_id-gr

Semantic ID-based GR introduces a serving pattern that is very different from chat-based LLM inference: long user context, short autoregressive decoding, and large beam widths over constrained item-token spaces. In practical Semantic ID workloads, a request may contain thousands of historical tokens, decode only 2–3 Semantic ID tokens, and use beam widths such as 128 or 256 to improve recommendation diversity.

Existing LLM serving systems such as vLLM, SGLang, and TensorRT LLM are primarily optimized for multi-user chat-style serving with paged KV cache, dynamic batching, and long decoding. They are powerful general frameworks, but do not naturally expose the core abstractions needed by Semantic ID-GR: shared request-level context KV, short per-beam decode KV, beam-path tracking, dynamic beam width, and item-constrained generation.

To address this,recsys\-examplesprovides a GR-specialized inference frameworkexamplefor Qwen based Semantic ID models. The framework separates KV cache to isolate beam dependent and independent components: long shared context intoContextKV, short decode history intoBeamKV, and logical beam ancestry intoBeamPath. This avoids treating every beam as a separate long sequence. The runtime also includes GR-native continuous batching, direct pool-view CUDA graph replay, item-constrainedtopK, and a dedicatedgr\-decode\_attenbackend that operates directly on the GR KV layout.

On a single NVIDIA H100 80GB GPU with Qwen3-1.7B, context lengths of 1,000 and 5,000 tokens, beam width of 256, and 3 output tokens, the GR-specialized path consistently outperforms the SGLang beam-search in the measured offline and online benchmarks.

MetricWorkloadGR ResultBaselineImprovementOffline latencyctx=1000, batch=4, beam=256, output=347.736 ms102.318 ms2.14x fasterOffline latencyctx=5000, batch=4, beam=256, output=3154.224 ms349.857 ms2.27x fasterOffline latencyctx=5000, batch=8, beam=256, output=3307.917 ms685.354 ms2.23x fasterOnline serving throughputctx=5000, concurrency=4, beam=256, output=3~19.7 req/s~10.7 req/s~1.85x higherOnline median latencyctx=5000, concurrency=4, beam=256, output=3~198 ms~370 ms~46% lowerOffline correctnessctx=1000/5000, batch=1/2/4/8, beam=256Top1 exact 1.000SGLang comparisonTopK overlap mean 0.960Table 2. Serving results for recsys-examples Semantic ID-GR inference framework

This makes Semantic ID-GR serving a natural complement to the HSTU and embedding components in recsys-examples: HSTU andDynamicEmbaddress production-scale training and embedding-heavy inference, while the Semantic ID-GR inference path targets autoregressive Semantic ID generation with long context, large beam search, and strict recommender-system latency requirements.

nv\-embedding\-cachehttps://developer.nvidia.com/blog/how-generative-recommenders-are-redefining-recsys-at-scale/#nv-embedding-cache

nv\-embedding\-cache(NVE) is an SDK for accelerating large-scale embedding table lookups and operations in recommender inference. It provides modular components including optimized kernels, software cache primitives, and PyTorch-compatible bindings, enabling low-latency access to embedding tables that exceed the capacity of a single GPU’s HBM.

Production recommender embedding tables routinely exceed the capacity of a single GPU’s HBM, forcing inference systems to stage embeddings across multiple memory tiers. Furthermore, recommenders often exhibit access patterns that are very favorable to caching.

NVE manages this through a hierarchical lookup flow, composed of GPU cache in HBM as the hot tier, CPU cache in DRAM as the warm tier, and a remote parameter store (typically Redis or RocksDB-backed). Hot keys promotion into the GPU cache is customizable, and a lockless invalidate-and-commit protocol lets lookups and cache modifications run concurrently on the GPU without stalling the lookup stream. Sharding across devices is handled via CUDA virtual memory, so a single logical table can span multiple GPUs or nodes.

Figure 3. Hierarchical memory look-up for NVE supported embedding tables NVE providesNVEmbeddingandNVEmbeddingBagas drop-in replacements fortorch\.nn\.Embeddingandtorch\.nn\.EmbeddingBagfor easy integration with the PyTorch ecosystem. The modules expose familiar arguments while adding cache specific configuration flags. Because the layers behave like standardnn\.Modulecomponents, they can be integrated into existing recommender models with minimal graph changes. For deployment, NVE registers its lookup operators against the LibTorch Stable ABI, providingAOTInductorsupport in the C++ runtime.

Recsys-examples DynamicEmb tables are supported in NVE, allowing an easy transition between training and inference. On DLRM v3, the MLPerf generative recommender benchmark based on HSTU, recsys-examples and NVE were able to achieve99,997 queries/secinference throughput in an online server scenario. An example of how to perform inference with these two libraries can be foundhere.

Get startedhttps://developer.nvidia.com/blog/how-generative-recommenders-are-redefining-recsys-at-scale/#get_started

Check out the following repositories for quick-start guides and more details:

• recsys-examples quick-start guide
• nv-embedding-cache overview