先优化玉米棒,再优化内核:面向生成式推荐推理的束搜索优化

arXiv cs.LG 论文

摘要

该论文提出了一套面向生成式推荐推理的整体优化流水线,利用树注意力(Tree Attention)及其他编译器级优化,消除冗余的 KV 缓存访问和主机端开销,无需定制内核,即可在昇腾 NPU 上将端到端延迟降至 vLLM 的最低 1/25。

arXiv:2611.23339v1 Announce Type: new Abstract: Generative recommendation (GR) inference differs substantially from conventional LLM serving, featuring long input sequences, wide beam search, and only a few fixed output tokens. Existing LLM serving engines such as vLLM process beams as batched independent sequences, causing redundant KV cache accesses over the shared prompt prefix. Furthermore, vLLM executes each beam search generation step as a separate engine invocation, incurring host-side beam processing overhead between decoding steps and substantial runtime KV block management in PagedAttention. Using Tree Attention, we concatenate beams along the sequence dimension to eliminate redundant KV accesses, and further optimize execution through pre-generated partial-attention masks, static KV cache allocation, loop-unrolled generation dispatch, TorchAir graph compilation, automatic operator fusion, and vocabulary-constrained decoding. Without the use of custom kernels, our approach reduces end-to-end latency by up to 25x over vLLM on Ascend NPUs, and achieves generation latency comparable to a SOTA solution using custom kernels, showing that the generation pipeline should be optimized holistically before resorting to custom kernel development.
查看原文
查看缓存全文

缓存时间: 2026/11/19 08:59

# Optimize the Cob, Then the Kernel: Optimizing Beam Search for Generative Recommendation Inference
Source: [https://arxiv.org/html/2611.23339](https://arxiv.org/html/2611.23339)
## Optimize the Cob, Then the Kernel: Optimizing Beam Search for Generative Recommendation InferenceThanks:†Corresponding authors: Shing Wai Pun \(shingwaipun@gmail\.com\) and Kai Wu \(eewukai@gmail\.com\)\.Thanks:Code will be made publicly available\.

Shing Wai Pun†, Kai Wu†, Pengfei Xia, Hui Zang, Wei Guo, Yong Liu

###### Abstract

Generative recommendation \(GR\) inference differs substantially from conventional LLM serving, featuring long input sequences, wide beam search, and only a few fixed output tokens\. Existing LLM serving engines such as vLLM process beams as batched independent sequences, causing redundant KV cache accesses over the shared prompt prefix\. Furthermore, vLLM executes each beam search generation step as a separate engine invocation, incurring host\-side beam processing overhead between decoding steps and substantial runtime KV block management in PagedAttention\. Using Tree Attention, we concatenate beams along the sequence dimension to eliminate redundant KV accesses, and further optimize execution through pre\-generated partial\-attention masks, static KV cache allocation, loop\-unrolled generation dispatch, TorchAir graph compilation, automatic operator fusion, and vocabulary\-constrained decoding\. Without the use of custom kernels, our approach reduces end\-to\-end latency by up to 25×\\timesover vLLM on Ascend NPUs, and achieves generation latency comparable to a SOTA solution using custom kernels, showing that the generation pipeline should be optimized holistically before resorting to custom kernel development\.

###### Index Terms:

Beam search, generative recommendation, inference optimization, Ascend NPU\.

## IIntroduction

Generative recommendation \(GR\) has recently emerged as an alternative paradigm to conventional retrieval\- and ranking\-based recommendation, formulating recommendation as a conditional sequence generation problem\. As shown in Fig\.[1](https://arxiv.org/html/2611.23339#S1.F1), given a tokenized representation of a user’s interaction history and contextual information, a generative recommendation model directly predicts recommended items as short sequences of discrete item tokens, commonly referred to as Semantic IDs \(SIDs\)\[[1](https://arxiv.org/html/2611.23339#bib.bib1)\]\. Item recommendations are then generated autoregressively through beam search\.

![Refer to caption](https://arxiv.org/html/2611.23339v1/gr_pipeline.png)Fig\. 1:The generation pipeline of generative recommendation models\.Unlike conventional LLM inference, which typically uses greedy decoding or narrow\-beam search to produce long, variable\-length outputs, GR inference is characterized by extremely wide beam search and short, fixed\-length outputs\. To generate a diverse set of recommended items, the beam width can often exceed thousands, while each candidate typically consists of only a few output tokens \(e\.g\., three SID tokens\)\. Recommendations are also conditioned on long user interaction histories, resulting in long input sequences that are shared across all beams\. Moreover, recommendation serving is highly latency\-sensitive, typically operating under inference end\-to\-end latency budgets on the order of 10–100 ms\.

Existing LLM serving engines, such as vLLM, are not designed for this unique combination of long shared prefixes, extremely wide beam search, and short, fixed decoding lengths\. In particular, vLLM processes beams as batched independent sequences, causing redundant KV cache accesses over the shared prefix\. Furthermore, each beam search generation step is executed as a separate engine invocation, incurring host\-side processing overhead between decoding steps, and substantial runtime KV block management in PagedAttention over each generation step\.

In this work, we design a GR inference pipeline that is explicitly optimized for GR’s workload characteristics\. Building upon Tree Attention, we concatenate beams along the sequence dimension and process them with a compile\-time pre\-generated tree attention mask that restricts token visibility between beams, eliminating redundant KV cache accesses over shared prefixes\. We further exploit the short and fixed decoding length of GR by allocating a static\-sized KV cache based on the known beam width and output length, thereby minimizing runtime memory\-management overhead and removing inter\-step host\-side control\. We also use TorchAir graph compilation, automatic operator fusion, and SID vocabulary\-constrained decoding to further improve end\-to\-end latency\.

Our evaluation shows that these system\-level optimizations substantially improve GR inference latencies without the need for custom kernel development\. Our approach reduces end\-to\-end latency by up to 25×\\timescompared with vLLM and achieves generation latency comparable to a SOTA GR serving solution that relies on custom operators\. More broadly, our results serve as a reminder of Amdahl’s law, where kernel\-level speedups are limited by the costs of their surrounding operators\. This motivates optimizing the full generation pipeline holistically, followed by targeted development of the individual kernels that remain bottlenecks—optimizing the “cob” first, then the kernel\.

Our main contributions are:

- •Building on Tree Attention, we develop a complete end\-to\-end inference pipeline and detail the system design choices used, including pre\-generated partial\-attention masks, static KV cache allocation, and token generation loop unrolling\.
- •We implement and optimize the pipeline on Ascend NPUs, detailing how graph compilation, operator fusion, and execution path optimization are applied within the Ascend software stack\.
- •We demonstrate up to 25×\\timeslower end\-to\-end latency than vLLM while achieving generation latency comparable to a SOTA implementation based on custom operators\.

## IIBackground and Motivation

### II\-ASemantic IDs in OpenOneRec

In OpenOneRec, each item is represented as a fixed\-length sequence of three Semantic\-ID \(SID\) tokens\[[2](https://arxiv.org/html/2611.23339#bib.bib6),[3](https://arxiv.org/html/2611.23339#bib.bib7),[4](https://arxiv.org/html/2611.23339#bib.bib3),[5](https://arxiv.org/html/2611.23339#bib.bib5)\]\. These SIDs are obtained by applying three\-layer residual\-quantization K\-means \(RQ\-Kmeans\) to multimodal item embeddings\. At each layer, K\-means constructs a codebook, and the nearest centroid’s index becomes the corresponding token\. Subtracting this centroid produces the residual passed to the next layer\. The tokens therefore describe the item from coarse semantic categories \(e\.g\., sports equipment\) to finer distinctions \(e\.g\., basketballs\), allowing semantically similar items to share prefixes\. The resulting three\-token SID sequence serves as the item representation used for autoregressive recommendation\.

### II\-BBeam Search in vLLM

vLLM is an open\-source inference and serving framework for LLMs, designed to support efficient generation through request batching and paged KV cache management\[[6](https://arxiv.org/html/2611.23339#bib.bib4)\]\. In current versions of vLLM, beam search is implemented as a step\-by\-step loop outside the generation engine, repeatedly invoking`generate\(\)`with`max\_tokens=1`\. This introduces three major inefficiencies for GR workloads\.

#### Bottleneck 1: Redundant KV cache accesses

Within each decoding step, vLLM processes active beams as independent sequences, logically repeating the shared prefix across the batch dimension, as illustrated in Fig\.[2](https://arxiv.org/html/2611.23339#S2.F2)\(lower\-left\)\. Attention is therefore evaluated separately for each beam’s query against the same prefix KV states\. This batched representation does not allow for the common prefix to be reused effectively across beam queries in the attention kernel, leading to redundant KV accesses that are exacerbated by the extremely wide beam search in GR\.

Enabling automatic prefix caching in vLLM can reduce physical KV storage duplication by allowing beams to reuse cached prefix blocks\. However, this doesn’t change vLLM’s attention kernel tiling strategy, which is unaware and therefore unable to take advantage of the shared prefix KV states across beam queries\.

#### Bottleneck 2: Runtime KV block management

vLLM manages the KV cache using PagedAttention to support general LLM workloads with variable sequence lengths\. Using a pre\-determined KV block size \(fixed to 128 tokens in vLLM Ascend\), vLLM manages KV blocks during runtime through block allocation, prefix\-cache block matching, block\-table updates, block reference\-count management, and block releases across each decoding step in beam search\. This mechanism is suboptimal in GR workloads, leading to memory fragmentation, repeated block re\-allocation and uncached KV suffix recomputation\. As each beam is treated as a separate generation request, each beam is allocated a separate KV block for its beam\-specific tokens\. Since generation is limited to a few tokens, most of the capacity in these 128\-token blocks remains unused, where the wasted capacity is multiplied by the large beam widths\. Furthermore, automatic prefix caching only retains fully filled KV blocks across generation steps, so each partially filled beam\-specific block is released and subsequently reallocated at the next generation step, requiring its previously computed KV states to be recomputed\.

#### Bottleneck 3: Host\-side beam processing

vLLM performs beam selection and processing on the host, requiring a large number of candidate token IDs and log probabilities to be transferred from the NPU after each decoding step\. The host then updates beam scores and selects the surviving beams before generation can resume, introducing synchronization overhead and gaps where the NPU is idle\.

### II\-CExisting GR Inference Systems

xGR\[[7](https://arxiv.org/html/2611.23339#bib.bib2)\]is a serving system optimized for GR inference\. Its custom xAttention operator separates the KV cache into a shared prompt region and beam\-specific regions, allowing all beams to reuse the prompt KV states without access duplication\. Attention over the two regions is computed separately and combined through online softmax to preserve the result of joint attention\. After beam selection, dedicated cache\-management operators reorder and replicate the beam\-specific KV states according to the selected parent indices for the next decoding step\.

UniSGR\[[5](https://arxiv.org/html/2611.23339#bib.bib5)\]proposes Semantic Tree Attention with Reorganized KV Cache \(STARK\), which uses Tree Attention, like ours\. Its key idea is to move beam expansion to the sequence dimension, using a precomputed tree attention mask to exploit shared prefixes while preserving beam search semantics\. However, the paper provides limited details on the practical implementation and system\-level optimizations\.

![Refer to caption](https://arxiv.org/html/2611.23339v1/graph.png)Fig\. 2:\(Upper\) Illustration of beam search with a beam width of two and three decode steps\. \(Lower\-left\) Processing logic and memory management in vLLM at the second decoding step\. \(Lower\-right\) Processing logic and memory management in Tree Attention at the second decoding step\.

## IIIProposed Method

### III\-ATree Attention

We address the first bottleneck of redundant KV cache accesses by applying Tree Attention to avoid replication of the shared prefix while preserving the logical structure of beam search\. As opposed to placing beams along the batch dimension in vLLM, the Tree Attention implementation appends the tokens selected at each decoding step contiguouslyalong the sequence dimensionto a single instance of the prompt\. This evaluates all the beams together with the prompt as one long sequence\. Figure[2](https://arxiv.org/html/2611.23339#S2.F2)illustrates this process for a beam width of 2 in the second decoding step\. Tokens E and F are selected as the top\-B​WBWtokens for the second decode step, and are consequently flattened and appended to the input sequence for the next decoding step\.

The use of a flattened tree structure requires a corresponding attention mask that only allows each generated token to attend to itself, its ancestors, and the shared prompt, but not to tokens in other beam branches\. The attention operation can be expressed mathematically as:

softmax⁡\(𝐐𝐊𝖳d−λ​𝐌\)​𝐕,λ→\+∞\.\\operatorname\{softmax\}\\left\(\\frac\{\\mathbf\{Q\}\\mathbf\{K\}^\{\\mathsf\{T\}\}\}\{\\sqrt\{d\}\}\-\\lambda\\mathbf\{M\}\\right\)\\mathbf\{V\},~~\\lambda\\rightarrow\+\\infty\.\(1\)where𝐐\\mathbf\{Q\},𝐊\\mathbf\{K\}and𝐕\\mathbf\{V\}denote the query, key and value matrices\. The attention mask𝐌\\mathbf\{M\}controls the visibility between tokens, and its elements are defined as

Mi,j=\{0​\(visible\),j∈𝒫∪Path⁡\(i\),1​\(masked\),otherwise,M\_\{i,j\}=\\begin\{cases\}0~\(\\text\{visible\}\),&j\\in\\mathcal\{P\}\\cup\\operatorname\{Path\}\(i\),\\\\ 1~\(\\text\{masked\}\),&\\text\{otherwise\},\\end\{cases\}\(2\)where𝒫\\mathcal\{P\}denotes the positions of the shared prompt, andPath⁡\(i\)\\operatorname\{Path\}\(i\)denotes the ancestor path of tokenii, including itself\. With this mask mechanism, each token observes the same attention causality as it would in an independently evaluated beam, while all beams are evaluated in a single attention operation\.

### III\-BPartial Mask Pre\-Generation

![Refer to caption](https://arxiv.org/html/2611.23339v1/mask_generation.png)Fig\. 3:Mask generation for the three\-token generation loop in our Tree Attention implementation with a beam width of 2\. Following the Ascend FlashAttention operator’s masking convention, 0 denotes a visible element and 1 denotes a masked element\. The parent update shown in this figure is based off the parent selection from Fig\.[2](https://arxiv.org/html/2611.23339#S2.F2)\.Constructing the attention mask during decoding introduces repeated tensor initialization, indexing, and comparison operations, motivating mask pre\-generation before decoding\. However, the complete mask cannot be fully pre\-generated because the ancestor paths depend on runtime top\-kkbeam selection\. We therefore decompose mask construction into a`static mask`and a`running mask`\. The static mask represents the predefined visibility pattern of each decoding step and is pre\-generated for every supported prompt length and beam width, while the running mask carries previously generated masks across decoding steps and updates each beam’s ancestors according to the parent indices selected at runtime\. The two masks are combined through a lightweight`logical AND`operation\. This efficiently preserves previously visible positions while exposing the newly valid positions of the current decoding step, avoiding full mask reconstruction at runtime\.

Figure[3](https://arxiv.org/html/2611.23339#S3.F3)illustrates the mask\-update process using the beam search example in Fig\.[2](https://arxiv.org/html/2611.23339#S2.F2)\. During prefill and the first decoding step, the prompt is processed using standard causal attention without an explicit beam\-ancestry mask\. At the second decoding step, the initialized running mask is combined with the corresponding static mask through an element\-wise`logical AND`operation, producing the attention mask that is subsequently used in FlashAttention\. Following the example in Fig\.[2](https://arxiv.org/html/2611.23339#S2.F2), at the end of the second decoding step, Tokens E and F are selected, both descending from parent A\. The running mask is therefore updated to unmask Token A while masking Token B\. This update is performed through a single gather operation on the mask using parent indices from top\-kkselection, allowing each beam’s ancestor visibility to be updated in parallel\. At the third decoding step, the updated running mask is again combined with the static mask through`logical AND`to produce the final mask passed to the FlashAttention operator\.

The memory overhead for mask pre\-generation is low\. For a prompt length of 1024 and beam width of 512, the static mask occupies 4 MB of HBM, while the running mask requires 1 MB per active request, which is negligible relative to model weights and KV cache\.

### III\-CStatic Cache Initialization

To address our second bottleneck of runtime KV block management overhead, we statically allocate a memory region of sizeP\+B​W×\(N−1\)P\+BW\\times\(N\-1\)for the KV states before graph capture, as shown in Fig\.[2](https://arxiv.org/html/2611.23339#S2.F2), wherePPis the prompt length,B​WBWis the beam width, andN−1N\-1is the number of decoding steps minus one, as the last token’s KV values do not need to be cached\. Since these dimensions are fixed for GR inference, the cache layout and per\-step write regions are predetermined, eliminating runtime block allocation, resizing, and block\-table management\.

Our use of an attention mask removes the need to reorganize KV states after top\-kkbeam selection\. All generated KV states remain at their original cache positions, including those from subsequently pruned branches, while the attention mask is updated to prevent future tokens from accessing invalid paths\. This replaces costly KV state rearrangement after each decoding step for each transformer layer with a fast mask update that is common to all layers\.

### III\-DFull Three\-Token Loop Unrolling

![Refer to caption](https://arxiv.org/html/2611.23339v1/vllm_profile.png)\(a\)Profiling trace of vLLM showing NPU idle gaps between generation steps\.
![Refer to caption](https://arxiv.org/html/2611.23339v1/treeatten_profile.png)\(b\)Profiling trace of our solution showing that loop unrolling eliminates NPU idle gaps between generation steps\.

Fig\. 4:NPU profiled execution timelines for three\-token generation\.To address the third bottleneck of host\-side beam processing, we exploit the fixed generation length in GR and unroll the entire decoding loop into a single NPU execution graph\. Intermediate candidate scores, beam indices, attention masks, and KV cache states remain on the NPU, allowing beam selection and state updates to proceed without returning control to the host between decoding steps\. This eliminates repeated engine invocations and host–device synchronization while exposing the complete fixed computation for further graph\-level optimization\.

Figure[4](https://arxiv.org/html/2611.23339#S3.F4)compares the execution timelines of vLLM and our implementation\. In vLLM, each decoding step requires a separate engine invocation\. After each step, control returns to the host for beam selection, request construction, scheduling, and KV cache management before the next step can begin\. These dependencies introduce substantial NPU idle gaps between decoding steps, particularly at large beam widths\. In contrast, our loop\-unrolled execution keeps the complete three\-step generation on the NPU, effectively eliminating inter\-step idle time\.

## IVFurther Optimizations

### IV\-ATorchAir Graph Compilation and Automatic Operator Fusion

The fixed execution path of GR inference provides an opportunity for graph\-level compilation and dispatch, replacing individual operator dispatches and reducing operator\-launch overhead\. We compile the entire decoding computation graph for Ascend NPUs with TorchAir\[[8](https://arxiv.org/html/2611.23339#bib.bib8)\], so that each request is executed as a single device\-side graph\. Dedicated graphs are compiled beforehand for predefined prompt\-length buckets, and the closest matching graph is selected at runtime\. Inputs are padded to the corresponding bucket size when necessary, enabling static tensor shapes and reusable compiled graphs at the cost of a small amount of redundant computation\. This static execution structure further allows TorchAir to optimize invariant tensor shapes and memory addresses, and to specialize operator tiling and autotuning for each predefined tensor shape\.

TorchAir provides various compiler optimization and kernel fusion features\. We enable tensor\-view optimization, graph\-pattern optimization, and Unified Buffer fusion for our implementation\. Tensor\-view optimization simplifies eligible reshape and dimension\-permutation sequences, avoiding unnecessary tensor copies\. For example, this is used in to avoid unnecessary tensor materialization during Q/K/V preparation and mask processing\. Graph\-pattern optimization fuses compatible operator sequences into a combined operator without writing to HBM, such as fusing residual addition followed by RMS normalization\. Unified Buffer fusion combines compatible operations into compiler\-generated kernels, allowing intermediate results to remain in the NPU’s on\-chip Unified Buffer and reducing kernel\-launch overhead and reads/writes to HBM\. The resulting implementation combines Ascend\-provided fused kernels that are explicitly selected in our code with kernels automatically fused by the compiler from compatible operation sequences, without requiring custom kernel implementations\.

### IV\-BSID\-restricted Decode

In GR, only a small subset of tokens from the model’s vocabulary are valid item related Semantic\-ID tokens, so we replace the final output projection over the full vocabulary with an output projection over only the valid subset of SID tokens\. The projection’s corresponding matrix multiplication operation, beam scoring, log\-softmax, and top\-kkselection therefore process 8,192 candidates per beam instead of 176,384 for OpenOneRec, reducing the candidate dimension by approximately21\.5×21\.5\\times, and therefore significantly lowering latency for these operations\.

## VEvaluation

We evaluate our implementation on OpenOneRec\-1\.7B and OpenOneRec\-8B\[[4](https://arxiv.org/html/2611.23339#bib.bib3)\]using a single Ascend 910B4 NPU withtorch\_npu2\.8\.0 and TorchAir 7\.3\.0\. Model weights, activations, and KV cache are stored in the model default`BFloat16`precision\. We compare against two inference serving implementations\. The first is vanilla vLLM, using vLLM Ascend v0\.13\.0 with prefix caching enabled\. vLLM Ascend has a fixed KV block size of 128 tokens\. The second is a SOTA custom vLLM implementation for GR that integrates the xAttention operator introduced in Sec\. 2C to reduce redundant KV accesses\. This solution also moves beam selection to the NPU, avoiding host–device synchronization between decoding steps\. We evaluate these solutions using inference latency, where compute graph compilation latencies and warm\-up overheads are excluded\. There is no generation accuracy loss observed across all three implementations\.

### V\-ALatency Comparisons

TABLE I:Latency results on an Ascend 910B4 NPU\. All values are reported in milliseconds\. E2E latency is the sum of TTFT and decode time\. NPU time measures token generation time excluding NPU idle time\.\(a\)OpenOneRec\-1\.7B, beam width 128, input length 1,024\.
\(b\)OpenOneRec\-1\.7B, beam width 512, input length 1,024\.
\(c\)OpenOneRec\-1\.7B, beam width 512, input length 10,880\.
\(d\)OpenOneRec\-8B, beam width 128, input length 1,024\.
\(e\)OpenOneRec\-8B, beam width 512, input length 1,024\.

Table[I](https://arxiv.org/html/2611.23339#S5.T1)compares the inference latency of the three implementations under various beam widths and input lengths\. End\-to\-end \(E2E\) latency measures the overall prefill and decode latency, including CPU processing, host–device synchronization, and NPU execution\. It consists of Time\-to\-First\-Token \(TTFT\), the time required to generate the first token, and Decode Time, the latency of generating the remaining two tokens\. NPU TTFT and NPU Time\-per\-Output\-Token \(TPOT\) measure only NPU execution time, where NPU TTFT denotes the NPU latency of the first token and NPU TPOT denotes the average NPU latency per token for the remaining two tokens\.

Both GR\-specific implementations achieve significant reductions in E2E latency relative to vanilla vLLM\. For example, for OpenOneRec\-1\.7B with a beam width of 128 and an input length of 1,024 tokens, Table[I](https://arxiv.org/html/2611.23339#S5.T1)\(a\) shows that the custom vLLM and our implementation achieve E2E latencies of 59\.9 ms and 59\.1 ms, respectively, compared with 244\.7 ms for vanilla vLLM, a speedup of approximately 4×\\times\. These gains come primarily from reductions in decode time\. Vanilla vLLM performs beam path management on the host, incurring host\-side processing and host\-device synchronization overhead between decoding steps\. This is reflected in the large difference between the NPU execution time for the last two tokens and the total decode latency\. In Table[I](https://arxiv.org/html/2611.23339#S5.T1)\(a\), this corresponds to \(28\.8×2=57\.628\.8\\times 2=57\.6ms\) for NPU time for the last 2 tokens, and 208\.2 ms for its decode time\. By unrolling the generation loop into a single device\-side graph dispatch using TorchAir and keeping beam updates on the NPU, our approach eliminates expensive data movement, reducing this overhead to a negligible level\. For the same model, increasing the beam width to 512 raises the speedup of our approach over vanilla vLLM to over 25×\\times, as shown in Table[I](https://arxiv.org/html/2611.23339#S5.T1)\(b\)\.

Furthermore, our approach achieves E2E latency comparable to the custom vLLM solution, with a difference of less than 3%\. This demonstrates that, with system\-level optimizations, we can achieve performance similar to a solution that uses custom operators\.

### V\-BComponent\-Level Breakdowns

TABLE II:Operator latencies for OpenOneRec\-1\.7B inference with an input length of 1,024 tokens and a beam width of 512\. Each entry reports the sum of NPU kernel latencies grouped by component\.\*vLLM performs KV cache initialization and beam\-path update & reconstruction on the host\. \+vLLM is adapted to execute`torch\.topk\(\)`on the NPU rather than on the host in our evaluation\.

Table[II](https://arxiv.org/html/2611.23339#S5.T2)shows a breakdown of the NPU execution time for OpenOneRec\-1\.7B with a beam width of 512 and an input length of 1,024 tokens\. Both GR\-specific implementations substantially reduce the attention kernel and beam scoring latencies relative to vanilla vLLM\. Across the second and third decoding steps, attention kernel time decreases from 114\.484 ms in vanilla vLLM to 7\.442 ms in custom vLLM and 9\.790 ms in our approach\. This is achieved by avoiding redundant attention computations over the shared prompt across beams in beam search\. Beam scoring latency also decreases from 24\.507 ms to 4\.640 ms in custom vLLM and 0\.984 ms in our approach\. Vanilla vLLM computes token probabilities over the full vocabulary, whereas custom vLLM restricts top\-kkselection to valid SID tokens\. Our implementation goes one step further and applies this restriction earlier, at the vocabulary projection stage, avoiding the computation of logits and probabilities for invalid tokens\.

Our approach has a higher attention kernel latency than the custom vLLM solution, but comparing attention kernel times alone does not reflect the additional cost of beam KV management that is unique to the chosen method of attention\. Furthermore, each attention operator has associated preprocessing and postprocessing \(i\.e\., transposes, slicing, and copies\) to satisfy the layout requirements of the attention operator\. Although our approach has higher attention kernel latency and surrounding processing overheads, we focus on minimizing KV cache operations, including KV cache initialization, beam\-path update and reconstruction overheads\. Custom vLLM uses xAttention, which requires an unshared KV cache to be initialized at runtime and updated between decoding steps to reflect the selected parent beams\. After the final decoding step, the selected beam sequences must also be reconstructed from the shared and unshared KV caches\. Initialization takes 0\.684 ms, while beam\-path updates and reconstruction take 2\.686 ms and 0\.971 ms after the second and third decoding steps, respectively\. Together, these operations take 4\.341 ms\. In contrast, our implementation retains KV states in their existing positions and updates attention masks to reflect beam ancestry, requiring only 0\.209 ms for mask construction, beam\-path updates and reconstruction combined\. This is achieved because updating attention masks requires substantially less data movement than reorganizing KV tensors\. Accounting for attention kernels, surrounding processing, and KV/mask management, our solution takes a total of 15\.189 ms and custom vLLM takes 15\.571 ms\.

These results highlight the importance of holistically optimizing the complete generation path, or the “cob”, before optimizing the attention “kernel” in isolation\. Our mask\-based design offsets the slower attention kernel and tensor preparation overheads, resulting in similar attention\-related latency\. As future work, a custom Tree Attention kernel that supports the existing tensor layouts could potentially further improve performance\.

## VIConclusion

This work presents a system\-level optimization approach for GR inference on Ascend NPUs\. By combining tree attention with efficient mask and KV cache management, loop\-unrolled generation, graph compilation, and vocabulary\-constrained decoding, our implementation reduces redundant KV accesses and host\-side overhead\. Without custom kernels, it achieves up to 25×\\timeslower end\-to\-end latency than vLLM and achieves performance comparable to a SOTA implementation using specialized operators\. These results demonstrate that system\-level optimization can deliver substantial gains while retaining flexibility across model configurations and decoding workflows\.

## References

- \[1\]S\. Rajput, N\. Mehta, A\. Singh, R\. H\. Keshavan, T\. Vu, L\. Heldt, L\. Hong, Y\. Tay, V\. Q\. Tran, J\. Samost, M\. Kula, E\. H\. Chi, and M\. Sathiamoorthy\(2023\)Recommender systems with generative retrieval\.External Links:2305\.05065,[Link](https://arxiv.org/abs/2305.05065)Cited by:[§I](https://arxiv.org/html/2611.23339#S1.p1.1)\.
- \[2\]S\. Rajput, N\. Mehta, A\. Singh, R\. Keshavan, T\. Vu, L\. Heidt, L\. Hong, Y\. Tay, V\. Q\. Tran, J\. Samost, M\. Kula, E\. H\. Chi, and M\. Sathiamoorthy\(2023\)Recommender systems with generative retrieval\.InProceedings of the 37th International Conference on Neural Information Processing Systems,NIPS ’23,Red Hook, NY, USA\.Cited by:[§II\-A](https://arxiv.org/html/2611.23339#S2.SS1.p1.1)\.
- \[3\]R\. He, L\. Heldt, L\. Hong, R\. Keshavan, S\. Mao, N\. Mehta, Z\. Su, A\. Tsai, Y\. Wang, S\. Wang, X\. Yi, L\. Baugher, B\. Cakici, E\. Chi, C\. Goodrow, N\. Han, H\. Ma, R\. Rosales, A\. V\. Soest, D\. Tandon, S\. Wu, W\. Yang, and Y\. Zheng\(2026\)PLUM: adapting pre\-trained language models for industrial\-scale generative recommendations\.InProceedings of the ACM Web Conference 2026,WWW ’26,New York, NY, USA,pp\. 8093–8104\.External Links:ISBN 9798400723070,[Link](https://doi.org/10.1145/3774904.3792802),[Document](https://dx.doi.org/10.1145/3774904.3792802)Cited by:[§II\-A](https://arxiv.org/html/2611.23339#S2.SS1.p1.1)\.
- \[4\]G\. Zhou, H\. Bao, J\. Huang, J\. Deng, J\. Zhang, J\. She, K\. Cai, L\. Ren, L\. Ren, Q\. Luo, Q\. Wang, Q\. Hu, R\. Zhang, R\. Tang, S\. Wang, W\. Li, X\. Wu, X\. Luo, X\. Wang, Y\. Hu, Y\. Wu, Z\. Liu, Z\. Zhang, Z\. Zhang, B\. Chen, B\. Wen, C\. Ma, C\. Song, C\. Chu, D\. Lian, F\. Yang, F\. Jiang, H\. Cheng, H\. Wang, K\. Gai, P\. Zheng, Q\. Wang, R\. Huang, S\. Mao, T\. Gao, W\. Yuan, Y\. Wang, Y\. Zhou, Y\. Su, Z\. Cheng, Z\. Ling, and Z\. Li\(2026\)OpenOneRec technical report\.External Links:2512\.24762,[Link](https://arxiv.org/abs/2512.24762)Cited by:[§II\-A](https://arxiv.org/html/2611.23339#S2.SS1.p1.1),[§V](https://arxiv.org/html/2611.23339#S5.p1.1)\.
- \[5\]J\. Sun, J\. Yang, Z\. Guo, D\. Xu, J\. Yan, L\. Deng, and X\. Zeng\(2026\)UniSGR: unified framework for semantic ID generation and ranking\.External Links:2607\.04068,[Link](https://arxiv.org/abs/2607.04068)Cited by:[§II\-A](https://arxiv.org/html/2611.23339#S2.SS1.p1.1),[§II\-C](https://arxiv.org/html/2611.23339#S2.SS3.p2.1)\.
- \[6\]W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica\(2023\)Efficient memory management for large language model serving with PagedAttention\.External Links:2309\.06180,[Link](https://arxiv.org/abs/2309.06180)Cited by:[§II\-B](https://arxiv.org/html/2611.23339#S2.SS2.p1.1)\.
- \[7\]Q\. Sun, T\. Liu, S\. Zhang, S\. Wu, P\. Yang, H\. Liang, M\. Li, X\. Ma, Z\. Liang, Z\. Ren, M\. Zhang, Y\. Wang, X\. Liu, K\. Zhang, H\. Yang, and D\. Qian\(2026\)xGR: efficient generative recommendation serving at scale\.External Links:2512\.11529,[Link](https://arxiv.org/abs/2512.11529)Cited by:[§II\-C](https://arxiv.org/html/2611.23339#S2.SS3.p1.1)\.
- \[8\]Huawei\(2024\)TorchAir\.Note:https://gitee\.com/ascend/torchairCited by:[§IV\-A](https://arxiv.org/html/2611.23339#S4.SS1.p1.1)\.

相似文章

Meta的优化版RecSys推理(58分钟阅读)

TLDR AI

Meta的内核内广播优化(IKBO)通过内核-模型-系统协同设计,消除了RecSys推理中的冗余用户嵌入广播,在H100 GPU上实现了高达2/3的延迟降低和约4倍加速,并成为Meta自适应排名模型的骨干。

AdaExplore:基于失败驱动的自适应与多样性保留搜索的高效内核生成

arXiv cs.CL

来自卡内基梅隆大学、华盛顿大学和Arm的研究人员提出了AdaExplore,这是一种用于GPU内核代码生成的LLM智能体框架。该框架通过失败驱动自适应与多样性保留搜索技术,在不进行额外微调的情况下,在KernelBench Level-2和Level-3基准测试中分别实现了3.12倍和1.72倍的加速。

优化模型以快速进行代码生成(8分钟阅读)

TLDR AI

Morph LLC描述了三种关键技术——基于编码输出训练投机模型、在廉价GPU上自动搜索内核、以及编写自定义互连——以大幅加速像Qwen和DeepSeek这样的开放模型在编码代理工作负载上的运行,实现了最高3倍的投机解码加速,并在7000美元的GPU上达到97-162 tok/s。