hLLM: Single Pass Decoding for Generative Reranking

# hLLM: Single Pass Decoding for Generative Reranking
Source: [https://arxiv.org/html/2609.01807](https://arxiv.org/html/2609.01807)
Prachi AgrawalMoe Kayali& Bixing YanAffiliation:Meta Platforms, Inc\.Email:[\{emillaftchiev,praa,kayali,bixing\}@meta\.com](mailto:)Qi XuZijie LeiChen QiuZhi HuaKe Li & Luke SimonAffiliation:Meta Platforms, Inc\.

###### Abstract

Large language models \(LLMs\) achieve state\-of\-the\-art generative ranking quality, but the ranking they produce must be*decoded*, and autoregressive decoding spends one sequential forward pass per emitted token\. We observe that the only tokens a ranker must emit are theNNordinal values naming the items in ranked order, and that this narrow, permutation\-structured output format admits decoding strategies which are much more efficient than left\-to\-right generation\. We introducehLLM\(Hungarian LLM\), a format\-specialized decoding strategy that decodes allNNordinals inO⁡\(1\)O\(1\)forward passes\.hLLMreads anN×KN\\times Kitem–position score matrix off the LLM’s prefill hidden states with a lightweight self\-attention head, then decodes the ordinals as the optimal bipartite assignment of that matrix via the Hungarian algorithm, yielding a valid permutation by construction rather than by repair\. Through a systematic study of training signals and backbone adaptation, we show that LoRA\-based fine\-tuning combined with teacher ranking distillation reaches28 msend\-to\-end inference, a speed\-up of64×64\\timeswhile maintaining ranking quality on par with the teacher\. We provide a complete ablation decomposing the contributions of architecture, training signal, and backbone adaptation\. Our framework connects generative ranking to combinatorial optimization, opening a path toward otherO⁡\(1\)O\(1\)\-decode mechanisms for real\-time ranking\.

## 1Introduction

Generative ranking, producing a complete ordering ofNNcandidates jointly rather than scoring each independently, is critical for recommendation, advertising, and search\. Autoregressive LLMs have recently emerged as powerful generative rankers, achieving superior quality by jointly reasoning over all candidates in context\([Sun et al\., 2023](https://arxiv.org/html/2609.01807#bib.bib4)\)\. The output of such a ranker is a permutation, and the model must*emit*that permutation as tokens: concretely, theNNordinal values that name the items in ranked order\. Ranking is therefore an unusually*format\-specialized*generation problem—the output alphabet is\{1,…,N\}\\\{1,\\dots,N\\\}, the output length is known before decoding begins, and every value must appear exactly once\. It is this structure that determines which decoding strategies are available, and it is the axis along which we organize both the prior work and our contribution\.

Autoregressive decoding, however, spends one sequential forward pass over the KV cache per emitted token, so decoding theNNordinals usesO⁡\(N⋅T\)O\(N\\cdot T\)passes, whereTTis the number of tokens per item identifier\. This is a bottleneck of the*decoder*, not of the model: the prefill that computes the candidate representations is already parallel and already optimized in its computational complexity\. As an example, in the experiments for this paper, a teacher with an explicit reasoning trace reaches18071807ms per request on an A100 GPU, of which2828ms is prefill\. We address the*re\-ranking*stage, in which a candidate set ofNNitems retrieved by an upstream system must be ordered by relevance under strict latency constraints\.

##### Decoding strategies\.

Because the ordinals are what must be decoded, the design space is a space of*decoders*over a fixed backbone\. Autoregressive decoding emits them left to right, one forward pass per token: exact and requiring no architectural change, but with a pass count that grows with the slate\. Parallel decoders—non\-autoregressive rerankers and multi\-token predictors\([Ren et al\., 2024](https://arxiv.org/html/2609.01807#bib.bib19)\), and diffusion\-based rerankers\([Liu et al\., 2026](https://arxiv.org/html/2609.01807#bib.bib27)\)—emit all positions at once and refine them over several steps, reducing the pass count but not to one, and without guaranteeing that the emitted ordinals form a permutation\. We ask whether the ordinals can be decoded in a single pass:

Can a ranker decode allNNordinals inO⁡\(1\)O\(1\)forward passes and still recover the ranking that an autoregressive LLM computes over hundreds of sequential passes?

We answer yes, withhLLM, which exploits a connection between ranking and optimal assignment\. The key insight is that an LLM’s prefill hidden states, computed in a single parallel forward pass over all candidates, already encode the comparative information that the ordinals express, via the backbone’s self\-attention layers\. Autoregressive decoding re\-serializes that information one token at a time;hLLMinstead decodes it in place\. A lightweight self\-attention head turns the prefill states into anN×KN\\times Kitem–position score matrix, and the Hungarian algorithm\([Kuhn, 1955](https://arxiv.org/html/2609.01807#bib.bib25)\)decodes allNNordinals from it as the optimal bipartite assignment, with a valid permutation guaranteed by construction\.

##### Contributions\.

1. 1\.A formulation of ordinal decoding as optimal assignment on hidden states\.We cast the emission of theNNitem ordinals as maximum\-weight bipartite matching over anN×KN\\times Kitem–position score matrix whose weights are read directly off the model’s hidden states rather than off the output logits\. This is the formulation that makes exact combinatorial decoding possible: the Hungarian algorithm recovers the optimal permutation at inference, from one pass\. Output validity is guaranteed by construction rather than learned\.
2. 2\.hLLM, anO⁡\(1\)O\(1\)\-pass decoding strategy for generative ranking\.A single prefill over all candidates yields per\-item hidden states; a lightweight self\-attention head maps them to the score matrix; the Hungarian solver decodes theNNordinals from it\. Decoding therefore takes a constant number of forward passes, independent ofNN, removing the sequential decode dependency that dominates autoregressive ranking latency\.
3. 3\.An empirical study on proprietary data and an open dataset\.We evaluate the performance and latency of the approach on a proprietary dataset, finding thathLLMachieves a speed\-up of64×64\\timesover an autoregressive LLM while maintaining ranking performance\. We also validate our approach on an open\-source ranking dataset, Amazon Beauty—and find similar speedups\. We further ablate the experiments across three scoring\-head configurations\. We also isolate the contributions of the backbone and the self\-attention layers\.

## 2Background and Related Work

### 2\.1LLM\-Based Ranking and Generative Retrieval

[Sun et al\. \(2023\)](https://arxiv.org/html/2609.01807#bib.bib4)introduced RankGPT, showing that LLMs can rerank generatively, decoding permutations autoregressively with a sliding window\.[Tay et al\. \(2022\)](https://arxiv.org/html/2609.01807#bib.bib5)pioneered generative retrieval with the Differentiable Search Index, encoding document identifiers directly in model parameters, and[De Cao et al\. \(2021\)](https://arxiv.org/html/2609.01807#bib.bib6)showed autoregressive entity generation is viable for structured retrieval\. Most relevant to our work,[Reddy et al\. \(2024\)](https://arxiv.org/html/2609.01807#bib.bib7)proposed FIRST, which uses first\-decoded\-token logits for ranking\. This is the closest prior approach to constant\-pass decoding, but it reads the ordering out of the output token distribution rather than out of the hidden states\.hLLMkeeps the same output as all of these—the item ordinals—and changes only how they are decoded\.

### 2\.2Decoding Strategies for Generative Ranking

Beyond autoregressive decoding, non\-autoregressive \(NAR\) methods for machine translation established that parallel generation can achieve substantial speedups at some degradation of quality\.[Gu et al\. \(2018\)](https://arxiv.org/html/2609.01807#bib.bib1)introduced NAR translation via fertility prediction;[Ghazvininejad et al\. \(2019\)](https://arxiv.org/html/2609.01807#bib.bib2)recovered quality through iterative refinement of masked positions; and[Qian et al\. \(2021\)](https://arxiv.org/html/2609.01807#bib.bib3)introduced curriculum\-based NAR training\. NAR methods for general text generation typically suffer more than10%10\\%quality degradation, and they still take several refinement passes per output\.hLLMsidesteps both by exploiting the permutation structure of ranking: rather than decoding arbitrary text in parallel, we decode the ordinals as the solution of a constrained combinatorial problem that admits an exact polynomial\-time solution, in a constant number of passes\.

### 2\.3Optimal Transport and Differentiable Permutation Learning

Our training procedure builds on differentiable relaxations of discrete permutations\.[Cuturi \(2013\)](https://arxiv.org/html/2609.01807#bib.bib8)introduced entropic regularization for efficient optimal transport, providing the algorithmic foundation for the Sinkhorn operator, and[Mena et al\. \(2018\)](https://arxiv.org/html/2609.01807#bib.bib9)extended this to learning latent permutations with Gumbel\-Sinkhorn networks\. Alternative differentiable sorting approaches include SoftSort\([Prillo and Eisenschlos, 2020](https://arxiv.org/html/2609.01807#bib.bib10)\)and differentiable sorting networks\([Petersen et al\., 2021](https://arxiv.org/html/2609.01807#bib.bib11)\)\. Our contribution is connecting these permutation\-learning tools to LLM hidden states\.

### 2\.4Knowledge Distillation for Ranking

[Hinton et al\. \(2015\)](https://arxiv.org/html/2609.01807#bib.bib12)introduced soft\-target distillation\. For ranking,[Hofstätter et al\. \(2020\)](https://arxiv.org/html/2609.01807#bib.bib13)demonstrated cross\-architecture distillation from cross\-encoders to bi\-encoders, and[Pradeep et al\. \(2023\)](https://arxiv.org/html/2609.01807#bib.bib14)distilled generative reranking ability into smaller models\. We differ in two ways: we distill full permutations via a Sinkhorn cross\-entropy objective, and our ablation reveals a previously undocumented interaction between signal richness and backbone capacity\.

### 2\.5Hungarian Assignment in Deep Learning

DETR\([Carion et al\., 2020](https://arxiv.org/html/2609.01807#bib.bib15)\)uses bipartite matching between predicted and ground\-truth detections for set\-based loss computation\.[Berthet et al\. \(2020\)](https://arxiv.org/html/2609.01807#bib.bib16)provided theoretical foundations for learning with differentiable perturbed combinatorial optimizers\. In mechanism design,[Dütting et al\. \(2019\)](https://arxiv.org/html/2609.01807#bib.bib17)learn optimal multi\-item auctions via neural networks, and for ad allocation[Zheng et al\. \(2025\)](https://arxiv.org/html/2609.01807#bib.bib18)introduced non\-autoregressive generative auctions with position\-query cross\-attention\. Critically, we use Hungarian matching*not*for loss computation as in DETR, but as the inference\-time decoder that replaces autoregressive generation\. Similarly, DiffuRank\([Liu et al\., 2026](https://arxiv.org/html/2609.01807#bib.bib27)\)applies the Hungarian algorithm to the decoded output of the LLM\. We instead apply it to the hidden states, as the decoder itself, so the permutation constraint is enforced during decoding rather than restored afterwards\.

### 2\.6Efficient LLM Inference

Speculative decoding\([Leviathan et al\., 2023](https://arxiv.org/html/2609.01807#bib.bib20)\)uses a small draft model verified by the target LLM for lossless speedup\. Medusa\([Cai et al\., 2024](https://arxiv.org/html/2609.01807#bib.bib21)\)and EAGLE\([Li et al\., 2024](https://arxiv.org/html/2609.01807#bib.bib22)\)extend this with multiple parallel decoding heads, and LayerSkip\([Elhoushi et al\., 2024](https://arxiv.org/html/2609.01807#bib.bib23)\)enables early\-exit inference\. These methods all reduce the computational overhead of autoregressive decoding while preserving its left\-to\-right form; the pass count still scales with the output\.hLLMis orthogonal and composable: it changes the decoding strategy itself, so theNNordinals emerge from a single pass\.

Despite this breadth of prior work, no existing method decodes a complete ranking from an LLM backbone in a constant number of forward passes, at real\-time latency, with near\-lossless quality and a permutation guaranteed by construction\.

## 3Problem Formulation

### 3\.1Notation and Setup

Letuudenote a user context andC=\{c1,…,cN\}C=\\\{c\_\{1\},\\dots,c\_\{N\}\\\}a slate ofNNcandidate items retrieved by an upstream system\. The re\-ranking task is to produce a permutationπ∈ΠN\\pi\\in\\Pi\_\{N\}that orders items by relevance\. An autoregressive teacher generatesπ\\piviaTTsequential decode passes, incurringO⁡\(N⋅T\)O\(N\\cdot T\)latency\. We seek a studentfθf\_\{\\theta\}that decodes the sameNNordinals, and hence the sameπ\\pi, inO⁡\(1\)O\(1\)decode passes—i\.e\. a single forward pass\.

##### Latency decomposition\.

For an autoregressive ranker, inference computation is dominated by the decode phase: each token requires a full forward pass over the KV\-cache\. The following examples are from the experiments in this paper\. With reasoning enabledT≈657T\\approx 657tokens; without reasoningT≈39T\\approx 39\. In both cases the prefill phase, which processes allNNcandidates in parallel, takes a fixedtprefill≈28t\_\{\\mathrm\{prefill\}\}\\approx 28ms\. Our key observation: if the ranking can be extracted from the prefill hidden states alone, theO⁡\(T\)O\(T\)*sequential*decode computational complexity collapses toO⁡\(1\)O\(1\)—the ordinals are still decoded, but not one at a time\.

### 3\.2Decoding a Ranking as Optimal Assignment

Define a bipartite graphG=\(I∪J,E\)G=\(I\\cup J,E\)whereI=\{1,…,N\}I=\\\{1,\\dots,N\\\}indexes items andJ=\{1,…,K\}J=\\\{1,\\dots,K\\\}indexes rank positions\. Let𝐌∈ℝN×K\\mathbf\{M\}\\in\\mathbb\{R\}^\{N\\times K\}be a score matrix in which𝐌i​j\\mathbf\{M\}\_\{ij\}is the affinity of itemiifor rank positionjj\. The optimal ranking is the maximum\-weight bipartite matching

π∗=arg​maxπ∈ΠN∑i=1N𝐌i,π⁡\(i\)\.\\pi^\{\*\}\\;=\\;\\argmax\_\{\\pi\\in\\Pi\_\{N\}\}\\;\\sum\_\{i=1\}^\{N\}\\mathbf\{M\}\_\{i,\\pi\(i\)\}\.\(1\)Equation \([1](https://arxiv.org/html/2609.01807#S3.E1)\) is the linear assignment problem \(LAP\), solvable exactly inO⁡\(N3\)O\(N^\{3\}\)by the Hungarian algorithm\([Kuhn, 1955](https://arxiv.org/html/2609.01807#bib.bib25)\), or by the Jonker–Volgenant variant \(LAPJV\)\([Jonker and Volgenant, 1987](https://arxiv.org/html/2609.01807#bib.bib26)\)which achievesO⁡\(N2\)O\(N^\{2\}\)average\-case complexity via shortest augmenting paths\.

##### Complexity comparison\.

ForN=50N=50, the autoregressive teacher requiresT=657T=657sequential forward passes\. The Hungarian solver requiresO⁡\(N3\)≈1\.25×105O\(N^\{3\}\)\\approx 1\.25\\times 10^\{5\}arithmetic operations, roughly8​μ8\\,\\mus on CPU\. The fullhLLMpipeline \(one prefill plus one Hungarian solve, i\.e\.O⁡\(1\)O\(1\)passes for allNNordinals\) is therefore bounded by the computational complexity of prefill, making the combinatorial optimization effectively free \(Section[5\.5](https://arxiv.org/html/2609.01807#S5.SS5)\)\.

### 3\.3Differentiable Relaxation via Sinkhorn

The Hungarian algorithm is non\-differentiable: it returns a discrete permutation\. For gradient\-based training we require a continuous relaxation\. We adopt the Sinkhorn operator\([Cuturi, 2013](https://arxiv.org/html/2609.01807#bib.bib8);[Mena et al\., 2018](https://arxiv.org/html/2609.01807#bib.bib9)\), which projects a non\-negative matrix onto the Birkhoff polytope of doubly\-stochastic matrices via alternating row and column normalization:

𝐒\(0\)=exp⁡\(𝐌/τ\),𝐒\(l\+1\)=𝐃r−1​𝐃c−1​𝐒\(l\),\\mathbf\{S\}^\{\(0\)\}=\\exp\(\\mathbf\{M\}/\\tau\),\\qquad\\mathbf\{S\}^\{\(l\+1\)\}=\\mathbf\{D\}\_\{r\}^\{\-1\}\\,\\mathbf\{D\}\_\{c\}^\{\-1\}\\,\\mathbf\{S\}^\{\(l\)\},\(2\)where𝐃r\\mathbf\{D\}\_\{r\}and𝐃c\\mathbf\{D\}\_\{c\}are diagonal normalization matrices enforcing unit row and column sums, andτ\>0\\tau\>0is a temperature controlling relaxation sharpness \(τ→0\\tau\\to 0recovers a permutation matrix\)\.

##### Training objective\.

Given a teacher permutation matrix𝐏teacher∈\{0,1\}N×K\\mathbf\{P\}^\{\\mathrm\{teacher\}\}\\in\\\{0,1\\\}^\{N\\times K\}, the Sinkhorn cross\-entropy loss is

ℒ=−∑i,j𝐏teacheri​jlog𝐒\(L\)i​j\.\\mathcal\{L\}\\;=\\;\-\\sum\_\{i,j\}\\mathbf\{P\}^\{\\mathrm\{teacher\}\}\_\{ij\}\\,\\log\\mathbf\{S\}^\{\(L\)\}\_\{ij\}\.\(3\)This provides smooth gradients to𝐌θ\\mathbf\{M\}\_\{\\theta\}while driving the student’s output toward a valid permutation\. At inference we bypass Sinkhorn entirely and apply the Hungarian algorithm to𝐌\\mathbf\{M\}directly, guaranteeing an exact permutation\.

## 4Method:hLLM

### 4\.1Architecture Overview

hLLMcomprises three components applied in sequence \(Figure[1](https://arxiv.org/html/2609.01807#S4.F1)\)\. Figure[1](https://arxiv.org/html/2609.01807#S4.F1)depicts the winning configuration \(self\-attention head,L=2L=2, LoRA\-adapted backbone\); alternative head variants are ablated in Section[5\.4](https://arxiv.org/html/2609.01807#S5.SS4)\.

Promptuser context\+\+NNcandidatesLLM Backbonesingle forward pass\(prefill only\)0\.6B params, LoRA\-adaptedh1h\_\{1\}h2h\_\{2\}⋮\\vdotshNh\_\{N\}per\-itemhidden statesSelf\-AttentionHead\(L=2L\{=\}2\)items attend toall other itemsScore Matrix𝐌∈ℝN×K\\mathbf\{M\}\\in\\mathbb\{R\}^\{N\\times K\}HungarianAlgorithm\(LAPJV\)Ranked List\[B,D,A,E,C\]\[B,D,A,E,C\]\(item ordinals\)28 ms totalAllNNordinals decoded in one pass

Figure 1:hLLMarchitecture\. A single prefill pass over allNNcandidates yields per\-item hidden states, which the self\-attention head maps to anN×KN\\times Kscore matrix\. The Hungarian algorithm decodes allNNitem ordinals from that matrix in one shot:O⁡\(1\)O\(1\)forward passes, no sequential decode\. End\-to\-end latency is2828ms\.##### Backbone \(prefill\-only, LoRA\-adapted\)\.

The backbone is a0\.60\.6B\-parameter decoder\-only transformer \(2828layers, hidden dimension10241024,1616attention heads\) pretrained for language modeling and fine\-tuned for ranking\. It processes the full prompt containing the user context and allNNcandidates in a single forward pass\. We gather the hidden state at each candidate’s readout position \(the final token of its description\), yielding\{𝐡1,…,𝐡N\}∈ℝN×D\\\{\\mathbf\{h\}\_\{1\},\\dots,\\mathbf\{h\}\_\{N\}\\\}\\in\\mathbb\{R\}^\{N\\times D\}\.

##### Scoring head \(self\-attention,L=2L=2\)\.

A lightweight transformer encoder withL=2L=2self\-attention layers operates over theNNcandidate vectors, allowing each item to attend to all others and thereby perform explicit pairwise comparison\. A linear projection maps each refined representation toKKrank\-position scores, producing𝐌∈ℝN×K\\mathbf\{M\}\\in\\mathbb\{R\}^\{N\\times K\}\.

##### Assignment decoder \(Hungarian\)\.

Unlike DETR\([Carion et al\., 2020](https://arxiv.org/html/2609.01807#bib.bib15)\), which uses Hungarian matching during training for loss computation, we apply the Hungarian algorithm at inference as the decoding mechanism itself\. Applied to𝐌\\mathbf\{M\}it yields the optimal one\-to\-one assignment: a valid permutation decoded in one shot\. The Hungarian solver is thushLLM’s decoder in the same sense that a sampling loop is an autoregressive decoder—it is the component that turns model outputs into theNNemitted ordinals\. During training, Equation \([3](https://arxiv.org/html/2609.01807#S3.E3)\) supplies gradients\.

### 4\.2Scoring Head Variants

We study three head architectures to isolate the contribution of cross\-item comparison \(Figure[2](https://arxiv.org/html/2609.01807#S4.F2), Table[1](https://arxiv.org/html/2609.01807#S4.T1)\)\.

\(a\) Linear Probeh1h\_\{1\}h2h\_\{2\}hNh\_\{N\}⋯\\cdotsLinear\(D,512\)\(D,512\)Linear\(512,K\)\(512,K\)Linear\(D,512\)\(D,512\)Linear\(512,K\)\(512,K\)Linear\(D,512\)\(D,512\)Linear\(512,K\)\(512,K\)each item scored*independently*∼\\sim550K params\(b\) Self\-Attention \(L=2L\{=\}2\)h1h\_\{1\}h2h\_\{2\}hNh\_\{N\}⋯\\cdotsSelf\-Attention Layer 1Self\-Attention Layer 2Linear\(512,K\)\(512,K\)items compare to*each other*∼\\sim4\.2M params\(c\) Slot\-Queryh1h\_\{1\}h2h\_\{2\}hNh\_\{N\}⋮\\vdotst1t\_\{1\}t2t\_\{2\}tKt\_\{K\}⋮\\vdotslearnable positionembeddingscross\-attentiondot product→𝐌\\to\\mathbf\{M\}positions*query*items∼\\sim4\.0M paramsInput \(shared\):per\-item hidden statesh1,…,hNh\_\{1\},\\dots,h\_\{N\}from one prefill passOutput:score matrix𝐌⁡\[N×K\]\\mathbf\{M\}\\,\[N\\times K\]→\\toSinkhorn \(train\) / Hungarian \(eval\)

Figure 2:Scoring head variants\. \(a\) A linear probe scores each item independently; all cross\-item context arises implicitly from the backbone’s self\-attention during prefill\. \(b\) The self\-attention head performs explicit item\-to\-item comparison before scoring\. \(c\) The slot\-query head inverts the formulation:KKlearnable position embeddings cross\-attend over items\. On a frozen backbone \(a\) is competitive; once the backbone is LoRA\-adapted, \(b\) is strongest\.Table 1:Scoring head variants\. All operate on identical frozen or LoRA\-adapted backbone hidden states\.
### 4\.3Training: Offline Teacher Ranking Distillation

Training proceeds in two phases\.

Phase 1 \(offline, one\-time\)\.The teacher, a well\-trained autoregressive generative ranking model, decodes a ranking for every training slate via greedy decoding in direct\-output mode \(no chain\-of\-thought\)\. These target permutations are computed once and stored\.

Phase 2 \(student training\)\.The student minimizes Equation \([3](https://arxiv.org/html/2609.01807#S3.E3)\) against the pre\-computed teacher permutations\. There is no online interaction between teacher and student; the labels are fixed\. This follows from the decoding strategy:hLLM’s decoder has no intermediate state and no rollout—𝐌\\mathbf\{M\}is conditioned only on the prompt, and the Hungarian solve is a deterministic function of𝐌\\mathbf\{M\}—so the train\-time and inference\-time input distributions are identical by construction\. There is no exposure bias for on\-policy distillation to correct and no student trajectory for the teacher to score, and Equation \([3](https://arxiv.org/html/2609.01807#S3.E3)\) is already a whole\-permutation objective rather than a per\-token surrogate\. One offline teacher pass over the training set therefore suffices\.

### 4\.4Backbone Adaptation via LoRA

We apply Low\-Rank Adaptation\([Hu et al\., 2022](https://arxiv.org/html/2609.01807#bib.bib24), LoRA,\)with rank6464to all linear projections in the backbone\. This allows the backbone to reshape its hidden states to encode ranking\-relevant information while preserving pretrained knowledge\. As Section[5\.3](https://arxiv.org/html/2609.01807#S5.SS3)shows, this adaptation is the enabler: without it, even rich teacher signals cannot be absorbed by the scoring head\.

## 5Experiments

### 5\.1Setup

Internal dataset\.A proprietary re\-ranking dataset \(≈189\\approx 189K training slates,236236K test requests\), with up to150150items per slate\.Teacher:a well\-trained autoregressive generative ranking model \(0\.60\.6B\-parameter decoder\-only transformer, fine\-tuned for ranking\), evaluated both with and without an explicit reasoning trace\.

Amazon Beauty dataset\.The “beauty” products subset of the 2014 Amazon product dataset\([He and McAuley, 2016](https://arxiv.org/html/2609.01807#bib.bib28)\)\. Slates comprise 50 candidate items to rank, with 21,245 training and 1,118 evaluation slates\.Teacher:32B Qwen model used for ranking\. We also test a 0\.6B Qwen model for comparison\.

Student\.The same0\.60\.6B architecture, adapted via LoRA \(r=64r=64\)\. Trained viasftfrom the respective teacher for each dataset\.

Metrics\.AUC, Recall@\{1,10\}\\\{1,10\\\}, NDCG@11\.Hardware\.NVIDIA A100 80GB; LAPJV solver on CPU\.

### 5\.2Main Results

ModelLatencySpeed\-upDecodeAUCR@1R@10NDCG@1passesInternal DatasetTeacher \(with reasoning\)18071807ms1\.0×1\.0\\times7687680\.59110\.59110\.16340\.16340\.78770\.78770\.17760\.1776Teacher \(no reasoning\)8888ms20\.5×20\.5\\times39390\.5912\\mathbf\{0\.5912\}0\.16350\.16350\.78920\.78920\.17790\.1779hLLM\(ours\)𝟐𝟖\\mathbf\{28\}ms64\.5×\\mathbf\{64\.5\}\\boldsymbol\{\\times\}𝟏\\mathbf\{1\}0\.59070\.59070\.1652\\mathbf\{0\.1652\}0\.7912\\mathbf\{0\.7912\}0\.1791\\mathbf\{0\.1791\}Amazon Beauty DatasetQwen3\-32B50755075ms1\.0×1\.0\\times78780\.6292\\mathbf\{0\.6292\}0\.1384\\mathbf\{0\.1384\}0\.42320\.42320\.1384\\mathbf\{0\.1384\}Qwen3\-0\.6B921921ms5\.5×5\.5\\times26260\.51220\.51220\.02320\.02320\.20620\.20620\.01790\.0179hLLM0\.6B \(ours\)𝟏𝟏𝟑\\mathbf\{113\}ms44\.9×\\mathbf\{44\.9\}\\boldsymbol\{\\times\}𝟏\\mathbf\{1\}0\.61680\.61680\.13210\.13210\.4268\\mathbf\{0\.4268\}0\.13470\.1347Table 2:Main results\. Every row decodes the same output: theNNitem ordinals naming the ranked slate\.Decode passescounts the sequential forward passes required to do so; for autoregressive teachers under greedy decoding this equals the number of emitted tokens, whilehLLMdecodes the ordinals out of the prefill pass itself, so its count is constant inNN\. Speed\-up is relative to the teacher within each dataset\.We present the main results and a series of ablations below, note that ablations were performed on successive datasets\. Table[2](https://arxiv.org/html/2609.01807#S5.T2)compareshLLMagainst both teacher configurations\. Removing the reasoning trace reduces teacher latency from18071807ms to8888ms, a20×20\\timesreduction in decode passes, at essentially no reduction in quality, confirming that reasoning tokens are not load\-bearing for ranking\.hLLMdecodes the same ordinals in a single pass, reaching2828ms:64×64\\timesfaster than the reasoning teacher and3\.1×3\.1\\timesfaster than the no\-reasoning teacher\. Note what the second comparison isolates\. The no\-reasoning teacher has already stripped every non\-essential token and decodes only the ranking itself; the remaining3\.1×3\.1\\timesis attributable purely to the decoding strategy, not to shorter output\. Quality is lossless on list\-level metrics:hLLMis indistinguishable from both teachers on NDCG@1 \(0\.17910\.1791hLLMvs\.0\.17760\.1776/0\.17790\.1779\) and Recall@1 \(0\.16520\.1652hLLMvs\.0\.16340\.1634/0\.16350\.1635\)\. AUC is the metric wherehLLMretains99\.9%99\.9\\%of the teacher’s quality\. We attribute the list\-level gains to two factors: the Hungarian assignment enforces a globally consistent one\-to\-one ordering that autoregressive decoding only approximates, and distillation across the full training set averages out the teacher’s sampling noise \(Section[6\.1](https://arxiv.org/html/2609.01807#S6.SS1)\)\.

### 5\.3Ablation: Training Signal×\\timesBackbone Adaptation

Table 3:Signal richness interacts with backbone capacity\. Rich permutation targets*hurt*a frozen backbone \(Track 2<<Track 1\) but become the strongest signal once the backbone can adapt \(Track 3\)\.Table[3](https://arxiv.org/html/2609.01807#S5.T3)isolates the interaction between signal richness and backbone adaptability\. We find that freezing the backbone causes a significant penalty in ranking performance \(row 2\)\. Similarly, training on click labels only is not sufficient: the poor ranking performance indicates that the click\-signal is too sparse a signal for good learning \(row 1\)\. Neither ingredient suffices alone; only their combination \(Track 3\) succeeds\.

### 5\.4Ablation: Head Architecture

Table[4](https://arxiv.org/html/2609.01807#S5.T4)shows our ablations on the three different head architectures described\. The self\-attention architecture described above performs best\. Removing the self\-attention layers and replacing them with a linear probe reduces the ranking performance, as does adding learned position\-dependent embeddings\. On the backbone that is LoRA\-adapted, the self\-attention head is the strongest configuration, consistent with our interpretation in Section[6\.1](https://arxiv.org/html/2609.01807#S6.SS1)\.

Table 4:Ablation on the different scoring head architectures\.
### 5\.5Solver Overhead

Table 5:End\-to\-end latency decomposition\. The combinatorial solver contributes under0\.03%0\.03\\%of total latency\.Table[5](https://arxiv.org/html/2609.01807#S5.T5)decomposes end\-to\-end latency\. The backbone prefill accounts for essentially all of it; the scoring head adds under0\.10\.1ms and the LAPJV solver0\.0080\.008ms, under0\.03%0\.03\\%of the total\. This is the central efficiency argument: although the linear assignment problem isO⁡\(N3\)O\(N^\{3\}\)in the worst case, at slate sizes ofN≤50N\\leq 50it resolves in microseconds on CPU, fully overlapped by the GPU forward pass\. The apparent overhead of introducing a combinatorial solver is therefore illusory\. More importantly, the decomposition shows thathLLMhas moved the bottleneck from a*sequential*dependency \(autoregressive decode, which cannot be parallelized\) to a*parallel*one \(a single prefill over all candidates\)\. Decoding allNNordinals has an impact of0\.0080\.008ms of that budget\. Any further latency reduction must come from the backbone itself, via pruning, quantization, or early exit, rather than from the ranking machinery, which is already effectively free\.

## 6Analysis

### 6\.1The Capacity Gap

Reproducing a fullNN\-item permutation requires hidden states that encode fine\-grained relative preferences between every pair of items\. Click labels are binary and supply a simple gradient \(“push clicked items up”\)\. Teacher rankings demand that the score matrix reproduce exact orderings, which requires the backbone to encode pairwise preferences absent from its language\-modeling\-optimized representations\. LoRA closes this capacity gap with roughly44M trainable parameters on a600600M backbone\. This explains the counterintuitive ordering in Table[3](https://arxiv.org/html/2609.01807#S5.T3): a frozen backbone cannot reshape its representations to serve the richer target, so the richer signal becomes harmful rather than helpful\.

### 6\.2Structural Guarantees

By constructionhLLMyields \(i\)100%100\\%valid outputs, since the Hungarian algorithm always returns a valid permutation—the constraint is discharged inside the decoder rather than repaired after it; \(ii\) no degenerate solutions, since the one\-to\-one constraint forbids assigning multiple items to the same position; and \(iii\) deterministic inference, with no sampling temperature or beam\-search artifacts\. These properties are guaranteed rather than learned, and they account for part of the list\-level advantage over the autoregressive teacher, which can and does emit invalid or repeated orderings\.

## 7Conclusion

We presentedhLLM, a format\-specialized decoding strategy that decodes a complete ranking—allNNitem ordinals—inO⁡\(1\)O\(1\)forward passes, at2828ms end to end\. We formalize the use of the Hungarian algorithm to decode a ranking directly from hidden states\. We empirically validate this approach on a proprietary dataset and an open\-source benchmark, finding near\-lossless ranking quality compared to autoregressive LLMs, at a4545–64×64\\timesinference speedup\. We profile the performance of the different parts of the architecture and our systematic ablations decompose the contributions between the backbone and the self\-attention layers\.

#### Acknowledgments

The authors would like to thank Yufei Li for providing the upstream data assets and model checkpoints used in this work, and for helpful discussions on evaluation methodology\.

## References

- Berthetet al\.\(2020\)Q\. Berthet, M\. Blondel, O\. Teboul, M\. Cuturi, J\. Vert, and F\. BachLearning with differentiable perturbed optimizers\.InNeurIPS,Cited by:[§2\.5](https://arxiv.org/html/2609.01807#S2.SS5.p1.1)\.
- Caiet al\.\(2024\)T\. Cai, Y\. Li, Z\. Geng, H\. Peng, J\. D\. Lee, D\. Chen, and T\. DaoMedusa: simple llm inference acceleration framework with multiple decoding heads\.arXiv:2401\.10774\.Cited by:[§2\.6](https://arxiv.org/html/2609.01807#S2.SS6.p1.1)\.
- Carionet al\.\(2020\)N\. Carion, F\. Massa, G\. Synnaeve, N\. Usunier, A\. Kirillov, and S\. ZagoruykoEnd\-to\-end object detection with transformers\.InECCV,Cited by:[§2\.5](https://arxiv.org/html/2609.01807#S2.SS5.p1.1),[§4\.1](https://arxiv.org/html/2609.01807#S4.SS1.SSS0.Px3.p1.1)\.
- Cuturi \(2013\)M\. CuturiSinkhorn distances: lightspeed computation of optimal transport\.InNeurIPS,Cited by:[§2\.3](https://arxiv.org/html/2609.01807#S2.SS3.p1.1),[§3\.3](https://arxiv.org/html/2609.01807#S3.SS3.p1.1)\.
- De Caoet al\.\(2021\)N\. De Cao, G\. Izacard, S\. Riedel, and F\. PetroniAutoregressive entity retrieval\.InICLR,Cited by:[§2\.1](https://arxiv.org/html/2609.01807#S2.SS1.p1.1)\.
- Düttinget al\.\(2019\)P\. Dütting, Z\. Feng, H\. Narasimhan, D\. C\. Parkes, and S\. S\. RavindranathOptimal auctions through deep learning\.InICML,Cited by:[§2\.5](https://arxiv.org/html/2609.01807#S2.SS5.p1.1)\.
- Elhoushiet al\.\(2024\)M\. Elhoushi, A\. Shrivastava, D\. Liskovich, B\. Hosmer, B\. Wasti, L\. Lai, A\. Mahmoud, B\. Acun, S\. Agarwal, A\. Roman, A\. A\. Aly, B\. Chen, and C\. WuLayerSkip: enabling early exit inference and self\-speculative decoding\.arXiv:2404\.16710\.Cited by:[§2\.6](https://arxiv.org/html/2609.01807#S2.SS6.p1.1)\.
- Ghazvininejadet al\.\(2019\)M\. Ghazvininejad, O\. Levy, Y\. Liu, and L\. ZettlemoyerMask\-predict: parallel decoding of conditional masked language models\.InEMNLP,Cited by:[§2\.2](https://arxiv.org/html/2609.01807#S2.SS2.p1.1)\.
- Guet al\.\(2018\)J\. Gu, J\. Bradbury, C\. Xiong, V\. O\.K\. Li, and R\. SocherNon\-autoregressive neural machine translation\.InICLR,Cited by:[§2\.2](https://arxiv.org/html/2609.01807#S2.SS2.p1.1)\.
- He and McAuley \(2016\)R\. He and J\. McAuleyUps and downs: modeling the visual evolution of fashion trends with one\-class collaborative filtering\.InProceedings of the 25th International Conference on World Wide Web,pp\. 507–517\.Cited by:[item Amazon Beauty dataset\.](https://arxiv.org/html/2609.01807#S5.I1.ix2.p1.1)\.
- Hintonet al\.\(2015\)G\. Hinton, O\. Vinyals, and J\. DeanDistilling the knowledge in a neural network\.arXiv:1503\.02531\.Cited by:[§2\.4](https://arxiv.org/html/2609.01807#S2.SS4.p1.1)\.
- Hofstätteret al\.\(2020\)S\. Hofstätter, S\. Althammer, M\. Schröder, M\. Sertkan, and A\. HanburyImproving efficient neural ranking models with cross\-architecture knowledge distillation\.arXiv:2010\.02666\.Cited by:[§2\.4](https://arxiv.org/html/2609.01807#S2.SS4.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.arXiv:2106\.09685\.Cited by:[§4\.4](https://arxiv.org/html/2609.01807#S4.SS4.p1.1)\.
- Jonker and Volgenant \(1987\)R\. Jonker and A\. VolgenantA shortest augmenting path algorithm for dense and sparse linear assignment problems\.Computing38,pp\. 325–340\.Cited by:[§3\.2](https://arxiv.org/html/2609.01807#S3.SS2.p1.2)\.
- Kuhn \(1955\)H\. W\. KuhnThe hungarian method for the assignment problem\.Naval Research Logistics Quarterly2,pp\. 83–97\.Cited by:[§1](https://arxiv.org/html/2609.01807#S1.SS0.SSS0.Px1.p3.1),[§3\.2](https://arxiv.org/html/2609.01807#S3.SS2.p1.2)\.
- Leviathanet al\.\(2023\)Y\. Leviathan, M\. Kalman, and Y\. MatiasFast inference from transformers via speculative decoding\.InICML,Cited by:[§2\.6](https://arxiv.org/html/2609.01807#S2.SS6.p1.1)\.
- Liet al\.\(2024\)Y\. Li, F\. Wei, C\. Zhang, and H\. ZhangEAGLE: speculative sampling requires rethinking feature uncertainty\.arXiv:2401\.15077\.Cited by:[§2\.6](https://arxiv.org/html/2609.01807#S2.SS6.p1.1)\.
- Liuet al\.\(2026\)Q\. Liu, K\. Ai, J\. Mao, Y\. Zhang, M\. Li, D\. Long, P\. Xie, F\. Zhu, and J\. WenDiffuRank: effective document reranking with diffusion language models\.External Links:2602\.12528,[Link](https://arxiv.org/abs/2602.12528)Cited by:[§1](https://arxiv.org/html/2609.01807#S1.SS0.SSS0.Px1.p1.1),[§2\.5](https://arxiv.org/html/2609.01807#S2.SS5.p1.1)\.
- Menaet al\.\(2018\)G\. Mena, D\. Belanger, S\. Linderman, and J\. SnoekLearning latent permutations with gumbel\-sinkhorn networks\.InICLR,Cited by:[§2\.3](https://arxiv.org/html/2609.01807#S2.SS3.p1.1),[§3\.3](https://arxiv.org/html/2609.01807#S3.SS3.p1.1)\.
- Petersenet al\.\(2021\)F\. Petersen, C\. Borgelt, H\. Kuehne, and O\. DeussenDifferentiable sorting networks for scalable sorting and ranking supervision\.InICML,Cited by:[§2\.3](https://arxiv.org/html/2609.01807#S2.SS3.p1.1)\.
- Pradeepet al\.\(2023\)R\. Pradeep, S\. Sharifymoghaddam, and J\. LinRankZephyr: effective and robust zero\-shot listwise reranking is a breeze\!\.arXiv:2312\.02724\.Cited by:[§2\.4](https://arxiv.org/html/2609.01807#S2.SS4.p1.1)\.
- Prillo and Eisenschlos \(2020\)S\. Prillo and J\. M\. EisenschlosSoftSort: a continuous relaxation for the argsort operator\.InICML,Cited by:[§2\.3](https://arxiv.org/html/2609.01807#S2.SS3.p1.1)\.
- Qianet al\.\(2021\)L\. Qian, H\. Zhou, Y\. Bao, M\. Wang, L\. Qiu, W\. Zhang, Y\. Yu, and L\. LiGlancing transformer for non\-autoregressive neural machine translation\.InACL,Cited by:[§2\.2](https://arxiv.org/html/2609.01807#S2.SS2.p1.1)\.
- Reddyet al\.\(2024\)R\. G\. Reddy, J\. Doo, Y\. Xu, M\. A\. Sultan, D\. Swain, A\. Sil, and H\. JiFIRST: faster improved listwise reranking with single token decoding\.InEMNLP,Cited by:[§2\.1](https://arxiv.org/html/2609.01807#S2.SS1.p1.1)\.
- Renet al\.\(2024\)Y\. Ren, Q\. Yang, Y\. Wu, W\. Xu, Y\. Wang, and Z\. ZhangNon\-autoregressive generative models for reranking recommendation\.InKDD,Cited by:[§1](https://arxiv.org/html/2609.01807#S1.SS0.SSS0.Px1.p1.1)\.
- Sunet al\.\(2023\)W\. Sun, L\. Yan, X\. Ma, P\. Ren, D\. Yin, and Z\. RenIs chatgpt good at search? investigating large language models as re\-ranking agents\.arXiv:2304\.09542\.Cited by:[§1](https://arxiv.org/html/2609.01807#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.01807#S2.SS1.p1.1)\.
- Tayet al\.\(2022\)Y\. Tay, V\. Q\. Tran, M\. Dehghani, J\. Ni, D\. Bahri, H\. Mehta, Z\. Qin, K\. Hui, Z\. Zhao, J\. Gupta, T\. Schuster, W\. W\. Cohen, and D\. MetzlerTransformer memory as a differentiable search index\.InNeurIPS,Cited by:[§2\.1](https://arxiv.org/html/2609.01807#S2.SS1.p1.1)\.
- Zhenget al\.\(2025\)Z\. Zheng, Z\. Wang, F\. Yang, W\. Ye, W\. Huang, W\. He, T\. Zhang, and X\. WangNGA: non\-autoregressive generative auction with global externalities for advertising systems\.arXiv:2506\.05685\.Cited by:[§2\.5](https://arxiv.org/html/2609.01807#S2.SS5.p1.1)\.