Retrieval, Scoring, and Decoding Shape Performance and Stability in LLM-based Conversational Recommendation

# Retrieval, Scoring, and Decoding Shape Performance and Stability in LLM-based Conversational Recommendation
Source: [https://arxiv.org/html/2609.00086](https://arxiv.org/html/2609.00086)
Conference:Proceedings of the 35th ACM International Conference on Information and Knowledge Management; November 07–11, 2026; Rome, ItalyProceedings of the 35th ACM International Conference on Information and Knowledge Management \(CIKM ’26\), November 07–11, 2026, Rome, ItalyDOI:[10\.1145/3799682\.3840066](https://doi.org/10.1145/3799682.3840066)ISBN:979\-8\-4007\-2539\-5/2026/11CCS:Information systems Recommender systemsCCS:Information systems Retrieval models and rankingCCS:Information systems Evaluation of retrieval results,Tomislav Duricicemail:[tduricic@infobip\.com](mailto:[email protected])Affiliation:Infobip,Zagreb,Croatia,Andro Mercepemail:[amercep@infobip\.com](mailto:[email protected])Affiliation:Infobip,Zagreb,CroatiaandEmanuel Lacicemail:[emlacic@infobip\.com](mailto:[email protected])Affiliation:Infobip,Zagreb,Croatia

© cc

###### Abstract\.

Large language models \(LLMs\) are increasingly used as rerankers in conversational recommender systems, yet measured gains depend strongly on the retrieval and inference protocol\. On the ReDial conversational movie recommendation benchmark, we compare proprietary, open\-weight, and fine\-tuned LLM rerankers with collaborative\-filtering and sequential baselines in a shared retrieve\-then\-rerank pipeline\. We vary candidate\-pool size, first\-stage retriever, and decoding temperature\. With a shared semantic top\-250 candidate pool and strict candidate\-aware scoring, the best proprietary reranker reaches NDCG@10 of 0\.1497, compared with 0\.0939 for the strongest non\-LLM baseline\. The same reranker reaches 0\.2925 in zero\-shot generation, showing that unconstrained scoring can yield a much larger apparent advantage than matched\-pool evaluation\. No evaluated open\-weight LLM outperforms the tuned shallow autoencoder baseline under this protocol\. For the strongest proprietary and open\-weight rerankers, switching from semantic to collaborative\-filtering candidates raises NDCG@10 by more than 50%, showing that measured reranker performance is highly sensitive to candidate generation\. For the best proprietary reranker, raising temperature from 0 to 1\.0 increases top\-10 Jaccard distance from 0\.0900 to 0\.1240 while mean NDCG@10 changes negligibly, whereas weaker LLMs show larger degradation\. These ReDial results support treating candidate generation, candidate\-pool size, scoring policy, and decoding configuration as required reporting fields rather than implementation details\.

###### Keywords:

conversational recommender systems, large language models, reranking, candidate generation, evaluation, stability

††cc\-license:by## 1\.Introduction

Conversational recommender systems \(CRS\) infer preferences from dialogue rather than from dense interaction histories\([Jannach et al\., 2021](https://arxiv.org/html/2609.00086#bib.bib12)\)\. Large language models \(LLMs\) have shown promising zero\-shot ranking ability on recommendation tasks\([Hou et al\., 2024](https://arxiv.org/html/2609.00086#bib.bib11)\)and can also be fine\-tuned for the role\([Bao et al\., 2023](https://arxiv.org/html/2609.00086#bib.bib3)\), which is why recent CRS work increasingly relies on them at the ranking stage\. Reported gains over earlier baselines are often substantial, but those numbers depend on pipeline choices that many tend to under\-report, i\.e\., which items are retrieved as candidates, how many of them are shown to the LLM, whether evaluation restricts credit to the candidate set or scores any generated catalog title, and how the model is decoded\. On a single benchmark, these choices can move ranking quality by more than the gap separating top systems, so the same model can appear strong or mediocre depending on how it is evaluated\.

Two\-stage retrieve\-then\-rerank is standard for LLM\-based ranking\([Sun et al\., 2023](https://arxiv.org/html/2609.00086#bib.bib27)\), and recent LLM rerankers add user\-preference retrieval or graph signals on top of an explicit candidate stage\([Zhang et al\., 2025](https://arxiv.org/html/2609.00086#bib.bib31);[Wei et al\., 2024](https://arxiv.org/html/2609.00086#bib.bib28)\)\. Yet, LLM CRS comparisons are seldom controlled for candidate generation, and reproducibility studies show that recommender comparisons are sensitive to such protocol choices as well as to baseline strength\([Dacrema et al\., 2019](https://arxiv.org/html/2609.00086#bib.bib5);[Krichene and Rendle, 2020](https://arxiv.org/html/2609.00086#bib.bib15)\)\. Stochastic decoding further makes LLM outputs variable across repeated samples and sensitive to the order in which candidates appear in the prompt\([Ma et al\., 2025](https://arxiv.org/html/2609.00086#bib.bib18);[Bito et al\., 2026](https://arxiv.org/html/2609.00086#bib.bib4)\), yet list\-level stability is rarely reported alongside accuracy in LLM CRS evaluations\.

We study how these factors jointly shape observed effectiveness on ReDial\([Li et al\., 2018](https://arxiv.org/html/2609.00086#bib.bib17)\), a CRS benchmark of seeker\-recommender movie dialogues in which preferences must be inferred from the conversation alone\. We compare proprietary, open\-weight, and fine\-tuned LLM rerankers against collaborative\-filtering \(CF\) and sequential rerankers in a shared two\-stage pipeline11footnotetext:Equal contribution\.111Code, prompts, configurations, and outputs:[https://github\.com/infobip/crs\-performance](https://github.com/infobip/crs-performance)\. Matched\-pool comparisons hold the retrieved candidates fixed while we vary candidate\-pool size, first\-stage retriever, and decoding temperature\. We answer the following research questions:

- RQ1Under a matched semantic candidate pool, how do LLM rerankers compare to CF and sequential rerankers?
- RQ2How sensitive is LLM reranking quality to candidate\-pool size, from zero\-shot generation to full\-catalog reranking?
- RQ3How does the choice of first\-stage retriever \(i\.e\., semantic, CF, or sequential\) affect the quality of an LLM reranker?
- RQ4How stable are LLM recommendation lists under decoding\-temperature sampling, in terms of both ranking quality and list\-level agreement?

Four proprietary LLMs significantly outperform EASE \(0\.0939\), led by Claude Opus 4\.6 at 0\.1497, while every evaluated open\-weight reranker falls below it\. Expanding the semantic pool from 250 items to the full catalog raises proprietary NDCG@10 by 57–89%\. Replacing semantic with EASE candidates raises NDCG@10 by 52–59% across the two tested rerankers\. With higher temperature, Claude Opus 4\.6 maintains mean accuracy while Jaccard distance@10 rises from 0\.0900 to 0\.1240, whereas for Llama\-3\.3\-70B it rises from 0\.0230 to 0\.7600\. These results identify retrieval strategy, candidate\-pool size, scoring policy, and decoding configuration as core experimental variables\.

## 2\.Methodology

### 2\.1\.Dataset and Task

We evaluate on ReDial’s standard test split \(1,025 dialogues\) and 6,924\-movie catalog\([Li et al\., 2018](https://arxiv.org/html/2609.00086#bib.bib17)\)\. For each dialogue, accepted recommendations are masked and used as ground\-truth targets\. Each model returns ten ranked movie titles from family\-specific inputs\.

LLM prompts contain liked titles, the masked dialogue, and, outside zero\-shot generation, a shuffled candidate list\. We label settings by candidate count: cKKdenotesKKcandidates, c0 no candidate list, and cAll the full catalog\. Candidate\-aware prompts require outputs to use only listed titles\. For CF models, liked movies form the implicit interaction history, while accepted recommendations define validation and test targets\. For sequential models, we preserve item order by placing liked movies before accepted recommendations and expanding these histories into the pre\-augmented RecBole sequence format\([Zhao et al\., 2021](https://arxiv.org/html/2609.00086#bib.bib32)\)\. The targets stay fixed while the input representation matches each model family\. Matching pools controls item availability but not model information: LLMs use raw dialogue and pretrained knowledge, whereas CF and sequential models use interaction histories\. Our comparisons therefore evaluate systems rather than reranking under identical information\.

### 2\.2\.Two\-stage Pipeline and Candidate Generation

We use a two\-stage pipeline with two roles: a candidate generator returns a poolCuKC\_\{u\}^\{K\}ofKKcatalog items for dialogueuu, and a reranker orders items from this pool and emits the top\-10 recommendation list\. The same model can fill either role\. In our experiments, CF and sequential models act as rerankers over a matched semantic pool and also produce top\-KKpools that LLMs then rerank\.

The primary candidate generator is content\-based filtering \(CBF\) over movie metadata\. We embed each catalog item with all\-mpnet\-base\-v2, a Sentence\-BERT model\([Reimers and Gurevych, 2019](https://arxiv.org/html/2609.00086#bib.bib23)\), and L2\-normalize the resulting vectors𝐞i\\mathbf\{e\}\_\{i\}\. For dialogueuuwith liked setLuL\_\{u\}and disliked setDuD\_\{u\}, we form centroids

\(1\)𝐜u\+=norm⁡\(1\|Lu\|​∑j∈Lu𝐞j\),𝐜u−=norm⁡\(1\|Du\|​∑j∈Du𝐞j\),\\displaystyle\\mathbf\{c\}^\{\+\}\_\{u\}=\\mathrm\{norm\}\\left\(\\frac\{1\}\{\|L\_\{u\}\|\}\\sum\_\{j\\in L\_\{u\}\}\\mathbf\{e\}\_\{j\}\\right\),\\qquad\\mathbf\{c\}^\{\-\}\_\{u\}=\\mathrm\{norm\}\\left\(\\frac\{1\}\{\|D\_\{u\}\|\}\\sum\_\{j\\in D\_\{u\}\}\\mathbf\{e\}\_\{j\}\\right\),and score each catalog itemi∉Lu∪Dui\\notin L\_\{u\}\\cup D\_\{u\}by

\(2\)su,i=λ\+​⟨𝐞i,𝐜u\+⟩−λ−​⟨𝐞i,𝐜u−⟩,\\displaystyle s\_\{u,i\}=\\lambda\_\{\+\}\\left\\langle\\mathbf\{e\}\_\{i\},\\,\\mathbf\{c\}^\{\+\}\_\{u\}\\right\\rangle\\;\-\\;\\lambda\_\{\-\}\\left\\langle\\mathbf\{e\}\_\{i\},\\,\\mathbf\{c\}^\{\-\}\_\{u\}\\right\\rangle,withλ\+=1\\lambda\_\{\+\}=1andλ−=0\.5\\lambda\_\{\-\}=0\.5\. The negative term is dropped whenDuD\_\{u\}is empty\. The top\-KKitems bysu,is\_\{u,i\}formCuKC\_\{u\}^\{K\}, which we shuffle before insertion into the<CANDIDATES\>block of LLM prompts to reduce prompt\-position effects\.

CBF pools of sizeK=250K=250drive the main reranker comparison\. For candidate\-pool sensitivity, we also evaluate zero\-shot generation \(no candidate list\), CBF pools atK∈\{500,1000\}K\\in\\\{500,1000\\\}, and full\-catalog reranking, where all 6,924 catalog titles are placed in a single prompt\. Additionally, we build top\-250 pools with EASE as the CF retriever and SASRec as the sequential retriever for the same LLM rerankers, isolating first\-stage effects\.

### 2\.3\.Rerankers, Inference, and Evaluation

#### Rerankers

We compare proprietary API LLMs, open\-weight LLMs, and a fine\-tuned open\-weight LLM against CF and sequential rerankers\. LLM rerankers receive the dialogue context and candidate titles then generate an ordered list of movie titles\. CF rerankers score the same candidate items using collaborative signals learned from the RecBole interaction data\([Zhao et al\., 2021](https://arxiv.org/html/2609.00086#bib.bib32)\)\. Sequential rerankers score candidates using the corresponding ordered dialogue/user sequence\. We also include the unreranked CBF order and a popularity baseline\. The fine\-tuned reranker is Qwen2\.5\-7B\-Instruct adapted with LoRA \(rank 16,α=32\\alpha=32, 3 epochs, learning rate×10−52\\\!\\times\\\!10^\{\-5\}\) on the c250 training prompts\. For the main reranking runs, LLMs are decoded with temperature 0 and top\-p left at its default of 1\.0\. The repository contains full prompt templates, exact provider model identifiers, run configurations, outputs, and parsing code\.

#### Metrics

Our primary metric is NDCG@10\([Järvelin and Kekäläinen, 2002](https://arxiv.org/html/2609.00086#bib.bib13)\)with binary relevance over the accepted target movies\. We also report Hit@10, item coverage@10, and average training\-set popularity@10\. LLM outputs are parsed as ordered title lists and matched to ReDial catalog items after Unicode\-normalized, lowercased title matching with trailing punctuation stripped\. Titles that cannot be matched to a catalog item are kept in the raw artifact but receive no metric credit\. In candidate\-constrained settings, the prompt instructs the LLM to choose only from the candidate list\. Generated titles outside that list receive no metric credit, even if they match a catalog item\.

#### Retrieval diagnostics and significance

CandRecall@250 is the mean fraction of ground\-truth items retrieved into the top\-250 candidate pool\. Oracle NDCG@10 is the best NDCG@10 attainable by a reranker that can only rank items from that pool\. Confidence intervals use bootstrap resampling over dialogue\-level examples\. For the main reranker comparison, we test NDCG@10 against EASE with paired Wilcoxon signed\-rank tests and Holm correction\([Holm, 1979](https://arxiv.org/html/2609.00086#bib.bib10)\)\.

### 2\.4\.Temperature sensitivity and list stability

For each LLM reranker we run 20 generations per prompt at each temperature in\{0,0\.5,1\.0,2\.0\}\\\{0,0\.5,1\.0,2\.0\\\}over a fixed 30\-prompt subset of the c250 test set, with top\-p=1\.0p=1\.0\. Anthropic models \(Claude Opus 4\.6, Claude Sonnet 4\.6\) are capped atT=1\.0T=1\.0by the provider\. We report mean NDCG@10 and dialogue\-level variation across repeated generations\. List changes are measured with Jaccard distance@10 over the top\-10 sets and position disagreement@10, the mean fraction of rank\-aligned positions whose items differ across paired generations\.

## 3\.Results

The results show three main patterns\. Proprietary LLMs lead under a fixed semantic candidate pool, first\-stage retrieval can change NDCG@10 as much as model choice, and decoding temperature affects list stability more than average ranking quality\.

Table 1\.Reranker effectiveness on ReDial under zero\-shot scoring \(c0, no candidate list\) and strict candidate\-aware scoring over the semantic top\-250 pool \(c250\)\. Rows group LLM, collaborative\-filtering, sequential, and popularity rerankers\.Cov\. is item coverage of the top\-10 list\. Pop\. is mean training\-set popularity of top\-10 items\.∗marks NDCG@10 significantly above the EASE c250 baseline by paired Wilcoxon signed\-rank test with Holm correction \(p<0\.05p<0\.05\)\. c0 and c250 LLM scores are tested separately against that baseline\. CF, sequential, and popularity baselines require a candidate set and have no c0 entry\. Qwen2\.5\-7B\-FT was fine\-tuned on c250 prompts and is reported only for that policy\.

#### RQ1\.

Under the matched semantic top\-250 pool \(Table[1](https://arxiv.org/html/2609.00086#S3.T1)\), Claude Opus 4\.6 achieves the highest strict NDCG@10 at 0\.1497, with GPT\-5\.2, Claude Sonnet 4\.6, and GPT\-4\.1 clustered behind between 0\.1283 and 0\.1335\. These four proprietary models are the only rerankers significantly above EASE \(0\.0939\) after Holm correction\. GPT\-4\.1 Mini is numerically higher than EASE but not significant, and every open\-weight reranker falls below EASE\. Zero\-shot scoring changes this picture\. Claude Opus 4\.6 reaches 0\.2925, nearly twice its strict c250 score, and additional models become significant against the EASE baseline\. Table[1](https://arxiv.org/html/2609.00086#S3.T1)also shows differences in coverage and popularity\. Under c250, GPT\-4\.1 and GPT\-4\.1 Mini have the highest coverage among proprietary LLMs, while EASE has lower coverage and higher average popularity\. The strongest proprietary rerankers improve NDCG without relying only on the most popular items\. Open\-weight models show mixed behavior: some cover a broad part of the catalog but still rank less accurately\.

Figure 1\.NDCG@10 as a function of candidate\-pool size across LLM rerankers with semantic retrieval\. Settings span c0 \(zero\-shot, no candidates\), strict candidate\-aware top\-kkscoring at c250, c500, and c1000, and full\-catalog \(cAll\)\.Line plot of NDCG at 10 across c0, c250, c500, c1000, and cAll candidate settings for the evaluated LLMs\.
#### RQ2\.

Figure[1](https://arxiv.org/html/2609.00086#acmlabel1)shows how candidate access reshapes the same rerankers\. Every proprietary model gains substantially as the pool expands\. Strict NDCG@10 rises by 57% \(GPT\-4\.1 Mini\) to 89% \(GPT\-5\.2\) between c250 and cAll, and cAll is the best strict setting for each\. c0, scored without a candidate list, lands in the same range and slightly exceeds cAll for Opus \(0\.2925 vs 0\.2644\) and GPT\-4\.1 Mini \(0\.1937 vs 0\.1600\)\. Open\-weight rerankers do not benefit similarly, with NDCG dropping from c250 to c500 for every evaluated open\-weight model\. Because pool size simultaneously changes target availability, candidate composition, prompt length, and ranking difficulty, this experiment measures their combined pipeline effect rather than isolating reranking ability\.

Table 2\.Effect of the first\-stage retriever on LLM reranking with a fixed top\-250 candidate pool\.CandR@250 is the mean fraction of ground\-truth movies retrieved into the candidate pool\. Oracle NDCG@10 is the best NDCG@10 achievable by a perfect reranker restricted to that pool, and upper\-bounds the downstream reranker columns\.

#### RQ3\.

Table[2](https://arxiv.org/html/2609.00086#S3.T2)changes the first\-stage retriever while holding candidate\-pool size and reranker fixed\. Semantic retrieval places only 28\.8% of relevant items in the top\-250 pool, versus 48\.6% for EASE and 50\.9% for SASRec\. Oracle NDCG@10 follows the same order\. Retriever choice carries through to the reranker\. Switching from semantic to EASE lifts Claude Opus 4\.6 by 52% \(0\.1497 to 0\.2277\) and Llama\-3\.3\-70B by 59% \(0\.0770 to 0\.1223\)\. SASRec has the highest recall and oracle ceiling, while EASE yields the best NDCG for both rerankers\. Retrieval opportunity alone therefore does not determine downstream performance, though this design does not separate candidate composition from reranker–pool compatibility\.

Figure 2\.NDCG@10 \(A\) and Jaccard distance@10 \(B\) as functions of decoding temperature on the fixed semantic c250 stability subset of 30 prompts\.Two\-panel line plot showing strict NDCG at 10 and Jaccard distance at 10 as a function of decoding temperature for all evaluated LLM rerankers\.
#### RQ4\.

Figure[2](https://arxiv.org/html/2609.00086#acmlabel2)reports the stability subset of 30 prompts, so absolute NDCG levels are not comparable to the full\-test numbers in Table[1](https://arxiv.org/html/2609.00086#S3.T1)\. Across API models, mean strict NDCG@10 is roughly flat in temperature while Jaccard distance@10 rises with each step\. The Anthropic models cap atT=1\.0T=1\.0by the provider, so their traces stop there\. Claude Opus 4\.6 is the most stable\. Its Jaccard distance moves from 0\.0900 atT=0T=0to 0\.1240 atT=1\.0T=1\.0, position disagreement@10 rises from 0\.2700 to 0\.3500, and NDCG@10 changes little\. Open\-weight rerankers are less stable\. Llama\-3\.3\-70B begins almost deterministic atT=0T=0\(Jaccard 0\.0230, position disagreement 0\.0250\) and reaches 0\.7600 and 0\.7930 atT=2\.0T=2\.0\. Its NDCG@10 also falls from 0\.0750 to 0\.0490\.

## 4\.Discussion and Conclusion

#### Retrieval choices are as important as the reranker\.

Switching first\-stage retrievers improves NDCG@10 by 52–59% across both rerankers, while larger candidate pools also substantially improve proprietary models\. Yet the higher\-recall SASRec pool trails EASE after reranking, showing that candidate availability, composition, and reranker–pool compatibility jointly determine downstream quality\. Treating candidate generation as an implementation detail thus conflates pipeline and reranker quality\.

#### Candidate access and scoring policy shape headline LLM gains\.

Claude Opus 4\.6 reaches 0\.2925 in zero\-shot generation but 0\.1497 under strict candidate\-aware scoring on the matched semantic top\-250 pool\. Because c0 and c250 differ in both candidate access and scoring policy, this contrast does not isolate either factor\. It instead shows how strongly the evaluation protocol affects the apparent advantage over EASE\. Under the matched c250 pool, only four proprietary models significantly outperform EASE, while every open\-weight reranker falls below it\. These are system\-level comparisons rather than tests of reranking under identical information\.

#### Stability is a separate axis from accuracy\.

Mean accuracy and list identity respond differently to decoding temperature\. For the strongest proprietary models, NDCG@10 changes little between temperature 0 and 1, while Jaccard distance@10 and position disagreement@10 increase\. The model can therefore return lists with similar accuracy but different items and rankings\. Llama\-3\.3\-70B is more temperature\-sensitive, with Jaccard distance@10 rising from 0\.0230 to 0\.7600 and NDCG@10 falling from 0\.0750 to 0\.0490\. Reporting only mean NDCG hides this deployment\-relevant behavior\.

#### Scope and future work\.

Our experiments use ReDial, a single movie\-domain CRS benchmark with a 6,924\-item catalog\. Future work will test whether these findings generalize to larger catalogs, other domains, and conversational settings\. We did not systematically measure cost or latency\. Future evaluations should compare quality, cost, and latency, particularly for large and full\-catalog candidate pools\.

#### Toward a reporting norm\.

LLM\-based CRS evaluations should report the retriever, candidate\-pool size, scoring policy for off\-candidate and unmatched generations, decoding configuration, prompt template, and exact model and provider identifier\. Headline results should be interpreted together with this protocol because pipeline choices can affect measured gains as much as model choice\.

###### Acknowledgements\.

This research was supported in part by the project Infobip Global Communication Platform \(PK\.1\.1\.07\.0001\), part of the Important Project of Common European Interest on Next Generation Cloud Infrastructure and Services \(IPCEI\-CIS\) consortium\.

## GenAI Usage Disclosure

Generative AI tools assisted in a supporting capacity: Claude Code \(Opus 4\.7\) for code implementation and data analysis, ChatGPT 5\.5 forLaTeXediting and grammar checking, and Google Scholar Labs for related\-work identification\. All AI\-assisted outputs were reviewed and verified by the authors, who take full responsibility for the work\.

## References

- Anthropic \(2026a\)AnthropicClaude Opus 4\.6 System Card\.Anthropic\.Note:Technical reportAccessed: 2026\-06\-04External Links:[Link](https://www.anthropic.com/claude-opus-4-6-system-card)Cited by:[Table 1](https://arxiv.org/html/2609.00086#S3.T1.4.6.1.1)\.
- Anthropic \(2026b\)AnthropicClaude Sonnet 4\.6 System Card\.Anthropic\.Note:Technical reportAccessed: 2026\-06\-05External Links:[Link](https://www-cdn.anthropic.com/78073f739564e986ff3e28522761a7a0b4484f84.pdf)Cited by:[Table 1](https://arxiv.org/html/2609.00086#S3.T1.4.8.1.1)\.
- Baoet al\.\(2023\)K\. Bao, J\. Zhang, Y\. Zhang, W\. Wang, F\. Feng, and X\. HeTALLRec: an effective and efficient tuning framework to align large language model with recommendation\.InProceedings of the 17th ACM Conference on Recommender Systems,RecSys ’23,New York, NY, USA,pp\. 1007–1014\.External Links:[Document](https://dx.doi.org/10.1145/3604915.3608857),[Link](https://doi.org/10.1145/3604915.3608857)Cited by:[§1](https://arxiv.org/html/2609.00086#S1.p1.1)\.
- Bitoet al\.\(2026\)E\. Bito, Y\. Ren, and E\. HeOne pass, any order: position\-invariant listwise reranking for LLM\-based recommendation\.Note:arXiv preprint arXiv:2604\.27599To appear in SIGIR ’26External Links:2604\.27599,[Document](https://dx.doi.org/10.48550/arXiv.2604.27599),[Link](https://arxiv.org/abs/2604.27599)Cited by:[§1](https://arxiv.org/html/2609.00086#S1.p2.1)\.
- Dacremaet al\.\(2019\)M\. F\. Dacrema, P\. Cremonesi, and D\. JannachAre we really making much progress? a worrying analysis of recent neural recommendation approaches\.InProceedings of the 13th ACM Conference on Recommender Systems,RecSys ’19,New York, NY, USA,pp\. 101–109\.External Links:[Document](https://dx.doi.org/10.1145/3298689.3347058),[Link](https://doi.org/10.1145/3298689.3347058)Cited by:[§1](https://arxiv.org/html/2609.00086#S1.p2.1)\.
- Gemma Team \(2024\)Gemma TeamGemma 2: improving open language models at a practical size\.Note:arXiv preprint arXiv:2408\.00118External Links:2408\.00118,[Document](https://dx.doi.org/10.48550/arXiv.2408.00118),[Link](https://arxiv.org/abs/2408.00118)Cited by:[Table 1](https://arxiv.org/html/2609.00086#S3.T1.4.14.1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan, A\. Yang, A\. Fan, A\. Goyal, A\. Hartshorn, A\. Yang, A\. Mitra, A\. Sravankumar, A\. Korenev, A\. Hinsvark, A\. Rao, A\. Zhang, A\. Rodriguez, A\. Gregerson, A\. Spataru, B\. Rozière,et al\.The llama 3 herd of models\.Note:arXiv preprint arXiv:2407\.21783External Links:2407\.21783,[Document](https://dx.doi.org/10.48550/arXiv.2407.21783),[Link](https://arxiv.org/abs/2407.21783)Cited by:[Table 1](https://arxiv.org/html/2609.00086#S3.T1.4.15.1.1)\.
- Heet al\.\(2020\)X\. He, K\. Deng, X\. Wang, Y\. Li, Y\. Zhang, and M\. WangLightGCN: simplifying and powering graph convolution network for recommendation\.InProceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval,SIGIR ’20,New York, NY, USA,pp\. 639–648\.External Links:[Document](https://dx.doi.org/10.1145/3397271.3401063),[Link](https://doi.org/10.1145/3397271.3401063)Cited by:[Table 1](https://arxiv.org/html/2609.00086#S3.T1.4.21.1.1)\.
- Hidasiet al\.\(2016\)B\. Hidasi, A\. Karatzoglou, L\. Baltrunas, and D\. TikkSession\-based recommendations with recurrent neural networks\.Note:4th International Conference on Learning Representations \(ICLR 2016\), San Juan, Puerto RicoExternal Links:1511\.06939,[Link](https://arxiv.org/abs/1511.06939)Cited by:[Table 1](https://arxiv.org/html/2609.00086#S3.T1.4.25.1.1)\.
- Holm \(1979\)S\. HolmA simple sequentially rejective multiple test procedure\.Scandinavian Journal of Statistics6\(2\),pp\. 65–70\.Cited by:[§2\.3](https://arxiv.org/html/2609.00086#S2.SS3.SSS0.Px3.p1.1)\.
- Houet al\.\(2024\)Y\. Hou, J\. Zhang, Z\. Lin, H\. Lu, R\. Xie, J\. McAuley, and W\. X\. ZhaoLarge language models are zero\-shot rankers for recommender systems\.InAdvances in Information Retrieval,Lecture Notes in Computer Science, Vol\.14609,Cham, Switzerland,pp\. 364–381\.External Links:[Document](https://dx.doi.org/10.1007/978-3-031-56060-6%5F24),[Link](https://doi.org/10.1007/978-3-031-56060-6_24)Cited by:[§1](https://arxiv.org/html/2609.00086#S1.p1.1)\.
- Jannachet al\.\(2021\)D\. Jannach, A\. Manzoor, W\. Cai, and L\. ChenA survey on conversational recommender systems\.ACM Computing Surveys54\(5\),pp\. 1–36\.External Links:[Document](https://dx.doi.org/10.1145/3453154),[Link](https://doi.org/10.1145/3453154)Cited by:[§1](https://arxiv.org/html/2609.00086#S1.p1.1)\.
- Järvelin and Kekäläinen \(2002\)K\. Järvelin and J\. KekäläinenCumulated gain\-based evaluation of IR techniques\.ACM Transactions on Information Systems20\(4\),pp\. 422–446\.External Links:[Document](https://dx.doi.org/10.1145/582415.582418),[Link](https://doi.org/10.1145/582415.582418)Cited by:[§2\.3](https://arxiv.org/html/2609.00086#S2.SS3.SSS0.Px2.p1.1)\.
- Kang and McAuley \(2018\)W\. Kang and J\. McAuleySelf\-attentive sequential recommendation\.In2018 IEEE International Conference on Data Mining,Piscataway, NJ, USA,pp\. 197–206\.External Links:[Document](https://dx.doi.org/10.1109/ICDM.2018.00035),[Link](https://doi.org/10.1109/ICDM.2018.00035)Cited by:[Table 1](https://arxiv.org/html/2609.00086#S3.T1.4.24.1.1)\.
- Krichene and Rendle \(2020\)W\. Krichene and S\. RendleOn sampled metrics for item recommendation\.InProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining,KDD ’20,New York, NY, USA,pp\. 1748–1757\.External Links:[Document](https://dx.doi.org/10.1145/3394486.3403226),[Link](https://doi.org/10.1145/3394486.3403226)Cited by:[§1](https://arxiv.org/html/2609.00086#S1.p2.1)\.
- Liet al\.\(2017\)J\. Li, P\. Ren, Z\. Chen, Z\. Ren, T\. Lian, and J\. MaNeural attentive session\-based recommendation\.InProceedings of the 2017 ACM on Conference on Information and Knowledge Management,CIKM ’17,New York, NY, USA,pp\. 1419–1428\.External Links:[Document](https://dx.doi.org/10.1145/3132847.3132926),[Link](https://doi.org/10.1145/3132847.3132926)Cited by:[Table 1](https://arxiv.org/html/2609.00086#S3.T1.4.26.1.1)\.
- Liet al\.\(2018\)R\. Li, S\. E\. Kahou, H\. Schulz, V\. Michalski, L\. Charlin, and C\. PalTowards deep conversational recommendations\.InAdvances in Neural Information Processing Systems,Vol\.31,Red Hook, NY, USA,pp\. 9748–9758\.External Links:[Link](https://proceedings.neurips.cc/paper/2018/hash/800de15c79c8d840f4e78d3af937d4d4-Abstract.html)Cited by:[§1](https://arxiv.org/html/2609.00086#S1.p3.1),[§2\.1](https://arxiv.org/html/2609.00086#S2.SS1.p1.1)\.
- Maet al\.\(2025\)T\. Ma, Y\. Cheng, Z\. Zheng, H\. Zhu, and H\. XiongLarge language models are not stable recommender systems: a position bias perspective\.InKnowledge Science, Engineering and Management,Lecture Notes in Computer Science, Vol\.15919,Singapore,pp\. 415–429\.External Links:[Document](https://dx.doi.org/10.1007/978-981-95-3001-4%5F31),[Link](https://doi.org/10.1007/978-981-95-3001-4_31)Cited by:[§1](https://arxiv.org/html/2609.00086#S1.p2.1)\.
- Meta \(2024a\)MetaLlama 3\.2 3B Instruct Model Card\.Hugging Face\.Note:Model cardAccessed: 2026\-06\-06External Links:[Link](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct)Cited by:[Table 1](https://arxiv.org/html/2609.00086#S3.T1.4.17.1.1)\.
- Meta \(2024b\)MetaLlama 3\.3 70B Instruct Model Card\.Hugging Face\.Note:Model cardAccessed: 2026\-06\-05External Links:[Link](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct)Cited by:[Table 1](https://arxiv.org/html/2609.00086#S3.T1.4.13.1.1)\.
- OpenAI \(2025a\)OpenAIIntroducing GPT\-4\.1 in the API\.OpenAI\.Note:OpenAI blogAccessed: 2026\-06\-05External Links:[Link](https://openai.com/index/gpt-4-1/)Cited by:[Table 1](https://arxiv.org/html/2609.00086#S3.T1.4.10.1.1),[Table 1](https://arxiv.org/html/2609.00086#S3.T1.4.9.1.1)\.
- OpenAI \(2025b\)OpenAIUpdate to GPT\-5 system card: GPT\-5\.2\.OpenAI\.Note:Technical reportAccessed: 2026\-06\-04External Links:[Link](https://openai.com/index/gpt-5-system-card-update-gpt-5-2/)Cited by:[Table 1](https://arxiv.org/html/2609.00086#S3.T1.4.7.1.1)\.
- Reimers and Gurevych \(2019\)N\. Reimers and I\. GurevychSentence\-BERT: sentence embeddings using siamese BERT\-networks\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing,Hong Kong, China,pp\. 3982–3992\.External Links:[Document](https://dx.doi.org/10.18653/v1/D19-1410),[Link](https://doi.org/10.18653/v1/D19-1410)Cited by:[§2\.2](https://arxiv.org/html/2609.00086#S2.SS2.p2.1),[Table 1](https://arxiv.org/html/2609.00086#S3.T1.4.4.1.1)\.
- Rendleet al\.\(2009\)S\. Rendle, C\. Freudenthaler, Z\. Gantner, and L\. Schmidt\-ThiemeBPR: bayesian personalized ranking from implicit feedback\.InProceedings of the Twenty\-Fifth Conference on Uncertainty in Artificial Intelligence,UAI ’09,Arlington, Virginia, USA,pp\. 452–461\.External Links:[Link](https://dl.acm.org/doi/10.5555/1795114.1795167)Cited by:[Table 1](https://arxiv.org/html/2609.00086#S3.T1.4.22.1.1)\.
- Sarwaret al\.\(2001\)B\. M\. Sarwar, G\. Karypis, J\. A\. Konstan, and J\. RiedlItem\-based collaborative filtering recommendation algorithms\.InProceedings of the 10th International Conference on World Wide Web,WWW ’01,New York, NY, USA,pp\. 285–295\.External Links:[Document](https://dx.doi.org/10.1145/371920.372071),[Link](https://doi.org/10.1145/371920.372071)Cited by:[Table 1](https://arxiv.org/html/2609.00086#S3.T1.4.20.1.1)\.
- Steck \(2019\)H\. SteckEmbarrassingly shallow autoencoders for sparse data\.InThe World Wide Web Conference,WWW ’19,New York, NY, USA,pp\. 3251–3257\.External Links:[Document](https://dx.doi.org/10.1145/3308558.3313710),[Link](https://doi.org/10.1145/3308558.3313710)Cited by:[Table 1](https://arxiv.org/html/2609.00086#S3.T1.4.19.1.1)\.
- Sunet al\.\(2023\)W\. Sun, L\. Yan, X\. Ma, S\. Wang, P\. Ren, Z\. Chen, D\. Yin, and Z\. RenIs ChatGPT good at search? investigating large language models as re\-ranking agents\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 14918–14937\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.923),[Link](https://doi.org/10.18653/v1/2023.emnlp-main.923)Cited by:[§1](https://arxiv.org/html/2609.00086#S1.p2.1)\.
- Weiet al\.\(2024\)W\. Wei, X\. Ren, J\. Tang, Q\. Wang, L\. Su, S\. Cheng, J\. Wang, D\. Yin, and C\. HuangLLMRec: large language models with graph augmentation for recommendation\.InProceedings of the 17th ACM International Conference on Web Search and Data Mining,WSDM ’24,New York, NY, USA,pp\. 806–815\.External Links:[Document](https://dx.doi.org/10.1145/3616855.3635853),[Link](https://doi.org/10.1145/3616855.3635853)Cited by:[§1](https://arxiv.org/html/2609.00086#S1.p2.1)\.
- Wuet al\.\(2019\)S\. Wu, Y\. Tang, Y\. Zhu, L\. Wang, X\. Xie, and T\. TanSession\-based recommendation with graph neural networks\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.33,Palo Alto, CA, USA,pp\. 346–353\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v33i01.3301346),[Link](https://doi.org/10.1609/aaai.v33i01.3301346)Cited by:[Table 1](https://arxiv.org/html/2609.00086#S3.T1.4.27.1.1)\.
- Yanget al\.\(2024\)A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. QiuQwen2\.5 technical report\.Note:arXiv preprint arXiv:2412\.15115External Links:2412\.15115,[Document](https://dx.doi.org/10.48550/arXiv.2412.15115),[Link](https://arxiv.org/abs/2412.15115)Cited by:[Table 1](https://arxiv.org/html/2609.00086#S3.T1.4.12.1.1),[Table 1](https://arxiv.org/html/2609.00086#S3.T1.4.16.1.1)\.
- Zhanget al\.\(2025\)H\. Zhang, Q\. Zhu, and Z\. DouEnhancing reranking for recommendation with LLMs through user preference retrieval\.InProceedings of the 31st International Conference on Computational Linguistics,Abu Dhabi, UAE,pp\. 658–671\.External Links:[Link](https://aclanthology.org/2025.coling-main.45/)Cited by:[§1](https://arxiv.org/html/2609.00086#S1.p2.1)\.
- Zhaoet al\.\(2021\)W\. X\. Zhao, S\. Mu, Y\. Hou, Z\. Lin, Y\. Chen, X\. Pan, K\. Li, Y\. Lu, H\. Wang, C\. Tian, Y\. Min, Z\. Feng, X\. Fan, X\. Chen, P\. Wang, W\. Ji, Y\. Li, X\. Wang, and J\. WenRecBole: towards a unified, comprehensive and efficient framework for recommendation algorithms\.InProceedings of the 30th ACM International Conference on Information & Knowledge Management,CIKM ’21,New York, NY, USA,pp\. 4653–4664\.External Links:[Document](https://dx.doi.org/10.1145/3459637.3482016),[Link](https://doi.org/10.1145/3459637.3482016)Cited by:[§2\.1](https://arxiv.org/html/2609.00086#S2.SS1.p2.1),[§2\.3](https://arxiv.org/html/2609.00086#S2.SS3.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2609.00086#S3.T1.4.29.1.1)\.