Rethinking Literature Search Evaluation: Deep Research Helps, and Human Citation Lists Are Not a Ground Truth

# Deep Research Helps, and Human Citation Lists Are Not a Ground Truth
Source: [https://arxiv.org/html/2605.29234](https://arxiv.org/html/2605.29234)
## Rethinking Literature Search Evaluation: Deep Research Helps, and Human Citation Lists Are Not a Ground Truth

Gaurav Sahu1,2, Laurent Charlin1,2,4, Christopher Pal1,3,4,5,6 1Mila – Quebec AI Institute2HEC Montréal 3ServiceNow Research4Canada CIFAR AI Chair 5Université de Montréal6Polytechnique Montréal

###### Abstract

We study large\-scale literature search from two complementary angles: improving the retrieval pipeline, and stress\-testing the human reference list as an evaluation target\. First, we implement a*Deep Research*pipeline that processes the full query paper and expands the retrieved results breadth\-first along their bibliographies, and show that it substantially outperforms vanilla API\-only search, raising recall onRollingEval\-Jun25\(a 250\-paper literature\-search benchmark\) from below 20% to above 80%\. Second, we use a neutral LLM\-as\-a\-judge to determine if human references are sound ground truth for the task\. We find significant limitations: only 51% of human citations are judged moderately relevant or higher, against 86–88% for the strongest AI\-based re\-rankers\. We study this gap on the OpenAlex co\-authorship graph, finding that humans are 2\.5×\\timesmore likely than the best AI re\-rankers to cite a direct collaborator\. Together, our results argue against single\-axis literature\-search evaluation: recall, topical\-relevance scoring, ranked\-list diversity, and a co\-authorship\-distance diagnostic each measure complementary properties of citation quality and should be reported jointly\.

Rethinking Literature Search Evaluation: Deep Research Helps, and Human Citation Lists Are Not a Ground Truth

## 1Introduction

Literature search remains a foundational step of scientific work, yet it is hard to evaluate well\. The community typically reports precision and recall against the papers that the query paper itself cites\(Ajithet al\.,[2024](https://arxiv.org/html/2605.29234#bib.bib4); Hanget al\.,[2025](https://arxiv.org/html/2605.29234#bib.bib24)\), treating that reference list as ground truth\. This convention has two problems\. First, vanilla API search based on keywords misses most of the relevant literature: single and multiple keyword queries to a scholarly index are brittle, and the long\-tail structure of citation graphs hides relevant work behind chains of references\(Kinneyet al\.,[2023](https://arxiv.org/html/2605.29234#bib.bib7); Priemet al\.,[2022](https://arxiv.org/html/2605.29234#bib.bib8)\)\. Second, the reference list itself is incomplete and idiosyncratic\(Agarwalet al\.,[2025](https://arxiv.org/html/2605.29234#bib.bib1)\), including foundational paper citations and software attribution, in addition to topically relevant papers\.

We address both problems and report two findings\. \(1\) A*Deep Research*pipeline that ingests the full query paper and recursively expands its bibliography increases recall by an order of magnitude over vanilla API search\. \(2\) Treating human reference list as a clean ground truth is problematic: a neutral LLM judge\(Zhenget al\.,[2023](https://arxiv.org/html/2605.29234#bib.bib13)\)rates only 51% of human citations as moderately relevant or higher, compared with 86–88% for the strongest automated re\-rankers\. We probe this gap through the OpenAlex co\-authorship graph\(Priemet al\.,[2022](https://arxiv.org/html/2605.29234#bib.bib8)\)and find that humans cite direct collaborators 2\.5×\\timesmore often than the strongest re\-rankers\.

#### Contributions\.

\(i\) A modular Deep Research retrieval pipeline \(Figure[1](https://arxiv.org/html/2605.29234#S1.F1)\) for full\-text literature search, with order\-of\-magnitude recall gains over traditional literature search baselines, along withRollingEval\-Jun25, a 250\-paper benchmark that mitigates data contamination\. \(ii\) An analysis of human reference lists that documents systematic non\-topical citing and provides a framework for re\-thinking literature\-search ground truth\.

![Refer to caption](https://arxiv.org/html/2605.29234v1/figures/new_fig1.png)Figure 1:Deep Research pipeline\. The system ingests the full query paper, generates LLM\-constructed keyword queries, retrieves seed papers from scholarly APIs, expands the candidate set along citation links, and re\-ranks the pooled candidates\. Full algorithm and prompts in Appendix[B](https://arxiv.org/html/2605.29234#A2)\.![Refer to caption](https://arxiv.org/html/2605.29234v1/x1.png)Figure 2:Precision@KK\(left\) and Recall@KK\(right\),KKon log scale\. Deep Research raises recall by an order of magnitude over normal search;Qwen3embeddings give the strongest top\-KKprecision\.

## 2Deep Research Pipeline

The system has two phases: a high\-recall retrieval phase and a re\-ranking phase\.Phase 1prompts an LLM to draft diverse keyword queries for the query document, translates them into provider\-specific syntax for arXiv, OpenAlex, and Semantic Scholar\(Kinneyet al\.,[2023](https://arxiv.org/html/2605.29234#bib.bib7); Priemet al\.,[2022](https://arxiv.org/html/2605.29234#bib.bib8)\), and collects candidates under rate\-limit\-aware concurrency\.Phase 2\(Deep Research\) expands the candidate set breadth\-first along bibliographic references up to a configurable depth and paper budget\. One of three interchangeable modules then re\-ranks the expanded pool:Qwen3embeddings\(Zhanget al\.,[2025a](https://arxiv.org/html/2605.29234#bib.bib6)\); an LLM\-based debate ranking mechanism that produces arguments for and against inclusion before assigning a 0–100 relevance score; or a Debate\+\+Qwen3ensemble that combines the two\. Algorithm pseudo\-code, prompts, and full caching/concurrency details are in Appendix[B](https://arxiv.org/html/2605.29234#A2)\.

## 3Setup

Benchmark\.We constructRollingEval\-Jun25, a June 2025 snapshot of theRollingEvalseries proposed originally by\(Agarwalet al\.,[2025](https://arxiv.org/html/2605.29234#bib.bib1)\)to avoid data contamination issues when benchmarking LLMs for literature search\. The benchmark comprises 250 computer science arXiv submissions with cleaned full\-text and bibliographies\. We choose June 2025 as it post\-dates the training\-data cutoff of every LLM evaluated in this work, mitigating contamination, and the cleaned full\-text and bibliography pipeline supports the matched\-input head\-to\-head re\-ranker comparison in Section[4](https://arxiv.org/html/2605.29234#S4)\. We fetch references through the OpenAlex and SemanticScholar APIs, and fall back on an LLM\-based parser that extracts bibliographies directly from PDFs\.

Metrics\.We report:\(a\)Precision and Recall against the human reference list,\(b\)Semantic Relevance \(SR\), a 0–100 score from an LLM judge\(Zhenget al\.,[2023](https://arxiv.org/html/2605.29234#bib.bib13)\), and\(c\)α\\alpha\-nDCG\(Clarkeet al\.,[2008](https://arxiv.org/html/2605.29234#bib.bib9)\), the novelty\-aware variant of nDCG\(Järvelin and Kekäläinen,[2002](https://arxiv.org/html/2605.29234#bib.bib10)\)that measures how well a ranked list covers diverse topics without redundancy\. Detailed metric definitions are in Appendix[D](https://arxiv.org/html/2605.29234#A4)\.

The SR judge isGPT\-OSS\-120Bserved via vLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2605.29234#bib.bib12)\); for each pair it conditions on the query paper’s full text \(title, abstract, body\) and the candidate’s title and abstract, and returns a rubric grade in\{0,…,5\}\\\{0,\\ldots,5\\\}\(see Prompt[8](https://arxiv.org/html/2605.29234#A3.T8)\), which we then multiply by 20\. The re\-rankers also score candidates by comparing the query paper’s full text with the candidate’s title and abstract\.

## 4Retrieval Results

Deep Research yields an order\-of\-magnitude recall gain\.Figure[2](https://arxiv.org/html/2605.29234#S1.F2)plots precision and recall against the human reference list\. Vanilla API search \(*No Rerank, no DR*\) tops out near 15% recall only atK≈105K\{\\approx\}10^\{5\}and never exceeds it, while every Deep Research variant exceeds 80% recall byK=104K\{=\}10^\{4\}\.Qwen3embeddings achieve the strongest top\-KKprecision \(0\.37 atK=1K\{=\}1\), closely followed by the Debate\+Qwen ensemble\. We also includePaSa\(Heet al\.,[2025](https://arxiv.org/html/2605.29234#bib.bib2)\)andScholarQA\(Singhet al\.,[2025](https://arxiv.org/html/2605.29234#bib.bib3)\)as deep\-research baselines\. We run both onRollingEval\-Jun25with the query paper’s abstract as input; each system then produces its ranking through its native retrieval and scoring pipeline\. Table[1](https://arxiv.org/html/2605.29234#S4.T1)summarizes precision and recall at four representative cutoffs\. The best Deep Research variant nearly triples recall atK=100K\{=\}100over an arXiv\-API baseline and reaches 51\.9% recall atK=1000K\{=\}1000\.

Table 1:Precision and recall atK∈\{20,100,1,000\}K\\in\\\{20,100,1\{,\}000\\\}\. Best column in bold\.#### Diversity and ranking quality\.

Beyond precision and recall, we ask whether highly relevant items are concentrated near the top of each ranked list\. Table[2](https://arxiv.org/html/2605.29234#S4.T2)reportsα\\alpha\-nDCG\(Clarkeet al\.,[2008](https://arxiv.org/html/2605.29234#bib.bib9)\)and the LLM\-judged Semantic Relevance score atK∈\{10,100,1000\}K\\in\\\{10,100,1000\\\}\. Debate \(without Qwen\) achieves the strongestα\\alpha\-nDCG atK=100K\{=\}100andK=1000K\{=\}1000, suggesting that prompt\-based scoring leads to higher diversity and relevant items more evenly through the ranked list than embedding similarity\. The Debate\+\+Qwen3ensemble has the highest top\-10 Semantic Relevance \(67\.9\), validating it as the strongest top\-of\-list retriever, whileQwen3alone maintains the highest Semantic Relevance atK=100K\{=\}100andK=1000K\{=\}1000, indicating better long\-tail behavior\. The ground\-truth row anchors the comparison: the human reference list itself sits at 47–51 on Semantic Relevance acrossKK, well below the AI re\-rankers in the top\-10/100 buckets, a discrepancy we examine next\.

Table 2:α\\alpha\-nDCG and Semantic Relevance atK∈\{10,100,1000\}K\\in\\\{10,100,1000\\\}\. Higher is better\.

## 5Human References = Ground Truth?

Recall against the human reference list is a coherent metric only if that list is itself reliable\. We test this by asking a neutral LLM judge to rate the topical relevance of every cited paper, then comparing the resulting distribution against the same judge’s ratings of AI re\-ranker outputs at matched list length\.

Setup\.For each of the 250 query papersPPwithNPN\_\{P\}human references, we evaluate all human citations and the top\-NPN\_\{P\}predictions from each re\-ranker\. This list\-length parity is essential: a longer AI list would trivially include more low\-scoring tail items\. We aggregate 9,204 human \(query, reference\) pairs and 36,692 AI pairs\.

Table[3](https://arxiv.org/html/2605.29234#S5.T3)shows the matched\-KKdistribution and Figure[3](https://arxiv.org/html/2605.29234#S5.F3)the full curve\. Restricted to the same list length, human references score≥60\\geq 60in 51\.4% of cases, whileQwen3and Debate\+Qwen reach 86\.0% and 87\.8%\. The disparity concentrates at score=40=40, which captures 38\.2% of human references but only 10\.2–13\.7% of what the top re\-rankers retrieve\. On the LLM judge’s topical\-relevance axis, the strongest re\-rankers’ top\-NPN\_\{P\}lists score higher than the human\-curated list at matched length; this speaks to one axis of citation quality, not an overall verdict on the human list\. Figure[3](https://arxiv.org/html/2605.29234#S5.F3)makes this concrete acrossKK: the AI re\-rankers start well above the human saturation level and remain above it throughout theK≤1000K\\leq 1000scoring window, with their cumulative scores gradually diluting toward the matched\-KKmeans in Table[3](https://arxiv.org/html/2605.29234#S5.T3)as the top\-of\-list signal blends with lower\-relevance tail items\.

Table 3:Semantic relevance distribution at matched top\-NPN\_\{P\}\.nnis the total number of \(query, reference\) pairs\.![Refer to caption](https://arxiv.org/html/2605.29234v1/x2.png)Figure 3:Cumulative semantic relevance over the top\-KKretrieved candidates, with the SR judge applied to at mostKmax=1000K\_\{\\max\}\{=\}1000items per query\. Human references saturate near 51 byK=50K\{=\}50; AI re\-rankers start substantially higher \(73–75 atK=1K\{=\}1\) and stay above the human curve across the fullK≤1000K\\leq 1000scoring window\.Is the gap a citation\-collusion artifact?A natural hypothesis is that humans cite collaborators\. We test this on the OpenAlex co\-authorship graph\(Priemet al\.,[2022](https://arxiv.org/html/2605.29234#bib.bib8)\), classifying each \(query, candidate\) pair by the minimum hop distance between author sets,d∈\{0,1,2,3,≥4\}d\\in\\\{0,1,2,3,\{\\geq\}4\\\}\(formal definition and BFS data requirements in Appendix[E](https://arxiv.org/html/2605.29234#A5)\)\. Table[4](https://arxiv.org/html/2605.29234#S5.T4)reports the distribution\.

The bias concentrates at short hops and decays with radius\. Atd=0d\{=\}0\(direct co\-authorship\) humans are 2\.5×\\timesmore likely than the strongest re\-rankers to cite themselves or a co\-author \(5\.13% vs\. 1\.9–2\.1%\)\. The cumulative within\-one\-hop rate is 12\.2% for humans vs\. 6\.9–7\.4% for the best AI methods, a 1\.6×\\timesgap\. Byd≤2d\{\\leq\}2the gap narrows to 1\.1×\\times\(53\.6% vs\. 46\.6–48\.7%\), and byd≤3d\{\\leq\}3all sources converge near 80–83%, consistent with the small\-world structure of the academic graph\. Conditional mean scores are essentially flat across distance classes for humans \(56\.1, 52\.6, 51\.2, 51\.2, 50\.9 ford=0d\{=\}0throughd≥4d\{\\geq\}4\): the network proximity of a citation does not predict its topical relevance\.

Table 4:Co\-authorship\-graph distance at matched top\-NPN\_\{P\}, OpenAlex graph\.d=0d\{=\}0: direct co\-authorship; higherdd: shortest co\-author path\.Interpretation\.Both effects are individually defensible but together explain the gap\. The network bias is concentrated at direct and one\-hop collaborators, where humans diverge sharply from the re\-rankers \(5\.13% vs\. 1\.9–2\.1% atd=0d\{=\}0; 12\.2% vs\. 6\.9–7\.4% within one hop\); fromd≥2d\{\\geq\}2onward the distributions converge, so network proximity drives a clean but bounded share of the disparity\. Citing close collaborators is appropriate where the work is genuinely relevant, but at this magnitude it suggests authors should put proportionally more effort into capturing the broader landscape beyond their immediate clique\. On the topical axis, foundational background \(Transformer, ResNet, BERT\(Vaswaniet al\.,[2017](https://arxiv.org/html/2605.29234#bib.bib16); Heet al\.,[2016](https://arxiv.org/html/2605.29234#bib.bib17); Devlinet al\.,[2019](https://arxiv.org/html/2605.29234#bib.bib14)\)\), classical algorithms \(Adam\(Kingma and Ba,[2015](https://arxiv.org/html/2605.29234#bib.bib15)\)\), and software or library attribution are all defensible reasons to cite a paper whose abstract carries no topical signal; what is harder to defend is that citations of this kind make up nearly half \(48\.6%\) of human reference lists at matched length, against 12–14% for the strongest re\-rankers\. Eighteen representative examples are in Appendix[F](https://arxiv.org/html/2605.29234#A6)\. The practical implication is two\-fold: literature\-search benchmarks should not treat human reference lists as a clean ground truth, and authors should treat the topical\-relevance share of their bibliography as a property worth checking\. Future work should also pair the topical rubric with citation\-context spans from the query paper, or otherwise model the reasons humans cite\.

## 6Discussion and Conclusion

Our results argue for separating two questions the community usually conflates\.*Did the system retrieve the papers the authors cited?*is a coverage question against a known\-imperfect target\.*Did the system retrieve papers that a neutral reader would judge relevant?*is a different,*complementary*question\. The two diverge sharply: Deep Research dominates on the first, while the human\-curated list scores lower than the top re\-rankers on the second\. These two findings are intertwined rather than competing: the order\-of\-magnitude recall gain matters precisely because it expands the candidate pool from which any retrieval system, including SR\-aligned ones, must draw, so a higher\-recall pipeline raises the achievable ceiling on whichever metric the community ultimately adopts as ground truth\. Neither metric is strictly better: recall against human citations captures dimensions an LLM\-judge cannot justify \(foundational and tool/library attribution, methodological lineage\), while the topical rubric captures dimensions that human reference lists undersample \(long\-tail topically\-related work, non\-collaborator authors\)\. We argue for reporting both, alongside the OpenAlex\-graph diagnostic, rather than treating either in isolation as ground truth\. We will releaseRollingEval\-Jun25, the Deep Research pipeline, the LLM\-judge prompts, and the OpenAlex distance analysis as a reproducible scaffold for re\-examining literature\-search ground truth\.

## Limitations

The LLM judge is a single neutral model; rubric\-conditioned scoring may drift across model and prompt versions, and we do not currently model citation context \(in\-text spans surrounding the citation\)\. Validating against human relevance ratings on a labelled subsample remains a natural next step\. The co\-authorship graph fetch caps each author at 1,000 most recent works, so we under\-count distances through hyper\-prolific intermediaries, which biases the reported gap in a conservative direction\. Bibliography extraction relies on LLM agents and can carry residual noise\. TheRollingEvalbenchmark consists only of computer science papers; extending it to biomedical, social\-science, and humanities papers would test cross\-domain generality\. Finally, the analysis covers only arXiv\-indexed references; we exclude non\-arXiv citations\.

## Potential Risks

The Semantic Relevance judge and Deep Research pipeline we release target legitimate literature discovery, but the same topical\-relevance rubric could be misapplied to gate citations during peer review, to filter submissions, or to automate rejection, uses for which a single LLM judge is not a defensible standard\. Our results show the rubric and the human reference list disagree systematically along measurable axes; both should serve as complementary diagnostics, not as evaluation thresholds\. The OpenAlex co\-authorship analysis surfaces network proximity between authors that is computable from public data but that researchers may reasonably consider private; we report only aggregate distributions and avoid disclosing individual\-paper proximities\. Finally,RollingEval\-Jun25comprises arXiv preprints chosen to postdate the training cutoff of every LLM we evaluate; its contamination\-resistance guarantee weakens as models update, so the snapshot will need periodic refreshes\. Additionally, extending it to include articles from other domains medical, biology, or physics might help study its generalizability\.

## References

- S\. Agarwal, G\. Sahu, A\. Puri, I\. H\. Laradji, K\. D\. Dvijotham, J\. Stanley, L\. Charlin, and C\. Pal \(2025\)LitLLMs, LLMs for literature review: are we there yet?\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=heeJqQXKg7)Cited by:[Appendix A](https://arxiv.org/html/2605.29234#A1.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2605.29234#S1.p1.1),[§3](https://arxiv.org/html/2605.29234#S3.p1.1)\.
- Litsearch: a retrieval benchmark for scientific literature search\.InProceedings of the 2024 Conference on EMNLP,pp\. 15068–15083\.Cited by:[Appendix A](https://arxiv.org/html/2605.29234#A1.SS0.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2605.29234#S1.p1.1)\.
- C\. L\. Clarke, M\. Kolla, G\. V\. Cormack, O\. Vechtomova, A\. Ashkan, S\. Büttcher, and I\. MacKinnon \(2008\)Novelty and diversity in information retrieval evaluation\.InProceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval,pp\. 659–666\.Cited by:[Appendix D](https://arxiv.org/html/2605.29234#A4.SS0.SSS0.Px3.p1.3),[§3](https://arxiv.org/html/2605.29234#S3.p2.1),[§4](https://arxiv.org/html/2605.29234#S4.SS0.SSS0.Px1.p1.9)\.
- A\. Cohan, S\. Feldman, I\. Beltagy, D\. Downey, and D\. Weld \(2020\)SPECTER: document\-level representation learning using citation\-informed transformers\.InProceedings of the 58th Annual Meeting of the ACL,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 2270–2282\.External Links:[Link](https://aclanthology.org/2020.acl-main.207/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.207)Cited by:[Appendix A](https://arxiv.org/html/2605.29234#A1.SS0.SSS0.Px2.p1.1)\.
- J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova \(2019\)Bert: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 conference of the NAACL: HLT,pp\. 4171–4186\.Cited by:[§5](https://arxiv.org/html/2605.29234#S5.p6.2)\.
- C\. N\. Hang, P\. Yu, C\. Tan, and D\. Chiu \(2025\)Beyond search: measuring LLM performance for scientific literature discovery\.InIEEE TALE,External Links:[Document](https://dx.doi.org/10.1109/TALE66047.2025.11346619)Cited by:[Appendix A](https://arxiv.org/html/2605.29234#A1.SS0.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2605.29234#S1.p1.1)\.
- K\. He, X\. Zhang, S\. Ren, and J\. Sun \(2016\)Deep residual learning for image recognition\.InProceedings of CVPR,pp\. 770–778\.Cited by:[Appendix F](https://arxiv.org/html/2605.29234#A6.p1.1),[§5](https://arxiv.org/html/2605.29234#S5.p6.2)\.
- Y\. He, G\. Huang, P\. Feng, Y\. Lin, Y\. Zhang, H\. Li, and W\. E \(2025\)PaSa: an LLM agent for comprehensive academic paper search\.InProceedings of ACL,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 11663–11679\.External Links:[Link](https://aclanthology.org/2025.acl-long.572/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.572),ISBN 979\-8\-89176\-251\-0Cited by:[Appendix A](https://arxiv.org/html/2605.29234#A1.SS0.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2605.29234#S4.p1.6)\.
- K\. Järvelin and J\. Kekäläinen \(2002\)Cumulated gain\-based evaluation of ir techniques\.ACM Transactions on Information Systems \(TOIS\)20\(4\),pp\. 422–446\.Cited by:[Appendix D](https://arxiv.org/html/2605.29234#A4.SS0.SSS0.Px3.p1.3),[§3](https://arxiv.org/html/2605.29234#S3.p2.1)\.
- V\. Karpukhin, B\. Oguz, S\. Min, P\. Lewis, L\. Wu, S\. Edunov, D\. Chen, and W\. Yih \(2020\)Dense passage retrieval for open\-domain question answering\.InProceedings of EMNLP,B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 6769–6781\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.550/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.550)Cited by:[Appendix A](https://arxiv.org/html/2605.29234#A1.SS0.SSS0.Px2.p1.1)\.
- D\. P\. Kingma and J\. Ba \(2015\)Adam: a method for stochastic optimization\.InICLR,Cited by:[Appendix F](https://arxiv.org/html/2605.29234#A6.p1.1),[§5](https://arxiv.org/html/2605.29234#S5.p6.2)\.
- R\. Kinney, C\. Anastasiades, R\. Authur, I\. Beltagy, J\. Bragg, A\. Buraczynski, I\. Cachola, S\. Candra, Y\. Chandrasekhar, A\. Cohan,et al\.\(2023\)The semantic scholar open data platform\.arXiv preprint arXiv:2301\.10140\.Cited by:[Appendix A](https://arxiv.org/html/2605.29234#A1.SS0.SSS0.Px1.p1.1),[§B\.1](https://arxiv.org/html/2605.29234#A2.SS1.p1.1),[Appendix G](https://arxiv.org/html/2605.29234#A7.p1.1),[§1](https://arxiv.org/html/2605.29234#S1.p1.1),[§2](https://arxiv.org/html/2605.29234#S2.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of SOSP,pp\. 611–626\.Cited by:[1st item](https://arxiv.org/html/2605.29234#A2.I1.i1.p1.1),[Appendix G](https://arxiv.org/html/2605.29234#A7.p1.1),[§3](https://arxiv.org/html/2605.29234#S3.p3.1)\.
- J\. Lála, O\. O’Donoghue, A\. Shtedritski, S\. Cox, S\. G\. Rodriques, and A\. D\. White \(2023\)PaperQA: retrieval\-augmented generative agent for scientific research\.arXiv preprint arXiv:2312\.07559\.Cited by:[Appendix A](https://arxiv.org/html/2605.29234#A1.SS0.SSS0.Px3.p1.1)\.
- Y\. Lei, S\. Si, W\. Wang, Y\. Wu, G\. Chen, F\. Qi, and M\. Sun \(2025\)RhinoInsight: improving deep research through control mechanisms for model behavior and context\.arXiv preprint arXiv:2511\.18743\.Cited by:[Appendix A](https://arxiv.org/html/2605.29234#A1.SS0.SSS0.Px3.p1.1)\.
- L\. McInnes, J\. Healy, S\. Astels,et al\.\(2017\)Hdbscan: hierarchical density based clustering\.\.Journal of Open Source Software2\(11\),pp\. 205\.Cited by:[Appendix D](https://arxiv.org/html/2605.29234#A4.SS0.SSS0.Px3.p1.6),[Appendix G](https://arxiv.org/html/2605.29234#A7.p1.1)\.
- J\. Priem, H\. Piwowar, and R\. Orr \(2022\)OpenAlex: a fully\-open index of scholarly works, authors, venues, institutions, and concepts\.arXiv preprint arXiv:2205\.01833\.Cited by:[Appendix A](https://arxiv.org/html/2605.29234#A1.SS0.SSS0.Px1.p1.1),[§B\.1](https://arxiv.org/html/2605.29234#A2.SS1.p1.1),[Appendix E](https://arxiv.org/html/2605.29234#A5.SS0.SSS0.Px6.p1.1),[Appendix G](https://arxiv.org/html/2605.29234#A7.p1.1),[§1](https://arxiv.org/html/2605.29234#S1.p1.1),[§1](https://arxiv.org/html/2605.29234#S1.p2.1),[§2](https://arxiv.org/html/2605.29234#S2.p1.1),[§5](https://arxiv.org/html/2605.29234#S5.p4.1)\.
- S\. Robertson and H\. Zaragoza \(2009\)The probabilistic relevance framework: BM25 and beyond\.Foundations and Trends in Information Retrieval3\(4\),pp\. 333–389\.Cited by:[Appendix A](https://arxiv.org/html/2605.29234#A1.SS0.SSS0.Px2.p1.1)\.
- A\. Singh, J\. C\. Chang, D\. Haddad, A\. Naik, J\. D\. Hwang, R\. Kinney, D\. S\. Weld, D\. Downey, and S\. Feldman \(2025\)Ai2 ScholarQA: organized literature synthesis with attribution\.InProceedings of ACL,P\. Mishra, S\. Muresan, and T\. Yu \(Eds\.\),Vienna, Austria,pp\. 513–523\.External Links:[Link](https://aclanthology.org/2025.acl-demo.49/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-demo.49),ISBN 979\-8\-89176\-253\-4Cited by:[Appendix A](https://arxiv.org/html/2605.29234#A1.SS0.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2605.29234#S4.p1.6)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.Advances in NeurIPS30\.Cited by:[§5](https://arxiv.org/html/2605.29234#S5.p6.2)\.
- J\. Zhang, J\. Du, Z\. Xue, G\. Ye, and Y\. Shao \(2026\)Tool\-augmented multi\-turn academic paper recommendation via reinforcement learning\.InIEEE BigComp,External Links:[Document](https://dx.doi.org/10.1109/BigComp68355.2026.00031)Cited by:[Appendix A](https://arxiv.org/html/2605.29234#A1.SS0.SSS0.Px3.p1.1)\.
- Y\. Zhang, M\. Li, D\. Long, X\. Zhang, H\. Lin, B\. Yang, P\. Xie, A\. Yang, D\. Liu, J\. Lin,et al\.\(2025a\)Qwen3 embedding: advancing text embedding and reranking through foundation models\.arXiv preprint arXiv:2506\.05176\.Cited by:[Appendix A](https://arxiv.org/html/2605.29234#A1.SS0.SSS0.Px2.p1.1),[1st item](https://arxiv.org/html/2605.29234#A2.I1.i1.p1.1),[§C\.3](https://arxiv.org/html/2605.29234#A3.SS3.p1.1),[Appendix D](https://arxiv.org/html/2605.29234#A4.SS0.SSS0.Px3.p1.6),[Appendix G](https://arxiv.org/html/2605.29234#A7.p1.1),[§2](https://arxiv.org/html/2605.29234#S2.p1.1)\.
- Y\. Zhang, R\. Yang, S\. Jiao, S\. Kang, and J\. Han \(2025b\)Scientific paper retrieval with LLM\-guided semantic\-based ranking\.arXiv preprint arXiv:2505\.21815\.Cited by:[Appendix A](https://arxiv.org/html/2605.29234#A1.SS0.SSS0.Px2.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging LLM\-as\-a\-judge with MT\-Bench and chatbot arena\.InNeurIPS Datasets and Benchmarks,Cited by:[Appendix D](https://arxiv.org/html/2605.29234#A4.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2605.29234#S1.p2.1),[§3](https://arxiv.org/html/2605.29234#S3.p2.1)\.

## Appendix ABackground and Related Work

#### Scholarly APIs and index\-based retrieval\.

Scholarly APIs such as OpenAlex\(Priemet al\.,[2022](https://arxiv.org/html/2605.29234#bib.bib8)\), Semantic Scholar\(Kinneyet al\.,[2023](https://arxiv.org/html/2605.29234#bib.bib7)\), and arXiv remain the backbone of scalable literature discovery, but require careful query design to maximize recall under strict rate limits, and a single API call against them typically saturates well below the topical neighborhood of the query paper\. We therefore use these APIs only as the entry\-point fetch layer, drafting an LLM\-constructed keyword set against them in Phase 1 and then expanding breadth\-first along bibliographic references in Phase 2 \(Section[2](https://arxiv.org/html/2605.29234#S2)\)\.

#### Dense retrievers and re\-rankers\.

Dense retrievers such as DPR\(Karpukhinet al\.,[2020](https://arxiv.org/html/2605.29234#bib.bib20)\)and BM25\(Robertson and Zaragoza,[2009](https://arxiv.org/html/2605.29234#bib.bib19)\), together with citation\-aware embeddings like SPECTER\(Cohanet al\.,[2020](https://arxiv.org/html/2605.29234#bib.bib5)\), provide stronger semantic matches than keyword search but trade off efficiency at scale; more recent LLM\-guided schemes such as SemRank\(Zhanget al\.,[2025b](https://arxiv.org/html/2605.29234#bib.bib26)\)graft concept\-level matching on top of a dense retriever\. We treat the re\-ranker as an interchangeable module:Qwen3\-Embedding\-8B\(Zhanget al\.,[2025a](https://arxiv.org/html/2605.29234#bib.bib6)\), an LLM\-based Debate ranker, and their ensemble plug into the same final stage of the pipeline and are compared head\-to\-head under matched inputs and matched candidate pools\.

#### LLM\-based retrieval agents\.

LLM\-based retrieval agents such as PaSa\(Heet al\.,[2025](https://arxiv.org/html/2605.29234#bib.bib2)\), Ai2 ScholarQA\(Singhet al\.,[2025](https://arxiv.org/html/2605.29234#bib.bib3)\), PaperQA\(Lálaet al\.,[2023](https://arxiv.org/html/2605.29234#bib.bib22)\), RhinoInsight\(Leiet al\.,[2025](https://arxiv.org/html/2605.29234#bib.bib25)\), and earlier LitLLM\(Agarwalet al\.,[2025](https://arxiv.org/html/2605.29234#bib.bib1)\)combine API search with LLM scoring, control mechanisms, or bibliography expansion, andZhanget al\.\([2026](https://arxiv.org/html/2605.29234#bib.bib27)\)extend the family to a multi\-turn, tool\-using policy trained with reinforcement learning\. We hold the orchestration policy constant \(keyword expansion followed by bibliography\-graph expansion\) and instead vary \(i\) full\-text input vs\. abstract\-only input and \(ii\) the choice of re\-ranker, which isolates the contribution of full\-paper context and citation\-graph reach from the choice of agent policy\.

#### Evaluating LLM literature search\.

On the evaluation side, theLitSearchbenchmark\(Ajithet al\.,[2024](https://arxiv.org/html/2605.29234#bib.bib4)\)and the three\-metric framework ofHanget al\.\([2025](https://arxiv.org/html/2605.29234#bib.bib24)\), which scores GPT\-4, Gemini 2\.5, and DeepSeek\-V3 in vanilla and “deep research” configurations, both anchor quality on recall against the human reference list\. Our analysis treats that recall as one diagnostic axis, pairs it with a semantic\-relevance rubric scored by a neutral LLM judge, and adds an OpenAlex co\-authorship\-graph distance metric \(Section[5](https://arxiv.org/html/2605.29234#S5)\); the first two metrics disagree systematically on real data, and the co\-authorship analysis attributes a measurable portion of the gap to network bias in human citing rather than to retrieval quality alone\.

## Appendix BSystem Details

### B\.1Phase 1: Retrieval with LLM\-Constructed Queries

Given a query document, we prompt an LLM to draft a diverse keyword query set, then normalize each query into the provider\-specific syntax for arXiv, OpenAlex, and Semantic Scholar\(Kinneyet al\.,[2023](https://arxiv.org/html/2605.29234#bib.bib7); Priemet al\.,[2022](https://arxiv.org/html/2605.29234#bib.bib8)\)\. We pre\-process inputs to remove inline citations and bibliography content before prompting, and cache all generated queries by input\- and model\-specific hash\.

### B\.2Phase 2: Citation Graph Expansion

We expand the seed set breadth\-first along bibliographic references up to a configurable depthDm​a​xD\_\{max\}\(=3 in our experiments\) and paper budgetNm​a​xN\_\{max\}\. When references from APIs like SemanticScholar and OpenAlex are incomplete, we fall back on LLM\-based PDF bibliography parsing, which first extracts the bibliography section from the paper and resolves each entry through its title and authors using SemanticScholar and OpenAlex\. Algorithm[1](https://arxiv.org/html/2605.29234#alg1)states the procedure\.

Algorithm 1Deep Research: Citation Graph Expansion0:Initial Seed Papers

Ps​e​e​dP\_\{seed\}, Max Depth

Dm​a​xD\_\{max\}, Max Papers

Nm​a​xN\_\{max\}
1:

Q←new Queue​\(\)Q\\leftarrow\\text\{new Queue\}\(\)
2:

Pf​i​n​a​l←Ps​e​e​dP\_\{final\}\\leftarrow P\_\{seed\}
3:

Vi​d​s←\{id​\(p\)​for​p∈Ps​e​e​d\}V\_\{ids\}\\leftarrow\\\{\\text\{id\}\(p\)\\text\{ for \}p\\in P\_\{seed\}\\\}
4:for each

p∈Ps​e​e​dp\\in P\_\{seed\}do

5:

Q\.enqueue​\(\(p,0\)\)Q\.\\text\{enqueue\}\(\(p,0\)\)
6:end for

7:while

QQis not emptyand

\|Pf​i​n​a​l\|<Nm​a​x\|P\_\{final\}\|<N\_\{max\}do

8:

\(pc​u​r​r,dc​u​r​r\)←Q\.dequeue​\(\)\(p\_\{curr\},d\_\{curr\}\)\\leftarrow Q\.\\text\{dequeue\}\(\)
9:if

dc​u​r​r≥Dm​a​xd\_\{curr\}\\geq D\_\{max\}then

10:continue

11:endif

12:

Pc​a​n​d←GetReferences​\(pc​u​r​r\)P\_\{cand\}\\leftarrow\\text\{GetReferences\}\(p\_\{curr\}\)
13:foreach

pn​e​w∈Pc​a​n​dp\_\{new\}\\in P\_\{cand\}do

14:if

id​\(pn​e​w\)∉Vi​d​s\\text\{id\}\(p\_\{new\}\)\\notin V\_\{ids\}then

15:

Vi​d​s\.add​\(id​\(pn​e​w\)\)V\_\{ids\}\.\\text\{add\}\(\\text\{id\}\(p\_\{new\}\)\)
16:

Pf​i​n​a​l\.add​\(pn​e​w\)P\_\{final\}\.\\text\{add\}\(p\_\{new\}\)
17:

Q\.enqueue​\(\(pn​e​w,dc​u​r​r\+1\)\)Q\.\\text\{enqueue\}\(\(p\_\{new\},d\_\{curr\}\+1\)\)
18:if

\|Pf​i​n​a​l\|≥Nm​a​x\|P\_\{final\}\|\\geq N\_\{max\}then

19:break

20:endif

21:endif

22:endfor

23:endwhile

24:return

Pf​i​n​a​lP\_\{final\}

### B\.3Re\-ranking Modules

- •Qwen3embeddings\(Zhanget al\.,[2025a](https://arxiv.org/html/2605.29234#bib.bib6)\)served via vLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2605.29234#bib.bib12)\), scoring each candidate by cosine similarity between the instruction\-conditioned query embedding and the bare candidate embedding\.
- •LLM debate ranking: an LLM agent scores each candidate \(0–100\) with arguments\-for/against conditioned on the query paper’s full text\. Batches run asynchronously under a concurrency semaphore\.
- •Debate\+\+Qwen3ensemble: averages the two normalized scores into a single per\-candidate ranking\.

### B\.4Engineering Notes

All experiments use asynchronous execution with bounded parallelism, query/result caching keyed by input hash, and YAML\-driven configuration\. API semaphores respect provider limits while maintaining throughput\.

## Appendix CPrompt Templates

This appendix lists the four prompts the Deep Research pipeline and LLM\-as\-a\-judge evaluation use: \(i\) keyword generation for Phase 1 retrieval \(Prompt[5](https://arxiv.org/html/2605.29234#A3.T5)\), \(ii\) LLM debate ranking \(Prompt[6](https://arxiv.org/html/2605.29234#A3.T6)\), \(iii\) the instruction prefix forQwen3embedding queries \(Prompt[7](https://arxiv.org/html/2605.29234#A3.T7)\), and \(iv\) the semantic\-relevance judge \(Prompt[8](https://arxiv.org/html/2605.29234#A3.T8)\) that produces all scores in Sections[4](https://arxiv.org/html/2605.29234#S4)–[5](https://arxiv.org/html/2605.29234#S5)\.

### C\.1Keyword Generation

Phase 1 prompts an LLM to draft a diverse set of search queries given the query paper’s abstract\.

Table 5:Keyword generation prompt \(Phase 1 of Deep Research\)\.
### C\.2Debate Ranking

The debate re\-ranker \(Table[6](https://arxiv.org/html/2605.29234#A3.T6)\) scores each candidate on a 0–100 scale by first writing arguments for and against citing it\. We batch candidates per LLM call and share the system prompt with the keyword\-generation step\.

Table 6:Debate ranking prompt\. Arguments\-for/against and a 0–100 score are produced per candidate\.
### C\.3Qwen3Embedding Instruction

Following the recommended usage ofQwen3\-Embedding\-8B\(Zhanget al\.,[2025a](https://arxiv.org/html/2605.29234#bib.bib6)\), we prefix each query paper with a one\-sentence task instruction before embedding, and embed candidate abstracts without one\. Cosine similarity between instruction\-conditioned query embeddings and bare candidate embeddings produces the final ranking\.

Table 7:Qwen3\-Embedding\-8Binstruction prefix applied to the query side only\.
### C\.4Semantic Relevance Judge

Every Semantic Relevance score we report \(the matched\-KKdistribution in Table[3](https://arxiv.org/html/2605.29234#S5.T3), the cumulative curves in Figure[3](https://arxiv.org/html/2605.29234#S5.F3), and the per\-method scores in Table[2](https://arxiv.org/html/2605.29234#S4.T2)\) comes from the prompt in Table[8](https://arxiv.org/html/2605.29234#A3.T8)\. We show the judge the query paper \(title, abstract, optionally full text\) and a single candidate \(title, abstract\); it returns an integer in\{0,1,2,3,4,5\}\\\{0,1,2,3,4,5\\\}, which we multiply by 20 to obtain the 0–100 scale used in the main text\.

Table 8:Semantic relevance judge prompt\. Returns an integer 0–5 per \(query, candidate\) pair; we multiply by 20 to obtain the 0–100 scale used throughout the paper\.

## Appendix DMetrics: Definitions

#### Precision and Recall\.

For retrieved setR​@​KR@Kand ground\-truth setGG,

Precision@K=\|R​@​K∩G\|\|R​@​K\|\\text\{Precision@K\}=\\frac\{\|R@K\\cap G\|\}\{\|R@K\|\}Recall@K=\|R​@​K∩G\|\|G\|\\text\{Recall@K\}=\\frac\{\|R@K\\cap G\|\}\{\|G\|\}We report curves forKKup to1\.5×1051\.5\{\\times\}10^\{5\}to assess long\-tail retrieval\.

#### LLM\-as\-a\-judge relevance\.

For each query–candidate pair, an LLM\(Zhenget al\.,[2023](https://arxiv.org/html/2605.29234#bib.bib13)\)assigns a graded relevance score on a 5\-point rubric, which we multiply by 20 to obtain a 0–100 scale\. The judge conditions on the query paper and the candidate’s title and abstract\.

#### 𝜶\\boldsymbol\{\\alpha\}\-nDCG\.

We extend nDCG\(Järvelin and Kekäläinen,[2002](https://arxiv.org/html/2605.29234#bib.bib10)\)with the novelty\-aware variant ofClarkeet al\.\([2008](https://arxiv.org/html/2605.29234#bib.bib9)\)\. For each retrieved documentddthat belongs to a ground\-truth clustercc, the contribution at rankrris

gain​\(d,r\)=1log2⁡\(r\+1\)⋅\(1−α\)​αnc,\\text\{gain\}\(d,r\)=\\frac\{1\}\{\\log\_\{2\}\(r\+1\)\}\\cdot\(1\-\\alpha\)\\,\\alpha^\{n\_\{c\}\},wherencn\_\{c\}is the number of earlier documents from clustercc\. We normalize by the ideal gain to obtainα\\alpha\-nDCG@K\. Clusters come from HDBSCAN\(McInneset al\.,[2017](https://arxiv.org/html/2605.29234#bib.bib11)\)over Qwen3 embeddings\(Zhanget al\.,[2025a](https://arxiv.org/html/2605.29234#bib.bib6)\)of candidate\-paper abstracts, with the defaultmin\_cluster\_size, so cluster identity reflects topical neighborhoods of the candidate pool\.

## Appendix ECo\-Authorship Graph Distance: Definition and Computation

This appendix formalizes the co\-authorship\-graph distance metric from Section[5](https://arxiv.org/html/2605.29234#S5)and documents the data we fetch to compute it\.

#### Graph and distance\.

LetG=\(V,E\)G=\(V,E\)be the undirected co\-authorship graph in whichVVis the set of academic authors and an edge\(a,b\)∈E\(a,b\)\\in Eexists wheneveraaandbbhave co\-authored at least one publication\. For a query paperQQwith author setAQA\_\{Q\}and a candidate paperCCwith author setACA\_\{C\}, the pair distance is

d​\(Q,C\)=mina∈AQ,b∈AC⁡dG​\(a,b\),d\(Q,C\)\\;=\\;\\min\_\{a\\in A\_\{Q\},\\;b\\in A\_\{C\}\}d\_\{G\}\(a,b\),\(1\)wheredG​\(⋅,⋅\)d\_\{G\}\(\\cdot,\\cdot\)is the standard shortest\-path distance onGG, withdG​\(a,a\)=0d\_\{G\}\(a,a\)=0anddG​\(a,b\)=∞d\_\{G\}\(a,b\)=\\inftyfor disconnected pairs\.

#### Hop neighborhoods\.

For a setS⊆VS\\subseteq V,

L0​\(S\)\\displaystyle L\_\{0\}\(S\)=S,\\displaystyle=S,Lk\+1​\(S\)\\displaystyle L\_\{k\+1\}\(S\)=⋃u∈Lk​\(S\)N​\(u\),\\displaystyle=\\bigcup\_\{u\\in L\_\{k\}\(S\)\}N\(u\),whereN​\(u\)=\{v:\(u,v\)∈E\}N\(u\)=\\\{v:\(u,v\)\\in E\\\}is the set of co\-authors ofuu\.

#### Distance test\.

We determine the pair distance by checking each level in increasing order:

d=0:\\displaystyle d\{=\}0:\\;\\;AQ∩AC≠∅,\\displaystyle A\_\{Q\}\\cap A\_\{C\}\\neq\\emptyset,d=1:\\displaystyle d\{=\}1:\\;\\;L1​\(AQ\)∩AC≠∅,\\displaystyle L\_\{1\}\(A\_\{Q\}\)\\cap A\_\{C\}\\neq\\emptyset,d=2:\\displaystyle d\{=\}2:\\;\\;\(L1​\(AQ\)∩L1​\(AC\)≠∅\)∨\\displaystyle\\bigl\(L\_\{1\}\(A\_\{Q\}\)\\cap L\_\{1\}\(A\_\{C\}\)\\neq\\emptyset\\bigr\)\\lor\(L2​\(AQ\)∩AC≠∅\),\\displaystyle\\bigl\(L\_\{2\}\(A\_\{Q\}\)\\cap A\_\{C\}\\neq\\emptyset\\bigr\),d=3:\\displaystyle d\{=\}3:\\;\\;L2​\(AQ\)∩L1​\(AC\)≠∅\.\\displaystyle L\_\{2\}\(A\_\{Q\}\)\\cap L\_\{1\}\(A\_\{C\}\)\\neq\\emptyset\.We record pairs failing all four tests asd≥4d\\geq 4\. Thed=2d\{=\}2disjunction is necessary because a length\-2 path takes either form: a shared collaborator, or a second\-hop reach\.

#### Data requirements\.

We materialize three subsets:

- •L1​\(AQ\)L\_\{1\}\(A\_\{Q\}\)requires the co\-author list of each of the 1,113 unique query authors\.
- •L1​\(AC\)L\_\{1\}\(A\_\{C\}\)requires the co\-author list of each candidate author that is not already inL1​\(AQ\)∪AQL\_\{1\}\(A\_\{Q\}\)\\cup A\_\{Q\}\(52,868 authors after exclusion\)\.
- •L2​\(AQ\)L\_\{2\}\(A\_\{Q\}\)requires the co\-author list of each author inL1​\(AQ\)L\_\{1\}\(A\_\{Q\}\)\(114,669 authors\), which is the dominant cost\.

d=3d\{=\}3detection requires*both*the candidate\-side fetch and the second\-hop expansion: thed=3d\{=\}3intersectionL2​\(AQ\)∩L1​\(AC\)L\_\{2\}\(A\_\{Q\}\)\\cap L\_\{1\}\(A\_\{C\}\)draws one factor from each side\.

#### Bounded\-degree approximation\.

We construct each author’s co\-author list by aggregating across the OpenAlex works in 5 pages of 200 results\. Under\-counting can only inflate the estimated distance, so the reported gap is a conservative lower bound\.

#### Identifier reconciliation\.

We resolve author identities through OpenAlex identifiers\(Priemet al\.,[2022](https://arxiv.org/html/2605.29234#bib.bib8)\), and map arXiv papers to OpenAlex Works via the batched DOI filter using the prefix10\.48550/arxiv\.\. Of 22,935 unique arXiv identifiers, 19,990 \(87\.2%\) resolve; we exclude pairs in which either side fails\.

#### Cost\.

OpenAlex’s batched author\-id filter accepts up to 50 IDs per request at $0\.0001 per request under the prepaid plan\. The complete pipeline costs approximately $1\.33 in API spend and 4–5 hours of wall\-clock time with parallel fetches\.

## Appendix FQualitative Examples of Low\-Scoring Human References

Table[9](https://arxiv.org/html/2605.29234#A6.T9)lists eighteen representative human\-curated references that receive a low semantic relevance score \(0 or 20 on the 0–100 rubric\)\. Examples are sampled to cover the recurring categories of non\-topical citation we observe across the 250\-paper set: foundational architectures \(Inception, ResNet\(Heet al\.,[2016](https://arxiv.org/html/2605.29234#bib.bib17)\), Llama\-3, DDPM\), classical algorithms \(Adam\(Kingma and Ba,[2015](https://arxiv.org/html/2605.29234#bib.bib15)\), CMA\-ES, GELU, the Concrete distribution\), software libraries \(PyTorch, scikit\-learn, CatBoost\), benchmarks \(ImageNet, DQN/Atari\), and cross\-domain references \(autonomous\-vehicle localization, asset pricing, OpenAlex\)\. These citations are common and typically well\-motivated in the body, yet their abstracts contain no topical signal that an abstract\-conditioned judge can use\.

Table 9:Example low\-scoring human references with the judge’s abridged rationale\. Score is on the 0–100 rubric\. All examples drawn from the 250\-paper human ground\-truth set used in Section[5](https://arxiv.org/html/2605.29234#S5)\.
## Appendix GArtifact Licenses and Use

We use the following pre\-existing artifacts in their published configurations:GPT\-OSS\-120B\(Apache 2\.0\),Qwen3\-Embedding\-8B\(Zhanget al\.,[2025a](https://arxiv.org/html/2605.29234#bib.bib6)\)\(Apache 2\.0\),vLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2605.29234#bib.bib12)\)\(Apache 2\.0\), andHDBSCAN\(McInneset al\.,[2017](https://arxiv.org/html/2605.29234#bib.bib11)\)\(BSD\-3\)\. Reference data comes from OpenAlex\(Priemet al\.,[2022](https://arxiv.org/html/2605.29234#bib.bib8)\)\(CC0\), Semantic Scholar\(Kinneyet al\.,[2023](https://arxiv.org/html/2605.29234#bib.bib7)\)\(standard API terms of service\), and arXiv preprints \(per\-author licenses selected at submission\)\. All uses align with the research purposes the source artifacts permit\. The artifacts we will release \(the Deep Research pipeline code, the LLM\-judge prompts, the OpenAlex distance\-analysis scripts, andRollingEval\-Jun25\) will be made available under Apache 2\.0 for code and CC\-BY\-4\.0 for the benchmark\.