LLMs for Academic Workflows: An Evaluation of Literature Reviews Generated with Short and Long Context Windows of LLMs
Summary
This paper evaluates literature reviews generated by large language models using short and long context windows, finding that longer contexts improve information incorporation but exacerbate issues like repetition and descriptiveness, necessitating human oversight for academic standards.
View Cached Full Text
Cached at: 08/28/26, 09:28 AM
# LLMs for Academic Workflows: An Evaluation of Literature Reviews Generated with Short and Long Context Windows of LLMs Source: [https://arxiv.org/abs/2608.26145](https://arxiv.org/abs/2608.26145) [View PDF](https://arxiv.org/pdf/2608.26145) > Abstract:Our research focuses on evaluating literature reviews generated in short and long context settings of large language models \(LLMs\) to investigate the impact of context window on the quality of AI\-generated literature reviews and the role of AI in supporting literature review writing\. Twenty AI\-generated literature reviews based on research sources from Semantic Scholar and Arxiv were evaluated by two researchers across 15 dimensions\. Our findings reveal that AI\-generated literature reviews require human oversight to meet academic publishing standards\. As context windows increase, LLMs can incorporate broader information and maintain coherence across longer inputs, but they also exacerbate issues such as content repetition, omission of critical work, and a tendency towards descriptiveness over synthesis\. Our work shows that AI\-generated reviews can provide foundational overviews, but their output must be critically evaluated and refined by domain experts\. Future research should consider integrating other LLMs and fine\-tuned models in different domains with hybrid approaches that combine human expertise with AI capabilities to address the limitations identified in this study\. ## Submission history From: Muhammad Chaudhry Mr \[[view email](https://arxiv.org/show-email/464865a0/2608.26145)\] **\[v1\]**Sun, 28 Jun 2026 19:03:56 UTC \(454 KB\)
Similar Articles
LLM Wiki v2 (16 minute read)
This post presents a pattern for building personal knowledge bases using LLMs, offering a structured approach for leveraging large language models in knowledge management.
Review Arcade: On the Human Alignment and Gameability of LLM Reviews
This paper investigates the alignment of LLM-generated reviews with human judgment using 1k real ACL 2025 submissions, finding limited agreement, instability across models/prompts, and a method to artificially inflate scores without meaningful changes. The authors advise against relying solely on LLM reviews and call for discussion on their use in handling increasing submission volumes.
Can LLMs Judge Better Than They Generate? Evaluating Task Asymmetry, Mechanistic Interpretability and Transferability for In-Context QA
This paper tests the assumption that LLMs judge better than they generate in in-context QA, finding generation accuracy exceeds self-evaluation on most benchmarks, with evaluation attending less to context. The findings challenge core assumptions in self-evaluation pipelines.
When Do LLMs Actually Help? Evaluating LLMs as Data Quality Annotators
This study evaluates LLMs as data quality annotators on e-commerce tasks, finding they outperform baselines when background knowledge is required but offer limited advantages for tasks with strong lexical signals, while demonstrating high consistency across runs.
Can LLMs Reason in a Legally Meaningful Manner? A Small-scale Study on European Court of Human Rights Cases
This small-scale study evaluates LLM reasoning in legal case forecasting using European Court of Human Rights cases, finding that models produce structurally complete but substantively shallow analyses and that LLM-based evaluators align weakly with human annotators.