OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling
Summary
Introduces OctoLong, a context engineering pipeline for curating dependency-rich cross-repository code contexts, and OctoLong-Instruct, a suite of long-context open LMs trained on this data. Experiments show that replacing 12% of traditional long-context corpora with OctoLong data yields substantial gains in long-range retrieval, state tracking, repository-level code understanding, and agentic tasks.
View Cached Full Text
Cached at: 08/06/26, 07:43 AM
# OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling Source: [https://arxiv.org/abs/2608.05141](https://arxiv.org/abs/2608.05141) [View PDF](https://arxiv.org/pdf/2608.05141) > Abstract:Context lengths of language models \(LMs\) have dramatically increased, driven by the demands for in\-context learning, self\-improvement, and long\-horizon agentic workflows\. Existing long\-context corpora, however, are dominated by books, academic articles, and code repositories, which are finite resources and often scarce in long\-distance dependencies\. In this work, we introduce OctoLong, a context engineering pipeline that instruments an AST parser, a language server backend, and a package manager to facilitate the recursive retrieval of code references, enabling the curation of dependency\-rich code contexts of millions of tokens in length\. We then train OctoLong\-Instruct, a suite of capable long\-context open LMs, derived from base models ranging in size from 600M to 14B parameters, via context\-extension mid\-training on a ~50B\-token mixture containing ~6\.2B tokens of OctoLong code contexts, followed by ~10B tokens of instruction tuning\. Our training ablations and experimental evaluations against 18 state\-of\-the\-art open\-weight long\-context LMs show that supplanting just 12% of traditional context\-extension corpora with OctoLong data yields substantial gains in long\-range retrieval, long\-term state tracking, repository\-level code understanding, and downstream agentic tasks, while also enhancing API usage in short\-context coding scenarios\. ## Submission history From: Indraneil Paul Mr\. \[[view email](https://arxiv.org/show-email/2d28635f/2608.05141)\] **\[v1\]**Wed, 5 Aug 2026 17:58:15 UTC \(640 KB\)
Similar Articles
LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning
LongAttnComp adapts AttnComp for long-context reasoning by fine-tuning lightweight cross-attention layers and introducing token-level chunking, a top-p algorithm, positional reordering, and a query parser. It achieves strong performance on long-context tasks like code debugging and transfers across multiple model families.
Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE
Jet-Long introduces a tuning-free zero-shot method for extending LLM context length by dynamically adjusting RoPE scaling, achieving strong performance on benchmarks up to 128K context with minimal inference overhead.
Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE
Introduces Jet-Long, a zero-shot method for long-context extension that dynamically adapts rescaling factors and uses a bifocal attention mechanism, achieving efficient and high-performance processing across varying sequence lengths without retraining.
Less Context, Better Agents: Efficient Context Engineering for Long-Horizon Tool-Using LLM Agents
This paper evaluates context engineering configurations for LLM agents in enterprise tool-use workflows, showing that summarization with selective pruning achieves 91.6% accuracy while reducing token usage by over 60% compared to full-context baselines.
GoLongRL: Capability-Oriented Long Context Reinforcement Learning with Multitask Alignment
GoLongRL presents an open-source approach for long-context reinforcement learning with diverse reward optimization through capability-oriented data construction and TMN-Reweight methodology.