Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE

Hugging Face Daily Papers Papers
long-context zero-shot rope attention-mechanism efficient-inference dynamic-scaling

Summary

Introduces Jet-Long, a zero-shot method for long-context extension that dynamically adapts rescaling factors and uses a bifocal attention mechanism, achieving efficient and high-performance processing across varying sequence lengths without retraining.

Modern LLMs are increasingly deployed in long-context applications such as retrieval-augmented generation, repository-level coding, and agentic workflows whose accumulated reasoning and tool traces routinely push the input an order of magnitude past the pretraining window, making zero-shot context extension the dominant deployment path for open-weight checkpoints. Most existing zero-shot methods fix a single rescaling factor up front, so an aggressive factor sacrifices short-context fidelity while a conservative one breaks down at long contexts. We propose Jet-Long, a tuning-free zero-shot method that pairs a local RoPE-faithful window with a long-range window whose rescaling factor adapts dynamically to the current sequence length, recovering the base model exactly at short inputs while extrapolating cleanly at long ones. An inclusion-exclusion attention merge and an on-the-fly RoPE correction rotation make the bifocal construction essentially free at inference; fused into a single CuTe kernel, long-context prefill reaches up to 1.39times FA2 throughput on H100 (approaching the Hopper-only FA4), and single-batch generation incurs le 4% overhead at every length. On Qwen3-1.7B/4B/8B up to 128K context, Jet-Long leads RULER by +4.79/+2.18/+2.03~pp over the strongest baseline at 1.7B/4B/8B, achieves the best overall accuracy on HELMET-RAG (a benchmark identified by HELMET as the most efficient predictor of downstream long-context performance) and attains the lowest PG-19 perplexity. Jet-Long also generalizes to hybrid attention architectures such as Jet-Nemotron for further long-context improvement without retraining, and remains hyperparameter-resilient for ease of deployment.
Original Article
View Cached Full Text

Cached at: 07/10/26, 06:17 AM

Paper page - Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE

Source: https://huggingface.co/papers/2607.07740

Abstract

A novel zero-shot method called Jet-Long enables efficient long-context processing for large language models by dynamically adapting rescaling factors and utilizing a bifocal attention mechanism that maintains high performance across varying sequence lengths.

ModernLLMsare increasingly deployed inlong-context applicationssuch asretrieval-augmented generation, repository-level coding, andagentic workflowswhose accumulated reasoning and tool traces routinely push the input an order of magnitude past the pretraining window, makingzero-shot context extensionthe dominant deployment path for open-weight checkpoints. Most existing zero-shot methods fix a single rescaling factor up front, so an aggressive factor sacrifices short-context fidelity while a conservative one breaks down at long contexts. We propose Jet-Long, a tuning-free zero-shot method that pairs a localRoPE-faithful windowwith along-range windowwhose rescaling factor adapts dynamically to the current sequence length, recovering the base model exactly at short inputs while extrapolating cleanly at long ones. An inclusion-exclusionattention mergeand an on-the-flyRoPE correction rotationmake thebifocal constructionessentially free at inference; fused into a singleCuTe kernel, long-context prefill reaches up to 1.39timesFA2throughput onH100(approaching theHopper-onlyFA4), and single-batch generation incurs le 4% overhead at every length. On Qwen3-1.7B/4B/8B up to 128K context, Jet-Long leadsRULERby +4.79/+2.18/+2.03~pp over the strongest baseline at 1.7B/4B/8B, achieves the best overall accuracy onHELMET-RAG(a benchmark identified by HELMET as the most efficient predictor of downstream long-context performance) and attains the lowestPG-19 perplexity. Jet-Long also generalizes tohybrid attention architecturessuch asJet-Nemotronfor further long-context improvement without retraining, and remains hyperparameter-resilient for ease of deployment.

View arXiv pageView PDFGitHub0Add to collection

Get this paper in your agent:

hf papers read 2607\.07740

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.07740 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.07740 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.07740 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

EndPrompt: Efficient Long-Context Extension via Terminal Anchoring

arXiv cs.CL

EndPrompt proposes a method for extending the context window of large language models using only short training sequences, by anchoring a terminal prompt with target-length positional indices. It achieves strong benchmark results with substantially less computation than full-length fine-tuning.

LongAttnComp: Cross-Family Context Compression for Long-Context Reasoning

Hugging Face Daily Papers

LongAttnComp adapts AttnComp for long-context reasoning by fine-tuning lightweight cross-attention layers and introducing token-level chunking, a top-p algorithm, positional reordering, and a query parser. It achieves strong performance on long-context tasks like code debugging and transfers across multiple model families.

OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling

arXiv cs.AI

Introduces OctoLong, a context engineering pipeline for curating dependency-rich cross-repository code contexts, and OctoLong-Instruct, a suite of long-context open LMs trained on this data. Experiments show that replacing 12% of traditional long-context corpora with OctoLong data yields substantial gains in long-range retrieval, state tracking, repository-level code understanding, and agentic tasks.