tencent/HiLS-Attention-7B · Hugging Face

Source: https://huggingface.co/tencent/HiLS-Attention-7B

HiLS-Attentionis a chunk-wise sparse attention mechanism that learns chunk selection end-to-end under the language-modeling loss, enabling native sparse training for efficient long-context modeling. This repository hosts the7Bcheckpoint continued-trained on top of an OLMo3-style backbone.

Model introduced in the paperHierarchical Sparse Attention Done Right: Toward Infinite Context Modeling.

https://huggingface.co/tencent/HiLS-Attention-7B#model-descriptionModel Description

Naive block sparse attention selects top-k chunks by their exact chunk mass, but computing all chunk masses requires full QK computation. HiLS-Attention instead usescompressed chunk keysto estimate a chunk-mass surrogate andfactorizes attention into inter-chunk and intra-chunk softmax, enabling end-to-end learning from the next-token prediction loss.

Overview of HiLS-Attention. Naive block sparse attention selects top-k chunks by their exact chunk mass, but computing all chunk masses requires full QK computation. HiLS-Attention instead uses compressed chunk keys to estimate a chunk-mass surrogate and factorizes attention into inter-chunk and intra-chunk softmax, enabling end-to-end learning from the next-token prediction loss.

• Parameters:~7B
• **Base architecture:**OLMo3-7B
• Paper:https://arxiv.org/abs/2607.02980
• Code:https://github.com/Tencent-Hunyuan/HiLS-Attention

https://huggingface.co/tencent/HiLS-Attention-7B#highlightsHighlights

After only50B continued-training tokens, HiLS-Attention inherits the capability of full attention while bringing two key advantages:

• Strong ultra-long context extrapolationbeyond the YaRN-extended 4× length.
• Faster inferencefor long sequences.

Meanwhile, it preserves comparable performance for short- and medium-context tasks, within both the original training length and the YaRN-extrapolated range.

After only 50B continued-training tokens, HiLS-Attention inherits the capability of full attention while bringing two key advantages:strong ultra-long context extrapolationbeyond the YaRN-extended 4× length (a) andfaster inference(b). Meanwhile, it preserves comparable performance for short- and medium-context tasks, within both the original training length and the YaRN-extrapolated range (c & d).

https://huggingface.co/tencent/HiLS-Attention-7B#intended-usesIntended Uses

• Long-context language modeling and inference (retrieval, QA, summarization over long documents).
• Research on native sparse-attention training and long-context extrapolation.

https://huggingface.co/tencent/HiLS-Attention-7B#out-of-scopeOut-of-Scope

Not aligned/instruction-tuned for safety-critical deployment. Outputs may be inaccurate or biased; validate before production use.

https://huggingface.co/tencent/HiLS-Attention-7B#how-to-useHow to Use

HiLS-Attention cannot be loaded directly via the standardtransformers``AutoModelAPI — the custom hierarchical sparse attention must be registered through theHiLS-Attentioncodebase. Please set up the environment and run the model following theGitHub repository, in particular theEvaluationsection (seeeval/andscripts/eval/), which shows how to load the checkpoint with the corresponding HiLS config and run generation / evaluation.

For efficient long-context serving, SGLang inference support is on the roadmap — see theGitHub repofor updates.

https://huggingface.co/tencent/HiLS-Attention-7B#evaluationEvaluation

The model is evaluated across short-, medium-, and long-context benchmarks. Full scripts are available in theGitHub repository.

https://huggingface.co/tencent/HiLS-Attention-7B#long-contextLong-context

BenchmarkDescriptionLongBench v1Long-context QA (21 tasks, up to 64K input)RULERSynthetic probing at 8K / 16K / 32K / 128KPPLPerplexity across 64 … 256K sequence lengths

https://huggingface.co/tencent/HiLS-Attention-7B#short-context-opencompass-11-benchmarksShort-context (OpenCompass, 11 benchmarks)

BenchmarkTypeMMLU, GPQA, HellaSwag, ARC-c, BoolQ, RACEFew-shot PPLGSM8K, CMathMath generation (CoT)HumanEval+, MBPP+, CRUXEval-OCode generation

https://huggingface.co/tencent/HiLS-Attention-7B#trainingTraining

• **Method:**Continued pre-training with native sparse-attention (HiLS-Attention) on top of an OLMo3-7B backbone.
• Tokens:~50B continued-training tokens.
• **Positional encoding / long-context:**HoPE with YaRN-based length extrapolation.

Training and continue-pre-training scripts are documented in theGitHub repository.

https://huggingface.co/tencent/HiLS-Attention-7B#limitations-and-biasLimitations and Bias

This is a pretrained base model without alignment or safety tuning. It may reflect biases present in the training corpus and can produce inaccurate or unsafe content. Users are responsible for evaluating suitability for their use case.

https://huggingface.co/tencent/HiLS-Attention-7B#citationCitation

@misc{hu2026hierarchicalsparseattentionright,
      title={Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling},
      author={Xiang Hu and Xinyu Wei and Hao Gu and Minshen Zhang and Tian Liang and Huayang Li and Lei Zhu and Yan Wang and Sirui Han and Yushi Bai and Kewei Tu and Haitao Mi and Leo Liang},
      year={2026},
      eprint={2607.02980},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2607.02980},
}