FocuSFT: Bilevel Optimization for Dilution-Aware Long-Context Fine-Tuning
Summary
The paper introduces FocuSFT, a bilevel optimization framework that enhances long-context language model performance by addressing attention dilution through parametric memory. It demonstrates significant improvements in accuracy and context engagement on benchmarks like BABILong and RULER.
View Cached Full Text
Cached at: 05/13/26, 08:12 AM
Paper page - FocuSFT: Bilevel Optimization for Dilution-Aware Long-Context Fine-Tuning
Source: https://huggingface.co/papers/2605.09932
Abstract
Training framework FocuSFT improves long-context language model performance by addressing attention allocation issues through bilevel optimization with parametric memory that focuses attention on semantically relevant content.
Large language models can now process increasingly long inputs, yet their ability to effectively use information spread across long contexts remains limited. We trace this gap to howattention budgetis spent duringsupervised fine-tuning(SFT) on long sequences:positional biasesandattention sinkscause the model to allocate most of its attention to positionally privileged tokens rather than semantically relevant content. This training-timeattention dilution(the starvation of content tokens in the attention distribution) weakens the gradient signal, limiting the model’s ability to learn robust long-context capabilities. We introduce FocuSFT, abilevel optimizationframework that addresses this problem at training time. An inner loop adapts lightweightfast-weight parameterson the training context to form aparametric memorythat concentrates attention on relevant content, and the outer loop performs SFT conditioned on this sharpened representation. Both loops apply bidirectional attention over context tokens while preservingcausal maskingfor responses, reducing the causal asymmetry that gives rise toattention sinksand aligning inner-outer behavior. On BABILong, FocuSFT improves accuracy by up to +14pp across 4K--32K context lengths; on RULER, it raises CWE aggregation from 72.9\% to 81.1\% at 16K; and on GPQA with agentic tool use, it yields a 24\% relative gain in pass@1. Attention analysis shows that FocuSFT reducesattention sink massby 529times and triplescontext engagementduring training. Code: https://github.com/JarvisPei/FocuSFT
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2605\.09932
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.09932 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.09932 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.09932 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
LiFT: Does Instruction Fine-Tuning Improve In-Context Learning for Longitudinal Modelling by Large Language Models?
LiFT is a longitudinal instruction fine-tuning framework that unifies diverse temporal NLP tasks under a shared instruction schema with curriculum-based training. Evaluated across OLMo, LLaMA, and Qwen models, LiFT consistently outperforms base-model in-context learning, especially on out-of-distribution data and rare change events.
Learnability-Informed Fine-Tuning of Diffusion Language Models
We propose LIFT, a learnability-informed fine-tuning algorithm for diffusion language models that aligns training with token difficulty and time step, achieving substantial gains on reasoning benchmarks.
MixSD: Mixed Contextual Self-Distillation for Knowledge Injection
MixSD proposes a self-distillation method for knowledge injection in language models that aligns supervision with the model's native distribution, reducing catastrophic forgetting during fine-tuning. It achieves near-perfect memorization while retaining up to 100% of base capabilities, vastly outperforming standard SFT.
Shared SFT Lessons Across Alignment, Model Organisms, and Toy Models
This paper studies transferring lessons about supervised fine-tuning (SFT) across alignment training, model organisms, and toy models, showing that techniques like training on reasons for behavior and mixing on-model data can improve generalization and capability preservation.
Foundation-Preserving Adaptation via Generalized Rayleigh-Quotient Optimization
Proposes FoLoRA, a forgetting-aware optimization framework for fine-tuning foundation models that balances task utility and forgetting penalty via generalized Rayleigh-quotient optimization, achieving better preservation of non-target capabilities.