FeatCal: Feature Calibration for Post-Merging Models
Summary
FeatCal is a calibration method that reduces performance gaps in post-merging models by layer-wise weight updates without gradient descent, achieving superior results on CLIP and GLUE benchmarks with high sample efficiency.
View Cached Full Text
Cached at: 05/14/26, 12:18 PM
Paper page - FeatCal: Feature Calibration for Post-Merging Models
Source: https://huggingface.co/papers/2605.13030
Abstract
Feature drift analysis in model merging leads to FeatCal, a calibration method that reduces performance gaps through layer-wise weight updates without gradient descent, achieving superior benchmark results and efficiency.
Model mergingcombinestask expertsinto one model and avoids joint training, retraining, or deploying many expert models, but the merged model often still underperformstask experts. We study this performance gap throughfeature drift, the difference between features produced by the merged model and by the expert on the same input. Our theory decomposes this drift into upstream propagation and local mismatch, tracks how it propagates and combines through later layers inforward order, and links finalfeature driftto output drift. This view motivates FeatCal, which uses a smallcalibration setto calibrate the mergedmodel weightslayer by layer inforward order, reducingfeature driftwhile staying close to merged weights and preserving the benefits ofmodel merging. FeatCal uses an efficientclosed-form solutionto updatemodel weights, with no gradient descent, iterative optimization, or extra modules. On the main CLIP and GLUE benchmarks, FeatCal beats Surgery and ProbSurgery, the closest post-merging calibration baselines: 85.5% vs. 77.0%/78.8% on CLIP-ViT-B/32 Task Arithmetic (TA) and 85.2% vs. 83.7%/82.2% on FLAN-T5-base GLUE. On CLIP-ViT-B/32, 8 examples per task reach 82.9%, and 256 examples per task take 53 seconds, about 4x faster than both baselines, showing bettersample efficiencyand lowercalibration cost.
View arXiv pageView PDFProject pageGitHub2Add to collection
Get this paper in your agent:
hf papers read 2605\.13030
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.13030 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.13030 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.13030 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
Before Fusion, Ask What to Keep: Contextual Calibration of Multimodal Signals
This paper introduces a plug-in calibration module that adjusts multimodal representations before fusion, using cross-modal context to suppress misleading signals and emphasize reliable ones, improving performance on multiple benchmarks.
FAIR-Calib: Frontier-Aware Instability-Reweighted Calibration for Post-Training Quantization of Diffusion Large Language Models
This paper proposes FAIR-Calib, a two-stage post-training quantization framework for diffusion large language models that addresses the instability of token commitments during iterative refinement. It achieves state-of-the-art results on LLaDA and Dream models under low-bit quantization.
Model Merging by Output-Space Projection
This paper presents a new framework for model merging that casts the problem as a convex quadratic program over residual updates, minimizing a squared-output calibration objective. It subsumes existing heuristic methods and provides a closed-form diagnostic to predict merge quality, showing consistent gains on language and vision benchmarks.
E-PMQ: Expert-Guided Post-Merge Quantization with Merged-Weight Anchoring
This paper introduces E-PMQ, an expert-guided post-merge quantization framework that addresses the combined deviations from merging and quantization, achieving significant accuracy improvements on multi-task merged models like CLIP-ViT and FLAN-T5.
LeapAlign: Post-Training Flow Matching Models at Any Generation Step by Building Two-Step Trajectories
LeapAlign is a post-training method that improves flow matching model alignment with human preferences by reducing computational costs through two-step trajectory shortcuts while enabling stable gradient propagation to early generation steps. The method outperforms state-of-the-art approaches when fine-tuning Flux models across various image quality and text-alignment metrics.