ReGain: Restoring Subject Fidelity in Personalization on Synthetic Images

Hugging Face Daily Papers Papers
diffusion-models personalization dreambooth classifier-free-guidance text-to-image training-free

Summary

ReGain 提出一种训练无关的采样时校正方法，通过按频率带缩放被放大的 CFG 引导，恢复基于合成图像微调的个性化扩散模型的主体保真度，在 SD1.5、SDXL 和 SD3.5 上均有效。

Text-to-image diffusion models are personalized to a subject by DreamBooth fine-tuning on a handful of its images. Increasingly, these images come from a diffusion model rather than a camera. We show that fine-tuning on such synthetic images degrades subject fidelity, producing oversaturated color and excess high-frequency detail. To isolate the cause, we fine-tune two models from the same base model with the same DreamBooth recipe, one on real photos of a subject and one on synthetic images of that subject generated by the first. We trace the degradation to classifier-free guidance (CFG). For the model personalized on synthetic images, the angle between the conditional and unconditional noise predictions, and with it the norm of their difference, is much larger than for the model personalized on real photos. This inflation grows toward high frequencies and also appears at other prompts semantically close to the subject, such as its class noun, but not at unrelated ones. We propose ReGain, a training-free correction applied at sampling time that measures how much each frequency band of the guidance is inflated relative to the base model and scales that band down accordingly. ReGain needs no real photos. On Stable Diffusion v1.5, ReGain closes 51-64% of the subject-fidelity gap to the model personalized on real photos, as measured by DINO, DINOv2 and CLIP-I. It also improves subject fidelity on SDXL and SD 3.5 and preserves text alignment on all three backbones.
Original Article
View Cached Full Text

Cached at: 10/09/26, 06:29 AM

Paper page - ReGain: Restoring Subject Fidelity in Personalization on Synthetic Images

Source: https://huggingface.co/papers/2609.38680

Abstract

Text-to-imagediffusionmodelsarepersonalizedtoasubjectbyDreamBoothfine-tuningonahandfulofitsimages.Increasingly,theseimagescomefromadiffusionmodelratherthanacamera.Weshowthatfine-tuningonsuchsyntheticimagesdegradessubjectfidelity,producingoversaturatedcolorandexcesshigh-frequencydetail.Toisolatethecause,wefine-tunetwomodelsfromthesamebasemodelwiththesameDreamBoothrecipe,oneonrealphotosofasubjectandoneonsyntheticimagesofthatsubjectgeneratedbythefirst.Wetracethedegradationtoclassifier-freeguidance(CFG).Forthemodelpersonalizedonsyntheticimages,theanglebetweentheconditionalandunconditionalnoisepredictions,andwithitthenormoftheirdifference,ismuchlargerthanforthemodelpersonalizedonrealphotos.Thisinflationgrowstowardhighfrequenciesandalsoappearsatotherpromptssemanticallyclosetothesubject,suchasitsclassnoun,butnotatunrelatedones.WeproposeReGain,atraining-freecorrectionappliedatsamplingtimethatmeasureshowmucheachfrequencybandoftheguidanceisinflatedrelativetothebasemodelandscalesthatbanddownaccordingly.ReGainneedsnorealphotos.OnStableDiffusionv1.5,ReGaincloses51-64%ofthesubject-fidelitygaptothemodelpersonalizedonrealphotos,asmeasuredbyDINO,DINOv2andCLIP-I.ItalsoimprovessubjectfidelityonSDXLandSD3.5andpreservestextalignmentonallthreebackbones.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2609\.38680

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.38680 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.38680 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.38680 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

tencentarc/gfpgan

Replicate Explore

GFPGAN is a practical face restoration model by Tencent ARC, available on Replicate. It restores old or low-quality face images with high fidelity.

Reusing Calibration Gains across Model Representations

arXiv cs.LG

该论文研究如何将一个模型表示上学到的量化校准增益复用到同一预训练模型的其他表示（如浮点与INT3量化）上，无需接收端优化即可保留大部分文本损失改善，量化了校准更新的可迁移性与接收端专属训练的剩余收益。