CtrlVTON: Controllable Virtual Try-On via Visual-Instance-Prompt Segmentation

Hugging Face Daily Papers Papers
virtual-try-on segmentation image-editing controllable garment computer-vision

Summary

This paper introduces VIP-SAM for instance-level garment segmentation and CtrlVTON, a controllable virtual try-on framework that treats try-on as an image editing problem, allowing precise control over garment layout, style, and placement. Both methods achieve state-of-the-art results on their respective tasks.

Virtual try-on (VTO) has made significant progress in realistically transferring garments onto a target person. Yet most systems give the user little control over how a garment should be worn -- its size (loose or fitted), style (e.g., tucked in or untucked, open or closed), and spatial placement on the body. We address this gap with two complementary contributions. First, we define and solve Visual-Instance-Prompt Segmentation via VIP-SAM: given a flatlay image of a garment, segment that specific instance in a photograph of a person wearing it. This is an instance-level task, distinct from the typically studied category-level segmentation. Second, we introduce CtrlVTON, a controllable VTO framework that recasts try-on as an image editing problem and adds segmentation masks as pixel-level control over garment layout, including style, size, and spatial placement on the body. VIP-SAM and CtrlVTON each achieve state-of-the-art results on their respective tasks. In particular, CtrlVTON generates images that follow user-provided layouts far more faithfully than the strongest proprietary editing systems while matching them on garment fidelity.
Original Article
View Cached Full Text

Cached at: 07/14/26, 08:13 AM

Paper page - CtrlVTON: Controllable Virtual Try-On via Visual-Instance-Prompt Segmentation

Source: https://huggingface.co/papers/2607.09362

Abstract

Virtualtry-on(VTO)hasmadesignificantprogressinrealisticallytransferringgarmentsontoatargetperson.Yetmostsystemsgivetheuserlittlecontroloverhowagarmentshouldbeworn--itssize(looseorfitted),style(e.g.,tuckedinoruntucked,openorclosed),andspatialplacementonthebody.Weaddressthisgapwithtwocomplementarycontributions.First,wedefineandsolveVisual-Instance-PromptSegmentationviaVIP-SAM:givenaflatlayimageofagarment,segmentthatspecificinstanceinaphotographofapersonwearingit.Thisisaninstance-leveltask,distinctfromthetypicallystudiedcategory-levelsegmentation.Second,weintroduceCtrlVTON,acontrollableVTOframeworkthatrecaststry-onasanimageeditingproblemandaddssegmentationmasksaspixel-levelcontrolovergarmentlayout,includingstyle,size,andspatialplacementonthebody.VIP-SAMandCtrlVTONeachachievestate-of-the-artresultsontheirrespectivetasks.Inparticular,CtrlVTONgeneratesimagesthatfollowuser-providedlayoutsfarmorefaithfullythanthestrongestproprietaryeditingsystemswhilematchingthemongarmentfidelity.

View arXiv pageView PDFAdd to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.09362 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.09362 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.09362 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Text-Vision Co-Instructed Image Editing

Hugging Face Daily Papers

A new framework called TV-Edit combines textual instructions and visual prompts for precise image editing, along with a benchmark TV-Edit-Bench for evaluation. The method achieves better spatial control and semantic faithfulness than existing approaches.