Geometry Matters: 3D Foundation Priors for Learning Semantic Correspondence

Hugging Face Daily Papers Papers

Summary

This paper introduces a post-training framework that leverages 3D priors from SAM3D to improve semantic correspondence in 2D foundation features, addressing issues like left-right confusion and repeated parts. The method uses instance-specific 3D reconstruction without pose annotations or spherical geometry shortcuts.

Foundation features from self-supervised vision models and text-to-image diffusion models have proven effective for semantic correspondence estimation. However, because these features are learned primarily from 2D image objectives, they lack explicit 3D awareness and often confuse symmetric object sides, repeated parts, and visually similar structures that are distinct in 3D. We introduce a 3D-aware post-training framework that goes beyond available 2D foundation features by incorporating priors from 3D foundation models. Given an image, our method uses SAM3D to estimate object geometry and pose, and refines the pose through render-and-compare optimization. Subsequently, we render PartField descriptors from the reconstructed geometry into the image plane based on the estimated object pose. The resulting geometry-aware feature maps complement DINO and Stable Diffusion features, while geodesic distances on the reconstructed shapes enable reliable filtering of candidate correspondences. We use the filtered matches as supervision to train a lightweight adapter on top of DINO and Stable Diffusion for semantic correspondence. In contrast to prior post-training approaches that require pose annotations and rely on coarse spherical geometry, our method automatically obtains instance-specific 3D structure and uses it to guide correspondence learning. Experiments show that our approach improves semantic correspondence over the prior methods while reducing manual geometric supervision. Code and model can be found at https:/github.com/GenIntel/3D-SC.
Original Article
View Cached Full Text

Cached at: 05/29/26, 11:01 AM

Paper page - Geometry Matters: 3D Foundation Priors for Learning Semantic Correspondence

Source: https://huggingface.co/papers/2605.30093 🧵New work: 3D-Aware Semantic Correspondence 2D foundation features (DINO, Stable Diffusion) are powerful for semantic correspondence — but they have a blind spot: they can’t tell left from right, or distinguish repeated parts that are clearly separate in 3D. We introduce a post-training framework that brings in 3D priors to fix this.

How it works: Given an image, we use SAM3D to reconstruct object geometry and estimate pose, then refine via render-and-compare. PartField descriptors are rendered into the image plane and combined with DINO + SD features. Geodesic distances on the reconstructed shape filter unreliable matches — and the filtered correspondences supervise a lightweight adapter.

What’s different from prior work: No pose annotations. No spherical geometry shortcuts. Instance-specific 3D structure, recovered automatically.

**Results:**Improved semantic correspondence over prior post-training methods, with less manual supervision. Code + model: github.com/GenIntel/3D-SC (Coming soon)

Similar Articles

See Before You Code: Learning Visual Priors for Spatially Aware Educational Animation Generation

arXiv cs.AI

This paper introduces OmniManim, a render-feedback-aware framework for generating educational animations from natural language descriptions using large language models. It addresses visual defects like element overlap and misalignment by incorporating explicit visual planning, post-render diagnostics, and localized repair, demonstrating improved render quality on newly constructed datasets.

MARCO: Navigating the Unseen Space of Semantic Correspondence

Hugging Face Daily Papers

MARCO introduces a compact, fast model for semantic correspondence that achieves state-of-the-art accuracy and generalization to unseen keypoints using a coarse-to-fine objective and self-distillation framework with DINOv2.