Geometry Matters: 3D Foundation Priors for Learning Semantic Correspondence
Summary
This paper introduces a post-training framework that leverages 3D priors from SAM3D to improve semantic correspondence in 2D foundation features, addressing issues like left-right confusion and repeated parts. The method uses instance-specific 3D reconstruction without pose annotations or spherical geometry shortcuts.
View Cached Full Text
Cached at: 05/29/26, 11:01 AM
Paper page - Geometry Matters: 3D Foundation Priors for Learning Semantic Correspondence
Source: https://huggingface.co/papers/2605.30093 🧵New work: 3D-Aware Semantic Correspondence 2D foundation features (DINO, Stable Diffusion) are powerful for semantic correspondence — but they have a blind spot: they can’t tell left from right, or distinguish repeated parts that are clearly separate in 3D. We introduce a post-training framework that brings in 3D priors to fix this.
How it works: Given an image, we use SAM3D to reconstruct object geometry and estimate pose, then refine via render-and-compare. PartField descriptors are rendered into the image plane and combined with DINO + SD features. Geodesic distances on the reconstructed shape filter unreliable matches — and the filtered correspondences supervise a lightweight adapter.
What’s different from prior work: No pose annotations. No spherical geometry shortcuts. Instance-specific 3D structure, recovered automatically.
**Results:**Improved semantic correspondence over prior post-training methods, with less manual supervision. Code + model: github.com/GenIntel/3D-SC (Coming soon)
Similar Articles
Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning
This paper proposes GASP, a framework that injects geometric priors into vision-language models via deep supervision with contrastive and depth consistency losses, achieving significant improvements on 3D spatial reasoning benchmarks without using 3D VQA data.
Geometric and Semantic Coupling for Interaction Understanding in 3D Scenes
This paper introduces Segment-Snap, a method that combines geometric and semantic cues to improve interaction understanding in 3D scenes, achieving significant gains in motion-gated AP and handle detection metrics.
One Scene, Two Depths: Probing Geometric Ambiguity in Monocular Foundation Models
Introduces MultiDepth-3k, a benchmark to evaluate depth-layer preferences in monocular depth foundation models, and shows Laplacian Visual Prompting can alter reported depth layers, suggesting complementary geometric hypotheses exist across models.
Beyond Spatio-Temporal Priors: A Generalizable Approach for Dense Correspondence Matching
论文提出 FreeMatching,一个结合生成式与语义基础表征、通过异构监督和教师引导迭代精化的通用密集对应匹配框架,在图像编辑与参考引导生成任务上显著提升对应质量,并可作为身份保持的量化评估指标。
See Before You Code: Learning Visual Priors for Spatially Aware Educational Animation Generation
This paper introduces OmniManim, a render-feedback-aware framework for generating educational animations from natural language descriptions using large language models. It addresses visual defects like element overlap and misalignment by incorporating explicit visual planning, post-render diagnostics, and localized repair, demonstrating improved render quality on newly constructed datasets.