Semantic correspondence is essential for handling diverse in-the-wild images lacking explicit correspondence annotations. While recent 2D foundation models offer powerful features, adapting them for unsupervised learning via nearest-neighbor pseudo-labels has key limitations: it operates locally, ignoring structural relationships, and consequently its reliance on 2D appearance fails to resolve geometric ambiguities arising from symmetries or repetitive features.
In this work, we address this by reformulating pseudo-label generation as a Fused Gromov-Wasserstein (FGW) problem, which jointly optimizes inter-feature similarity and intra-structural consistency. Our framework, Shape-of-You (SoY), leverages a 3D foundation model to define this intra-structure in the geometric space, resolving the abovementioned ambiguity. However, since FGW is a computationally prohibitive quadratic problem, we approximate it through anchor-based linearization. The resulting probabilistic transport plan provides a structurally consistent but noisy supervisory signal. Thus, we introduce a soft-target loss dynamically blending guidance from this plan with network predictions to build a learning framework robust to this noise.
SoY achieves state-of-the-art performance on SPair-71k and AP-10k datasets, establishing a new benchmark in semantic correspondence without explicit geometric annotations.
Overview of our pseudo-label generation pipeline. High-confidence anchors from an initial semantic match enable a tractable linear approximation of the otherwise quadratic Gromov-Wasserstein geometric cost. Fusing this geometric cost with the semantic cost yields a final cost matrix, solved via unbalanced optimal transport to produce a transport plan that serves as a structurally consistent pseudo-label.
Shape-of-You achieves the best average PCK@0.1 (67.9%), outperforming the previous state of the art (DINOv2 + SD, 63.5%) by +4.4%p, with consistent gains across nearly all 18 categories. Best and second-best per column are color-coded.
| Method |  |
 |
 |
avg | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ASIC | 57.9 | 25.2 | 68.1 | 24.7 | 35.4 | 28.4 | 30.9 | 54.8 | 21.6 | 45.0 | 47.2 | 39.9 | 26.2 | 48.8 | 14.5 | 24.5 | 49.0 | 24.6 | 36.9 |
| DINOv2 | 72.7 | 62.4 | 85.2 | 41.4 | 40.3 | 52.5 | 51.5 | 71.3 | 36.1 | 67.2 | 65.0 | 67.6 | 61.1 | 68.5 | 30.6 | 61.9 | 54.3 | 24.3 | 55.7 |
| DIFT | 63.5 | 54.5 | 80.8 | 34.5 | 46.2 | 52.7 | 48.3 | 77.7 | 39.0 | 76.0 | 54.9 | 61.3 | 53.3 | 46.0 | 57.8 | 57.1 | 71.1 | 63.4 | 57.7 |
| DistillDIFT† | 70.3 | 55.4 | 85.9 | 36.8 | 51.9 | 51.7 | 50.5 | 78.0 | 40.5 | 73.8 | 65.3 | 65.2 | 54.5 | 66.0 | 47.8 | 59.4 | 55.6 | 54.4 | 59.8 |
| DINOv2 + SD† | 72.9 | 63.4 | 86.4 | 40.5 | 52.6 | 55.4 | 53.3 | 78.4 | 45.2 | 77.1 | 64.7 | 69.4 | 62.9 | 68.5 | 56.8 | 67.0 | 65.9 | 51.8 | 63.5 |
| Shape-of-You (Ours) | 73.5 | 66.7 | 89.9 | 40.8 | 58.0 | 55.4 | 51.1 | 84.8 | 52.1 | 81.0 | 71.0 | 75.3 | 64.1 | 71.8 | 62.1 | 70.2 | 70.2 | 64.4 | 67.9 |
† denotes results re-evaluated under identical standard evaluation settings.
Comparison on challenging cases (viewpoint change, occlusion, pose variation). Shape-of-You produces accurate matches where 2D-only baselines confuse symmetric or repetitive parts.
Shape-of-You: Fused Gromov-Wasserstein Optimal Transport for Semantic Correspondence in-the-Wild
@inproceedings{im2026shapeofyou,
title={Shape-of-You: Fused Gromov-Wasserstein Optimal Transport for Semantic Correspondence in-the-Wild},
author={Im, Jiin and Liu, Sisung and Hong, Je Hyeong},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2026}
}