Probabilistic Warp Consistency for Weakly-Supervised Semantic Correspondences

Prune Truong, Martin Danelljan, Fisher Yu, Luc Van Gool
CVPR 2022

Probabilistic Warp Consistency for Weakly-Supervised Semantic Correspondences

Abstract

We propose Probabilistic Warp Consistency, a weakly-supervised learning objective for semantic matching. Our approach directly supervises the dense matching scores predicted by the network, encoded as a conditional probability distribution. We first construct an image triplet by applying a known warp to one of the images in a pair depicting different instances of the same object class. Our probabilistic learning objectives are then derived using the constraints arising from the resulting image triplet. We further account for occlusion and background clutter present in real image pairs by extending our probabilistic output space with a learnable unmatched state. To supervise it, we design an objective between image pairs depicting different object classes. We validate our method by applying it to four recent semantic matching architectures. Our weakly-supervised approach sets a new state-of-the-art on four challenging semantic matching benchmarks. Lastly, we demonstrate that our objective also brings substantial improvements in the strongly-supervised regime, when combined with keypoint annotations. We validate our method by applying it to four recent semantic matching architectures. Our weakly-supervised approach sets a new state-of-the-art on four challenging semantic matching benchmarks. Lastly, we demonstrate that our objective also brings substantial improvements in the strongly-supervised regime, when combined with keypoint annotations.

Citation

@inproceedings{probwarpc,
  author    = {Truong, Prune and Danelljan, Martin and Yu, Fisher and Van Gool, Luc},
  title     = {Probabilistic Warp Consistency for Weakly-Supervised Semantic Correspondences},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2022}
}

Related


Warp Consistency for Unsupervised Learning of Dense Correspondences

Warp Consistency for Unsupervised Learning of Dense Correspondences

ICCV 2021 Oral We propose Warp Consistency, an unsupervised learning objective for dense correspondence regression.


Robust Object Detection via Instance-Level Temporal Cycle Confusion

Robust Object Detection via Instance-Level Temporal Cycle Confusion

ICCV 2021 We study the effectiveness of auxiliary self-supervised tasks to improve the out-of-distribution generalization of object detectors.


Hierarchical Discrete Distribution Decomposition for Match Density Estimation

Hierarchical Discrete Distribution Decomposition for Match Density Estimation

CVPR 2019 We propose Hierarchical Discrete Distribution Decomposition (HD^3), a framework suitable for learning probabilistic pixel correspondences in both optical flow and stereo matching.