Textureless Deformable Object Tracking with Invisible Markers


International Conference on Computational Photography (ICCP) 2024

(Also in the TPAMI Special Issue)


Xinyuan Li1     Yu Guo2     Yubei Tu2      Yu Ji3      Yanchen Liu4      Jinwei Ye2      Changxi Zheng3,4
1Tencent USA     2George Mason University    
3LightThought LLC     4Columbia University
🎉🎉🎉 Paper, Rope and Hand are ready! Cloth will be ready soon! 🎉🎉🎉

Abstract

Tracking and reconstructing deformable objects with little texture is challenging due to lack of features. To overcome this problem, we use "invisible markers" to add features to surfaces without changing their appearance: our markers are UV fluorescent and can only be seen under UV light, thus the surface's appearance under normal lighting is unchanged. We design an imaging system for simultaneously capturing videos of deformable objects with and without markers, and leverage the markers for correspondences matching and tracking. Using marker features, we are able to obtain ground truth correspondences for textureless deformable surfaces. We collect a real-world dataset, called "DOT", with ground truth 2D and 3D correspondences, for deformable object tracking and reconstruction. Our dataset can be used for benchmarking, as well as training learning-based algorithms.

"Invisible" Markers

We use "invisble" markers to introduce features to surfaces, regardless of its original texture. The "invisible" markers are made with UV fluorescent dyes that are only visible under UV lighting. Since they are invisible under visible light, the surface appearance remains untouched in normal lighting conditions.

We use different patterns for different types objects. Here are some example patterns we use.

Time-multiplexed Imaging System

We develop a time-multiplexed multi-view imaging system to capture deformable motions with and without markers.

(Left) Conceptual illustration of our system with trigger scheme (green color: reference camera and triggered with UV light off, purple color: UV cameras and triggered with UV light on). (Center) The real physical setup of our system with a zoom-in view of the UV LED unit. (Right) Multi-view images of a hand scene.

Template-base 3D Reconstruction & Feature Warping

We use images with markers (under UV light) for accruate feature tracking and 3D reconstruction. We then use a template-based approach to map the tracked feature points back to the object's normal appearance. Our algorithmic pipeline is shown below.

Deformable Object Tracking (DOT) Dataset

We capture a large dataset for deformable object tracking (we call it DOT). Our dataset contains 4 types of objects (rope, paper, cloth and hand), with around 200 motion sequences. We provides videos with and without UV markers, 3D model, and feature correspondences in both 2D and 3D.

Reference

X. Li*, G. Yu*, Y. Tu *, Y. Ji, Y. Liu, J. Ye, and C. Zheng, "Textureless Deformable Object Tracking with Invisible Markers", IEEE International Conference on Computational Photograhpy (TPAMI Special Issue), 2024.

* indicates equal contribution.

@INPROCEEDINGS{Li2024DOT,
  author    = {Li, Xinyuan and Guo, Yu and Tu, Yubei and Ji, Yu and Liu, Yanchen and Ye, Jinwei and Zheng, Changxi},
  title     = {Textureless Deformable Object Tracking with Invisible Markers},
  journal   = {International Conference on Computational Photography (ICCP)},
  year      = {2024},
}