Tracking and reconstructing deformable objects with little texture is challenging due to lack of features. To overcome this problem, we use "invisible markers" to add features to surfaces without changing their appearance: our markers are UV fluorescent and can only be seen under UV light, thus the surface's appearance under normal lighting is unchanged. We design an imaging system for simultaneously capturing videos of deformable objects with and without markers, and leverage the markers for correspondences matching and tracking. Using marker features, we are able to obtain ground truth correspondences for textureless deformable surfaces. We collect a real-world dataset, called "DOT", with ground truth 2D and 3D correspondences, for deformable object tracking and reconstruction. Our dataset can be used for benchmarking, as well as training learning-based algorithms.
We use "invisble" markers to introduce features to surfaces, regardless of its original texture. The "invisible" markers are made with UV fluorescent dyes that are only visible under UV lighting. Since they are invisible under visible light, the surface appearance remains untouched in normal lighting conditions.
We use different patterns for different types objects. Here are some example patterns we use.
We develop a time-multiplexed multi-view imaging system to capture deformable motions with and without markers.
(Left) Conceptual illustration of our system with trigger scheme (green color: reference camera and triggered with UV light off, purple color: UV cameras and triggered with UV light on). (Center) The real physical setup of our system with a zoom-in view of the UV LED unit. (Right) Multi-view images of a hand scene.
We use images with markers (under UV light) for accruate feature tracking and 3D reconstruction. We then use a template-based approach to map the tracked feature points back to the object's normal appearance. Our algorithmic pipeline is shown below.
We capture a large dataset for deformable object tracking (we call it DOT). Our dataset contains 4 types of objects (rope, paper, cloth and hand), with around 200 motion sequences. We provides videos with and without UV markers, 3D model, and feature correspondences in both 2D and 3D.
X. Li*, G. Yu*, Y. Tu *, Y. Ji, Y. Liu, J. Ye, and C. Zheng, "Textureless Deformable Object Tracking with Invisible Markers", IEEE International Conference on Computational Photograhpy (TPAMI Special Issue), 2024.
* indicates equal contribution.
@INPROCEEDINGS{Li2024DOT,
author = {Li, Xinyuan and Guo, Yu and Tu, Yubei and Ji, Yu and Liu, Yanchen and Ye, Jinwei and Zheng, Changxi},
title = {Textureless Deformable Object Tracking with Invisible Markers},
journal = {International Conference on Computational Photography (ICCP)},
year = {2024},
}