Instant 3D Object Tracking with Applications in Augmented Reality

Tracking object poses in 3D is a crucial building block for Augmented Reality applications. We propose an instant motion tracking system that tracks an object's pose in space (represented by its 3D bounding box) in real-time on mobile devices. Our system does not require any prior sensory calibration or initialization to function. We employ a deep neural network to detect objects and estimate their initial 3D pose. Then the estimated pose is tracked using a robust planar tracker. Our tracker is capable of performing relative-scale 9-DoF tracking in real-time on mobile devices. By combining use of CPU and GPU efficiently, we achieve 26-FPS+ performance on mobile devices.

[1]  Adrian David Cheok,et al.  Augmented Reality Camera Tracking with Homographies , 2002, IEEE Computer Graphics and Applications.

[2]  V. Lepetit,et al.  EPnP: An Accurate O(n) Solution to the PnP Problem , 2009, International Journal of Computer Vision.

[3]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[4]  K. Madhava Krishna,et al.  Beyond Pixels: Leveraging Geometry and Shape Cues for Online Multi-Object Tracking , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[5]  Alex Fridman,et al.  Object as Distribution , 2019, ArXiv.

[6]  Yifan Wu,et al.  Planar Object Tracking in the Wild: A Benchmark , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[7]  Selim Benhimane,et al.  Real-time image-based tracking of planes using efficient second-order minimization , 2004, 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566).

[8]  Gerhard Reitmayr,et al.  Homography-based planar mapping and tracking for mobile phones , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[9]  Vladlen Koltun,et al.  Tracking Objects as Points , 2020, ECCV.

[10]  Shaojie Shen,et al.  Stereo Vision-based Semantic 3D Object and Ego-motion Tracking for Autonomous Driving , 2018, ECCV.

[11]  Tingbo Hou,et al.  MobilePose: Real-Time Pose Estimation for Unseen Objects with Weak Shape Supervision , 2020, ArXiv.

[12]  Bastian Leibe,et al.  Combined image- and world-space tracking in traffic scenes , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[13]  Tingbo Hou,et al.  Instant Motion Tracking and Its Applications to Augmented Reality , 2019, ArXiv.

[14]  Trevor Darrell,et al.  Joint Monocular 3D Vehicle Detection and Tracking , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[15]  Simon Baker,et al.  Lucas-Kanade 20 Years On: A Unifying Framework , 2004, International Journal of Computer Vision.