Human Pose as Calibration Pattern: 3D Human Pose Estimation with Multiple Unsynchronized and Uncalibrated Cameras

This paper proposes a novel algorithm of estimating 3D human pose from multi-view videos captured by unsynchronized and uncalibrated cameras. In a such configuration, the conventional vision-based approaches utilize detected 2D features of common 3D points for synchronization and camera pose estimation, however, they sometimes suffer from difficulties of feature correspondences in case of wide baselines. For such cases, the proposed method focuses on that the projections of human joints can be associated each other robustly even in wide baseline videos and utilizes them as the common reference points. To utilize the projections of joint as the corresponding points, they should be detected in the images, however, these 2D joint sometimes include detection errors which make the estimation unstable. For dealing with such errors, the proposed method introduces two ideas. The first idea is to relax the reprojection errors for avoiding optimizing to noised observations. The second idea is to introduce an geometric constraint on the prior knowledge that the reference points consists of human joints. We demonstrate the performance of the proposed algorithm of synchronization and pose estimation with qualitative and quantitative evaluations using synthesized and real data.

[1]  Michael Werman,et al.  Camera Calibration from Dynamic Silhouettes Using Motion Barcodes , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Hans-Peter Seidel,et al.  Markerless Motion Capture with unsynchronized moving cameras , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[3]  Edmond Boyer On Using Silhouettes for Camera Calibration , 2006, ACCV.

[4]  V. Lepetit,et al.  EPnP: An Accurate O(n) Solution to the PnP Problem , 2009, International Journal of Computer Vision.

[5]  Matthijs C. Dorst Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[6]  Christian Szegedy,et al.  DeepPose: Human Pose Estimation via Deep Neural Networks , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Marc Pollefeys,et al.  Camera Network Calibration and Synchronization from Silhouettes in Archived Video , 2010, International Journal of Computer Vision.

[9]  Varun Ramakrishna,et al.  Convolutional Pose Machines , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Peter V. Gehler,et al.  Poselet Conditioned Pictorial Structures , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  PollefeysMarc,et al.  Camera Network Calibration and Synchronization from Silhouettes in Archived Video , 2010 .

[13]  Sophia Blau,et al.  Visual Motion Of Curves And Surfaces , 2016 .

[14]  Mauro Barbieri,et al.  Synchronization of multi-camera video recordings based on audio , 2007, ACM Multimedia.

[15]  Andrew W. Fitzgibbon,et al.  On the Two-View Geometry of Unsynchronized Cameras , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Yuandong Tian,et al.  Exploring the Spatial Hierarchy of Mixture Models for Human Pose Estimation , 2012, ECCV.

[17]  Hideo Saito,et al.  Reconstructing the 3D Trajectory of a Ball with Unsynchronized Cameras , 2015, Int. J. Comput. Sci. Sport.

[18]  Marc Pollefeys,et al.  Joint Camera Pose Estimation and 3D Human Pose Estimation in a Multi-camera Setup , 2014, ACCV.

[19]  Silvio Savarese,et al.  Articulated part-based model for joint object detection and pose estimation , 2011, 2011 International Conference on Computer Vision.

[20]  Hongbin Zha,et al.  Camera Calibration from Periodic Motion of a Pedestrian , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Bernt Schiele,et al.  Pictorial structures revisited: People detection and articulated pose estimation , 2009, CVPR.

[22]  Varun Ramakrishna,et al.  Pose Machines: Articulated Pose Estimation via Inference Machines , 2014, ECCV.

[23]  Sanja Fidler,et al.  Sports Field Localization via Deep Structured Models , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Maher Moakher,et al.  To appear in: SIAM J. MATRIX ANAL. APPL. MEANS AND AVERAGING IN THE GROUP OF ROTATIONS∗ , 2002 .