A Methodology for Multi-Camera Surface-Shape Estimation of Deformable Unknown Objects

A novel methodology is proposed herein to estimate the three-dimensional (3D) surface shape of unknown, markerless deforming objects through a modular multi-camera vision system. The methodology is a generalized formal approach to shape estimation for a priori unknown objects. Accurate shape estimation is accomplished through a robust, adaptive particle filtering process. The estimation process yields a set of surface meshes representing the expected deformation of the target object. The methodology is based on the use of a multi-camera system, with a variable number of cameras, and range of object motions. The numerous simulations and experiments presented herein demonstrate the proposed methodology’s ability to accurately estimate the surface deformation of unknown objects, as well as its robustness to object loss under self-occlusion, and varying motion dynamics.

[1]  Albert Dipanda,et al.  Direct 3D Information Determination in an Uncalibrated Stereovision System by Using Evolutionary Algorithms , 2011, Int. J. Comput. Vis. Image Process..

[2]  Alessio Del Bue,et al.  Garment-based motion capture (GaMoCap): high-density capture of human shape in motion , 2015, Machine Vision and Applications.

[3]  Hans-Peter Seidel,et al.  Markerless Motion Capture with unsynchronized moving cameras , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  Brendan McCane,et al.  Better than SIFT? , 2015, Machine Vision and Applications.

[5]  Peter Leskovský,et al.  Constant-time monocular object detection using scene geometry , 2017, Pattern Analysis and Applications.

[6]  Bogdan Kwolek,et al.  Real-Time Multi-view Human Motion Tracking Using Particle Swarm Optimization with Resampling , 2012, AMDO.

[7]  Tomás Pajdla,et al.  Multi-view reconstruction preserving weakly-supported surfaces , 2011, CVPR 2011.

[8]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[9]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[10]  Xiaoqin Zhang,et al.  A swarm intelligence based searching strategy for articulated 3D human body tracking , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[11]  Max Mignotte,et al.  A biologically inspired framework for contour detection , 2017, Pattern Analysis and Applications.

[12]  Dmitry Chetverikov,et al.  Real-Time 4D Reconstruction of Human Motion , 2012, AMDO.

[13]  Lingling Chen,et al.  Statistical background model-based target detection , 2015, Pattern Analysis and Applications.

[14]  Dan Xia,et al.  A novel 3D convex surface reconstruction method based on visual hull , 2011, International Symposium on Multispectral Image Processing and Pattern Recognition.

[15]  Lionel Moisan,et al.  A Probabilistic Criterion to Detect Rigid Point Matches Between Two Images and Estimate the Fundamental Matrix , 2004, International Journal of Computer Vision.

[16]  Joé Lallemand,et al.  Human Pose Estimation in Stereo Images , 2014, AMDO.

[17]  Kok Kiong Tan,et al.  Vision-based approach towards lane line detection and vehicle localization , 2015, Machine Vision and Applications.

[18]  A. Laurentini,et al.  The Visual Hull Concept for Silhouette-Based Image Understanding , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  Lena Gorelick,et al.  GrabCut in One Cut , 2013, 2013 IEEE International Conference on Computer Vision.

[20]  Frédo Durand,et al.  Noise-optimal capture for high dynamic range photography , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[21]  Shiv Ram Dubey,et al.  Human Activity Recognition Using Gait Pattern , 2013, Int. J. Comput. Vis. Image Process..

[22]  Beno Benhabib,et al.  Multi-camera active surveillance of an articulated human form - An implementation strategy , 2011, Comput. Vis. Image Underst..

[23]  Daniel Cremers,et al.  Multiview Stereo and Silhouette Consistency via Convex Functionals over Convex Domains , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Wilfried Philips,et al.  High performance multi-camera tracking using shapes-from-silhouettes and occlusion removal , 2015, ICDSC.

[25]  Truong Q. Nguyen,et al.  Realistic surface geometry reconstruction using a hand-held RGB-D camera , 2016, Machine Vision and Applications.

[26]  Beno Benhabib,et al.  Simulation-based sensing-system configuration for dynamic dispatching , 2001, 2001 IEEE International Conference on Systems, Man and Cybernetics. e-Systems and e-Man for Cybernetics in Cyberspace (Cat.No.01CH37236).

[27]  J.C. Garcia,et al.  Comparing a Kalman Filter and a Particle Filter in a Multiple Objects Tracking Application , 2007, 2007 IEEE International Symposium on Intelligent Signal Processing.

[28]  Jérémie Allard,et al.  Multicamera Real-Time 3D Modeling for Telepresence and Remote Collaboration , 2010, Int. J. Digit. Multim. Broadcast..

[29]  De Xu,et al.  Research and latest development of Ping-Pong robot player , 2008, 2008 7th World Congress on Intelligent Control and Automation.

[30]  François Blais.  Review of 20 years of range sensor development , 2004, J. Electronic Imaging.

[31]  Jean-Michel Morel,et al.  ASIFT: An Algorithm for Fully Affine Invariant Comparison , 2011, Image Process. Line.

[32]  Yücel Yemez,et al.  A volumetric fusion technique for surface reconstruction from silhouettes and range data , 2007, Comput. Vis. Image Underst..

[33]  Philippos Mordohai,et al.  Consistent 3D Background Model Estimation from Multi-viewpoint Videos , 2015, 2015 International Conference on 3D Vision.

[34]  Maarten Speekenbrink,et al.  A tutorial on particle filters , 2016 .

[35]  Beno Benhabib,et al.  A Multi-Camera Active-Vision System for Deformable-Object-Motion Capture , 2014, J. Intell. Robotic Syst..

[36]  Francis Schmitt,et al.  Silhouette and stereo fusion for 3D object modeling , 2003, Fourth International Conference on 3-D Digital Imaging and Modeling, 2003. 3DIM 2003. Proceedings..

[37]  Stefano Soatto,et al.  Multi-View Stereo Reconstruction of Dense Shape and Complex Appearance , 2005, International Journal of Computer Vision.

[38]  Sherin Jabbar,et al.  Efficient 3D visual hull reconstruction based on marching cube algorithm , 2015, 2015 International Conference on Innovations in Information, Embedded and Communication Systems (ICIIECS).

[39]  Brian Y. Lattimer,et al.  Real-Time Classification of Water Spray and Leaks for Robotic Firefighting , 2015, Int. J. Comput. Vis. Image Process..

[40]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[41]  Yasuhiro Oue,et al.  A flexible 3D modeling system based on combining shape-from-silhouette with light-sectioning algorithm , 2005, Fifth International Conference on 3-D Digital Imaging and Modeling (3DIM'05).

[42]  Chao Liu,et al.  Efficient 3D Tracking for Motion Compensation in Beating Heart Surgery , 2008, MICCAI.

[43]  Jean Ponce,et al.  Accurate, Dense, and Robust Multiview Stereopsis , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[44]  C Tomasi,et al.  Shape and motion from image streams: a factorization method. , 1992, Proceedings of the National Academy of Sciences of the United States of America.

[45]  Steven M. Seitz,et al.  Photo tourism: exploring photo collections in 3D , 2006, ACM Trans. Graph..

[46]  Emiliano Gambaretto,et al.  Markerless Motion Capture through Visual Hull, Articulated ICP and Subject Specific Model Generation , 2010, International Journal of Computer Vision.

[47]  S. Y. Chen,et al.  Kalman Filter for Robot Vision: A Survey , 2012, IEEE Transactions on Industrial Electronics.

[48]  Francisco José Madrid-Cuevas,et al.  Stereo Pictorial Structure for 2D articulated human pose estimation , 2015, Machine Vision and Applications.

[49]  David Schacter,et al.  Multi-Camera Active-vision System Reconfiguration for Deformable Object Motion Capture , 2014 .

[50]  Roger Mohr,et al.  Optimal camera placement for accurate reconstruction , 2002, Pattern Recognit..

[51]  Beno Benhabib,et al.  TIME-VARYING-GEOMETRY OBJECT SURVEILLANCE USING A MULTI-CAMERA ACTIVE-VISION SYSTEM , 2008 .

[52]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[53]  Long Quan,et al.  A quasi-dense approach to surface reconstruction from uncalibrated images , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[54]  Cihan Altuntas,et al.  Pair-wise automatic registration of three-dimensional laser scanning data from historical building by created two-dimensional images , 2014 .

[55]  Beno Benhabib,et al.  Coordinated dispatching of proximity sensors for the surveillance of manoeuvring targets , 2003 .

[56]  Mohammed Bennamoun,et al.  A Gaussian Process Guided Particle Filter for Tracking 3D Human Pose in Video , 2013, IEEE Transactions on Image Processing.

[57]  Diego Borro,et al.  Real time non-rigid 3D surface tracking using particle filter , 2015, Comput. Vis. Image Underst..

[58]  Andrew Zisserman,et al.  Automatic and Efficient Human Pose Estimation for Sign Language Videos , 2013, International Journal of Computer Vision.

[59]  Philippe Poignet,et al.  Towards robust 3D visual tracking for motion compensation in beating heart surgery , 2011, Medical Image Anal..

[60]  M.D. Naish,et al.  Active-vision-based multisensor surveillance - an implementation , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[61]  Hans-Peter Seidel,et al.  Performance capture from sparse multi-view video , 2008, ACM Trans. Graph..

[62]  ZhiWu Li,et al.  A double circle structure descriptor and Hough voting matching for real-time object detection , 2016, Pattern Analysis and Applications.

[63]  Shengyong Chen,et al.  Active vision in robotic systems: A survey of recent developments , 2011, Int. J. Robotics Res..

[64]  Thomas Popham,et al.  Tracking 3D Surfaces Using Multiple Cameras: A Probabilistic Approach , 2010 .

[65]  Jean Ponce,et al.  Dense 3D motion capture for human faces , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[66]  Josiane Zerubia,et al.  A Marked Point Process Model Including Strong Prior Shape Information Applied to Multiple Object Extraction From Images , 2011, Int. J. Comput. Vis. Image Process..

[67]  C. Cobelli,et al.  A Markerless Motion Capture System to Study Musculoskeletal Biomechanics: Visual Hull and Simulated Annealing Approach , 2006, Annals of Biomedical Engineering.

[68]  Matthew A. Brown,et al.  Automatic Panoramic Image Stitching using Invariant Features , 2007, International Journal of Computer Vision.

[69]  Greg Welch,et al.  Welch & Bishop , An Introduction to the Kalman Filter 2 1 The Discrete Kalman Filter In 1960 , 1994 .

[70]  Jean Ponce,et al.  Carved Visual Hulls for Image-Based Modeling , 2006, International Journal of Computer Vision.

[71]  Derek Bradley,et al.  High resolution passive facial performance capture , 2010, ACM Trans. Graph..

[72]  Marc Pollefeys,et al.  Multi-object shape estimation and tracking from silhouette cues , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[73]  Pieter Abbeel,et al.  Tracking deformable objects with point clouds , 2013, 2013 IEEE International Conference on Robotics and Automation.

[74]  Mario Castelán,et al.  A photometric sampling method for facial shape recovery , 2016, Machine Vision and Applications.

[75]  Andrea Vedaldi,et al.  Vlfeat: an open and portable library of computer vision algorithms , 2010, ACM Multimedia.

[76]  Branko Ristic,et al.  Beyond the Kalman Filter: Particle Filters for Tracking Applications , 2004 .

[77]  Michael Goesele,et al.  Multi-View Stereo Revisited , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[78]  Derek Bradley,et al.  Markerless garment capture , 2008, SIGGRAPH 2008.

[79]  Adrian Hilton,et al.  An Empirical Study of Non-Rigid Surface Feature Matching of Human from 3D Video , 2010, J. Virtual Real. Broadcast..

[80]  Xiaojun Wu,et al.  Real-time 3D shape reconstruction, dynamic 3D mesh deformation, and high fidelity visualization for 3D video , 2004, Comput. Vis. Image Underst..

[81]  Maarten Vergauwen,et al.  Structure and motion from image sequences , 2001 .

[82]  Pawel Strumillo,et al.  Pedestrian tracking in video sequences: A particle filtering approach , 2015, 2015 Federated Conference on Computer Science and Information Systems (FedCSIS).

[83]  Russell C. Eberhart,et al.  A new optimizer using particle swarm theory , 1995, MHS'95. Proceedings of the Sixth International Symposium on Micro Machine and Human Science.

[84]  Beno Benhabib,et al.  Active-Vision for the Autonomous Surveillance of Dynamic, Multi-Object Environments , 2009, J. Intell. Robotic Syst..

[85]  H. Frank,et al.  Accurate prediction of interception positions for catching thrown objects in production systems , 2008, 2008 6th IEEE International Conference on Industrial Informatics.