Color and Depth Sensing Sensor Technologies for Robotics and Machine Vision

Robust scanning technologies that offer 3D view of the world in real time are critical for situational awareness and safe operation of robotic and autonomous systems. Color and depth sensing technologies play an important role in localization and navigation in unstructured environments. Most often, sensor technology must be able to deal with factors such as objects that have low textures or objects that are dynamic, soft, and deformable. Adding intelligence to the imaging system has great potential in simplifying some of the problems. This chapter discusses the important role of scanning technologies in the development of trusted autonomous systems for robotic and machine vision with an outlook for areas that need further research and development. We start with a review of sensor technologies for specific environments including autonomous systems, mining, medical, social, aerial, and marine robotics. Special focus is on the selection of a particular scanning technology to deal with constrained or unconstrained environments. Fundamentals, advantages, and limitations of color and depth (RGB-D) technologies such as stereo vision, time of flight, structured light, and shape from shadow are discussed in detail. Strategies to deal with lighting, color constancy, occlusions, scattering, haze, and multiple reflections are discussed. This chapter also introduces the latest developments in this area by discussing the potential of emerging technologies, such as dynamic vision and focus-induced photoluminescence.

[1]  Federico Tombari,et al.  CNN-SLAM: Real-Time Dense Monocular SLAM with Learned Depth Prediction , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Oisin Mac Aodha,et al.  Unsupervised Monocular Depth Estimation with Left-Right Consistency , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Antonio Sgorbissa,et al.  Path following and obstacle avoidance for an autonomous UAV using a depth camera , 2018, Robotics Auton. Syst..

[4]  Gabriele Peters,et al.  Fast and robust keypoint detection in unstructured 3-D point clouds , 2015, 2015 12th International Conference on Informatics in Control, Automation and Robotics (ICINCO).

[5]  R. A. Hamzah,et al.  Sum of Absolute Differences algorithm in stereo correspondence problem for stereo matching in computer vision application , 2010, 2010 3rd International Conference on Computer Science and Information Technology.

[6]  Yung-Chang Chen,et al.  High-Performance SIFT Hardware Accelerator for Real-Time Image Feature Extraction , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[7]  Wendy Flores-Fuentes,et al.  Energy Center Detection in Light Scanning Sensors for Structural Health Monitoring Accuracy Enhancement , 2014, IEEE Sensors Journal.

[8]  Ian D. Reid,et al.  Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[9]  Sven Behnke,et al.  RGB-D object recognition and pose estimation based on pre-trained convolutional neural network features , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[10]  Andreas Zell,et al.  On-Board Dual-Stereo-Vision for the Navigation of an Autonomous MAV , 2014, J. Intell. Robotic Syst..

[11]  Fabio Remondino,et al.  State-of-the-Art of TOF Range-Imaging Sensors , 2013 .

[12]  Houman Borouchaki,et al.  On uniqueness in triangulation based pattern for structured light reconstruction , 2013, 2013 International Conference on 3D Imaging.

[13]  Ming C. Wu,et al.  Lidar System Architectures and Circuits , 2017, IEEE Communications Magazine.

[14]  Wendy Flores-Fuentes,et al.  Accuracy improvement in 3D laser scanner based on dynamic triangulation for autonomous navigation system , 2017, 2017 IEEE 26th International Symposium on Industrial Electronics (ISIE).

[15]  Yu-Cheng Fan,et al.  VLSI Design of a Depth Map Estimation Circuit Based on Structured Light Algorithm , 2015, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[16]  E. Charbon,et al.  SPAD-based Sensors , 2013 .

[17]  Sébastien Ourselin,et al.  Combined 2D and 3D tracking of surgical instruments for minimally invasive and robotic-assisted surgery , 2016, International Journal of Computer Assisted Radiology and Surgery.

[18]  Davide Scaramuzza,et al.  Event-Based, 6-DOF Camera Tracking from Photometric Depth Maps , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Kyoung Mu Lee,et al.  V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Map , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[20]  Dan Schonfeld,et al.  Real-time wireless multisensory smart surveillance with 3D-HEVC streams for internet-of-things (IoT) , 2017, The Journal of Supercomputing.

[21]  Svorad Stolc,et al.  Variational Fusion of Light Field and Photometric Stereo for Precise 3D Sensing within a Multi-Line Scan Framework , 2018, 2018 24th International Conference on Pattern Recognition (ICPR).

[22]  Stefano Soatto,et al.  Fast Marching Method for Generic Shape from Shading , 2005, VLSM.

[23]  Faisal Mahmood,et al.  Deep learning and conditional random fields‐based depth estimation and topographical reconstruction from conventional endoscopy , 2017, Medical Image Anal..

[24]  Luming Zhang,et al.  Integrating 3D structure into traffic scene understanding with RGB-D data , 2015, Neurocomputing.

[25]  Lei Yang,et al.  Binocular Structured Light Stereo Matching Approach for Dense Facial Disparity Map , 2011, Australasian Conference on Artificial Intelligence.

[26]  Ryad Benosman,et al.  Event-Based Structured Light for Depth Reconstruction using Frequency Tagged Light Patterns , 2018, ArXiv.

[27]  Dongbing Gu,et al.  UnDeepVO: Monocular Visual Odometry Through Unsupervised Deep Learning , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[28]  Joachim Weickert,et al.  Why Is the Census Transform Good for Robust Optic Flow Computation? , 2013, SSVM.

[29]  Raquel Urtasun,et al.  Efficient Deep Learning for Stereo Matching , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Theocharis Theocharides,et al.  A Low-Cost Real-Time Embedded Stereo Vision System for Accurate Disparity Estimation Based on Guided Image Filtering , 2016, IEEE Transactions on Computers.

[31]  Rolf-Rainer Grigat,et al.  Structure from Motion Based Approaches to 3D Reconstruction in Minimal Invasive Laparoscopy , 2012, ICIAR.

[32]  Pierre Graebling,et al.  Robust Structured Light Coding for 3D Reconstruction , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[33]  Jilin Liu,et al.  Improved real-time correlation-based FPGA stereo vision system , 2010, 2010 IEEE International Conference on Mechatronics and Automation.

[34]  Kostas Daniilidis,et al.  Realtime Time Synchronized Event-based Stereo , 2018, ECCV.

[35]  W. Brockherde,et al.  1×80 pixel SPAD-based flash LIDAR sensor with background rejection based on photon coincidence , 2017, 2017 IEEE SENSORS.

[36]  Haidi Ibrahim,et al.  Stereo matching algorithm for 3D surface reconstruction based on triangulation principle , 2016, 2016 1st International Conference on Information Technology, Information Systems and Electrical Engineering (ICITISEE).

[37]  Daijin Kim,et al.  Robust human activity recognition from depth video using spatiotemporal multi-fused features , 2017, Pattern Recognit..

[38]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[39]  Yulia Sandamirskaya,et al.  An Active Approach to Solving the Stereo Matching Problem using Event-Based Sensors , 2018, 2018 IEEE International Symposium on Circuits and Systems (ISCAS).

[40]  In-So Kweon,et al.  Adaptive Support-Weight Approach for Correspondence Search , 2006, IEEE Trans. Pattern Anal. Mach. Intell..

[41]  Krzysztof Wegner,et al.  Fast depth estimation on mobile platforms and FPGA devices , 2015, 2015 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[42]  Gian Luca Foresti,et al.  Stereo rectification of uncalibrated and heterogeneous images , 2010, Pattern Recognit. Lett..

[43]  Fernando Morgado-Dias,et al.  NanEye-An Endoscopy Sensor With 3-D Image Synchronization , 2017, IEEE Sensors Journal.

[44]  Lu Sun,et al.  Three-Dimensional Reconstruction of Macrotexture and Microtexture Morphology of Pavement Surface Using Six Light Sources–Based Photometric Stereo with Low-Rank Approximation , 2017 .

[45]  Amaury Lendasse,et al.  Underwater 3D object reconstruction with multiple views in video stream via structure from motion , 2016, OCEANS 2016 - Shanghai.

[46]  Jason Geng,et al.  Structured-light 3D surface imaging: a tutorial , 2011 .

[47]  Hyon-Gon Choo,et al.  Depth and texture imaging using time-varying color structured lights , 2011, 2011 International Conference on 3D Imaging (IC3D).

[48]  Ali Farhadi,et al.  Deep3D: Fully Automatic 2D-to-3D Video Conversion with Deep Convolutional Neural Networks , 2016, ECCV.

[49]  Weisi Lin,et al.  3DHoPD: A Fast Low-Dimensional 3-D Descriptor , 2017, IEEE Robotics and Automation Letters.

[50]  Aly A. Farag,et al.  Integrating stereo and shape from shading , 1999, Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348).

[51]  B. Alaraimi,et al.  A Randomized Prospective Study Comparing Acquisition of Laparoscopic Skills in Three-Dimensional (3D) vs. Two-Dimensional (2D) Laparoscopy , 2014, World Journal of Surgery.

[52]  Junyu Dong,et al.  Combining SLAM with muti-spectral photometric stereo for real-time dense 3D reconstruction , 2018, ArXiv.

[53]  Yu Wang,et al.  Real-time high-quality stereo vision system in FPGA , 2013, 2013 International Conference on Field-Programmable Technology (FPT).

[54]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[55]  Robert J. Woodham,et al.  Photometric method for determining surface orientation from multiple images , 1980 .

[56]  Robert J. Woodham,et al.  Photometric Stereo: A Reflectance Map Technique For Determining Surface Orientation From Image Intensity , 1979, Optics & Photonics.

[57]  Alexander Simons,et al.  Multimodal Location Based Services - Semantic 3D City Data as Virtual and Augmented Reality , 2016, LBS.

[58]  Alex Pentland,et al.  A New Sense for Depth of Field , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[59]  Frédéric Champagnat,et al.  Deep Depth from Defocus: how can defocus blur improve 3D estimation using dense neural networks? , 2018, ECCV Workshops.

[60]  Ryad Benosman,et al.  Neuromorphic Event-Based Generalized Time-Based Stereovision , 2018, Front. Neurosci..

[61]  Lars Konge,et al.  Three-dimensional versus two-dimensional vision in laparoscopy: a systematic review , 2015, Surgical Endoscopy.

[62]  Nam Ik Cho,et al.  Stereo matching using hierarchical belief propagation along ambiguity gradient , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[63]  Daniel Santana-Cedrés,et al.  Estimation of the Lens Distortion Model by Minimizing a Line Reprojection Error , 2017, IEEE Sensors Journal.

[64]  Carsten Rother,et al.  REal-time local stereo matching using guided image filtering , 2011, 2011 IEEE International Conference on Multimedia and Expo.

[65]  Shinji Tanaka,et al.  Tissue Shape Acquisition with a Hybrid Structured Light and Photometric Stereo Endoscopic System , 2015, CARE@MICCAI.

[66]  Wendy Flores-Fuentes,et al.  Improve a 3D distance measurement accuracy in stereo vision systems using optimization methods’ approach , 2017 .

[67]  H. Hirschmüller.  Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information , 2005, CVPR.

[68]  Thomas Brox,et al.  DeMoN: Depth and Motion Network for Learning Monocular Stereo , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[69]  Jianxiong Xiao,et al.  SUN RGB-D: A RGB-D scene understanding benchmark suite , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[70]  Heiko Hirschmüller,et al.  Evaluation of Cost Functions for Stereo Matching , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[71]  Eliseo Clementini,et al.  Reactive Obstacle Avoidance for Multicopter UAVs via Evaluation of Depth Maps , 2017, COSIT.

[72]  M. Westoby,et al.  ‘Structure-from-Motion’ photogrammetry: A low-cost, effective tool for geoscience applications , 2012 .

[73]  Davide Scaramuzza,et al.  Low-latency event-based visual odometry , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[74]  Shahram Izadi,et al.  StereoNet: Guided Hierarchical Refinement for Real-Time Edge-Aware Depth Prediction , 2018, ECCV.

[75]  Brice Gayet,et al.  3D visualization reduces operating time when compared to high-definition 2D in laparoscopic liver resection: a case-matched study , 2016, Surgical Endoscopy.

[76]  Liang Lu,et al.  Three-Dimensional Reconstruction from Single Image Base on Combination of CNN and Multi-Spectral Photometric Stereo , 2018, Sensors.

[77]  Michael J. Brooks,et al.  Shape and Source from Shading , 1985, IJCAI.

[78]  Masatoshi Okutomi,et al.  Structure from motion using dense CNN features with keypoint relocalization , 2018, IPSJ Transactions on Computer Vision and Applications.

[79]  Junyu Dong,et al.  Demultiplexing Colored Images for Multispectral Photometric Stereo via Deep Neural Networks , 2018, IEEE Access.

[80]  Kim Sang‐Hwan,et al.  Pixel Aperture Technique in CMOS Image Sensors for 3D Imaging , 2017 .

[81]  M. Rodrigues,et al.  Structured Light Techniques for 3D Surface Reconstruction in Robotic Tasks , 2013, CORES.

[82]  Xi Liu,et al.  A method of stereo images rectification and its application in stereo vision measurement , 2010, 2010 Second IITA International Conference on Geoscience and Remote Sensing.

[83]  Fernando Morgado Dias,et al.  Distance Measurement System for Medical Applications Based on the NanEye Stereo Camera , 2018, 2018 International Conference on Biomedical Engineering and Applications (ICBEA).

[84]  Yann LeCun,et al.  Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches , 2015, J. Mach. Learn. Res..

[85]  Sebastian Scherer,et al.  VoxNet: A 3D Convolutional Neural Network for real-time object recognition , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[86]  B. Krishna Mohan,et al.  Image Matching Using SIFT Features and Relaxation Labeling Technique—A Constraint Initializing Method for Dense Stereo Matching , 2014, IEEE Transactions on Geoscience and Remote Sensing.

[87]  Hamid Krim,et al.  3D Surface Reconstruction Using Structured Circular Light Patterns , 2010, ACIVS.

[88]  A. Verri,et al.  A compact algorithm for rectification of stereo pairs , 2000 .

[89]  Ahmed Nabil Belbachir,et al.  Improved Cooperative Stereo Matching for Dynamic Vision Sensors with Ground Truth Evaluation , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[90]  Subhasis Chaudhuri,et al.  Depth estimation and image restoration using defocused stereo pairs , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[91]  Lyndon N. Smith,et al.  In vivo measurement of skin microrelief using photometric stereo in the presence of interreflections. , 2013, Journal of the Optical Society of America. A, Optics, image science, and vision.

[92]  Jonathan Tompson,et al.  MoDeep: A Deep Learning Framework Using Motion Features for Human Pose Estimation , 2014, ACCV.

[93]  Yongjie Li,et al.  Hypergraph based feature fusion for 3-D object retrieval , 2015, Neurocomputing.

[94]  Haicheng Xu,et al.  Binocular Stereo Vision Measuring System Based on Structured Light Extraction Algorithm , 2012, 2012 International Conference on Industrial Control and Electronics Engineering.

[95]  Toby P. Breckon,et al.  A comparative review of plausible hole filling strategies in the context of scene depth image completion , 2018, Comput. Graph..

[96]  Geoffrey E. Hinton,et al.  Deep Lambertian Networks , 2012, ICML.

[97]  Wendy Flores-Fuentes,et al.  Experimental image and range scanner datasets fusion in SHM for displacement detection , 2017 .

[98]  Huei-Yung Lin,et al.  A Structured Light RGB-D Camera System for Accurate Depth Measurement , 2018, International Journal of Optics.

[99]  Nabil Aouf,et al.  Histogram of distances for local surface description , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[100]  Hong Liu,et al.  View-based 3D object retrieval with discriminative views , 2017, Neurocomputing.

[101]  Sertac Karaman,et al.  Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[102]  Beiwen Li,et al.  Structured Light Techniques and Applications , 2016 .

[103]  Simone Bianco,et al.  Evaluating the Performance of Structure from Motion Pipelines , 2018, J. Imaging.

[104]  Christophe Collewet,et al.  Optimised De Bruijn patterns for one-shot shape acquisition , 2005, Image Vis. Comput..

[105]  Cordelia Schmid,et al.  SfM-Net: Learning of Structure and Motion from Video , 2017, ArXiv.

[106]  M. Bennamoun,et al.  3-D Face Recognition Using Curvelet Local Features , 2014, IEEE Signal Processing Letters.