An integrated system for automated 3D visualization and monitoring of vehicles

Recent technological advances in computer vision and the advent of commercial RGB-D sensors have certainly boosted 3D modeling applications. This work presents an integrated system that enables the digitization of big objects, like vehicles, with low-cost RGB-D sensors. The implemented system can be used for visualization and monitoring of vehicles in large fleets that currently require a time-consuming manual inspection process. The main objective is to achieve an efficient consolidation of multiple views of a vehicle inside a moving frame, to acquire color and depth data and generate its 3D representation. The proposed integrated system denoises the acquired depth maps, aligns the produced point clouds captured in different time instances, and builds the 3D-reconstructed mesh. Finally, we apply a texture mapping algorithm to acquire realistic texture details and remove any visible seams. We evaluate all modules of the implemented system by performing several experiments with scanned vehicles.

[1]  Dimitrios Tzovaras,et al.  A Deep Learning framework for simulation and defect prediction applied in microelectronics , 2020, Simul. Model. Pract. Theory.

[2]  Slobodan Ilic,et al.  PPFNet: Global Context Aware Local Features for Robust 3D Point Matching , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[3]  Dimitrios Tzovaras,et al.  Automated Mechanical Multi-sensorial Scanning , 2019, ICVS.

[4]  S. Nebiker,et al.  VISUALISATION OF COMPLEX 3D CITY MODELS ON MOBILE WEBBROWSERS USING CLOUD-BASED IMAGE PROVISIONING , 2015 .

[5]  Dimitrios Tzovaras,et al.  Realistic Texture Reconstruction Incorporating Spectrophotometric Color Correction , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[6]  Yasuhiro Aoki,et al.  PointNetLK: Robust & Efficient Point Cloud Registration Using PointNet , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Leonidas J. Guibas,et al.  One Point Isometric Matching with the Heat Kernel , 2010, Comput. Graph. Forum.

[8]  Victor S. Lempitsky,et al.  Seamless Mosaicing of Image-Based Texture Maps , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[9]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Matthias Nießner,et al.  3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Dimitrios Tzovaras,et al.  Color-Guided Adaptive Support Weights for Active Stereo Systems , 2019, ICVS.

[12]  Gary R. Bradski,et al.  Monte Carlo Pose Estimation with Quaternion Kernels and the Bingham Distribution , 2011, Robotics: Science and Systems.

[13]  Marc Levoy,et al.  Efficient variants of the ICP algorithm , 2001, Proceedings Third International Conference on 3-D Digital Imaging and Modeling.

[14]  Han-Ul Kim,et al.  Monocular Depth Estimation Using Whole Strip Masking and Reliability-Based Refinement , 2018, ECCV.

[15]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[16]  Maks Ovsjanikov,et al.  PCPNet Learning Local Shape Properties from Raw Point Clouds , 2017, Comput. Graph. Forum.

[17]  Daniel Cremers,et al.  Fight Ill-Posedness with Ill-Posedness: Single-shot Variational Depth Super-Resolution from Shading , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[18]  Michael Goesele,et al.  Let There Be Color! Large-Scale Texturing of 3D Reconstructions , 2014, ECCV.

[19]  Slobodan Ilic,et al.  SobolevFusion: 3D Reconstruction of Scenes Undergoing Free Non-rigid Motion , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[20]  ChaaraouiAlexandros Andre,et al.  Visual privacy protection methods , 2015 .

[21]  Alexandros André Chaaraoui,et al.  Visual privacy protection methods: A survey , 2015, Expert Syst. Appl..

[22]  Lizhen Wang,et al.  DDRNet: Depth Map Denoising and Refinement for Consumer Depth Cameras Using Cascaded CNNs , 2018, ECCV.

[23]  Roland Siegwart,et al.  Comparing ICP variants on real-world data sets , 2013, Auton. Robots.

[24]  Jean-Philippe Pons,et al.  Seamless image-based texture atlases using multi-band blending , 2008, 2008 19th International Conference on Pattern Recognition.

[25]  Andrea Tagliasacchi,et al.  Eurographics Symposium on Geometry Processing 2013 Sparse Iterative Closest Point , 2022 .

[26]  Peiyun Hu,et al.  Finding Tiny Faces , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Dimitrios Tzovaras,et al.  Fault Diagnosis in Microelectronics Attachment Via Deep Learning Analysis of 3-D Laser Scans , 2020, IEEE Transactions on Industrial Electronics.

[28]  Lei Yang,et al.  3D depth image analysis for indoor fall detection of elderly people , 2016, Digit. Commun. Networks.

[29]  George Michalos,et al.  A Machine Learning Approach for Visual Recognition of Complex Parts in Robotic Manipulation , 2017 .

[30]  Max Mignotte,et al.  Fall Detection from Depth Map Video Sequences , 2011, ICOST.

[31]  Michael M. Kazhdan,et al.  Screened poisson surface reconstruction , 2013, TOGS.

[32]  Joachim Giesen,et al.  Delaunay Triangulation Based Surface Reconstruction , 2006 .

[33]  SiegwartRoland,et al.  Comparing ICP variants on real-world data sets , 2013 .

[34]  Mingxi Wang,et al.  Measuring cloud point pressures by image analysis: A simple and reproducible alternative method to direct visual determination , 2019 .

[35]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[36]  D. Tzovaras,et al.  Using Activity-Related Behavioural Features towards More Effective Automatic Stress Detection , 2012, PloS one.

[37]  Jürgen Döllner,et al.  Concepts and techniques for web-based visualization and processing of massive 3D point clouds with semantics , 2019, Graph. Model..

[38]  Shahram Izadi,et al.  StereoNet: Guided Hierarchical Refinement for Real-Time Edge-Aware Depth Prediction , 2018, ECCV.

[39]  Michael G. Strintzis,et al.  Robust image watermarking in the subband or discrete cosine transform domain , 1998, 9th European Signal Processing Conference (EUSIPCO 1998).

[40]  Brian L. Steward,et al.  Automated crop plant detection based on the fusion of color and depth images for robotic weed control , 2019, J. Field Robotics.

[41]  Dimitrios Tzovaras,et al.  Managing Spatial Graph Dependencies in Large Volumes of Traffic Data for Travel-Time Prediction , 2016, IEEE Transactions on Intelligent Transportation Systems.

[42]  Kostas Daniilidis,et al.  Fully Automatic Registration of 3D Point Clouds , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[43]  Andrew Zisserman,et al.  Multiple View Geometry in Computer Vision (2nd ed) , 2003 .

[44]  Yinda Zhang,et al.  ActiveStereoNet: End-to-End Self-Supervised Learning for Active Stereo Systems , 2018, ECCV.

[45]  Nassir Navab,et al.  Coloured signed distance fields for full 3D object reconstruction , 2014, BMVC.

[46]  Nico Blodow,et al.  Fast Point Feature Histograms (FPFH) for 3D registration , 2009, 2009 IEEE International Conference on Robotics and Automation.

[47]  Oleksandr Semeniuta,et al.  Vision-based robotic system for picking and inspection of small automotive components , 2016, 2016 IEEE International Conference on Automation Science and Engineering (CASE).

[48]  Shuo Yang,et al.  WIDER FACE: A Face Detection Benchmark , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Richard K. Beatson,et al.  Reconstruction and representation of 3D objects with radial basis functions , 2001, SIGGRAPH.

[50]  Cordelia Schmid,et al.  DeepMatching: Hierarchical Deformable Dense Matching , 2015, International Journal of Computer Vision.

[51]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[52]  Michael G. Strintzis,et al.  3-D Model Search and Retrieval From Range Images Using Salient Features , 2010, IEEE Transactions on Multimedia.

[53]  Vladlen Koltun,et al.  Fast Global Registration , 2016, ECCV.

[54]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..