An Automated Training of Deep Learning Networks by 3D Virtual Models for Object Recognition

Small series production with a high level of variability is not suitable for full automation. So, a manual assembly process must be used, which can be improved by cooperative robots and assisted by augmented reality devices. The assisted assembly process needs reliable object recognition implementation. Currently used technologies with markers do not work reliably with objects without distinctive texture, for example, screws, nuts, and washers (single colored parts). The methodology presented in the paper introduces a new approach to object detection using deep learning networks trained remotely by 3D virtual models. Remote web application generates training input datasets from virtual 3D models. This new approach was evaluated by two different neural network models (Faster RCNN Inception v2 with SSD, MobileNet V2 with SSD). The main advantage of this approach is the very fast preparation of the 2D sample training dataset from virtual 3D models. The whole process can run in Cloud. The experiments were conducted with standard parts (nuts, screws, washers) and the recognition precision achieved was comparable with training by real samples. The learned models were tested by two different embedded devices with an Android operating system: Virtual Reality (VR) glasses, Cardboard (Samsung S7), and Augmented Reality (AR) smart glasses (Epson Moverio M350). The recognition processing delays of the learned models running in embedded devices based on an ARM processor and standard x86 processing unit were also tested for performance comparison.

[1]  Kamil Židek,et al.  Recognition of Assembly Parts by Convolutional Neural Networks , 2018, Advances in Manufacturing Engineering and Materials.

[2]  Kasthurirangan Gopalakrishnan,et al.  Deep Learning in Data-Driven Pavement Image Analysis and Automated Distress Detection: A Review , 2018, Data.

[3]  Didier Stricker,et al.  Trained 3D Models for CNN based Object Recognition , 2017, VISIGRAPP.

[4]  Petr Baron,et al.  AUTOMATIZATION PROCESS OF DESIGNING THE TECHNOLOGICAL DOCUMENTATION BY TOOLS OF COMPREHENSIVE CAD-CAM-CAE SYSTÉM , 2016 .

[5]  Kate Saenko,et al.  Learning Deep Object Detectors from 3D Models , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[6]  Yilun Shang,et al.  On the Delayed Scaled Consensus Problems , 2017 .

[7]  Xuan Li,et al.  Training and testing object detectors with virtual images , 2017, IEEE/CAA Journal of Automatica Sinica.

[8]  Vitalii Oleksandrovych Ivanov,et al.  Scientific and Methodological Approach for the Identification of Mathematical Models of Mechanical Systems by Using Artificial Neural Networks , 2018, Innovation, Engineering and Entrepreneurship.

[9]  Hannes Kaufmann,et al.  Extraction of Structural and Semantic Data from 2D Floor Plans for Interactive and Immersive VR Real Estate Exploration , 2018, Technologies.

[10]  Ján Piteľ,et al.  Machine learning algorithms implementation into embedded systems with web application user interface , 2017, 2017 IEEE 21st International Conference on Intelligent Engineering Systems (INES).

[11]  Yilun Shang,et al.  Deffuant model of opinion formation in one-dimensional multiplex networks , 2015 .

[12]  Kamil Židek,et al.  Diagnostics of Errors at Component Surface by Vision Recognition in Production Systems , 2014 .

[13]  Cong Hong Phong Nguyen,et al.  Triangular Mesh and Boundary Representation Combined Approach for 3D CAD Lightweight Representation for Collaborative Product Development , 2019, J. Comput. Inf. Sci. Eng..

[14]  Kamil Židek,et al.  Diagnostics of Surface Errors by Embedded Vision System and its Classification by Machine Learning Algorithms , 2015 .

[15]  Zhenhua Tan,et al.  A Case Study on Attribute Recognition of Heated Metal Mark Image Using Deep Convolutional Neural Networks , 2018, Sensors.

[16]  Vladislav Maxim,et al.  Embedded vision equipment of industrial robot for inline detection of product errors by clustering–classification algorithms , 2016 .

[17]  Ki-Ryong Kwon,et al.  Anti-3D Weapon Model Detection for Safe 3D Printing Based on Convolutional Neural Networks and D2 Shape Distribution , 2018, Symmetry.

[18]  Rastislav Pirník,et al.  Searching for collisions between mobile robot and environment , 2016 .

[19]  Long Chen,et al.  An Improved Augmented Reality Registration Method Based on Visual SLAM , 2017, Edutainment.

[20]  Kamil Židek,et al.  Wireless Device Based on MEMS Sensors and Bluetooth Low Energy (LE/Smart) Technology for Diagnostics of Mechatronic Systems , 2013 .

[21]  Lihui Wang,et al.  An AR-based Worker Support System for Human-Robot Collaboration , 2017 .

[22]  Marius Ionut Ripanu,et al.  Product engineering design enhancing by parameterizing the 3D solid model , 2018 .

[23]  Weizhi Nie,et al.  Convolutional deep learning for 3D object retrieval , 2017, Multimedia Systems.

[24]  Andrew Y. Ng,et al.  Convolutional-Recursive Deep Learning for 3D Object Classification , 2012, NIPS.

[25]  Vitalii Oleksandrovych Ivanov,et al.  Application of Artificial Neural Network for Identification of Bearing Stiffness Characteristics in Rotor Dynamics Analysis , 2018, Lecture Notes in Mechanical Engineering.

[26]  Leonidas J. Guibas,et al.  Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).