A Framework for Automatic Generation of Augmented Reality Maintenance & Repair Instructions based on Convolutional Neural Networks

Abstract In modern manufacturing world, MRO (Maintenance and Repair Operations) are the cornerstone for keeping industrial equipment in near-optimum condition. Successful completion of MRO has been benefited from Augmented Reality (AR), by considerably decreasing MTTR (Mean Time to Repair). To that end, AR delivers the digital tools that help on-the-field technicians to perform MRO easily and intuitively, however intense development is required for generating AR instructions. The latest advances in computer technologies, concretely in Convolutional Neural Networks, have enabled advanced computer vision. This research paper presents a framework for generating AR maintenance instructions, based on advanced computer vision and Convolutional Neural Networks (CNN). The applicability of the framework is tested in-vitro in a lab-based machine shop.

[1]  Li Lin,et al.  Remaining useful life estimation of engineered systems using vanilla LSTM neural networks , 2018, Neurocomputing.

[2]  Dimitris Mourtzis,et al.  Knowledge-based Estimation of Maintenance Time for Complex Engineered-to-Order Products Based on KPIs Monitoring: A PSS Approach , 2017 .

[3]  Liang Guo,et al.  A recurrent neural network based health indicator for remaining useful life prediction of bearings , 2017, Neurocomputing.

[4]  Wei Zhao,et al.  Remaining useful life prediction using multi-scale deep convolutional neural network , 2020, Appl. Soft Comput..

[5]  Mohammad Ebrahim Shiri Ahmad Abadi,et al.  Maintaining filter structure: A Gabor-based convolutional neural network for image analysis , 2020, Appl. Soft Comput..

[6]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2015, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  Dimitris Mourtzis,et al.  Augmented Reality Application to Support Remote Maintenance as a Service in the Robotics Industry , 2017 .

[8]  Ling Shao,et al.  From handcrafted to learned representations for human action recognition: A survey , 2016, Image Vis. Comput..

[9]  Robert X. Gao,et al.  Virtualization and deep recognition for system fault classification , 2017 .

[10]  José Manuel Ferrández,et al.  Image Processing with CNN in a FPGA-Based Augmented Reality System for Visually Impaired People , 2005, IWANN.

[11]  Jianhua Wang,et al.  Detection of weak transient signals based on unsupervised learning for bearing fault diagnosis , 2018, Neurocomputing.

[12]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[13]  Tao Lei,et al.  A review of Convolutional-Neural-Network-based action recognition , 2019, Pattern Recognit. Lett..

[14]  Wei-Yun Yau,et al.  Human Action Recognition With Video Data: Research and Evaluation Challenges , 2014, IEEE Transactions on Human-Machine Systems.

[15]  Ming J. Zuo,et al.  A dependence-based feature vector and its application on planetary gearbox fault classification , 2018, Journal of Sound and Vibration.

[16]  Suk-Ho Lee,et al.  3D Integral Imaging Based Augmented Reality with Deep Learning Implemented by Faster R-CNN , 2017 .

[17]  Yan Han,et al.  Multi-level wavelet packet fusion in dynamic ensemble convolutional neural network for fault diagnosis , 2018, Measurement.

[18]  Guanghua Xu,et al.  Health indicator construction of machinery based on end-to-end trainable convolution recurrent neural networks , 2020 .

[19]  Yan Han,et al.  An enhanced convolutional neural network with enlarged receptive fields for fault diagnosis of planetary gearboxes , 2019, Comput. Ind..

[20]  Yaguo Lei,et al.  Recurrent convolutional neural network: A new framework for remaining useful life prediction of machinery , 2020, Neurocomputing.

[21]  George Chryssolouris,et al.  Assembly precedence diagram generation through assembly tiers determination , 2016, Int. J. Comput. Integr. Manuf..