Indoor Augmented Reality Using Deep Learning for Industry 4.0 Smart Factories

This paper proposes to design, develop and implement a fast and markerless mobile augmented reality system to achieve the registration for, the visualization of, and the interaction with machines in indoor smart factories with Industry 4.0 vision. A lightweight deep-learning image detection module based on MobileNets running on mobile devices is used to detect/recognize different machines and different portions of machines. Internet of Things (IoT) networking allows machines and sensors in machines to report data, such as machine settings and machine states, to the cloud-side server. Thus, augmented information associated with a machine portion can be derived from the server and superimposed with the portion image shown on the device display. Furthermore, interaction methods based on touch gestures and distance calculation are also implemented. A prototype system is developed and tested in a mechanical workshop for the purpose of validation and evaluation. The system is shown to achieve high detection accuracy, intuitive visualization, and unique interaction modes.

[1]  José Luis Lerma,et al.  Augmented reality and photogrammetry: A synergy to visualize physical and virtual city environments , 2010 .

[2]  Shubhangi Singh,et al.  MQTT-Message Queuing Telemetry Transport protocol , 2016 .

[3]  Z. Rajnai,et al.  Assessing industry 4.0 readiness of enterprises , 2018, 2018 IEEE 16th World Symposium on Applied Machine Intelligence and Informatics (SAMI).

[4]  Zhiyuan Luo,et al.  On assessing the positioning accuracy of Google Tango in challenging indoor environments , 2017, 2017 International Conference on Indoor Positioning and Indoor Navigation (IPIN).

[5]  Kil To Chong,et al.  A Real-Time Marker-Based Visual Sensor Based on a FPGA and a Soft Core Processor , 2016, Sensors.

[6]  Kousik Sankar Ramasubramaniam,et al.  LcAR — Low cost augmented reality for the automotive industry , 2018, 2018 IEEE International Conference on Consumer Electronics (ICCE).

[7]  Qingyun Du,et al.  A Mobile Outdoor Augmented Reality Method Combining Deep Learning Object Detection and Spatial Relationships for Geovisualization , 2017, Sensors.

[8]  Doug Fisher,et al.  SCADA: Supervisory Control and Data Acquisition , 2015 .

[9]  Darlis Herumurti,et al.  Location based augmented reality game using Kudan SDK , 2017, 2017 11th International Conference on Information & Communication Technology and System (ICTS).

[10]  Mathias Schmitt,et al.  Human-machine-interaction in the industry 4.0 era , 2014, 2014 12th IEEE International Conference on Industrial Informatics (INDIN).

[11]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[12]  Tiago M. Fernández-Caramés,et al.  A Review on Industrial Augmented Reality Systems for the Industry 4.0 Shipyard , 2018, IEEE Access.

[13]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[14]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[15]  Paulo Leitão,et al.  Augmented reality experiments with industrial robot in industry 4.0 environment , 2016, 2016 IEEE 14th International Conference on Industrial Informatics (INDIN).

[16]  R. Schlagowski,et al.  Design of an assistant system for industrial maintenance tasks and implementation of a prototype using augmented reality , 2017, 2017 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM).

[17]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Jehn-Ruey Jiang,et al.  A Marker-Based Cyber-Physical Augmented-Reality Indoor Guidance System for Smart Campuses , 2016, 2016 IEEE 18th International Conference on High Performance Computing and Communications; IEEE 14th International Conference on Smart City; IEEE 2nd International Conference on Data Science and Systems (HPCC/SmartCity/DSS).