DeepHaul: a deep learning and reinforcement learning-based smart automation framework for dump trucks

In surface mining operations, the haul truck contributes the largest share to the overall operating cost, along with being a significant contributor to the injuries and fatalities to the operators; therefore, measures need to be taken for improving truck haulage safety and efficiency. In the absence of any major advancement in the automation technology for the mining sector, this study attempted to eliminate the existing technology lull by developing a novel DeepHaul framework for inducing smartness and intelligence within any dump truck, using the advanced algorithmic knowledge of artificial intelligence and machine learning. The DeepHaul framework consisted of two major components: first, inducing an object recognition ability by using deep learning methodology, for any dump truck. Experiments were conducted with different deep learning architectures, different training batch size ranges, and various image sizes for developing an optimum deep learning model with state-of-the-art performance for the haul truck. The second component consisted of the steering action decision making ability for the dump truck based on the given state of the haulage route. A reinforcement learning-based algorithm was designed, implemented, and tested for achieving the aforementioned objective. The algorithm exhibited an accuracy of 100% regarding safety and an average accuracy score of more than 97% regarding haulage efficiency. With the implementation of this DeepHaul framework, the existing autonomous haulage truck control technology can be greatly enhanced by inducing intelligence and smartness within the haul trucks, which would result in improved safety, efficiency, and the effectiveness of mining operations.

[1]  S.N. Balakrishnan,et al.  ETNAC Design Enabling Formation Flight at Liberation Points , 2019, 2019 American Control Conference (ACC).

[2]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Chetan Gupta,et al.  Dispatch with Confidence: Integration of Machine Learning, Optimization and Simulation for Open Pit Mines , 2017, KDD.

[4]  Dimitri Palaz,et al.  Analysis of CNN-based speech recognition system using raw speech as input , 2015, INTERSPEECH.

[5]  E. Finkelstein,et al.  Development and Validation of a Deep Learning System for Diabetic Retinopathy and Related Eye Diseases Using Retinal Images From Multiethnic Populations With Diabetes , 2017, JAMA.

[6]  M. He,et al.  Efficacy of a Deep Learning System for Detecting Glaucomatous Optic Neuropathy Based on Color Fundus Photographs. , 2018, Ophthalmology.

[7]  Alex Graves,et al.  Playing Atari with Deep Reinforcement Learning , 2013, ArXiv.

[8]  Subhashini Venugopalan,et al.  Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. , 2016, JAMA.

[9]  V. S. S. Nadendla,et al.  Effect of text message alerts on miners evacuation decisions , 2020 .

[10]  Saeid R. Dindarloo,et al.  Prediction of fuel consumption of mining dump trucks: A neural networks approach , 2015 .

[11]  Ronald M. Summers,et al.  Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning , 2016, IEEE Transactions on Medical Imaging.

[12]  Phil Blunsom,et al.  A Convolutional Neural Network for Modelling Sentences , 2014, ACL.

[13]  Qingmin Liao,et al.  Video-based construction vehicles detection and its application in intelligent monitoring system , 2016, CAAI Trans. Intell. Technol..

[14]  Lei Yang,et al.  The Use of a Machine Learning Method to Predict the Real-Time Link Travel Time of Open-Pit Trucks , 2018 .

[15]  Michael V. McConnell,et al.  Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning , 2017, Nature Biomedical Engineering.

[16]  Samuel Frimpong,et al.  DeepImpact: a deep learning model for whole body vibration control using impact force monitoring , 2020, Neural Computing and Applications.

[17]  Masaki Mori,et al.  Automatic Wheel-Loader in Asphalt Plant , 1992 .

[18]  Gerald Tesauro,et al.  Temporal difference learning and TD-Gammon , 1995, CACM.

[19]  Samuel Frimpong,et al.  Artificial intelligence, machine learning and process automation: existing knowledge frontier and way forward for mining sector , 2020, Artificial Intelligence Review.

[20]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[21]  Neil J. Joshi,et al.  Automated Grading of Age-Related Macular Degeneration From Color Fundus Images Using Deep Convolutional Neural Networks , 2017, JAMA ophthalmology.

[22]  Maohua Xiao,et al.  Prediction of Surface Roughness and Optimization of Cutting Parameters of Stainless Steel Turning Based on RSM , 2018, Mathematical Problems in Engineering.

[23]  A. Peters,et al.  A Deep Learning Algorithm for Prediction of Age-Related Eye Disease Study Severity Scale for Age-Related Macular Degeneration from Color Fundus Photography. , 2018, Ophthalmology.

[24]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[25]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Yue Wu,et al.  Deep-Learning Based, Automated Segmentation of Macular Edema in Optical Coherence Tomography , 2017, bioRxiv.

[27]  Daniel S W Ting,et al.  Clinical Applicability of Deep Learning System in Detecting Tuberculosis with Chest Radiography. , 2018, Radiology.

[28]  Les T. Zoschke,et al.  Experiences to Date with Unmanned Haul Trucks in Open Pit Mines , 2000 .

[29]  Hoo-Chang Hoo-Chang Shin Shin,et al.  Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning , 2016, Ieee Transactions on Medical Imaging.

[30]  Sebastian Thrun,et al.  Dermatologist-level classification of skin cancer with deep neural networks , 2017, Nature.

[31]  Juan Carlos Niebles,et al.  Vision-based action recognition of earthmoving equipment using spatio-temporal features and support vector machine classifiers , 2013, Adv. Eng. Informatics.

[32]  James M. Brown,et al.  Automated Diagnosis of Plus Disease in Retinopathy of Prematurity Using Deep Convolutional Neural Networks , 2018, JAMA ophthalmology.

[33]  Xiang Zhang,et al.  OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks , 2013, ICLR.

[34]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition , 2012 .

[35]  Raja V. Ramani,et al.  Surface Mining Technology: Progress and Prospects , 2012 .

[36]  Andrew H. Beck,et al.  Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer , 2017, JAMA.

[37]  Samuel Frimpong,et al.  A Computer Vision System for Terrain Recognition and Object Detection Tasks in Mining and Construction Environments , 2019 .

[38]  Gregory S. Corrado,et al.  Deep learning for predicting refractive error from retinal fundus images , 2017, Investigative ophthalmology & visual science.

[39]  Rishab Gargeya,et al.  Automated Identification of Diabetic Retinopathy Using Deep Learning. , 2017, Ophthalmology.

[40]  P. Lakhani,et al.  Deep Learning at Chest Radiography: Automated Classification of Pulmonary Tuberculosis by Using Convolutional Neural Networks. , 2017, Radiology.

[41]  Yoon Kim,et al.  Convolutional Neural Networks for Sentence Classification , 2014, EMNLP.

[42]  Samuel Frimpong,et al.  Artificial intelligence models for predicting the performance of hydro-pneumatic suspension struts in large capacity dump trucks , 2018, International Journal of Industrial Ergonomics.

[43]  Sven J. Dickinson,et al.  Server-Customer Interaction Tracker: Computer Vision-Based System to Estimate Dirt-Loading Cycles , 2013 .

[44]  Jian Sun,et al.  Accelerating Very Deep Convolutional Networks for Classification and Detection , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[45]  M. Abràmoff,et al.  Improved Automated Detection of Diabetic Retinopathy on a Publicly Available Dataset Through Integration of Deep Learning. , 2016, Investigative ophthalmology & visual science.

[46]  Darcy M. Bullock,et al.  A Laboratory Study of Force-Cognitive Excavation , 1989 .

[47]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups , 2012, IEEE Signal Processing Magazine.

[48]  Laura Martini,et al.  Machine-related injuries in the US mining industry and priorities for safety research , 2011, International journal of injury control and safety promotion.

[49]  Khaled Shaalan,et al.  Speech Recognition Using Deep Neural Networks: A Systematic Review , 2019, IEEE Access.

[50]  Juliana Parreira,et al.  An interactive simulation model to compare an autonomous haulage truck system with a manually-operated system , 2013 .

[51]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[52]  Jason Weston,et al.  Natural Language Processing (Almost) from Scratch , 2011, J. Mach. Learn. Res..

[53]  Fei-Yue Wang,et al.  Experimental robotic excavation with fuzzy logic and neural networks , 1996, Proceedings of IEEE International Conference on Robotics and Automation.

[54]  S. Jagannathan,et al.  Event Triggered Neuro-Adaptive Controller (ETNAC) Design for Uncertain Linear Systems , 2018, 2018 IEEE Conference on Decision and Control (CDC).