An Audio-Based Fault Diagnosis Method for Quadrotors Using Convolutional Neural Network and Transfer Learning

Quadrotor unmanned aerial vehicles (UAVs) have been developed and applied into several types of workplaces, such as warehouses, which usually involve human workers. The co-existence of human and UAVs brings new challenges to UAVs: potential failure of UAVs may cause risk and danger to surrounding human. Effective and efficient detection of such failure may provide early warning to the surrounding human workers and reduce such risk to human beings as much as possible. One of the most common reasons that cause the failure of the UAV’s flight is the physical damage to the propellers. This paper presents a method to detect the propellers damage only based on the audio noise caused by the UAV’s flight. The diagnostic model is developed based on convolutional neural network (CNN) and transfer learning techniques. The audio data is collected from the UAVs in real time, transformed into the time-frequency spectrogram, and used to train the CNN-based diagnostic model. The developed model is able to detect the abnormal features of the spectrogram and thus the physical damage of the propellers. To reduce the data dependence on the UAV’s dynamic models and enable the utilization of the training data from UAVs with different dynamic models, the CNN-based diagnostic model is further augmented by transfer learning. As such, the refinement of the well-trained diagnostic model ground on other UAVs only requires a small amount of UAV’s training data. Experimental tests are conducted to validate the diagnostic model with an accuracy of higher than 90%.

[1]  Xiao Liang,et al.  Image‐based post‐disaster inspection of reinforced concrete bridge systems using deep learning with Bayesian optimization , 2018, Comput. Aided Civ. Infrastructure Eng..

[2]  Wail Gueaieb,et al.  A UAV based system for real time flash flood monitoring in desert environments using Lagrangian microsensors , 2013, 2013 International Conference on Unmanned Aircraft Systems (ICUAS).

[3]  Victor Alves,et al.  Brain Tumor Segmentation Using Convolutional Neural Networks in MRI Images , 2016, IEEE Transactions on Medical Imaging.

[4]  Mogens Blanke,et al.  Diagnosis of Airspeed Measurement Faults for Unmanned Aerial Vehicles , 2014, IEEE Transactions on Aerospace and Electronic Systems.

[5]  Seyed Omid Sajedi,et al.  A Convolutional Cost-Sensitive Crack Localization Algorithm for Automated and Reliable RC Bridge Inspection , 2019, Risk-based Bridge Engineering.

[6]  Li Lin,et al.  Fault diagnosis and remaining useful life estimation of aero engine using LSTM neural network , 2016, 2016 IEEE International Conference on Aircraft Utility Systems (AUS).

[7]  Yang Liu,et al.  A hybrid feature model and deep learning based fault diagnosis for unmanned aerial vehicle sensors , 2018, Neurocomputing.

[8]  Francesco Ferracuti,et al.  Fault detection of a VTOL UAV using Acceleration Measurements , 2019, 2019 18th European Control Conference (ECC).

[9]  Ran Zhang,et al.  Transfer Learning With Neural Networks for Bearing Fault Diagnosis in Changing Working Conditions , 2017, IEEE Access.

[10]  Abdelhamid Rabhi,et al.  Actuator Fault-Diagnosis and Fault-Tolerant-Control using intelligent-Output-Estimator Applied on Quadrotor UAV , 2019, 2019 International Conference on Unmanned Aircraft Systems (ICUAS).

[11]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[12]  Jonathan A. Muse,et al.  IMU Sensor Fault Diagnosis and Estimation for Quadrotor UAVs , 2015 .

[13]  Jiong Tang,et al.  Preprocessing-Free Gear Fault Diagnosis Using Small Datasets With Deep Convolutional Neural Network-Based Transfer Learning , 2017, IEEE Access.

[14]  Tal Hassner,et al.  Age and gender classification using convolutional neural networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[15]  Behnam Ghalamchi,et al.  Vibration-Based Propeller Fault Diagnosis for Multicopters , 2018, 2018 International Conference on Unmanned Aircraft Systems (ICUAS).

[16]  Liang Gao,et al.  A New Deep Transfer Learning Based on Sparse Auto-Encoder for Fault Diagnosis , 2019, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[17]  Robert Babuska,et al.  Railway Track Circuit Fault Diagnosis Using Recurrent Neural Networks , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[18]  Lin Ma,et al.  Learning to Answer Questions from Image Using Convolutional Neural Network , 2015, AAAI.

[19]  Seyed Omid Sajedi,et al.  Intensity-Based Feature Selection for Near Real-Time Damage Diagnosis of Building Structures , 2019, ArXiv.

[20]  Xiao Liang,et al.  Vibration‐based semantic damage segmentation for large‐scale structural health monitoring , 2019, Comput. Aided Civ. Infrastructure Eng..

[21]  Xiao Liang,et al.  A data‐driven framework for near real‐time and robust damage diagnosis of building structures , 2019, Structural Control and Health Monitoring.

[22]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[23]  Zhiqiang Chen,et al.  Deep neural networks-based rolling bearing fault diagnosis , 2017, Microelectron. Reliab..

[24]  Minghui Zheng,et al.  A Preliminary Study on A Physical Model Oriented Learning Algorithm with Application to UAVs , 2019, Volume 3, Rapid Fire Interactive Presentations: Advances in Control Systems; Advances in Robotics and Mechatronics; Automotive and Transportation Systems; Motion Planning and Trajectory Tracking; Soft Mechatronic Actuators and Sensors; Unmanned Ground and.

[25]  Jose de Jesus Rangel-Magdaleno,et al.  Detection of unbalanced blade on UAV by means of audio signal , 2018, 2018 IEEE International Autumn Meeting on Power, Electronics and Computing (ROPEC).

[26]  Gino Iannace,et al.  Fault Diagnosis for UAV Blades Using Artificial Neural Network , 2019, Robotics.

[27]  Gaoliang Peng,et al.  A deep convolutional neural network with new training methods for bearing fault diagnosis under noisy environment and different working load , 2018, Mechanical Systems and Signal Processing.

[28]  Michal Pechoucek,et al.  Autonomous UAV Surveillance in Complex Urban Environments , 2009, 2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology.

[29]  Eleni I. Vlahogianni,et al.  Unmanned Aerial Aircraft Systems for Transportation Engineering: Current Practice and Future Challenges , 2016 .

[30]  Yangquan Chen,et al.  Low-cost Multi-UAV Technologies for Contour Mapping of Nuclear Radiation Field , 2013, J. Intell. Robotic Syst..

[31]  Adam Glowacz,et al.  Early fault diagnosis of bearing and stator faults of the single-phase induction motor using acoustic signals , 2018 .

[32]  Xiao Liang,et al.  A Scalable Model-Based Learning Algorithm With Application to UAVs , 2018, IEEE Control Systems Letters.

[33]  Vinayak K. Bairagi,et al.  Engine fault diagnosis using sound analysis , 2016, 2016 International Conference on Automatic Control and Dynamic Optimization Techniques (ICACDOT).

[34]  K. Mosalam,et al.  SIMULATION-BASED DATA-DRIVEN DAMAGE DETECTION FOR HIGHWAY BRIDGE SYSTEMS , 2017 .

[35]  Abdelhamid Rabhi,et al.  Sensor fault diagnosis and fault tolerant control using intelligent-output-estimator applied on quadrotor UAV , 2016, 2016 International Conference on Unmanned Aircraft Systems (ICUAS).

[36]  Liang Gao,et al.  A New Convolutional Neural Network-Based Data-Driven Fault Diagnosis Method , 2018, IEEE Transactions on Industrial Electronics.

[37]  R. I. A. Harmanny,et al.  Radar micro-Doppler mini-UAV classification using spectrograms and cepstrograms , 2015, International Journal of Microwave and Wireless Technologies.

[38]  Xiao-ping Jiang,et al.  Belt conveyor roller fault audio detection based on the wavelet neural network , 2015, 2015 11th International Conference on Natural Computation (ICNC).