Deep Learning: A Comprehensive Overview on Techniques, Taxonomy, Applications and Research Directions

Deep learning (DL), a branch of machine learning (ML) and artificial intelligence (AI) is nowadays considered as a core technology of today’s Fourth Industrial Revolution (4IR or Industry 4.0). Due to its learning capabilities from data, DL technology originated from artificial neural network (ANN), has become a hot topic in the context of computing, and is widely applied in various application areas like healthcare, visual recognition, text analytics, cybersecurity, and many more. However, building an appropriate DL model is a challenging task, due to the dynamic nature and variations in real-world problems and data. Moreover, the lack of core understanding turns DL methods into black-box machines that hamper development at the standard level. This article presents a structured and comprehensive view on DL techniques including a taxonomy considering various types of real-world tasks like supervised or unsupervised. In our taxonomy, we take into account deep networks for supervised or discriminative learning, unsupervised or generative learning as well as hybrid learning and relevant others. We also summarize real-world application areas where deep learning techniques can be used. Finally, we point out ten potential aspects for future generation DL modeling with research directions. Overall, this article aims to draw a big picture on DL modeling that can be used as a reference guide for both academia and industry professionals.

[1]  Iqbal H. Sarker,et al.  Data Science and Analytics: An Overview from Data-Driven Smart Computing, Decision-Making and Applications Perspective , 2021, SN Computer Science.

[2]  Joel J. P. C. Rodrigues,et al.  Hybrid Deep-Learning-Based Anomaly Detection Scheme for Suspicious Flow Detection in SDN: A Social Multimedia Perspective , 2019, IEEE Transactions on Multimedia.

[3]  Mohamed Abdel-Basset,et al.  Energy-Net: A Deep Learning Approach for Smart Energy Management in IoT-Based Smart Cities , 2021, IEEE Internet of Things Journal.

[4]  Iqbal H. Sarker,et al.  ContextPCA: Predicting Context-Aware Smartphone Apps Usage Based On Machine Learning Techniques , 2020, Symmetry.

[5]  Xishi Huang,et al.  From traditional to deep learning: Fault diagnosis for autonomous vehicles , 2021 .

[6]  Yue Liu,et al.  Domain Knowledge-Based Deep-Broad Learning Framework for Fault Diagnosis , 2021, IEEE Transactions on Industrial Electronics.

[7]  B. Ślusarczyk.  INDUSTRY 4.0 – ARE WE READY? , 2018, Polish Journal of Management Studies.

[8]  Xin Wang,et al.  A smart agriculture IoT system based on deep reinforcement learning , 2019, Future Gener. Comput. Syst..

[9]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[10]  J Anuradha,et al.  Big data based stock trend prediction using deep CNN with reinforcement-LSTM model , 2021 .

[11]  Mamta Mittal,et al.  Generative adversarial network: An overview of theory and applications , 2021, Int. J. Inf. Manag. Data Insights.

[12]  Pascal Vincent,et al.  Higher Order Contractive Auto-Encoder , 2011, ECML/PKDD.

[13]  M. Z. Islam,et al.  A combined deep CNN-LSTM network for the detection of novel coronavirus (COVID-19) using X-ray images , 2020, Informatics in Medicine Unlocked.

[14]  Iqbal H. Sarker,et al.  Effectiveness analysis of machine learning classification models for predicting personalized context-aware smartphone usage , 2019, Journal of Big Data.

[15]  Hirokazu Kameoka,et al.  Supervised Determined Source Separation with Multichannel Variational Autoencoder , 2019, Neural Computation.

[16]  Chao Yang,et al.  A Survey on Deep Transfer Learning , 2018, ICANN.

[17]  Tim Kraska,et al.  Sherlock: A Deep Learning Approach to Semantic Data Type Detection , 2019, KDD.

[18]  Sung-Bae Cho,et al.  Zero-day malware detection using transferred generative adversarial networks based on deep autoencoders , 2018, Inf. Sci..

[19]  Julio Navío-Marco,et al.  Analysis of computer user behavior, security incidents and fraud using Self-Organizing Maps , 2019, Comput. Secur..

[20]  Lina Yao,et al.  Multi-modality Sensor Data Classification with Selective Attention , 2018, IJCAI.

[21]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[22]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Chang Soo Kim,et al.  Multi-Step Short-Term Power Consumption Forecasting Using Multi-Channel LSTM With Time Location Considering Customer Behavior , 2020, IEEE Access.

[24]  Rohan Ramanath,et al.  An Attentive Survey of Attention Models , 2019, ACM Trans. Intell. Syst. Technol..

[25]  Giancarlo Fortino,et al.  CNN-Based Health Model for Regular Health Factors Analysis in Internet-of-Medical Things Environment , 2020, IEEE Access.

[26]  H. A. Ananya,et al.  Deep Learning based Large Scale Visual Recommendation and Search for E-Commerce , 2017, ArXiv.

[27]  Iqbal H. Sarker.  A Machine Learning based Robust Prediction Model for Real-life Mobile Phone Data , 2019, Internet Things.

[28]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[29]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[30]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[31]  Manuel López Martín,et al.  Application of deep reinforcement learning to intrusion detection for supervised problems , 2020, Expert Syst. Appl..

[32]  Iqbal H. Sarker.  Deep Cybersecurity: A Comprehensive Overview from Neural Network and Deep Learning Perspective , 2021, SN Computer Science.

[33]  Naomie Salim,et al.  Recommendation system based on deep learning methods: a systematic review and new directions , 2019, Artificial Intelligence Review.

[34]  Indranil Bose,et al.  On the platform but will they buy? Predicting customers' purchase behavior using deep learning , 2021, Decis. Support Syst..

[35]  Joelle Pineau,et al.  Domain Adversarial Reinforcement Learning , 2021, ArXiv.

[36]  Geoffrey E. Hinton,et al.  Learning to Represent Spatial Transformations with Factored Higher-Order Boltzmann Machines , 2010, Neural Computation.

[37]  Wei Liu,et al.  A Survey on Deep Learning Empowered IoT Applications , 2019, IEEE Access.

[38]  Vinod Vokkarane,et al.  A New Deep Learning-Based Food Recognition System for Dietary Assessment on An Edge Computing Service Infrastructure , 2018, IEEE Transactions on Services Computing.

[39]  Yonghong Yan,et al.  Attention-Based LSTM with Multi-Task Learning for Distant Speech Recognition , 2017, INTERSPEECH.

[40]  Lin Yang,et al.  A Survey on the Development of Self-Organizing Maps for Unsupervised Intrusion Detection , 2021, Mob. Networks Appl..

[41]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[42]  Ye Wang,et al.  Deep Learning Based Plant Disease Detection for Smart Agriculture , 2019, 2019 IEEE Globecom Workshops (GC Wkshps).

[43]  Zhen Zhang,et al.  An Optimization Method for Intrusion Detection Classification Model Based on Deep Belief Network , 2019, IEEE Access.

[44]  Domain-specific cues improve robustness of deep learning-based segmentation of CT volumes , 2019, Scientific Reports.

[45]  Chunhua Wang,et al.  Machine Learning and Deep Learning Methods for Cybersecurity , 2018, IEEE Access.

[46]  Trevor Darrell,et al.  Adversarial Feature Learning , 2016, ICLR.

[47]  Nando de Freitas,et al.  Inductive Principles for Restricted Boltzmann Machine Learning , 2010, AISTATS.

[48]  Ping-Huan Kuo,et al.  A Deep CNN-LSTM Model for Particulate Matter (PM2.5) Forecasting in Smart Cities , 2018, Sensors.

[49]  Dong Yu,et al.  Deep Learning: Methods and Applications , 2014, Found. Trends Signal Process..

[50]  Iqbal H. Sarker,et al.  BehavDT: A Behavioral Decision Tree Learning to Build User-Centric Context-Aware Predictive Model , 2019, Mobile Networks and Applications.

[51]  Amir H. Behzadan,et al.  Convolutional neural networks for object detection in aerial imagery for disaster response and recovery , 2020, Adv. Eng. Informatics.

[52]  Salvatore Cuomo,et al.  Predictive Analytics for Smart Parking: A Deep Learning Approach in Forecasting of IoT Data , 2021, ACM Trans. Internet Techn..

[53]  V. V. Karthikeyan,et al.  Soil monitoring and evaluation system using EDL‐ASQE: Enhanced deep learning model for IoT smart agriculture network , 2021, Int. J. Commun. Syst..

[54]  Mohammad Motiur Rahman,et al.  The architectural design of smart blind assistant using IoT with deep learning paradigm , 2020, Internet Things.

[55]  Petra Perner,et al.  Data Mining - Concepts and Techniques , 2002, Künstliche Intell..

[56]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2015, IEEE Trans. Pattern Anal. Mach. Intell..

[57]  Antonio Coronato,et al.  A reinforcement learning and deep learning based intelligent system for the support of impaired patients in home treatment , 2020, Expert Syst. Appl..

[58]  Carolyn R. Bertozzi,et al.  Methods and Applications , 2009 .

[59]  Tzuu-Hseng S. Li,et al.  CNN and LSTM Based Facial Expression Analysis Model for a Humanoid Robot , 2019, IEEE Access.

[60]  Pascal Vincent,et al.  Contractive Auto-Encoders: Explicit Invariance During Feature Extraction , 2011, ICML.

[61]  Martin L. Puterman,et al.  Markov Decision Processes: Discrete Stochastic Dynamic Programming , 1994 .

[62]  Yan Cui,et al.  Transfer Learning for Molecular Cancer Classification Using Deep Neural Networks , 2019, IEEE/ACM Transactions on Computational Biology and Bioinformatics.

[63]  Zhihan Lv,et al.  Cyber security in smart cities: A review of deep learning-based applications and case studies , 2020 .

[64]  Juha Karhunen,et al.  Chapter 7 – Unsupervised deep learning: A short review , 2015 .

[65]  Iqbal H. Sarker.  CyberLearning: Effectiveness Analysis of Machine Learning Security Modeling to Detect Cyber-Anomalies and Multi-Attacks , 2021, Internet Things.

[66]  Wilson Vicente Ruggiero,et al.  A Knowledge-Based Recommendation System That Includes Sentiment Analysis and Deep Learning , 2019, IEEE Transactions on Industrial Informatics.

[67]  Yaohui Jin,et al.  CNN-based feature cross and classifier for loan default prediction , 2020, Other Conferences.

[68]  Danilo P. Mandic,et al.  Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability , 2001 .

[69]  Iqbal H. Sarker.  Context-aware rule learning from smartphone data: survey, challenges and future directions , 2019, Journal of Big Data.

[70]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[71]  Yurong Liu,et al.  A survey of deep neural network architectures and their applications , 2017, Neurocomputing.

[72]  Ying Tan,et al.  Variational Autoencoder for Semi-Supervised Text Classification , 2017, AAAI.

[73]  Iqbal H. Sarker,et al.  ABC-RuleMiner: User behavioral rule-based machine learning method for context-aware intelligent services , 2020, J. Netw. Comput. Appl..

[74]  Teuvo Kohonen,et al.  The self-organizing map , 1990 .

[75]  Iqbal H. Sarker,et al.  RecencyMiner: mining recency-based personalized behavior from contextual smartphone data , 2019, Journal of Big Data.

[76]  Gyanendra K. Verma,et al.  Deep belief network based affect recognition from physiological signals , 2017, 2017 4th IEEE Uttar Pradesh Section International Conference on Electrical, Computer and Electronics (UPCON).

[77]  Hung-Hsuan Huang,et al.  Toward RNN Based Micro Non-verbal Behavior Generation for Virtual Listener Agents , 2019, HCI.

[78]  Iqbal H. Sarker,et al.  Mobile Data Science and Intelligent Apps: Concepts, AI-Based Modeling and Research Directions , 2020, Mobile Networks and Applications.

[79]  M.N.S. Swamy,et al.  Neural Networks and Statistical Learning , 2013 .

[80]  Ron Hoory,et al.  Efficient Emotion Recognition from Speech Using Deep Learning on Spectrograms , 2017, INTERSPEECH.

[81]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[82]  Muhammet Fatih Aslan,et al.  CNN-based transfer learning–BiLSTM network: A novel approach for COVID-19 infection detection , 2020, Applied Soft Computing.

[83]  Teuvo Kohonen,et al.  The self-organizing map , 1990, Neurocomputing.

[84]  Suat Özdemir,et al.  A deep learning model for air quality prediction in smart cities , 2017, 2017 IEEE International Conference on Big Data (Big Data).

[85]  Geoffrey E. Hinton.  Deep belief networks , 2009, Scholarpedia.

[86]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[87]  Juha Karhunen,et al.  Advances in Independent Component Analysis and Learning Machines , 2015 .

[88]  Kiran Rama,et al.  DNNRec: A novel deep learning based hybrid recommender system , 2020, Expert Syst. Appl..

[89]  Rajib Rana,et al.  Transfer Learning for Improving Speech Emotion Classification Accuracy , 2018, INTERSPEECH.

[90]  Yu Lasheng,et al.  Deep Learning Approach Combining Sparse Autoencoder With SVM for Network Intrusion Detection , 2018, IEEE Access.

[91]  Frederico G. Guimarães,et al.  A GPU deep learning metaheuristic based model for time series forecasting , 2017 .

[92]  Iqbal H. Sarker,et al.  Analytics : An Overview from Data ‐ Driven Smart Computing , Decision ‐ Making and Applications Perspective , 2021 .

[93]  Akbar Siami Namin,et al.  The Performance of LSTM and BiLSTM in Forecasting Time Series , 2019, 2019 IEEE International Conference on Big Data (Big Data).

[94]  Shervin Minaee,et al.  Deep-Sentiment: Sentiment Analysis Using Ensemble of CNN and Bi-LSTM Models , 2019, ArXiv.

[95]  Taghi M. Khoshgoftaar,et al.  A survey of transfer learning , 2016, Journal of Big Data.

[96]  Iqbal H. Sarker,et al.  Individualized Time-Series Segmentation for Mining Mobile Phone User Behavior , 2018, Comput. J..

[97]  Md. Milon Islam,et al.  A combined deep CNN-LSTM network for the detection of novel coronavirus (COVID-19) using X-ray images , 2020, Informatics in Medicine Unlocked.

[98]  Limin Luo,et al.  Automatic and Robust Object Detection in X-Ray Baggage Inspection Using Deep Convolutional Neural Networks , 2021, IEEE Transactions on Industrial Electronics.

[99]  Iqbal H. Sarker,et al.  Cybersecurity data science: an overview from machine learning perspective , 2020, Journal of Big Data.

[100]  Pascal Vincent,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010, J. Mach. Learn. Res..

[101]  Teuvo Kohonen,et al.  Essentials of the self-organizing map , 2013, Neural Networks.

[102]  Li Zhao,et al.  Attention-based LSTM for Aspect-level Sentiment Classification , 2016, EMNLP.

[103]  이상헌,et al.  Deep Belief Networks , 2010, Encyclopedia of Machine Learning.

[104]  Mahmood Yousefi-Azar,et al.  Text summarization using unsupervised deep learning , 2017, Expert Syst. Appl..

[105]  Iqbal H. Sarker,et al.  AI-Driven Cybersecurity: An Overview, Security Intelligence Modeling and Research Directions , 2021, SN Computer Science.

[106]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[107]  Jiafu Wan,et al.  Knowledge Reasoning with Semantic Data for Real-Time Data Processing in Smart Factory , 2018, Sensors.

[108]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[109]  Esa Alhoniemi,et al.  Clustering of the self-organizing map , 2000, IEEE Trans. Neural Networks Learn. Syst..

[110]  Liang-Chih Yu,et al.  Tree-Structured Regional CNN-LSTM Model for Dimensional Sentiment Analysis , 2020, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[111]  Lina Yao,et al.  A Survey on Deep Learning based Brain Computer Interface: Recent Advances and New Frontiers , 2019, ArXiv.

[112]  Keun Ho Ryu,et al.  Deep Autoencoder Based Neural Networks for Coronary Heart Disease Risk Prediction , 2019, Poly/DMAH@VLDB.

[113]  Xing Fang,et al.  Toward multi-label sentiment analysis: a transfer learning based approach , 2020, Journal of Big Data.

[114]  Iqbal H. Sarker,et al.  IntruDTree: A Machine Learning Based Cyber Security Intrusion Detection Model , 2020, Symmetry.

[115]  Hui Yu,et al.  A review on the attention mechanism of deep learning , 2021, Neurocomputing.

[116]  Liang-Bi Chen,et al.  A Deep Learning-Based Intelligent Medicine Recognition System for Chronic Patients , 2019, IEEE Access.

[117]  Aurélien Géron,et al.  Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems , 2017 .

[118]  Li Deng,et al.  A tutorial survey of architectures, algorithms, and applications for deep learning , 2014, APSIPA Transactions on Signal and Information Processing.

[119]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[120]  Chen Chen,et al.  A Real-Time Collision Prediction Mechanism With Deep Learning for Intelligent Transportation System , 2020, IEEE Transactions on Vehicular Technology.

[121]  Diederik P. Kingma,et al.  An Introduction to Variational Autoencoders , 2019, Found. Trends Mach. Learn..

[122]  M. Shamim Hossain,et al.  Deep learning-based intelligent face recognition in IoT-cloud environment , 2020, Comput. Commun..

[123]  François Chollet,et al.  Xception: Deep Learning with Depthwise Separable Convolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[124]  Mooi Choo Chuah,et al.  New attacks on RNN based healthcare learning system and their detections , 2018 .

[125]  Yoshua Bengio,et al.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling , 2014, ArXiv.

[126]  Wei Wang,et al.  Effective android malware detection with a hybrid model based on deep autoencoder and convolutional neural network , 2018, Journal of Ambient Intelligence and Humanized Computing.

[127]  Xiaohui Yuan,et al.  A review of deep learning methods for semantic segmentation of remote sensing imagery , 2021, Expert Syst. Appl..

[128]  Danilo P. Mandic,et al.  Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability , 2001 .

[129]  Yadigar Imamverdiyev,et al.  Deep Learning Method for Denial of Service Attack Detection Based on Restricted Boltzmann Machine , 2018, Big Data.

[130]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[131]  Rajiv Kumar,et al.  An intelligent Chatbot using deep learning with Bidirectional RNN and attention model , 2020, Materials Today: Proceedings.

[132]  Rashid Mehmood,et al.  Disaster Management in Smart Cities by Forecasting Traffic Plan Using Deep Learning and GPUs , 2017 .

[133]  Nicole Gruber,et al.  Are GRU Cells More Specific and LSTM Cells More Sensitive in Motive Classification of Text? , 2020, Frontiers in Artificial Intelligence.

[134]  Marc Peter Deisenroth,et al.  Deep Reinforcement Learning: A Brief Survey , 2017, IEEE Signal Processing Magazine.

[135]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[136]  Iqbal H. Sarker,et al.  AppsPred: Predicting Context-Aware Smartphone Apps using Random Forest Learning , 2019, Internet Things.

[137]  Guijuan Zhang,et al.  A survey of autoencoder-based recommender systems , 2019, Frontiers of Computer Science.

[138]  Roger Zimmermann,et al.  Aspect-Based Financial Sentiment Analysis using Deep Learning , 2018, WWW.

[139]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.