A Survey on Fuzzy Deep Neural Networks

Deep neural networks are a class of powerful machine learning model that uses successive layers of non-linear processing units to extract features from data. However, the training process of such networks is quite computationally intensive and uses commonly used optimization methods that do not guarantee optimum performance. Furthermore, deep learning methods are often sensitive to noise in data and do not operate well in areas where data are incomplete. An alternative, yet little explored, method in enhancing deep learning performance is the use of fuzzy systems. Fuzzy systems have been previously used in conjunction with neural networks. This survey explores the different ways in which deep learning is improved with fuzzy logic systems. The techniques are classified based on how the two paradigms are combined. Finally, the real-life applications of the models are also explored.

[1]  Vikas Kumar,et al.  FUZZY NEURAL NETWORKS , 2012 .

[2]  Abdollah Homaifar,et al.  A deep learning based approach for long-term drought prediction , 2017, SoutheastCon 2017.

[3]  James M. Keller Fuzzy logic rules in low and mid level computer vision tasks , 1996, Proceedings of North American Fuzzy Information Processing.

[4]  Xiaohui Wu,et al.  Detection of driver drowsiness using wearable devices: A feasibility study of the proximity sensor. , 2017, Applied ergonomics.

[5]  Ronald R. Yager,et al.  Pythagorean Membership Grades in Multicriteria Decision Making , 2014, IEEE Transactions on Fuzzy Systems.

[6]  Madan M. Gupta,et al.  On the principles of fuzzy neural networks , 1994 .

[7]  Radford M. Neal Pattern Recognition and Machine Learning , 2007, Technometrics.

[8]  Yuichi Motai,et al.  Respiratory Motion Estimation With Hybrid Implementation of Extended Kalman Filter , 2012, IEEE Transactions on Industrial Electronics.

[9]  Lutz Prechelt,et al.  Early Stopping - But When? , 2012, Neural Networks: Tricks of the Trade.

[10]  Youyong Kong,et al.  A Hierarchical Fused Fuzzy Deep Neural Network for Data Classification , 2017, IEEE Transactions on Fuzzy Systems.

[11]  Ronald R. Yager,et al.  Decision making with fuzzy probability assessments , 1999, IEEE Trans. Fuzzy Syst..

[12]  Nikhil S. Shirwandkar,et al.  Extractive Text Summarization Using Deep Learning , 2018, 2018 Fourth International Conference on Computing Communication Control and Automation (ICCUBEA).

[13]  Yuichi Motai,et al.  Respiratory Motion Estimation with Hybrid Implementation , 2014 .

[14]  Yuichi Motai,et al.  Customized Prediction of Respiratory Motion , 2014 .

[15]  Sebastian Ruder,et al.  An overview of gradient descent optimization algorithms , 2016, Vestnik komp'iuternykh i informatsionnykh tekhnologii.

[16]  Mohammad Hassan Moradi,et al.  Mortality prediction in intensive care units (ICUs) using a deep rule-based fuzzy classifier , 2018, J. Biomed. Informatics.

[17]  U. F. Eze,et al.  Fuzzy Logic Model for Traffic Congestion , 2014 .

[18]  Hayat Al-Dmour,et al.  A clustering fusion technique for MR brain tissue segmentation , 2018, Neurocomputing.

[19]  Yuichi Motai,et al.  Intra- and Inter-Fractional Variation Prediction of Lung Tumors Using Fuzzy Deep Learning , 2016, IEEE Journal of Translational Engineering in Health and Medicine.

[20]  Qionghai Dai,et al.  Visual Words Assignment Via Information-Theoretic Manifold Embedding , 2014, IEEE Transactions on Cybernetics.

[21]  Geoffrey E. Hinton,et al.  Deep Boltzmann Machines , 2009, AISTATS.

[22]  Iddo Greental,et al.  Genetic algorithms for evolving deep neural networks , 2014, GECCO.

[23]  Erich Peter Klement,et al.  Fuzzy probability measures , 1981 .

[24]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[25]  Yongdong Zhang,et al.  A Highly Parallel Framework for HEVC Coding Unit Partitioning Tree Decision on Many-core Processors , 2014, IEEE Signal Processing Letters.

[26]  Hisao Ishibuchi,et al.  Learning by fuzzified neural networks , 1995, Int. J. Approx. Reason..

[27]  Yu-Jun Zheng,et al.  Airline Passenger Profiling Based on Fuzzy Deep Machine Learning , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[28]  Chih-Hong Lin,et al.  Self-constructing fuzzy neural network speed controller for permanent-magnet synchronous motor drive , 2001, IEEE Trans. Fuzzy Syst..

[29]  Shinji Watanabe,et al.  Structure discovery of deep neural network based on evolutionary algorithms , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[30]  Yaochu Jin,et al.  Fuzzy Systems in Bioinformatics and Computational Biology , 2009, Fuzzy Systems in Bioinformatics and Computational Biology.

[31]  Nicolas Le Roux,et al.  Representational Power of Restricted Boltzmann Machines and Deep Belief Networks , 2008, Neural Computation.

[32]  Qinghua Hu,et al.  Hybrid attribute reduction based on a novel fuzzy-rough model and information granulation , 2007, Pattern Recognit..

[33]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[34]  Minqiang Li,et al.  Learning Subspace-Based RBFNN Using Coevolutionary Algorithm for Complex Classification Tasks , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[35]  H. Robbins A Stochastic Approximation Method , 1951 .

[36]  Nishchal K. Verma,et al.  Developing deep fuzzy network with Takagi Sugeno fuzzy inference system , 2017, 2017 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE).

[37]  Craig B. Borkowf,et al.  Time-Series Forecasting , 2002, Technometrics.

[38]  Yuichi Motai,et al.  Customized prediction of respiratory motion with clustering from multiple patient interaction , 2013, ACM Trans. Intell. Syst. Technol..

[39]  Michio Sugeno,et al.  Fuzzy identification of systems and its applications to modeling and control , 1985, IEEE Transactions on Systems, Man, and Cybernetics.

[40]  Ning Qian,et al.  On the momentum term in gradient descent learning algorithms , 1999, Neural Networks.

[41]  Vladimir G Ivancevic,et al.  Neuro-Fuzzy Associative Machinery for Comprehensive Brain and Cognition Modelling (Studies in Computational Intelligence) , 2007 .

[42]  Danuta Rutkowska,et al.  Neuro-Fuzzy Architectures and Hybrid Learning , 2002, Studies in Fuzziness and Soft Computing.

[43]  Lotfi A. Zadeh,et al.  Toward a theory of fuzzy information granulation and its centrality in human reasoning and fuzzy logic , 1997, Fuzzy Sets Syst..

[44]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[45]  J. Buckley,et al.  Fuzzy neural networks: a survey , 1994 .

[46]  Ren Zhang,et al.  A model with Fuzzy Granulation and Deep Belief Networks for exchange rate forecasting , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[47]  Meera Narvekar,et al.  Hybrid auto text summarization using deep neural network and fuzzy logic system , 2017, 2017 International Conference on Inventive Computing and Informatics (ICICI).

[48]  Li Fu,et al.  A novel fuzzy deep-learning approach to traffic flow prediction with uncertain spatial–temporal data features , 2018, Future Generation Computer Systems.

[49]  Jaouad Boumhidi,et al.  Fuzzy deep learning based urban traffic incident detection , 2017, Cognitive Systems Research.

[50]  Youyong Kong,et al.  Sparse Coding-Inspired Optimal Trading System for HFT Industry , 2015, IEEE Transactions on Industrial Informatics.

[51]  Shengyong Chen,et al.  Acceleration Strategies in Generalized Belief Propagation , 2012, IEEE Transactions on Industrial Informatics.

[52]  Minho Lee,et al.  A fuzzy convolutional neural network for text sentiment analysis , 2018, J. Intell. Fuzzy Syst..

[53]  Yixu Song,et al.  Real-time Tumor Tracking with Respiratory Motion Based on Short-term Prediction , 2017 .

[54]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning, 2006 , 2012 .

[55]  野崎 賢,et al.  Generating Fuzzy Rules from Numerical Data , 1995 .

[56]  Liping Fu,et al.  Development of a global road safety performance function using deep neural networks , 2017 .

[57]  Yi Shang,et al.  EvoAE -- A New Evolutionary Method for Training Autoencoders for Deep Learning Networks , 2015, 2015 IEEE 39th Annual Computer Software and Applications Conference.

[58]  Wei Li,et al.  Single and Multiple Object Tracking Using a Multi-Feature Joint Sparse Representation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[59]  Nitish Srivastava,et al.  Multimodal learning with deep Boltzmann machines , 2012, J. Mach. Learn. Res..

[60]  Jeffrey D Bradley,et al.  A semi-automatic method for peak and valley detection in free-breathing respiratory waveforms. , 2006, Medical physics.

[61]  Mohamed Chtourou,et al.  A Hybrid Training Algorithm for Feedforward Neural Networks , 2006, Neural Processing Letters.

[62]  Chin-Teng Lin,et al.  Support-vector-based fuzzy neural network for pattern classification , 2006, IEEE Transactions on Fuzzy Systems.

[63]  José M. N. Vieira,et al.  Neuro-Fuzzy Systems: A Survey , 2004 .

[64]  Yuichi Motai,et al.  Irregular Breathing Classification From Multiple Patient Datasets Using Neural Networks , 2012, IEEE Transactions on Information Technology in Biomedicine.

[65]  Hsien-I Lin,et al.  Human hand gesture recognition using a convolution neural network , 2014, 2014 IEEE International Conference on Automation Science and Engineering (CASE).

[66]  Matthew D. Zeiler ADADELTA: An Adaptive Learning Rate Method , 2012, ArXiv.

[67]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[68]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[69]  Debi Prosad Dogra,et al.  Surveillance scene representation and trajectory abnormality detection using aggregation of multiple concepts , 2018, Expert Syst. Appl..

[70]  Lotfi A. Zadeh,et al.  Fuzzy Algorithms , 1968, Inf. Control..

[71]  Yan Shi,et al.  Developing fast predictors for large-scale time series using fuzzy granular support vector machines , 2013, Appl. Soft Comput..

[72]  Kunikazu Kobayashi,et al.  Time series forecasting using a deep belief network with restricted Boltzmann machines , 2014, Neurocomputing.

[73]  Qingcai Chen,et al.  Fuzzy deep belief networks for semi-supervised sentiment classification , 2014, Neurocomputing.

[74]  I Ketut Eddy Purnama,et al.  Vehicle License Plate Image Segmentation System Using Cellular Neural Network Optimized by Adaptive Fuzzy and Neuro-Fuzzy Algorithms , 2016, MUE 2016.

[75]  X. Li,et al.  Application of a new Restricted Boltzmann Machine to Radar Target Recognition , 2016, 2016 Progress in Electromagnetic Research Symposium (PIERS).

[76]  Martin J Murphy,et al.  Optimization of an adaptive neural network to predict breathing. , 2008, Medical physics.

[77]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[78]  Daphna Weinshall,et al.  On The Power of Curriculum Learning in Training Deep Networks , 2019, ICML.

[79]  Juhan Nam,et al.  Multimodal Deep Learning , 2011, ICML.

[80]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[81]  Yuanjie Zheng,et al.  An evolving recurrent interval type-2 intuitionistic fuzzy neural network for online learning and time series prediction , 2019, Appl. Soft Comput..

[82]  Licheng Jiao,et al.  A Semi-Supervised CNN With Fuzzy Rough C-Mean for Image Classification , 2019, IEEE Access.

[83]  Cor J. Veenman,et al.  Kernel Codebooks for Scene Categorization , 2008, ECCV.

[84]  Marius George Linguraru,et al.  Automatic kidney segmentation in 3D pediatric ultrasound images using deep neural networks and weighted fuzzy active shape model , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[85]  João Paulo Papa,et al.  Fine-tuning Deep Belief Networks using Harmony Search , 2016, Appl. Soft Comput..

[86]  Surya Ganguli,et al.  Identifying and attacking the saddle point problem in high-dimensional non-convex optimization , 2014, NIPS.

[87]  Paul J Keall,et al.  Measuring interfraction and intrafraction lung function changes during radiation therapy using four-dimensional cone beam CT ventilation imaging. , 2015, Medical physics.

[88]  Marco Tomassini,et al.  Soft computing - integrating evolutionary, neural, and fuzzy systems , 2001 .

[89]  Ronald R. Yager,et al.  Pythagorean fuzzy subsets , 2013, 2013 Joint IFSA World Congress and NAFIPS Annual Meeting (IFSA/NAFIPS).

[90]  Navneet Walia,et al.  ANFIS: Adaptive Neuro-Fuzzy Inference System- A Survey , 2015 .

[91]  Hisao Ishibuchi,et al.  Fuzzy neural networks with fuzzy weights and fuzzy biases , 1993, IEEE International Conference on Neural Networks.

[92]  Youyong Kong,et al.  Deep Direct Reinforcement Learning for Financial Signal Representation and Trading , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[93]  Yie-Chien Chen,et al.  A model reference control structure using a fuzzy neural network , 1995 .

[94]  Zeshui Xu,et al.  Extension of TOPSIS to Multiple Criteria Decision Making with Pythagorean Fuzzy Sets , 2014, Int. J. Intell. Syst..

[95]  Shen Furao,et al.  Forecasting exchange rate with deep belief networks , 2011, The 2011 International Joint Conference on Neural Networks.

[96]  Chen Hu,et al.  Recognizing human activity in smart home using deep learning algorithm , 2014, Proceedings of the 33rd Chinese Control Conference.

[97]  Shui Yu,et al.  Network Traffic Prediction Based on Deep Belief Network in Wireless Mesh Backbone Networks , 2017, 2017 IEEE Wireless Communications and Networking Conference (WCNC).

[98]  Xin Yao,et al.  A review of evolutionary artificial neural networks , 1993, Int. J. Intell. Syst..

[99]  Hon Keung Kwan,et al.  A fuzzy neural network and its application to pattern recognition , 1994, IEEE Trans. Fuzzy Syst..

[100]  Zhendong Wu,et al.  Damaged fingerprint classification by Deep Learning with fuzzy feature points , 2016, 2016 9th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI).

[101]  Dan Simon,et al.  Biogeography-Based Optimization , 2022 .

[102]  Zeshui Xu,et al.  Pythagorean fuzzy TODIM approach to multi-criteria decision making , 2016, Appl. Soft Comput..

[103]  Radhe Mohan,et al.  Initial evaluation of treatment-related pneumonitis in advanced-stage non-small-cell lung cancer patients treated with concurrent chemotherapy and intensity-modulated radiotherapy. , 2007, International journal of radiation oncology, biology, physics.

[104]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[105]  Krassimir T. Atanassov,et al.  Intuitionistic fuzzy sets , 1986 .

[106]  Yu-Jun Zheng,et al.  A Pythagorean-Type Fuzzy Deep Denoising Autoencoder for Industrial Accident Early Warning , 2017, IEEE Transactions on Fuzzy Systems.

[107]  Pankaj Gupta,et al.  Fuzzy Chance-Constrained Multiobjective Portfolio Selection Model , 2014, IEEE Transactions on Fuzzy Systems.

[108]  Zhenhong Du,et al.  Deep-Learning-Based Approach for Prediction of Algal Blooms , 2016 .

[109]  R Mohan,et al.  Predicting respiratory motion for four-dimensional radiotherapy. , 2004, Medical physics.

[110]  Joel J. P. C. Rodrigues,et al.  Evolutionary radial basis function network for gestational diabetes data analytics , 2017, J. Comput. Sci..

[111]  F. S. Wong,et al.  Fuzzy Neural Systems for Stock Selection , 1992 .

[112]  Y. Nesterov A method for unconstrained convex minimization problem with the rate of convergence o(1/k^2) , 1983 .

[113]  Quoc V. Le,et al.  Adding Gradient Noise Improves Learning for Very Deep Networks , 2015, ArXiv.

[114]  C. L. Philip Chen,et al.  Fuzzy Restricted Boltzmann Machine for the Enhancement of Deep Learning , 2015, IEEE Transactions on Fuzzy Systems.

[115]  Jordi Casas,et al.  Unsupervised Incident Detection Model in Urban and Freeway Networks , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[116]  Yiyu Yao,et al.  Information granulation for web-based information support systems , 2003, SPIE Defense + Commercial Sensing.

[117]  Elisabeth Weiss,et al.  Respiratory motion variability of primary tumors and lymph nodes during radiotherapy of locally advanced non-small-cell lung cancers , 2015, Radiation oncology.

[118]  Edward T. Lee,et al.  Fuzzy Neural Networks , 1975 .

[119]  Hugo Larochelle,et al.  Efficient Learning of Deep Boltzmann Machines , 2010, AISTATS.

[120]  Tapani Raiko,et al.  Improved Learning of Gaussian-Bernoulli Restricted Boltzmann Machines , 2011, ICANN.

[121]  Yoshua Bengio,et al.  Greedy Layer-Wise Training of Deep Networks , 2006, NIPS.

[122]  Philip M. Wolfe,et al.  The amalgamation of neural networks and fuzzy logic systems—a survey , 1995 .

[123]  Geoffrey E. Hinton,et al.  Deep Belief Networks for phone recognition , 2009 .

[124]  O. C. Akinyokun,et al.  Fingerprint Singular Point Detection Based on Modified Poincare Index Method , 2014 .

[125]  Atsuto Maki,et al.  A systematic study of the class imbalance problem in convolutional neural networks , 2017, Neural Networks.

[126]  Paul Smolensky,et al.  Information processing in dynamical systems: foundations of harmony theory , 1986 .

[127]  Sushmita Mitra,et al.  Neuro-fuzzy rule generation: survey in soft computing framework , 2000, IEEE Trans. Neural Networks Learn. Syst..