A deep contractive autoencoder for solving multiclass classification problems

Contractive auto encoder (CAE) is on of the most robust variant of standard Auto Encoder (AE). The major drawback associated with the conventional CAE is its higher reconstruction error during encoding and decoding process of input features to the network. This drawback in the operational procedure of CAE leads to its incapability of going into finer details present in the input features by missing the information worth consideration. Resultantly, the features extracted by CAE lack the true representation of all the input features and the classifier fails in solving classification problems efficiently. In this work, an improved variant of CAE is proposed based on layered architecture following feed forward mechanism named as deep CAE. In the proposed architecture, the normal CAEs are arranged in layers and inside each layer, the process of encoding and decoding take place. The features obtained from the previous CAE are given as inputs to the next CAE. Each CAE in all layers are responsible for reducing the reconstruction error thus resulting in obtaining the informative features. The feature set obtained from the last CAE is given as input to the softmax classifier for classification. The performance and efficiency of the proposed model has been tested on five MNIST variant-datasets. The results have been compared with standard SAE, DAE, RBM, SCAE, ScatNet and PCANet in term of training error, testing error and execution time. The results revealed that the proposed model outperform the aforementioned models.

[1]  Lin Sun,et al.  Laplacian Auto-Encoders: An explicit learning of nonlinear data manifold , 2015, Neurocomputing.

[2]  François Chollet,et al.  Keras: The Python Deep Learning library , 2018 .

[3]  Rozaida Ghazali,et al.  An Improved Back Propagation Neural Network Algorithm on Classification Problems , 2010, FGIT-DTA/BSBT.

[4]  Jiwen Lu,et al.  PCANet: A Simple Deep Learning Baseline for Image Classification? , 2014, IEEE Transactions on Image Processing.

[5]  Rozaida Ghazali,et al.  An Improved Hybrid Firefly Algorithm for Solving Optimization Problems , 2018, SCDM.

[6]  Abdul Salam Shah,et al.  Statistical Features Based Approach (SFBA) for Hourly Energy Consumption Prediction Using Neural Network , 2017 .

[7]  Pascal Vincent,et al.  Contractive Auto-Encoders: Explicit Invariance During Feature Extraction , 2011, ICML.

[8]  Richa Singh,et al.  Group sparse autoencoder , 2017, Image Vis. Comput..

[9]  Keun-Chang Kwak,et al.  Design of Ensemble Stacked Auto-Encoder for Classification of Horse Gaits with MEMS Inertial Sensor Technology , 2018, Micromachines.

[10]  Fatos Xhafa,et al.  Geometrical and topological approaches to Big Data , 2017, Future Gener. Comput. Syst..

[11]  Nazri Mohd Nawi,et al.  Auto-Encoder Variants for Solving Handwritten Digits Classification Problem , 2020, Int. J. Fuzzy Log. Intell. Syst..

[12]  Tongwei Lu,et al.  Character Recognition Based on PCANet , 2016, 2016 15th International Symposium on Parallel and Distributed Computing (ISPDC).

[13]  Zhaohui Wu,et al.  Robust feature learning by stacked autoencoder with maximum correntropy criterion , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[14]  Jacek M. Zurada,et al.  Learning Understandable Neural Networks With Nonnegative Weight Constraints , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[15]  Zhongzhi Shi,et al.  Denoising Laplacian multi-layer extreme learning machine , 2016, Neurocomputing.

[16]  Xiaoqing Feng,et al.  Multimodal video classification with stacked contractive autoencoders , 2016, Signal Process..

[17]  Mary Inaba,et al.  Bayesian AutoEncoder: Generation of Bayesian Networks with Hidden Nodes for Features , 2016, AAAI.

[18]  Jochen J. Steil,et al.  Efficient online learning of a non-negative sparse autoencoder , 2010, ESANN.

[19]  Nazri Mohd Nawi,et al.  An Optimized Back Propagation Learning Algorithm with Adaptive Learning Rate , 2017 .

[20]  Fazli Wahid,et al.  An Efficient Artificial Intelligence Hybrid Approach for Energy Management in Intelligent Buildings , 2019, KSII Trans. Internet Inf. Syst..

[21]  Fahimeh Biglari,et al.  Limited memory BFGS method based on a high-order tensor model , 2015, Comput. Optim. Appl..

[22]  Chunyan Miao,et al.  Online multimodal deep similarity learning with application to image retrieval , 2013, ACM Multimedia.

[23]  Shiguang Shan,et al.  Deeply Coupled Auto-encoder Networks for Cross-view Classification , 2014, ArXiv.

[24]  Klaus-Robert Müller,et al.  Wasserstein Training of Restricted Boltzmann Machines , 2016, NIPS.

[25]  A. Izenman.  Linear Discriminant Analysis , 2013 .

[26]  Yoshua Bengio,et al.  An empirical evaluation of deep architectures on problems with many factors of variation , 2007, ICML '07.

[27]  Meng Wang,et al.  Multimodal Deep Autoencoder for Human Pose Recovery , 2015, IEEE Transactions on Image Processing.

[28]  Jeevan Kanesan,et al.  PCANet-Based Convolutional Neural Network Architecture for a Vehicle Model Recognition System , 2019, IEEE Transactions on Intelligent Transportation Systems.

[29]  Ruifang Liu,et al.  Stacked denoising autoencoder and dropout together to prevent overfitting in deep neural network , 2015, 2015 8th International Congress on Image and Signal Processing (CISP).

[30]  Rozaida Ghazali,et al.  Hybrid of firefly algorithm and pattern search for solving optimization problems , 2018, Evol. Intell..

[31]  Jane You,et al.  HSAE: A Hessian regularized sparse auto-encoders , 2016, Neurocomputing.

[32]  Pascal Vincent,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010, J. Mach. Learn. Res..

[33]  Jacek M. Zurada,et al.  Deep Learning of Part-Based Representation of Data Using Sparse Autoencoders With Nonnegativity Constraints , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[34]  Jun Miao,et al.  Hierarchical Extreme Learning Machine for unsupervised representation learning , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[35]  Zhiguo Jiang,et al.  A Cloud Detection Method for Landsat 8 Images Based on PCANet , 2018, Remote. Sens..

[36]  Pascal Vincent,et al.  Higher Order Contractive Auto-Encoder , 2011, ECML/PKDD.

[37]  Sung Bum Pan,et al.  An EigenECG Network Approach Based on PCANet for Personal Identification from ECG Signal , 2018, Sensors.

[38]  Wenjing Jin,et al.  Enhanced Restricted Boltzmann Machine With Prognosability Regularization for Prognostics and Health Assessment , 2016, IEEE Transactions on Industrial Electronics.