A New Fast Training Algorithm for Autoencoder Neural Networks based on Extreme Learning Machine

Autoencoders are neural networks that are characterized by having the same inputs and outputs. This kind of Neural Networks aim to estimate a nonlinear transformation whose parameters allow to represent the input patterns to the network. The Extreme Learning Machine (ELM-AE) Autoencoders have random weights and biases in the hidden layer, and compute the output parameters by solving an overdetermined linear system using the Moore-Penrose Pseudoinverse. ELM-AE training is based on the Fast Iterative Shrinkage-Thresholding (FISTA). In this paper, we propose to improve the convergence speed obtained by FISTA considering the use of two algorithms of the Shrinkage-Thresholding class, namely Greedy FISTA and Linearly-Convergent FISTA. 6 frequently used public machine learning datasets were considered: MNIST, NORB, CIFAR10, UMist, Caltech256, Stanford Cars. Experiments were carried out varying the number of neurons in the hidden layer of the Autoencoders, considering the 3 algorithms, for all the databases. The experimental results showed that Greedy FISTA and Linearly-Convergent FISTA presented higher convergence speed, increasing the speed of ELM-Autoencoder training, maintaining a comparable generalization error between the three Shrinkage-Thresholding algorithms.

[1]  Ping Guo,et al.  DensePILAE: a feature reuse pseudoinverse learning algorithm for deep stacked autoencoder , 2021, Complex & Intelligent Systems.

[2]  Tiejun Yang,et al.  Application of the improved fast iterative shrinkage-thresholding algorithms in sound source localization , 2021 .

[3]  Shuzhong Zhang,et al.  Linearly-Convergent FISTA Variant for Composite Optimization with Duality , 2021, Journal of Scientific Computing.

[4]  Peizhi Yan,et al.  Non-iterative online sequential learning strategy for autoencoder and classifier , 2021, Neural Computing and Applications.

[5]  Rui Li,et al.  Hierarchical extreme learning machine with L21-norm loss and regularization , 2020, International Journal of Machine Learning and Cybernetics.

[6]  Derya Soydaner,et al.  Hyper Autoencoders , 2020, Neural Processing Letters.

[7]  Fjodor van Veen,et al.  The Neural Network Zoo , 2020, Proceedings.

[8]  Guang-Bin Huang,et al.  Unsupervised feature selection based extreme learning machine for clustering , 2020, Neurocomputing.

[9]  Jun Liang,et al.  Discriminant autoencoder for feature extraction in fault diagnosis , 2019, Chemometrics and Intelligent Laboratory Systems.

[10]  Rui Li,et al.  Representation learning by hierarchical ELM auto-encoder with double random hidden layers , 2019, IET Comput. Vis..

[11]  Yao Yao,et al.  Stacked Denoising Extreme Learning Machine Autoencoder Based on Graph Embedding for Feature Representation , 2019, IEEE Access.

[12]  Paul Honeine,et al.  Correntropy-based robust multilayer extreme learning machines , 2018, Pattern Recognition.

[13]  Yurii Nesterov,et al.  Lectures on Convex Optimization , 2018 .

[14]  Tao Luo,et al.  Improving "Fast Iterative Shrinkage-Thresholding Algorithm": Faster, Smarter, and Greedier , 2018, SIAM J. Sci. Comput..

[15]  Chi-Man Vong,et al.  Empirical kernel map-based multilayer extreme learning machines for representation learning , 2018, Neurocomputing.

[16]  Nan Zhang,et al.  Research of stacked denoising sparse autoencoder , 2018, Neural computing & applications (Print).

[17]  Hongwei Liu,et al.  A Review of the Autoencoder and Its Variants: A Comparative Perspective from Target Recognition in Synthetic-Aperture Radar Images , 2018, IEEE Geoscience and Remote Sensing Magazine.

[18]  Jiuwen Cao,et al.  Kernel-Based Multilayer Extreme Learning Machines for Representation Learning , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[19]  Chunxia Zhang,et al.  Generalized extreme learning machine autoencoder and a new deep neural network , 2017, Neurocomputing.

[20]  Pierre Baldi,et al.  Learning in the machine: Random backpropagation and the deep learning channel , 2016, Artif. Intell..

[21]  Yu Xue,et al.  Research on denoising sparse autoencoder , 2016, International Journal of Machine Learning and Cybernetics.

[22]  Guang-Bin Huang,et al.  Extreme Learning Machine for Multilayer Perceptron , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[23]  Giuseppe Durisi,et al.  Quantized Massive MU-MIMO-OFDM Uplink , 2015, IEEE Transactions on Communications.

[24]  Guang-Bin Huang,et al.  Trends in extreme learning machines: A review , 2015, Neural Networks.

[25]  Richard G. Baraniuk,et al.  A Field Guide to Forward-Backward Splitting with a FASTA Implementation , 2014, ArXiv.

[26]  Cheng Wu,et al.  Semi-Supervised and Unsupervised Extreme Learning Machines , 2014, IEEE Transactions on Cybernetics.

[27]  Danwei Wang,et al.  Sparse Extreme Learning Machine for Classification , 2014, IEEE Transactions on Cybernetics.

[28]  Hongming Zhou,et al.  Extreme Learning Machine for Regression and Multiclass Classification , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[29]  Hujun Bao,et al.  A Variance Minimization Criterion to Feature Selection Using Laplacian Regularization , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[31]  Narasimhan Sundararajan,et al.  A Fast and Accurate Online Sequential Learning Algorithm for Feedforward Networks , 2006, IEEE Transactions on Neural Networks.

[32]  Shunming Li,et al.  A Novel Symmetric Stacked Autoencoder for Adversarial Domain Adaptation Under Variable Speed , 2022, IEEE Access.

[33]  Stanley Kok,et al.  Multi-layer Online Sequential Extreme Learning Machine for Image Classification , 2016 .

[34]  A. Beck,et al.  Gradient-based algorithms with applications to signal-recovery problems , 2010, Convex Optimization in Signal Processing and Communications.

[35]  Yonina C. Eldar,et al.  Convex Optimization in Signal Processing and Communications , 2009 .

[36]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..