Autoencoder Node Saliency: Selecting Relevant Latent Representations

The autoencoder is an artificial neural network that learns hidden representations of unlabeled data. With a linear transfer function it is similar to the principal component analysis (PCA). While both methods use weight vectors for linear transformations, the autoencoder does not come with any indication similar to the eigenvalues in PCA that are paired with eigenvectors. We propose a novel supervised node saliency (SNS) method that ranks the hidden nodes, which contain weight vectors for transformations. SNS is able to indicate the nodes specialized in a learning task. The latent representations of a hidden node can be described using a one-dimensional histogram. We apply normalized entropy difference (NED) to measure the ”interestingness” of the histograms, and conclude a property for NED values to identify a good classifying node. By applying our methods to real datasets, we demonstrate their ability to find valuable nodes and explain the learned tasks in autoencoders.

[1]  Ethem Alpaydin,et al.  Unsupervised feature extraction with autoencoder trees , 2017, Neurocomputing.

[2]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[3]  Michael Brady,et al.  Saliency, Scale and Image Description , 2001, International Journal of Computer Vision.

[4]  Aleix Prat Aparicio.  Comprehensive molecular portraits of human breast tumours , 2012 .

[5]  Avanti Shrikumar,et al.  Learning Important Features Through Propagating Activation Differences , 2017, ICML.

[6]  Honglak Lee,et al.  Sparse deep belief net model for visual area V2 , 2007, NIPS.

[7]  Yoshua Bengio,et al.  What regularized auto-encoders learn from the data-generating distribution , 2012, J. Mach. Learn. Res..

[8]  Paul E. Utgoff,et al.  Many-Layered Learning , 2002, Neural Computation.

[9]  Thorsteinn Rögnvaldsson,et al.  Predicting Air Compressor Failures with Echo State Networks , 2016 .

[10]  Simon J. Doran,et al.  Stacked Autoencoders for Unsupervised Feature Learning and Multiple Organ Detection in a Pilot Study Using 4D Patient Data , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Andrew Zisserman,et al.  Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps , 2013, ICLR.

[12]  Pascal Vincent,et al.  A Connection Between Score Matching and Denoising Autoencoders , 2011, Neural Computation.

[13]  Ching Y. Suen,et al.  A novel hybrid CNN-SVM classifier for recognizing handwritten digits , 2012, Pattern Recognit..

[14]  Stefan Byttner,et al.  Consensus self-organized models for fault detection (COSMO) , 2011, Eng. Appl. Artif. Intell..

[15]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[16]  Geoffrey E. Hinton.  Connectionist Learning Procedures , 1989, Artif. Intell..

[17]  Kurt Hornik,et al.  Neural networks and principal component analysis: Learning from examples without local minima , 1989, Neural Networks.

[18]  Yoshua Bengio,et al.  Extracting and composing robust features with denoising autoencoders , 2008, ICML '08.

[19]  F. Markowetz,et al.  The genomic and transcriptomic architecture of 2,000 breast tumours reveals novel subgroups , 2012, Nature.

[20]  Gregory P. Way,et al.  Extracting a Biologically Relevant Latent Space from Cancer Transcriptomes with Variational Autoencoders , 2017, bioRxiv.

[21]  Norbert Jankowski,et al.  Survey of Neural Transfer Functions , 1999 .

[22]  Sung-Hyuk Cha.  Comprehensive Survey on Distance/Similarity Measures between Probability Density Functions , 2007 .

[23]  R. Giryes,et al.  Autoencoders , 2021, Deep Learning in Science.

[24]  Casey S. Greene,et al.  Unsupervised Feature Construction and Knowledge Extraction from Genome-Wide Assays of Breast Cancer with Denoising Autoencoders , 2014, Pacific Symposium on Biocomputing.

[25]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[26]  Ivan Laptev,et al.  Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[27]  Ching Y. Suen,et al.  A trainable feature extractor for handwritten digit recognition , 2007, Pattern Recognit..

[28]  Florian Metze,et al.  Extracting deep bottleneck features using stacked auto-encoders , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[29]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[30]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[31]  Martin A. Riedmiller,et al.  Deep auto-encoder neural networks in reinforcement learning , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[32]  Yoshua Bengio,et al.  How transferable are features in deep neural networks? , 2014, NIPS.

[33]  Arun Ross,et al.  On automated source selection for transfer learning in convolutional neural networks , 2018, Pattern Recognit..

[34]  Pascal Vincent,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010, J. Mach. Learn. Res..

[35]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[36]  Meng Wang,et al.  Multimodal Deep Autoencoder for Human Pose Recovery , 2015, IEEE Transactions on Image Processing.

[37]  Yoshua Bengio,et al.  Scaling learning algorithms towards AI , 2007 .

[38]  Chandrika Kamath,et al.  Salient points for tracking moving objects in video , 2005, IS&T/SPIE Electronic Imaging.

[39]  Jürgen Schmidhuber,et al.  Stacked Convolutional Auto-Encoders for Hierarchical Feature Extraction , 2011, ICANN.