Sparse Multilayer Autoencoders Based on the Small World Network

Unlike most of manifold learning algorithms, multilayer autoencoders construct a bi-directional mapping between the original data and the low dimensional representation. However, the time complexity of multilayer autoencoders is much higher since the neurons between the adjacent layers are fully connected in multilayer autoencoders. According to the small world network model, we build sparse multilayer autoencoders. To obtain a sparse network, the connections between the nodes in some layer and the nodes in their adjacent upper layer are firstly removed with a probability p. Then the connections are established between the nodes deleted the connections and the nodes in higher layer. The higher layer connected is selected with the method called limited random numbers. Experiments demonstrate that the reconstruction quality and the recognition accuracy from the sparse network are very near to those from the original network. However, the executing efficiency of the sparse network is improved.

[1]  Yoshua Bengio,et al.  Non-Local Manifold Tangent Learning , 2004, NIPS.

[2]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[3]  Michel Verleysen,et al.  Curvilinear Distance Analysis versus Isomap , 2002, ESANN.

[4]  Yuancheng Li,et al.  An Intrusion Detection Approach using Improved Manifold learning and BVM , 2012 .

[5]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[6]  Ben J. A. Kröse,et al.  Fast nonlinear dimensionality reduction with topology representing networks , 2002, ESANN.

[7]  Hongyuan Zha,et al.  Isometric Embedding and Continuum ISOMAP , 2003, ICML.

[8]  Mikhail Belkin,et al.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation , 2003, Neural Computation.

[9]  B. Nadler,et al.  Diffusion maps, spectral clustering and reaction coordinates of dynamical systems , 2005, math/0503445.

[10]  Kilian Q. Weinberger,et al.  Learning a kernel matrix for nonlinear dimensionality reduction , 2004, ICML.

[11]  Qian He,et al.  Research of High Dimensional Data Corresponding Based on Semi Supervised Manifold Learning , 2011 .

[12]  Robert Pless,et al.  A Survey of Manifold Learning for Images , 2009, IPSJ Trans. Comput. Vis. Appl..

[13]  Huafeng Xu,et al.  A self-organizing principle for learning nonlinear manifolds , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[14]  Eric O. Postma,et al.  Dimensionality Reduction: A Comparative Review , 2008 .

[15]  Matti Pietikäinen,et al.  Classification of handwritten digits using supervised locally linear embedding algorithm and support vector machine , 2003, ESANN.

[16]  Joshua B. Tenenbaum,et al.  Global Versus Local Methods in Nonlinear Dimensionality Reduction , 2002, NIPS.

[17]  Xiaofei He,et al.  Locality Preserving Projections , 2003, NIPS.

[18]  Jiao Wang,et al.  Auto-Associative Neural Network System for Recognition , 2007, 2007 International Conference on Machine Learning and Cybernetics.

[19]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[20]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[21]  H. Hotelling.  Analysis of a complex of statistical variables into principal components. , 1933 .

[22]  Geoffrey E. Hinton,et al.  Stochastic Neighbor Embedding , 2002, NIPS.

[23]  Duncan J. Watts,et al.  Collective dynamics of ‘small-world’ networks , 1998, Nature.

[24]  Hongyuan Zha,et al.  Adaptive Manifold Learning , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Bernhard Schölkopf,et al.  A kernel view of the dimensionality reduction of manifolds , 2004, ICML.

[26]  H. Zha,et al.  Principal manifolds and nonlinear dimensionality reduction via tangent space alignment , 2004, SIAM J. Sci. Comput..

[27]  J. Kruskal.  Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis , 1964 .

[28]  Li Yang.  Sammon's nonlinear mapping using geodesic distances , 2004, ICPR 2004.

[29]  H. Zha,et al.  Local smoothing for manifold learning , 2004, CVPR 2004.

[30]  Junhua Gu,et al.  A Fast Manifold Learning Algorithm , 2012 .

[31]  Anil K. Jain,et al.  Nonlinear Manifold Learning for Data Stream , 2004, SDM.