Sparse Combinatorial Autoencoders

Recent research has shown that employing unsupervised pretraining often produces well-conditioned neural network initializations that lead to better local optima attained during training. One commonly used pretraining method involves hierarchically stacking sparse autoencoders (SAs) and learning the network parameters layer by layer using unlabeled data. Large network sizes and the amount of data required to properly pretrain a deep network make pretraining computationally intensive and the training bottleneck. To alleviate this problem, we propose a novel warm-start procedure for the SA capable of rapidly initializing large SAs in parameter regions yielding fast convergence to good local optima. At the heart of our approach lies the sparse combinatorial autoencoder (SCA), a novel method to regularize neural networks that allows us to train an SA withH features in O( √ H) time. We present a comprehensive series of experiments demonstrating the effectiveness of the warm-start procedure, called fast initialization with SCAs (FISCA), on the STL-10 and the MNIST datasets. Our experiments consider untied sigmoid and tied soft-rectified SAs of various sizes and demonstrate that FISCA ultimately yields significantly reduced training times compared to widely prevalent initialization techniques. For example, on the MNIST dataset, FISCAinitialized soft-rectified SAs with 10K hidden neurons converge over 20× faster to notably better local optima than SAs initialized with alternate methods.