Class sparsity signature based Restricted Boltzmann Machine

Abstract Restricted Boltzmann Machines (RBMs) have been extensively utilized in machine learning as core units in constructing deep learning architectures such as Deep Boltzmann Machines (DBMs) and Deep Belief Networks (DBNs). However, they are prone to overfitting and several regularization techniques have been proposed to mitigate this effect. In this paper, we propose the semi-supervised class sparsity signature based RBM formulation by combining unsupervised generative training of the RBM with a supervised sparsity regularizer. The proposed approach, termed as cssRBM, enforces sparsity at the class level to ensure that coherent and discriminative representations are learnt during training. Combining unsupervised learning with supervised learning allows the model to utilize external training data to learn better generative features while the supervised learning enables fine-tuning for discrimination using the learned features. We construct both DBMs and DBNs with cssRBM units and evaluate the performance on multiple publicly available benchmark datasets. Experiments on the MNIST and CIFAR-10 databases demonstrate that the proposed approaches are comparable with state-of-the-art deep learning architectures in the literature. We also evaluate the performance on one of the most challenging face databases, i.e., the Point and Shoot Challenge dataset. The results show that the proposed approaches improve state-of-the-art results by 15% on the PaSC database.