Bidirectional Covariance Matrices: A Compact and Efficient Data Descriptor for Image Set Classification

Symmetric Positive Definite (SPD) matrices have been widely used in many computer vision tasks. Recently, there are growing interests in applying covariance matrices to image set classification due to their benefit of encoding image features as a data descriptor. Since SPD matrices follow a non-linear Riemannian geometry, exploiting an appropriate Riemannian metric is the key to successful classification. Adopting Riemannian metrics to classify covariance matrices of image sets is nontrivial, since such matrices are usually singular matrices. Besides, the computational complexity is intolerable while dealing with high dimensional covariance matrices. This paper proposes to use bidirectional covariance matrices instead of covariance matrices as a data descriptor. We model image sets both from the row and column directions of images and these bidirectional covariance matrices are proved to be compact and efficient. Improved accuracy and efficiency are obtained through experiments on standard datasets for comparing bidirectional covariance matrices with covariance matrices.