Cell Subclass Identification in Single-Cell RNA-Sequencing Data Using Orthogonal Nonnegative Matrix Factorization

Identification of cell subclasses using single-cell RNA-Sequencing (scRNA-Seq) data is of paramount importance since it uncovers the hidden biological processes within the cell population. While the nonnegative matrix factorization (NMF) model has been reported to be effective in various unsupervised clustering tasks, it may still produce inappropriate results for some scRNA-Seq datasets with heterogeneous structures. In this paper, we propose the use of an orthogonally constrained NMF (ONMF) model for the subclass identification problem of scRNA-Seq datasets. The ONMF model in general can provide improved clustering performance, but is challenging to solve. We present a computationally efficient algorithm based on optimization techniques of variable splitting and alternating direction method of multipliers (ADMM). Through two scRNA-Seq datasets, we show that the proposed method can yield promising performance in identifying cell subclasses and detecting key genes over the existing methods. Moreover, the key genes identified by the proposed method are shown biologically significant via the gene set enrichment analysis.