MarkovHC: Markov hierarchical clustering for the topological structure of high-dimensional single-cell omics data

Distinguishing cell types and cell states is one of the fundamental questions in single-cell studies. Meanwhile, exploring the lineage relations among cells and finding the path and critical points in the cell fate transition are also of great importance. Existing unsupervised clustering methods and lineage trajectory reconstruction methods often face several challenges such as clustering data of arbitrary shapes, tracking precise trajectories and identifying critical points. Certain adaptive landscape approach1–3, which constructs a pseudo-energy landscape of the dynamical system, may be used to explore such problems. Thus, we propose Markov hierarchical clustering algorithm (MarkovHC), which reconstructs multi-scale pseudo-energy landscape by exploiting underlying metastability structure in an exponentially perturbed Markov chain4. A Markov process describes the random walk of a hypothetically traveling cell in the corresponding pseudo-energy landscape over possible gene expression states. Technically, MarkovHC integrates the tasks of cell classification, trajectory reconstruction, and critical point identification in a single theoretical framework consistent with topological data analysis (TDA)5. In addition to the algorithm development and simulation tests, we also applied MarkovHC to diverse types of real biological data: single-cell RNA-Seq data, cytometry data, and single-cell ATAC-Seq data. Remarkably, when applying to single-cell RNA-Seq data of human ESC derived progenitor cells6, MarkovHC not only could successfully identify known cell types, but also discover new cell types and stages. In addition, when using MarkovHC to analyze single-cell RNA-Seq data of human preimplantation embryos in early development7, the hierarchical structure of the lineage trajectories was faithfully reconstituted. Furthermore, the critical points representing important stage transitions had also been identified by MarkovHC from early gastric cancer data8. In summary, these results demonstrate that MarkovHC is a powerful tool based on rigorous metastability theory to explore hierarchical structures of biological data, to identify a cell sub-population (basin) and a critical point (stage transition), and to track a lineage trajectory (differentiation path). Highlights MarkovHC explores the topology hierarchy in high-dimensional data. MarkovHC can find clusters (basins) and cores (attractors) of clusters in different scales. The trajectory of state transition (transition paths) and critical points in the process of state transition (critical points) among clusters can be tracked. MarkovHC can be applied on diverse types of single-cell omics data.