Interaction-driven topological phase diagram of twisted bilayer MoTe$_2$

Twisted bilayer MoTe$_2$ is a promising platform to investigate the interplay between topology and many-body interaction. We present a theoretical study of its interaction-driven quantum phase diagrams based on a three-orbital model, which can be viewed as a generalization of the Kane-Mele-Hubbard model with an additional orbital and realistic Coulomb repulsion. We predict a cascade of phase transitions tuned by the twist angle $\theta$. At the hole filling factor $\nu=1$ (one hole per moir\'e unit cell), the ground state can be in the multiferroic phase with coexisting spontaneous layer polarization and magnetism, the quantum anomalous Hall phase, and finally the topologically trivial magnetic phases, as $\theta$ increases from $1.5^{\circ}$ to $5^{\circ}$. At $\nu=2$, the ground state can have a second-order phase transition between an antiferromagnetic phase and the quantum spin Hall phase as $\theta$ passes through a critical value. The dependence of the phase boundaries on model parameters such as the gate-to-sample distance, the dielectric constant, and the moir\'e potential amplitude is examined. The predicted phase diagrams can guide the search for topological phases in twisted transition metal dichalcogenide homobilayers.