Interacting topological phases and modular invariance