Exotic quantum phase transitions of strongly interacting topological insulators