Spectral form factors of topological phases