Insights into non-axisymmetric instabilities in three-dimensional rotating supernova models with neutrino and gravitational-wave signatures