The downplayed role of secular processes in the co-evolution of galaxies and black holes