AMUSE-VIRGO. II. DOWN-SIZING IN BLACK HOLE ACCRETION