The Scale Dependence of Mass Assembly in Galaxies