Fractional Brownian motion with random diffusivity: emerging residual nonergodicity below the correlation time