Fractional Brownian motion and the halo mass function