Intrinsic limits on the detection of the anisotropies of the Stochastic Gravitational Wave Background