$L^2$-stability near equilibrium for the $4$ waves kinetic equation