Irreducibility of the Gorenstein loci of Hilbert schemes via ray families