Residual-Charge Induced Memory Effect of Electric Polarization in Multiferroic CuFe1-xGaxO2 as Seen via Polarized Neutron Diffraction