Hysteresis in photoelectric charging of dust particles in a complex plasma