Onset of Optical Instabilities in the Nonlinear Optical Transmission of Heliconical Cholesteric Liquid Crystals