Rayleigh-Taylor instability in nonlinear optics