Self-pulsing instabilities in an optical parametric oscillator: Experimental observation and modeling of the mechanism