Experimental investigation of three-dimensional flow instabilities in a rotating lid-driven cavity