Experimental studies of three-dimensional instability of boundary layers