Aiming at the problem of that it is impossible to extract the amplitude of the sinusoidal response signal submerged in the strong noise to analyze the winding deformation fault by online frequency response method for detection of the transformer winding distortion, this paper proposes a new filtering algorithm based on the statistical eigenvalues of the correlation function of random signals, and the target sinusoidal signal is successfully extracted from the noisy signal. The implementation methods in time domain and frequency domain of the algorithm are respectively introduced, and the detection effects under the high-order harmonic and Gaussian white noise interference are simulated by MATLAB. Finally, the field measured data of the transformer winding deformation test is simulated by the online frequency response method. The results validates that the method can extract the sinusoidal signal with very low SNR(Signal to Noise Ratio) under strong noise background.