Speaker Extraction Using Stacked BLSTM Optimized with Frequency-domain Differentiated Spectrum Loss

We propose a novel speaker extraction method that extracts the target speaker from the mixture of the target speaker and an unknown interferer in a semi-supervised scheme. The dynamic nature of the speech signal urges the use of Recurrent Neural Networks (RNN). For this work, we use two stacked layers of Bidirectional Long Short-Term Memory (BLSTM) to account for the continuity between the consecutive speech frames. Moreover, to preserve the spectrum changes of the target speaker in the frequency domain, we propose a loss function that differentiates between the neighboring frequency bins in a frame. This loss function is a weighted sum of the frequency-dependent loss function and Mean Squared Error (MSE). The evaluation results on the Speech Separation Challenge (SSC) dataset show that the proposed approach outperforms the baseline method, which uses a Deep Neural Network (DNN), in terms of three measurement scores such as Speech to Distortion Ratio (SDR), Perceptual Evaluation of Speech Quality (PESQ) and Short-Time Objective Intelligibility (STOI). Also, By comparing the aforementioned evaluation metrics from two similar networks, where one is optimized by the proposed loss function and the other optimized by MSE, we show that the new loss function provides better results for speaker extraction applications.