Speaker adaptation using speaker-normalized DNN based on speaker codes

Recently, deep neural network (DNN) becomes one of the main streams of acoustic modeling for automatic speech recognition. Further, speaker adaptation techniques have been tested for DNN-based speech recognition, including one based on a framework of bias adaptation using speaker codes. This paper introduces speaker-normalized training to this framework and experimentally shows its effectiveness. In the conventional method using speaker codes, two kinds of networks of speaker-independent (SI) DNNs and subnetworks for speaker adaptation were trained sequentially. We expect that, by training the SI networks and the subnetworks simultaneously, this method can be tuned so that it can handle both SI information and speaker-dependent (SD) information more adequately. Further, different from the conventional method, the speaker code vector is generated through networks from a 1-of-N speaker representation. This will reduce the training cost of the SI models and the subnetworks and avoid the over-fitting problem. Experimental evaluations using the TIMIT database demonstrate that our proposed training method can reduce the phoneme error rate by 5.7% relative.