Unsupervised adaptive sign language recognition based on hypothesis comparison guided cross validation and linguistic prior filtering

Abstract Signer adaptation is important for sign language recognition systems because a fixed system cannot perform well on all kinds of signers. In supervised signer adaptation, the labeled adaptation data must be collected explicitly. To skip the data collecting process in signer adaptation, we propose a novel unsupervised adaptation method, namely the hypothesis comparison guided cross validation method. The method not only addresses the problem of the overlap between the data set to be labeled and the data set for adaptation, but also employs an additional hypothesis comparison step to decrease the noise rate of the adaptation data set. We also utilize linguistic prior knowledge to down sample the adaptation data list to further decrease the noise rate. To evaluate the effectiveness of the proposed method, the CASIIE-SL-Database is formed, which is the first specialized data set for unsupervised signer adaptation to the best of our knowledge. Experimental results show that the proposed method can achieve relative word error rate reductions of 3.93% and 4.05% respectively compared with self-teaching method and cross validation method. Though the method is proposed for signer adaptation, it can also be applied to speaker adaptation and writer adaptation directly.