One in a hundred: Select the best predicted sequence from numerous candidates for streaming speech recognition

The RNN-Transducers and improved attention-based sequence-to-sequence models are widely applied to streaming speech recognition. Compared with these two end-to-end models, the CTC model is more efficient in training and inference. However, it cannot capture the dependencies between the output tokens. We assume that we can pick out the best sequence among the first N candidates predicted by the CTC model. Enough number and enough diversity of candidates can compensate it for the lack of language modeling ability. Therefore, we improve the hybrid CTC and attention model, and introduce a two-stage inference method named one-in-a-hundred (OAH). During inference, we first generate many candidates by the CTC decoder in a streaming fashion. Then the transformer decoder selects the best candidate based on the corresponding acoustic encoded states. All the experiments are conducted on a Chinese Mandarin dataset AISHELL-1. The results show that our proposed model can implement stream decoding in a fast and straightforward way. Our model can achieve up to 20% reduction in the character error rate than the baseline CTC model. In addition, our model can also perform non-streaming decoding.