Mobile Device-Based Speech Enhancement System Using Lip-Reading

This paper describes our preliminary study towards a new type of speech enhancement system. To avoid using odd-looking electrolarynx, we used lip-reading function. Our final image is to use a smart phone with camera and audio output to be able to convert the lip motion to speech output. We tested MLP, CNN, and MobileNets image recognition methods. 3k image datasets for training and testing were recorded from five persons. The preliminary experiment indicated that the MobileNets is the most adequate algorithm for smart phone apps. in terms of the recognition accuracy and the calculation cost.