An Environmental Audio{Based Context Recognition System Using Smartphones

Environmental sound/audio is a rich source of information that can be used to infer a person's context in daily life. Almost every activity produces some sound patterns, e.g., speaking, walking, washing, or typing on computer. Most locations have usually a specific sound pattern too, e.g., restaurants, offices or streets. This thesis addresses the design and development of an application for real-time detection and recognition of user activities using audio signals on mobile phones. The audio recognition application increases the capability, intelligence and feature of the mobile phones and, thus, increases the convenience of the users. For example, a smartphone can automatically go into a silent mode while entering a meeting or provide information customized to the location of the user. However, mobile phones have limited power and capabilities in terms of CPU, memory and energy supply. As a result, it is important that the design of audio recognition application meets the limited resources of the mobile phones. In this thesis we compare performance of different audio classifiers (k-NN, SVM and GMM) and audio feature extraction techniques based on their recognition accuracy and computational speed in order to select the optimal ones. We evaluate the performance of the audio event recognition techniques on a set of 6 daily life sound classes (coffee machine brewing, water tape (hand washing), walking, elevator, door opening/closing, and silence ). Test results show that the k-NN classifier (when used with mel-frequency cepstral coefficients (MFCCs), spectral entropy (SE) and spectral centroid (SC) audio features) outperforms other audio classifiers in terms of recognition accuracy and execution time. The audio features are selected based on simulation results and proved to be optimal features. An online audio event recognition application is then implemented as an Android app (on mobile phones) using the k-NN classifier and the selected optimal audio features. The application continuously classifies audio events (user activities) by analyzing environmental sounds sampled from smartphone's microphone. It provides a user with real-time display of the recognized context (activity). The impact of other parameters such as analysis window and overlapping size on the performance of audio recognition is also analyzed. The test result shows that varying the parameters does not have significant impact on the performance of the audio recognition technique. Moreover, we also compared online audio recognition results of the same classifier set (i.e.,k-NN) with that of the off-line classification results.