Automatic Voicemail Summarisation for Mobile Messaging

One way to overcome the display and bandwidth limitations of today’s mobile environments is to reduce the amount of data transmitted to end-users by selecting only principal content, at the cost of introducing a tolerable information loss. This thesis provides a framework that allows systematic comparisons and integration of patterns present in voicemail messages for constructing summaries suitable for mobile messaging applications. In order to derive a text representation of voicemail messages we employ automatic speech recognition technology. IBM Voicemail Corpus-Part I is used in our experiments which represents a large vocabulary task over telephone lines with problems that include unknown channels, spontaneous and topic independent speech. We use a hybrid connectionist/HMM approach with a combination of front-ends and multi-style language modelling. This results into a compact system with competitive accuracy relative to that produced by more complex systems based on Gaussian mixture models. Voicemail summarisation differs from conventional text summarisation, since it does not assume a perfect transcription and is concerned with summarising brief spoken messages into terse summaries. We have adopted a word-extractive approach with each word in the transcribed message represented as a vector of features. The initial realisation of the summarisation component is based on lexical feature weighting given some summary length restrictions or compression rates. Frequent message terms are compacted and any terms classified as less informative are excluded from the summaries. The resulting summaries are then converted into a format suitable for transmission over narrowband wireless networks. The platform of choice is WAP Push over SMS that offers a proactive way to transmit data from servers to mobile devices without explicit user requests as well as easy and immediate access to particular voicemail messages. Machine learning methods are then used to investigate the extent to which lexical and prosodic features can be associated with content in voicemail messages. Prosodic features concern the way in which speech sounds are acoustically realised and the ones we extracted can be broadly grouped as referring to pitch, energy, word duration and pauses. One can identify many potentially relevant but also interrelated features for this task. We employ a feature selection approach in which we aim to use the data to guide us to an optimal subset of features. Instead of specifying a single classifier and feature set – optimised for a particular precision/recall trade-off – we maintain a set of classifiers/feature sets, optimising for all possible precision/recall trade-offs. We achieve this by considering the ROC curves of the trained classifiers (with respect to development data) and forming the convex hull of those ROC curves. The relative contribution of a large number of features and derived subsets is compared within two summarisation tasks, namely the binary decision and multi-class tasks. In the former, the goal is to classify words into those carrying principal content and those that do not, while in the latter the goal is to further classify the principal content words into proper names, telephone numbers, dates/times and other. A series of objective and subjective evaluations using unseen data is also presented. The objective evaluations show significant improvements over the baseline systems while the subjective evaluations show that users are able to determine the message priority and content fairly accurately. The perceived difference in quality of summarisation is affected more by errors resulting from automatic transcription, than by the automatic summarisation process. This suggests that the provision of accurate transcriptions is essential for successful speech summarisation applications. Finally, an evaluation framework is proposed with the aim to determine which metrics maximise summary quality and minimise delivery costs by combining user data and comparing system configurations that make different trade-offs.