Multilingual Spoken Term Detection: Finding and Testing New Pronunciations

When you listen to the evening news, or read a newspaper, book or web site, there is a good chance that you will hear or see a term — perhaps a name, perhaps a technical term — that you have never seen before. Such words are often novel or rare and are often names (of people, places, organizations. . . ). They are hard for humans to process, but they are even harder for automatic speech and language processing systems. For a single language, a speech recognition or text-to-speech system needs to know how to pronounce a word to recognize or say it. For two languages, in particular a pair with different writing systems, a search engine or document summarizer needs to know how to transcribe one word to another to retrieve or distill across languages. For example, the soccer player written in English text as Maciej Zurawski would appear as 마치에이주라브스키 (ma-chi-e-i ju-ra-beu-seu-ki) in Korean In this project we attack both problems – unusual term pronunciation and term transcription. For pronunciation, we make use of the huge numbers of pronunciations that are now available in various forms on the web to mine pronunciations. This ranges from straightforward, such as dictionary sites and Wikipedia entries where people use a fairly strict phonetic transcription system such as IPA, to difficult such as: • Trio Shares Nobel Prize in Medicine – The Nobel is a particularly striking achievement for Capecchi (pronounced kuh-PEK’-ee). Here we need to look in the vicinity of the name “Capecchi” to find the pronunciation, make use of the word “pronounced”, and then interpret the writer’s attempt to render the pronunciation using an Englishbased ad-hoc “phonetic” orthography. The problem is therefore one of entity extraction, where the entities to extract can be either relatively easy or relatively hard. A relatively easy case is Wikipedia, which uses standard IPA transcriptions that are clearly delimited by markup. On general web pages, tokens with Unicode IPA characters are potential pronunciations. Data extracted from Wikipedia can be matched against these tokens to provide training material for entity extraction. Statistical entity extractors for the more difficult case of ad-hoc phonetic transcriptions (such as “kuh-PEK’-ee” above) can be bootstrapped from unannotated web pages containing patterns such as “pronounced as”. These entity extractors make use of both the textual environment and the letter-to-sound constraints between the candidate pronunciation and its corresponding orthography. We also use speech data to test possible pronunciation variants by comparing the performance of spoken term detection systems using these different variants. Pronunciations mined from the web are used to suggest pronunciations for spoken term detection; transcription are used to suggest reasonable candidates to search for in a speech stream in another language. We use a novel technique called delayed-decision testing to test candidate pronunciations in speech, and to choose the best one from a set of candidates via a sequential testing procedure, with the associated null hypothesis stating that all candidate pronunciations exhibit the same performance on average. Spoken term detection are in turn used for automatic labeling of practice data acquired to test this null hypothesis; however, this automatic labeling procedure inevitably induces false alarms as well as correct detections. Delayed-decision testing are then used to choose the correct pronunciation in spite of these false alarms, leading to improved pronunciations for newly identified terms. For transcription, we use available resources – dictionaries, and text corpora – as well as methods for phonetic matching across scripts and tracking names across time in comparable corpora (such as news sources).