Mutual disambiguation of recognition errors in a multimodel architecture

As a new generation of multimodal/media systems begins to defineitself, researchers are attempting to learn how to combinedifferent modes into strategically integrated whole systems. Intheory, well designed multimodal systems should be able tointegrate complementary modalities in a manner that supports mutualdisambiguation (MD) of errors and leads to more robust performance.In this study, over 2,000 multimodal utterances by both native andaccented speakers of English were processed by a multimodal system,and then logged and analyzed. The results confirmed that multimodalsystems can indeed support significant levels of MD, and alsohigher levels of MD for the more challenging accented users. As aresult, although speech recognition as a stand-alone performed farmore poorly for accented speakers, their multimodal recognitionrates did not differ from those of native speakers. Implicationsare discussed for the development of future multimodalarchitectures that can perform in a more robust and stable mannerthan individual recognition technologies. Also discussed is thedesign of interfaces that support diversity in tangible ways, andthat function well under challenging real-world usageconditions,