Collection and Analysis of Data from Real Users: Implications for Speech Recognition/Understanding Systems

Performance estimates given for speech recognition/understanding systems are typically based on the assumption that users will behave in ways similar to the observed behavior of laboratory volunteers. This includes the acoustic/phonetic characteristics of the speech they produce as well as their willingness and ability to constrain their input to the device according to instructions. Since speech recognition devices often do not perform as well in the field as they do in the laboratory, analyses of real user behavior have been undertaken. The results of several field trials suggest that real user compliance with instructions is dramatically affected by the particular details of the prompts supplied to the user. A significant amount of real user speech data has been collected during these trials (34,000 utterances, 29 hours of data). These speech databases are described along with the results of an experiment comparing the performance of a speech recognition system on real user vs. laboratory speech.