Automated Regression Testing Approach To Expansion And Refinement Of Speech Recognition Grammars

This thesis describes an approach to automated regression testing for speech recognition grammars. A prototype Audio Regression Tester called ART has been developed using Microsoft's Speech API and C#. ART allows a user to perform any of three tasks: automatically generate a new XML-based grammar file from standardized SQL database entries, record and cross-reference audio files for use by an underlying speech recognition engine, and perform regression tests with the aid of an oracle grammar. ART takes as input a wave sound file containing speech and a newly created XML grammar file. It then simultaneously executes two tests: one with the wave file and the new grammar file and the other with the wave file and the oracle grammar. The comparison result of the tests is used to determine whether the test was successful or not. This allows rapid exhaustive evaluations of additions to grammar files to guarantee forward process as the complexity of the voice domain grows. The data used in this research to derive results were taken from the LifeLike project. However, the capabilities of ART extend beyond LifeLike. The results gathered have shown that using a person's recorded voice to do regression testing is as effective as having the person do live testing. A cost-benefit analysis, using two published equations, one for Cost and the other for Benefit, was also performed to determine if automated regression testing is really more effective than manual testing. Cost captures the salaries iv of the engineers who perform regression testing tasks and Benefit captures revenue gains or losses related to changes in product release time. ART had a higher benefit of $21461.08 when compared to manual regression testing which had a benefit of $21393.99. Coupled with its excellent error detection rates, ART has proven to be very efficient and cost-effective in speech grammar creation and refinement. v To my Mom and Dad for their unlimited love and support vi ACKNOWLEDGMENTS

[1]  Steve Young.  A review of large-vocabulary continuous-speech , 1996 .

[2]  John Makhoul,et al.  BYBLOS: The BBN continuous speech recognition system , 1987, ICASSP '87. IEEE International Conference on Acoustics, Speech, and Signal Processing.

[3]  D.I. Moldovan,et al.  PASS: a parallel speech understanding system , 1993, Proceedings of 9th IEEE Conference on Artificial Intelligence for Applications.

[4]  Rolf P. Würtz,et al.  Emergent Graphs with PCA‐features for Improved Face Recognition , 2006 .

[5]  Gregg Rothermel,et al.  Analyzing Regression Test Selection Techniques , 1996, IEEE Trans. Software Eng..

[6]  Geoffrey Zweig,et al.  Advances in speech transcription at IBM under the DARPA EARS program , 2006, IEEE Transactions on Audio, Speech, and Language Processing.

[7]  Lee J. White.  Regression testing of GUI event interactions , 1996, 1996 Proceedings of International Conference on Software Maintenance.

[8]  Frank K. Soong,et al.  High performance connected digit recognition, using hidden Markov models , 1988, ICASSP-88., International Conference on Acoustics, Speech, and Signal Processing.

[9]  R. F. DeMara,et al.  The SNAP-1 parallel AI prototype , 1991, [1991] Proceedings. The 18th Annual International Symposium on Computer Architecture.

[10]  Brian S. Stensrud,et al.  Context-Based Reasoning: A Revised Specification , 2004, FLAIRS.

[11]  Biing-Hwang Juang,et al.  Hidden Markov Models for Speech Recognition , 1991 .

[12]  Geoffrey Zweig,et al.  Advances in Mandarin Broadcast Speech Transcription at IBM Under the DARPA GALE Program , 2006, ISCSLP.

[13]  Hareton Leung,et al.  A study of integration testing and software regression at the integration level , 1990, Proceedings. Conference on Software Maintenance 1990.

[14]  Hareton K. N. Leung,et al.  A cost model to compare regression test strategies , 1991, Proceedings. Conference on Software Maintenance 1991.

[15]  Sigrid Eldh.  Software Testing Techniques , 2007 .

[16]  Aaron E. Rosenberg,et al.  Evaluation of a word recognition system using syntax analysis , 1977 .

[17]  Gregg Rothermel,et al.  An empirical study of regression testing techniques incorporating context and lifetime factors and improved cost-benefit models , 2006, SIGSOFT '06/FSE-14.

[18]  Juergen Luettin,et al.  Audio-Visual Speech Modeling for Continuous Speech Recognition , 2000, IEEE Trans. Multim..

[19]  D. Sheskin.  The Pearson Product-Moment Correlation Coefficient , 2003 .

[20]  David S. Rosenblum,et al.  TESTTUBE: a system for selective regression testing , 1994, Proceedings of 16th International Conference on Software Engineering.

[21]  Henry Lieberman,et al.  How to wreck a nice beach you sing calm incense , 2005, IUI.

[22]  Rajiv Gupta,et al.  An approach to regression testing using slicing , 1992, Proceedings Conference on Software Maintenance 1992.

[23]  A. E. Rosenberg,et al.  Evaluation of a word recognition system using syntax analysis , 1978, The Bell System Technical Journal.

[24]  Atif M. Memon,et al.  DART: a framework for regression testing "nightly/daily builds" of GUI applications , 2003, International Conference on Software Maintenance, 2003. ICSM 2003. Proceedings..

[25]  Bogdan Korel,et al.  Automated regression test generation , 1998, ISSTA '98.

[26]  Hareton Leung,et al.  A firewall concept for both control-flow and data-flow in regression integration testing , 1992, Proceedings Conference on Software Maintenance 1992.

[27]  Christiane Fellbaum,et al.  Book Reviews: WordNet: An Electronic Lexical Database , 1999, CL.

[28]  Gregg Rothermel,et al.  A safe, efficient regression test selection technique , 1997, TSEM.

[29]  Henrik Wann Jensen,et al.  Light diffusion in multi-layered translucent materials , 2005, ACM Trans. Graph..

[30]  Nancy J. Wahl.  An overview of regression testing , 1999, SOEN.

[31]  B. Lowerre,et al.  Dynamic speaker adaptation in the Harpy speech recognition system , 1977 .

[32]  David Chenho Kung,et al.  Or Regression Testing of Object-Oriented Programs , 1996, J. Syst. Softw..

[33]  Boris Beizer,et al.  Software Testing Techniques , 1983 .

[34]  J. Baker,et al.  The DRAGON system--An overview , 1975 .

[35]  D. J. Robson,et al.  Approaches to regression testing , 1988, Proceedings. Conference on Software Maintenance, 1988..

[36]  Hsiao-Wuen Hon,et al.  An overview of the SPHINX speech recognition system , 1990, IEEE Trans. Acoust. Speech Signal Process..

[37]  Gregg Rothermel,et al.  Selecting regression tests for object-oriented software , 1994, Proceedings 1994 International Conference on Software Maintenance.

[38]  Avelino J. Gonzalez,et al.  Towards Interactive Training with an Avatar-based Human-Computer Interface , 2008 .