The Impact of Asr Accuracy on the Performance of an Automated Scoring Engine for Spoken Responses

Automated scoring to assess speaking proficiency depends to a great extent on the availability of appropriate tools for speech processing (such as speech recognizers). These tools not only must be robust to the sorts of speech errors and nonstandard pronunciations exhibited by language learners, but must provide metrics which can be used as a basis for assessment. One major strand of current research in the area of automated scoring of spoken responses is the effort to develop deeper measures of the grammatical, discourse, and semantic structure of learners' speech, in order to model a broader speaking proficiency construct than one which focuses primarily on phonetic and timing characteristics of speech. The quality of speech recognition systems is especially crucial to this research goal, as errors in speech recognition lead to downstream errors in the computation of higher‐level linguistic structures that can be used to calculate construct‐relevant features. The goal of this paper is to provide a case study illustrating the effects of speech recognition accuracy on what can be achieved in automated scoring of speech. A comparison of speech features calculated on the basis of competing speech recognition systems demonstrates that scoring accuracy is strongly dependent on using the most accurate models available, both for open‐ended tasks and for more restricted speaking tasks.