Measurements in support of research accomplishments

(NIST) has provided support to other federal governmental agencies' speech research programs , and the university and industrial research communities, by developing, and implementing , test protocols, and reference test sets to provide quantitative measures of the progress of research accomplishments. These test sets are distributed to participants in the Measurements IN Accomplishments How the National Institute of Standards and Technology provides a basis for evaluating these novel systems. course of NIST's implementations of community wide tests, and the results of the tests are widely quoted. Throughout this period, progress was traditionally measured in terms of word error rate—the percentage of substitution , deletion, and insertion word errors, relative to the number of words in an accurately transcribed set of recorded speech test material. Word error rates have shown a marked decline as the technology has improved and additional materials become available for system training. Program managers have generally challenged the research community to undertake more difficult tasks, as it becomes evident that substantial progress has occurred. Figure 1 shows some of the historic word error rate data associated with the development of large vocabulary continuous speech recognition technology within the Defense Advanced Research Projects Agency (DARPA) research community. From approximately 1992, research results were reported on speech recorded from " read " texts derived from electronic news-wire services (the Wall Street Journal and North American Business News corpora). These corpora provided a well-controlled reference condition, but limited the challenge to the research community because of the absence of other phenomena such as spontaneity , the use of different microphones, tele