Efficient combination of type-in and wizard-of-oz tests in speech interface development process

Effective methods and tools are needed for designing, developing and testing speech-enabled applications. Traditional methods have been collecting natural dialogues, using Wizardof-Oz experiments and different type-in tests. Collecting and analysing natural dialogues is always time consuming and in many situations even impossible. Wizard-of-Oz studies need well defined test scenarios and a lot of time to collect and analyse data. Type-in studies are one possibility to collect big amount of data quickly and analyse it semi-automatically. This paper discusses Wizard-of-Oz and type-in data collection methods as implemented at our site. The test set-ups and the identified discrepancies are discussed in details. As our study indicates, testing an initial version of a dialogue flow and the corresponding grammar with the type-in method offers quite reliable measures about the final implementation of the interface for a speech-enabled navigation application.