Subjective evaluation of spoken dialogue systems using SER VQUAL method

There is demand for subjective metrics in spoken dialogue system evaluation. SERVQUAL is a service quality evaluation method developed by marketing academics. It produces a subjective measure of the gap between expectations and perceptions in five service quality dimensions common for all services. We present how the method was applied to spoken dialogue system evaluation. In order to improve the suitability of the original method, we modified the test questionnaire and the test process. We demonstrate how the modified method was successfully used in an evaluation of a telephone-based e-mail application. The evaluation gave us directions for further development of the system. In addition, we found some interesting phenomena, such as the variation between genders. We present how the method can be further improved, for example, by dividing the questionnaire into two parts.