Context and Culture affect the Psychometrics of Questionnaires evaluating Speech-based Assistants

Intelligent Personal Assistants (IPAs) have grown into technologically mature systems. However, instruments used for evaluating the usability and user experience of IPAs were developed two decades ago. This bears the danger for research and development to apply inadequate measurements to a novel technology. In this study, more recent scales from human-robot-interaction were used for evaluating speech-based assistants in vehicles in a Chinese sample. However, it cannot be assumed, that adapting a questionnaire from another context and culture leads to objective, reliable and valid measurements. Therefore, data was examined regarding internal consistency and factor structure. Cronbach's alpha was considerably high. However, a factor analysis did not support the assumed four-factor structure but rather a two-factorial solution. Findings suggest that adapting a questionnaire from a different context and culture, can affect its psychometrics. Consequently, the underlying postulated constructs should be treated with caution.