Utility assessment in TRANSTAC: using a set of complementary methods

This paper describes the methods used during formative utility assessments for speech-to-speech, real-time translation systems intended for tactical use. The systems, while still prototypical, had hardware platforms that could be exercised indoors or outdoors by study participants with pertinent backgrounds and skills that are similar to the intended target users. English-speaking and foreign language subjects participated in exercises using the systems exchanging information without the aid of a human interpreter. Feedback on subjects' experiences during the dialog interactions was collected via two primary methods: survey questionnaires and semi-structured interviews. In this paper, we describe our human-centered approach to utility assessment, how we combined the use of two feedback gathering methods, and discuss findings from use of this approach.