Automated comprehensive evaluation approach for user interface satisfaction based on concurrent think-aloud method

Abstract The concurrent think-aloud protocol (CTA) is an effective method for collecting abundant product comments related to user satisfaction during the execution of evaluation tasks. However, manual analysis of these audio comments is time-consuming and labor-intensive. This paper aims to propose an approach for automated comprehensive evaluation of user interface (UI) satisfaction. It takes advantage of text mining and sentiment analysis (SA) techniques instead of manual analysis in order to assess user comments collected by the CTA. Based on the results of the SA, the proposed approach makes use of the analytic hierarchy process (AHP) method to evaluate the overall satisfaction and support developers for UI design improvements. In order to enhance the objectivity of evaluation, a sentiment matrix originating from text mining and SA on user comments is used to replace the criteria and the relative weights of the AHP method which were previously defined by experts. A comparison between the questionnaire survey method and the proposed approach in the empirical study suggested that the latter can efficiently evaluate UI satisfaction with high accuracy and provide designers abundant and specific information directly related to defects in design. It is argued that the proposed approach could be used as an automated framework for handling any type of comments.