CONTEMPORARY MULTIMODAL DATA COLLECTION METHODOLOGY FOR RELIABLE INFERENCE OF AUTHENTIC SURPRISE

The need for intelligent systems that can understand and convey human emotional expression is becoming increasingly prevalent. Unfortunately, most datasets for developing such systems rely on acted or exaggerated emotions, or utilize subjective labels obtained from possibly unreliable sources. This paper reports on an innovative data collection methodology for capturing multimodal human signals of authentic surprise. We introduce two tasks with a facilitator to elicit genuine reactions of surprise while co-collecting data from three human modalities: speech, facial expressions, and galvanic skin response. Our work highlights the methodological potential of biophysical measurement-based validation for enabling reliable inference. A case study is presented which provides baseline results for Random Forest classification. Using features gathered from the three modalities, our baseline system is able to identify surprise instances with approximately 20% absolute increase in accuracy compared to random assignment on a balanced dataset.