BAREM: A multimodal dataset of individuals interacting with an e-service platform

The use of e-service platforms has become essential for many applications (administrative documents, online shopping, reservations). Although these platforms have improved significantly the user experience, unexpected and stressful situations can occur. Navigation problems (latency, missing information, poor ergonomics) are not always reported to the designers. To address this problem, we propose a multimodal dataset (video, audio, and physiological data) to help implicitly quantify the impact of navigation problems on users when using an e-service platform. A scenario has been designed to generate various navigation problems which can lead to changes in user behaviour. A baseline is proposed to spot changes in user behaviour, opening the way towards automatically qualifying user experiences while using e-service platforms.

[1]  Hichem Snoussi,et al.  Abnormal event detection via covariance matrix for optical flow based feature , 2018, Multimedia Tools and Applications.

[2]  Ashish Kapoor,et al.  Automatic prediction of frustration , 2007, Int. J. Hum. Comput. Stud..

[3]  Zhi-Hua Zhou,et al.  Isolation Forest , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[4]  Jonathan P. Rowe,et al.  Improving Affect Detection in Game-Based Learning with Multimodal Data Fusion , 2020, AIED.

[5]  Hans-Peter Kriegel,et al.  LOF: identifying density-based local outliers , 2000, SIGMOD '00.

[6]  Zhuowen Tu,et al.  Aggregated Residual Transformations for Deep Neural Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Mohammad H. Mahoor,et al.  DISFA: A Spontaneous Facial Action Intensity Database , 2013, IEEE Transactions on Affective Computing.

[8]  Soyeong Jeong,et al.  Development of Speech Emotion Recognition Algorithm using MFCC and Prosody , 2020, 2020 International Conference on Electronics, Information, and Communication (ICEIC).

[9]  Diane Kelly,et al.  Engaged or Frustrated?: Disambiguating Emotional State in Search , 2017, SIGIR.

[10]  Fabien Ringeval,et al.  Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[11]  Kyandoghere Kyamakya,et al.  Emotion Recognition Involving Physiological and Speech Signals: A Comprehensive Review , 2018 .

[12]  Björn W. Schuller,et al.  AVEC 2013: the continuous audio/visual emotion and depression recognition challenge , 2013, AVEC@ACM Multimedia.

[13]  Ying He,et al.  Spotting Macro-and Micro-expression Intervals in Long Video Sequences , 2020, 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020).

[14]  Fillia Makedon,et al.  Deep Visual Attributes vs. Hand-Crafted Audio Features on Multidomain Speech Emotion Recognition , 2017, Comput..

[15]  Maja Pantic,et al.  This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. IEEE TRANSACTIONS ON AFFECTIVE COMPUTING , 2022 .

[16]  Abhinav Dhall,et al.  Depression Scale Recognition from Audio, Visual and Text Analysis , 2017, ArXiv.

[17]  Kristy Elizabeth Boyer,et al.  Automatically Recognizing Facial Indicators of Frustration: A Learning-centric Analysis , 2013, 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

[18]  P. Ekman,et al.  What the face reveals : basic and applied studies of spontaneous expression using the facial action coding system (FACS) , 2005 .

[19]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[20]  Chunhua Shen,et al.  Self-Trained Deep Ordinal Regression for End-to-End Video Anomaly Detection , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Moi Hoon Yap,et al.  MEGC2020 - The Third Facial Micro-Expression Grand Challenge , 2020, 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020).

[22]  Klaus R. Scherer,et al.  Vocal communication of emotion: A review of research paradigms , 2003, Speech Commun..

[23]  Meike Jipp,et al.  Recognizing Frustration of Drivers From Face Video Recordings and Brain Activation Measurements With Functional Near-Infrared Spectroscopy , 2018, Front. Hum. Neurosci..

[24]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.