Action-perception cycle learning for incremental emotion recognition in a movie clip using 3D fuzzy GIST based on visual and EEG signals

Emotions are regarded as the complex programs of internal actions triggered by the perception of visual stimuli. To understand human emotions in a more natural situation, we use dynamic stimuli such as movies for the analysis. Electroencephalography EEG signals evoked while watching the movie clip are also used to understand subject specific emotions for the movies. To benefit from the integrated ways that human perceive emotions, this paper proposes a mathematical framework to incorporate the link between two modalities to highly interact with in an action-perception cycle, which uses incremental concepts for understanding the complex human emotions over time. Incremental adaptive neuro-fuzzy inference system ANFIS is used to autonomously learn new emotional states from the information available over time. The system automatically adjusts or increases the rules for clustering the features in a fuzzy domain based on the interactions. After improving the recognition of individual sub-systems, the emotional descriptors from both channels are concatenated to be used as inputs in the incremental ANFIS in the next stage in order to classify a movie clip into a positive or negative emotion. Utilizing the action-perception cycle, the system can autonomously develop the ability to recognize complex human emotions through interactions with the environment. The mean opinion score MOS is used as ground truth to evaluate the performance of the proposed emotion recognition system.