Bidirectional Associative Memory for Multimodal Fusion : a Depression Evaluation Case Study

In this study, we step aside from traditional fusion strategies and propose to use a bidirectional associative memory to combine abstract representations of several modalities. The innovative contribution of our strategy is the fact that the fusion is performed neither at the input data level nor at the output score level. The model achieves outperforming results on depression evaluation from the videos of the AVEC2014 challenge corpus.

[1]  Roland Göcke,et al.  Diagnosis of depression by behavioural signals: a multimodal approach , 2013, AVEC@ACM Multimedia.

[2]  Derek T. Anderson,et al.  Fuzzy Choquet Integration of Deep Convolutional Neural Networks for Remote Sensing , 2018 .

[3]  Hélène Paugam-Moisy,et al.  Prototype-based Classifier for Automatic Diagnosis of Depressive Mood , 2018, 2018 International Conference on Biomedical Engineering and Applications (ICBEA).

[4]  Fan Zhang,et al.  Artificial Intelligent System for Automatic Depression Level Analysis Through Visual and Vocal Expressions , 2018, IEEE Transactions on Cognitive and Developmental Systems.

[5]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.

[6]  Renato A. Krohling,et al.  Aggregation of neural classifiers using Choquet integral with respect to a fuzzy measure , 2018, Neurocomputing.

[7]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[8]  Tanaya Guha,et al.  Multimodal Prediction of Affective Dimensions and Depression in Human-Computer Interactions , 2014, AVEC '14.

[9]  H. Paugam-Moisy,et al.  A multiple BAM for hetero-association and multisensory integration modelling , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[10]  Xinhua Zhuang,et al.  A general model for bidirectional associative memories , 1998, IEEE Trans. Syst. Man Cybern. Part B.

[11]  Heekuck Oh,et al.  Adaptation of the relaxation method for learning in bidirectional associative memory , 1994, IEEE Trans. Neural Networks.

[12]  A. Beck,et al.  Beck Depression Inventory–II , 2011 .

[13]  Stephen Grossberg,et al.  Competitive Learning: From Interactive Activation to Adaptive Resonance , 1987, Cogn. Sci..

[14]  Andrew Zisserman,et al.  Deep Face Recognition , 2015, BMVC.

[15]  Björn W. Schuller,et al.  AVEC 2014: 3D Dimensional Affect and Depression Recognition Challenge , 2014, AVEC '14.

[16]  Dongmei Jiang,et al.  Multimodal Measurement of Depression Using Deep Learning Models , 2017, AVEC@ACM Multimedia.

[17]  Fabien Ringeval,et al.  AVEC 2017: Real-life Depression, and Affect Recognition Workshop and Challenge , 2017, AVEC@ACM Multimedia.

[18]  Sung-Bae Cho,et al.  Fuzzy aggregation of modular neural networks with ordered weighted averaging operators , 1995, Int. J. Approx. Reason..

[19]  Fernando De la Torre,et al.  Detecting depression from facial actions and vocal prosody , 2009, 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops.

[20]  Thomas F. Quatieri,et al.  Vocal and Facial Biomarkers of Depression based on Motor Incoordination and Timing , 2014, AVEC '14.

[21]  BART KOSKO,et al.  Bidirectional associative memories , 1988, IEEE Trans. Syst. Man Cybern..

[22]  Didier Puzenat.  Parallelisme et modularite des modeles connexionnistes , 1997 .

[23]  T. Strine,et al.  The PHQ-8 as a measure of current depression in the general population. , 2009, Journal of affective disorders.

[24]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..