Deep Modular Multimodal Fusion on Multiple Sensors for Volcano Activity Recognition

Nowadays, with the development of sensor techniques and the growth in a number of volcanic monitoring systems, more and more data about volcanic sensor signals are gathered. This results in a need for mining these data to study the mechanism of the volcanic eruption. This paper focuses on Volcano Activity Recognition (VAR) where the inputs are multiple sensor data obtained from the volcanic monitoring system in the form of time series. And the output of this research is the volcano status which is either explosive or not explosive. It is hard even for experts to extract handcrafted features from these time series. To solve this problem, we propose a deep neural network architecture called VolNet which adapts Convolutional Neural Network for each time series to extract non-handcrafted feature representation which is considered powerful to discriminate between classes. By taking advantages of VolNet as a building block, we propose a simple but effective fusion model called Deep Modular Multimodal Fusion (DMMF) which adapts data grouping as the guidance to design the architecture of fusion model. Different from conventional multimodal fusion where the features are concatenated all at once at the fusion step, DMMF fuses relevant modalities in different modules separately in a hierarchical fashion. We conducted extensive experiments to demonstrate the effectiveness of VolNet and DMMF on the volcanic sensor datasets obtained from Sakurajima volcano, which are the biggest volcanic sensor datasets in Japan. The experiments showed that DMMF outperformed the current state-of-the-art fusion model with the increase of F-score up to 1.9% on average.

[1]  Li Wei,et al.  Fast time series classification using numerosity reduction , 2006, ICML.

[2]  M. I. Seht,et al.  Detection and identification of seismic signals recorded at Krakatau volcano (Indonesia) using artificial neural networks , 2008 .

[3]  Javier Ramírez,et al.  Continuous HMM-Based Seismic-Event Classification at Deception Island, Antarctica , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[4]  R. Sparks,et al.  Forecasting volcanic eruptions , 2003 .

[5]  Jukka Saarinen,et al.  Prediction with Multilayer Perceptron , FIR and Elman Neural Networks , 1996 .

[6]  Xiaoli Li,et al.  An integrated framework for human activity classification , 2012, UbiComp.

[7]  Christopher Joseph Pal,et al.  EmoNets: Multimodal deep learning approaches for emotion recognition in video , 2015, Journal on Multimodal User Interfaces.

[8]  Yusaku Ohta,et al.  Characteristics of Volcanic Activity at Sakurajima Volcano's Showa Crater During the Period 2006 to 2011(＜Special Section＞Sakurajima Special Issue) , 2013 .

[9]  Bernt Schiele,et al.  A tutorial on human activity recognition using body-worn inertial sensors , 2014, CSUR.

[10]  Robert P. W. Duin,et al.  Classification of Volcano Events Observed by Multiple Seismic Stations , 2010, 2010 20th International Conference on Pattern Recognition.

[11]  Jukka Saarinen,et al.  Time Series Prediction with Multilayer Perception, FIR and Elman Neural Networks , 1996 .

[12]  Eamonn J. Keogh,et al.  Dimensionality Reduction for Fast Similarity Search in Large Time Series Databases , 2001, Knowledge and Information Systems.

[13]  Eamonn J. Keogh,et al.  A Complexity-Invariant Distance Measure for Time Series , 2011, SDM.

[14]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[15]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[16]  Juhan Nam,et al.  Multimodal Deep Learning , 2011, ICML.

[17]  Sergey Malinchik,et al.  SAX-VSM: Interpretable Time Series Classification Using SAX and Vector Space Model , 2013, 2013 IEEE 13th International Conference on Data Mining.

[18]  Dimitri Palaz,et al.  Analysis of CNN-based speech recognition system using raw speech as input , 2015, INTERSPEECH.

[19]  Xiaoli Li,et al.  Deep Convolutional Neural Networks on Multichannel Time Series for Human Activity Recognition , 2015, IJCAI.

[20]  Marielle Malfante,et al.  Machine Learning for Volcano-Seismic Signals: Challenges and Perspectives , 2018, IEEE Signal Processing Magazine.

[21]  Daniel Roggen,et al.  Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition , 2016, Sensors.

[22]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[23]  Yi Zheng,et al.  Time Series Classification Using Multi-Channels Deep Convolutional Neural Networks , 2014, WAIM.