Time Series Data Mining for Multimodal Bio-Signal Data

Summary It is believed that there is a close correlation between the physical and mental activities of human body. These physical activities can be measured using various multimodal bio-signals employing either invasive or non-invasive sensors. In contrast, although mental activities have been modeled from various standpoints, they have been associated mostly with brain activities. We have focused on mapping physical multimodal bio-signals as time series data with mental states such as somnolence, fatigue, and concentration. We will discuss two mathematical data mining tools: (1) the interval cross-correlation coefficient; and (2) the interval cross- covariance function coefficient. Given a multimodal time series bio-signal data acquired at a given frequency using non-invasive sensors attached to a human volunteer, our methods aimed to predict the mental state of a human subject. The primary objective was to examine the feasibility of our methods in predicting the mental state of students during lessons in a university classroom. Previous attempts to predict mental states from bio-signals have mostly been based on electroencephalogram (EEG), electrocardiogram (ECG), or electrooculogram (EOG), but have not tried to combine them. However, it is often difficult to obtain stable EEG signals from students in a university classroom because of artifacts arising from body and eye movements. Based on this observation, we considered simultaneous multimodal bio-signals with their combination, and introduced interval cross- correlation coefficient and interval cross-covariance function coefficient as data mining tools for mapping the physical and mental states of the human body. We conducted experiments using subjects equipped with multiple sensors, and compared the results with the outputs of our data mining methods. Preliminary experiments show that our method produces reasonable results and allows us to control the experimental parameters to cope with individual variations. Our method is also applicable to monitoring personal health care, vehicle drivers, and individuals in business group meetings.