Exploring the Potential of Short-Time Fourier Transforms for Analyzing Skin Conductance and Pupillometry in Real-Time Applications

The development of real-time predictors of mental workload is critical for the practical application of augmented cognition to human-machine systems. This paper explores a novel method based on a short-time Fourier transform (STFT) for analyzing galvanic skin conductance (SC) and pupillometry time-series data to extract estimates of mental workload with temporal bandwidth high-enough to be useful for augmented cognition applications. We tested the method in the context of a process control task based on the DURESS simulation developed by Vincente and Pawlak (1994; ported to Java by Cosentino,& Ross, 1999). SC, pupil dilation, blink rate, and visual scanning patterns were measured for four participants actively engaged in controlling the simulation. Fault events were introduced that required participants to diagnose errors and make control adjustments to keep the simulator operating within a target range. We were interested in whether the STFT of these measures would produce visible effects of the increase in mental workload and stress associated with these events. Graphical exploratory data analysis of the STFT showed visible increases in the power spectrum across a range of frequencies directly following fault events. We believe this approach shows potential as a relatively unobtrusive, low-cost, high bandwidth measure of mental workload that could be particularly useful for the application of augmented cognition to human-machine systems.