Spontaneous Expression Detection from 3D Dynamic Sequences by Analyzing Trajectories on Grassmann Manifolds

In this paper, we propose a framework for online spontaneous emotion detection, such as happiness or physical pain, from depth videos. Our approach consists on mapping the video streams onto a Grassmann manifold (i.e., space of <inline-formula><tex-math notation="LaTeX">$k$</tex-math><alternatives> <inline-graphic xlink:href="benamor-ieq1-2623718.gif"/></alternatives></inline-formula>-dimensional linear subspaces) to form time-parameterized trajectories. To this end, depth videos are decomposed into short-time subsequences, each approximated by a <inline-formula><tex-math notation="LaTeX">$k$</tex-math><alternatives> <inline-graphic xlink:href="benamor-ieq2-2623718.gif"/></alternatives></inline-formula>-dimensional linear subspace, which is in turn a point on the Grassmann manifold. Then, the temporal evolution of subspaces gives rise to a precise mathematical representation of trajectories on the underlying manifold. In the final step, extracted spatio-temporal features based on computing the velocity vectors along the trajectories, termed Geometric Motion History (GMH), are fed to an early event detector based on Structured Output SVM, which enables online emotion detection from partially-observed data. Experimental results obtained on the publicly available Cam3D Kinect and BP4D-spontaneous databases validate the proposed solution. The first database has served to exemplify the proposed framework using depth sequences of the upper part of the body collected using depth-consumer cameras, while the second database allowed the application of the same framework to physical pain detection from high-resolution and long 3D-face sequences.