Interactive display using depth and RGB sensors for face and gesture control

This paper introduces an interactive display system guided by a human observer's gesture, facial pose, and facial expression. The Kinect depth sensor is used to detect and track an observer's skeletal joints while the RGB camera is used for detailed facial analysis. The display consists of active regions that the observer can manipulate with body gestures and secluded regions that are activated through head pose and facial expression. The observer receives realtime feedback allowing for intuitive navigation of the interface. A storefront interactive display was created and feedback was collected from over one hundred subjects. Promising results demonstrate the potential of the proposed approach for human-computer interaction applications.

[1]  Lawrence Cayton,et al.  Algorithms for manifold learning , 2005 .

[2]  Jie-Chi Yang,et al.  Development and Evaluation of an Interactive Mobile Learning Environment with Shared Display Groupware , 2010, J. Educ. Technol. Soc..

[3]  Ali Ghodsi,et al.  Dimensionality Reduction A Short Tutorial , 2006 .

[4]  Andreas E. Savakis,et al.  Facial Expression Recognition Using Facial Features and Manifold Learning , 2010, ISVC.

[5]  Dong Li,et al.  New research advances of facial expression recognition , 2009, 2009 International Conference on Machine Learning and Cybernetics.

[6]  Andreas E. Savakis,et al.  Facial pose estimation using a symmetrical feature model , 2009, 2009 IEEE International Conference on Multimedia and Expo.

[7]  Alex Pentland,et al.  Human computing and machine understanding of human behavior: a survey , 2006, ICMI '06.

[8]  Mohan M. Trivedi,et al.  Head Pose Estimation in Computer Vision: A Survey , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Shaogang Gong,et al.  Appearance Manifold of Facial Expression , 2005, ICCV-HCI.

[10]  Du-Sik Park,et al.  3D user interface combining gaze and hand gestures for large-scale display , 2010, CHI EA '10.

[11]  Andreas Holzinger,et al.  Finger Instead of Mouse: Touch Screens as a Means of Enhancing Universal Access , 2002, User Interfaces for All.

[12]  Xiaofei He,et al.  Locality Preserving Projections , 2003, NIPS.

[13]  Beat Fasel,et al.  Automati Fa ial Expression Analysis: A Survey , 1999 .

[14]  Patrick J. F. Groenen,et al.  A graphical shopping interface based on product attributes , 2007, Decis. Support Syst..

[15]  Nicu Sebe,et al.  Human-Computer Intelligent Interaction: A Survey , 2007, ICCV-HCI.

[16]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.