Real-time single camera natural user interface engine development

Natural user interfaces (NUIs) provide human computer interaction (HCI) with natural and intuitive operation interfaces, such as using human gestures and voice. We have developed a real-time NUI engine architecture using a web camera as a means of implementing NUI applications. The system captures video via the web camera, implements real-time image processing using graphic processing unit (GPU) programming. This paper describes the architecture of the engine and the real-virtual environment interaction methods, such as foreground segmentation and hand gesture recognition. These methods are implemented using GPU programming in order to realize real-time image processing for HCI. To verify the efficacy of our proposed NUI engine, we utilized it in the development and implementation of several mixed reality games and touch-less operation applications, using the developed NUI engine and the DirectX SDK. Our results confirm that the methods implemented by the engine operate in real time and the interactive operations are intuitive.

[1]  Janusz Konrad,et al.  A gesture-driven computer interface using Kinect , 2012, 2012 IEEE Southwest Symposium on Image Analysis and Interpretation.

[2]  Qiang Ling,et al.  A background modeling and foreground segmentation approach based on the feedback of moving objects in traffic surveillance systems , 2014, Neurocomputing.

[3]  Jeong-Ick Lee.  Safety Equipment System for Bicycle , 2013 .

[4]  M. Betke,et al.  The Camera Mouse: visual tracking of body features to provide computer access for people with severe disabilities , 2002, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[5]  Gueesang Lee,et al.  Small Object Segmentation Based on Visual Saliency in Natural Images , 2013, J. Inf. Process. Syst..

[6]  Kanad K. Biswas,et al.  Gesture recognition using Microsoft Kinect® , 2011, The 5th International Conference on Automation, Robotics and Applications.

[7]  Kahlil Muchtar,et al.  Real-time background modeling based on a multi-level texture description , 2014, Inf. Sci..

[8]  Sheng-Siang Huang,et al.  Natural user interface for interactive television , 2013, 2013 IEEE International Symposium on Consumer Electronics (ISCE).

[9]  Hrvoje Benko,et al.  Combining multiple depth cameras and projectors for interactions on, above and between surfaces , 2010, UIST.

[10]  Andrew Hunter,et al.  A spatially distributed model for foreground segmentation , 2009, Image Vis. Comput..

[11]  Akhtar Hussain,et al.  Detecting & interpreting self-manipulating hand movements for student’s affect prediction , 2012, Human-centric Computing and Information Sciences.

[12]  Radek Martinek,et al.  Development and testing of a visualization application software, implemented with wireless control system in smart home care , 2014, Human-centric Computing and Information Sciences.

[13]  Yo-Sung Ho.  Challenging Technical Issues of 3D Video Processing , 2013 .

[14]  Christian Stolte,et al.  The Molecular Control Toolkit: Controlling 3D molecular graphics via gesture and voice , 2013, 2013 IEEE Symposium on Biological Data Visualization (BioVis).

[15]  Mubarak Shah,et al.  Human Action Recognition in Videos Using Kinematic Features and Multiple Instance Learning , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Wei Song,et al.  Intuitive Terrain Reconstruction Using Height Observation-Based Ground Segmentation and 3D Object Boundary Estimation , 2012, Sensors.

[17]  Junsong Yuan,et al.  Robust Part-Based Hand Gesture Recognition Using Kinect Sensor , 2013, IEEE Transactions on Multimedia.

[18]  Jan L. Plass,et al.  Moved to learn: The effects of interactivity in a Kinect-based literacy game for beginning readers , 2014, Comput. Educ..

[19]  Hui Li,et al.  Kinect Based Dynamic Hand Gesture Recognition Algorithm Research , 2012, 2012 4th International Conference on Intelligent Human-Machine Systems and Cybernetics.

[20]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[21]  Kwan-Hee Yoo,et al.  Abstract: A Gesture based Camera Controlling Method in the 3D Virtual Space , 2012 .