Virtual Conversation with Real-Time Prediction of Body Moments/Gestures on Video Streaming Data

The exisitng conversation system where the user interacts with the virtual system with voice and virtual system replies to the user based on what user speaks. In this context whenever user makes some gestures to communicate with the virtual system, the virtual system will miss out those communications. For example, user instead of speaking, may nod head for “yes” or “no” and user can also use hand signals to respond to the virtual system. If these events are not addressed then the conversation is not very interactive and natural human-like interaction will start losing important information. The paper describes how the user body moments/gestures will help effective conversation with the virtual system and virtual conversation system can understand the user misspelled conversation, missed conversation effectively with user gesture/body movements.

[1]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[2]  K.C. Ng,et al.  Music via motion: transdomain mapping of motion and sound for interactive performances , 2004, Proceedings of the IEEE.

[3]  Mohan M. Trivedi,et al.  Hand Gesture Recognition in Real Time for Automotive Interfaces: A Multimodal Vision-Based Approach and Evaluations , 2014, IEEE Transactions on Intelligent Transportation Systems.

[4]  James J. Kuffner,et al.  Goal-Directed Navigation for Animated Characters Using Real-Time Path Planning and Control , 1998, CAPTECH.

[5]  Hong Wei,et al.  A survey of human motion analysis using depth imagery , 2013, Pattern Recognit. Lett..

[6]  Tieniu Tan,et al.  Real-time hand tracking using a mean shift embedded particle filter , 2007, Pattern Recognit..

[7]  Pietro Zanuttigh,et al.  Hand gesture recognition with leap motion and kinect devices , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[8]  Z. Liu,et al.  A real time system for dynamic hand gesture recognition with a depth sensor , 2012, 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO).

[9]  Hideki Koike,et al.  Real-time human motion forecasting using a RGB camera , 2018, VRST.

[10]  Norman I. Badler,et al.  Real-time virtual humans , 1997, Proceedings The Fifth Pacific Conference on Computer Graphics and Applications.

[11]  Pavlo Molchanov,et al.  Hand gesture recognition with 3D convolutional neural networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).