Connecting users to virtual worlds within MPEG-V standardization

Virtual world such as Second life and 3D internet/broadcasting services have been increasingly popular. A life-scale virtual world presentation and the intuitive interaction between the users and the virtual worlds would provide more natural and immersive experience for users. The emergence of novel interaction technologies, such as facial-expression/body-motion tracking and remote interaction for virtual object manipulation, could be used to provide a strong connection between users in the real world and avatars in the virtual world. For the wide acceptance and the use of the virtual world, various types of novel interaction devices should have a unified interaction format between the real world and the virtual world. Thus, MPEG-V Media Context and Control (ISO/IEC 23005) standardizes such connecting information. The paper provides an overview and its usage example of MPEG-V from the real world to the virtual world (R2V) on interfaces for controlling avatars and virtual objects in the virtual world by the real world devices. In particular, we investigate how the MPEG-V framework can be applied for the facial animation and hand-based 3D manipulation using intelligent camera. In addition, in order to intuitively manipulate objects in a 3D virtual environment, we present two interaction techniques using motion sensors such as a two-handed spatial 3D interaction approach and a gesture-based interaction approach.

[1]  Chin-Hui Lee,et al.  Application of hidden Markov models for recognition of a limited set of words in unconstrained speech , 1989, International Conference on Acoustics, Speech, and Signal Processing,.

[2]  Du-Sik Park,et al.  3D remote interface for smart displays , 2011, CHI EA '11.

[3]  Fadi Dornaika,et al.  On Appearance Based Face and Facial Action Tracking , 2006, IEEE Transactions on Circuits and Systems for Video Technology.

[4]  Xavier Provot,et al.  Deformation Constraints in a Mass-Spring Model to Describe Rigid Cloth Behavior , 1995 .

[5]  Xia Liu,et al.  Sign recognition using depth image streams , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[6]  Hyunjeong Lee,et al.  Remote interaction for 3D manipulation , 2010, CHI EA '10.

[7]  Michel Beaudouin-Lafon,et al.  Charade: remote control of objects using free-hand gestures , 1993, CACM.

[8]  Jiangwen Deng,et al.  An HMM-based approach for gesture segmentation and recognition , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[9]  Keechul Jung,et al.  Recognition-based gesture spotting in video games , 2004, Pattern Recognit. Lett..

[10]  Franck Davoine,et al.  A linear estimation method for 3D pose and facial animation tracking , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Taehyun Rhee,et al.  Real-time facial animation from live video tracking , 2011, SCA '11.

[12]  Ronald Fedkiw,et al.  Level set methods and dynamic implicit surfaces , 2002, Applied mathematical sciences.

[13]  R. Oka,et al.  Spotting Recognition of Human Gestures from Motion Images , 1994 .

[14]  Marius Preda,et al.  Avatar interoperability and control in virtual Worlds , 2013, Signal Process. Image Commun..

[15]  Hyong-Euk Lee,et al.  High-precision 6-DOF motion tracking architecture with compact low-cost sensors for 3D manipulation , 2012, 2012 IEEE International Conference on Consumer Electronics (ICCE).

[16]  Ayoub Al-Hamadi,et al.  Posture Recognition using Combined Statistical and Geometrical Feature Vectors based on SVM , 2009 .

[17]  Simon Baker,et al.  Active Appearance Models Revisited , 2004, International Journal of Computer Vision.

[18]  Xia Liu,et al.  Hand gesture recognition using depth data , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[19]  Andrew S. Forsberg,et al.  Image plane interaction techniques in 3D immersive environments , 1997, SI3D.

[20]  Ulrich Neumann,et al.  Real-time Hand Pose Recognition Using Low-Resolution Depth Images , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[21]  Daijin Kim,et al.  Simultaneous Gesture Segmentation and Recognition based on Forward Spotting Accumulative HMMs , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[22]  Jin-Hyung Kim,et al.  An HMM-Based Threshold Model Approach for Gesture Recognition , 1999, IEEE Trans. Pattern Anal. Mach. Intell..