Real-time facial tracking in virtual reality

Virtual reality (VR) emerges as the next social computing platform. For realizing immersive social interactions, projecting facial expressions onto the virtual avatar a crucial component. This is a challenge in VR as it requires capturing the facial motions behind the VR head mounted displays (HMDs). In this paper, we present a real-time facial expression tracking system in VR HMDs. The core of the system is a 3D camera attached to the HMDs, capturing motions on the lower half of the face, which enables users to track and retarget their facial animations in real-time onto CG avatars. The system is capable of capturing 20 facial expression parameters and transfer it onto the 3D character in real-time.