A Prototype for Practical Eye-Gaze Corrected Video Chat on Graphics Hardware

We present a fully functional prototype to convincingly restore eye contact between two video chat participants, with a minimal amount of constraints. The proposed six-fold camera setup is easily integrated into the monitor frame, and is used to interpolate an image as if its virtual camera captured the image through a transparent screen. The peer user has a large freedom of movement, resulting in system specifications that enable genuine practical usage. Our software framework thereby harnesses the powerful computational resources inside graphics hardware, to achieve real-time performance up to 30 frames per second for 800×600 resolution images. Furthermore, an optimal set of finetuned parameters are presented, that optimizes the end-to-end performance of the application, and therefore is still able to achieve high subjective visual quality.

[1]  Tomás Svoboda,et al.  A Convenient Multicamera Self-Calibration for Virtual Environments , 2005, Presence: Teleoperators & Virtual Environments.

[2]  Liang-Gee Chen,et al.  Efficient stereo video coding system for immersive teleconference with two-stage hybrid disparity estimation algorithm , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[3]  Jens H. Krüger,et al.  A Survey of General‐Purpose Computation on Graphics Hardware , 2007, Eurographics.

[4]  Luc Van Gool,et al.  Extended view interpolation by parallel use of the GPU and the CPU , 2005 .

[5]  L. Van Gool,et al.  Fast interpolated cameras by combining a GPU based plane sweep with a max-flow regularisation algorithm , 2004 .

[6]  Dean Brown,et al.  Decentering distortion of lenses , 1966 .

[7]  Ruigang Yang,et al.  Real‐Time Consensus‐Based Scene Reconstruction Using Commodity Graphics Hardware † , 2003, Comput. Graph. Forum.

[8]  Ruigang Yang,et al.  Fast Image Segmentation and Smoothing Using Commodity Graphics Hardware , 2002, J. Graphics, GPU, & Game Tools.

[9]  Kentaro Toyama,et al.  Gaze-awareness for Videoconferencing: A Software Approach , 2000 .

[10]  Bang Jun Lei,et al.  Real-Time Multi-Step View Reconstruction for a Virtual Teleconference System , 2002, EURASIP J. Adv. Signal Process..

[11]  Jiangbo Lu,et al.  A Scalable End-to-End Optimized Real-Time Image-Based Rendering Framework on Graphics Hardware , 2008, 2008 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[12]  Didier Arquès,et al.  Real-time Plane-sweep with Local Strategy , 2006, J. WSCG.

[13]  Andrew Blake,et al.  Gaze manipulation for one-to-one teleconferencing , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[14]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[15]  Thomas Malzbender,et al.  The Coliseum Immersive Teleconferencing System , 2002 .

[16]  Oliver Schreer,et al.  A virtual 3D Video-Conference System Providing Semi-Immersive Telepresence: A Real-Time Solution in Hardware and Software , 2001 .

[17]  Ruigang Yang,et al.  Multi-resolution real-time stereo on commodity graphics hardware , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[18]  Anil K. Jain,et al.  Face Detection in Color Images , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  Wen Gao,et al.  Motion vector prediction in multiview video coding , 2005, IEEE International Conference on Image Processing 2005.