A color-based face tracking algorithm for enhancing interaction with mobile devices

A color-based face tracking algorithm is proposed to be used as a human-computer interaction tool on mobile devices. The solution provides a natural means of interaction enabling a motion parallax effect in applications. The algorithm considers the characteristics of mobile use-constrained computational resources and varying environmental conditions. The solution is based on color comparisons and works on images gathered from the front camera of a device. In addition to color comparisons, the coherency of the facial pixels is considered in the algorithm. Several applications are also demonstrated in this work, which use the face position to determine the viewpoint in a virtual scene, or for browsing large images. The accuracy of the system is tested under different environmental conditions such as lighting and background, and the performance of the system is measured in different types of mobile devices. According to these measurements the system allows for accurate (7% RMS error) face tracking in real time (20–100 fps).

[1]  Vidya Setlur,et al.  Camera-Based Virtual Environment Interaction on Mobile Devices , 2006, ISCIS.

[2]  Nicu Sebe,et al.  Multimodal Human Computer Interaction: A Survey , 2005, ICCV-HCI.

[3]  King Ngi Ngan,et al.  Face segmentation using skin-color map in videophone applications , 1999, IEEE Trans. Circuits Syst. Video Technol..

[4]  Colin Ware,et al.  Space perception and the display of data in space , 2004 .

[5]  Janne Heikkilä,et al.  Adaptive Motion-Based Gesture Recognition Interface for Mobile Phones , 2008, ICVS.

[6]  Sami Huttunen,et al.  Motion-based finger tracking for user interaction with mobile devices , 2007 .

[7]  Peter Shirley,et al.  Fundamentals of computer graphics , 2018 .

[8]  Jari Hannuksela,et al.  A vision based motion interface for mobile phones , 2007 .

[9]  S. Kamolphiwong,et al.  Blink and click , 2007, i-CREATe '07.

[10]  Vladimir Vezhnevets,et al.  A Survey on Pixel-Based Skin Color Detection Techniques , 2003 .

[11]  Ka-Ping Yee,et al.  Peephole displays: pen interaction on spatially aware handheld computers , 2003, CHI '03.

[12]  Colin Ware,et al.  Information Visualization: Perception for Design , 2000 .

[13]  Alex Waibel,et al.  Face locating and tracking for human-computer interaction , 1994, Proceedings of 1994 28th Asilomar Conference on Signals, Systems and Computers.

[14]  Tomas Akenine-Möller,et al.  The State of the Art in Mobile Graphics Research , 2008, IEEE Computer Graphics and Applications.

[15]  Tolga K. Çapin,et al.  A Face Tracking Algorithm for User Interaction in Mobile Devices , 2009, 2009 International Conference on CyberWorlds.

[16]  Nikolaos G. Bourbakis,et al.  A survey of skin-color modeling and detection methods , 2007, Pattern Recognit..

[17]  Jason Brand,et al.  A comparative assessment of three approaches to pixel-level human skin-detection , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[18]  Erik Hjelmås,et al.  Face Detection: A Survey , 2001, Comput. Vis. Image Underst..

[19]  Tolga K. Çapin,et al.  Mobile Camera-Based User Interaction , 2005, ICCV-HCI.

[20]  Janne Heikkilä,et al.  A Vision-Based Approach for Controlling User Interfaces of Mobile Devices , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[21]  Gary Bradski,et al.  Computer Vision Face Tracking For Use in a Perceptual User Interface , 1998 .