Interacting with Objects in the Environment by Gaze and Hand Gestures

A head-mounted wireless gaze tracker in the form of gaze tracking glasses is used here for continuous and mobile monitoring of a subject’s point of regard on the surrounding environment. We combine gaze tracking and hand gesture recognition to allow a subject to interact with objects in the environment by gazing at them, and controlling the object using hand gesture commands. The gaze tracking glasses was made from low-cost hardware consisting of a safety glasses’ frame and wireless eye tracking and scene cameras. An open source gaze estimation algorithm is used for eye tracking and user’s gaze estimation. A visual markers recognition library is used to identify objects in the environment through the scene camera. A hand gesture classification algorithm is used to recognize hand-based control commands. When combining all these elements the emerging system permits a subject to move freely in an environment, select the object he wants to interact with using gaze (identification) and transmit a command to it by performing a hand gesture (control). The system identifies the target for interaction by using visual markers. This innovative HCI paradigm opens up new forms of interaction with objects in smart environments.

[1]  Dan Witzner Hansen,et al.  Parallax error in the monocular head-mounted eye trackers , 2012, UbiComp.

[2]  Dan Witzner Hansen,et al.  Eye-based head gestures , 2012, ETRA.

[3]  Francisco B. Rodríguez,et al.  Extending the bioinspired hierarchical temporal memory paradigm for sign language recognition , 2012, Neurocomputing.

[4]  Manolis I. A. Lourakis,et al.  Real-Time Tracking of Multiple Skin-Colored Objects with a Possibly Moving Camera , 2004, ECCV.

[5]  I. Scott MacKenzie,et al.  Eye typing using word and letter prediction and a fixation algorithm , 2008, ETRA.

[6]  Francisco B. Rodríguez,et al.  Optimizing Hierarchical Temporal Memory for Multivariable Time Series , 2010, ICANN.

[7]  Shumin Zhai,et al.  Manual and gaze input cascaded (MAGIC) pointing , 1999, CHI '99.

[8]  Albrecht Schmidt,et al.  Interacting with the Computer Using Gaze Gestures , 2007, INTERACT.

[9]  Poika Isokoski,et al.  Text input methods for eye trackers using off-screen targets , 2000, ETRA.

[10]  J. A. Adam,et al.  Virtual reality is for real , 1993 .

[11]  Maribeth Gandy Coleman,et al.  The Gesture Pendant: A Self-illuminating, Wearable, Infrared Computer Vision System for Home Automation Control and Medical Monitoring , 2000, Digest of Papers. Fourth International Symposium on Wearable Computers.

[12]  John Paulin Hansen,et al.  Single gaze gestures , 2010, ETRA '10.

[13]  Francisco B. Rodríguez,et al.  Low cost remote gaze gesture recognition in real time , 2012, Appl. Soft Comput..

[14]  Rajeev Sharma,et al.  Experimental evaluation of vision and speech based multimodal interfaces , 2001, PUI '01.

[15]  Andy Cockburn,et al.  FingARtips: gesture based direct manipulation in Augmented Reality , 2004, GRAPHITE '04.

[16]  Howell O. Istance,et al.  Designing gaze gestures for gaming: an investigation of performance , 2010, ETRA.

[17]  Didier Stricker,et al.  Detection and Identification Techniques for Markers Used in Computer Vision , 2010, VLUDS.

[18]  Francisco B. Rodríguez,et al.  Gliding and saccadic gaze gesture recognition in real time , 2012, TIIS.

[19]  Colin Ware,et al.  An evaluation of an eye tracker as a device for computer input2 , 1987, CHI 1987.

[20]  Alexander De Luca,et al.  Evaluation of eye-gaze interaction methods for security enhanced PIN-entry , 2007, OZCHI '07.

[21]  Loren G. Terveen,et al.  The sound of one hand: a wrist-mounted bio-acoustic fingertip gesture interface , 2002, CHI Extended Abstracts.

[22]  Richard A. Bolt.  Eyes at the interface , 1982, CHI '82.

[23]  Dan Witzner Hansen,et al.  Mobile gaze-based screen interaction in 3D environments , 2011, NGCA '11.

[24]  Francisco B. Rodríguez,et al.  Gaze Gesture Recognition with Hierarchical Temporal Memory Networks , 2011, IWANN.