Communication via eye blinks and eyebrow raises: video-based human-computer interfaces

Two video-based human-computer interaction tools are introduced that can activate a binary switch and issue a selection command. “BlinkLink,” as the first tool is called, automatically detects a user’s eye blinks and accurately measures their durations. The system is intended to provide an alternate input modality to allow people with severe disabilities to access a computer. Voluntary long blinks trigger mouse clicks, while involuntary short blinks are ignored. The system enables communication using “blink patterns:” sequences of long and short blinks which are interpreted as semiotic messages. The second tool, “EyebrowClicker,” automatically detects when a user raises his or her eyebrows and then triggers a mouse click. Both systems can initialize themselves, track the eyes at frame rate, and recover in the event of errors. No special lighting is required. The systems have been tested with interactive games and a spelling program. Results demonstrate overall detection accuracy of 95.6% for BlinkLink and 89.0% for EyebrowClicker.

[1]  Marco La Cascia,et al.  Fast, Reliable Head Tracking under Varying Illumination: An Approach Based on Registration of Texture-Mapped 3D Models , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  David G. Stork,et al.  Pattern Classification , 1973 .

[3]  G A Rinard,et al.  An infrared system for determining ocular position. , 1980, ISA transactions.

[4]  Shumeet Baluja,et al.  Non-Intrusive Gaze Tracking Using Artificial Neural Networks , 1993, NIPS.

[5]  Larry S. Davis,et al.  A probabilistic framework for rigid and non-rigid appearance based tracking and recognition , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[6]  Margrit Betke,et al.  Active Detection of Eye Scleras in Real Time , 2000 .

[7]  Azriel Rosenfeld,et al.  Computer Vision , 1988, Adv. Comput..

[8]  Margrit Betke,et al.  Evaluation of tracking methods for human-computer interaction , 2002, Sixth IEEE Workshop on Applications of Computer Vision, 2002. (WACV 2002). Proceedings..

[9]  Alexander H. Waibel,et al.  Estimating focus of attention based on gaze and sound , 2001, PUI '01.

[10]  Carlos Hitoshi Morimoto,et al.  Real-time multiple face detection using active illumination , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[11]  Margrit Betke,et al.  Communication via eye blinks - detection and duration analysis in real time , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[12]  Shumin Zhai,et al.  Manual and gaze input cascaded (MAGIC) pointing , 1999, CHI '99.

[13]  R.J.K. Jacob,et al.  Hot topics-eye-gaze computer interfaces: what you look at is what you get , 1993, Computer.

[14]  Stanley T. Birchfield,et al.  Elliptical head tracking using intensity gradients and color histograms , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[15]  Margrit Betke,et al.  THE CAMERA MOUSE: PRELIMINARY INVESTIGATION OF AUTOMATED VISUAL TRACKING FOR COMPUTER ACCESS , 2000 .

[16]  L. Young,et al.  Survey of eye movement recording methods , 1975 .

[17]  Worthy N. Martin,et al.  Human-computer interaction using eye-gaze input , 1989, IEEE Trans. Syst. Man Cybern..

[18]  Irfan A. Essa,et al.  Detecting and tracking eyes by using their physiological properties, dynamics, and appearance , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[19]  David G. Stork,et al.  Pattern Classification (2nd ed.) , 1999 .

[20]  M. Betke,et al.  The Camera Mouse: visual tracking of body features to provide computer access for people with severe disabilities , 2002, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[21]  Ashish Kapoor,et al.  Real-time, fully automatic upper facial feature tracking , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[22]  Takeo Kanade,et al.  Dual-state parametric eye tracking , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[23]  Shin Yamamoto,et al.  BLINK MEASUREMENT BY IMAGE PROCESSING AND APPLICATION TO WARNING OF DRIVER'S DROWSINESS IN AUTOMOBILES , 1998 .

[24]  Alex Pentland,et al.  Facial expression recognition using a dynamic model and motion energy , 1995, Proceedings of IEEE International Conference on Computer Vision.

[25]  Margrit Betke,et al.  Experiments with a Camera-Based Human-Computer Interface System , 2002 .

[26]  Walter H. Chang,et al.  The new design of an infrared-controlled human-computer interface for the disabled. , 1999, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[27]  J. Kleinman,et al.  Speaking, thinking, and blinking , 1981, Psychiatry Research.

[28]  Richard B. Reilly.  Applications of face and gesture recognition for human-computer interaction , 1998, MULTIMEDIA '98.

[29]  W. J. Perkins,et al.  Control units for operation of computers by severely physically handicapped persons. , 1986, Journal of medical engineering & technology.

[30]  Constantine Stephanidis,et al.  Universal Access in the Information Society: Methods, Tools, and Interaction Technologies , 2001, Universal Access in the Information Society.

[31]  James L. Crowley,et al.  Multi-modal tracking of faces for video communications , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[32]  James Gips,et al.  Using EagleEyes—an electrodes based device for controlling the computer with your eyes—to help people with special needs , 1996 .

[33]  Michael J. Black,et al.  Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion , 1995, Proceedings of IEEE International Conference on Computer Vision.

[34]  Dorin Comaniciu,et al.  Robust detection and tracking of human faces with an active camera , 2000, Proceedings Third IEEE International Workshop on Visual Surveillance.

[35]  Hiroshi Ueno,et al.  Development of drowsiness detection system , 1994, Proceedings of VNIS'94 - 1994 Vehicle Navigation and Information Systems Conference.

[36]  Dimitris N. Metaxas,et al.  The integration of optical flow and deformable models with applications to human face shape and motion estimation , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[37]  Ravi Kothari,et al.  Detection of eye locations in unconstrained visual images , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[38]  P Blenkhorn,et al.  Controlling mouse pointer position using an infrared head-operated joystick. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[39]  James W. Davis,et al.  The Recognition of Human Movement Using Temporal Templates , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[40]  Linda G. Shapiro,et al.  Computer Vision , 2001 .

[41]  Bernd Girod,et al.  Model-based face tracking for view-independent facial expression recognition , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[42]  LUIGI STRINGA,et al.  Eyes detection for face recognition , 1993, Appl. Artif. Intell..

[43]  James W. Davis,et al.  The representation and recognition of human movement using temporal templates , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[44]  James W. Davis,et al.  The Representation and Recognition of Action Using Temporal Templates , 1997, CVPR 1997.

[45]  Timothy F. Cootes,et al.  Learning to identify and track faces in image sequences , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[46]  Margrit Betke,et al.  Early experiences using visual tracking for computer access by people with profound physical disabilities , 2001, HCI.