Optical Mouse Sensor for Eye Blink Detection and Pupil Tracking: Application in a Low-Cost Eye-Controlled Pointing Device

In this paper, a new application of the optical mouse sensor is presented. The optical mouse is used as a main low-cost infrared vision system of a new proposal of a head-mounted human-computer interaction (HCI) device controlled by eye movements. The default optical mouse sensor lens and illumination source are replaced in order to improve its field of view and capture entire eye images. A complementary 8-bit microcontroller is used to acquire and process these images with two optimized algorithms to detect forced eye blinks and pupil displacements which are translated to computer pointer actions. This proposal introduces an inexpensive and approachable plug and play (PnP) device for people with severe disability in the upper extremities, neck, and head. The presented pointing device performs standard computer mouse actions with no extra software required. It uses the human interface device (HID) standard class of the universal serial bus (USB) increasing its compatibility for most computer platforms. This new device approach is aimed at improving comfortability and portability of the current commercial devices with simple installation and calibration. Several performance tests were done with different volunteer users obtaining an average pupil detection error of 0.34 pixels with a successful detection in 82.6% of all mouse events requested by means of pupil tracking.

[2]  Miguel Carrasco,et al.  Evaluation of an eye-pointer interaction device for human-computer interaction , 2018, Heliyon.

[3]  Pawel Strumillo,et al.  Eye-blink detection system for human–computer interaction , 2011, Universal Access in the Information Society.

[4]  Zahra Hakimi,et al.  SET: a pupil detection method using sinusoidal approximation , 2015, Front. Neuroeng..

[5]  Dongheng Li,et al.  Starburst: A hybrid algorithm for video-based eye tracking combining feature-based and model-based approaches , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[6]  Takashi Watanabe,et al.  Gyro-Mouse for the Disabled: 'Click' and 'Position' Control of the Mouse Cursor , 2007 .

[7]  Margrit Betke,et al.  Communication via eye blinks and eyebrow raises: video-based human-computer interfaces , 2003, Universal Access in the Information Society.

[8]  Leo Pauly,et al.  Non Intrusive Eye Blink Detection from Low Resolution Images Using HOG-SVM Classifier , 2016, International Journal of Image, Graphics and Signal Processing.

[9]  Thiago Santini,et al.  PuReST: robust pupil tracking for real-time pervasive eye tracking , 2018, ETRA.

[10]  Wilbert McClay,et al.  A Magnetoencephalographic/Encephalographic (MEG/EEG) Brain-Computer Interface Driver for Interactive iOS Mobile Videogame Applications Utilizing the Hadoop Ecosystem, MongoDB, and Cassandra NoSQL Databases , 2018, Diseases.

[11]  Fatimah Khalid,et al.  Noncircular iris segmentation based on weighted adaptive hough transform using smartphone database , 2018 .

[12]  Ashok A. Ghatol,et al.  Iris recognition: an emerging biometric technology , 2007 .

[13]  Thiago Santini,et al.  Brightness- and motion-based blink detection for head-mounted eye trackers , 2016, UbiComp Adjunct.

[14]  Maysam Ghovanloo,et al.  Comparing the Use of Single Versus Multiple Combined Abilities in Conducting Complex Computer Tasks Hands-Free , 2018, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[15]  Jordi Palacín,et al.  The optical mouse for indoor mobile robot odometry measurement , 2006 .

[16]  Myung Jin Chung,et al.  Eye-mouse under large head movement for human-computer interface , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[17]  Marcel Tresanchez,et al.  Implementation of a robust absolute virtual head mouse combining face detection, template matching and optical flow algorithms , 2013, Telecommun. Syst..

[18]  Marcel Tresanchez,et al.  Using the Optical Mouse Sensor as a Two-Euro Counterfeit Coin Detector , 2009, Sensors.

[19]  Kennedy O. Okokpujie,et al.  An Improved Iris Segmentation Technique Using Circular Hough Transform , 2018 .

[20]  Sung Ho Cho,et al.  Human–computer interaction using radio sensor for people with severe disability , 2018, Sensors and Actuators A: Physical.

[21]  Howell O. Istance,et al.  Snap clutch, a moded approach to solving the Midas touch problem , 2008, ETRA.

[22]  Eduard Clotet,et al.  An inexpensive wireless smart camera system for IoT applications based on an ARM Cortex-M7 microcontroller , 2019, J. Ubiquitous Syst. Pervasive Networks.

[23]  Marcel Tresanchez,et al.  The optical mouse sensor as an incremental rotary encoder , 2009 .

[24]  T. W. Ng.  The optical mouse as a two-dimensional displacement sensor , 2003 .

[25]  Robert J. K. Jacob,et al.  Eye Movement-Based Human-Computer Interaction Techniques: Toward Non-Command Interfaces , 2003 .

[26]  J. Palacin,et al.  Preliminary study of pupil detection and tracking with low cost optical flow sensors , 2012, 2012 IEEE International Instrumentation and Measurement Technology Conference Proceedings.

[27]  R B Reilly,et al.  Adaptive noncontact gesture-based system for augmentative communication. , 1999, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[28]  Dongheng Li,et al.  openEyes: a low-cost head-mounted eye-tracking solution , 2006, ETRA.

[29]  Hazim G. Daway,et al.  Pupil Detection Based on Color Difference and Circular Hough Transfor , 2018, International Journal of Electrical and Computer Engineering (IJECE).

[30]  Andreas Bulling,et al.  Pupil: an open source platform for pervasive eye tracking and mobile gaze-based interaction , 2014, UbiComp Adjunct.

[31]  Vicente Alarcón Aquino,et al.  Gyroscope-Driven Mouse Pointer with an EMOTIV® EEG Headset and Data Analysis Based on Empirical Mode Decomposition , 2013, Sensors.

[32]  Bin Li,et al.  Etracker: A Mobile Gaze-Tracking System with Near-Eye Display Based on a Combined Gaze-Tracking Algorithm , 2018, Sensors.

[33]  Wenhao Zhang,et al.  Gender and gaze gesture recognition for human-computer interaction , 2016, Comput. Vis. Image Underst..

[34]  Carlos Hitoshi Morimoto,et al.  Episcleral surface tracking: challenges and possibilities for using mice sensors for wearable eye tracking , 2016, ETRA.

[35]  Wolfgang Nutt,et al.  Tongue-mouse for quadriplegics , 1998 .

[36]  Marcel Tresanchez,et al.  Measuring yarn diameter using inexpensive optical sensors , 2010 .

[37]  Zeeshan Ali Haq,et al.  Eye-blink rate detection for fatigue determination , 2016, 2016 1st India International Conference on Information Processing (IICIP).

[38]  Neil A. Dodgson,et al.  Robust real-time pupil tracking in highly off-axis images , 2012, ETRA.

[39]  Marcel Tresanchez,et al.  Using the image acquisition capabilities of the optical mouse sensor to build an absolute rotary encoder , 2010 .

[40]  Thomas S. Huang,et al.  Face as mouse through visual face tracking , 2007, Comput. Vis. Image Underst..

[41]  Mohammad Shahram Moin,et al.  A new approach for iris localization in iris recognition systems , 2008, 2008 IEEE/ACS International Conference on Computer Systems and Applications.

[42]  Mohammad Pooyan,et al.  A New Method for Iris Recognition Systems Based on Fast Pupil Localization , 2008, ICCSA.

[43]  Wanda Benesova,et al.  Eye blink detection based on motion vectors analysis , 2016, Comput. Vis. Image Underst..

[44]  Xueliang Huo,et al.  A Magneto-Inductive Sensor Based Wireless Tongue-Computer Interface , 2008, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[45]  John Daugman,et al.  How iris recognition works , 2002, IEEE Transactions on Circuits and Systems for Video Technology.

[46]  Chern-Sheng Lin,et al.  An eye-tracking and head-control system using movement increment-coordinate method , 2007 .

[47]  P Blenkhorn,et al.  Controlling mouse pointer position using an infrared head-operated joystick. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[48]  Mitsuo Shimotani,et al.  A study of blink detection using bright pupils , 1998 .

[49]  Dong Ming,et al.  Study on EEG-based mouse system by using brain-computer interface , 2009, 2009 IEEE International Conference on Virtual Environments, Human-Computer Interfaces and Measurements Systems.

[50]  Chang-Soo Kim,et al.  Versatile Optochemical Quantification with Optical Mouse , 2017, J. Sensors.

[51]  Daniela Iacoviello,et al.  Robust real time eye tracking for computer interface for disabled people , 2009, Comput. Methods Programs Biomed..

[52]  John Daugman,et al.  High Confidence Visual Recognition of Persons by a Test of Statistical Independence , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[53]  M. Mazo,et al.  Experiences in assisted mobility: the SIAMO project , 2002, Proceedings of the International Conference on Control Applications.

[54]  Chu-Hsing Lin,et al.  Design and implementation of a mouth-controlled mouse , 2011, 2011 IEEE EUROCON - International Conference on Computer as a Tool.

[55]  Wolfgang Rosenstiel,et al.  ExCuSe: Robust Pupil Detection in Real-World Scenarios , 2015, CAIP.

[56]  Thiago Santini,et al.  ElSe: ellipse selection for robust pupil detection in real-world environments , 2015, ETRA.

[57]  Suman Deb,et al.  Brain Computer Interfaced Single Key Omni Directional Pointing and Command System: a Screen Pointing Interface for Differently-abled Person , 2018 .

[58]  Dong-Hoon Yi,et al.  Afocal Optical Flow Sensor for Reducing Vertical Height Sensitivity in Indoor Robot Localization and Navigation , 2015, Sensors.

[59]  Enkelejda Kasneci,et al.  Pupil detection for head-mounted eye tracking in the wild: an evaluation of the state of the art , 2016, Machine Vision and Applications.

[60]  Jiang-Wei Li,et al.  Eye blink detection based on multiple Gabor response waves , 2008, 2008 International Conference on Machine Learning and Cybernetics.

[61]  Mohammed Benaissa,et al.  Robust Eye Blink Detection Based on Eye Landmarks and Savitzky-Golay Filtering , 2018, Inf..

[62]  Oleg V. Komogortsev,et al.  Power-efficient and shift-robust eye-tracking sensor for portable VR headsets , 2019, ETRA.