Eye-Tracking Metrics Predict Perceived Workload in Robotic Surgical Skills Training

Objective The aim of this study is to assess the relationship between eye-tracking measures and perceived workload in robotic surgical tasks. Background Robotic techniques provide improved dexterity, stereoscopic vision, and ergonomic control system over laparoscopic surgery, but the complexity of the interfaces and operations may pose new challenges to surgeons and compromise patient safety. Limited studies have objectively quantified workload and its impact on performance in robotic surgery. Although not yet implemented in robotic surgery, minimally intrusive and continuous eye-tracking metrics have been shown to be sensitive to changes in workload in other domains. Methods Eight surgical trainees participated in 15 robotic skills simulation sessions. In each session, participants performed up to 12 simulated exercises. Correlation and mixed-effects analyses were conducted to explore the relationships between eye-tracking metrics and perceived workload. Machine learning classifiers were used to determine the sensitivity of differentiating between low and high workload with eye-tracking features. Results Gaze entropy increased as perceived workload increased, with a correlation of .51. Pupil diameter and gaze entropy distinguished differences in workload between task difficulty levels, and both metrics increased as task level difficulty increased. The classification model using eye-tracking features achieved an accuracy of 84.7% in predicting workload levels. Conclusion Eye-tracking measures can detect perceived workload during robotic tasks. They can potentially be used to identify task contributors to high workload and provide measures for robotic surgery training. Application Workload assessment can be used for real-time monitoring of workload in robotic surgical training and provide assessments for performance and learning.

[1]  M. Zenati,et al.  Systematic review of measurement tools to assess surgeons' intraoperative cognitive workload , 2018, The British journal of surgery.

[2]  J. Bakdash,et al.  Repeated Measures Correlation , 2017, Front. Psychol..

[3]  Denny Yu,et al.  Impact of novel shift handle laparoscopic tool on wrist ergonomics and task performance , 2016, Surgical Endoscopy.

[4]  E. N. Corlett,et al.  Evaluation of human work : a practical ergonomics methodology , 1991 .

[5]  Herbert A. Colle,et al.  Estimating a Mental Workload Redline in a Simulated Air-to-Ground Combat Mission , 2005 .

[6]  J. Beatty,et al.  The pupillary system. , 2000 .

[7]  D. L. Forkey,et al.  The effect of laparoscopic instrument working angle on surgeons’ upper extremity workload , 2001, Surgical Endoscopy.

[8]  A. Meraney,et al.  da Vinci Skills Simulator construct validation study: correlation of prior robotic experience with overall score and time score simulator performance. , 2012, Urology.

[9]  D. Kahneman,et al.  Pupillary changes in two memory tasks , 1966 .

[10]  Pierre Jannin,et al.  A Framework for the Recognition of High-Level Surgical Tasks From Video Images for Cataract Surgeries , 2012, IEEE Transactions on Biomedical Engineering.

[11]  Sarah Sharples,et al.  Physiological Parameter Response to Variation of Mental Workload , 2017, Hum. Factors.

[12]  Jasbir Kaur,et al.  Eye tracking based driver fatigue monitoring and warning system , 2011, India International Conference on Power Electronics 2010 (IICPE2010).

[13]  Guang-Zhong Yang,et al.  Eye tracking for skills assessment and training: a systematic review. , 2014, The Journal of surgical research.

[14]  Zied Elouedi,et al.  Naive Bayes vs decision trees in intrusion detection systems , 2004, SAC '04.

[15]  Trevor Hastie,et al.  Model Assessment and Selection , 2009 .

[16]  Donald L Fisher,et al.  Eye Tracking: A Novel Approach for Evaluating and Improving the Safety of Healthcare Processes in the Simulated Setting , 2016, Simulation in healthcare : journal of the Society for Simulation in Healthcare.

[17]  Hasan Ayaz,et al.  Optical brain monitoring for operator training and mental workload assessment , 2012, NeuroImage.

[18]  Jonathan Allsop,et al.  Flying under pressure: Effects of anxiety on attention and gaze behavior in aviation , 2014 .

[19]  A. Lanfranco,et al.  Robotic Surgery: A Current Perspective , 2004, Annals of surgery.

[20]  Yili Liu,et al.  Queueing Network-Model Human Processor (QN-MHP): A computational architecture for multitask performance in human-machine systems , 2006, TCHI.

[21]  Ahmed M. Zihni,et al.  Ergonomic analysis of robot-assisted and traditional laparoscopic procedures , 2014, Surgical Endoscopy.

[22]  G Hubens,et al.  What Have we Learnt after Two Years Working with the Da Vinci Robot System in Digestive Surgery ? , 2004, Acta chirurgica Belgica.

[23]  M. S. Atkins,et al.  Analysis of eye gaze: Do novice surgeons look at the same location as expert surgeons during a laparoscopic operation? , 2012, Surgical Endoscopy.

[24]  Mark R. Wilson,et al.  Psychomotor control in a virtual laparoscopic surgery training environment: gaze control parameters differentiate novices from experts , 2010, Surgical Endoscopy.

[25]  Andrew McCallum,et al.  A comparison of event models for naive bayes text classification , 1998, AAAI 1998.

[26]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[27]  Ying Wang,et al.  The Impact of Systematic Variation of Cognitive Demand on Drivers' Visual Attention across Multiple Age Groups , 2010 .

[28]  Prithima R Mosaly,et al.  Subjective and objective quantification of physician's workload and performance during radiation therapy planning tasks. , 2013, Practical radiation oncology.

[29]  Albi,et al.  Cognitive load measurement while driving , 2013 .

[30]  Guang-Zhong Yang,et al.  Collaborative eye tracking: a potential training tool in laparoscopic surgery , 2012, Surgical Endoscopy.

[31]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[32]  S. Horgan,et al.  Solving the surgeon ergonomic crisis with surgical exosuit , 2017, Surgical Endoscopy.

[33]  Licheng Jiao,et al.  Feature Scaling for Kernel Fisher Discriminant Analysis Using Leave-One-Out Cross Validation , 2006, Neural Computation.

[34]  L. L. Di Stasi,et al.  Gaze entropy reflects surgical task load , 2016, Surgical Endoscopy.

[35]  R. Lyons,et al.  Quantifying Intraoperative Workloads Across the Surgical Team Roles: Room for Better Balance? , 2016, World Journal of Surgery.

[36]  Mark R. Wilson,et al.  Surgeons’ display reduced mental effort and workload while performing robotically assisted surgical tasks, when compared to conventional laparoscopy , 2015, Surgical Endoscopy.

[37]  P. Jorna.  Spectral analysis of heart rate and psychological state: A review of its validity as a workload index , 1992, Biological Psychology.

[38]  A. Moinzadeh,et al.  Face, content, and construct validity of dV-trainer, a novel virtual reality simulator for robotic surgery. , 2009, Urology.

[39]  K. Catchpole,et al.  Effects of Flow Disruptions on Mental Workload and Surgical Performance in Robotic-Assisted Surgery , 2018, World Journal of Surgery.

[40]  L. L. Di Stasi,et al.  Quantifying the cognitive cost of laparo-endoscopic single-site surgeries: Gaze-based indices. , 2017, Applied ergonomics.

[41]  P. Mosaly,et al.  Quantification of baseline pupillary response and task-evoked pupillary response during constant and incremental task load , 2017, Ergonomics.

[42]  A. Hemal,et al.  Ergonomic problems associated with laparoscopy. , 2001, Journal of endourology.

[43]  Andrew L. Kun,et al.  Estimating cognitive load using remote eye tracking in a driving simulator , 2010, ETRA.

[44]  A. Darzi,et al.  Dexterity enhancement with robotic surgery , 2004, Surgical Endoscopy And Other Interventional Techniques.

[45]  A. H. Roscoe,et al.  Heart rate as a psychophysiological measure for in-flight workload assessment. , 1993, Ergonomics.

[46]  S. Padmavathi.  Applying Naive Bayes Data Mining Technique for Classification of Agricultural Land Soils , 2009 .

[47]  M. Golz,et al.  Evaluation of PERCLOS based current fatigue monitoring technologies , 2010, 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology.

[48]  Yu Tian,et al.  Eye Tracking for Assessment of Mental Workload and Evaluation of RVD Interface , 2018, Man-Machine-Environment System Engineering.

[49]  Robin K. Morris,et al.  Eye movement guidance in reading: the role of parafoveal letter and space information. , 1990, Journal of experimental psychology. Human perception and performance.

[50]  E. Granholm,et al.  Pupillometric measures of cognitive and emotional processes. , 2004, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[51]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[52]  N. Dingemanse,et al.  Quantifying individual variation in behaviour: mixed-effect modelling approaches. , 2013, The Journal of animal ecology.

[53]  R Darin Ellis,et al.  NASA TLX: Software for assessing subjective mental workload , 2009, Behavior research methods.

[54]  Prithima R Mosaly,et al.  Relating physician's workload with errors during radiation therapy planning. , 2014, Practical radiation oncology.

[55]  Xianta Jiang,et al.  Detection of Changes in Surgical Difficulty , 2015, Surgical innovation.

[56]  Gyusung I. Lee,et al.  Comparative assessment of physical and cognitive ergonomics associated with robotic and traditional laparoscopic surgeries , 2014, Surgical Endoscopy.

[57]  R J Fairbanks,et al.  RESEARCH ON VEHICLE-BASED DRIVER STATUS/PERFORMANCE MONITORING; DEVELOPMENT, VALIDATION, AND REFINEMENT OF ALGORITHMS FOR DETECTION OF DRIVER DROWSINESS. FINAL REPORT , 1994 .

[58]  Michela Terenzi,et al.  A Random Glance at the Flight Deck: Pilots' Scanning Strategies and the Real-Time Assessment of Mental Workload , 2007 .

[59]  B. Cain.  A Review of the Mental Workload Literature , 2007 .

[60]  M. Susan Hallbeck,et al.  Overview of Human Factors and Ergonomics in the OR, with an Emphasis on Minimally Invasive Surgeries , 2014 .

[61]  M. Pomplun,et al.  Pupil Dilation as an Indicator of Cognitive Workload in Human-Computer Interaction , 2003 .

[62]  W. B. Seales,et al.  Assessing Mental Workload During Laparoscopic Surgery , 2005, Surgical innovation.

[63]  M. Stella Atkins,et al.  Workload assessment of surgeons: correlation between NASA TLX and blinks , 2012, Surgical Endoscopy.

[64]  Peter A Hancock,et al.  State of science: mental workload in ergonomics , 2015, Ergonomics.

[65]  J. Beatty.  Task-evoked pupillary responses, processing load, and the structure of processing resources. , 1982, Psychological bulletin.

[66]  Allen Newell,et al.  The model human processor: An engineering model of human performance. , 1986 .

[67]  Tjerk de Greef,et al.  Eye Movement as Indicators of Mental Workload to Trigger Adaptive Automation , 2009, HCI.

[68]  Garth H Ballantyne,et al.  The pitfalls of laparoscopic surgery: challenges for robotics and telerobotic surgery. , 2002, Surgical laparoscopy, endoscopy & percutaneous techniques.

[69]  Rebecca A. Grier.  How High is High? A Meta-Analysis of NASA-TLX Global Workload Scores , 2015 .

[70]  R. van Hillegersberg,et al.  Minimally invasive surgery compared to open procedures in esophagectomy for cancer: a systematic review of the literature. , 2009, Minerva chirurgica.

[71]  James C. Christensen,et al.  Classifying Workload with Eye Movements in a Complex Task , 2012 .

[72]  Peter Nickel,et al.  Sensitivity and Diagnosticity of the 0.1-Hz Component of Heart Rate Variability as an Indicator of Mental Workload , 2003, Hum. Factors.

[73]  Laura Drudi,et al.  Validation of the da Vinci Surgical Skill Simulator across three surgical disciplines: A pilot study. , 2013, Canadian Urological Association journal = Journal de l'Association des urologues du Canada.

[74]  R L Harris,et al.  Visual scanning behavior and pilot workload. , 1982, Aviation, space, and environmental medicine.

[75]  M. Hallbeck,et al.  Human factors in robotic assisted surgery: Lessons from studies 'in the Wild'. , 2019, Applied ergonomics.

[76]  Somayeh B. Shafiei,et al.  Understanding Cognitive Performance During Robot-Assisted Surgery. , 2015, Urology.

[77]  Joost C. F. de Winter,et al.  Review of eye-related measures of drivers’ mental workload , 2015 .

[78]  Gregory Wilding,et al.  Cognitive skills assessment during robot‐assisted surgery: separating the wheat from the chaff , 2015, BJU international.

[79]  Mark R. Wilson,et al.  Gaze training enhances laparoscopic technical skill acquisition and multi-tasking performance: a randomized, controlled study , 2011, Surgical Endoscopy.

[80]  Ramon Berguer,et al.  A comparison of the physical effort required for laparoscopic and open surgical techniques. , 2003, Archives of surgery.

[81]  Jani Koskinen,et al.  Blink-Based Estimation of Suturing Task Workload and Expertise in Microsurgery , 2018, 2018 IEEE 31st International Symposium on Computer-Based Medical Systems (CBMS).

[82]  Nir Friedman,et al.  Bayesian Network Classifiers , 1997, Machine Learning.

[83]  J. Dankelman,et al.  Surgical flow disturbances in dedicated minimally invasive surgery suites: an observational study to assess its supposed superiority over conventional suites , 2016, Surgical Endoscopy.

[84]  M. A. Recarte,et al.  Effects of verbal and spatial-imagery tasks on eye fixations while driving. , 2000, Journal of experimental psychology. Applied.

[85]  Mansour Rahimi,et al.  Techniques in mental workload assessment. , 1995 .

[86]  S. Horgan,et al.  A prospective analysis of 211 robotic-assisted surgical procedures , 2003, Surgical Endoscopy And Other Interventional Techniques.

[87]  P G Jorna,et al.  Heart rate and workload variations in actual and simulated flight. , 1993, Ergonomics.

[88]  Pedro M. Domingos,et al.  On the Optimality of the Simple Bayesian Classifier under Zero-One Loss , 1997, Machine Learning.

[89]  R. Muradore,et al.  Robotic Surgery , 2011, IEEE Robotics & Automation Magazine.

[90]  Jacques Felblinger,et al.  The virtual reality simulator dV-Trainer® is a valid assessment tool for robotic surgical skills , 2012, Surgical Endoscopy.

[91]  M. Forsman,et al.  Intraoperative workload in robotic surgery assessed by wearable motion tracking sensors and questionnaires , 2017, Surgical Endoscopy.

[92]  S. Miyake.  Multivariate workload evaluation combining physiological and subjective measures. , 2001, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[93]  S. Sawilowsky.  New Effect Size Rules of Thumb , 2009 .

[94]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[95]  N. Laird,et al.  Using the general linear mixed model to analyse unbalanced repeated measures and longitudinal data. , 1997, Statistics in medicine.

[96]  Sandra G. Hart,et al.  Nasa-Task Load Index (NASA-TLX); 20 Years Later , 2006 .

[97]  T. Robinson,et al.  Minimally invasive surgery , 1999, European Surgical Research.

[98]  J. Bisley,et al.  Evaluating tactile feedback in robotic surgery for potential clinical application using an animal model , 2016, Surgical Endoscopy.