The Virtual Operative Assistant: An explainable artificial intelligence tool for simulation-based training in surgery and medicine

Simulation-based training is increasingly being used for assessment and training of psychomotor skills involved in medicine. The application of artificial intelligence and machine learning technologies has provided new methodologies to utilize large amounts of data for educational purposes. A significant criticism of the use of artificial intelligence in education has been a lack of transparency in the algorithms’ decision-making processes. This study aims to 1) introduce a new framework using explainable artificial intelligence for simulation-based training in surgery, and 2) validate the framework by creating the Virtual Operative Assistant, an automated educational feedback platform. Twenty-eight skilled participants (14 staff neurosurgeons, 4 fellows, 10 PGY 4–6 residents) and 22 novice participants (10 PGY 1–3 residents, 12 medical students) took part in this study. Participants performed a virtual reality subpial brain tumor resection task on the NeuroVR simulator using a simulated ultrasonic aspirator and bipolar. Metrics of performance were developed, and leave-one-out cross validation was employed to train and validate a support vector machine in Matlab. The classifier was combined with a unique educational system to build the Virtual Operative Assistant which provides users with automated feedback on their metric performance with regards to expert proficiency performance benchmarks. The Virtual Operative Assistant successfully classified skilled and novice participants using 4 metrics with an accuracy, specificity and sensitivity of 92, 82 and 100%, respectively. A 2-step feedback system was developed to provide participants with an immediate visual representation of their standing related to expert proficiency performance benchmarks. The educational system outlined establishes a basis for the potential role of integrating artificial intelligence and virtual reality simulation into surgical educational teaching. The potential of linking expertise classification, objective feedback based on proficiency benchmarks, and instructor input creates a novel educational tool by integrating these three components into a formative educational paradigm.

[1]  R. Aggarwal,et al.  Assessing Technical Competence in Surgical Trainees: A Systematic Review. , 2015, Annals of surgery.

[2]  Qiang Liu,et al.  Endoscopic Image Classification Based on DWT-CM and Improved BNN for Surgical Tool Appearances , 2007, 2007 International Conference on Machine Learning and Cybernetics.

[3]  D. Norman.  The Design of Everyday Things: Revised and Expanded Edition , 2013 .

[4]  Abdulgadir Bugdadi,et al.  The force pyramid: a spatial analysis of force application during virtual reality brain tumor resection. , 2017, Journal of neurosurgery.

[5]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[6]  K. Bajunaid,et al.  Assessing Neurosurgical Psychomotor Performance : Role of Virtual Reality Simulators , Current and Future Potential , 2015 .

[7]  Huan Liu,et al.  Toward integrating feature selection algorithms for classification and clustering , 2005, IEEE Transactions on Knowledge and Data Engineering.

[8]  Jeffrey S. Raskin,et al.  Use of Risk Model for Assessment of Residents' Perception of Complexity of Surgical Steps: Example of Modular Component Steps of Lumbar Spinal Fusion Surgery. , 2015, Operative neurosurgery.

[9]  Bernard F. Buxton,et al.  Drug Design by Machine Learning: Support Vector Machines for Pharmaceutical Data Analysis , 2001, Comput. Chem..

[10]  R. Calvo,et al.  Positive Computing: Technology for Wellbeing and Human Potential , 2014, Psychology Teaching Review.

[11]  G. Sutherland,et al.  Supplementary Educational Models in Canadian Neurosurgery Residency Programs , 2016, Canadian Journal of Neurological Sciences / Journal Canadien des Sciences Neurologiques.

[12]  Abdulgadir Bugdadi,et al.  Virtual Reality Tumor Resection: The Force Pyramid Approach. , 2018, Operative neurosurgery.

[13]  J. Sweller.  COGNITIVE LOAD THEORY, LEARNING DIFFICULTY, AND INSTRUCTIONAL DESIGN , 1994 .

[14]  RossArun,et al.  Score normalization in multimodal biometric systems , 2005 .

[15]  James H. Block,et al.  1: Mastery Learning , 1976 .

[16]  Kewei Cheng,et al.  Feature Selection , 2016, ACM Comput. Surv..

[17]  L. Ladha,et al.  FEATURE SELECTION METHODS AND ALGORITHMS , 2011 .

[18]  K. Borgwardt,et al.  Machine Learning in Medicine , 2015, Mach. Learn. under Resour. Constraints Vol. 3.

[19]  Arun Ross,et al.  Score normalization in multimodal biometric systems , 2005, Pattern Recognit..

[20]  J. Ende.  Feedback in clinical medical education. , 1983, JAMA.

[21]  Nykan Mirchi,et al.  Machine Learning Identification of Surgical and Operative Factors Associated With Surgical Expertise in Virtual Reality Simulation. , 2019, JAMA network open.

[22]  R. Del Maestro,et al.  A Comparison of Visual Rating Scales and Simulated Virtual Reality Metrics in Neurosurgical Training: A Generalizability Theory Study. , 2019, World neurosurgery.

[23]  D. Fenner,et al.  Surgical skills assessment: a blinded examination of obstetrics and gynecology residents. , 2002, American journal of obstetrics and gynecology.

[24]  Yadong Wang,et al.  Improving fuzzy c-means clustering based on feature-weight learning , 2004, Pattern Recognit. Lett..

[25]  Nykan Mirchi,et al.  Artificial Intelligence in Medical Education: Best Practices Using Machine Learning to Assess Surgical Expertise in Virtual Reality Simulation. , 2019, Journal of surgical education.

[26]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[27]  Pheng-Ann Heng,et al.  Augmented Reality-Based Personalized Virtual Operative Anatomy for Neurosurgical Guidance and Training , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[28]  Robert DiRaddo,et al.  NeuroTouch: A Physics-Based Virtual Simulator for Cranial Microneurosurgery Training , 2012, Neurosurgery.

[29]  J. Pine,et al.  Chunking mechanisms in human learning , 2001, Trends in Cognitive Sciences.

[30]  Glenn Regehr,et al.  Slowing down when you should: a new model of expert judgment. , 2007, Academic medicine : journal of the Association of American Medical Colleges.

[31]  B. Karlik,et al.  Artificial Neural Networks to Assess Virtual Reality Anterior Cervical Discectomy Performance. , 2019, Operative neurosurgery.

[32]  Ben Williamson,et al.  Digital education governance: data visualization, predictive analytics, and ‘real-time’ policy instruments , 2016 .

[33]  Ahmed El-Henawy,et al.  A comparative Analytical Studies onAcaciapolyacantha gum Samples collected from three different locations in Sudan , 2014 .

[34]  Cristina Conati,et al.  AI in Education needs interpretable machine learning: Lessons from Open Learner Modelling , 2018, ArXiv.

[35]  Sri Ramakrishna,et al.  FEATURE SELECTION METHODS AND ALGORITHMS , 2011 .

[36]  David J. McArthur,et al.  The roles of artificial intelligence in education : current progress and future prospects: , 2005 .

[37]  Andre Esteva,et al.  A guide to deep learning in healthcare , 2019, Nature Medicine.

[38]  Ima Hajshirmohammadi,et al.  Using fuzzy set theory to objectively evaluate performance on minimally invasive surgical simulators , 2006 .

[39]  John McCarthy,et al.  A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence, August 31, 1955 , 2006, AI Mag..

[40]  R. Del Maestro,et al.  Development of a performance model for virtual reality tumor resections. , 2019, Journal of neurosurgery.

[41]  Sotiris B. Kotsiantis,et al.  Supervised Machine Learning: A Review of Classification Techniques , 2007, Informatica.

[42]  Karen Mann,et al.  Orienting Teaching Toward the Learning Process , 2004, Academic medicine : journal of the Association of American Medical Colleges.

[43]  Jo Herstad,et al.  Feedback from Digital Systems Used in Higher Education: An Inquiry into Triggered Emotions - Two Universal Design Oriented Solutions for a Better User Experience. , 2018, Studies in health technology and informatics.

[44]  Christopher R. Wagner,et al.  The Benefit of Force Feedback in Surgery: Examination of Blunt Dissection , 2007, PRESENCE: Teleoperators and Virtual Environments.

[45]  Ravindra Hegadi,et al.  A Systematic Review on Orthopedic Simulators for Psycho-Motor Skill and Surgical Procedure Training , 2018, Journal of Medical Systems.

[46]  Kwanjeera Wanichthanarak,et al.  Rise of Deep Learning for Genomic, Proteomic, and Metabolomic Data Integration in Precision Medicine , 2018, Omics : a journal of integrative biology.

[47]  Huan Liu,et al.  Efficient Feature Selection via Analysis of Relevance and Redundancy , 2004, J. Mach. Learn. Res..

[48]  Léon Bottou,et al.  Large-Scale Machine Learning with Stochastic Gradient Descent , 2010, COMPSTAT.

[49]  R. Del Maestro,et al.  Artificial Intelligence Distinguishes Surgical Training Levels in a Virtual Reality Spinal Task , 2019, The Journal of bone and joint surgery. American volume.

[50]  SchmidhuberJürgen.  Deep learning in neural networks , 2015 .