Twin-S: A Digital Twin for Skull-base Surgery

PURPOSE Digital twins are virtual replicas of real-world objects and processes, and they have potential applications in the field of surgical procedures, such as enhancing situational awareness. We introduce Twin-S, a digital twin framework designed specifically for skull base surgeries. METHODS Twin-S is a novel framework that combines high-precision optical tracking and real-time simulation, making it possible to integrate it into image-guided interventions. To guarantee accurate representation, Twin-S employs calibration routines to ensure that the virtual model precisely reflects all real-world processes. Twin-S models and tracks key elements of skull base surgery, including surgical tools, patient anatomy, and surgical cameras. Importantly, Twin-S mirrors real-world drilling and updates the virtual model at frame rate of 28. RESULTS Our evaluation of Twin-S demonstrates its accuracy, with an average error of 1.39 mm during the drilling process. Our study also highlights the benefits of Twin-S, such as its ability to provide augmented surgical views derived from the continuously updated virtual model, thus offering additional situational awareness to the surgeon. CONCLUSION We present Twin-S, a digital twin environment for skull base surgery. Twin-S captures the real-world surgical progresses and updates the virtual model in real time through the use of modern tracking technologies. Future research that integrates vision-based techniques could further increase the accuracy of Twin-S.

[1]  P. Kazanzides,et al.  Fully Immersive Virtual Reality for Skull-base Surgery: Surgical Training and Beyond , 2023, ArXiv.

[2]  Russell H. Taylor,et al.  TAToo: Vision-based Joint Tracking of Anatomy and Tool for Skull-base Surgery , 2022, ArXiv.

[3]  Weixin Si,et al.  Synergistic Digital Twin and Holographic Augmented-Reality-Guided Percutaneous Puncture of Respiratory Liver Tumor , 2022, IEEE Transactions on Human-Machine Systems.

[4]  Clayton D. Scott,et al.  IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Ken Goldberg,et al.  A Digital Twin Framework for Telesurgery in the Presence of Varying Network Quality of Service , 2022, 2022 IEEE 18th International Conference on Automation Science and Engineering (CASE).

[6]  Rahuman S. Malik-Sheriff,et al.  Building digital twins of the human immune system: toward a roadmap , 2022, npj Digital Medicine.

[7]  P. Kazanzides,et al.  CaRTS: Causality-driven Robot Tool Segmentation from Vision and Kinematics Data , 2022, MICCAI.

[8]  P. Nakaji,et al.  Current status of augmented reality in cerebrovascular surgery: a systematic review , 2022, Neurosurgical Review.

[9]  Peter Kazanzides,et al.  Virtual reality for synergistic surgical training and data generation , 2021, Comput. methods Biomech. Biomed. Eng. Imaging Vis..

[10]  Russell H. Taylor,et al.  On the Sins of Image Synthesis Loss for Self-supervised Depth Estimation , 2021, ArXiv.

[11]  M. Holland,et al.  Hidden Error in Optical Stereotactic Navigation Systems and Strategy to Maximize Accuracy , 2021, Stereotactic and Functional Neurosurgery.

[12]  Rashid M. Yasin,et al.  Evaluation of Hybrid Control and Palpation Assistance for Situational Awareness in Telemanipulated Task Execution , 2021, IEEE Transactions on Medical Robotics and Bionics.

[13]  Xingtong Liu,et al.  Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformers , 2020, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[14]  Jan Egger,et al.  A review on the applications of virtual reality, augmented reality and mixed reality in surgical simulation: an extension to different kinds of surgery , 2020, Expert review of medical devices.

[15]  R. Beck,et al.  A Randomized Trial of Closed-Loop Control in Children with Type 1 Diabetes. , 2020, The New England journal of medicine.

[16]  Russell H. Taylor,et al.  Anatomical Mesh-Based Virtual Fixtures for Surgical Robots* , 2020, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[17]  Jason Yon,et al.  Characterising the Digital Twin: A systematic literature review , 2020, CIRP Journal of Manufacturing Science and Technology.

[18]  Manoel Jacobsen Teixeira,et al.  Augmented reality and physical hybrid model simulation for preoperative planning of metopic craniosynostosis surgery. , 2020, Neurosurgical focus.

[19]  Mika Gustafsson,et al.  Digital twins to personalize medicine , 2019, Genome Medicine.

[20]  Russell H. Taylor,et al.  Image-Guided Mastoidectomy with a Cooperatively Controlled ENT Microsurgery Robot , 2019, Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery.

[21]  Abdulmotaleb El-Saddik,et al.  Cardio Twin: A Digital Twin of the human heart running on the edge , 2019, 2019 IEEE International Symposium on Medical Measurements and Applications (MeMeA).

[22]  Heikki Laaki,et al.  Prototyping a Digital Twin for Real Time Remote Control Over Mobile Networks: Application of Remote Surgery , 2019, IEEE Access.

[23]  Russell H. Taylor,et al.  Preliminary Evaluation of an Online Estimation Method for Organ Geometry and Tissue Stiffness , 2018, IEEE Robotics and Automation Letters.

[24]  Peter Kazanzides,et al.  Force-Controlled Exploration for Updating Virtual Fixture Geometry in Model-Mediated Telemanipulation , 2017 .

[25]  Hannes Sommer,et al.  Evaluation of Combined Time-Offset Estimation and Hand-Eye Calibration on Robotic Datasets , 2017, FSR.

[26]  Russell H. Taylor,et al.  Accuracy assessment and kinematic calibration of the robotic endoscopic microsurgical system , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[27]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[28]  Russell H. Taylor,et al.  Concurrent nonparametric estimation of organ geometry and tissue stiffness using continuous adaptive palpation , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[29]  Ziv Yaniv,et al.  Which pivot calibration? , 2015, Medical Imaging.

[30]  R. Agha,et al.  The role and validity of surgical simulation. , 2015, International surgery.

[31]  Luis Serra,et al.  DEX‐RAY: AUGMENTED REALITY NEUROSURGICAL NAVIGATION WITH A HANDHELD VIDEO PROBE , 2009, Neurosurgery.

[32]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[33]  Michel Dhome,et al.  Hand-eye calibration , 1997, Proceedings of the 1997 IEEE/RSJ International Conference on Intelligent Robot and Systems. Innovative Robotics for Real-World Applications. IROS '97.

[34]  Fadi Dornaika,et al.  Hand-Eye Calibration , 1995, Int. J. Robotics Res..

[35]  Paul S. Schenker,et al.  Sensor fusion IV: Control paradigms and data structures; Proceedings of the Meeting, Boston, MA, Nov. 12-15, 1991 , 1992 .

[36]  Gérard G. Medioni,et al.  Object modeling by registration of multiple range images , 1991, Proceedings. 1991 IEEE International Conference on Robotics and Automation.

[37]  Peter Kazanzides,et al.  An Image-directed Robotic System For Precise Orthopaedic Surgery , 1990, [1990] Proceedings of the Twelfth Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[38]  W. Kabsch.  A solution for the best rotation to relate two sets of vectors , 1976 .