HoloYolo: A proof‐of‐concept study for marker‐less surgical navigation of spinal rod implants with augmented reality and on‐device machine learning

Existing surgical navigation approaches of the rod bending procedure in spinal fusion rely on optical tracking systems that determine the location of placed pedicle screws using a hand‐held marker.

[1]  Eric W Nottmeier,et al.  Timing of Paired Points and Surface Matching Registration in Three-Dimensional (3D) Image-guided Spinal Surgery , 2007, Journal of spinal disorders & techniques.

[2]  Juan Liu,et al.  Real-time mixed-reality telepresence via 3D reconstruction with HoloLens and commodity depth sensors , 2017, ICMI.

[3]  Pascal Fua,et al.  Simultaneous Recognition and Pose Estimation of Instruments in Minimally Invasive Surgery , 2017, MICCAI.

[4]  Naoki Ishiguro,et al.  Epidemiological trends in spine surgery over 10 years in a multicenter database , 2018, European Spine Journal.

[5]  Giacomo Placella,et al.  SURGICAL TREATMENT , 1947 .

[6]  Dieter Fox,et al.  PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes , 2017, Robotics: Science and Systems.

[7]  Florentin Liebmann,et al.  Augmented Reality Navigation for Spinal Pedicle Screw Instrumentation using Intraoperative 3D Imaging. , 2020, The spine journal : official journal of the North American Spine Society.

[8]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Klaus Schöffmann,et al.  Learning laparoscopic video shot classification for gynecological surgery , 2018, Multimedia Tools and Applications.

[10]  Manuel Moser,et al.  Accuracy of patient-specific template-guided vs. free-hand fluoroscopically controlled pedicle screw placement in the thoracic and lumbar spine: a randomized cadaveric study , 2017 .

[11]  Kanako Harada,et al.  Single-Shot Pose Estimation of Surgical Robot Instruments’ Shafts from Monocular Endoscopic Images , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[12]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[13]  Sergei Vassilvitskii,et al.  k-means++: the advantages of careful seeding , 2007, SODA '07.

[14]  Florian Richter,et al.  SuPer Deep: A Surgical Perception Framework for Robotic Tissue Manipulation using Deep Learning for Feature Extraction , 2020, 2021 IEEE International Conference on Robotics and Automation (ICRA).

[15]  W. Dhert,et al.  Surgical Treatment of Traumatic Fractures of the Thoracic and Lumbar Spine: A Systematic Review of the Literature on Techniques, Complications, and Outcome , 2004, Spine.

[16]  Ali Farhadi,et al.  YOLO9000: Better, Faster, Stronger , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Sohail K. Mirza,et al.  Trends in Lumbar Fusion Procedure Rates and Associated Hospital Costs for Degenerative Spinal Diseases in the United States, 2004 to 2015 , 2019, Spine.

[18]  Danail Stoyanov,et al.  Articulated Multi-Instrument 2-D Pose Estimation Using Fully Convolutional Networks , 2018, IEEE Transactions on Medical Imaging.

[19]  Florentin Liebmann,et al.  Augmented reality-assisted rod bending in spinal surgery. , 2019, The spine journal : official journal of the North American Spine Society.

[20]  Nassir Navab,et al.  Deep Residual Learning for Instrument Segmentation in Robotic Surgery , 2017, MLMI@MICCAI.

[21]  Guang-Zhong Yang,et al.  Self-Supervised Siamese Learning on Stereo Image Pairs for Depth Estimation in Robotic Surgery , 2017, ArXiv.

[22]  Р Ю Чуйков,et al.  Обнаружение транспортных средств на изображениях загородных шоссе на основе метода Single shot multibox Detector , 2017 .

[23]  Michael Söderman,et al.  Pedicle Screw Placement Using Augmented Reality Surgical Navigation With Intraoperative 3D Imaging , 2018, Spine.

[24]  Didier Mutter,et al.  Weakly supervised convolutional LSTM approach for tool tracking in laparoscopic videos , 2018, International Journal of Computer Assisted Radiology and Surgery.

[25]  Clifford B Tribus,et al.  Degenerative Lumbar Scoliosis: Evaluation and Management , 2003, The Journal of the American Academy of Orthopaedic Surgeons.

[26]  Davide Scaramuzza,et al.  Pedicle screw navigation using surface digitization on the Microsoft HoloLens , 2019, International Journal of Computer Assisted Radiology and Surgery.

[27]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Antoine Tohmeh,et al.  Long Construct Pedicle Screw Reduction and Residual Forces are Decreased Using a Computer-Assisted Spinal Rod Bending System , 2014 .

[29]  Ève Coste-Manière,et al.  3D reconstruction of the operating field for image overlay in 3D-endoscopic surgery , 2001, Proceedings IEEE and ACM International Symposium on Augmented Reality.

[30]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[31]  Berthold K. P. Horn,et al.  Closed-form solution of absolute orientation using orthonormal matrices , 1988 .

[32]  Mathias Unberath,et al.  CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer-Assisted Interventions , 2019, Proceedings of the IEEE.

[33]  René Schmidt,et al.  Cervical Pedicle Screws: Conventional Versus Computer-Assisted Placement of Cannulated Screws , 2005, Spine.

[34]  Zeng-Guang Hou,et al.  RASNet: Segmentation for Tracking Surgical Instruments in Surgical Videos Using Refined Attention Segmentation Network , 2019, 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[35]  Thomas Probst,et al.  Automatic Tool Landmark Detection for Stereo Vision in Robot-Assisted Retinal Surgery , 2018, IEEE Robotics and Automation Letters.

[36]  R. Deyo,et al.  Systematic Review of Randomized Trials Comparing Lumbar Fusion Surgery to Nonoperative Care for Treatment of Chronic Back Pain , 2007, Spine.

[37]  Alexander Rakhlin,et al.  Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning , 2018, bioRxiv.

[38]  Yingying Guo,et al.  Surgical Navigation in Orthopedics: Workflow and System Review. , 2018, Advances in experimental medicine and biology.

[39]  Nassir Navab,et al.  Towards clinical translation of augmented orthopedic surgery: from pre-op CT to intra-op x-ray via RGBD sensing , 2018, Medical Imaging.

[40]  Katsushi Takeshita,et al.  Surgical treatment of scoliosis: a review of techniques currently applied , 2008, Scoliosis.

[41]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.