Single-Shot Deep Volumetric Regression for Mobile Medical Augmented Reality

Augmented reality for medical applications allows physicians to obtain an inside view into the patient without surgery. In this context, we present an augmented reality application running on a standard smartphone or tablet computer, providing visualizations of medical image data, overlaid with the patient, in a video see-through fashion. Our system is based on the registration of medical imaging data to the patient using a single 2D photograph of the patient. From this image, a 3D model of the patient’s face is reconstructed using a convolutional neural network, to which a pre-operative CT scan is automatically registered. For efficient processing, this is performed on a server PC. Finally, anatomical and pathological information is sent back to the mobile device and can be displayed, accurately registered with the live patient, on the screen. Hence, our cost-effective, markerless approach needs only a smartphone and a server PC for image processing. We present a qualitative and quantitative evaluation using real patient photos and CT from the clinical routine in facial surgery, reporting overall processing times and registration errors.

[1]  Mauricio Reyes,et al.  Comparison of actual surgical outcomes and 3-dimensional surgical simulations. , 2010, Journal of oral and maxillofacial surgery : official journal of the American Association of Oral and Maxillofacial Surgeons.

[2]  Ming Zhu,et al.  Precision of a Novel Craniofacial Surgical Navigation System Based on Augmented Reality Using an Occlusal Splint as a Registration Strategy , 2019, Scientific Reports.

[3]  Antonio Pepe,et al.  A Marker-Less Registration Approach for Mixed Reality–Aided Maxillofacial Surgery: a Pilot Evaluation , 2019, Journal of Digital Imaging.

[4]  Zhijian Song,et al.  A new markerless patient-to-image registration method using a portable 3D scanner. , 2014, Medical physics.

[5]  H. Lameckera,et al.  Automatic Segmentation of Mandibles in Low-Dose CT-Data , 2006 .

[6]  Tong Xi,et al.  Toward Holographic-Guided Surgery , 2018, Surgical innovation.

[7]  Christina Gsaxner,et al.  A review on multiplatform evaluations of semi-automatic open-source based image segmentation for cranio-maxillofacial surgery , 2019, Comput. Methods Programs Biomed..

[8]  Jan Egger,et al.  Development of a surgical navigation system based on augmented reality using an optical see-through head-mounted display , 2015, J. Biomed. Informatics.

[9]  Antonio Pepe,et al.  Markerless Image-to-Face Registration for Untethered Augmented Reality in Head and Neck Surgery , 2019, MICCAI.

[10]  Andrew Horowitz,et al.  Applications of 3-dimensional virtual computerized tomography technology in oral and maxillofacial surgery: current therapy. , 2010, Journal of oral and maxillofacial surgery : official journal of the American Association of Oral and Maxillofacial Surgeons.

[11]  Bostjan Likar,et al.  A review of 3D/2D registration methods for image-guided interventions , 2012, Medical Image Anal..

[12]  Christina Gsaxner,et al.  Facial model collection for medical augmented reality in oncologic cranio-maxillofacial surgery , 2019, Scientific Data.

[13]  Antonio Pepe,et al.  Depth-Awareness in a System for Mixed-Reality Aided Surgical Procedures , 2019, ICIC.

[14]  Shuo Yang,et al.  A practical marker-less image registration method for augmented reality oral and maxillofacial surgery , 2019, International Journal of Computer Assisted Radiology and Surgery.

[15]  Nico Blodow,et al.  Fast Point Feature Histograms (FPFH) for 3D registration , 2009, 2009 IEEE International Conference on Robotics and Automation.

[16]  Georgios Tzimiropoulos,et al.  Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[17]  Vladlen Koltun,et al.  Robust reconstruction of indoor scenes , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Arnold W. M. Smeulders,et al.  Interaction in the segmentation of medical images: A survey , 2001, Medical Image Anal..

[19]  Gérard G. Medioni,et al.  Object modelling by registration of multiple range images , 1992, Image Vis. Comput..

[20]  William E. Lorensen,et al.  Marching cubes: A high resolution 3D surface construction algorithm , 1987, SIGGRAPH.

[21]  David Black,et al.  A Novel Mixed Reality Navigation System for Laparoscopy Surgery , 2018, MICCAI.

[22]  N. Navab,et al.  Advanced Medical Displays: A Literature Review of Augmented Reality , 2008, Journal of Display Technology.

[23]  Jürgen Wallner,et al.  A Client/Server based Online Environment for the Calculation of Medical Segmentation Scores , 2019, 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[24]  R. Marmulla,et al.  Image-to-patient registration techniques in head surgery. , 2006, International journal of oral and maxillofacial surgery.

[25]  Michael Unser,et al.  Fast 3D reconstruction method for differential phase contrast X-ray CT. , 2016, Optics express.

[26]  M. Nagane,et al.  Smart Glasses for Neurosurgical Navigation by Augmented Reality. , 2018, Operative neurosurgery.

[27]  Jan Egger,et al.  Client/Server Based Online Environment for Manual Segmentation of Medical Images , 2019, ArXiv.

[28]  Dieter Schmalstieg,et al.  Pattern Recognition and Mixed Reality for Computer-Aided Maxillofacial Surgery and Oncological Assessment , 2018, 2018 11th Biomedical Engineering International Conference (BMEiCON).

[29]  Sven Behnke,et al.  Registration with the Point Cloud Library: A Modular Framework for Aligning in 3-D , 2015, IEEE Robotics & Automation Magazine.

[30]  Dieter Schmalstieg,et al.  Clinical evaluation of semi-automatic open-source algorithmic software segmentation of the mandibular bone: Practical feasibility and assessment of a new course of action , 2018, PloS one.

[31]  Jaesung Hong,et al.  Tracking Accuracy of a Stereo Camera-Based Augmented Reality Navigation System for Orthognathic Surgery. , 2019, Journal of oral and maxillofacial surgery : official journal of the American Association of Oral and Maxillofacial Surgeons.