Combining Augmented Reality and 3D Printing to Display Patient Models on a Smartphone.

Augmented reality (AR) has great potential in education, training, and surgical guidance in the medical field. Its combination with three-dimensional (3D) printing (3DP) opens new possibilities in clinical applications. Although these technologies have grown exponentially in recent years, their adoption by physicians is still limited, since they require extensive knowledge of engineering and software development. Therefore, the purpose of this protocol is to describe a step-by-step methodology enabling inexperienced users to create a smartphone app, which combines AR and 3DP for the visualization of anatomical 3D models of patients with a 3D-printed reference marker. The protocol describes how to create 3D virtual models of a patient's anatomy derived from 3D medical images. It then explains how to perform positioning of the 3D models with respect to marker references. Also provided are instructions for how to 3D print the required tools and models. Finally, steps to deploy the app are provided. The protocol is based on free and multi-platform software and can be applied to any medical imaging modality or patient. An alternative approach is described to provide automatic registration between a 3D-printed model created from a patient's anatomy and the projected holograms. As an example, a clinical case of a patient suffering from distal leg sarcoma is provided to illustrate the methodology. It is expected that this protocol will accelerate the adoption of AR and 3DP technologies by medical professionals.

[1]  Nolan Ung,et al.  Utilizing virtual and augmented reality for educational and clinical enhancements in neurosurgery , 2017, Journal of Clinical Neuroscience.

[2]  Daniel K Sodickson,et al.  Patient-specific 3D printed and augmented reality kidney and prostate cancer models: impact on patient education , 2019, 3D Printing in Medicine.

[3]  Terry M. Peters,et al.  Training for Planning Tumour Resection: Augmented Reality and Human Factors , 2015, IEEE Transactions on Biomedical Engineering.

[4]  Samuel R. Barber,et al.  Augmented Reality, Surgical Navigation, and 3D Printing for Transcanal Endoscopic Approach to the Petrous Apex , 2018, OTO Open.

[5]  Eliot Siegel,et al.  Implementing Virtual and Augmented Reality Tools for Radiology Education and Training, Communication, and Clinical Care. , 2019, Radiology.

[6]  Kai Lawonn,et al.  Comparison of Projective Augmented Reality Concepts to Support Medical Needle Insertion , 2019, IEEE Transactions on Visualization and Computer Graphics.

[7]  Weiwei Deng,et al.  Easy-to-Use Augmented Reality Neuronavigation Using a Wireless Tablet PC , 2013, Stereotactic and Functional Neurosurgery.

[8]  KARYA SENI,et al.  Do it yourself , 1991, Nature.

[9]  Chaozheng Zhou,et al.  Robot-Assisted Surgery for Mandibular Angle Split Osteotomy Using Augmented Reality: Preliminary Results on Clinical Animal Experiment , 2017, Aesthetic Plastic Surgery.

[10]  Rubén Pérez-Mañanes,et al.  Three dimensional-printed patient-specific cutting guides for femoral varization osteotomy: Do it yourself. , 2017, The Knee.

[11]  A. Jones,et al.  American Academy of Otolaryngology-Head and Neck Surgery Foundation, Inc. , 2004 .

[12]  Javier Pascau,et al.  Augmented reality in computer-assisted interventions based on patient-specific 3D printed reference , 2018, Healthcare technology letters.

[13]  Graham Lawton,et al.  Through the HoloLens™ looking glass: augmented reality for extremity reconstruction surgery using 3D vascular models with perforating vessels , 2018, European Radiology Experimental.

[14]  A. Torbicki,et al.  Augmented reality and three-dimensional printing in percutaneous interventions on pulmonary arteries. , 2019, Quantitative imaging in medicine and surgery.

[15]  Hongen Liao,et al.  3D interactive surgical visualization system using mobile spatial information acquisition and autostereoscopic display , 2017, J. Biomed. Informatics.

[16]  Rubén Pérez-Mañanes,et al.  3D patient-specific surgical printing cutting blocks guides and spacers for open- wedge high tibial osteotomy (HTO) - do it yourself , 2016 .

[17]  Jan-Michael Frahm,et al.  Development of augmented‐reality applications in otolaryngology–head and neck surgery , 2019, The Laryngoscope.

[18]  D. C. Henckel,et al.  Case report. , 1995, Journal.

[19]  Hiroshi Kato,et al.  Utilizing a low-cost desktop 3D printer to develop a “one-stop 3D printing lab” for oral and maxillofacial surgery and dentistry fields , 2018, 3D Printing in Medicine.

[20]  Darwin G. Caldwell,et al.  Integrating Haptics with Augmented Reality in a Femoral Palpation and Needle Insertion Training Simulation , 2011, IEEE Transactions on Haptics.

[21]  P. Liacouras,et al.  Using 3D models in orthopedic oncology: presenting personalized advantages in surgical planning and intraoperative outcomes , 2018, 3D Printing in Medicine.